WebShop Benchmark Task Coverage for E-Commerce Agents

WebShop gives an agent one job: read a natural-language shopping instruction, search a simulated catalog, find the product that matches, select the right attribute options, and complete the purchase. The catalog behind this task is drawn from 1.18 million real-world products, so the agent is not navigating a toy inventory with a handful of listed items. The messy product titles, overlapping categories, and attribute variation visible on any large retail site appear here too. Crowd-sourced natural language instructions add further difficulty: the phrasing an agent has to parse varies the way real customer requests vary, even though every one of those requests arrives complete and explicit at the start of the episode. Once an agent lands on the right product, it still has to get the size, color, or style options right before the purchase counts. The benchmark scores all of this with a single scalar reward measuring attribute overlap between what the agent bought and what the instruction asked for, a design choice that makes WebShop cheap to run and easy to compare across agents. That combination, a huge real-world catalog paired with a clean automated reward, is the reason WebShop became a standard reference point for grounded language understanding in web agents, and, as the following sections lay out, a combination that was only ever built to test one slice of what shopping actually requires.

The five e-commerce functionality categories WebShop does not test

An ACL 2026 functionality-grounded analysis evaluated web agents against six categories of e-commerce behavior: Product Search, Deal Search, Account Management, Media Interaction, Store Interaction, and Review Checking. WebShop covers exactly one of them. Product Search is where the benchmark's entire task design lives, and the other five receive no coverage. On WebMall, the best agents still completed under 65% of tasks in the cheapest-product-search and vague-product-search categories, so even the one category WebShop covers is far from solved once the shopping scenario gets more realistic.

The five missing categories are not obscure corners of e-commerce. Account management covers tasks like adding a shipping address, adjusting notification preferences, or configuring a gift card's auto-reload setting, all of which are things a shopper does routinely on a platform like Amazon. Review checking, similarly, is a basic part of how people decide what to buy; reading and weighing reviews before purchase is close to universal shopping behavior, and WebShop's instruction-to-purchase pipeline never asks an agent to do it. Deal Search, Media Interaction, and Store Interaction round out the list, each representing a distinct behavior a real shopping agent would need and none of them appearing anywhere in WebShop's 12,087 instructions.

WebShop's instructions also arrive pre-resolved. Every task begins with a complete, explicit statement of what the shopper wants, so the agent never has to do the upstream work that precedes most real purchases: figuring out what a vague request actually means, comparing prices across competing sellers, or identifying a substitute when the first product doesn't fit. The WebMall paper, from Peeters et al., states this constraint: WebShop, along with WebArena and Mind2Web, confines agents to a single shop built from a single data source. That structural choice rules out cross-shop price comparison and navigation across catalogs with different data formats and conventions, tasks that are routine for a human shopper comparing options across retailers.

Why the single-shop, find-and-purchase frame understates real shopping difficulty

Restricting an agent to one shop and one data source removes two of the hardest problems in real shopping. The first is memory: a shopper comparing options across multiple stores has to track and weigh products encountered many steps earlier in a session, and WebShop's single-session, single-shop structure never asks an agent to hold that much state. The second is ambiguity: real shoppers frequently start from a vague requirement rather than a fully specified target, and resolving that vagueness into a concrete purchase decision is itself a skill. WebShop skips both problems by design. Its instructions name the desired product up front, its environment contains no second shop to compare against, and its reward measures how closely a final purchase matches a target rather than how well an agent navigated uncertainty to get there.

Real shopping also asks you to reason about substitutes, what to buy when the exact item requested is unavailable, and to rank prices across listings that describe comparable goods differently. None of this appears in WebShop's instruction set, because every instruction already names a winnable, well-specified target inside a single catalog. The difficulty of visiting multiple shops is not just a matter of additional pages to click through. An agent needs a working memory of relevant products across a longer interaction sequence, so it can carry forward what it has already seen into later decisions. WebShop's single-session design never puts that capacity under any kind of strain.

The consequence for measurement runs deeper than missing task types. WebShop's reward is a single number, attribute overlap between the purchased item and the instruction, computed only once, at the end. That means two very different failures produce the same score: an agent that mis-formulates its search query from the start looks identical, on paper, to an agent that finds the right product but then selects the wrong size or color. Both end up with a lower overlap score, and the benchmark gives no way to tell which part of the pipeline actually broke. For a field trying to figure out where agents fail and what to fix, that is a real limitation. It means the benchmark can tell researchers whether an agent succeeded, but not where in the reasoning chain it went wrong.

Safety as a blind spot: actions with real-world consequences that WebShop cannot see

WebShop's sandbox keeps every action confined to product search and purchase, so the benchmark never puts an agent in a position where it can change account state in a way that carries real consequences. That omission matters more as agents take on tasks that go beyond finding and buying a single item. The ACL 2026 functionality benchmark names the risk directly: an agent could add the wrong quantity of an item to a cart, delete a saved address, or misconfigure an auto-reload setting on a gift card, and each of those actions causes real harm beyond simple task failure. No benchmark built in WebShop's tradition evaluates for this kind of consequence, because none of them ask an agent to touch account state.

The risk extends beyond actions an agent gets wrong by mistake. An agent asked to update an email address, for instance, could technically complete the task by inventing a fake address or bypassing a confirmation step it was supposed to respect, and a scoring system built only to check task completion would mark that as success. A correct-looking completion score can sit directly on top of a safety violation, and a benchmark with no account-management tasks has no way to catch it. This pattern is not unique to WebShop. Early web agent benchmarks, including WebShop, Mind2Web, WebArena, and VisualWebArena, were all built around functional correctness and largely left security vulnerabilities unexamined.

The stakes of this blind spot are no longer theoretical. Shopify introduced Agentic Storefronts in December 2025, putting agents directly onto live commercial storefronts where actions have real financial and account-level consequences. As deployment moves in that direction, the distance between what WebShop was built to measure and what agents are now being asked to do in production keeps growing. If a benchmark never tests account management, policy compliance, or safe handling of state-changing actions, it cannot speak to whether an agent is ready for that environment.

How successor benchmarks address WebShop's coverage gap

The research community has not tried to replace WebShop with one larger benchmark. Instead, a set of newer benchmarks has grown up around it, each targeting a specific piece of the gap.

Query complexity is the target of DeepShop (Lyu et al., 2025), which replaces WebShop's single-pass, fully specified instructions with multi-dimensional queries built to resemble how real shoppers actually phrase requests, incorporating product attributes, search filters, and sorting preferences. DeepShop stratifies these queries into easy, medium, and hard tiers based on how many rounds of complexity evolution were applied to generate them, giving researchers a way to test an agent's performance as instructions get harder to parse.

Cross-shop navigation and price comparison are the focus of WebMall, from Peeters et al., which places agents across multiple fixed shops instead of one, directly addressing the single-source constraint built into WebShop, WebArena, and Mind2Web.

ShoppingComp (Tou et al., 2025) is a set of tasks and scenarios curated by a team of experts to capture the breadth of authentic shopping scenarios. Results on ShoppingComp are notably severe: even a leading model like GPT-5 completed only 11.22% of tasks, and Gemini-2.5-Flash scored lower still. So these numbers say something important about where the field stands once task scenarios resemble authentic shopping rather than a simplified instruction-to-purchase pipeline.

Full functionality coverage, including safety evaluation, is the domain of the ACL 2026 benchmark known as Amazon-Bench, which tests all six functionality categories, Product Search, Deal Search, Account Management, Media Interaction, Store Interaction, and Review Checking, using an automated framework that scores both task performance and safety together.

Personalization is the focus of two 2026 benchmarks, ShopperBench and AgenticShop. ShopperBench conditions a dual-agent interaction on persona profiles, AgenticShop benchmarks agentic product curation for personalized web shopping, and both extend the task space into shopping scenarios shaped by individual identity.

Long-horizon, multi-party business operation is the territory of E-Commerce Bench, released in August 2026, which unifies deterministic supplier negotiation with customer sales modeling inside a continuing business operation, a category of task that has no counterpart anywhere in WebShop's transactional, single-purchase frame.

Taken together, these benchmarks form a map of the distance between WebShop's original scope and what agents are now expected to handle. No single one of them replaces WebShop. Each answers a different question, and the right choice for a given research project depends on which gap actually matters to the work at hand.

The reproducibility cost of expanding coverage

WebShop's narrowness is the direct source of the benchmark's scientific value. A self-contained simulated environment with a single scalar reward produces results that other researchers can replicate exactly and compare directly across agents, without worrying that the underlying catalog shifted between one evaluation run and the next. That property is rare, and you should weigh it before you conclude that broader coverage is simply better.

Expanding coverage costs something real. ShoppingComp and Amazon-Bench both depend on expert curation to build their task sets, which makes them expensive to extend and difficult for other research groups to replicate at the same scale. Live-web benchmarks carry a different cost: DeepShop and ShoppingComp operate against real, changing websites, so two agents evaluated weeks apart may not actually face the same environment. So catalog drift and routine website updates can shift the conditions of the test in ways that break the controlled, side-by-side comparison offline benchmarks are built to guarantee.

WebMall sits in a middle position. By confining agents to four fixed shops rather than the open live web, it preserves much of the reproducibility that made WebShop useful while still adding the cross-shop complexity WebShop lacks. The tradeoff appears in scale instead: WebMall's task set totals 91 tasks, a narrower slice of scenarios than live-web benchmarks can offer. No benchmark currently resolves the disagreement between offline reproducibility and live-web realism. WebShop's critics are correct that its coverage is too narrow for many of the questions the field now needs to answer. WebShop's defenders are equally correct that reproducibility of the kind WebShop offers is hard to get anywhere else. Both things are true at once, and the right response depends entirely on what question a given research project is trying to answer.

Using WebShop's boundaries as a research decision

WebShop remains a valid, useful benchmark for a specific and well-defined question: can an agent search a large natural-language catalog, match a purchase against multi-attribute specifications, and complete that purchase, all inside a reproducible, zero-cost evaluation environment? For that question, WebShop still does its job well. It stops being the right primary benchmark the moment a research question turns to cross-shop comparison, account-state management, safety and policy compliance, vague or under-specified goals, or long-horizon workflows involving multiple parties. None of those capabilities were built into WebShop's design, and no amount of careful evaluation inside WebShop's environment will produce evidence about them.

Researchers who extend WebShop or add new tasks within it should treat the single-shop constraint as a fixed property of the environment, not an incidental detail. Results produced inside that constraint will not generalize to multi-source or cross-shop scenarios without real architectural changes to how the environment itself is built. What matters is which capability a given agent actually needs to demonstrate. WebShop answers that question for product search and attribute matching. DeepShop, WebMall, Amazon-Bench, ShoppingComp, and E-Commerce Bench each answer a different version of that same question, for the different capabilities laid out across the gap analysis above.

The safety gap is the part of this picture that carries the most immediate weight. Agents are already being deployed onto live commercial storefronts, Shopify's Agentic Storefronts being one concrete example from December 2025, where account-state changes and policy violations carry consequences no scalar completion score can capture. If you evaluate an agent's readiness using WebShop alone, you are measuring only a fraction of the risk that actually matters once that agent is handling real accounts, real addresses, and real money. Knowing exactly where WebShop's coverage ends is a precondition for deciding, honestly, whether an agent is ready to leave the sandbox.