AI Shoppers Failed 42.8% of Visa's Test, and Often Still Bought

The Visa Agentic Commerce Benchmark, released on 5 November 2026, tested 22 AI models from 12 labs as shopping assistants, and their runs failed 42.8% of the time. In 54% of the failed runs, the shopper still ended up buying something that missed a requirement. The failures happen early, in discovery: the models often do not ask the question that would reveal what the shopper actually needs.

Visa's announcement gives the headline numbers. The model-by-model results sit on Visa's benchmark site, labelled an "October 2026 snapshot". This analysis draws on both, including the published results file behind the site's charts.

What Is the Visa Agentic Commerce Benchmark?

The Visa Agentic Commerce Benchmark is a Visa Research test of whether an AI shopping assistant can work out what a shopper needs, search a market and recommend the right product. Each test, which Visa calls an episode, is a conversation between two AI models: one plays the shopper, the other is the shopping assistant being tested.

The shopper starts with a few stated preferences and holds the rest back. A hidden fact comes out only when the assistant asks a relevant question or takes an action that reveals it. The assistant searches a simulated market that is identical for every model, and each run is scored against an answer key no model sees.

Visa publishes one episode in full. A shopper called Sam needs a 65-inch TV with at least 30-day returns. Questioning reveals a budget of up to $1,200 all in, and checking with the family adds Dolby Vision, HDR10+ and native 120 Hz for an Xbox Series X. Sam's market holds 265 TVs; each requirement found narrows it, down to 20. Visa has made this episode playable as an early demo, so readers can question Sam and pick a TV themselves. The other 14 episodes stay private. According to the release, the tasks cover TVs, laptops and personal care appliances.

What counts as a pass?

A run passes only if the decision is right and the assistant earned it. In Visa's words, a purchase passes "when the product meets every requirement in the answer key and each requirement was established during the run." Declining to buy passes only when nothing on the market fits. A lucky fit does not pass, and a run that hits the tool-call limit or returns a reply in the wrong format counts as a failure.

Every model ran each of the 15 episodes three times, giving 990 runs in total.

What Did Visa's AI Shopping Benchmark Find?

Source: Visa release and benchmark site, 5 November 2026.

Two numbers matter most. The discovery gap (70.2% vs 44.4%) shows that asking better questions is what separates the stronger models. The consistency gap (57.2% per run, 34.2% for all three runs) shows that one good run proves little. Visa's site adds that no model passed all three runs on more than 67% of the episodes.

Which AI Model Is Best at Shopping?

On pass rate, Gemini 4 Argon from Google and Muse Wonder from Meta tied at the top with 82.2%, 37 of 45 runs each. Visa itself cautions against reading this as a clear winner: "No model clearly leads." Each rate carries a wide 95% confidence interval, and Visa says the intervals of the next eleven models overlap the leaders'.

Source: Visa benchmark results file, October 2026 snapshot (cost is what the tested model spent per run, as its provider reported it).

Same score, three times the cost

The tie at the top hides a cost gap. Gemini 4 Argon averaged $0.54 per run and Muse Wonder $1.55, so Visa's cost-per-pass figure is about $0.66 for Argon against $1.88 for Wonder. At the other end, GPT-6 Luna passed 53.3% of runs for about $0.02 each. For a merchant or platform choosing a model to run shopping conversations at volume, cost per correct outcome is the number to compare, and the leaderboard alone does not show it.

Brand is not a proxy for performance

Results vary widely inside a single lab. Google's models span 82.2% (Gemini 4 Argon) to 35.6% (Gemini 3.1 Pro). Anthropic's Claude Sonnet 5.5 scored 64.4% at $0.39 a run, slightly ahead of Claude Opus 5.5 at 62.2%. "We tested Gemini" or "we use Claude" says little about shopping quality until the exact model is named.

How Do AI Shopping Agents Fail?

Visa found that models fail in different ways, and the way a model fails matters as much as how often.

• Declining when a fit existed. OpenAI's models end the most runs with no purchase although a suitable product was on the market: GPT-6.1 Terra in 38% of runs, GPT-6.1 Sol in 33% and GPT-6 Astra in 29%. Visa says no model from another lab exceeds 16%.
• Buying the wrong thing. Gemini 3.1 Pro made a purchase that did not pass in 51% of its runs.
• Running out of road. MiniMax M3.1 Flash hit the tool-call limit in 38% of runs, and Inkling's reply was not in the accepted format in 36%.

Cautious agents lose sales; eager agents create returns

The split between declining and buying wrong is the most practical finding in the data, because the two failures cost a merchant in different ways. Visa's results file shows the contrast clearly. All 11 of Claude Fable 5.1's failed runs were purchases that missed a requirement; it never declined a suitable product or ran out of tool calls. GPT-6.1 Sol failed the other way: 15 of its 17 failed runs ended with no purchase although a fit existed, and only 2 were wrong purchases.

An eager agent that buys the wrong product turns into a return, a refund or a complaint. A cautious agent that wrongly says nothing fits turns into a sale that never happens, and the shopper may never know a suitable product was available. Pass rate alone treats both as the same failure. The benchmark also leaves open who bears the cost when an agent's wrong purchase becomes a dispute, because its current stage stops before payment.

Why one good run proves little

Visa measures reliability with pass^k, a method it credits to the τ-bench research benchmark: the chance that k runs drawn at random from the same episode all pass, averaged over the 15 episodes. Across the 22 models, a single run passed 57% of the time, two runs of the same episode both passed 42% of the time, and all three 34%. Visa's whitepaper, as summarised on its site, counts a capability as demonstrated only when it holds up across runs. For anyone deploying a shopping agent, that is the right standard: shoppers experience every run, not the average.

Visa Research's diagnosis is precise: capable models "seldom misinterpret a fact once it's in their context." The failures come earlier. Visa lists five patterns the episodes are built to catch:

1. Meeting every stated requirement while never asking about unstated ones.
2. Treating a value missing from a product listing as acceptable.
3. Saying no suitable product exists while one sits unexamined in an earlier search result.
4. Talking the shopper out of a correct choice over a concern no requirement supports.
5. Agreeing on a bundle in conversation, then finalising a selection without it.

What Does This Mean for Merchants?

The common reading of agentic commerce puts the risk at checkout: will the payment go through, is the agent authorised? Visa's data points somewhere else. The weak step in this benchmark is discovery, before any payment happens, and 54% of failures still ended in a sale. For a merchant, that means a share of agent-driven orders could arrive for products that do not fit the buyer, with the returns and service cost that follow. This is an inference from a simulated market, not measured merchant data.

Failure pattern 2 is the one merchants can act on. If an agent treats a missing attribute as acceptable, a listing without a stated return window, HDR format or refresh rate can be bought on a false assumption or passed over in favour of a competitor's complete listing. Complete, explicit product attributes are the part of this problem a merchant controls. Visa's benchmark measures model behaviour, not listing quality, so this is a reading of its failure patterns rather than a tested result.

Shopper trust is the other constraint. Visa's Trust Index, cited in the release, found that 72% of US consumers had used an AI assistant but only 23% trusted AI to handle payments on their behalf; that survey is covered in the agentic checkout trust gap.

What Has Visa Not Tested Yet?

The October 2026 snapshot covers discovery and intent only. Visa says it will extend testing to "checkout, payment, and order placement" and add more scenarios and product categories, with post-checkout listed as a later stage on its site. Three limits apply to the current results:

• Fourteen of the fifteen episodes are private, so outsiders can inspect only Sam's TV.
• The market is simulated and the shopper is an AI model, not a person.
• Visa runs a card network with its own agentic commerce products. Its release says the benchmark is not directly part of Visa Intelligent Commerce, and that it has been shared with leading model providers for feedback.

Rubail Birwadker, Visa's Global Head of Growth, framed the release around trust: "AI agents have the potential to power the next era of commerce, but trust is key to making it work." He was blunter with Fortune: "We're not there yet from a trust or technology perspective."

Most agentic commerce debate so far has been about whether an agent is allowed to spend. Visa's data shifts the question to whether it knows what to buy. On the October 2026 snapshot, the best models get that right about four times in five, and none does so reliably across repeated runs.