AI Agentic Shopping Not Yet Delivering the Goods: Study

A new benchmark study that puts leading AI models through realistic buying challenges finds such apps are struggling to complete that task. The research, by credit-card giant Visa, tested whether AI could uncover customer needs and make an informed purchase decision. The result: the models failed 42.8% of such test runs.

“Shopping is a difficult task for AI because consumers often do not clearly know what they want at the outset—for example, they may know they want a TV, but not all the required specifications,” Visa’s summary of the test notes. “Under Visa’s new benchmark, AI models are required to identify the needs of a consumer based on a conversation. If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.”

The first round of tests involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a TV, laptop, or personal care appliance.

The results:

• There’s major room for improvement: Across testing, selected AI models achieved an overall benchmark pass rate of about half (57.2%). The best performing models passed 82.2% of attempts, the worst 28.9%.
• The more you know: The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a 70.2% pass rate. The five that uncovered the least passed less than half (44.4%) of the time.
• Pricy models win, but do not guarantee success: The five models with the highest average cost per run passed 67.1% of runs, compared with 50.2% for the five least expensive.
• Consistency is an issue: Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time.
• Buyer beware: About 1-in-2 (54%) of failed runs still ended in the AI making a purchase. Either the AI recommended a purchase that did not meet the customer’s identified requirements, or it failed to uncover all the customer’s requirements.

“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. “

Visa said plans to add other shopping scenarios and product categories to Visa Agentic Commerce Benchmark, and will extend testing to checkout, payment, and order placement. Visa has shared the benchmark with leading model providers for feedback.