Visa’s shopping test shows why AI agents need to know when not to buy

A helpful shopping assistant needs to recognise when no available product fits.

What happened

Visa evaluated 22 AI models from 12 labs across 15 shopping scenarios. The overall pass rate was 57.2%, meaning 42.8% of test runs failed. In 54% of failed runs, the model still recommended a purchase despite an unsuitable choice or incomplete requirements.

These are results from a benchmark, not a measured failure rate for real customer transactions.

Why it matters

An agent needs to uncover the buyer’s requirements before deciding what to recommend. A model that confidently picks something too early can look useful while missing the purpose of the task.

The bigger picture

Payment infrastructure alone will not make automated shopping dependable. Developers also need to test the decisions leading up to checkout, including whether an agent asks enough questions and can stop without buying.