- Of twenty large US storefronts probed on September 13, 2026, eighteen served a robots.txt file to a browser.
- Fifteen of the eighteen disallow at least one cart, checkout, basket or add-to-bag path to the wildcard group.
- Ten of the eighteen name no AI agent anywhere, so an agent inherits the crawler rules.
- OpenAI documents that for user-triggered fetches, robots.txt rules may not apply. [1]
- Six of the twenty served a desktop browser and refused the published agent string minutes later.
- The agent feed carries price and availability, and nothing in it expires by itself. [2]
- Cloudflare separates search, agent and training behavior. Most merchants have chosen none of the three. [4]
The argument about shopping agents is usually conducted at the level of strategy, which is a shame, because most of it can be settled with a text file and two HTTP requests. So we made them. Twenty large US storefronts, on one morning, asked the two questions an agent would ask before it could ever buy anything: what does your robots.txt permit, and will you answer me at all?
The method, so you can repeat it and disagree with it. We requested /robots.txt from each site with a desktop Chrome user agent, parsed the User-agent: * group, and counted disallow rules whose path contains a buying segment. Order history was not counted, because reading past orders is not buying. Then we asked each homepage for a page twice, once with the same browser string and once with the ChatGPT-User string OpenAI publishes. Two storefronts refused the robots.txt request outright, which is why the denominators differ.
This is one connection on one day, and bot management is probabilistic. Your own result will differ, which is the point: the only version of this measurement worth acting on is the one you run against your own domain.
The line in the file is a preference
Fifteen storefronts tell the default crawler group to stay out of the cart. Read that as intent and it sounds decisive. Read the vendor documentation and it stops being decisive at all.
OpenAI publishes three named agents and describes them differently. GPTBot crawls for model training. OAI-SearchBot surfaces sites in ChatGPT search results. Then there is ChatGPT-User, the agent that visits a page when a person asks a question, about which the docs say two things in one paragraph: that it is not used for crawling the web in an automatic fashion, and that because these actions are initiated by a user, robots.txt rules may not apply. [1]
Nobody is hiding a loophole there. It is an honest description of what robots.txt has always been: a convention addressed to crawlers, agreed by the well-behaved, enforced by nobody. It governs indexing. It has never governed a person, and an agent fetching a page because a person asked for it sits on the person’s side of that line.
The thing that does bind was configured for scrapers
Six storefronts served a desktop browser and refused the agent string minutes later. Not a challenge page, not a slow path: a refusal. Nothing about those sites is broken. Bot management is doing what it was configured to do, back when every non-browser client was a scraper, a price-watcher or a sneaker bot, and nothing has since forced anyone to revisit that.
Cloudflare’s documentation now separates the cases by behavior rather than by vendor. Search collects or indexes content to answer questions about it later. Agent is automated activity acting in real time on a person’s behalf to get something done. Training crawls content to train or fine-tune a model. One bot can have more than one behavior. [4]
Three behaviors, three separate commercial answers, and they are configured in one place. [5] A merchant almost always wants training refused, search allowed, and a buyer’s agent let through to a readable product page with a price on it. Ten of our eighteen storefronts name no AI agent anywhere, so all three behaviors get whatever the wildcard group happens to say. That is not a policy but a default nobody chose, the same failure mode as letting the feed decide which auctions you enter without reading it.
Nothing in the feed expires
Suppose the agent gets through. It now needs a price it can commit to, and here the failure is quieter and more expensive.
The Agentic Commerce Protocol, the feed contract Stripe and OpenAI publish for merchants, requires an availability value and a price on every row, rejects the row if availability is missing or unrecognized, and tells merchants to submit the current price and update the feed when a sale starts or ends. [2] It is explicit about what it will not do for you: sale-window dates, expiration dates and availability dates do not schedule price or availability changes, and merchants are told not to rely on uploaded dates to remove a product. [2]
The consequence is plain. A price in an agent feed never goes stale, because nothing in the format knows what stale means. It stays exactly as true as the last upload, and the only clock is the one you run. On a nightly batch, a promotion that ends at noon is quoted to buyers for the rest of the day by a system with no way of doubting it.
| Layer | What the agent meets | What to check |
|---|---|---|
| robots.txt | A wildcard disallow written for crawlers, which the vendor says may not apply [1] | Whether your file names agent behaviors, or leaves them to * |
| Edge | A rule set built when every non-browser client was a scraper | The status code your product page returns to an agent string |
| Product page | Price and availability that may exist only after rendering | Whether both sit in the HTTP response [3] |
| Feed | The last values you uploaded, with no expiry [2] | Whether it is rewritten on the event or on a schedule |
None of this is new discipline. Google already requires that the landing page show the same price, currency and availability you submitted, reports the mismatch by name when it does not, and recommends putting product information, particularly price and availability, in the HTTP response so it does not depend on the page being rendered. [3][6] Merchants who took Shopping feed hygiene seriously are most of the way there. What changes is the tolerance. A human who meets a wrong price argues, shrugs, or buys anyway. An agent reconciles two records, finds they disagree, and stops.
Six questions, in the order they fail
Could a buyer’s agent complete a purchase on your storefront today?
Six properties decide it, weighted by how often each is the thing that ends the session. “Ready” means an agent acting for a real customer can read a price it can trust and reach a cart. “Exposed” means it leaves, and nothing records that it was there.
The first two are the gate because they are binary and they come first. An agent that cannot fetch the page never reaches the price, and an agent that fetches a page with no price in the response has fetched nothing it can use. Everything below them is optimization on a session that already ended.
What this is not
This is not an argument that agent traffic is large. It is not, yet. The figures in circulation about how much of it converts come from vendor panels using different definitions of both “agent” and “checkout”, which is why none of them appear above. The most-cited telemetry on the subject sits behind a page that asked us to press and hold to confirm we were human, which is a fair summary of the whole problem.
It is an argument that the cost of being wrong is asymmetric and the fix is cheap. Blocking a buyer’s agent by accident costs you a sale you never see, because a refused request leaves no session, no cart and no row in any funnel report. That is what separates this from ordinary conversion work, where a drop-off at least leaves a trace. Here the failure is silent by construction.
So run the two requests. Ask your own storefront for a product page as a browser and as an agent, read both status codes, then compare the price in the response against the price in the feed. It takes an afternoon and the answer is yours rather than a vendor’s. If it all comes back clean, you have lost an afternoon and learned you are ready.
Sources
- OpenAI · Overview of OpenAI CrawlersThe three named agents, and the statement that for user-initiated fetches robots.txt rules may not apply.
- OpenAI and Stripe · Agentic Commerce Protocol: product feed specificationRequired price and availability, the instruction to update the feed when a sale starts or ends, and the rule that uploaded dates schedule nothing.
- Google Merchant Center Help · About landing page requirementsThe landing page must match the submitted price, currency and availability, and price and availability belong in the HTTP response.
- Cloudflare · What is a botThe search, agent and training behaviors, and the note that one bot can have more than one.
- Cloudflare · AI Crawl ControlWhere per-behavior allow, charge and block decisions are actually made on a Cloudflare site.
- Google Merchant Center Help · Inaccurate price due to inconsistency between feed and landing pageThe named failure when the feed and the page disagree, and how Google reports it.
Frequently asked questions
What exactly did you measure?
Two things, on September 13, 2026, across twenty large US storefronts. First, whether the robots.txt file disallows a cart, checkout, basket, or add-to-bag path to the default user-agent group. Second, whether the homepage answers a request that identifies itself as a desktop browser and a request that identifies itself with the ChatGPT-User string OpenAI publishes. One request each, from one connection.
What did the robots.txt files say?
Eighteen of the twenty served a robots.txt file to a browser user agent. Of those eighteen, fifteen disallow at least one buying path to the wildcard group, and eight name at least one AI agent anywhere in the file. The other ten say nothing about AI agents at all, which means an agent inherits whatever the wildcard group says.
Does robots.txt actually stop a shopping agent?
Not reliably, and the vendor says so. OpenAI documents that ChatGPT-User is used for fetches a person triggers, that it is not used for crawling the web in an automatic fashion, and that because these actions are initiated by a user, robots.txt rules may not apply. A disallow line is a request to well-behaved crawlers. It was never an access control.
So what does stop the agent?
The edge. Six of the twenty storefronts served a desktop browser user agent and refused the published agent string, mostly with a 403. That is bot management doing its job as configured, and the configuration predates anyone’s decision about buyer agents. The block is a side effect rather than a policy.
Is a single request per site a fair test?
It is a demonstration, not a league table, and it should be read that way. Bot management is probabilistic and sensitive to network reputation, so a different connection or a different hour can return a different answer for the same storefront. What the probe establishes is that the divergence exists and is easy to reproduce on your own domain, which is the only version of this result that should change your plans.
Why do stale prices break an agent sale in particular?
Because the agent commits to a number before it reaches you. The Agentic Commerce Protocol feed spec requires price and availability on every row and tells merchants to submit the current price and update the feed when a sale starts or ends. It also states that sale-window, expiration, and availability dates do not schedule price or availability changes. Nothing in the feed expires on its own, so a price is whatever you last uploaded until you upload again.
Does this overlap with Shopping feed hygiene we already do?
It uses the same discipline and a stricter tolerance. Google already requires that the price and availability on the landing page match what you submitted, and recommends putting price and availability in the HTTP response so the information does not depend on rendering. An agent that reads the initial HTML has exactly the same dependency, with less patience and no human to notice the discrepancy.
Should I block AI agents or let them through?
Decide it deliberately, per behavior, rather than inheriting a default. Cloudflare classifies AI bots by what they do rather than who sent them: search, agent, and training are separate behaviors and one bot can have more than one. A merchant usually wants training refused, search allowed, and a buyer’s agent allowed through to a readable product page. That is three decisions, and most storefronts have currently made none of them.
Mikołaj Salecki
Writes about media, tech, and AI business for people who actually run digital. Former agency lead. Skeptic of frameworks that read better than they perform.
