▦ Checklane
How it works Our own audit What we check FAQ Guides
Free audit

Checklane Guides

Should You Block AI Crawlers? A Store Owner's Guide to robots.txt

No technical background required. About a 4-minute read.

Somewhere in your website sits a small file called robots.txt. It tells automated visitors which parts of your site they may read. In the last couple of years, a new crowd of visitors showed up: AI crawlers with names like GPTBot, ClaudeBot, and PerplexityBot. Many store owners blocked them on reflex. If you want AI shoppers to find you, that reflex may be costing you sales.

What robots.txt does

robots.txt is a plain-text file at yoursite.com/robots.txt. It lists crawlers by name and says "you may read these pages" or "stay out." Respectful crawlers obey it. It was built for search engines, but AI companies' crawlers read it too.

A typical block looks like this:

User-agent: GPTBot Disallow: /

Those two lines tell OpenAI's crawler to stay off your entire site.

Which AI crawlers exist

The names you will see most: GPTBot (OpenAI, feeds ChatGPT), ClaudeBot (Anthropic, feeds Claude), PerplexityBot (Perplexity), and Google-Extended (Google's control for AI training use). There are others, and new ones appear regularly. You can see exactly who you are blocking by opening your own robots.txt in a browser.

"Every crawler you block is an AI assistant that will never learn your store exists."

The tradeoff, honestly

Blocking is not crazy. Crawlers cost you bandwidth, and some store owners object to their content training AI models. Those are legitimate concerns, and for a blog or a publisher they may win.

But a store is not a publisher. Your product pages are advertisements you pay to keep online. An AI crawler reading them is a potential shopper doing research. Blocking it does not protect a trade secret; it hides your storefront from the fastest-growing kind of window shopper.

The math is simple: if you want AI assistants to recommend your products and AI agents to buy them, those systems have to be allowed to read your site. Blocked pages cannot be recommended, cited, or purchased from.

What we recommend for stores

1. Open yoursite.com/robots.txt and read it. Look for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
2. If you sell things online and want AI-driven traffic, do not disallow those crawlers from your product and category pages.
3. Keep blocking what should stay private: admin pages, carts, checkout flows, and customer accounts. Those were never meant for crawlers of any kind.
4. Re-check after any site migration or redesign. Blocks have a way of reappearing.

One nuance worth knowing

Some AI features do not rely on these crawlers at all. Shopping agents that drive a real browser, for example, visit your site the way a person would and never check robots.txt. Unblocking crawlers helps the assistants that read and cite the web. It does not hurt you with the agents that browse directly. There is no scenario where allowing product-page reads makes you worse off with AI shoppers.

Run your free audit

Checklane checks your robots.txt for AI crawler blocks, plus every other signal AI shoppers read. Paste any public URL for your free 0-100 score.

Get my free score

▦ Checklane helps small businesses get found by AI shoppers.

Guides · Services · Terms · Privacy · Refund policy

Checklane grades what’s technically verifiable about your site at audit time. A score is not a promise of sales or AI recommendations.