Your Site Looks Fine in Google but Is Invisible to AI Crawlers

Your site ranks fine in Google. It's indexed, it gets impressions, Search Console shows no errors. Then you ask ChatGPT about your own service category and you're simply not there. That gap is a technical SEO for AI problem, and it's increasingly common because AI crawlers behave nothing like Googlebot.

Server log data shows AI bots now generate roughly 40 to 50% of Googlebot-level activity across the web, according to log analysis from thousands of sites. Yet most sites are technically unprepared for them. Only 14% of brands track AI citation visibility, even though 89% appear in AI Overviews, per a survey of 100+ SEO professionals.

The core issue: AI crawlers are less capable than Googlebot. They don't render JavaScript, so a React or client-side-rendered page that looks perfect in your browser can appear empty to GPTBot or PerplexityBot. As one technical analysis puts it, AI crawlers often fail on JavaScript-heavy pages, and if they hit technical barriers, they simply skip your content.

Ranking in Google no longer guarantees AI visibility. Only 17% of AI Overview citations come from pages ranking in the organic top 10, down from roughly 76% in late 2024, per BrightEdge data. Citation has decoupled from ranking, so a site can hold position 3 for its money keyword and still be absent from ChatGPT and AI Overviews.

Here's where sites most often go wrong, usually without knowing it:

• JavaScript-only content renders fine for Google but returns a blank shell to AI crawlers, which don't execute scripts or click buttons.
• AI bots ignore canonical tags and meta noindex, so content you've hidden from Google can leak into AI training data, while robots.txt is the only reliable control.
• CDN and WAF bot protection from Cloudflare, Akamai, or Fastly frequently blocks AI crawlers by accident, locking legitimate bots out at the firewall layer.

There's a subtlety most teams miss: being crawled by AI bots isn't the same as appearing in AI answers. Log data distinguishes training bots (GPTBot, ClaudeBot), search bots (OAI-SearchBot, Claude-SearchBot), and user bots that fetch answers for real people. A site can be heavily crawled by training bots and still never surface in a single AI answer, because visibility is driven by user-bot activity.

The commercial stakes make this worth fixing. Brands cited in AI Overviews earn 35% more organic clicks than non-cited competitors, and AI-referred visitors convert at 4.4x the rate of traditional organic visitors, per aggregated performance data. Meanwhile, unprepared brands face projected traffic declines of 20 to 50% from traditional channels, according to McKinsey research.

At AI SEO Consultants, we regularly audit sites that pass every Google check yet fail all four of these AI crawler tests. If that sounds like your site, start with the basics: verify your robots.txt rules for GPTBot, ClaudeBot, and PerplexityBot, confirm your key content exists in server-rendered HTML, and check your WAF logs for blocked AI crawler requests. A free AI crawler audit can flag all three in a single pass, covering your robots.txt, schema, and crawlability so your site is genuinely AI-search ready.

The Robots.txt Playbook: Control GPTBot, ClaudeBot, PerplexityBot, and Google-Extended

Your robots.txt file is now the single most consequential piece of technical SEO for AI you control. AI bots already account for roughly 40–50% of Googlebot-level activity across the web, per server log analysis, and robots.txt is the only mechanism they reliably respect.

Before writing rules, you need to understand the three bot types. Training bots (GPTBot, ClaudeBot) crawl broadly to build models. Search bots (OAI-SearchBot, Claude-SearchBot) behave like Googlebot and power live answers. User bots fetch pages on behalf of actual people asking questions, and that third group is what drives real visibility.

Why blanket allow-or-block decisions fail: blocking GPTBot doesn't block ChatGPT Search if OAI-SearchBot is still allowed, and vice versa. OpenAI and Anthropic separate these crawlers, so you can control each independently, per provider documentation. A blanket rule almost always blocks visibility you wanted or opens content you didn't.

Here are the copy-paste configurations:

Allow GPTBot everywhere:

User-agent: GPTBot
Disallow:

Block training but keep ChatGPT Search:

User-agent: OAI-SearchBot
Disallow:

User-agent: GPTBot
Disallow: /

A selective strategy, which is what most sites actually need:

User-agent: GPTBot
Disallow: /intern/
Disallow: /suche/
Allow: /ratgeber/

The selective approach blocks internal search pages, test environments, customer areas, and paid content while leaving public specialist content open. That's the pattern our team at AI SEO Consultants configures most often, because it protects what matters without sacrificing AI search optimization.

• OAI-SearchBot powers ChatGPT Search; allowing GPTBot alone doesn't guarantee visibility there.
• PerplexityBot is not used to train base models, so blocking it costs you citations without protecting much.
• Google-Extended controls Gemini training and grounding, but it does not affect standard Google Search or AI Overviews eligibility.
• AI bots ignore canonical tags and meta noindex; content hidden from Google stays visible to AI crawlers if internally linked.

Google-Extended deserves its own decision. Blocking it changes what Gemini can draw on, but your regular Google rankings and AI Overviews presence run through the standard Search index, as Google's own guidance confirms. Treat it as a separate lever, not a package deal.

One caveat you can't skip: robots.txt is not a security barrier. It sets rules for cooperative crawlers but protects nothing from a bad actor, and user-agent strings can be spoofed. Keep confidential data behind authentication, always. If you'd rather have someone verify all of this against your actual logs, get a free AI crawler audit: our team will analyze your robots.txt, schema, and crawlability to make your site AI-search ready. Book your consultation today.

Skip llms.txt, Fix What Actually Matters: Rendering, Speed, and Crawl Depth

You've optimized your robots.txt for GPTBot, PerplexityBot, and ClaudeBot. Now the harder question: should you also publish an llms.txt file? The honest answer, backed by both Google and raw server log data, is no. Spend that effort on rendering, speed, and crawl depth instead.

Google's official AI optimization guide says it plainly: you don't need llms.txt or similar machine-readable files because "Google Search itself doesn't use them" (per Google's documentation). The server-side evidence agrees. Log analysis across thousands of sites shows major AI bots never request the file at all. It sits unread while the things that actually determine AI visibility go unfixed.

The one narrow exception: B2B developer documentation. Coding agents like Claude Code can request llms.txt when working with API docs (practitioners report), so if your audience is developers, a quick test costs little. For everyone else, deprioritize it.

What actually moves the needle is technical accessibility, because AI crawlers are less capable than Googlebot. They don't render JavaScript, they don't click buttons, and when they hit a technical barrier they simply skip your content. A site that looks fine in Google Search can be invisible to an AI engine.

The benchmarks to hit, drawn from real crawler log data:

• Server-rendered HTML. All AI-critical content must exist in the raw HTML response, not behind client-side JavaScript.
• TTFB under 600 ms and HTML under 200 KB, with server responses under 3 seconds.
• Key pages within 2–3 clicks of the homepage. AI search bot visits drop to roughly one per page at 3+ clicks deep.

Speed matters for a subtle reason. As one log-analysis practitioner put it, ChatGPT crawls a lot but only uses the fastest pages in answers; slow pages get crawled but never selected. Being crawlable gets you fetched. Being fast gets you cited.

Crawl depth deserves equal attention. If your best content lives four clicks from the homepage, AI search bots barely reach it. Flatten your architecture, use descriptive anchor text, and link important pages from high-authority hub pages. Also check your CDN or WAF: Cloudflare, Akamai, and Fastly bot protections frequently block legitimate AI crawlers without you knowing.

One more thing worth verifying: your robots.txt and Google-Extended settings. Allowing GPTBot alone doesn't guarantee ChatGPT Search visibility, since OAI-SearchBot must be allowed separately, and Google-Extended controls Gemini training without affecting standard Search.

If you'd rather not audit all of this yourself, AI SEO Consultants offers a free AI crawler audit: our team will analyze your robots.txt, schema, and crawlability to make your site AI-search ready. Book your consultation today.

Schema That Earns Citations: Attribute-Rich Beats Generic

Adding schema markup is one of the most common "AI search optimization" recommendations, and it's also one of the most misunderstood. The honest answer from the data is nuanced: slapping generic schema on a page does nothing for AI visibility, but a specific kind of schema does measurably help.

An SSRN study analyzing 730 AI citations across 75 queries and 1,006 pages found that schema presence alone doesn't predict whether a page gets cited by AI systems. The odds ratio was 0.678 and the result wasn't statistically significant. If your technical SEO for AI strategy is "add schema everywhere," the data says that's wasted effort.

But the same study found a real signal. Pages with attribute-rich Product and Review schema were cited at 61.7%, versus 41.6% for pages using generic schema types. That's a statistically significant gap (p = .012), and it makes intuitive sense: AI systems extract facts to compose answers, and structured attributes like price, rating, and specifications are exactly the facts they need.

The advantage was strongest for lower-authority domains (DR 60 and below). If you're not a household-name site with tens of thousands of backlinks, attribute-rich schema is one of the few levers that levels the field. That matters, because sites with 32,000+ referring domains are 3.5x more likely to be cited by ChatGPT than sites with 200 or fewer, per aggregated AI search statistics.

The difference is depth. A bare Product type with a name and image is generic. Attribute-rich means the fields AI systems actually want to quote are populated:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Wi-Fi Smart Thermostat Pro",
  "description": "Programmable smart thermostat with room sensors and app control.",
  "offers": {
    "@type": "Offer",
    "price": "249.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.7",
    "reviewCount": "1284"
  },
  "brand": { "@type": "Brand", "name": "ThermoHome" }
}

Populate pricing, aggregateRating, availability, and specifications. Empty or minimal markup is where the citation advantage disappears.

Google's AI optimization guide is blunt: structured data is not required for generative AI features, and there's no special schema.org markup you should add just for AI. Google's AI Overviews and AI Mode are rooted in core Search ranking and retrieval, so schema's role is the one it's always had: earning rich results and making your facts machine-readable.

The practical takeaway for technical SEO for AI: keep schema in your workflow for the rich-result benefit, but invest your effort in attribute-rich markup on the pages whose facts you'd want an AI answer to quote. That's the version the data supports, and it's the version we build into every AI crawler audit at AI SEO Consultants, because markup that's thin is markup that gets ignored.

Measure What Matters: Search Console, Query Fan-Outs, and Your AI Visibility Audit

In the rush to optimize for AI answers, most brands are tracking rankings but missing the actual signals that determine visibility. Research shows only 14% of brands systematically track AI/LLM citation visibility, even though 89% appear in AI Overviews. This gap means many site owners have no data on whether their content is being surfaced, cited, or ignored by the engines their audiences actually use.

Google’s AI features operate on "query fan-outs", concurrent related queries that pull from the Search index. To be eligible for these features, pages must be indexed and eligible for a snippet, and the site must be included in Search generative AI features within Search Console. This makes Search Console the primary diagnostic tool for understanding how AI systems discover your content. Within the console, you can spot query fan-outs, long-tail, multi-word search strings that reveal the specific questions AI systems are trying to answer. These typically run at least 8 to 9 words in length and appear as impressions or clicks from user bot activity, offering a direct line into what AI engines find relevant.

Crawlability remains the bedrock of AI visibility. AI crawlers like GPTBot and PerplexityBot often fail on JavaScript-heavy pages, gated resources, or complex navigation structures. If a page hits a technical barrier, the crawler simply skips it. A site that renders perfectly in a standard browser may be effectively invisible to an AI engine if the critical content isn't in server-rendered HTML. Page speed also factors into selection: AI systems favor fast-loading pages, with benchmarks suggesting a time-to-first-byte under 600 milliseconds and HTML file sizes ideally under 200 KB. Beyond speed, site architecture matters. AI bots tend to drop off after 2 to 3 clicks from the homepage, so important pages should be within that depth with descriptive anchor text guiding the way.

Not all crawlers behave the same, and robots.txt is the primary lever for controlling them. OpenAI maintains separate user-agents: GPTBot for training and OAI-SearchBot for search. Blocking GPTBot alone will not guarantee ChatGPT Search visibility, OAI-SearchBot must also be allowed. A selective robots.txt strategy is more effective than a blanket allow or block. For example, you might block internal search pages, test environments, and paid content while allowing public specialist content. Here is a copy-paste example for allowing GPTBot:

User-agent: GPTBot
Disallow:

And for blocking GPTBot training while keeping ChatGPT Search active:

User-agent: OAI-SearchBot
Disallow:

User-agent: GPTBot
Disallow: /

Google-Extended controls Gemini training and grounding but does not affect standard Google Search. PerplexityBot, conversely, is not used to train base models. Monitoring your server logs and Google Search Console is essential, as user-agent strings can be spoofed, and CDN or WAF layers frequently block AI crawlers accidentally. Whitelisting known AI crawler IP ranges ensures your content remains accessible.

Schema markup is another area where technical SEO intersects with AI optimization. While generic schema does not independently predict AI citation, attribute-rich Product or Review schema with populated pricing, aggregate ratings, and specifications fields is cited at substantially higher rates, 61.7% versus 41.6% for generic types. This advantage is most pronounced for lower-authority domains. Implementing this schema as part of a broader SEO strategy helps AI systems extract and cite your content more effectively.

Only 14% of brands track AI citations, leaving a significant opportunity for those who do. Using Google Search Console to monitor query fan-outs and user bot activity provides the data needed to refine your strategy. If you are ready to see where your site stands, a free AI crawler audit can analyze your robots.txt, schema markup, and overall crawlability to identify gaps. There are no guarantees, only a clear diagnostic to help your site become AI-search ready.

Get a free AI crawler audit, our team will analyze your robots.txt, schema, and crawlability to make your site AI-search ready. Book your consultation today.