84.2% of websites have no policy on AI crawlers at all. Not blocking them. Not helping them. Just invisible to the question, because nobody on the team ever opened robots.txt to check.
That gap is the whole opportunity here. Getting cited by ChatGPT or Google’s AI Overviews starts with a boring, unglamorous requirement. The AI crawler has to actually reach your page, read it, and make sense of it. No amount of great content fixes a page a crawler can’t access. This checklist covers exactly what to fix, in order, including the honest truth about llms.txt, which isn’t quite what most SEO blogs are telling you.
Why Crawlability Is the Foundation Everything Else Sits On
A review of 54 independent studies on AI citation ranked every factor by strength of evidence. Crawlability and access came out on top, ahead of backlinks, ahead of content length, ahead of almost everything else marketers obsess over. A page that ranks well but can’t be cleanly read by a crawler never gets a chance to compete on any other factor.
This is easy to overlook because it’s invisible in normal reporting. Google Analytics shows you human visitors. Search Console shows you indexing status for Googlebot specifically. Neither tool tells you whether GPTBot, ClaudeBot, or PerplexityBot ever successfully read a single page on your site. A team can spend months polishing content while an AI crawler quietly fails to load it every single time.
AI crawlers don’t behave like Googlebot either, which is where most technical checklists fall short:
- They rarely execute JavaScript, so content rendered client-side is often invisible to them.
- Several ignore standard rendering entirely and just parse raw HTML.
- Some, like Bytespider, are known to disregard robots.txt directives altogether.
- Most send far less traffic back than they take, so many publishers block them on principle, not performance.
The Crawlers Worth Knowing by Name
| Crawler | Operator | What It Does |
|---|---|---|
| GPTBot | OpenAI | Crawls content for model training |
| OAI-SearchBot | OpenAI | Powers ChatGPT’s live search results |
| ClaudeBot | Anthropic | Crawls for training and retrieval |
| PerplexityBot | Perplexity | Powers real-time answer citations |
| Google-Extended | Controls Gemini and AI Overview training access | |
| Bytespider | ByteDance | Crawls broadly, known to ignore robots.txt |
One Missing Line, One Invisible Site
A mid-market SaaS company once asked why ChatGPT never mentioned them, despite ranking well on Google for their core terms. Their robots.txt looked clean. No AI crawler was disallowed by name.
The real problem was one directive higher up: a blanket rule blocking a specific bot pattern, added years earlier to stop a scraper unrelated to AI. GPTBot happened to match that old pattern. Nobody had touched the file since, and nobody thought to check it against a list of AI user agents that didn’t exist when the rule was written. One line, written for a completely different reason, had quietly kept them out of every AI answer for months.
Is Your Site Accidentally Blocking AI Crawlers?
Most blocks aren’t a deliberate decision. A Q3 2026 crawl of 1,744 sites found that 9.9% block GPTBot directly in robots.txt, while 84.2% have no AI crawler policy of any kind, meaning nobody has ever decided one way or the other. That silence isn’t neutral. A crawler that hits an unclear signal often just moves on to a site that’s easier to parse.
The blocks that hurt most aren’t even in robots.txt. Check these three spots first:
- CDN and firewall rules. Security teams sometimes block AI bots at the CDN layer without telling marketing, so robots.txt looks fine while the real block sits upstream.
- JavaScript-only rendering. If your content only appears after a script runs, several AI crawlers never see it at all.
- Login walls and gated content. Anything behind a form is invisible to every crawler, human or AI.
Not sure what’s actually blocking your site. Our Technical Foundation work audits robots.txt, CDN rules, and rendering to find exactly what’s keeping AI crawlers out.
Schema Markup: The Machine-Readable Layer
Once a crawler can reach your page, schema tells it what that page actually is. Pages with proper schema markup are about 2.5 times more likely to appear in AI-generated answers than pages without it.
Priority Schema Types for a Software Site
- Organization schema– establishes your brand as a real, identifiable entity.
- Article schema– marks blog and guide content with author, date, and publisher.
- FAQ schema– wraps question-and-answer content for direct extraction.
- Person schem- ties named authors to a verifiable identity, with sameAs links to their profiles.
llms.txt, Honestly
Most write-ups sell llms.txt as a must-have AI SEO tactic. The actual 2026 data tells a more complicated story.
Adoption sits around one site in ten. An SE Ranking study of 300,000 domains found a 10.13% adoption rate, and among the fifty most AI-cited domains, only one had the file at all. Major crawlers barely fetch it once it’s published. And Google has said, on the record, that it won’t use the file to shape AI Overviews.
So why bother? Three honest reasons, none of which is “it drives citations.” None of them replace real crawl access or clean schema either, but each one is worth the small effort it takes:
- It costs almost nothing. Twenty minutes buys you a foothold if the standard ever gains real support.
- Writing one is a useful audit. Summarizing your most important pages in one sentence each often exposes duplicate or orphaned content you didn’t know you had.
- It’s built for AI agents, not search citations. It’s a business-to-agent surface, aimed at tools that need to know what your site offers, not a ranking lever.
Ship one because it’s cheap, not because it’s magic.
The Technical SEO Checklist for AI Crawlability
Everything above rolls up into one working list. Work through it in order, since each row depends on the one before it. Access comes first. Structure comes second. Curated extras like llms.txt come last, because they only matter once the fundamentals are already solid.
| Check | What to Verify | Why It Matters |
| robots.txt policy | GPTBot, ClaudeBot, and PerplexityBot are explicitly allowed | 84.2% of sites have no policy at all, and default silence isn’t the same as access |
| Rendering | Key content appears in raw HTML, not just after JavaScript runs | Several AI crawlers don’t execute scripts before reading a page |
| Organization + Article schema | Present and valid on every key page | Schema-marked pages are about 2.5x more likely to appear in AI answers |
| FAQ schema | Applied to genuine Q&A content | Matches the exact format AI answers pull from |
| Author + Person schema | Named authors linked to verifiable profiles | Gives models a trust signal to check against |
| llms.txt | Published, curated, kept current | Near-zero cost; not proven to drive citations yet |
| Crawl logs | Reviewed monthly for AI bot activity | Tells you what’s actually happening, not just what you configured |
Sources: Zyppy’s 54-study AI citation meta-analysis, Stackmatix schema markup study
How to Actually Verify It’s Working
Configuration isn’t the same as confirmation. Pull your server logs and filter for GPTBot, ClaudeBot, and PerplexityBot user agents. If you see zero hits after a month, something upstream is still blocking access, no matter what robots.txt says.
Most hosting dashboards let you search raw access logs by user agent string without any special tooling. Look for a real request that returns a 200 status code, not just an entry in the log. A crawler blocked at the CDN layer often still shows up in application logs. It just carries a 403 or 429 response instead. That looks like activity, but it actually confirms the block is working exactly as configured, just not the way you want.
This same gap shows up in content strategy too, not just crawl access. It’s exactly why we built our post on programmatic SEO without the spam around real verification, not assumptions.
Get Your Site’s Crawlability Checked Before You Publish Another Page
A brilliant piece of content sitting behind a JavaScript wall or an accidental CDN block never gets a chance to be cited. It doesn’t matter how well it’s written, how well it ranks on Google, or how much research went into it. This checklist fixes that, in the order that actually moves the needle.
Book your free growth audit and we’ll run every row above against your live site, then send you a prioritized fix list you can hand straight to your dev team.
FAQ
Do I need to block AI crawlers to protect my content?
That’s a business decision, not a technical one. Blocking GPTBot stops training use but can also cut you out of citations, since some platforms use the same or related crawlers for search.
Is llms.txt worth implementing in 2026?
Yes, but only because it’s cheap and low-risk. The evidence doesn’t yet show it improves AI citations, so treat it as a small bet, not a strategy.
What’s the single highest-priority fix on this checklist?
Confirm your robots.txt actually allows the crawlers you want. It’s the most common accidental block and the fastest one to fix.
Does schema markup guarantee my content gets cited?
No. It raises the odds significantly, but crawlability and content quality still have to be in place first.
How often should I recheck this checklist?
Quarterly at minimum. AI crawler names, behavior, and even company ownership have shifted multiple times in the past year alone.
Can I check any of this myself without a developer?
Yes. Robots.txt is a plain text file at yourdomain.com/robots.txt, and Google’s Rich Results Test will validate your schema markup for free.
