SourceCited / Phase 0 · Audit
0 Phase 0
Before optimizing anything, find what's silently blocking you. A perfect page behind a blocked GPTBot or a stray noindex earns zero citations.
Audit in dependency order: crawler access → indexing health → schema → answer-readiness. Fixing on-page copy is pointless if a bot can't fetch the page or a noindex is hiding it, so verify the plumbing first. Claude can run the whole pass against your files.
Audit in dependency order, because a fix downstream is worthless if something upstream blocks it. Check them in this sequence: (1) AI-crawler access — can the bots even fetch you; (2) indexing health — stray noindex, crawl traps, orphan pages; (3) schema coverage — what structured data you emit and whether it's accurate; (4) answer-readiness — do your pages front-load an extractable answer. Claude can run all four against your files.
You are auditing my website for SEO/AEO/GEO readiness. I'll paste (or point Claude Code at) my files. Work in this exact order and report findings as a checklist with PASS / PARTIAL / FAIL and the specific fix for each: 1. robots.txt — are the AI retrieval crawlers explicitly allowed (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Google-Agent)? Any that return Disallow, or a missing Sitemap line? Then check server/CDN logs: do these bots actually get 200s? (CDNs like Cloudflare block AI crawlers by default for new domains since July 2025 — the edge can override robots.txt.) 2. Indexing — list any pages with a noindex meta or header, any orphan pages (in the sitemap but not linked from the homepage/nav), and any broken sitemap reference. 3. Schema — inventory every JSON-LD @type emitted and flag inaccurate or missing ones (Organization on home, Article/HowTo/FAQPage on content, dateModified everywhere). 4. Answer-readiness — for each cornerstone page, does the first ~200 words contain a self-contained answer with a number or named entity? Quote the current opening and rate it. End with the 5 highest-leverage fixes ranked by impact.
Tip: keep this prompt in a Claude Project alongside your brand facts so every re-audit is consistent.
Retrieval bots are how AI answers get their live sources — block them and you're invisible for that surface no matter how good the page is. These are the ones that matter in 2026. Allow all of them unless you have a specific reason not to; note that the retrieval bot and the training bot are often different user-agents for the same company.
| User-agent | Company | Feeds | Allow? |
|---|---|---|---|
Googlebot | Search + AI Overviews / AI Mode | Yes | |
Google-Extended | Gemini grounding & training | Yes | |
OAI-SearchBot / ChatGPT-User | OpenAI | ChatGPT search & live retrieval | Yes |
GPTBot | OpenAI | Training corpus | Yes (usually) |
ClaudeBot / Claude-SearchBot | Anthropic | Claude retrieval & training | Yes |
PerplexityBot | Perplexity | Perplexity answers | Yes |
Applebot-Extended | Apple | Apple Intelligence | Yes |
CCBot | Common Crawl | Downstream training sets | Yes |
Google-Agent | User-triggered agent fetches (added Mar 2026) | Yes |
Verify before you pasteCrawler names change. Re-check each user-agent against the provider's own docs before editing robots.txt — a typo'd rule silently blocks a bot.
Blocking AI bots in robots.txt to "save crawl budget." You're opting out of every AI answer for that content.
A sitemap URL that 404s or points at the wrong file. We've seen a robots.txt advertise sitemap.xml while the real file was sitemap.php with the rewrite commented out — crawlers followed a dead link.
Orphan pages. A page in the sitemap but linked from nowhere reads as low-priority. Ask Claude to diff sitemap URLs against links on the homepage and nav.
A stray noindex left over from staging. One line can deindex a whole section. Grep for it first, every audit.
Your CDN blocking bots your robots.txt allows. Cloudflare has blocked AI crawlers by default for new domains since July 2025 — the edge can 403 a bot your rules welcome. Verify in server/CDN logs that retrieval bots actually receive 200s, not just that the file says Allow.
Yes — give it your robots.txt, sitemap, and a few representative pages, or point Claude Code at the repo so it reads everything directly. For a live site, you can also paste the rendered HTML of key pages.
A retrieval crawler that's blocked or a sitemap that doesn't resolve. Both make otherwise-good pages invisible to AI answers, and both are one-line fixes.
Quarterly for a full pass, plus a quick crawler-and-noindex check after any deploy. Google's update cadence is roughly quarterly and each one can shift what matters.