Generate the technical files with Claude

AI-crawler robots.txt

Your robots.txt should explicitly allow the retrieval crawlers and point to a sitemap that actually resolves. Explicit Allow rules remove ambiguity, and a Crawl-delay on the heavier bots keeps them polite. Have Claude generate it from the current crawler list, then diff it against what you have. One 2026 trap: your CDN can override you — Cloudflare has default-blocked AI crawlers on new domains since July 2025, so confirm in edge logs that these bots actually receive 200s.

Generate a robots.txt for my site at DOMAIN that:
- Allows all standard search + AI retrieval crawlers explicitly (Googlebot, Bingbot,
  Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot,
  PerplexityBot, Applebot-Extended, CCBot, Bytespider, Google-Agent), with Crawl-delay: 2 on the heavy ones.
- Disallows only real non-content paths I'll list (e.g. /api/, /assets/).
- Declares the Sitemap URL and references /llms.txt in a comment.
Then tell me which of these bots my current robots.txt (pasted below) is missing or blocking.

A real llms.txt

An llms.txt is a Markdown map at your domain root: an H1 title, a one-line blockquote summary, then ## sections linking your most important pages with short descriptions. Be clear-eyed about what it does: the citation evidence is negative — SE Ranking modeled ~300,000 domains and found no correlation with AI visibility, Ahrefs found 97% of published files received zero requests, and Google Search says it isn't needed. Yet Chrome's Lighthouse now audits for it in its Agentic Browsing category (default since May 2026) and Perplexity says it reads it. So ship it — ten minutes of agent-readiness and forward-compat — but never mistake it for a citation lever. Google itself splits: Search says skip it, Chrome audits for it.

From the sitemap / page list I'm pasting, write a spec-compliant llms.txt:
- H1 with the site name.
- A one-sentence blockquote describing what the site is and who it's for.
- ## sections grouping the most important pages, each as: - [Title](URL): one-line description.
- Only include genuinely important pages; skip utility and legal pages.
Keep descriptions concrete and factual — no marketing adjectives.

Accurate JSON-LD

First, Google's own position (official guide, May 2026): no special files, markup, or schema are needed for its AI features. Structured data serves entity understanding and the classic rich results that remain — not AI citation directly (an Ahrefs causal test on 1,885 pages found no significant citation lift from adding JSON-LD). Two deprecations to know: FAQ rich results ended May 7, 2026, and HowTo went in 2023 — both types stay valid for machine parsing but earn no SERP feature. Emit what matches the page, keep dateModified honest, and let the table guide what still pays:

The 2026 pivot is from schema-as-trick to entity as substrate: give your organization a Wikidata entry (there is no notability bar), point sameAs at it, add Person schema for real authors, and claim your Google Business Profile — the Knowledge Graph these feed is what grounds Gemini. Schema clarifies who you are; citations still come from what you publish and who corroborates it.

Generate JSON-LD (schema.org) for the page I'm pasting. Include:
- The correct primary type (Article or HowTo), with headline, description, author (Organization),
  publisher, datePublished, dateModified, inLanguage, and canonical url.
- FAQPage ONLY if the page has a visible FAQ, using the exact on-page questions and answers.
- BreadcrumbList reflecting its place in my site.
Output valid, minified JSON-LD in a single <script type="application/ld+json"> block, and flag any
field you had to guess so I can correct it.

Freshness signals

Answer engines favor current pages. Keep dateModified honest, show a visible "Last updated" date, and when something real changes in your topic, add a dated "recent developments" section rather than quietly editing — a specific, current fact is exactly what gets pulled into an answer about a breaking topic.