Does blocking GPTBot actually prevent ChatGPT from citing my site?

Yes, with nuance. Blocking GPTBot prevents OpenAI from using your content for training and from crawling for real-time search. If you want citations in ChatGPT's browsing and search responses, allow ChatGPT-User separately. GPTBot and ChatGPT-User are distinct user-agents with different functions, so you can block training crawls while keeping retrieval open.

Can AI engines read content behind login walls or paywalls?

No. AI crawlers access only publicly available HTML. Any content gated behind authentication is invisible to every major AI engine. If you want paywalled content to earn citations, consider adding a freely accessible summary or abstract that is schema-marked and crawlable.

What is the difference between llms.txt and robots.txt for AI crawlers?

robots.txt is an access control file that can block or allow crawlers at the URL level. llms.txt is a Markdown navigation file that helps AI systems find and prioritize your best content, but it cannot block any crawler or restrict access. Both serve different functions and should be implemented together. See the full llms.txt explainer for implementation details.

Does page speed matter if my content is excellent?

Yes. AI crawlers have request timeouts, and pages that respond slowly risk partial extraction or being dropped from the crawl cycle entirely. Pages meeting Google's Core Web Vitals Good thresholds also appear more consistently in AI Overviews. Performance is a prerequisite layer, not a nice-to-have.

Should I implement HowTo or FAQ schema even though they no longer generate rich results?

Yes. Google removed HowTo rich results in 2023 and FAQ rich results in May 2026, but both schema types remain valid structured data. AI engines use FAQPage schema to extract discrete Q&A pairs for inclusion in answers, and HowTo schema helps AI systems understand step sequences. The value has shifted from SERP decoration to AI extraction.

How often do AI crawlers revisit my pages?

Revisit frequency varies by crawler and page authority. GPTBot revisits high-traffic pages roughly every 2.4 days; other crawlers are slower. Keeping your lastmod sitemap timestamps accurate and your internal link structure robust signals content freshness and increases the chance of timely re-crawls after updates.

What is the biggest technical mistake brands make in GEO?

Client-side-only rendering is the single most common blocker. If your framework delivers an empty HTML shell and populates content via JavaScript after load, most AI crawlers never see your content. Fix SSR or pre-rendering first, before any other GEO tactic, because everything else is downstream of whether the crawler can read the page at all.

How do I know if my site is being crawled by AI engines?

Check your server access logs filtered by the AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Grok-Bot, etc.). Most analytics platforms do not track bots, so log-level analysis is required. A free baseline check is available at /tools/geo-audit, which surfaces common crawlability failures without requiring log access.

Technical GEO: Making Your Site Crawlable and Parseable for AI Engines