Pillar 1 of 5 · Crawl
Can AI systems find and retrieve your content?
Crawl is whether AI systems can find and retrieve your pages at all. AI systems run their sub-queries against Google and Bing, so your content has to be indexable in both. If a crawler is blocked, or the page only renders after JavaScript, you are invisible before the rest of the framework even matters.
How it maps to Query Fan Out
Pages must be indexable by Google and Bing, where the sub-queries run.
What to do
- Allow GPTBot, ClaudeBot, and PerplexityBot in robots.txt.
- Submit your sitemap to Bing Webmaster Tools. 87% of SearchGPT citations match Bing’s top organic results.
- Serve content server-side so crawlers see it without executing JavaScript.
- Add Schema.org markup and define a canonical URL.