AI search
AI Crawler
Also called: GPTBot, PerplexityBot, ClaudeBot
An AI crawler is an automated program that fetches web pages on behalf of an AI company, either to train a model or to retrieve current pages while answering a question. Each one identifies itself with a user-agent string, such as GPTBot for OpenAI or PerplexityBot for Perplexity, and each can be allowed or blocked in a site's robots.txt file independently of Googlebot.
The distinction between training crawlers and retrieval crawlers matters when deciding what to allow. Blocking a training crawler keeps content out of a future model. Blocking a retrieval crawler prevents the assistant from fetching the page while answering a live question, which removes any chance of being cited in that answer.
A common and costly mistake is blocking these crawlers by accident. Some hosting platforms and security products block unfamiliar bots by default, so a site can be unreachable to AI assistants without anyone having chosen that.
Most of these crawlers do not run JavaScript. A page whose content is assembled in the browser after load will be fetched as a nearly empty document, which reads as a page with nothing on it.
Why it matters
A site that quietly blocks AI crawlers cannot be cited by any AI assistant, no matter how good its pages are.
Covered in depth in AI Search Visibility for Real Estate Agents.
Related terms
Related reading
Last reviewed 2026-09-04. All Saige glossary terms.