Search engines
robots.txt
robots.txt is a plain text file at the root of a website that tells automated crawlers which paths they may request. It works by user-agent, so a site can allow Googlebot while blocking a specific AI crawler, or the reverse. It controls crawling rather than indexing: a blocked page can still appear in results if other sites link to it, because the block prevents fetching, not listing.
Because the file is public, anyone can read a site's robots.txt by adding /robots.txt to the domain. It is worth reading a site's own file directly rather than assuming what it contains, since hosting platforms and plugins write rules into it without announcing them.
To keep a page out of search results, the correct tool is a noindex meta tag on the page, which requires the crawler to be able to fetch the page to see it. Blocking that page in robots.txt prevents the crawler from ever reading the noindex, which produces the opposite of the intended result.
For AI visibility, robots.txt is the file that decides whether GPTBot, PerplexityBot, and similar crawlers are permitted. A default-deny rule here quietly removes a site from every AI assistant that respects it.
Why it matters
One wrong line in this file can remove an entire website from search engines and AI assistants at once.
Related terms
Related reading
Last reviewed 2026-09-04. All Saige glossary terms.