Technical
Making Your Agent Website Readable by AI Crawlers
The retrieval checks that decide whether an assistant can see your site at all, in the order they matter.
Part of the AI Search Visibility for Real Estate Agents guide · Last reviewed 2026-08-09
Before an AI assistant can quote your website, it has to fetch the page and find your content in the response. Two problems block this on most real estate agent sites: a robots.txt or firewall rule that excludes the assistant's named crawler, and content that only appears after JavaScript runs, which is common wherever an IDX feed supplies listings. Both are checkable in minutes, and both are decided before any writing matters. This page covers how to test each one and what the fix looks like.
Key takeaways
- View-source is the test that matters: if a sentence is not in the raw HTML, a retrieval pass does not have it.
- AI crawlers are named and separate from Googlebot, so a site can rank on Google while being invisible to ChatGPT.
- Bot-filtering at the CDN or firewall blocks AI crawlers silently, without any robots.txt rule to find.
- JavaScript-rendered IDX content is the single most common reason an agent site has no readable listing text.
- Server logs are the only direct confirmation that a named AI crawler is actually fetching your pages.
Check one: can the crawler fetch the page at all
Start with robots.txt at yoursite.com/robots.txt. You are looking for Disallow rules that apply to the AI crawlers by name (OAI-SearchBot, ChatGPT-User, PerplexityBot) or a blanket rule under User-agent: * that catches them.
Then check the layer robots.txt does not show you. Cloudflare and similar services offer AI bot blocking as a toggle, and some hosts enable it by default or as part of a security preset. That produces a hard block with nothing in robots.txt to explain it. If your site is behind such a service, the setting has to be inspected in that dashboard directly.
A blanket disallow is worth a second look for another reason: rules written years ago to stop scrapers now also exclude the assistants that could be recommending you.
Check two: is your content in the HTML
This is the check that finds the real problem on most agent sites, and it needs no tools. Open one of your pages, view the page source, and use the browser's find function to look for a sentence you can read on screen.
Present in the source: the crawler has it. Absent: the text is being inserted by JavaScript after the page loads, and a text-only retrieval pass sees the empty shell. Run this on your homepage, your bio or about page, one listing page, and one neighbourhood page, because the answer often differs between them. IDX-supplied pages are where it usually fails.
- Homepage: check your headline and your service-area sentence.
- Bio or about page: check your licence, years active, and brokerage.
- A listing page: check the address, price, and description text.
- A neighbourhood page: check the body copy, not just the heading.
What to do when the content is JavaScript-only
There is no configuration flag that fixes this. The content has to be present in the server's response, which means one of a small number of real changes: render the pages on the server, prerender them at build time, or replace the client-side widget with a feed-driven build that outputs real HTML.
One thing worth knowing before you spend money on it: Google can render JavaScript, so a JS-only site may rank acceptably while remaining invisible to assistants that do not render. That gap is why an agent can be told their SEO is fine and still never appear in an AI answer. The two systems are not testing the same thing.
Check three: are the crawlers actually visiting
Everything above is a permission check. Server logs tell you what happened. Filter your access logs for the crawler names and look for successful fetches rather than 403 or 404 responses.
Two patterns are worth acting on. No AI crawler entries at all suggests a block you have not found yet, or a site too new or too unlinked to have been discovered. Entries returning errors point at a firewall rule or a broken path rather than a content problem. Either way this is direct evidence, which puts it ahead of any inference from your rankings.
The order to work in
Do these in sequence, because each one gates the next. There is no value in improving your writing while the crawler is receiving a 403.
- Read robots.txt and remove rules that block the assistants you want to reach.
- Check your CDN or firewall for an AI bot blocking setting.
- Run the view-source test on four different page types.
- If content is JavaScript-only, decide how the pages will be rendered before writing anything new.
- Confirm in server logs that named AI crawlers are fetching successfully.
- Only then move on to what the pages say.
| What you observe | Likely cause | First thing to do |
|---|---|---|
| Ranks on Google, never appears in ChatGPT | AI crawler blocked, or content requires JavaScript | Read robots.txt, then run the view-source test |
| Listings invisible, static pages fine | IDX widget rendering client-side | Decide on server-side rendering or prerendering for listing pages |
| No AI crawler hits in server logs | Firewall or CDN bot filtering, or site not discovered | Check the CDN bot settings, then confirm the site is in a sitemap and linked |
| AI crawler hits returning 403 | Security rule rejecting the user agent | Allow the named crawler in the firewall rule |
Frequently asked questions
How do I know if my site is JavaScript-only?
Open the page, view the page source, and search the source for a sentence you can see on the rendered page. If the sentence is not in the source, that content is being added by JavaScript and a text-only crawler will not see it.
Will blocking AI crawlers protect my listing photos?
It changes who can fetch them from your site, but it also removes you from the assistants' results. It is worth separating the two decisions: training crawlers and search crawlers are different user agents, so you can usually decline training while remaining findable. Check each vendor's crawler documentation for which name does which.
Does my IDX provider handle this for me?
Ask them directly whether their pages are server-rendered, and then verify with the view-source test rather than taking the answer on trust. Many IDX integrations are explicitly client-side widgets, which is convenient to install and invisible to text retrieval.
Do I need a sitemap for AI crawlers?
A sitemap helps any crawler discover pages, so it is worth having and keeping accurate. It is a discovery aid, not a substitute for the page returning readable content when fetched.
Related pages in this guide
Related reading
Sources
Every claim on this page that could be checked against a primary source is linked below. Where something is not publicly documented by a vendor, the page says so rather than filling the gap with an estimate.
- OpenAI crawlers and user agents (OpenAI)
- PerplexityBot crawler documentation (Perplexity)
- Introduction to robots.txt (Google Search Central)