Why AI crawlers may not see half your website

There is a quiet gap on a lot of modern websites. A person opens the page and sees everything: the services, the prices, the opening hours, the reviews. A crawler opens the same page and receives an empty box and a script it does not run.

For years this mattered less, because Google eventually renders JavaScript. The AI crawlers that feed assistants and answer engines are different. Vercel's analysis of AI crawler traffic found that the major AI crawlers do not execute JavaScript at all (Vercel, "The rise of the AI crawler"). Whatever is not in the HTML the server sends back is invisible to them.

#What the gap looks like

Imagine a physiotherapy clinic whose services list is built in the browser. The raw HTML looks something like this:

<h2>Our services</h2>
<ul id="services"></ul>
<script src="/services.js"></script>

A visitor sees six services. An AI crawler sees a heading and an empty list. When someone asks an assistant which clinic near them treats running injuries, that clinic's own website has told the assistant nothing.

#Check your own site in five minutes

You do not need special tools. Pick your three most important pages, such as the home page, a service page and the contact page, and try these:

1. View the source. In your browser, open the page and choose View Page Source (not Inspect). Search for a sentence you can see on the page. If it is not in the source, a crawler that does not run JavaScript will not see it either.
2. Turn JavaScript off. In your browser's site settings, block JavaScript for your domain and reload. What remains is roughly what a non-rendering crawler gets.
3. Check the important facts. Look specifically for prices, opening hours, your address, your services and your phone number. These are the facts assistants are most often asked about.
4. Check robots.txt. Open yourdomain.com/robots.txt and make sure you are not blocking the crawlers you want to read you, such as OAI-SearchBot, PerplexityBot or ClaudeBot.

#What to do about it

The goal is simple to state: every fact that matters should be in the HTML your server sends.

• Server-render key pages. Most modern frameworks can render pages on the server or ahead of time. Service pages, pricing and contact details are the place to start.
• Keep facts in text, not images. A price inside a graphic is invisible to crawlers and to agents trying to compare options.
• Add structured data. Markup for your organisation, services and products tells machines exactly what each fact is.
• Re-check after every redesign. This gap often appears when a site moves to a new theme or builder, long after anyone checked.

#Why it matters more each month

AI tools are now an ordinary way to find a business. OpenAI said at DevDay 2026 that 1.2 billion people use ChatGPT every week. Each of those people can ask for a shortlist of businesses like yours, and the shortlist is written from what the tool can read.

Our website score runs this check for you: it reads your pages as an AI crawler does, counts how much text is there before any script runs, and tells you which AI crawlers your robots.txt lets in.