Most AI crawlers don’t run JavaScript. Here’s what that means for sponsored content.

A software company pays for a sponsored module on a review site’s “best CRM” guide. A reader loads the page and sees it: a logo, three bullet points, a button. A minute later an AI crawler fetches the same URL to help answer “What’s a good CRM for a small team?” It sees an empty <div>.

Nobody did anything wrong. The module loads with JavaScript, and the crawler doesn’t run JavaScript. That gap matters more as buyers do more of their research through AI assistants: in a G2 survey of 1,076 buyers, 51% of B2B software buyers said they now start software research in an AI chatbot more often than in Google (G2, April 2026).

How AI crawlers read a page

An AI crawler is a program that requests a URL, takes whatever the server sends back, and extracts the text. It identifies itself with a user agent string, such as GPTBot, ClaudeBot or PerplexityBot, names that OpenAI, Anthropic and Perplexity publish in their crawler documentation. Some crawlers collect pages for training. Others fetch pages in real time to answer a specific question.

What most of them don’t do is behave like a browser. A browser downloads the HTML, then fetches and runs every script on the page, and those scripts can add, remove or rewrite content after the fact. Running a full browser for every page is slow and expensive at crawl scale, and most AI crawlers skip it. They read the HTML the server sent and move on. When Vercel analyzed AI crawler traffic across its network, none of the major AI crawlers, including OpenAI’s, Anthropic’s and Perplexity’s, rendered JavaScript. Google’s Gemini and Apple’s crawler were the exceptions (Vercel, December 2024).

So every page effectively has two versions: the HTML the server sends, and the page a browser builds after the scripts run. People see the second. Most AI crawlers see the first.

Why sponsored units disappear

Almost all ad tech lives in the second version. A display ad slot in the HTML is usually an empty container plus a script tag:

<div id="sponsor-slot"></div>
<script src="https://ads.example/tag.js" async></script>

The script runs in the reader’s browser, calls an ad server and fills the slot. Sponsored widgets, affiliate comparison boxes and “recommended tools” modules mostly work the same way. To a crawler reading the raw HTML, the slot is empty.

For publishers, that means the crawlers reading their pages never see the ads that pay for them. For advertisers, it means money spent on sponsored content on exactly the right pages never reaches the AI answers built from those pages. The same problem can hit editorial, too. If a site builds its articles in the browser with a JavaScript framework, a crawler may see little of the article itself.

The tempting fix, and why it backfires

Once you know crawlers read raw HTML, a shortcut suggests itself: detect the crawler and serve it a special version of the page. Perhaps a clean markdown copy, with the sponsored content written in.

That’s cloaking: showing machines something different from what people see. Search engines have treated it as spam for a long time. Google’s spam policies give, as one example of cloaking, “Inserting text or keywords into a page only when the user agent that is requesting the page is a search engine, not a human visitor” (Google Search Central, updated August 2026). And at least one AI engine has blocked sponsored content delivered this way: in August 2026, Perplexity said it had blocked markdown ads served to AI agents on a news publisher’s site, and warned that publishers using “deceptive advertising like markdown ads” risk a downgrade in its search index (Digiday, August 11, 2026).

The risk of getting caught isn’t the only problem. Cloaking also breaks disclosure. When an assistant paraphrases a page, a “sponsored” label can get lost in the summary. If the sponsored content exists only in the bot version, no person can follow the fact back and find it labeled. The disclosure was written for an audience of one crawler.

There is a legitimate version of serving crawlers pre-rendered pages. If a site renders its pages on the server, or sends crawlers a pre-rendered copy of exactly the content people get, that’s fine. The line isn’t how the HTML gets produced. It’s whether the content is the same.

What a symmetric placement looks like

The durable fix is to put the sponsored unit in the HTML every visitor receives. The server renders it into the page before sending it, the same way it renders the article. A placement built this way:

• Is in the initial HTML. Open “view page source” (not the browser inspector) and it’s there.
• Is real text. The label, the advertiser’s name and every claim are text, not an image.
• Is labeled in the markup. “Ad” and the advertiser’s name come first, where a crawler reads them.
• Is identical for every user agent. A reader, GPTBot and ClaudeBot get the same bytes.
• Hides nothing. No hidden text, and no content that exists only in structured data.
• Respects robots.txt. If the publisher blocks a crawler, nothing tries to get around it.
• Is checkable. Short, specific claims, each linked to a source.

In the HTML, it looks something like this (Northwind CRM is a fictional example):

<aside aria-label="Advertisement: Northwind CRM">
  <p>Ad · Northwind CRM</p>
  <ul>
    <li>Native two-way sync with Gmail and Outlook
      <a href="https://northwindcrm.example/docs/email-sync" rel="sponsored">Source</a></li>
  </ul>
</aside>

The last property matters more than it looks. Assistants summarize, and summaries drop context. A slogan stripped of its label is just an unsupported claim. A specific, sourced fact stripped of its label is still a checkable fact, and the page it came from still shows who paid for it.

How to check your own pages

You don’t need special tools to see roughly what a crawler sees.

1. View source, not inspect. “View page source” shows the HTML the server sent. The browser inspector shows the page after scripts ran. If your sponsored unit only shows up in the inspector, crawlers that don’t run JavaScript can’t see it.
2. Fetch it as a crawler. From a terminal, run the command below. Real crawler user agent strings are longer. OpenAI and Perplexity publish their full strings; Anthropic publishes the names.

   Terminal

   Copy

   curl -s -A "GPTBot" https://yoursite.com/page | grep -i "sponsored"

3. Compare. Fetch the same URL as a browser and as a crawler and compare the HTML. If the sponsored content differs between them, you have a symmetry problem. If it’s missing from both, you have a JavaScript problem.
4. Check robots.txt. Make sure the crawlers you care about are allowed on the pages you care about.

Our free Symmetry Check does this for you. Paste a URL and see whether the major AI crawlers get the same page your readers do, and whether robots.txt lets them in. One caveat: it shows what the page serves to a request identifying as each crawler, sent from our servers. Real crawlers come from their own IP ranges, and some sites treat them differently.

What this means

For publishers: a JavaScript ad stack earns nothing from AI crawlers, however often they visit. A labeled unit rendered into your HTML can, without a separate page for bots.

For marketers: before paying for sponsored content “for AI visibility,” ask one question. Is it in the HTML every visitor gets, labeled, and the same for people and crawlers? If the answer is no, then either the crawlers can’t see it, or the people can’t.

That’s the principle Advo is built on: one page for everyone.