The Short Version
TL;DR
→AI crawlers don’t run JavaScript. Vercel logged 500 million-plus crawls with zero JS executed. GPTBot, ClaudeBot, and PerplexityBot read only the raw HTML.
→AI-referred visitors convert at 14.2% versus 2.8% for a standard Google click. You only get that traffic if the machine can read your site.
→Three 60-second checks: View Source for your headline, confirm server-side rendering, and open robots.txt for the AI search bots. Rendering, structure, and access are usually fixable — unless a page-builder is the ceiling.
Start Here: 60 Seconds
Answer these three honestly. Count your yes’s.
1Open your homepage, right-click, and choose “View Page Source” (Ctrl+U on Windows, Cmd+Option+U on Mac). Search that raw code for your main headline. Is the exact text there?
2Do you know for a fact your core content is server-side rendered, baked into the page before it loads, not drawn in afterward by code?
3Have you confirmed your robots.txt allows the AI search crawlers (OAI-SearchBot, Claude-SearchBot)?
0 yes: you’re likely invisible to AI search. Start at Rendering. 1–2 yes: partial. Work the section you failed. All 3: you’ve cleared a bar most contractors haven’t.
Why This Is Worth Sixty Seconds
AI-referred visitors convert at 14.2% versus 2.8% for a standard Google click (Lantern, 2026). The traffic that comes out of ChatGPT and Perplexity is far better than a cold Google click, but you only get it if the machine can read your site in the first place.
14.2% conversion on AI-referred visits vs 2.8% for a Google click.
Lantern, 2026
And most can’t. Vercel analyzed 500 million-plus AI-crawler fetches and found zero JavaScript executed. GPTBot, ClaudeBot, and PerplexityBot read only the raw HTML that arrives before any code runs. That’s the problem: templated sites assemble their content with JavaScript after the page loads, and that covers most Wix, Squarespace, and WordPress page-builder builds. The crawler shows up, the code hasn’t run yet, and it sees a near-empty page where your services and service areas should be.
500 million+ AI-crawler fetches. Zero JavaScript executed.
Vercel & MERJ — The Rise of the AI Crawler
Sources: Vercel & MERJ — The Rise of the AI Crawler · Lantern — AI Crawlers Do Not Render JavaScript
The Do’s and Don’ts
01 — Rendering: Can the Machine Read Your Words?
Do
✓Serve your core content (services, service areas, phone, answers) in the raw HTML, before any code runs
✓Ask your developer to server-side render or pre-render key pages (their terms: SSR, SSG, prerendering)
✓Judge the page by View Source, not the browser preview. The raw code is what bots receive
Don’t
✗Rely on JavaScript to draw in your main content after load. AI crawlers never see it
✗Assume “it looks fine in my browser” means AI sees it. Your browser runs the code; the bots don’t
✗Assume ranking on Google means AI can read you. Google renders JavaScript; the AI crawlers don’t
02 — Structure: Can It Understand What It Reads?
Do
✓Put one clear, direct answer near the top of every service page. Lead with it
✓Use real headings and lists so the structure itself is machine-readable, not just styled text
✓Add LocalBusiness and FAQ schema (machine-readable tags describing your business) as a supporting layer
Don’t
✗Expect schema alone to buy citations. It reinforces visible text; it can’t replace thin content
✗Bury the answer four paragraphs down. The old SEO instinct is the AI-skip instinct
✗Build pages as one unlabeled block. Parsers can’t tell your services from your footer
Sources: Ahrefs via Stan Ventures — Schema markup has no meaningful standalone lift in AI citations
03 — Access: Are You Locking the Bots Out?
Do
✓Open yourdomain.com/robots.txt and confirm the search bots are allowed (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
✓Separate the bots if you want: block the training crawlers while allowing the search ones
✓Keep your Bing house in order: roughly 92% of ChatGPT’s web answers pull from Bing’s index
Don’t
✗Blanket-block every AI bot and quietly kill your eligibility to be cited at all
✗Rely on llms.txt as an access gate. As of 2026 it has no confirmed effect on what AI retrieves
✗Set robots.txt once at launch and never look again. Plugins and platform updates quietly rewrite it
Sources: DigitalApplied — AI Crawler Access Control 2026
The honest part
The platform can still be the ceiling.
Rendering, structure, and access are usually fixable without leaving your platform.
But if your site is built on a page-builder that injects content with JavaScript, you may be patching around a ceiling the platform itself sets. That’s a build problem, not a copy problem. Last week’s five website leaks ended on this same ceiling.
From the team at DUO
Rendering, structure, and access we can fix on almost any site, and we’ll tell you honestly which ones don’t need a rebuild. But when a page-builder is the thing standing between you and the AI crawlers, no amount of copy tweaking clears it. We build custom-coded sites that render as clean HTML the machines read natively, findable for the future of search, not just today’s Google. Want to see exactly what ChatGPT and Claude pull from your site right now? We’ll run the crawl and show you.
Show me what AI sees on my site →Do this today
- ☐Run the View Source test on your homepage now: Ctrl+U (Windows) or Cmd+Option+U (Mac), then search for your headline.
- ☐Open yourdomain.com/robots.txt. Confirm OAI-SearchBot and Claude-SearchBot are not disallowed.
- ☐Ask your developer or platform one question: “Is our main page content server-side rendered, or drawn in by JavaScript?”
- ☐Verify LocalBusiness schema exists on your homepage and top service pages.
- ☐Pick your worst-scoring section above and make time for one fix this week.
