Why your website is invisible to AI
Four common reasons the AI systems cannot see your website, and how to check each one in ten minutes.
In short
- Content loaded with JavaScript is invisible to most AI crawlers.
- Many hosts and security products block them automatically, without you knowing.
- Content behind a login or a form does not get read.
- Pictures of text are unreadable. A price list that is a PDF or an image is invisible.
- You can check three of these yourself in ten minutes.
A business that never gets named in AI answers usually assumes the problem is the content. Sometimes it is technical and simpler than that: the machine cannot reach the content at all.
1. The text loads with JavaScript
Plenty of modern sites send an empty page and then fill it in with JavaScript. Google can usually cope with that. Most AI crawlers do not run JavaScript, so what they see is an empty page.
The test: switch JavaScript off in your browser and load the site. Whatever disappears is invisible to the machine.
2. robots.txt blocks them
Go to yourdomain.is/robots.txt. If it says Disallow for GPTBot, ClaudeBot, PerplexityBot or Google-Extended, then someone, often the content management system itself, has shut them out.
Sometimes that is a deliberate decision to protect content. But if you want people to find you through AI, you are closing the door you want them to walk through.
3. The hosting or the firewall blocks them
Some security products and content delivery networks block AI crawlers automatically, and that setting is often the default without anyone having chosen it. It does not show up in robots.txt, you have to look in the settings at your provider.
4. The content is hidden
- A price list as a PDF or an image. The machine reads it badly or not at all.
- Information behind a login or a form.
- Important text inside images, for example scanned menus.
- Content that only appears when you click a tab that loads it afterwards.
The ten minute test
- Switch JavaScript off and load your front page. Is the text there?
- Open yourdomain.is/robots.txt and look for Disallow.
- Search for your most important prices on Google with a site: search. Do they come up as text?
- Ask ChatGPT what your business does and what it costs. Compare the answer with reality.
If any of these fail, it is fixable, usually in a few hours. That is far cheaper than writing more articles nobody reads.
Frequently asked questions
Should I let AI in?
If you make a living from people finding you, then yes. If you sell the content itself, subscription material for instance, that is a different discussion.
Does fixing this hurt my speed?
No, quite the opposite. A site that sends real text straight away is both faster for people and more readable for machines.
What if my site is on a closed platform?
You can still fix robots.txt and add llms.txt. If the platform will not send text in the HTML, that is a bigger decision and we can work through it with you.
Related service
Vefrækt and visibility
This is the service this article relates to. Fixed price and a first draft within 48 hours.
See the serviceYour business in ChatGPT: how AI decides who it recommends
More and more people ask an AI instead of googling. Here is how these systems pick who they recommen…
ReadWhat is GEO? Search optimization when the search engine is an AI
SEO is about a position in a list. GEO is about being the source the machine cites. Here is the diff…
Readllms.txt and agents.md: the files that tell AI who you are
Two small text files that take half an hour to write and tell AI systems exactly what your business …
ReadAbout the author
Kristján Gilbert is co-founder of galdrasmiðjan. He builds websites and online stores for Icelandic companies and writes here about what actually works: speed, search, visibility in AI answers and automation.
Questions about this article go to hallo@galdrasmidjan.is.
Want us to look at your site?
We take a short call, look at where things stand and tell you what is worth fixing first. It costs nothing.
Get a fixed quote