Skip to content
AI for SMBs

Technical Audit: Is Your Website Ready for AI Agents?

Wouter·21 August 2026·9 min read
TL;DR

This technical audit goes through six areas in your website's code: llms.txt, schema, Open Graph, semantic HTML, loading speed and robots.txt. Each area says which source is behind it. Where we check something without a source asking for it, we call that our own choice. For one area OpenAI and Anthropic name a consequence themselves: if you block OpenAI's search robot you are not shown in ChatGPT search answers, and if you block Anthropic's, your visibility in search results may go down.

ChatGPT, Perplexity and the AI features in Google show links to web pages in their answers. None of the three explains in full how they choose from those pages, in the documentation we read. What they do describe: which robots they send and what a site owner can control.

Corrected on 14 November 2026: this article said that according to Gartner (2025), 30% of all search queries in 2026 go through AI assistants, and 50% by 2028. We have not found a Gartner publication with those figures, and we could not retrieve Gartner's own pages. We have removed the paragraph.

At Mindsora we go through six technical areas in an audit. Below, each area says what we look at and which source is behind it. Where we check something without a source asking for it, we call that our own choice.

The six areas of the audit

1. llms.txt: a description of your site in plain text

llms.txt is a text file on your website that describes who you are and where the most important pages are. It comes from a proposal from 2024 that was revised in August 2026. The current version describes itself like this: "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website." Unlike robots.txt, it does not control access.

Example of a made-up business, in the format the proposal asks for:

# Van Dam Bakery
> Artisan bakery in Delft since 1987. Bread, pastries and cakes for consumers and hospitality.

## Pages
- [Order a cake](https://www.example.com/order): order form
- [Products](https://www.example.com/products)
- [Contact](https://www.example.com/contact)

What the file delivers is only partly known:

  • Google writes in its guide to generative AI in Search: "For Google Search, you can ignore tactics like "chunking" content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions."
  • Ahrefs looked at 137,210 domains that use its own analytics tool and received traffic in May 2026. About the llms.txt files it found there, Ahrefs writes: "97% of those files received zero traffic in May 2026. Nothing fetched them at all." Ahrefs adds that its customers skew more technical than the web at large, and for that reason calls its figure for how many sites have such a file an upper bound.

Our own choice: we check the file anyway. It takes little work, and Lighthouse, Chrome's auditing tool, has a check for it in a category Chrome itself calls experimental. If the file is missing, that check counts as not applicable. How to set it up is in our llms.txt guide.

Corrected on 14 November 2026: this section said that less than 2% of Dutch SMB websites have an llms.txt, that the file is a competitive advantage and that it will be a hygiene standard in two years. We have a source for none of the three. The Ahrefs study above covers a different group of sites.

2. Schema.org markup: structured data

Schema.org markup is code in your HTML that says in a fixed format: this is a business, these are the opening hours, this is the address, these are the services we offer.

Example: a LocalBusiness schema with name, address, phone number and opening hours, and a Service for each service.

What Google writes about it: for the AI features in Google Search, structured data is not a requirement. Google writes in its guide to generative AI in Search: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." According to the same guide it does help with being eligible for rich results in Google Search. In its documentation on AI features, Google gives as an example of SEO fundamentals that remain worthwhile: "Making sure your structured data matches the visible text on the page".

The pages of OpenAI, Anthropic and Perplexity about their robots say nothing about schema markup.

Our own choice: we check schema markup anyway, even though Google says it is not required for its AI features.

Corrected on 14 November 2026: this line said "cited up to 40% more often in AI answers", with a source we have not been able to find.

3. Open Graph tags: the title and the image of your page

The Open Graph protocol describes itself like this: "The Open Graph protocol enables any web page to become a rich object in a social graph." According to the same page it was created at Facebook. With the tags you state what your page is called, what kind of page it is, at which address it lives and which image goes with it. The protocol names four required properties: og:title, og:type, og:image and og:url.

Example:

<meta property="og:title" content="Van Dam Bakery, artisan bread from Delft">
<meta property="og:type" content="website">
<meta property="og:url" content="https://www.example.com/">
<meta property="og:image" content="https://www.example.com/og-image.jpg">
<meta property="og:description" content="Since 1987. Bread, pastries and cakes. Order online or visit the shop.">

We found in none of the sources in this article that AI services use these tags. Our own choice: we check them because it takes little work, and because the protocol records which title and which image belong to your page.

4. Semantic HTML: your page's structure

Semantic elements such as <header>, <main>, <article>, <section>, <nav> and <footer> say what a part of the page is. According to web.dev, a Google site for developers, agents see a website in three ways: "Agents can view your website in 3 primary ways: screenshots, raw HTML, and the accessibility tree." The last one, the accessibility tree, is made by the browser itself from the structure of your page. As an example of what an agent misses when it relies on one of the three, the same article says: in the HTML an agent might see a <div> without knowing you have configured it as a working button with CSS and JavaScript. According to the article, that is why agents combine the three.

That article is about agents that operate a site in a browser. It says nothing about search robots. The advice in that article is about buttons and links: use <button> and <a> in place of a modified <div> or <span>. The article does not name the six elements in the table. Checking those is our own choice, and the descriptions in the table are ours.

Element What the element indicates
<header> Navigation and logo live here
<main> This is the main content
<article> This is a standalone piece of content
<section> This is a thematic block
<nav> These are navigation links
<footer> Contact info and legal links
  • In order: semantic elements from which the structure of the page can be read.
  • Work: everything in <div> elements with no meaning.

5. Loading speed: the Core Web Vitals

Google measures the experience of visitors with three metrics, the Core Web Vitals: LCP (Largest Contentful Paint) for loading, INP (Interaction to Next Paint) for the response to what a visitor does, and CLS (Cumulative Layout Shift) for layout that shifts. The thresholds for "good" are on web.dev. The other two columns come from web.dev's pages on INP and CLS and from the Web Almanac below:

Metric Good Needs work Poor
LCP 2.5 s or less 2.5-4 s over 4 s
INP 200 ms or less 200-500 ms over 500 ms
CLS 0.1 or less 0.1-0.25 over 0.25

web.dev advises looking at these metrics at the 75th percentile of page loads, separately for mobile and desktop.

The Web Almanac 2025 by HTTP Archive shows how websites do on the first metric: "On phone devices, 62% of websites have a good LCP, 25% need improvement, and 13% perform poorly." We found no separate figures for Dutch SMB sites.

What this has to do with AI: Google writes that no specific optimization is required for AI Overviews and AI Mode, and gives as an example of SEO fundamentals that remain worthwhile: "Providing a great page experience for users". In its guide Google counts "reducing latency" as part of that experience. A time limit within which an AI robot must have loaded your page is not in the pages of OpenAI, Anthropic, Perplexity and Google that we read for this article.

Corrected on 14 November 2026: this section said AI agents have timeouts and that a page that takes more than 3 seconds to load may not be fully indexed, and that the average Dutch SMB website has an LCP of 4.2 seconds (HTTPArchive, 2025). We found a source for neither. The table listed FID; that metric was replaced by INP in March 2024.

6. robots.txt: which robots do you allow?

Your robots.txt says which robots may fetch your site. What a block costs depends on which robot you block.

Good:

User-agent: *
Allow: /

This can cost you visibility in AI search:

User-agent: OAI-SearchBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /

Watch which robot you block. GPTBot and ClaudeBot collect content that may be used for training. For search, OpenAI uses a different robot: according to OpenAI, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic names Claude-SearchBot for search, and Claude-User for pages Claude fetches when a user asks something. According to Anthropic, blocking either one may reduce your visibility in search results.

Allowing search and blocking training can go side by side. According to OpenAI, a site owner "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot".

For its own search engine, Google also names the layer in front of robots.txt: "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure". OpenAI recommends also allowing requests from its published IP ranges. So ask your web developer as well if a CDN or firewall is stopping these robots.

Corrected on 14 November 2026: this line said you won't appear in ChatGPT and Claude if you block GPTBot and ClaudeBot. According to OpenAI and Anthropic those are the robots for training.

What we note in an audit

Per area we note if it is in order or if there is work to do. That classification is our own:

  • llms.txt. In order: present, in the format of the proposal. Work: missing, or present without links.
  • Schema markup. In order: present and matching the visible text. Work: missing, or different from the page.
  • Open Graph. In order: title, type, address and image. Work: missing or incomplete.
  • Semantic HTML. In order: elements with meaning. Work: everything in div elements.
  • Loading speed (LCP). In order: 2.5 seconds or less. Work: over 2.5 seconds.
  • robots.txt. In order: search robots allowed. Work: search robots blocked.

Whether a site that is in order on all six is mentioned more often in AI answers, and how much more often, we have not measured.

Corrected on 14 November 2026: this line said "cited 5-8x more often in AI answers on average". We have no measurement or source for that.

How AI-ready is your business?

Take the free AI Readiness Quiz: 7 questions, 2 minutes.

Take the quiz →

What did we find on our own site?

We went through mindsora.io in November 2026. For three of the six areas, this is what we found:

  • llms.txt. Our file existed, and Lighthouse failed it: it did not contain a single Markdown link. It now has 69. Of the 130 lines in the file that start with a dash, 82 still have no link.
  • Loading speed. A lab measurement with PageSpeed Insights on 13 November 2026 gave the homepage on mobile an LCP of 2.3 seconds and a CLS of 0. There is no field data. The site has too little traffic for that.
  • robots.txt. Our file allows OAI-SearchBot, PerplexityBot, Googlebot and Bingbot.
  • This article itself. The previous version cited figures without a source and a metric Google has replaced. What we have taken back is in the lines starting with "Corrected".

What does the Agent-Ready Audit do?

At Mindsora, we offer the Agent-Ready Audit. In a detailed report, we show exactly where your website stands on all six areas, with concrete recommendations and priorities.

Agent-Ready Audit: what you get

  • Score per area (1-10) with explanation
  • Comparison with your direct competitors
  • Priority list: what delivers the most impact to fix first
  • Estimated impact on AI discoverability
  • Implementation instructions (DIY or let us handle it)

The price is on the services page.

Measure it yourself

Our free Agent-Ready Grader tests five of the six areas: llms.txt, structured data, Open Graph, semantic HTML and robots.txt. It also looks at the title, the meta description and the canonical of your page. It does not measure loading speed. For that there is Google's PageSpeed Insights.

The score from 0 to 100 follows our own weighting: 20 points for an llms.txt and 30 for structured data, two things Google does not require for its AI features. Apart from the score it requests your homepage three times, each time with a user-agent string that carries the name of one robot: OAI-SearchBot, Claude-SearchBot and CCBot. It does not test whether an AI answer cites you.

Caveats on the grader. If you block GPTBot and allow OAI-SearchBot, the grader still reports that your robots.txt blocks AI crawlers. According to OpenAI that setup is possible. If you block only Claude-SearchBot or Claude-User, the robots.txt check reports no block, because it does not know those names. The same goes for Googlebot and Bingbot. And for two areas the grader checks something other than the list above: for Open Graph the title, description and image, and for llms.txt only that the file is there.


Want to work out how many hours a recurring task costs you? The free AI Readiness Scan does that: 7 questions about your business processes.

Ready to automate?

Book a free consultation and discover what AI automation can do for your business. Or take the 2-minute AI Readiness Quiz first.

Book a call

More articles