|
Hi there,
You can write perfectly extractable content and still be invisible — because the agent never got to read it. This edition is the plumbing underneath: the infrastructure that decides whether an AI system can access and afford your pages at all.
Failure that never shows up in analytics
When an agent can't parse or fit your page, nothing errors. No 404, no alert. In your analytics it looks like a 0.4-second visit with 100% bounce — indistinguishable from a bored human. Beatrice calls this silent failure, and it is the default state of the web right now. You lose the opportunity and never know it happened, because traditional analytics were never built to see an agent.
Silent failure is the default state of the web. The only thing that changed is whether your site was built for this kind of visitor. |
Agents parse tokens against a hard budget
Agents don't browse; they read sequentially against a fixed token budget. A token is about four characters, so you can estimate any page as characters ÷ 4 — paste a page into a plain-text editor, read the character count, divide by four. Past the ceiling, content is truncated or skipped, not politely summarised. In one of Beatrice's worked examples a bloated page comes back at 193,000 tokens against a 200,000 window and is silently discarded, producing a confident, wrong recommendation that never mentions the best option — you.
Three fixes, in order
You don't fix this all at once. You fix it in sequence. First, robots.txt: a blanket block written years ago, or one that only allows Googlebot, is the number one silent killer of agent traffic — add explicit allow rules for the crawlers you want (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Second, publish an llms.txt: a Markdown index of your key pages with a one-line, outcome-focused description and a token count each, so an agent can budget its context before fetching. Third, make high-value pages lean — front-load the answer in the first 100 tokens, and use tables, which cost roughly 80% fewer tokens than the same data in prose. The full walk-through is here: your robots.txt might be the silent killer of your AI traffic.
Four ways a page dies over budget
“Too heavy” is not one failure, it is four — and none of them fire an alert. The page can be truncated (cut at the token limit mid-thought), skipped entirely (the agent decides it can't afford it), lost to chunking overhead (the useful part never reaches the context window), or it can trigger a hallucination (the model answers from a stale memory instead of your current page). The remedy for all four is the same discipline: fewer tokens, answer first.
It also helps to know who is actually knocking. The crawlers to recognise in your logs and robots.txt are GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (which feeds Gemini and AI Overviews) and CCBot (Common Crawl). Adding explicit allow rules for the ones you want — and confirming with a quick curl using their user-agent that you get real content back — is a ten-minute job that most sites simply never did.
For the pages themselves, an inverted pyramid is the reliable template: the first ~100 tokens are a one-sentence summary of the answer, the next ~200 say what the reader will accomplish, then prerequisites, then the deep explanation. A human can still scroll for depth; an agent that reads only the first 500 tokens already has something it can act on. Test it literally — read the first 500 tokens of a page and ask whether that is enough to act on.
The most advanced move here is the skill.md file. Where llms.txt says where your content lives, skill.md declares what your product or service actually does, so an agent can match its task before spending a single token reading your site. In Beatrice's worked example that collapses a decision from tens of thousands of tokens to a few hundred and lifts accuracy from the 70s into the high 90s — the 150× figure on the graphic above.
Finally, treat this as maintenance, not a one-off. The teams that stay visible add a line to their publishing checklist: update llms.txt whenever a meaningful page ships, keep the human and the agent version of a page in step, and measure token counts before announcing anything. Silent failure is silent precisely because nothing tells you when a page slipped over budget — so you build the check into the process rather than waiting for a drop that never shows up.
|
✎ Exercise 1 — the 20-minute infrastructure audit
- Select all on your homepage, paste into a plain-text editor, read the character count, divide by 4. North of ~25,000 tokens? That page needs restructuring.
- Open yourdomain.com/robots.txt and look for rules blocking AI crawlers. Unblock the ones you want to reach.
- Move the actual answer on your five highest-value pages into the first 100 words.
- Find one long paragraph of specs or comparisons and turn it into a table.
|
|
✎ Exercise 2 — ship your first llms.txt
- List your 10–15 most important pages, organised by task/intent (not by nav menu).
- Give each a one-line description focused on the outcome a reader gets.
- Estimate a token count per entry (characters ÷ 4) and keep the whole file under ~5,000 tokens.
- Upload it to your site root and load the URL to confirm it resolves. Re-check it whenever you publish something important.
|
In the course, Beatrice takes this further — the skill.md capability file (up to a 150× token reduction in her worked example) and a full three-week rollout you can run as a literal project plan.
Next week: structured data — and why, to an agent, a missing field is a disqualifier.
— Lazarina
P.S. The characters ÷ 4 trick is worth automating in your build pipeline once. It turns “is this page too heavy for an agent?” into a number you can watch.
|
