# Make a website easy for agents to find, read, and use.

> An evidence-based guide to crawler access, HTML, structured data, Markdown, llms.txt, feeds, and real tool descriptions.

By George Kelly · Published 2026-09-05 · Updated 2026-09-05

Canonical: https://www.iamgeorgekelly.com/field-guide/agent-discoverability

## Short answer

Start with public, indexable HTML, stable URLs, internal links, and clear text. Add Markdown and a concise llms.txt index as alternate reading paths. Describe real tools with real contracts. No discoverability file guarantees an AI citation or referral.

## The foundation is still the published page

Google documents that its AI search features use the existing search foundation: pages need to be indexable and eligible for a snippet. It does not require special AI text files or a special schema. Structure the visible answer so a person or retrieval system can identify the question, the answer, and the evidence.

For this library, each guide has a short answer, named sections, definitions, an applicable example or decision table, sources, and a visible editorial date. These are publishing choices intended to reduce ambiguity. They are not a claim that a particular heading length or paragraph shape is a ranking factor.

Sources: [Google Search: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

## Distinguish search discovery from model training

OpenAI documents separate OAI-SearchBot and GPTBot purposes for search and training. Anthropic also distinguishes search, user-requested retrieval, and training crawlers. Perplexity documents search and user-triggered retrieval agents. A site owner should set the intended policy per purpose, then check that hosting or firewall rules do not contradict it.

A permissive robots.txt is permission to crawl, not evidence that a crawler visited. Some user-initiated fetchers have different robots behavior from automated search crawlers. Consult the provider’s current documentation rather than treating every bot name as equivalent.

Sources: [OpenAI: Overview of crawlers](https://developers.openai.com/api/docs/bots); [Anthropic: Web crawler controls](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler); [Perplexity: Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)

## Give each format a useful job

HTML remains the canonical publication. Markdown removes navigation and presentation overhead for tools that prefer text. A JSON index lets a client enumerate the corpus without scraping cards. Feeds expose publication and revision information to subscribers. Keep these exports derived from the same record so they do not disagree.

| Surface | Useful job | Boundary |
| --- | --- | --- |
| robots.txt | Declare automated crawler access policy | Does not force crawling or index inclusion. |
| sitemap.xml | List canonical public pages | Discovery hint, not an indexing guarantee. |
| Article and breadcrumb structured data | Describe visible authorship and page relationships | Must agree with the actual page. |
| llms.txt | Offer a small curated reading index | An emerging convention, not a universal AI search requirement. |
| Markdown and JSON | Provide alternate representations of the published content | Keep sources, caveats, and dates in every representation. |
| RSS and JSON Feed | Expose a subscription and update stream | Feeds do not themselves create an audience. |
| OpenAPI | Describe an implemented callable API | Do not advertise tools that do not exist. |

## Where llms.txt helps—and what is still unproved

The llms.txt proposal describes a concise Markdown guide to a site’s useful material. Chrome’s Lighthouse documentation now includes an audit for it and explicitly treats the file as optional. That is evidence of practical tooling support. It is not evidence that adding the file increases rankings or guarantees citations.

This site publishes a curated llms.txt, a full text bundle, and per-guide Markdown. The aim is to make a deliberate agent visit efficient. A source link resolves to the same content a person can read; there is no separate set of promotional claims hidden for bots.

Sources: [The llms.txt proposal](https://llmstxt.org/); [Chrome Lighthouse: llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)

## Make the destination worth opening

A useful answer can earn a citation and still remove the need to visit. A site should also offer something a reader can do: compare assumptions, calculate a result, download a working example, or inspect the underlying evidence. This library pairs the profit guide with a local calculator and a documented read-only calculation API.

Do not add agent descriptors merely because their filenames sound relevant. An OpenAPI document describes an HTTP API. An MCP server or an A2A agent has a different runtime contract. A portfolio with a content library does not become either service by publishing a manifest.

## Keep the evidence current as the library grows

Record the source behind provider-specific behavior and the date it was checked. Update the editorial date when the substance changes, not on every deployment. Validate all exported representations and the canonical live URLs after a release.

Use a topic backlog built around distinct decisions rather than automatically generating near-identical keyword pages. Publish when a page contributes a new framework, worked example, artifact, or verified explanation. Google warns that generating many pages without added value can fall within scaled-content abuse; production volume alone is not the objective.

Then measure the result in separate layers: access, citations, referrals, and qualified outcomes. The measurement guide explains what each observation can and cannot establish.

Sources: [Google Search: Using generative AI content](https://developers.google.com/search/docs/fundamentals/using-gen-ai-content)

## Sources and scope

- [Google Search: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) — Supports baseline search requirements and the absence of a special AI file requirement. Checked 2026-09-05.

- [OpenAI: Overview of crawlers](https://developers.openai.com/api/docs/bots) — Documents separate search, training, and user-initiated crawler purposes. Checked 2026-09-05.

- [Anthropic: Web crawler controls](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Documents Anthropic crawler purposes and site-owner controls. Checked 2026-09-05.

- [Perplexity: Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) — Documents search and user-initiated retrieval agents. Checked 2026-09-05.

- [The llms.txt proposal](https://llmstxt.org/) — An emerging convention for a concise site index and alternate text representations. Checked 2026-09-05.

- [Chrome Lighthouse: llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) — Documents optional llms.txt audit behavior; does not establish a ranking benefit. Checked 2026-09-05.

- [Google Search: Using generative AI content](https://developers.google.com/search/docs/fundamentals/using-gen-ai-content) — Supports prioritizing added value over mass generation without value. Checked 2026-09-05.

## Related reading

- [Measure AI visibility without confusing it with traffic.](https://www.iamgeorgekelly.com/field-guide/ai-visibility-measurement/index.md)

- [Product data an agent can use without guessing.](https://www.iamgeorgekelly.com/field-guide/product-data-for-agents/index.md)

- [Agentic commerce starts with an operating model.](https://www.iamgeorgekelly.com/field-guide/agentic-commerce/index.md)

## Editorial note

AI-assisted research and drafting. Provider-specific claims link to primary sources. Frameworks are editorial proposals; worked examples are illustrative and are not employer performance results.