The short answer
Start with public, indexable HTML, stable URLs, internal links, and clear text. Add Markdown and a concise llms.txt index as alternate reading paths. Describe real tools with real contracts. No discoverability file guarantees an AI citation or referral.
The foundation is still the published page
Google documents that its AI search features use the existing search foundation: pages need to be indexable and eligible for a snippet. It does not require special AI text files or a special schema. Structure the visible answer so a person or retrieval system can identify the question, the answer, and the evidence.
For this library, each guide has a short answer, named sections, definitions, an applicable example or decision table, sources, and a visible editorial date. These are publishing choices intended to reduce ambiguity. They are not a claim that a particular heading length or paragraph shape is a ranking factor.
Distinguish search discovery from model training
OpenAI documents separate OAI-SearchBot and GPTBot purposes for search and training. Anthropic also distinguishes search, user-requested retrieval, and training crawlers. Perplexity documents search and user-triggered retrieval agents. A site owner should set the intended policy per purpose, then check that hosting or firewall rules do not contradict it.
A permissive robots.txt is permission to crawl, not evidence that a crawler visited. Some user-initiated fetchers have different robots behavior from automated search crawlers. Consult the provider’s current documentation rather than treating every bot name as equivalent.
Source: OpenAI: Overview of crawlers; Anthropic: Web crawler controls; Perplexity: Crawlers
Give each format a useful job
HTML remains the canonical publication. Markdown removes navigation and presentation overhead for tools that prefer text. A JSON index lets a client enumerate the corpus without scraping cards. Feeds expose publication and revision information to subscribers. Keep these exports derived from the same record so they do not disagree.
| Surface | Useful job | Boundary |
|---|---|---|
| robots.txt | Declare automated crawler access policy | Does not force crawling or index inclusion. |
| sitemap.xml | List canonical public pages | Discovery hint, not an indexing guarantee. |
| Article and breadcrumb structured data | Describe visible authorship and page relationships | Must agree with the actual page. |
| llms.txt | Offer a small curated reading index | An emerging convention, not a universal AI search requirement. |
| Markdown and JSON | Provide alternate representations of the published content | Keep sources, caveats, and dates in every representation. |
| RSS and JSON Feed | Expose a subscription and update stream | Feeds do not themselves create an audience. |
| OpenAPI | Describe an implemented callable API | Do not advertise tools that do not exist. |
Where llms.txt helps—and what is still unproved
The llms.txt proposal describes a concise Markdown guide to a site’s useful material. Chrome’s Lighthouse documentation now includes an audit for it and explicitly treats the file as optional. That is evidence of practical tooling support. It is not evidence that adding the file increases rankings or guarantees citations.
This site publishes a curated llms.txt, a full text bundle, and per-guide Markdown. The aim is to make a deliberate agent visit efficient. A source link resolves to the same content a person can read; there is no separate set of promotional claims hidden for bots.
Source: The llms.txt proposal; Chrome Lighthouse: llms.txt audit
Make the destination worth opening
A useful answer can earn a citation and still remove the need to visit. A site should also offer something a reader can do: compare assumptions, calculate a result, download a working example, or inspect the underlying evidence. This library pairs the profit guide with a local calculator and a documented read-only calculation API.
Do not add agent descriptors merely because their filenames sound relevant. An OpenAPI document describes an HTTP API. An MCP server or an A2A agent has a different runtime contract. A portfolio with a content library does not become either service by publishing a manifest.
Keep the evidence current as the library grows
Record the source behind provider-specific behavior and the date it was checked. Update the editorial date when the substance changes, not on every deployment. Validate all exported representations and the canonical live URLs after a release.
Use a topic backlog built around distinct decisions rather than automatically generating near-identical keyword pages. Publish when a page contributes a new framework, worked example, artifact, or verified explanation. Google warns that generating many pages without added value can fall within scaled-content abuse; production volume alone is not the objective.
Then measure the result in separate layers: access, citations, referrals, and qualified outcomes. The measurement guide explains what each observation can and cannot establish.
Sources & scope
Primary references checked for this edition. The notes below distinguish source-backed facts from the frameworks and examples proposed in this guide.
- Google Search: AI features and your website ↗
Supports baseline search requirements and the absence of a special AI file requirement.
Checked September 5, 2026 - OpenAI: Overview of crawlers ↗
Documents separate search, training, and user-initiated crawler purposes.
Checked September 5, 2026 - Anthropic: Web crawler controls ↗
Documents Anthropic crawler purposes and site-owner controls.
Checked September 5, 2026 - Perplexity: Crawlers ↗
Documents search and user-initiated retrieval agents.
Checked September 5, 2026 - The llms.txt proposal ↗
An emerging convention for a concise site index and alternate text representations.
Checked September 5, 2026 - Chrome Lighthouse: llms.txt audit ↗
Documents optional llms.txt audit behavior; does not establish a ranking benefit.
Checked September 5, 2026 - Google Search: Using generative AI content ↗
Supports prioritizing added value over mass generation without value.
Checked September 5, 2026
AI-assisted research and drafting. Provider-specific claims link to primary sources. Frameworks are editorial proposals; worked examples are illustrative and are not employer performance results.
Editorial policy & corrections ↗