Improve agent/LLM discoverability of the docs site#24831
Open
dvdksn wants to merge 3 commits intodocker:mainfrom
Open
Improve agent/LLM discoverability of the docs site#24831dvdksn wants to merge 3 commits intodocker:mainfrom
dvdksn wants to merge 3 commits intodocker:mainfrom
Conversation
The CloudFront Lambda rewrote `/` with `Accept: text/markdown` to an invalid `.md` URI, causing a 502 LambdaValidationError. Homepage has no flattened markdown equivalent, so route to `/llms.txt` — the agent-oriented markdown index of the site. Same handling for `/index.html`. Also set `Content-Type: text/markdown` on `llms.txt` at upload time, so content negotiation responses and direct requests both advertise the correct media type. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add Content-Signal directive opting in to AI training, search indexing, and AI input (RAG/grounding). Docker's official docs benefit from wide distribution through AI assistants, so permitting training and grounding is aligned with the site's purpose. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Publish an MCP Server Card at /.well-known/mcp/server-card.json pointing at https://mcp-docs.docker.com/mcp, so agents doing MCP discovery can find the official Docker docs endpoint. Mirror the serverInfo and capabilities reported by the server's initialize response. Also add a one-line pointer near the top of llms.txt so LLMs fetching the index learn about the structured-access alternative. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
✅ Deploy Preview for docsdocker ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
Author
|
/review |
![docker-agent[bot]](https://avatars.githubusercontent.com/in/2787757?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Assessment: 🟢 APPROVE
This PR correctly addresses homepage markdown content negotiation (CloudFront Lambda + Content-Type header on llms.txt), adds AI content-usage preferences to robots.txt, and publishes an MCP Server Card. The changes are well-scoped and accurate.
One minor note: the version field in the new MCP Server Card JSON is hardcoded. See inline comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Address legitimate findings from an agent-readiness audit of docs.docker.com:
/withAccept: text/markdownto an invalid.mdURI, returning a 502. It now routes to/llms.txt, and the deploy workflow setsContent-Type: text/markdownon that file so the response media type matches the request.Content-Signal: ai-train=yes, search=yes, ai-input=yestorobots.txt. Docker's official docs benefit from being trained on and used as RAG input./.well-known/mcp/server-card.jsonpointing athttps://mcp-docs.docker.com/mcp, and adds a pointer inllms.txt.Out of scope
Several audit findings don't apply to a docs site (OAuth/OIDC discovery, OAuth Protected Resource Metadata, WebMCP — we have no protected APIs or site "tools" to expose) or are too early-stage to invest in (Agent Skills index).
Link headers (RFC 8288) deserves a specific note: the audit flags their absence as a pattern, but docs.docker.com is a multi-product site without a single canonical API to advertise via
rel="service-doc"/rel="service-desc". Agents that follow thellms.txtconvention already find/llms.txtwithout a hint, and MCP clients now find the server via the.well-known/mcp/server-card.jsonadded here. Adding Link headers would also require a new viewer-response Lambda trigger — real infrastructure cost for no marginal discovery value. Skipped intentionally.Generated by Claude Code