robots.txt, sitemap.xml, and llms.txt — The 2026 Crawl Bar

Crawl files are product

A sitemap that 200s HTML is a lie. A robots that blocks CSS on Googlebot-Image is how share cards vanish.

Minimum bar

Allow /. Sitemap with every public URL. lastmod. Image loc for the OG card. JSON-LD on the HTML those locs point at.

LLM-friendly without giving away ops

llms.txt: who we are, what we fix, how to contact, article list. No secrets, no fake review counts.

This host

asap-today.com ships robots, sitemap, llms.txt, and llms-full.txt for the article bodies.

Questions

Is llms.txt official Google?

No. It is how answer engines and researchers find a clean summary. Google still wants robots + sitemap + HTML.

Should I Disallow /privacy?

No. Privacy should be crawlable. We Allow it.

More notes

• Website Down Today: What We Fix in the First Hour
• SSL Certificate Expired: Browsers Block, Calls Stop
• Contact Form Not Sending: The Quiet Lead Leak
• DNS Cutover Same Day Without a Mystery Outage