Use robots.txt to set public and private boundaries
A good robots.txt file should allow public editorial and product pages while blocking app, admin, checkout, API and duplicate query-parameter URLs.
Review rules for traditional crawlers and AI-related user agents, but remember that robots.txt is a crawl directive rather than a security control.
Add llms.txt for compact AI guidance
llms.txt gives AI systems and agents a compact summary of the site: what the product does, which URLs matter, which pages should be preferred and which private paths should be ignored.
It should link to canonical public pages, guide hubs, tools, product descriptions and policy pages. Keep it concise and accurate.
Keep sitemaps complete and focused
The XML sitemap should include canonical, indexable public pages with current lastmod dates. Avoid adding app, checkout, API, admin, thin search result pages or parameter URLs.
Use schema to clarify entities
Organization, WebSite, BreadcrumbList, Article, FAQPage, SoftwareApplication and Service schema can help machines connect pages to the right brand and topic when the structured data reflects visible content.
FAQ
Should I allow all AI crawlers?
Most sites should allow public marketing, guide and product pages while keeping private, app, checkout and API paths blocked. The exact policy depends on your business and content strategy.
Is llms.txt required for AI visibility?
No. It is not a universal ranking requirement, but it can provide a helpful machine-readable summary and canonical page map.
Should API routes be indexed?
No. API, account, dashboard and checkout routes should usually be blocked from crawling and excluded from sitemaps.