robots.txt Generator
Select crawler groups, add your sitemap URL and copy a ready-to-edit robots.txt draft.
Preset
How it works
Create crawler rules in three steps.
1. Choose bots
Choose the search and AI-related user agents you want to address.
2. Add sitemap
Add your sitemap so crawlers can discover canonical URLs faster.
3. Copy output
Publish the output as /robots.txt at the root of your domain.
Related guides
Learn more
AI Bots & robots.txt
Learn the concept behind this tool and strengthen your AI visibility setup.
llms.txt Guide
Learn the concept behind this tool and strengthen your AI visibility setup.
Schema.org & JSON-LD
Learn the concept behind this tool and strengthen your AI visibility setup.
FAQ
robots.txt Generator FAQ
Should I allow AI bots?
For AI search visibility, crawler access should be intentional. Allow the bots that fit your strategy and block only where you have a clear reason.
Where should robots.txt be uploaded?
Publish it at the root of your domain, for example https://example.com/robots.txt.