Robots.txt Generator
Build your robots.txt for search and AI crawlers by choosing options; add blocked folders and your sitemap, then download it.
| Crawler | Default | Allow | Block |
|---|---|---|---|
GooglebotGoogle |
|||
BingbotBing |
|||
YandexBotYandex |
|||
DuckDuckBotDuckDuckGo |
|||
Googlebot-ImageGoogle Images |
|||
GPTBotOpenAI (training) · AI |
|||
OAI-SearchBotChatGPT search · AI |
|||
ChatGPT-UserChatGPT browsing · AI |
|||
ClaudeBotAnthropic (training) · AI |
|||
Claude-UserClaude browsing · AI |
|||
Google-ExtendedGemini training · AI |
|||
PerplexityBotPerplexity · AI |
|||
CCBotCommon Crawl · AI |
|||
Applebot-ExtendedApple AI training · AI |
This tool runs entirely in your browser. What you enter is not sent to a server, not saved and not stored in cookies.
robots.txt is a small text file at the root of your site that tells bots which parts to crawl. One wrong line can take a whole site out of search; a good file points crawl budget at the pages that matter and lets you decide how much access AI crawlers get.
How to use it
- Pick the default rule for all bots; for a live site it is almost always “allow”.
- Write the paths you do not want crawled, one per line (e.g. /admin/, /cart/).
- Add your sitemap URL.
- If needed, allow or block crawlers one by one; download the file and upload it to the site root.
Frequently asked questions
Where does robots.txt go?
At the root of the domain: site.com/robots.txt. A file in a subfolder is ignored; every subdomain needs its own.
Can I remove a page from Google with robots.txt?
No. A blocked page can still appear if other pages link to it. To remove it, allow crawling and add a noindex tag.
Should I block AI crawlers?
It is your call. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) limits use of your content for model training. Blocking crawlers that fetch pages for answers (OAI-SearchBot, ChatGPT-User, PerplexityBot) can reduce your visibility in AI search.
Should I use Crawl-delay?
Most sites don’t need it. Google ignores it; Bing and Yandex respect it. Use a small value only if your server is really struggling.
Is blocking CSS and JavaScript a problem?
Yes. Google renders pages to understand them; blocking these files makes it harder to see how the page looks.
What robots.txt does, and what it does not
- It is a request, not a lock. Well-behaved crawlers follow it; it does not protect private pages. Use a password or
noindexfor those. - Blocking is not de-indexing. A blocked URL can still appear in Google if other sites link to it. To remove a page, let it be crawled and add
noindex. - Don’t block CSS or JavaScript. Google renders pages; blocking their files can hurt how your pages are understood.
- AI crawlers: training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) and the ones that fetch pages for answers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) are separate. Blocking training does not remove you from AI search.
- Upload the file as
/robots.txtat the root of the domain; each subdomain needs its own.