← All tools

Robots.txt Generator

Build your robots.txt for search and AI crawlers by choosing options; add blocked folders and your sitemap, then download it.

Google ignores this; Bing and Yandex respect it.
One per line, starting with /. Applies to all bots.
Crawler by crawler
CrawlerDefaultAllowBlock
GooglebotGoogle
BingbotBing
YandexBotYandex
DuckDuckBotDuckDuckGo
Googlebot-ImageGoogle Images
GPTBotOpenAI (training) · AI
OAI-SearchBotChatGPT search · AI
ChatGPT-UserChatGPT browsing · AI
ClaudeBotAnthropic (training) · AI
Claude-UserClaude browsing · AI
Google-ExtendedGemini training · AI
PerplexityBotPerplexity · AI
CCBotCommon Crawl · AI
Applebot-ExtendedApple AI training · AI

This tool runs entirely in your browser. What you enter is not sent to a server, not saved and not stored in cookies.

robots.txt is a small text file at the root of your site that tells bots which parts to crawl. One wrong line can take a whole site out of search; a good file points crawl budget at the pages that matter and lets you decide how much access AI crawlers get.

How to use it

  1. Pick the default rule for all bots; for a live site it is almost always “allow”.
  2. Write the paths you do not want crawled, one per line (e.g. /admin/, /cart/).
  3. Add your sitemap URL.
  4. If needed, allow or block crawlers one by one; download the file and upload it to the site root.

Frequently asked questions

Where does robots.txt go?

At the root of the domain: site.com/robots.txt. A file in a subfolder is ignored; every subdomain needs its own.

Can I remove a page from Google with robots.txt?

No. A blocked page can still appear if other pages link to it. To remove it, allow crawling and add a noindex tag.

Should I block AI crawlers?

It is your call. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) limits use of your content for model training. Blocking crawlers that fetch pages for answers (OAI-SearchBot, ChatGPT-User, PerplexityBot) can reduce your visibility in AI search.

Should I use Crawl-delay?

Most sites don’t need it. Google ignores it; Bing and Yandex respect it. Use a small value only if your server is really struggling.

Is blocking CSS and JavaScript a problem?

Yes. Google renders pages to understand them; blocking these files makes it harder to see how the page looks.

What robots.txt does, and what it does not