Robots.txt Generator
Build your robots.txt for search and AI crawlers by choosing options; add blocked folders and your sitemap, then download it.
| Crawler | Default | Allow | Block |
|---|---|---|---|
GooglebotGoogle |
|||
BingbotBing |
|||
YandexBotYandex |
|||
DuckDuckBotDuckDuckGo |
|||
Googlebot-ImageGoogle Images |
|||
GPTBotOpenAI (training) · AI |
|||
OAI-SearchBotChatGPT search · AI |
|||
ChatGPT-UserChatGPT browsing · AI |
|||
ClaudeBotAnthropic (training) · AI |
|||
Claude-UserClaude browsing · AI |
|||
Google-ExtendedGemini training · AI |
|||
PerplexityBotPerplexity · AI |
|||
CCBotCommon Crawl · AI |
|||
Applebot-ExtendedApple AI training · AI |
This tool runs entirely in your browser. What you enter is not sent to a server, not saved and not stored in cookies.
robots.txt is a small text file at the root of your site that tells bots which parts to crawl. One wrong line can take a whole site out of search; a good file points crawl budget at the pages that matter and lets you decide how much access AI crawlers get.
How to use it
- Pick the default rule for all bots; for a live site it is almost always “allow”.
- Write the paths you do not want crawled, one per line (e.g. /admin/, /cart/).
- Add your sitemap URL.
- If needed, allow or block crawlers one by one; download the file and upload it to the site root.
Frequently asked questions
Where does robots.txt go?
At the root of the domain: site.com/robots.txt. A file in a subfolder is ignored; every subdomain needs its own.
Can I remove a page from Google with robots.txt?
No. A blocked page can still appear if other pages link to it. To remove it, allow crawling and add a noindex tag.
Should I block AI crawlers?
It is your call. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) limits use of your content for model training. Blocking crawlers that fetch pages for answers (OAI-SearchBot, ChatGPT-User, PerplexityBot) can reduce your visibility in AI search.
Should I use Crawl-delay?
Most sites don’t need it. Google ignores it; Bing and Yandex respect it. Use a small value only if your server is really struggling.
Is blocking CSS and JavaScript a problem?
Yes. Google renders pages to understand them; blocking these files makes it harder to see how the page looks.
Rather scan your whole site with an AI skill than check links one by one?
technical-seo-skill is an open-source skill I built that works like a technical SEO specialist inside your AI agent (Claude Code or Codex). It discovers every URL on your domain and subdomains, checks robots.txt, llms.txt and sitemaps, titles, descriptions, canonicals, H1s, hreflang and schema, finds broken internal links, orphan pages and oversized images, and hands you an Excel report with a prioritised task list.
Give the link to your AI
In Claude Code or Codex, paste a request like this and change the domain. The agent installs the skill and runs the audit.
Install the skill at https://github.com/AndacGuven/technical-seo-skill and run a technical SEO audit for https://example.com. Give me the Excel report and the most urgent fixes first.
Install it yourself
Clone the repository into your agent’s skills folder and restart the agent. Or download the v1.0.0-alpha release from GitHub’s Releases section and put SKILL.md and technical_seo_skill.py in the same folder.
Claude Code · macOS / Linuxgit clone https://github.com/AndacGuven/technical-seo-skill ~/.claude/skills/technical-seo
git clone https://github.com/AndacGuven/technical-seo-skill ~/.codex/skills/technical-seo
git clone https://github.com/AndacGuven/technical-seo-skill $HOME\.claude\skills\technical-seo
What robots.txt does, and what it does not
- It is a request, not a lock. Well-behaved crawlers follow it; it does not protect private pages. Use a password or
noindexfor those. - Blocking is not de-indexing. A blocked URL can still appear in Google if other sites link to it. To remove a page, let it be crawled and add
noindex. - Don’t block CSS or JavaScript. Google renders pages; blocking their files can hurt how your pages are understood.
- AI crawlers: training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) and the ones that fetch pages for answers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) are separate. Blocking training does not remove you from AI search.
- Upload the file as
/robots.txtat the root of the domain; each subdomain needs its own.