Processed locally in your browser — not uploaded
Robots.txt & AI Crawler Config Generator
Build a standards-compliant robots.txt configuration for your website. Features one-click controls for AI and LLM training web crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot, Meta-ExternalAgent), custom Allow/Disallow path rules, Crawl-delay directives, XML Sitemap references, and 1-click presets.
AI & LLM Model Web Crawlers
Control whether AI training bots and autonomous LLM scrapers are permitted to index your content.
# ==================================================================== # ROBOTS.TXT CONFIGURATION & AI CRAWLER DIRECTIVES # Generated via Moayyad Faris Developer Tools # ==================================================================== # BLOCK SPECIFIC AI SCRAPERS & TRAINING BOTS User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / # GLOBAL DEFAULT CRAWLER RULES User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /api/ # SITEMAP LOCATION Sitemap: https://example.com/sitemap.xml
How to use
- Select a pre-configured starter preset (e.g., Block AI Crawlers, Standard Search Engine Friendly, Strict Privacy).
- Toggle AI and LLM training bot permissions individually (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, etc.).
- Add custom Disallow and Allow directory path rules (e.g., /admin/, /private/, /api/).
- Specify your XML Sitemap URL and Crawl-delay setting.
- Copy or download the generated robots.txt file and place it in your website root folder (/public_html/robots.txt).
Understanding the output
The generator creates a standard robots.txt directives file. It formats User-agent headers, Disallow/Allow directives, Crawl-delay, and Sitemap declarations in the exact order parsed by search engines (Googlebot, Bingbot) and AI web crawlers (GPTBot, ClaudeBot).
Common issues & tips
- •Disallow rules apply to public web crawlers only. Do not rely on robots.txt to secure sensitive data; use server authentication instead.
- •Place robots.txt in the root document folder of your domain (https://example.com/robots.txt). Search engine crawlers look for it strictly at the root path.
- •Note that blocking AI crawlers in robots.txt prevents them from indexing or training on your content, but does not block web scrapers that ignore robots.txt standards.
Frequently asked questions
- Where do I upload the generated robots.txt file?
- Upload it to your domain's root web directory (e.g. https://example.com/robots.txt). Search engine and AI web crawlers look for it strictly at that root location.
- Does blocking AI crawlers in robots.txt guarantee they won't scrape my site?
- Legitimate AI companies (OpenAI, Anthropic, Google, Perplexity) honor robots.txt standards. However, rogue scrapers that ignore standards will still require firewall or CDN level blocking.
- Should I block all AI bots if I want search engine traffic?
- Blocking AI training bots like GPTBot or CCBot does not affect standard Google Search or Bing Search indexing (Googlebot and Bingbot). You can allow search engines while blocking LLM scrapers.
Related tools
WordPress & Apache Security .htaccess Generator
Build production-hardened .htaccess files: Force HTTPS, 301/302 redirects, IP blocking, WordPress shields, basic auth, and security headers.
URL Parser
Break a URL into protocol, host, path and query parameters — then edit and rebuild it.
RFC 9116 security.txt File Generator
Generate standardized RFC 9116 vulnerability disclosure security.txt files.