Configure Robots.txt

WordPress
Shopify
Blogger
Static Site
Custom
Rule #1

Live Preview

# robots.txt generated by Free SEO Tool Hub # https://freeseotoolhub.com/robots-txt-generator.html User-agent: * Allow: / Disallow: Sitemap: https://freeseotoolhub.com/sitemap.xml # Free SEO Tool Hub - All files allowed for indexing # Created for: https://freeseotoolhub.com/ # Date: 2024-12-27
Total lines: 8 | Size: 412 bytes
🔥 50+ High-Traffic Robots.txt Keywords

What is robots.txt?

Robots.txt is a text file that tells search engine crawlers which pages to crawl and which to ignore. It's placed in your website's root directory.

Common Directives

  • User-agent: * (all bots)
  • Disallow: /admin/
  • Allow: /public/
  • Sitemap: /sitemap.xml
  • Crawl-delay: 5

Best Practices

  • Don't block CSS/JS files
  • Always include sitemap
  • Test before deploying
  • Use specific user-agents

Popular Bots

  • Googlebot (Google)
  • Bingbot (Bing)
  • Slurp (Yahoo)
  • DuckDuckBot (DuckDuckGo)

Robots.txt Examples for Popular Platforms

WordPress:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php
Sitemap: /sitemap.xml
Shopify:
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Sitemap: /sitemap.xml
Static Site:
User-agent: *
Allow: /
Disallow: /private/
Disallow: /temp/
Sitemap: /sitemap.xml