BuildUtilities

Robots.txt Generator – Free Online Tool

Generate robots.txt files to control search engine crawler access.

✦ FreeNo SignupInstantWebHTML & Web Optimization Guide
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

Example

Input

Allow: /, Disallow: /admin, Sitemap: /sitemap.xml

Output

User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

A robots.txt file that allows crawling everywhere except /admin and specifies the sitemap.

How to Use Robots.txt Generator

  1. 1

    Configure Rules

    Set allow/disallow rules for different user agents.

  2. 2

    Add Sitemap

    Optionally add your sitemap URL.

  3. 3

    Download

    Download the generated robots.txt file.

Common Use Cases

  • ✦Setting up SEO for new websites
  • ✦Blocking sensitive directories from crawlers
  • ✦Managing search engine indexing

Embed This Tool

Add this tool to your website or blog with a simple iframe embed.

<div style="max-width:480px"><iframe src="https://buildutilities.com/embed/robots-txt-generator?ref=embed" width="480" height="360" frameborder="0" style="border-radius:8px;border:1px solid #2a2e3a;width:100%"></iframe><p style="font:11px sans-serif;color:#888;margin:4px 0 0;text-align:right">Powered by <a href="https://buildutilities.com/robots-txt-generator?ref=embed" target="_blank" rel="noopener">BuildUtilities</a></p></div>

Share

Share a link to this tool. Results can be passed via URL parameters.

https://buildutilities.com/share/robots-txt-generator

More Web Tools

HTML, URL, and meta tag tools for web development.

< HTML Escape / Unescape?= Query String Parser<>↓ HTML Minifier<>↑ HTML Formatter🏷️ Meta Tag GeneratorOG Open Graph Preview
View all Web tools
Tip Jar