ToolZeno

Robots.txt Generator

Create search-engine-friendly robots.txt rules instantly. Select custom crawler templates, specify allow/disallow paths, append sitemaps, resolve rule conflicts, and download your production-ready file.

Website Information

Crawl Directives By Crawler (1)

Crawl Paths Rules

XML Sitemaps

Sitemap #1

Clean-Param Directives

Helps Yandex and Mail.ru crawlers avoid indexing duplicate content pages caused by query params (e.g. tracking ref IDs, sorting filters).

No clean-param parameters specified.

Compilation Options

Generated robots.txt Preview
Loading editor...

Crawl Syntax Valid

Excellent! Your robots.txt rules contain no duplicate or conflicting syntax.

Robots.txt Best Practices

  • Size Limits: Keep your robots.txt under 512 KB. Search bots like Googlebot truncate larger files.
  • Casing Rules: DIRECTIVES (User-agent, Disallow, Allow) are case-insensitive, but file paths are case-sensitive.
  • Crawl-delay Support: Googlebot completely ignores crawl-delay directives; use Google Search Console rate throttlers. Yandex, Bingbot, and Baidu still respect it.
  • Implicit Allow: By default, anything not explicitly disallowed is allowed to be crawled.

A Guide to Search Engine Crawl Optimization

Optimizing your site's indexing starts with a properly configured robots.txt file. While it seems simple, syntax errors or conflicting directories can prevent your core articles and landing pages from appearing in Google search results.

Understanding Robots.txt Syntax

The file is parsed line by line. The structure is built around user-agent rules and directive lines:

  • User-agent: Defines which search crawler the following block applies to. A wildcard (*) targets all crawlers.
  • Disallow: Specifies path patterns that the crawler must not visit. For example, Disallow: /admin/ blocks search crawlers from entering the admin folder.
  • Allow: Specifies exceptions to a Disallow block. For instance, if you block an assets folder, you can use Allow: /assets/logo.png to allow indexing of your logo.
  • Sitemap: Provides search engines with the exact location of your XML Sitemap. Sitemaps should always use absolute URLs and be declared at the end of the file.

Common Mistakes in Robots.txt Configuration

  • Blocking CSS and JS: Modern search engines render pages like a browser to verify mobile friendliness and design layouts. Blocking folders like /css/ or /js/ can damage your rankings.
  • Using path protocols: Disallow and Allow directives must start with a slash (/) or wildcard. Do not include your domain name in these paths (e.g. use /private/ instead of https://example.com/private/).
  • Duplicate Blocks: Defining rules for the same user agent (e.g., Googlebot) in separate, disjointed blocks can cause engines to ignore some rules or get confused. Merge them into a single section.

Frequently Asked Questions

A robots.txt file is a simple text file placed in the root directory of a website. It instructs search engine crawlers (bots) which parts of the site they are allowed or not allowed to visit and index. It forms part of the Robots Exclusion Protocol (REP) standard.

Robots.txt is crucial for managing your website's crawl budget. By preventing search engine crawlers from indexing duplicate content, private admin panels, user accounts, and utility scripts, you direct crawler resources toward your most important content pages, improving crawl frequency and index health.

The Disallow directive blocks search bots from crawling specified folders, patterns, or files. The Allow directive is used to override a Disallow block, telling search bots they can index a subfolder or specific page within a parent blocked directory. For example, you can disallow /wp-admin/ but allow /wp-admin/admin-ajax.php.

No. Major search engines like Googlebot completely ignore Crawl-delay. Instead, Google determines crawl rate dynamically. However, Yandex, Bingbot, Baiduspider, and various other utilities still recognize Crawl-delay, which helps prevent slow page load performance when crawlers hit database-heavy pages.

No. Robots.txt is public and can be read by anyone. Disallowing a directory tells search engines not to crawl it, but it does not restrict access. Never use robots.txt to hide sensitive information. Instead, use password protection, authentication, or 'noindex' meta tags.

No. Like all operations on ToolZeno, the Robots.txt Generator operates entirely client-side inside your browser. No website URLs, paths, sitemap links, or custom templates are uploaded or stored on any server, ensuring complete security and data privacy.

Related Tools

Slug Generator

Generate SEO-friendly URL slugs instantly. Convert titles to lowercase, strip emojis, map accented characters, handle stop words, and customize separators.

Open Tool

Meta Tag Generator

Generate and optimize SEO meta titles, meta descriptions, and social media Open Graph cards. Preview Google search result snippets and social posts in real-time.

Open Tool

Robots.txt Tester

Analyze, validate, and simulate robots.txt rules in real-time. Verify which URLs are allowed or blocked for Googlebot, Bingbot, and others client-side.

Open Tool

Sitemap Validator

Validate XML sitemaps client-side against sitemap protocols and Google requirements. Instantly check syntax, duplicate URLs, priorities, and extract statistics offline.

Open Tool