Text & web utilities

robots.txt file generator

Prepare a simple all-crawler policy with optional path exclusions and an absolute sitemap URL.

Your settings

Calculated on your device. Your inputs aren’t sent to us.

Your result

Ready when you are

Enter your inputs and select Generate result.

Report an issue

Try it, understand it

Explore the result

Compare an exclusion list with an all-path crawling restriction.

Worked example comparison
ExampleOutputWhat to notice
Exclude private path prefixesDefault crawlingAllowed except listed pathsThe default group disallows /private/ and /admin/.
Allow everythingDefault crawlingAllowed except listed pathsAn empty Disallow value adds no restriction.
Disallow all crawlingDefault crawlingDisallowedThe slash rule covers every path for the group.

Check your understanding

Does Disallow: / remove all existing search results?

No. It requests no crawling for the group; it is not a guaranteed removal instruction.

Are excluded paths passwords or access controls?

No. Use authentication and server permissions for private resources.

Behind the tool

How this tool works

Choose the default policy and enter any excluded paths beginning with a slash. The tool writes a plain-text file for compliant crawlers. The sitemap address is absolute, so it remains unambiguous when a crawler reads robots.txt from your site root. All-path disallow supersedes narrower exclusions.

THE OUTPUT RULEUser-agent: * followed by Disallow rules and an optional Sitemap directive

A worked example

Allow mode with /private/ blocks that path prefix for the * group while leaving other paths available. A completely empty Disallow value imposes no restriction. Disallow: / requests no crawling under that group.

Good to know

A simple single-group generator, not a complete robots-rule tester or security mechanism. Blocked crawling does not guarantee deindexing. It generates a downloadable file and never changes QuickCalculators’ own crawling policy.

Crawl instructions belong at a site’s root

A compliant crawler requests robots.txt at the origin root and interprets the groups and path rules it contains. This generator creates a single User-agent: * group for a simple policy. In allow mode, only the listed prefixes are excluded; in disallow-all mode, the slash rule covers all paths.

The optional sitemap directive uses a complete address. The bulk URL to XML converter can prepare the referenced file, but you still need to publish that XML and place robots.txt at the appropriate root. Generating either file in a browser does not change your hosting configuration.

A crawler restriction is not a private area or deletion request

robots.txt can describe paths, but it does not authenticate visitors or prevent a person from opening them. Do not use it to protect confidential files. A blocked URL may also remain known to a search engine through links even if its content cannot be crawled.

For a page that should disappear from search, choose the appropriate response or indexing directive and consider whether crawlers can observe it. Blanket disallow can prevent them from seeing a noindex instruction. This utility deliberately avoids inventing a Noindex robots.txt rule or claiming that a generated file guarantees deindexing.

A few questions, answered

Does Disallow: / remove all existing search results?

No. It requests no crawling for the group; it is not a guaranteed removal instruction.

Are excluded paths passwords or access controls?

No. Use authentication and server permissions for private resources.

Find your calculator

Start typing to explore the library.