How this tool works
Choose the default policy and enter any excluded paths beginning with a slash. The tool writes a plain-text file for compliant crawlers. The sitemap address is absolute, so it remains unambiguous when a crawler reads robots.txt from your site root. All-path disallow supersedes narrower exclusions.
User-agent: * followed by Disallow rules and an optional Sitemap directiveA worked example
Allow mode with /private/ blocks that path prefix for the * group while leaving other paths available. A completely empty Disallow value imposes no restriction. Disallow: / requests no crawling under that group.
Good to know
A simple single-group generator, not a complete robots-rule tester or security mechanism. Blocked crawling does not guarantee deindexing. It generates a downloadable file and never changes QuickCalculators’ own crawling policy.
Reference: Google: creating robots.txt (opens in a new tab) ↗
Crawl instructions belong at a site’s root
A compliant crawler requests robots.txt at the origin root and interprets the groups and path rules it contains. This generator creates a single User-agent: * group for a simple policy. In allow mode, only the listed prefixes are excluded; in disallow-all mode, the slash rule covers all paths.
The optional sitemap directive uses a complete address. The bulk URL to XML converter can prepare the referenced file, but you still need to publish that XML and place robots.txt at the appropriate root. Generating either file in a browser does not change your hosting configuration.
A crawler restriction is not a private area or deletion request
robots.txt can describe paths, but it does not authenticate visitors or prevent a person from opening them. Do not use it to protect confidential files. A blocked URL may also remain known to a search engine through links even if its content cannot be crawled.
For a page that should disappear from search, choose the appropriate response or indexing directive and consider whether crawlers can observe it. Blanket disallow can prevent them from seeing a noindex instruction. This utility deliberately avoids inventing a Noindex robots.txt rule or claiming that a generated file guarantees deindexing.
A few questions, answered
Does Disallow: / remove all existing search results?
No. It requests no crawling for the group; it is not a guaranteed removal instruction.
Are excluded paths passwords or access controls?
No. Use authentication and server permissions for private resources.