Free robots.txt tester: Your robots.txt checked, for Google and AI crawlers
Enter your website address. Mark reads your robots.txt file, flags each error with its line number, tests your pages for Googlebot, Bingbot and AI crawlers, then writes the recommended file, ready to copy.
The essentials in 30 seconds
A robots.txt tester or robots.txt checker shows you which crawlers your file lets in and which ones it blocks.
- A robots.txt file sits at the root of each host and gives crawling rules to bots.
- It controls crawling. It can't remove a page from Google's index.
- Google ignores crawl-delay and noindex lines in robots.txt: it reads only User-agent, Allow, Disallow and Sitemap.
- AI crawlers have separate names for training, search and on-demand fetches.
- If you want answer engines to cite you without training on your content, allow search crawlers and block training crawlers.
- This free robots.txt tester reads your file, lists problems by line number and writes a recommended file that never lets AI crawlers read more than Googlebot.
Why your file needs a robots.txt tester
Your robots.txt file is short, but it decides how Google, Bing and AI crawlers reach your site. A single misplaced slash can block your whole blog. A robots.txt tester catches that error before Google acts on it, and a robots.txt checker also tells you which line blocks a given URL.
The file now carries a second decision. AI companies send several crawlers, and each one has its own job. Some collect pages to train models. Others fetch pages so an answer engine can cite you.
Use this guide with the free tool on this page. You can test up to 10 addresses, pick a policy for AI crawlers and get a recommended file. You don't need to sign up.
What a robots.txt file is and what it doesn't do
A robots.txt file is a plain text file that tells crawlers which parts of your site they may visit. It follows a public protocol, standardized as RFC 9309. It lives at the root of each host, for example https://www.example.com/robots.txt. Crawlers look for it there and nowhere else.
Each host reads its own file. The www version, the non-www version and every subdomain need separate files. The protocol and the port count too: Google applies a file only to the protocol, host and port that serve it.
Robots.txt manages crawling, and indexing is a separate step. If other sites link to a blocked page, Google can still index its URL without reading the content, and the result shows no description. To remove a page from results, let Google crawl it and add a noindex meta robots tag.
Keep in mind that the file is public and voluntary. Anyone can read it, and a bot that doesn't follow the standard can ignore it. Protect private pages with a password, not with robots.txt.
Google reads 4 fields: User-agent, Allow, Disallow and Sitemap. Other lines, such as crawl-delay, have no effect on it.
| Directive | Role | Example |
|---|---|---|
| User-agent | Names the crawler that the group applies to | User-agent: Googlebot |
| Disallow | Blocks the URLs that start with this path | Disallow: /admin/ |
| Allow | Opens a path inside a blocked folder | Allow: /admin/help/ |
| Sitemap | Gives the full address of your XML sitemap | Sitemap: https://www.example.com/sitemap.xml |
| * | Matches any characters in a path, or every crawler in User-agent | Disallow: /*?sort= |
| $ | Marks the end of a URL | Disallow: /*.pdf$ |
Why robots.txt matters for SEO and AI crawlers
For SEO, robots.txt steers crawlers toward the pages you want to rank. On a large site, you keep bots away from internal search results, filters and sorting parameters. Google then spends its visits on your product pages and articles.
The file can also hurt you. If you block a folder by mistake, Google stops visiting the pages inside it. If you block CSS or JavaScript, Google can't render your pages the way visitors see them.
The HTTP code of /robots.txt matters as much as its content. A 404 means no file, so everything is open. A server error is worse: Google stops crawling the site for 12 hours, then relies on the last version it knows for up to 30 days.
For AI crawlers, the stakes are different. Your file decides whether answer engines can read your pages and cite them. It also decides whether your content can enter training datasets. Marketing should own this choice with the web team, because it shapes how your brand shows up in AI answers.
Robots.txt syntax explained with an example
Read the file as a set of groups. Each group starts with 1 or more User-agent lines, followed by its rules.
# Rules for all crawlers
User-agent: *
# Keep internal search results out of the crawl
Disallow: /search
# Block URLs that only sort a list
Disallow: /*?sort=
# Block PDF files in the drafts folder
Disallow: /drafts/*.pdf$
# Reopen 1 public file inside that folder
Allow: /drafts/press-kit.pdf
# Rules for OpenAI's training crawler only
User-agent: GPTBot
Disallow: /
# Full address of the sitemap, valid for every crawler
Sitemap: https://www.example.com/sitemap.xmlA few rules explain how crawlers read this file.
- A crawler follows the group that names it. If no group names it, it follows the * group. Never both.
- GPTBot follows only its own group here, so it ignores the rules under * and reads no page.
- A rule matches the start of the path:
Disallow: /searchalso blocks /search?q=chairs. - When 2 rules match a URL, the longer rule wins, and Allow wins a tie. That's why the Allow line reopens the press kit.
- Paths are case-sensitive, so /Search and /search are different paths.
- An empty
Disallow:line allows everything for that group. - Sitemap lines can sit anywhere, and they apply to all crawlers. Google reads the first 500 KiB of the file and ignores the rest.
The main AI crawlers and how to treat them
AI companies use different crawler names for different jobs: training models, finding sources for an answer engine, or fetching a page when a person asks about it. The tester checks these 11:
| Crawler | Company | Use |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | Search |
| ChatGPT-User | OpenAI | On-demand visit |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Search |
| Claude-User | Anthropic | On-demand visit |
| PerplexityBot | Perplexity | Search |
| Google-Extended | Training and Gemini answers | |
| Applebot-Extended | Apple | Training |
| Meta-ExternalAgent | Meta | Training and indexing |
| CCBot | Common Crawl | Open dataset, often reused for training |
Google-Extended and Applebot-Extended work differently from the others. They don't crawl: Googlebot and Applebot do, and these tokens tell Google and Apple whether your content may be used for their AI models. Blocking them keeps your pages in Google Search and in Apple's results, but blocking Google-Extended also keeps your pages out of some Gemini answers grounded in Google Search.
On-demand fetchers visit a page when a person asks a question about it, and some treat robots.txt differently: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Companies also rename, add or retire crawlers. Check each company's crawler documentation before you change your file.
Be cited or refuse training: pick your policy
The tool offers 4 policies for these 11 crawlers.
- Get cited, no training: you allow search and on-demand crawlers, and you block GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot.
- Allow all: every AI crawler of the table may read your public pages, never more than Googlebot.
- Block all: you block the 11 crawlers, so answer engines can't read your pages to cite them.
- Keep my rules: the tool tests your current file and changes nothing about AI crawlers.
If you want to stay visible in AI answers without feeding training sets, the first policy is a sensible place to start.
How to use a robots.txt tester in 5 steps
- Open your live file. Type your domain followed by /robots.txt in a browser. Check that the file loads as plain text on the version your visitors use, and note any redirect from www to non-www or from http to https. After 5 redirects, Google acts as if there were no file.
- List the URLs that matter. Pick your homepage, a key product page, a blog article and a resource such as a white paper. Add a URL you expect to be blocked, like an internal search page. These tests show whether the file does what you intended.
- Pick your AI crawler policy. Decide with your team whether you want to be cited, block everything or keep your current rules. Write the decision down, so the web team knows why the file changes.
- Read the problems line by line. Start with problems that block the whole site or your CSS and JavaScript. Then fix ignored directives and the sitemap address. Each problem comes with its line number, so you find it fast.
- Deploy and test again. Replace the file on your server with the version you reviewed. Google generally caches it for up to 24 hours, so the change applies at its next read. Run the test again on the same URLs, and repeat after every site migration or CMS change.
Common mistakes a robots.txt validator catches
Blocking the whole site
A single Disallow: / under User-agent: * blocks every page for every crawler that follows that group. This often happens when a staging file goes live by mistake. Check the file after each release.
Blocking CSS and JavaScript
Rules like Disallow: /assets/ can hide the files that build your layout. Google then sees an incomplete page. Let crawlers load the CSS and JavaScript files that your pages need.
Using noindex in robots.txt
Google doesn't support noindex in robots.txt, so the line does nothing. Use a noindex meta robots tag or an X-Robots-Tag HTTP header instead, and leave the page open to crawlers so they see it.
Writing a relative sitemap address
Google needs the full address in a Sitemap line: Sitemap: /sitemap.xml doesn't work, while Sitemap: https://www.example.com/sitemap.xml does.
Adding crawl-delay for Google
Google ignores crawl-delay. Bing reads it, so keep the line if you need it for Bing. If Googlebot puts too much load on your server, Google suggests returning a 500, 503 or 429 code for a few hours, 1 to 2 days at most.
Forgetting the www or non-www version
Each host reads its own file. If www redirects to non-www, crawlers follow the redirect and read the file on the final host. If both versions answer without a redirect, check both files: the tester compares them for you.
Before and after: the Verdano example
Verdano is a fictional company, invented for this example: it sells office furniture to businesses. Its team wants to be cited by answer engines and refuse training.
The file before
Disallow: /search/
User-agent: *
Disallow: /admin/
Disallow: /assets/
Crawl-delay: 10
Noindex: /thank-you/
User-agent: GPTBot
Disallow: /
Sitemap: /sitemap.xmlThe tool flags 5 problems, and its AI crawler table shows a 6th point.
- Line 1: the rule sits outside any group, so crawlers ignore it.
- Line 5: the home page loads its CSS and JavaScript files from /assets/, and this rule blocks them for Googlebot.
- Line 6: Google ignores Crawl-delay, but Bing reads it.
- Line 7: Google ignores Noindex.
- Line 12: the sitemap address is relative.
- AI crawlers: only GPTBot is blocked, so ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot can still collect pages.
The recommended file
With the “Get cited, no training” policy, the tester writes this file:
# Disallow: /search/ (outside any User-agent group: robots ignore it)
User-agent: *
Disallow: /admin/
Disallow: /assets/
Crawl-delay: 10
# Noindex: /thank-you/ (ignored by Google: use the meta robots tag or the X-Robots-Tag header)
User-agent: GPTBot
Disallow: /
# AI robots: training refused, search and on-demand visits allowed
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
Sitemap: https://www.verdano.example/sitemap.xmlThe 5 training crawlers that could still read the site join a group that blocks everything, and GPTBot keeps its own. Search and on-demand crawlers such as OAI-SearchBot and PerplexityBot still follow the * group. The rule outside any group and the Noindex line become comments that say why, and the sitemap gets its full address. Crawl-delay stays for Bing.
The recommended file never changes what Googlebot may read, so the /assets/ line stays, flagged in the problems. Verdano removes it after checking that the folder holds nothing private, moves the /search/ rule into the * group if it wants it applied, and adds a noindex meta robots tag to its /thank-you/ page.
Robots.txt, the meta robots tag and llms.txt compared
These 3 tools solve different problems. Use them together.
| Criterion | robots.txt | Meta robots tag | llms.txt |
|---|---|---|---|
| What it is | Plain text file of crawling rules | HTML tag in the head of a page, or an X-Robots-Tag header | Markdown file that lists your key pages for LLMs |
| Where it lives | Root of each host | Inside each page | Root of the site |
| What it controls | Which paths a crawler may visit | Indexing and snippets for 1 page | Which content you suggest LLMs read first |
| Removes a page from the index | No | Yes, with noindex | No |
| Status | Standard, RFC 9309 | Followed by major search engines | Proposal, with uneven support |
One caution: if robots.txt blocks a page, crawlers never read its meta robots tag.
How the free robots.txt checker works
- Enter your site address. Add up to 10 addresses to test, such as your key pages, a CSS file and a page you expect to be blocked.
- Pick a policy for AI crawlers. Choose between getting cited without training, allowing all, blocking all or keeping your rules.
- Read the report. The tool follows redirects, records the HTTP code and checks the size and encoding of the file. It lists each problem with its line number and tests each address for Googlebot, Bingbot and the 11 AI crawlers, with the rule that decides. A table shows which AI crawlers are allowed and which are blocked, today and with the recommended file.
- Copy or download the recommended file. The tool highlights the changed lines, so your web team reviews them before deployment.
The recommended file follows 3 rules.
- What Googlebot may read never changes. A rule that blocks your pages or your CSS is flagged, and you fix it yourself.
- An AI crawler you allow never reads more than Googlebot. It gets the rules Googlebot follows: the * group when Googlebot follows it, else a copy of Googlebot's own group. When Googlebot itself can't read the site, as on a staging site, AI crawlers stay as they are and the page tells you why.
- Crawl-delay stays for Bing. Noindex and host lines become comments that say why, relative sitemaps get their full address, and a typo Google reads anyway gets its right spelling, so the rule then applies to every crawler.
The tool computes every rule from fixed checks based on how Google reads the file. No AI writes or guesses the result: the same file always gives the same report. The tool is free, and you don't need to sign up.
What the tool doesn't do
- It doesn't change anything on your server. You deploy the file yourself.
- It can't guarantee that every crawler follows your file.
- It tests 10 addresses at most per run, on top of the page entered.
- It suggests no file while your robots.txt answers with a server error: fix the server first, then run the test again.
How the recommended file differs from a robots.txt generator
A robots.txt generator starts from a blank page and can drop rules your site needs. This tool starts from your live file and changes only the lines that need it, so the rules you wrote stay in place.
Related free tools
- The llms.txt generator builds an llms.txt file that points LLMs to your key pages.
- The GEO audit measures your visibility in AI answers, against your competitors.
- The free SEO audit scores your website out of 100, from indexing to content.
After the test: pages answer engines cite
Opening the door to crawlers isn't enough: they cite the pages that clearly answer your customers' questions. Mark AI writes your articles, pages and product descriptions in your brand voice, with direct answers, and your team reviews before publishing.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file that tells crawlers which parts of your site they may visit. It groups rules by crawler name with User-agent, Disallow and Allow lines. Major search engines and AI crawlers read it before they crawl.
Where is the robots.txt file?
The file lives at the root of each host, for example https://www.example.com/robots.txt. Each subdomain, and each version with or without www, needs its own file. Crawlers don't look for it in any other folder.
How do you test a robots.txt file?
Enter your site address and the URLs you care about in a robots.txt tester. The tool reads your live file and checks each URL for each crawler. It also lists syntax problems with their line numbers.
How do you block GPTBot?
Add a group with User-agent: GPTBot followed by Disallow: /. This blocks OpenAI's training crawler. OAI-SearchBot uses a separate name, so your pages can still appear in ChatGPT search answers.
Should you block AI crawlers?
It depends on your goal. If you want answer engines to cite you, allow search and on-demand crawlers. If you want to keep your content out of training sets, block training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot.
Does robots.txt stop indexing?
No. Robots.txt stops crawling, and Google can still index a blocked URL if other pages link to it. To keep a page out of results, use a noindex meta robots tag and let Google crawl the page.
Is the robots.txt tester free?
Yes. The robots.txt tester is free and you don't need to sign up. You can test up to 10 addresses per run and copy or download the recommended file.
Test your file before your next release
Your robots.txt file shapes how search engines and AI crawlers see your site. Enter your site address in the free robots.txt checker above, review the problems line by line and get your recommended file.
Test my robots.txt