Robots.txt tester
Find out whether a crawler can fetch a page, and exactly which line of your robots.txt decides it. Check search engines and AI crawlers side by side, and catch mistakes that make crawlers ignore your rules.
Leave empty to test the home page. A full URL works too.
How to use it
- 1Enter your website and choose Fetch, or switch to Paste to test a robots.txt you have not published yet.
- 2Type the page or path you want to check, and pick a crawler such as Googlebot.
- 3Read the result: it shows whether the crawler is allowed and which line decided it.
- 4Fix any problems listed, then test again.
Frequently asked questions
Does robots.txt keep a page out of Google?
No. It stops crawlers fetching the page, but Google can still list a blocked URL if other sites link to it. To keep a page out of search results, use a noindex robots meta tag or an X-Robots-Tag header, and let the page be crawled so Google can see it.
What happens if my site has no robots.txt?
If the file returns "not found", crawlers assume they may crawl everything. If it returns a server error, Google treats the whole site as blocked until it recovers, which is worse.
Which rule wins when several match?
The most specific one: the longest matching path. If an Allow and a Disallow match with the same length, Allow wins. A crawler with its own group ignores the rules under "User-agent: *" completely.
Should I block AI crawlers?
It depends on your goals. Blocking stops well-behaved AI crawlers from using your pages, but may also mean you are cited less in AI answers. robots.txt is voluntary, so it does not stop crawlers that ignore it.
More free SEO tools
Want this checked automatically across your whole site?
Opptym runs SEO checks on every page of your website, keeps the history, and tells you exactly what to fix first.