A robots.txt file tells search engine crawlers which parts of your site they may access. It is useful, but a single wrong line can stop Google crawling important pages. The most common mistakes are blocking the whole site by accident, using robots.txt to hide pages that should use noindex, and blocking files Google needs to see your pages properly. This guide explains each mistake and how to check your own file.
Most WordPress bloggers never need to touch robots.txt. WordPress and SEO plugins handle sensible defaults.
Problems usually start when someone edits the file after reading a tip, or when a setting from development is left on after launch.
Key takeaways
- robots.txt controls crawling, not indexing.
- Google says it is not a way to keep a page out of search results. Use noindex for that.
- A page blocked by robots.txt can still appear in search if other pages link to it.
- One line can block your entire site, so check the file after any change.
- Do not block CSS and JavaScript files Google needs to render your pages.
- Use Search Console’s robots.txt report to check how Google reads your file.
What robots.txt does
Google’s documentation says a robots.txt file tells search engine crawlers which URLs they can access on your site. It is mainly used to manage crawler traffic, for example to stop crawlers wasting time on unimportant pages.
The file sits at the root of your domain, for example yourdomain.com/robots.txt. You can open that address in a browser to see yours.
Mistake 1: Blocking the whole site
These two lines block all crawlers from the entire site:
User-agent: *Disallow: /
This sometimes gets added while a site is being built and is forgotten at launch. If your pages suddenly stop appearing in Google, check this first.
Note that a single slash after Disallow blocks everything. An empty Disallow line blocks nothing.
Mistake 2: Using robots.txt to hide pages
This is the most common misunderstanding. Google says robots.txt is not a mechanism for keeping a page out of Google. To do that, use a noindex rule or password protection.
Worse, if you block a page in robots.txt, Google cannot crawl it to see a noindex rule. Google also says a blocked page can still appear in search results if other pages link to it, just without a description.
The right tool for each job
Use robots.txt to manage what crawlers spend time on. Use noindex to keep a page out of search results. In WordPress, SEO plugins such as Rank Math let you set noindex on individual posts and pages.
Mistake 3: Blocking CSS, JavaScript or images
Google renders pages much like a browser to understand them. If robots.txt blocks the theme or plugin files a page needs, Google may not see the page as visitors do.
Avoid blocking folders such as wp-content or wp-includes as a whole. Default WordPress setups do not need this.
Mistake 4: Leaving “Discourage search engines” switched on
WordPress has a setting under Settings, then Reading, called “Search engine visibility”. Ticking “Discourage search engines from indexing this site” asks search engines not to index the site.
It is handy while building a site, but it should be unticked at launch. Check it if a new site is not appearing in Google at all. My guide on why your blog is not showing on Google covers this and other causes.
Mistake 5: Typos and wrong paths
- Rules are matched against the URL path, so a small typo can block the wrong section or nothing at all.
- Paths are case-sensitive.
- Each rule should be on its own line.
Mistake 6: Forgetting the sitemap line
Adding a Sitemap line to robots.txt helps crawlers find your sitemap. SEO plugins often add it automatically. It does not replace submitting your sitemap in Search Console. See how to submit a sitemap to Google.
Does robots.txt work for every search engine?
Not always. Google says robots.txt rules may not be supported by all search engines, and different crawlers can interpret the syntax differently. Well-known search engines generally follow it, but it is a request, not a lock.
How to check your robots.txt
- Open the file. Visit yourdomain.com/robots.txt.
- Look for “Disallow: /”. Make sure you are not blocking the whole site.
- Check Search Console. The robots.txt report under Settings shows the file Google found and any problems.
- Inspect key URLs. Use URL Inspection on important pages to confirm they can be crawled.
How to edit robots.txt in WordPress
WordPress generates a basic virtual robots.txt if no file exists. Many SEO plugins, including Rank Math, provide a robots.txt editor in their settings. Change it only if you have a clear reason, and check the file and Search Console afterwards.
Frequently asked questions
What is robots.txt used for?
To tell crawlers which URLs on your site they can access, mainly to manage crawling.
Can robots.txt remove a page from Google?
No. Google says to use noindex or password protection to keep a page out of search results.
Can a page blocked by robots.txt still appear in Google?
Yes, if other pages link to it. It may show without a description.
Where is robots.txt in WordPress?
At yourdomain.com/robots.txt. WordPress creates a virtual one if no file exists, and SEO plugins often let you edit it.
Do all search engines follow robots.txt?
Google says not all search engines support its rules, and crawlers can interpret them differently.

Nayan Dhumal