How to Edit Robots.txt on WordPress

Robots.txt is one of those thing that many SEOs have heard of, but few of us really understand. The good news is that the robots.txt file does not need to be complicated. The better news is that it’s straightforward to edit robots.txt on WordPress.

If you’re already familiar with the robots exclusion protocol, skip down to the video below for a walkthrough of updating robots.txt using Yoast. Otherwise, here’s some high level background.

What is Robots.txt?

Robots.txt – also called the robots exclusion protocol – is a mechanism for websites to communicate with search engine crawlers and other bots on the internet.

It tells search engine crawlers which URLs they should access on your website. For example, if you have a private section on your site for logged in members, you can use robots.txt to exclude that portion of your site from being accessed, crawled, and indexed. You’d typically exclude a subdomain or subfolder in cases like this. On an ecommerce website, you could use robots.txt to exclude your cart and checkout pages.

Understanding robots.txt is an important part of the SEO ecosystem, and is generally considered to be in the “technical SEO” field (unlike Local SEO or On Page SEO optimization).

How to Edit Robots.txt on WordPress

One of the simplest ways to edit robots.txt on WordPress is to use the Yoast SEO plugin. Yoast is a Swiss Army knife of SEO for WordPress sites. You can use it for everything from editing your sitemap to blocking pages from being indexed to changing how your pages appear on the Google SERP (Search Engine Results Page).

By default, WordPress will generate a robots.txt that looks like this.

The first line means that the instruction is for all bots and crawlers. The asterisk is a regular expression (regex) that means everyone is included. If there was some reason you wanted to have different instructions for Google (Googlebot) or Bing (Bingbot) or Yahoo (Slurp), you could do so. Here’s a guide to the 10 most popular search engine bots, including DuckDuckGo, Baidu, and more.

The video linked below walks through how to access robots.txt on a WordPress site and how to add your sitemap to robots.txt.

Not a video person? No problem. You can check out the 5 steps detailed below.

How to Add Your Sitemap to Robots.txt in 5 Steps With Yoast

It’s really pretty quick. Maybe even quicker than the 3 minute video linked above.

1. After you’ve logged in to your WordPress site, click on “SEO” in the left-hand navigation to access the Yoast SEO menu.
2. Click on Tools.
3. Click on File Editor.
4. Make the changes to the robots.txt file. If you’ve used Yoast to build your sitemap, you can add it here to make it easily accessible to all search engines. Simply go to a new line and type Sitemap: https://www.yoursite.com/sitemap_index.xml
5. Hit the button to “Save changes to robots.txt” as shown immediately below.

Can You Use Robots.txt to Remove Pages from Google’s Index?

No. This is a common misconception with robots.txt. According to Google’s developer guide, you should block indexing with a noindex tag or password-protect the page. That’s because a page “blocked” by robots.txt could still be indexed by Google if there are other pages that are linking to that page. A noindex tag solves that problem.

It’s also worth noting that Google treats robots.txt as a suggestion as opposed to a directive from websites. Furthermore, some other bots (especially malware bots) may ignore robots.txt completely. They aren’t really the rule following type.

How Long Does Robots.txt Take to Update?

According to Google, it won’t take more than a day. “Google’s crawlers notice changes you made to your robots.txt file and update the cached version every 24 hours.”

Not bad!

If you’ve got some time on your hands while you’re waiting, consider checking out more free resources related to SEO, Google Analytics, Google Tag Manager, and Google Ads over on the Root and Branch YouTube channel. This one about using Google Tag Manager and Google Analytics to set up link click tracking is a personal favorite of mine.

If you like what you see and decide to subscribe, there’s normally at least few new analytics and SEO videos with tips and guides each month.

Thanks! Have fun out there!