There are a number of reasons why you may want to prevent search engines from indexing your website. You could want to stay away from web crawlers temporarily (if you’re working on developing your site in a live environment) or permanently (if your WordPress website is actually a private blog).
Thankfully, there are a number of ways you can prevent search engines from indexing your site from adding code snippets to the robots.txt file to using plugins and configuring your Google Search Console settings.
In this post, we’ll walk you through three different ways of preventing Google from indexing your WordPress website. By the time you’re done reading you’ll have a better idea of how you can stay clear of search engine results pages.
Let’s put everything into context before we begin.
Why Would I Want Google to Stop Indexing My WordPress Website?
Search engines (like Google) bring in a lot of traffic. They constantly have spiders crawling the web and caching pages for their index. Though it may sound counter-intuitive, there are some scenarios in which you may want to prevent Google from indexing your WordPress website.
But before we get into that, let’s quickly discuss the difference between being listed and being indexed. When we say a site has been listed it means that the website will show up in the search engine results pages. However, indexing is the process of caching a web page’s content to the search engine’s server aka its index.
Here’s why you might want to stop Google from indexing your website:
- You’re developing in a live environment. It’s common practice for WordPress developers to skip setting up a local development environment and start off by putting their website on a live domain instead.
- You have a private blog/project management site. Project managers and private blog owners find it easier to collaborate with team members using WordPress. It’s unlikely that they want their website’s content to show up in search engine results.
Aside from these two scenarios there may be other reasons why you want to prevent Google from indexing your WordPress site. The good news is that it is possible to make sure search engines don’t index your site.
How Can I Prevent My Website From Being Indexed by Google?
In this section, we’ll walk you through three different methods of preventing your website from being indexed by search engines.
Option 1: Discourage Search Engines From Indexing Your Site
The easiest way to go about preventing search engines from indexing your site is by instructing them from crawling it in the first place. Here are three different ways you can do this:
Using WordPress’ Built-in Feature
Log in to your WordPress website and navigate to Settings > Reading from the admin panel. From the Reading Settings screen, scroll down to the Search Engine Visibility option and check the box that reads Discourage search engines from indexing this site.
Click the Save Changes button at the bottom of the screen to continue. Once you’ve done that, WordPress will automatically modify your website’s robots.txt file and add rules that will disallow search engines from indexing your site.
Add a Meta Robots Tag
However, if you prefer taking the DIY route then you can achieve the same end result by modifying your site’s robots.txt file manually. Here’s how:
Log in to your cPanel account (or FTP client) and head over to your site’s root directory. Next, locate your robots.txt file and open it up in a text (or code) editor. Add the following lines of code to it:
https://gist.github.com/rafaysansari/9bc61fc5bd5599d2f096fac1ccb3781e
Doing this will not only prevent search engines from crawling your site but it will also make sure they stay away from it in the future.
Finally, add the following tag to the pages that you’d like to prevent Google from indexing using the Insert Headers and Footers plugin.
https://gist.github.com/rafaysansari/d15ac025dfcc23c256b4a82b40de0c8a
Add a X-Robots-Tag HTTP Header
Adding a meta robots tag to every single page on your WordPress website can be a daunting task if you have lots of pages that you need to add it to. Thankfully, there’s a simple(r) workaround for it: the X-Robots-Tag HTTP Header.
The X-Robots-Tag HTTP Header allows website owners to specify an HTTP header called the X-Robots-Tag and give it the same value that you would assign to the meta robots tag. All you have to do to implement this is add the following line of code to your WordPress website’s .htaccess file:
https://gist.github.com/rafaysansari/53143fd8ac7648b11d0d9c05674f1195
This will ensure that your entire WordPress website won’t show up in search engine results pages.
Option 2: Password Protect Your Website
The concept behind our second method is simple: both search engines and web crawlers can’t access files that are password protected. There are two ways you can password protect your website – using cPanel and by using a plugin.
Using cPanel
If you have access to your WordPress website’s cPanel account then we recommend password protecting your site’s files manually. It’s incredibly easy to do and it’s one of those things that you don’t need to install a plugin for.
To get started, log in to your cPanel account and scroll down to the Security section. From there, click on the Directory Password icon.
Next, you will be prompted to click the name of the folder that you want to password protect. Select the directory where you have WordPress installed. In most cases, this will be public_html.
In the following screen, you’ll be asked to configure the Security Settings. Check the box next to the Password protect this directory option and enter a name for the directory you want to protect. Hit the Save button once you’re done.
Finally, you’ll have to create a user who will be able to access the protected files after entering the login credentials. Once you’re done entering the username and password, click the Add Or Modify The Authorized User to proceed.
Using a Plugin
Alternatively, you can use the Password Protected plugin for WordPress to password protect your site. Here’s how:
Log in to your WordPress website and navigate to Plugins > Add New from the admin panel. Next, search for Password Protected.
Click the Install Now button to begin installing the plugin and once the plugin is installed click the Activate button to activate it. After the plugin is activated to your website, navigate to Settings > Password Protected. From the Password Protected settings screen:
- Check the box next to the Password Protected Status option.
- Add a password in the New Password option.
Once you’ve done that, click the Save Changes button at the bottom of the screen to proceed.
Option 3: Remove Pages That Have Already Been Indexed
If your website has already been indexed by Google then you’ll need to take a slightly different approach.
Log in to your Google Search Console account and navigate to Google Index > Remove URLs from the admin dashboard. From the Remove URLs screen, click the Temporarily hide button and enter the URL on your site that you’d like to hide. Once you’ve done that, click the Continue button.
That’s all there is to it. If you need to remove multiple URLs then all you have to do is repeat the same step for all of the URLs that you’d like to temporarily remove.
Conclusion
Whether you’re running a private blog or are still working on developing your WordPress website you might want to consider preventing search engines from indexing it. We covered some different ways you can get the job done and hopefully, you’re in a good position now to take things further.
Let’s quickly recap the three main methods you can implement to prevent Google from indexing your WordPress website:
- Discourage search engines from indexing your website from the WordPress back-end.
- Password protect your website either through the cPanel or by using the Password Protected plugin.
- Remove pages that have already been indexed.
Why do you want to prevent Google from indexing your WordPress website? Let us know by commenting below!
Mark Anthony
December 7, 2017 at 8:52 pmHi, do you guys provide custom work on your themes that you sold?
Sarah Bishop
December 20, 2017 at 7:04 amHi Mark, at this time we do not provide any custom work. If you need help with certain small changes our support staff would be happy to help.