Indexing as many pages in your web site as attainable could be very tempting for entrepreneurs who’re making an attempt to boost their search engine authority.

However, whereas it’s true that publishing extra pages which can be related for a selected key phrase (assuming they’re additionally prime quality) will enhance your rating for that key phrase, generally there’s really extra worth in maintaining sure pages in your web site out of a search engine’s index.

… Say what?!

Stick with us, people. This publish will stroll you thru why you may need to take away sure webpages from the SERPs (search engine outcomes pages), and precisely how you can go about doing it.

Free Guide: How to Run a Technical SEO Audit

De-Indexing a Web page from Google

There are a couple of events the place you could need to exclude a webpage — or a portion of a webpage — from search engine crawling and indexing like:

  • To prevent duplicate content (when there may be a couple of model of a web page listed by the various search engines, as in a printer-friendly model of your content material) from being listed
  • To deal with admin and login pages for inside use except they’re meant for use by a group
  • For a thank-you page (i.e., the web page a customer lands on after converting on one of your landing pages) the place the customer will get entry to no matter supply that touchdown web page promised, corresponding to a hyperlink to an e book PDF

Here is what the thank-you web page for our SEO tips ebook seems like, for instance:

HubSpot SEO Starter Pack thank-you page

You need anybody who lands in your thank-you pages to get there as a result of they’ve already stuffed out a kind on a touchdown web page — not as a result of they discovered your thank-you web page in search.

Why not? As a result of anybody who finds your thank-you web page in search can entry your lead-generating affords immediately — with out having to offer you their data to go by your lead-capture kind. Any marketer who understands the value of landing pages understands how vital it’s to seize these guests as leads first, earlier than they’ll entry your affords.

Backside line: In case your thank-you pages are simply discoverable by a easy Google search, you could be leaving priceless leads on the desk.

What’s worse, you could even discover that a few of your highest-ranking pages for a few of your long-tail key phrases may be your thank-you pages — which implies you would be inviting a whole bunch of potential results in bypass your lead-capture types. That is a fairly compelling purpose why you’d need to take away a few of your internet pages from SERPs.

So, how do you go about “de-indexing” sure pages from serps? Listed below are 3 ways to do it.

3 Methods to De-Index a Webpage From Search Engines

Robots.txt to De-Index

Use if: You need extra management over what you de-index, and you’ve got the required technical assets.

One strategy to take away a web page from search engine outcomes is by including a robots.txt file to your website. The benefit of utilizing this methodology is you can get extra management over what you might be permitting bots to index. The outcome? You possibly can proactively maintain undesirable content material out of search outcomes.

Inside a robots.txt file, you may specify whether or not you’d like to dam bots from a single web page, a complete listing, and even only a single picture or file. There’s additionally an possibility to forestall your website from being crawled whereas nonetheless enabling Google AdSense advertisements to work when you have them.

That being stated, of the 2 choices accessible to you, this one requires probably the most technical kung fu. To learn to create a robots.txt file, you will need to read through this article that explains exactly how to do it..

HubSpot clients: You possibly can learn to set up a robots.txt file in your web site here, and learn to customise the contents of the Robots.txt file here.

If you happen to don’t want all of the management of a robots.txt file and are in search of a neater, much less technical resolution, then this second possibility is for you.

Htaccess No Index No Observe to De-Index

Use if: Your web site is working on Apache, and mod_headers is enabled, this can be a fast resolution.

On this circumstance, you would connect this single line to your .htaccess file:

Header set X-Robots-Tag “noindex, nofollow”

To suggest that your web site could be listed, however by no means proven in any Google search outcomes.

Meta No Index No Observe to De-Index

Use if: You need a neater resolution to de-indexing a whole webpage, and/or de-indexing the hyperlinks on a whole webpage.

Utilizing a metatag to forestall a web page from showing in SERPs — and/or the hyperlinks on a web page — is each straightforward and efficient. It requires solely a tiny little bit of technical know-how — actually, it is actually only a copy/paste job for those who’re utilizing the appropriate content management system.

The tags that allow you to do these items are referred to as “noindex” and “nofollow.” Earlier than I get into how you can add these tags, let’s take a second to outline and distinguish between the 2. They’re, in spite of everything, two fully completely different directives — they usually can be utilized both on their very own, or alongside each other.

What’s a “noindex” tag?

Once you add a “noindex” metatag to a webpage, it tells a search engine that though it could actually crawl the web page, it can’t add the web page into its search index.

So any web page with the “noindex” directive on it’s going to not go into the search engine’s search index, and might due to this fact not be proven in search engine outcomes pages.

What’s a “nofollow” tag?

Once you add a “nofollow” metatag to a webpage, it disallows serps from crawling the hyperlinks on that web page. This additionally implies that any rating authority the web page has on SERPs will not be handed on to pages it hyperlinks to.

So any web page with a “nofollow” directive on it’s going to have all its hyperlinks ignored by Google and different serps.

Like I stated earlier than, you may add a “noindex” directive both by itself, or along with a “nofollow” directive. You may also add a “nofollow” directive by itself, too.

When to Use “noindex” and “nofollow” Individually

Add solely a “noindex” tag if you do not desire a search engine to index your internet web page in search, however you do need it to comply with the hyperlinks on that web page — thereby giving rating authority to the opposite pages your web page hyperlinks to.

Paid touchdown pages are an important instance of this. You don’t need serps to index touchdown pages that individuals are presupposed to pay to see, however you may want the pages it hyperlinks to learn from its authority.

Add solely a “nofollow” tag if you do desire a search engine to index your internet web page in search, however you do not need it to comply with the hyperlinks on that web page.

There aren’t too many examples of if you’d add a “nofollow” tag to a complete web page with out additionally including a “noindex” tag. Once you’re determining what to do on a given web page, it is extra a query of whether or not so as to add your “noindex” tag with or and not using a “nofollow” tag.

When to Use “noindex, nofollow” Collectively

Add each a “noindex” and “nofollow” tag if you do not need serps to index a webpage in search, and you don’t need it to comply with the hyperlinks on that web page.

Thank-you pages are an important instance of this case. You don’t need serps to index your thank-you web page, nor would you like them to comply with the hyperlink to your supply and begin indexing the content material of that provide, both.

Add a “noindex” and/or a “nofollow” Metatag

Step 1: Copy one of many following tags.

For “noindex”:

<META NAME=”robots” CONTENT=”noindex”>

For “nofollow”:

<META NAME=”robots” CONTENT=”nofollow”>

For each “noindex” and “nofollow”:

<META NAME=”robots” CONTENT=”noindex,nofollow”>

Step 2: Add the tag to the <head> part of your web page’s HTML, a.okay.a. the web page’s header.

If you happen to’re a HubSpot buyer, that is tremendous straightforward — click on right here or scroll down for these directions particular to HubSpot customers.

If you happen to’re not a HubSpot buyer, then you definitely’ll have to stick this tag into the code in your webpage manually. Don’t be concerned — it is fairly easy. Here is the way you do it.

First, open the supply code of the net web page you are making an attempt to de-index. Then, paste the total tag into a brand new line inside the <head> part of your web page’s HTML, often called the web page’s header. The screenshots under will stroll you thru it.

The <head> tag signifies the start of your header:

head tag at beginning of header

Here is the metatag for each “noindex” and “nofollow” pasted inside the header:

meta tag in header for noindex nofollow

And the </head> tag signifies the tip of the header:

end head tag

Growth! That’s it. This tag tells a search engine to show round and go away, leaving the web page out of any search outcomes.

No Index No Observe in HubSpot

Including the “noindex” and “nofollow” meta tags is even simpler. All you must do is open the HubSpot device to the web page you need to add these tags to and select the “Settings” tab.

Subsequent, below Superior Choices and click on into “Head HTML.” Within the window under, paste the suitable code snippet. Within the instance under, I’ve added each a “noindex” and a “nofollow” tag since it is a thank-you web page.

no index no follow in hubspot

Press “Save,” and also you’re golden.

Efficiently No Index No Following a Web page

You’ve simply magically erased your web page from search engine outcomes. Now, you can begin capturing extra of these misplaced leads once more.

Now, needless to say you will not see outcomes instantaneously. Your modifications will not kick in till the following time a search engine crawls your web page. Relying on how usually you sometimes publish new pages in your web site, it really may take a couple of weeks. The extra continuously you publish content material, the extra usually serps will crawl your website. The easiest way to maintain monitor of how continuously Google visits your web site is to look at your crawl stats in Google Webmaster Tools.

Backside line: If you happen to discover that your web page remains to be showing in Google’s search outcomes even with the “noindex” tag, it is in all probability as a result of Google hasn’t crawled your website because you added the tag. You possibly can request that Google recrawl your web page utilizing the Fetch as Google tool.

Additionally, be aware that some serps’ internet crawlers may interpret these directives in another way than Google does, so your web page may nonetheless seem in outcomes from different serps. However for Google, it’s going to work simply tremendous — as soon as it does get round to crawling your web site.

Regardless, you can sleep slightly simpler realizing you’ve finally made your web site a greater place in your advertising.

Editor’s be aware: This text was initially revealed July 2016 and has been up to date for comprehensiveness.

seo audit