Google’s John Mueller answered a question on Reddit about a commonly used bot meta tag and what would happen if it wasn’t missing. Mueller’s answer, while it makes sense and is documented, may still surprise many publishers and SEOs.
Robot meta tag
The HTML meta element communicates metadata. Metadata is machine-readable information that can be read by a crawler like Googlebot.
There are many types of meta elements such as the description meta element, but the robots meta element is different because it can control search engine crawlers.
The information that the robots meta tag communicates is called a directive, which means that the crawler robots are bound to obey the instructions in the robots meta tag.
There are many directions to go through the bots meta, but the following meta tag is one that is relevant to the question John Mueller answered.
The meta tag noindex, no follow:
The meta tag above tells search engine crawlers not to index the content of the web page and not to follow any links.
One of the most common meta tags is this one, which instructs search engines to index your content and follow all links:
Although the above meta tag is common, there are a lot of misunderstandings about it. There is a line of reasoning that since Google supports nofollow, it must imply that Google supports the follow directive.
I found a number of authoritative websites that say Google uses the meta robots index, follow meta tag.
But that’s not actually how Google uses these directives, as John Mueller makes clear in his answer.
What is the effect of omitting the Meta Robots index tag?
The person on reddit he asked the following question:
“I’m a little confused about a website I’m working on.
So, this is what the meta snippets look like on most of the websites I work on:
But the website in question is missing the “index” tag.
My question is: what is the effect of the site not having the “index” tag?
John Mueller replied:
“The ‘index’ meta tag for bots has no function (at least on Google), it’s completely ignored. Also ‘follow’.”
Google has & to document meta tags that have functions. You can use anything else, it will be ignored. it’s an option, if you want to throw people out.”
Why Google Ignores Robots Index & Follow
The simple reason why Google ignores the bot index and meta tag follows is because indexing and tracking are the default values.
Indexing and following links is what search engine bots do, they don’t need to be told to index content and follow links because that’s their purpose.
from Google documentation on bot tags advises:
“Defaults are index, follow and do not need to be specified.”
The complete list of valid directives for Google is here.
If the meta bot you want to use isn’t listed there, Googlebot will ignore it.
Is the index, tracking completely useless?
It is true, documented and official that when it comes to Googlebot, it’s a waste of HTML space and is ignored by Googlebot.
Bing treats the index in a similar way but with a slight difference, as described in oofficial Bing documentation for meta tags.
Here’s what Bing’s documentation says about the index directive:
“By default we assume ‘index’, but if needed you can use to explicitly indicate that we can index the page.”
And this is what it says about the following directive:
“By default we assume ‘follow’, but you can explicitly state ‘follow’ if you wish.”
In my 20+ years working in SEO building websites, optimizing and ranking them, I have always considered it a good policy to give the bots what they expect and try not to give them anything unexpected. So if a meta description isn’t necessary, my impulse would be to leave it out because the goal of optimizing is to make it as easy as possible for search engines to index and understand your content, which means getting rid of anything which could work against this goal.
In this case, it is very likely that it will have no effect one way or another.
But… There is another way to follow and index people.
Some publishers use this robots meta tag:
Some sites advise that if the page is not indexed, using the “follow” directive forces the search engine to follow the links.
But this is not true if there is a “noindex” directive for the simple reasons that Google cannot follow a link on a page that is not indexed. If not in the index, the links on these pages are not in the index.
Featured image by Shutterstock/Bangun Stock Productions
[ad_2]
Source link
