<\/p>\n
Google Search Relations answered several questions about web page indexing on the latest episode of the “Search Off The Record” podcast.<\/p>\n
Topics covered were how to block Googlebot from crawling specific sections of a page and how to prevent Googlebot from accessing a site.<\/p>\n
Google’s John Mueller and Gary Illyes answered the questions examined in this article.<\/p>\n
Mueller says yes impossible<\/strong> when asked how to prevent Googlebot from crawling specific sections of web pages, such as “also purchased” areas on product pages.<\/p>\n “The short version is that you can’t block a specific section of an HTML page from being crawled,” Mueller said.<\/p>\n He went on to offer two potential strategies to address the problem, neither of which, he stressed, are ideal solutions.<\/p>\n Mueller suggested using the data-nosnippet HTML attribute to prevent the text from appearing in a search snippet.<\/p>\n Alternatively, you can use an iframe or JavaScript with the source blocked by robots.txt, although he warned that this is not a good idea.<\/p>\n “Using a bot or JavaScript iframe can cause crawling and indexing issues that are difficult to diagnose and resolve,” Mueller said.<\/p>\n He assured everyone listening that if the content in question is being reused on multiple pages, it’s not a problem that needs to be fixed.<\/p>\n “There is no need to prevent Googlebot from seeing this type of duplication,” he added.<\/p>\n In response to a question about how to prevent Googlebot access none<\/strong> part of a site, Illyes provided an easy-to-follow solution.<\/p>\n “The easiest way is robots.txt: if you add a ban: \/ for the Googlebot user agent, Googlebot will leave your site alone as long as you maintain this rule,” Illyes explained.<\/p>\n For those looking for a more robust solution, Illyes offers another method:<\/p>\n “If you want to block even access to the network, you have to create firewall rules that load our IP ranges into a deny rule,” he said.<\/p>\n See Google’s official documentation<\/a> to get a list of Googlebot IP addresses.<\/p>\n While it’s impossible to prevent Googlebot from accessing specific sections of an HTML page, methods such as using the data-nosnippet attribute can provide control.<\/p>\n When considering blocking Googlebot from your site entirely, a simple disallow rule in your robots.txt file will do the trick. However, more extreme measures are also available, such as creating specific firewall rules.<\/p>\n Featured image generated by the author via Midjourney. <\/p>\n source<\/strong>: Google search off the record<\/a><\/p>\nBlocking Googlebot from accessing a website<\/h2>\n
To sum up<\/h2>\n