{"id":19743,"date":"2023-12-28T19:54:17","date_gmt":"2023-12-28T19:54:17","guid":{"rendered":"https:\/\/afreeurl.com\/?p=19743"},"modified":"2023-12-28T19:54:20","modified_gmt":"2023-12-28T19:54:20","slug":"google-says-that-bot-protection-services-sometimes-do-not-index","status":"publish","type":"post","link":"https:\/\/afreeurl.com\/?p=19743","title":{"rendered":"Google says that bot protection services sometimes do not index"},"content":{"rendered":"<p><\/p>\n<p>Did you know that sometimes these bot protection services will serve the noindex guidelines of Google and other search engines?  Google&#8217;s John Mueller had this to say <a href=\"https:\/\/twitter.com\/JohnMu\/status\/1739734474751938926\" target=\"_blank\" rel=\"noopener\">X<\/a>saying &#8220;Sometimes there&#8217;s a bot protection (or login, interstitial, etc.) running that doesn&#8217;t have an index.&#8221;<\/p>\n<p>He also added that it would be better if they provided 503 server status codes. &#8220;It would be better to use 503 for server-side blocks like bot protection,&#8221; he wrote.<\/p>\n<p>In the end the person he replied to had a different problem causing the noindex, it wasn&#8217;t bot protection.  He said it was &#8220;some shenanigans with NextJS doing client-side rendering instead of server-side rendering.&#8221;  &#8220;The HTML had a noindex tag, which was removed after rendering in the browser. The HTML displayed in GSC is AFTER the JavaScript rendering, so I couldn&#8217;t find it and thought it was a bug of GSC,&#8221; he added.<\/p>\n<p>This was the error I was seeing:<\/p>\n<p><a href=\"https:\/\/s3.amazonaws.com\/images.seroundtable.com\/gsc-noindex-shot-1703679941.jpg\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" src=\"https:\/\/s3.amazonaws.com\/images.seroundtable.com\/gsc-noindex-shot-1703679941.jpg\" style=\"max-width: 100%\" alt=\"Gsc Noindex Shot\" \/><\/a><\/p>\n<p>Here are those posts:<\/p>\n<p lang=\"en\" dir=\"ltr\" xml:lang=\"en\">wtf is going on here?  why isn&#8217;t google indexing this one? <\/p>\n<p>They say there is a &#8220;noindex&#8221; tag, but there isn&#8217;t one.  robots.txt looks good too.<\/p>\n<p>No one? <a href=\"https:\/\/twitter.com\/hashtag\/seo?src=hash&#038;ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">#seo<\/a> help <a href=\"https:\/\/t.co\/DYzJsLQ4Yk\">pic.twitter.com\/DYzJsLQ4Yk<\/a><\/p>\n<p>\u2014 Guilherme (@goenning) <a href=\"https:\/\/twitter.com\/goenning\/status\/1739580495795474699?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">December 26, 2023<\/a> <\/p>\n<p lang=\"en\" dir=\"ltr\" xml:lang=\"en\">Thanks John, I finally found it thanks to <a href=\"https:\/\/twitter.com\/HamilcDev?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">@HamilcDev<\/a> <\/p>\n<p>Some cheats with NextJS doing client-side rendering instead of server-side rendering<\/p>\n<p>The HTML had a noindex tag, which was removed after rendering to the browser.  The HTML displayed in GSC is AFTER rendering the JavaScript,\u2026 <a href=\"https:\/\/t.co\/NffU7suVLM\">https:\/\/t.co\/NffU7suVLM<\/a><\/p>\n<p>\u2014 Guilherme (@goenning) <a href=\"https:\/\/twitter.com\/goenning\/status\/1739739298151178403?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">December 26, 2023<\/a> <\/p>\n<p>Discussion in the forum a <a href=\"https:\/\/twitter.com\/JohnMu\/status\/1739734474751938926\" target=\"_blank\" rel=\"noopener\">X<\/a>.<\/p>\n<p>\t    <script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><br \/>\n<br \/>[ad_2]<br \/>\n<br \/><a href=\"https:\/\/www.seroundtable.com\/google-bot-protection-services-serve-noindex-36624.html\" target=\"_blank\" rel=\"noopener\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Did you know that sometimes these bot protection services will serve the noindex guidelines of Google and other search engines? Google&#8217;s John Mueller had this to say Xsaying &#8220;Sometimes there&#8217;s a bot protection (or login, interstitial, etc.) running that doesn&#8217;t have an index.&#8221; He also added that it would be better if they provided 503 server status codes. &#8220;It would be better to use 503 for server-side blocks like bot protection,&#8221; he wrote. In the&#8230; <\/p>\n","protected":false},"author":1,"featured_media":19744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-19743","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo-news"],"_links":{"self":[{"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/posts\/19743","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/afreeurl.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=19743"}],"version-history":[{"count":1,"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/posts\/19743\/revisions"}],"predecessor-version":[{"id":19745,"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/posts\/19743\/revisions\/19745"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/afreeurl.com\/index.php?rest_route=\/wp\/v2\/media\/19744"}],"wp:attachment":[{"href":"https:\/\/afreeurl.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=19743"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/afreeurl.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=19743"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/afreeurl.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=19743"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}