.Google.com's John Mueller answered an inquiry regarding why Google.com marks web pages that are actually disallowed from creeping through robots.txt as well as why the it is actually secure to neglect the associated Search Console files regarding those crawls.Bot Visitor Traffic To Inquiry Criterion URLs.The individual asking the question documented that bots were actually generating hyperlinks to non-existent question parameter Links (? q= xyz) to web pages along with noindex meta tags that are actually likewise obstructed in robots.txt. What urged the concern is that Google is creeping the web links to those web pages, receiving blocked out through robots.txt (without noticing a noindex robotics meta tag) after that getting shown up in Google Look Console as "Indexed, though blocked out by robots.txt.".The person asked the observing concern:." However listed here is actually the large inquiry: why would certainly Google.com index webpages when they can not also observe the material? What is actually the advantage because?".Google's John Mueller verified that if they can not creep the webpage they can not observe the noindex meta tag. He additionally helps make an intriguing mention of the web site: search operator, urging to ignore the results due to the fact that the "ordinary" users won't find those outcomes.He wrote:." Yes, you're correct: if our company can not crawl the page, our team can't see the noindex. That stated, if our company can not creep the webpages, then there's certainly not a great deal for our team to mark. So while you may view a few of those web pages with a targeted site:- inquiry, the typical customer won't see all of them, so I would not fuss over it. Noindex is additionally alright (without robots.txt disallow), it just implies the Links are going to wind up being actually crawled (and find yourself in the Browse Console report for crawled/not listed-- neither of these statuses create issues to the rest of the internet site). The vital part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the restrictions in using the Web site: search accelerated search driver for analysis factors. Some of those causes is actually considering that it's certainly not connected to the regular search index, it is actually a separate thing entirely.Google's John Mueller talked about the site hunt driver in 2021:." The brief response is actually that a web site: inquiry is not meant to become complete, nor utilized for diagnostics purposes.A site inquiry is actually a specific sort of search that limits the results to a certain internet site. It's generally only the word web site, a digestive tract, and then the site's domain.This query confines the end results to a certain website. It is actually not implied to become a detailed selection of all the web pages coming from that website.".2. Noindex tag without using a robots.txt is actually great for these kinds of situations where a robot is linking to non-existent pages that are obtaining discovered through Googlebot.3. URLs with the noindex tag are going to generate a "crawled/not catalogued" entry in Browse Console and that those will not have an adverse result on the rest of the site.Read through the question as well as answer on LinkedIn:.Why will Google index pages when they can't even see the web content?Included Photo by Shutterstock/Krakenimages. com.