Source.

Time for another brief installment in the series of posts on search, indexing, and Google Search Console. Last year I linked a post and some HN discussion about changes in indexing by Big Search. The discussion digressed into grievances about Google removing personal pages from its index. These were echoed in a more recent post that asserts that these observations indicate a new search strategy by Google.

Vincent Schmalbach

Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.

The comments section pointed out the obvious:

crazygringo

This post seems to be based entirely on personal anecdotal experience. There isn't a shred of hard data to support the headline claim that Google now "defaults to not indexing content". Google never indexed everything, removing duplicates, blogspam, useless pages, etc. Maybe they've changed their thresholds or maybe not. But this post provides zero evidence of anything. It's pure speculation without any facts at all.

OP showed up to say, "you're not wrong, crazygringo":

Vincent Schmalbach

You're right, this post is based on personal anecdotal experience. I have access to Google Search Console data for over 100 websites, and most have many pages in the "Discovered - currently not indexed" and "Crawled - currently not indexed" categories, despite ranking well for some keywords and getting traffic. This wasn't the case 10 years ago. Regarding "Google never indexed everything" - I'd say it came close. They did manual de-indexing for heavy spam sites and would even send an email when they did this. Apart from that, nearly everything was in the index, including duplicates. De-duplication happened at the ranking stage, not the indexing stage.

Whether or not Vincent has accurately described Google's new strategy, crazygringo is denying an actual trend simply because there's no public announcement or leaked memo. In a way, this only matters to small-time web publishers and SEOs. And yet it's also just weird to think that this fixture of the internet could be moving from "index everything" to "America Online was right all along".

aiauthoritydev

I think the problem is not Google specific rather the internet has grown far too large with too much of crap floating around. Google, in my opinion has done the best job of getting relevant information followed by Reddit. While OpenAI etc. is pretty good (so does Google Gemini) what is OpenAI like interfaces prevent me from doing is to segue from a focused topic to related areas to discover knowledge on the periphery, which is the most important aspect of learning in my opinion which chatbots today are not able to do that well.

Moment of Zen

My web index pointed me to another Chris who also has a characters page.

Related / internal

Related / external