Google Search: the lifecycle of a codified system
from simple and successful to complex and sclerotic
Recently, an increasing number of users have begun to notice that Google Search does not work as well as it used to. Search results are increasingly irrelevant, especially for search terms that are even remotely monetisable. A popular hack to get relevant results is to add "site:http://reddit.com" to the search term to get relevant results.
So why is this so?
Optimisation
Some argue that the primary reason for the deterioration is that Google continues to climb up the local hill which maximises its ad revenue. On popular search terms, Google shows more paid ads at the top of the page and for most such terms, improving the quality of the organic results is simply not revenue-maximising for Google. It may even be optimal to show worse organic results to some extent as Sergey Brin and Larry Page identified in their original paper on the Google algorithm:
advertising income often provides an incentive to provide poor quality search results
The same deterioration is also evident in the product search results on Amazon where sponsored results not only dominate the results view, they make the organic results worse. This neglect of organic search results is enabled by the market dominance of Google and Amazon.
Adaptive Arms Race
However, the primary reason for the deterioration in Google’s search results is the adaptive arms race that affects all codified systems. The adaptive arms race is the primary reason why all codified systems transition from simple and successful to complex and sclerotic and Google’s search engine is a great example of this phenomenon.
Google’s search engine started as a simple, effective algorithm that measures and ranks a system that has not yet adapted to its influence. The success of the algorithm meant that Google could monetise it via paid search ads. However, almost from inception, the success of the system means that those being measured and ranked begin to adapt to the system that measures it. An entire field of Search Engine Optimisation (SEO) was born with the sole purpose of gaming Google’s search algorithm. As websites change themselves to get higher in Google’s search rankings, Google in turn adapts to prevent itself being gamed. Over time, the system becomes more and more complex and full of kludges on both sides i.e. both the algorithm itself and the websites that it indexes and ranks become more complex. However, more complex control cannot overcome Goodhart’s Law.
Google’s size and impact mean that it shapes the system it measures, i.e. it shapes the entire internet. The internet itself becomes worse as a source of information due to the nature of this arms race. The Internet is full of mediocre SEO-driven content that has made Google worse partly because it has made the internet itself worse (which is why the Dead Internet Theory' exists). The websites Google now indexes are content-free pages written solely to the needs of Google’s criteria for ranking web pages.
And the increasing ability of AI to generate content will only make this problem worse. The future of the internet and search is AI-generated content trying to fool Google into thinking that it is written by a human and Google trying to distinguish AI-written content from human content. If the internet isn't already dead, it soon will be.