Perhaps the problem you're describing is actually Google's bittersweet solution to the problem of not having unlimited storage space to properly index the whole internet.
The whole text of the internet from day 1 and the index machinery for it can be stored in some number of petabytes. I assume the CIA and the Chinese state are in possession of such stores and it is a minor budget item.
