"These sites should be penalised for click-baiting and they have been doing it for years."
If SEO works and a result appears closer to result #1 in the SERPs, then the "true", non SEO-assisted result it is displacing would appear further from result #1. Apply this across the board and what we have are many, many non-SEO results that are pushed down in Google's ranking. No one is "penalising" these pages, however they suffer visibility problems because they have not engaegd in SEO. The incentives created by Google's secretive ranking system and online advertising commercial focus are perverse or at least in conflict with the user's goals. Google discourages and even prevents any user from looking at results that were hits but were not ranked high. Pages that may not succombed to the the influence of such incentives may "disappear".
What if a user understands this and wants ignore the Google ranking system. What if the user wants to see the true, non-SEO results. Google actively limits the user's ability to see those displaced results. For example if a user searches for a common term, such as "example", she will not be able to view more than 200-300 results. Elsewhere in this thread someone also noted even with a paid API, Google limits users to 1000 results. If the user wants to the see the full range of pages that have hits for the word "example", she cannot do so. If the user would like to perform a single search for all pages containing the term "example" and then sort by some other objective criteria such as alphabetical by domainname, date, page size, etc., she cannot do so.
Under Google's model of the web, pages that do not acquiesce to an online advertising company's secretive ranking system may become nondiscoverable, despite the fact that they may indeed match the user's query. Computers assist us in searching through data but "relevance" is ultimately decided by the user. That is why we can have HN threads that claim search result quality is declining. Though they may be slower, humans can determine relevance better than any computer. From the disclosures of Matt Cutts and others we know that humans are involved in Google's ranking implementation. Penalties are used. The search process is not 100% math/computer-based. However, in Google's model of the web, filtering results is the exclusive domain of the online advertising company and only the humans on its payroll, not the user performing the search. There is no option to disable the online advertising company's "assistance" in filtering.
If SEO works and a result appears closer to result #1 in the SERPs, then the "true", non SEO-assisted result it is displacing would appear further from result #1. Apply this across the board and what we have are many, many non-SEO results that are pushed down in Google's ranking. No one is "penalising" these pages, however they suffer visibility problems because they have not engaegd in SEO. The incentives created by Google's secretive ranking system and online advertising commercial focus are perverse or at least in conflict with the user's goals. Google discourages and even prevents any user from looking at results that were hits but were not ranked high. Pages that may not succombed to the the influence of such incentives may "disappear".
What if a user understands this and wants ignore the Google ranking system. What if the user wants to see the true, non-SEO results. Google actively limits the user's ability to see those displaced results. For example if a user searches for a common term, such as "example", she will not be able to view more than 200-300 results. Elsewhere in this thread someone also noted even with a paid API, Google limits users to 1000 results. If the user wants to the see the full range of pages that have hits for the word "example", she cannot do so. If the user would like to perform a single search for all pages containing the term "example" and then sort by some other objective criteria such as alphabetical by domainname, date, page size, etc., she cannot do so.
Under Google's model of the web, pages that do not acquiesce to an online advertising company's secretive ranking system may become nondiscoverable, despite the fact that they may indeed match the user's query. Computers assist us in searching through data but "relevance" is ultimately decided by the user. That is why we can have HN threads that claim search result quality is declining. Though they may be slower, humans can determine relevance better than any computer. From the disclosures of Matt Cutts and others we know that humans are involved in Google's ranking implementation. Penalties are used. The search process is not 100% math/computer-based. However, in Google's model of the web, filtering results is the exclusive domain of the online advertising company and only the humans on its payroll, not the user performing the search. There is no option to disable the online advertising company's "assistance" in filtering.