In recent news, Google has put forth a proposal known as the "Web Environment Integrity Explainer", authored by four of its engineers. On the surface, it
It would stop beneficial bots like the ones I create¹ as a small-time hobbyist because the little guy does not have the resources for this arms race. You may be right when it comes to large-scale scraping ops that are done by a business (e.g. scraping RyanAir or Southwest airlines so an airfare consolidation site can show more fares).
① e.g. I wrote a bot that scraped the real estate market sites, scraped the public transport sites, and found me a house with the shortest public transport commute.
It would stop beneficial bots like the ones I create¹ as a small-time hobbyist because the little guy does not have the resources for this arms race. You may be right when it comes to large-scale scraping ops that are done by a business (e.g. scraping RyanAir or Southwest airlines so an airfare consolidation site can show more fares).
① e.g. I wrote a bot that scraped the real estate market sites, scraped the public transport sites, and found me a house with the shortest public transport commute.