Posting to Autocat
On 22/08/2011 04:15, Thomas Krichel wrote:
James Weinheimer writes
Again, with materials that are already on the web, everything is different since if Google had to get prior permission from each and every website before they could crawl any content, the web would be impossible to use.
I beg to differ. It would be more difficult to search, since indexing would be much more limited. But you could still use link collections, starting from a s(ite that you know, say for example your local library web page. Such link collections were popular in the early days of the web when search engines were not as good as they are today.
I see your point, and I don’t want to belabor the issue, but if all we had were those horrible link collections like in the old days, the web would not be nearly as useful as it is now. I remember how much I hated them, and found them next to useless even then, when there were in fact, a lot fewer sites. Certainly, if that were all people had at their disposal, they would be used occasionally because it would be that or nothing, just as the library catalog was used much more when it was the only tool available. Lots of people would still choose nothing however, just as they often did back then with the web, clicking from document to document randomly, just as when they trolled (and still troll today) the shelves randomly in the library without consulting the catalog.
Anyway, what I was getting at is that if a web search engine had to get one-to-one permission from each website *before* they could index it, the practical issues would make it impossible, just as the same issue with printed materials may kill the Google books settlement. It is so often difficult even to find out who to ask and to get a working address (just like the printed materials), but with websites, the responsibility is squarely in the hands of the owners of the website to opt-out of Google indexing. If they do not add a robots.txt exception, it is assumed that they want their sites to be indexed.
It seems as the only answer will be new copyright laws, but seeing how matters stand now in almost all the world’s legislatures, such new laws will be a very long time in coming. Again, it is sad because of the tremendous boon it will mean to society when it is done–not if, since it will happen sooner or later–but it may take a lot later now, to the disadvantage of everyone, including the publishers and the authors themselves, whose works will be far more difficult to discover and therefore will be less used.