1. 14
  1.  

  2. 3

    Related: I always had the idea that one could create a reverse-robots.txt-search engine (aka it indexes only stuff robots.txt tells it shouldn’t index). Not aware of anyone ever did it, but it probably would be interesting.

    I’d expect that plenty of people mistakenly think robots.txt is a security mechanism.

    1. 1

      Unfortunately (DMCA, etc) I would assume this would likely be considered a criminal act, and Americans or US entities anywhere in the world may be found to be committing crimes by even linking to the results of such an engine.

      It sure would be neat to see however, there apepar very few places it would be both ‘safe’ and stable enough to run such a site.