Robots Database Export

Here you can obtain the raw data from The Robots Database.

Conditions of Use

• You may use this data freely.
• This data is contributed by third parties, sometimes anonymously. We make no guarantees to the validity of the data, or its suitability for any purpose.
• You must not download this data more often than once in 24 hours.
• If you use this data to create content which you then publish, we would appreciate a link back to http://www.robotstxt.org/
• If you want to refer others to this data, please link to http://www.robotstxt.org/db.html rather than the data file.

Format

The data is in the old robotsdb format, see schema.txt.

Download

Download all.txt.