The Web Robot Pages is an information resource dedicated to web robots. Initially hosted at WebCrawler in 1995, it moved to this dedicated site hosted by independent robotstxt.org in 2000. It underwent a modernisation in 2007.
Please support this site
The site is maintained by volunteers, and running costs are partly funded by (AdBlock friendly) advertising. If you've found the information here beneficial please consider making a donation towards our web hosting expenses.
At this time we do not offer advertising opportunities to new partners, nor are we interested in selling the domain.
To contact the administrators of this site regarding technical issues relating to the operation of this site only, please use the contact page.
The logo is based on this cartoon donated by Diesel Sweeties: