Can a /robots.txt be used in a court of law?
There is no law stating that /robots.txt must be obeyed, nor does it constitute a binding contract between site owner and user, but having a /robots.txt can be relevant in legal cases.
Obviously, IANAL, and if you need legal advice, obtain professional services from a qualified lawyer.
Some high-profile cases involving /robots.txt:
- Healthcare Advocates vs The Internet Archive
- Use of web archive did not constitute hacking, says US court
- Federal Judge Clears Law Firm Accused of Hacking Opponents' Web Archives
- US law firm cleared of robots.txt DMCA hacking charge
- Healthcare Advocates -- The Incredible "Lawyers as Hackers" Case
- More on Silly Lawsuits - Internet Archive and the BBC Flap
- Copiepresse (Belgium) vs Google
- AFP vs Google
- eBay vs Bidder's Edge
And you'll find lots of discussion over at Groklaw.