How do I prevent robots scanning my site?
The quick way to prevent robots visiting your site is put these two lines into the /robots.txt file on your server:
User-agent: * Disallow: /
but this only helps with well-behaved robots.
The quick way to prevent robots visiting your site is put these two lines into the /robots.txt file on your server:
User-agent: * Disallow: /
but this only helps with well-behaved robots.