How do I use /robots.txt on a shared host?
If you share a host with other people, and you have a URL like http://www.example.com/~username/ or http://www.example.com/username, then you can't have your own /robots.txt file. If you want to use /robots.txt you'll have to ask the host administrator to help you.
If you want more control, switch to a provider with a virtual host.