What about further development of /robots.txt?

There are no efforts on this site to further develop /robots.txt, and I am not aware of technical standards bodies like the IETF or W3C working in this area.

There are some industry efforts to extend /robots.txt. See for example Google's support for wildcards in Disallow and Sitemap line. But given the inherent limitations of /robots.txt, it's potential is limited.

More interesting are newer mechanisms such as Sitemaps.