Network Security or Website Security?

[Tanooki]

This is just more of a general question for anyone who does network security or knows much about securing websites...

I was wondering if anyone who has some time might be able to explain the robots.txt used for websites... If anyone had any spare time to discuss this with me it would be greatly appreciated.

 

[Lafinass]

Essentially robots.txt specifies which pages or data should not be accessible by which bots. It is an exclusionary list, so anything not on that list that would be otherwise accessible is fair game. Of course, this all stands upon the compliance of the bots.

If, for example, I run a community that has a roster page with names on it and I don't want that page showing up in Google searches I would specify that the page is off-limits for Googlebot.

Again though, it depends on the bot complying. Public content on a website can't reasonably be protected from bots that ignore robots.txt. Robots.txt is basically just a 'do not trespass' sign, it won't stop the burglar from hopping your fence.

Check the wiki page for some more specific examples:

http://en.wikipedia.org/wiki/Robots.txt

 

[Flippy the Squirrel]

You can also check out http://www.robotstxt.org ... as that's a good reference for that stuff...

Also check out the specification at http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

 

[Tanooki]

Alright thanks guys! I understand it a lot better now. The main security guy at my college just randomly came into our class during my Server 2003 lecture and started talking about more of the security side of things... I found it really interesting... and then he got on the topic about robots.txt which he just briefly explained and I didn't comprehend it. This makes it much more clearer, thanks!