I use the who's online add-on and occasionaly I've noticed it shows several guests - sometimes as many as 3 - being logged in, they normaly dissappear very quickly after I have logged in as admin. This is unusual as I do not have the tick in Allow non-members to browse the forum.

Are these normal web-bots (search engines and the like) showing up or something more sinister?

I don't think so, I'm the only one who knows the current URL as it is in testing on my server.

Yup, these are search engine bots usually Microsoft spiders, I check the IP addresses log and they point to microsoft corporation.
Does not seem to matter what's in the robots.txt file, they keep on a-comin' and indexing.

Gotta love Microsoft's blatant disregard for everything.