Here is a url you can visit to learn about the file attached in this post: The Web Robots Pages
And here is the file you have to install on the root of your vBulletin forum path to make it work: robots.txt
Why this file is useful for your site?
Because most of the time, you do not want the web crawlers like Google trying to post a new event in your calendar or trying to enter the modCP…
Web Crawlers have no idea of what they do on these pages… they just follow each and every link in your pages, and the access to the calendar is usually free to all in your navbar… they check that link, visit the calendar and visit the “create new event” link as well, so they seems to create a new event, but they just visit the link they see, nothing more, there is not mad intent in that.
The file i attach with this post is already known in the vBulletin community, and you have to use a robots.txt file in the root of your forum to make the web crawlers aware of your forum structure and what-not.
You can read the file, it helps understand what is block (Disabled) so if you want to enable one of these files, just delete it from the list… but remember that the list i provide is the one that makes more sense… nobody want the robots to visit the admincp…
