Robots Standards Reference
One very useful tool for Web Administrators lies in the Robots Exclusion Protocol. It’s a handy way to tell search engines certain actions you want them to take when crawling your site(s). This is typically done via a robots.txt file or various META tags within the (x)html. I knew about these tools and have been using them for a couple of years. What I did not know is that Yahoo!, Google, and Microsoft have been collaborating on these efforts for a while now.
Over the last couple of years, we’ve been collaborating with Google and Microsoft to make webmasters’ efforts more effective across the major search engines. By bringing standards such as Sitemaps and improvements like auto-discovery and cross-host submission, webmasters can simplify their account management across the different search engines.
Yahoo!, in their Yahoo! Search Blog, has outlined all the details needed to successfully create this data. Take a look at their site because it’s a reference you may need!
One Standard Fits All: Robots Exclusion Protocol for Yahoo!, Google and Microsoft
If you enjoyed this post, please consider to leave a comment or subscribe to the feed and get future articles delivered to your feed reader.
Comments
No comments yet.
Leave a comment