SiteMaps & Robots & SEO Oh My!
In November of 2006 the 3 major search engines at the time adopted the sitemaps protocol for better indexing initiatives. The purpose of a sitemap is to allocate one place where every URL link to every resource in your website can be found. Effectively accomplishing a couple general goals that will increase your search engine friendly internet business marketing. First by allowing a search engine spiders a centralized "map" to access all the information on your website. As well as provide your visitor traffic with a quick reference resource to quickly locate areas of your site if they get lost.
There are a couple ways of effectively employing the use of a site map in your optimization efforts. The more helpful method is quite a bit more technical, and may require you to learn a little more about site maps and how search engines utilize the protocol. For example how google uses sitemaps is based on the most current schema as set by sitemaps.org and is entirely XML at it's best. For a better example, check out the XML Site Map for this website.
Again, this is a beginner's guide to search engine optimization so we aren't going to completely dissect and detail the benefits of a search engine friendly sitemap, but let's take a quick closer look before moving on. If you peeked at our XML SiteMap you saw numerous listings formatted like this sample.
http://www.example.com/
2009-01-01
monthly
0.8
- SiteMap Must Have's
- Begin with an opening tag and end with a closing tag.
- Include a entry for each URL as a parent XML tag.
- Include a child entry for each parent tag.
If you want to know more about XML sitemaps please utilize the links listed above or refer to another article we have done for a more in depth explanation. For now though we need to move on, so let's talk about the other type of search engine friendly site map implementation.
This site map is for the human website visitors that provide your search engine traffic. It's an easily found page on your website that provides links to as much of your webpage's content as possible but avoid repetitive links. The idea is very simple. A well titled link for each page of your most prime content in an easily navigated and well organized spot. Even though you should utilize effective navigation to your web design that shouldn't give your visitor a reason to use it. A good text and link based site map can help the robots and spiders that crawl your website, accomplishing better search engine optimization. For an example of what we're talking about click here for ours so you can get a better idea. Moving on...
 What is the Robots.txt file? More importantly, how to use it.
In a nutshell, webmasters use the robots.txt to provide instructions on how search engine robots can index your site. It's called the Robots Exclusion Protocol and this is how it works. A search engine robot visits your website and wants to behave like normal traffic. Except chances are there are some elements of your site that you don't want to be indexed. So the robots will first check for a robots.txt file to make sure there's nothing you would like them to skip.
Remember two main points about using the robots.txt, the search engine robots can ignore them. Especially in the case of potentially malicious robots harvesting your data. Also they're publicly available, so anyone can see the directories you want to keep from being seen by using it. The first step is to actually create the robots.txt file to upload to your website's root directory via FTP or however you like. That means it should be accessible from your root domain as such : http://www.professionalsweb.net/robots.txt
If you look at our example you will notice some very simple text commands that allow the search engine robots to know they can skip certain directories. Let's brake the example down real quick.
User-agent: * is telling all your robot visitors that the following statements apply to them by using the asterisk wildcard. If you wanted to disallow certain directories to be indexed by individual robots you just enter a seperate entry with it's own paramaters for that specific robot. For example User-agent: Googlebot or User-agent: Googlebot-Image will tell those specific search engine robots to follow the statements you declare under it. You can also use it to keep potentially bad search robots from indexing your site as well. We won't cover what we mean by "bad robots" for now though look to the example robots.txt below to see how to do it if you needed. Disallow: /directory/ will tell the robot that you want it to skip that directory all together. You can name as many directories as you like. This is extremely helpful for keeping copyrighted images and backend elements from being indexed unnecessarily. If for some reason you don't want the search engine robots to index anything on your site, just use the asterisk wildcard and declare Disallow: /*
robots.txt Example
User-agent: /*
Disallow: /images/
Disallow: /admin/
[ This statement would tell all robots to disallow indexing of the images and admin directories, as well as all content contained in them. ]
User-agent: BadSearchRobot
Disallow: /*
[ This statement would tell a search engine robot that you have identified as unhelpful in any way that you don't want to be indexed by them. ]
You get the general idea by now I'm sure. This file is just one more small element that can improve your search engine optimization efforts and allow you to censor any content you don't need indexed. It's a simple concept but if you need any further information try the official robots.txt file project page. Or see how google uses the robots.txt file for more information. Whenever you're ready to continue the SEO lesson go ahead and continue for more fun and excitement. :)
Better SEO Introduction | Keywords and Keyphrase | Title Relavency | Dynamic Content | Navigation / Association | Meta Tags Explained | SiteMaps and Robots | HTML Optimization | Inbound & Outbound Links | Link Building | Search Engine Submission
|