A Few Words About Search Engine Spiders

As a website owner and someone concerned about Search Engine Optimization, you don't have to suffer from Arachnophobia. Search Engine Spiders are your friends!

The search engines deploy programs called spiders to transverse the web and index all content available on the internet. These programs follow a set of rules defined by each search engines algorithm. Spiders are in search of information, and SEO is the process of providing information to these spiders in a manner that will boost relevancy.

Search Engine Spiders follow links on your website to other pages within your website, and to other websites to which your website links. Factors that influence spiders, and crawlabilty: Robots.txt file - most robots will look at this to determine where they can and cannot go on the website. In some cases if a spider cannot find this file it will not index the website.

Of course, don't forget about a Google sitemap. (www.xml-sitemaps.com ) is a good starting point. A Google sitemap doesn't guarantee inclusion by Google, but it most certainly doesn't hurt. It enables Google to properly and easily index your site.

It's also a good idea to have a separate XML site map and an HTML version for your visitors to see and utilize. See http://www.kaboodleventures.com/sitemap.xml for an example.

* URL's - Clean urls make it easy for the spider to index content. Variables and long url strings will hinder the spiders, if you site is built in ASP or PHP you may need to do a ModRewrite to create clean urls on the fly while still enabling your Content Management System features to be utilized.

*Good: http://www.somedomain.com/content/subcontent/pagename.html
*Bad: http://www.somedomain.com/index.html?catid=8&location_page=somevariable-44
*Place your primary keyword terms near the top of the document
*Place your keyword terms in the body surrounded by other relevant terms to establish context
* Place your keyword terms again in the bottom of the document
* Keep individual pages under 80k
* It's not a bad idea to place a "navigation bar" above your JavaScript, Flash or Form elements, and maybe even again at the bottom of each page
* Spiders do not read JavaScript, Flash, or Form elements
* Spiders will not view pages accessible only by forms

Although there are slight differences between the algorithms, all the search engines look for the same things that the human visitor values.
* A simple, cleanly coded design
* Well-thought-out, intuitive navigation
* Well-written, keyword rich descriptive copy
* Titles and Meta tags that help identify relevant keyword phrases
* Links that accurately describe what can be found at the site

Spiders, regardless of their crawling ability and their sometimes "creepiness" are really your friends. If you treat them right, feed them what they need, they will most definitely be your friend.