Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to the site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.

Two clients that hired me for recent work saw their rankings shoot to the top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve the same instant ranking boost overnight.

Those clients that do succeed in this way are often thrilled with the results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.

WHY DO NEW SITES SUFFER?

What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in the form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet the site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all the titles, tags & page text, created a complete site map and placed links to all these resources on the front page.

I know that the site is not being crawled because Google's cached copy of the front page shows it before we did the work four months ago, without the new links and without title tags. We've submitted the site by hand, (manually) once a month for three months via the Google Add URL page. http://www.google.com/addurl.html When the hand submission failed to get it re-indexed for four months, we submitted the sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!

Part of the problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to the site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in the URL. This is to avoid crashing the server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1

The solution to this dynamic URL problem has been discussed widely in search engine forums and solutions have been bandied about including software provided by SEO's, URL re-write techniques for dynamic pages on APACHE servers http://www.alistapart.com/articles/urls/ and PHP pages http://www.stargeek.com/php-seo.php to generate search engine friendly URL's. Others recommend simply adding static HTML sitemap pages as alternatives for the search engine spiders. In this instance the client's developer simply said "I can't do that (PHP solution) on this server". So we resorted to putting up the static HTML sitemap pages with hard-coded

URLS to the main 54 pages of the site at http://lawfirm411.com/Law-Firm-411-sitemap.html This should get at least those fifty pages crawled by Googlebot, but Googles' spider appears not to be crawling this site at all. How do we know this? See for yourself by using the following query in the search box at Google: allinurl:www.lawfirm411.com where the result page shows ONE page in the results. If you try that query on your own site (replace your own domain name for lawfirm411.com), you'll see the results lists ALL your pages.

The site home page was crawled by Google four months ago, when they took their "Cached Snapshot" of the page. You can see this by visiting the Google cached page here: http://66.102.7.104/search?sourceid=navclient&;ie=UTF-8&q=cache:www.lawfirm411.com where the date of this snapshot is "Apr 20, 2004 07:42:19 GMT" and they haven't been back since. The page in that snapshot has none of the newly added links, an outdated title tag, and old content.

This problem is not unique to this site. One client we worked with two years ago had a dynamically generated, framed site! Those two site structures have always given search engines trouble. Their site was not crawled at all and only the front page showed up. Our solution was to create a second domain (owned by the client), which had static HTML pages that precisely mirrored the content of the client's framed, dynamically generated site. Guess what happened after Googlebot crawled the static site? Google indexed the framed site in full and then banned the static site from the index! Not an approach we advocate, but the one that worked for this client.

We're still searching for ways to get Googlebot back to LawFirm411.com before creating that new static site, but decided to share this odd experience with the SEO community before going to any extremes. Google provides over 70% of most search engine referred traffic to ALL of our clients and we realized we can't site idly by and see a major client languish because Googlebot didn't like what it found at the client site on the first visit four months ago. This issue dogs newer sites in other places as well. The Open Directory Project has also become notoriously slow in adding new sites to the directory and in this case, has not picked up this site even after 6 regular monthly submissions. The web playing field may have begun tilting toward older, established sites and away from new ones.

------------------------------------------------------------

Mike Banks Valentine is the SEO for http://www.lawfirm411.com

Contact him at http://www.seoptimism.com/SEO_ Contact.htm

Improve Your Small Business Online at our Ecommerce Tutorial

http://website101.com/Fr ee-Tutorials/index.html

------------------------------------------------------------

Search Engine Optimization for Beginners

If you are confused about terms like "search engine optimization" or having a "search engine... Read More

Keyword Popularity Check

Before using keywords for your site, it is always better to check for the popularity... Read More

SEO and the Outsourcing of Inbound Link Building

Search Engine Optimization nowadays has a lot to do with building inbound links to your... Read More

Great Site Ranking in Google The Secrets Out

How many years did you register your domain name for? If it was only one... Read More

Introduction to Google Page Rank (PR)

For anyone looking to enhance their Google Page Rank (PR) to get a better place... Read More

Google Groups

Some very early users of the Internet - not the worldwide web as we know... Read More

Is Google Fair?

If you are the owner of a new website, trying to get a decent ranking... Read More

Were You Bamboozled by Google?

It's still happening. "Florida," "Boston," "Ginger," and "Brandy" are just a few of the many... Read More

Google ? A Bit of History

The first question most people have is, "What the heck is a "Google?" It is... Read More

New Site and Sandbox: How to Get Rid of It

You have put lot of sweat in making your site. Now you want it to... Read More

Getting Honest With The Search Engines

Getting Honest With The Search EnginesI spend a lot of time reading newsletters and forum... Read More

How to Boost Your Traffic and Profits with Content!

Are you aware of how vitally important and valuable CONTENT is to your online business?... Read More

Are You Making These Deadly SEO Mistakes?

Black Hat SEO: Web Spamming and Linking to Bad NeighborhoodsSo you want to exchange links... Read More

Why Pay-Per-Inclusion Search Engines are Dying

A Pay-Per-Inclusion search engine is a service in which a search engine charges you a... Read More

Link Popularity Explained and How To Build Links

Link popularity is the single most influential factor for determining how well a web site... Read More

Creating a Google SiteMap For Your Work At Home Business Web Page

Search engine traffic is the best traffic You can get for your online Business. So... Read More

SEO Expert Guide - Sitewide Optimization (part 4/10)

In parts 1 and 2 you learnt how to develop your online business proposition and... Read More

Cache in the Bank: Understanding Googles Advanced Operators

If you would like to know when your site was last indexed by Google, you... Read More

Search Engine Marketing 101: What Search Engines See When They Visit Your Web Site

If you have a Web site, have you ever wondered what a search engine sees... Read More

20 Ideas for Creating Traffic Rich, Search Engine Friendly Pages

Sometimes questions will arise around the subject of gateway information pages or doorway pages. People... Read More

Would You Like More PageRank?

The higher the PR of your site the higher will be its search engine position.... Read More

How to Top Google by Writing Articles

Search engines determine their rankings based on two things: Is your site relevant? (Optimized... Read More

Do-It-Yourself Search Engine Optimization

Search engine optimization, or SEO, is big business. If you rely on search engines to... Read More

Owning Your Category Online - Is Our Website Broken?

"Owning your business category online is more than just a few keywords... Is your website... Read More

Give the Folks at Google What They Want

Recent developments on the Google front have web marketers and SEO specialists talking even more... Read More

Keyword Optimization - How To Achieve It

The most important thing you can do for your business and your website is develop... Read More

Is New MSN Search More Precise? Just Ask Google

MSN finally unleashed its new search technology to the world on Monday. The official announcement... Read More

To Understand the Success of Website Ranking

Time is a factorTo obtain positive results is not very fast to achieve. It always... Read More

Got Spiders?

Many internet marketers blow mountains of start-up cash on their websites just trying to break... Read More

The Real Search Engine Optimization Guide

Nowadays, there is so much talk about SEO (search engine optimization) that it has become... Read More

Search Engine Spiders Lost Without Guidance - Post This Sign!

The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them... Read More

Hens Lay Eggs (HLE) by Silke Stahl

"HLE" is a bit of a joke term, (or possible tongue in cheek could be... Read More

Determining the Value of Your SEO Service

Every once in a while--and probably more often than we should--we find ourselves reviewing our... Read More

Link Popularity - Basic Overview

There are many techniques that SEM/SEO experts use to optimize Web sites. One of the... Read More

Link Popularity

In today scenario when we talk about Search Engine Optimization, we also talk about one... Read More

SEO Trade Secrets - 8 Great Tools for Search Engine Optimization

About 80% of website traffic comes through search engines. And research shows, if you're not... Read More

How To Become an SEO Expert - 5 Secrets That Will Allow You to Outperform 95% of All Webmasters

Becoming an SEO expert, or a search engine optimization expert, is something that is worth... Read More

Linking Strategies to Skyrocket to the Top of Google

If you don't know already, one of the key success factors to getting loads of... Read More

How Google Indexes Content From Your Web Directory

In a fluke, I was able to notice something about the way Google indexes content... Read More

Are You Hung-Up on Page Rank and Back Links?

It's unfortunate that many website owners are so hung-up on Page Rank, they'll rarely if... Read More

Duplicate Content Penalty - How to Lose Google Ranking Fast!

Duplicate content penalty. Ever heard of it? This penalty is applied by Google and possibly... Read More

Finding Targeted Keyword Phrases Your Competitors Miss

Finding Targeted Keyword Phrases Your Competitors MissFinding keyword phrases your competition is missing is easier... Read More

Google Contest - Nigritude Ultramarine

Search engine optimization experts are having fun with Google. Experts, with DarkBlue.com at... Read More

The Hottest Tip On Getting A Web Site Listed Quickly by Yahoo, Inktomi, MSN, FAST and AltaVista

Every webmaster knows that the free search engines are a major source of traffic.They also... Read More

Designing a Better System for Search Engines

Designing a Better System for Search Engines and Information Distribution, information and knowledge are power.... Read More

An Easy Way Not to Get Banned by Google

Strategic search engine optimization involves far more than keyword research, META tags and content.If you... Read More

Do Search Engines Scare You?

What do the words "Search Engine" make you think of? I get immediate mind-pictures which... Read More

Website Copy-writing for Higher Sales and Higher Search Engine Ranking

Why, you ask? Mainly, because search engines want to provide their users with good content.... Read More

Types of Links

Internal Linking ? An introductionThe web as they call it is a mesh of links.... Read More

Keywords, Ranking, & Search Engine Optimization Fun

I am a Search Engine Optimization newbie. I have read a little on various forums,... Read More