Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to the site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.

Two clients that hired me for recent work saw their rankings shoot to the top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve the same instant ranking boost overnight.

Those clients that do succeed in this way are often thrilled with the results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.

WHY DO NEW SITES SUFFER?

What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in the form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet the site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all the titles, tags & page text, created a complete site map and placed links to all these resources on the front page.

I know that the site is not being crawled because Google's cached copy of the front page shows it before we did the work four months ago, without the new links and without title tags. We've submitted the site by hand, (manually) once a month for three months via the Google Add URL page. http://www.google.com/addurl.html When the hand submission failed to get it re-indexed for four months, we submitted the sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!

Part of the problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to the site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in the URL. This is to avoid crashing the server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1

The solution to this dynamic URL problem has been discussed widely in search engine forums and solutions have been bandied about including software provided by SEO's, URL re-write techniques for dynamic pages on APACHE servers http://www.alistapart.com/articles/urls/ and PHP pages http://www.stargeek.com/php-seo.php to generate search engine friendly URL's. Others recommend simply adding static HTML sitemap pages as alternatives for the search engine spiders. In this instance the client's developer simply said "I can't do that (PHP solution) on this server". So we resorted to putting up the static HTML sitemap pages with hard-coded

URLS to the main 54 pages of the site at http://lawfirm411.com/Law-Firm-411-sitemap.html This should get at least those fifty pages crawled by Googlebot, but Googles' spider appears not to be crawling this site at all. How do we know this? See for yourself by using the following query in the search box at Google: allinurl:www.lawfirm411.com where the result page shows ONE page in the results. If you try that query on your own site (replace your own domain name for lawfirm411.com), you'll see the results lists ALL your pages.

The site home page was crawled by Google four months ago, when they took their "Cached Snapshot" of the page. You can see this by visiting the Google cached page here: http://66.102.7.104/search?sourceid=navclient&;ie=UTF-8&q=cache:www.lawfirm411.com where the date of this snapshot is "Apr 20, 2004 07:42:19 GMT" and they haven't been back since. The page in that snapshot has none of the newly added links, an outdated title tag, and old content.

This problem is not unique to this site. One client we worked with two years ago had a dynamically generated, framed site! Those two site structures have always given search engines trouble. Their site was not crawled at all and only the front page showed up. Our solution was to create a second domain (owned by the client), which had static HTML pages that precisely mirrored the content of the client's framed, dynamically generated site. Guess what happened after Googlebot crawled the static site? Google indexed the framed site in full and then banned the static site from the index! Not an approach we advocate, but the one that worked for this client.

We're still searching for ways to get Googlebot back to LawFirm411.com before creating that new static site, but decided to share this odd experience with the SEO community before going to any extremes. Google provides over 70% of most search engine referred traffic to ALL of our clients and we realized we can't site idly by and see a major client languish because Googlebot didn't like what it found at the client site on the first visit four months ago. This issue dogs newer sites in other places as well. The Open Directory Project has also become notoriously slow in adding new sites to the directory and in this case, has not picked up this site even after 6 regular monthly submissions. The web playing field may have begun tilting toward older, established sites and away from new ones.

------------------------------------------------------------

Mike Banks Valentine is the SEO for http://www.lawfirm411.com

Contact him at http://www.seoptimism.com/SEO_ Contact.htm

Improve Your Small Business Online at our Ecommerce Tutorial

http://website101.com/Fr ee-Tutorials/index.html

------------------------------------------------------------

Yahoo Listing Still Worth It?

In October 2002, the Yahoo! portal changed the way it delivers search results. In the... Read More

Why You Should NOT Submit Your Site On Search Engines

Before to answer to this question we have to know what is the difference between... Read More

What are My Chances to Get the First Place in Search Engine Listings?

You must have heard the stories how people became rich and famous with their websites.... Read More

7 Free Search Engine Resources You Should be Using Now

Ask any business person who's website is at the top of the search engines if... Read More

Tips On Optomizing Your Website For Search Engines

I wish to give a few little tips, about optimizing. You may find it quite... Read More

Link Building in Light of Vision-based Page Segmentation

The days of basing a successful link building strategy on link quantity and anchor text... Read More

SEM - Research Measures Success

SEM - Research Measures SuccessSearch engine marketing success comes from good research. By applying... Read More

Your Website Title Could Be Costing You Money

Nothing could be simpler than the title you give to your web pages right? Unfortunately,... Read More

Arrogant Overture Placing Greed Ahead Of Their Customers Needs

According to the dictionary, the definition of the word "overture" is as follows:"An opening or... Read More

Search Engine Spam

Running an online business relies to a greater or lesser extent on search engine traffic.... Read More

The Simple Formula To Search Engines

Search engines are one of the best tools to bring targeted traffic to your business.... Read More

How To Syndicate Your Content

Note: In order to preserve the proper format for the HTML code snippet in this... Read More

How to Increase Alexa Ranking of Your Website

Alexa toolbar also useful to Browse expired websites database. Many times when we... Read More

Surviving Googles Aging Delay

Google has always been the search industry's innovator and that's just what Google's aging delay... Read More

SEO Expert Guide - Keyword Analysis (part 3/10)

If you imagine that building an optimized site is like cooking a meal, then keywords... Read More

Is a High Search Engine Ranking Important for YOU?

In the process of designing a website for clients the question inevitibly arises "How do... Read More

Search Engine Optimization Tips For 2005 - Part One

Anybody who has their own website or is involved in business online understands the importance... Read More

Link Building: To Link, or Not to Link, That is the Question

Lately, there have been a lot of heated discussions regarding link building. Is it ethical... Read More

Optimze Your Web Site on a Shoe String Budget

Let me start off by saying I'm not a marketing guru or work for... Read More

10 Steps To Higher Search Engine Positioning

There is perhaps no more level playing field in business than the Internet. It is... Read More

Search Engine Traffic: Winning With Content

Targeted traffic is the lifeblood of any online business. The best source of free-targeted traffic... Read More

The 7 Points of Do-It-Yourself SEO

Ever felt intimidated at the convoluted, jargon-ridden information about Internet marketing for small businesses... Read More

The Secret Benefit Of Search Engine Optimisation: Increased Usability

A higher search ranking is what many website owners dream of. What they don't realise... Read More

Google Patent Application - User Data As Part of Ranking Process

In this third article, we continue to dig into the patent application of Google regarding... Read More

Supply & Demand - Stop Attacking Good SEO Companies!

As I read the latest news online about what Google has done to many webmasters... Read More

SEO 101

SEO (search engine optimization) is a booming businees these days. Everyone wants to obtain top... Read More

See No Google, Hear No Google, Speak No Google

That's right - I dreamt of a World Wide Web without the Googlopoly. And let... Read More

How to Get the Best Deal on Your SEO Project

If you own or manage a business Website, chances are you are at least somewhat... Read More

How My Page Rank Went From 0 to 5 In One Update - How Yours Can Too

Increase Your Google Page Rank!Page Rank. We all know what is. We all want to... Read More

Googles New SEO Rules

Google has recently made some pretty significant changes in its ranking algorithm. The latest update,... Read More

Even More Tips For Getting A Top 10 Search Engine Ranking!

I have spent some time discussing the 5 different options you have for getting your... Read More

Branding Versus SEO

Branding versus search engine optimization is a marketing dilemma that larger companies will need... Read More

Creating a Google SiteMap For Your Work At Home Business Web Page

Search engine traffic is the best traffic You can get for your online Business. So... Read More

How To Get Listed In Google For Free

Google does not accept payment for inclusion in their main index ? so anyone that's... Read More

Opinion ? Search Engine Success

This article is actually the summary to a book soon to be released by the... Read More

Spamglish; A Search Engine Comedy With A Language All Its Own

When the movie Spanglish hit the screens in 2004, it was dubbed "A comedy with... Read More

How To Make Your Website More Successful? (Part II)

In part I of our series of how to make your website more successful we... Read More

Searching The Internet Without Search Engines

The World Wide Web contains more than ten million websites with thousands more being added... Read More

Professional SEO: Hand Off to Bob or Outsource the Job

We are often asked if professional SEO (search engine optimization) can be done effectively utilizing... Read More

Search Engine Strategies for Affiliate Websites

The major search engines are always on the lookout to improve their search results and... Read More

How to Boost Your Traffic and Profits with Content!

Are you aware of how vitally important and valuable CONTENT is to your online business?... Read More

Are You Getting Nuked By Google Lately?

Since the last Google update, there have been many instances and examples of the Google... Read More

Reciprocal Link Exchange Management Software

When shopping online for Link Management Software that can help create and automate your... Read More

Put the Full Power of Google to Work with 11 Google Power Search Tips

Google has many ways to help you find want you want easily. Here are 11... Read More

How to Create Killer Keywords to Get FREE Traffic to Your Website

When exploring good keyword choices there are several steps one must consider. The most important... Read More

Search Engine Traffic Domination - How to Turn the Search Engines into Your 24/7 Salesmen

Getting your website returned by the search engines as a high result is a significant... Read More

King + Queen = Royal Success

Ever wonder how or why your competitor gets better search engine rankings than you do?... Read More

The SEO Game - Do You Play It?

Most people do not think of SEO as a game. Let me take 5 minutes... Read More

Keyword Popularity Check

Before using keywords for your site, it is always better to check for the popularity... Read More

Is Page Not Found Making Google Tell The World Site Not Found?

Search Engines are hard to tame, that's for sure. But if you can get on... Read More