Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?

As a search engine optimization specialist I often optimize existing web pages for small business clients, upload them to the site and see pages re-indexed by Google within a week. This only happens with existing business sites that have been online for a few years. Google seems to be updating their index as often as every other week at this point and older established sites that are already indexed seem to be re- crawled on that twice a month schedule on a fairly routine basis.

Two clients that hired me for recent work saw their rankings shoot to the top for a newly targeted search phrase in a weekend when I did optimization on a Thursday and they were ranked instantly by Saturday. Now keep in mind that this doesn't happen for everyone, only those that have been online for some period and already have significant content that simply needs tweaking and proper title and metatag information added. They usually have relatively good existing PageRank and do well for other RELEVANT search phrases already. I offer that warning only to avoid instilling false hopes in anyone hoping to achieve the same instant ranking boost overnight.

Those clients that do succeed in this way are often thrilled with the results accomplished in such short order. I'd love to be able to offer that type of ranking boosts to everyone, but some are more equal than others when it comes to easy, inexpensive SEO tune-ups that rev up your rankings overnight. Your mileage may vary.

WHY DO NEW SITES SUFFER?

What is going on with newer sites that don't get crawled for months? I've got a client, a newer attorney directory that offers tons of great information in the form of articles on specific areas of law, links to incredibly valuable and relevant legal sites and over 600,000 attorneys listed by practice area and state. Yet the site has not been re-crawled by Google for over 3 months! Now this would not be such a big issue for many sites, but this site is relatively new and we've optimized all the titles, tags & page text, created a complete site map and placed links to all these resources on the front page.

I know that the site is not being crawled because Google's cached copy of the front page shows it before we did the work four months ago, without the new links and without title tags. We've submitted the site by hand, (manually) once a month for three months via the Google Add URL page. http://www.google.com/addurl.html When the hand submission failed to get it re-indexed for four months, we submitted the sitemap page, which has not been crawled at all. Google shows only ONE page on this site, when in fact it has thousands of pages, a sitemap and dozens static pages!

Part of the problem is that this site must be dynamic, since a database of over 632,000 attorneys must be accessed, retrieved and served for any of those law firms searched for to be returned to the site visitor. Google warns owners of dynamic sites that Googlebot may not crawl dynamically generated pages with "?"" question marks in the URL. This is to avoid crashing the server with too many concurrent page requests from Google's spider. http://www.google.com/webmasters/2.html#A1

The solution to this dynamic URL problem has been discussed widely in search engine forums and solutions have been bandied about including software provided by SEO's, URL re-write techniques for dynamic pages on APACHE servers http://www.alistapart.com/articles/urls/ and PHP pages http://www.stargeek.com/php-seo.php to generate search engine friendly URL's. Others recommend simply adding static HTML sitemap pages as alternatives for the search engine spiders. In this instance the client's developer simply said "I can't do that (PHP solution) on this server". So we resorted to putting up the static HTML sitemap pages with hard-coded

URLS to the main 54 pages of the site at http://lawfirm411.com/Law-Firm-411-sitemap.html This should get at least those fifty pages crawled by Googlebot, but Googles' spider appears not to be crawling this site at all. How do we know this? See for yourself by using the following query in the search box at Google: allinurl:www.lawfirm411.com where the result page shows ONE page in the results. If you try that query on your own site (replace your own domain name for lawfirm411.com), you'll see the results lists ALL your pages.

The site home page was crawled by Google four months ago, when they took their "Cached Snapshot" of the page. You can see this by visiting the Google cached page here: http://66.102.7.104/search?sourceid=navclient&;ie=UTF-8&q=cache:www.lawfirm411.com where the date of this snapshot is "Apr 20, 2004 07:42:19 GMT" and they haven't been back since. The page in that snapshot has none of the newly added links, an outdated title tag, and old content.

This problem is not unique to this site. One client we worked with two years ago had a dynamically generated, framed site! Those two site structures have always given search engines trouble. Their site was not crawled at all and only the front page showed up. Our solution was to create a second domain (owned by the client), which had static HTML pages that precisely mirrored the content of the client's framed, dynamically generated site. Guess what happened after Googlebot crawled the static site? Google indexed the framed site in full and then banned the static site from the index! Not an approach we advocate, but the one that worked for this client.

We're still searching for ways to get Googlebot back to LawFirm411.com before creating that new static site, but decided to share this odd experience with the SEO community before going to any extremes. Google provides over 70% of most search engine referred traffic to ALL of our clients and we realized we can't site idly by and see a major client languish because Googlebot didn't like what it found at the client site on the first visit four months ago. This issue dogs newer sites in other places as well. The Open Directory Project has also become notoriously slow in adding new sites to the directory and in this case, has not picked up this site even after 6 regular monthly submissions. The web playing field may have begun tilting toward older, established sites and away from new ones.

------------------------------------------------------------

Mike Banks Valentine is the SEO for http://www.lawfirm411.com

Contact him at http://www.seoptimism.com/SEO_ Contact.htm

Improve Your Small Business Online at our Ecommerce Tutorial

http://website101.com/Fr ee-Tutorials/index.html

------------------------------------------------------------

Linking for Fun and Profit

Well actually, linking isn't fun at all. In fact, it's quite tedious. Seeking, responding to... Read More

New Site and Sandbox: How to Get Rid of It

You have put lot of sweat in making your site. Now you want it to... Read More

Do Search Engines Like Your Web Site?

Between 75% and 98.8% of visitors to Web sites come from searches made at search... Read More

Googles Next Big Move

November 2003 might go down in history as the month that Google shook a lot... Read More

Most Overlooked Search Engine Optimization Technique

Some of the basics when it comes to SEO and getting high ranking in search... Read More

Link Reputation - How to Improve Search Engine Rankings

One of the most overlooked strategies to improve search engine rankings is building your link... Read More

The Benefits of Organic SEO (Search Engine Optimization)

If you're going to sell any type of product or service online, you're eventually going... Read More

How To Get Number 1 Spot In Google, Every Time, Guaranteed

Woaah! Wait a minute. Before I exclusively reveal these insider secrets can I ask you... Read More

Dealing With Search Engine Stress In A Home-Based Business

As a member of several search engine optimization forums, I have recently noticed (especially since... Read More

Get Better Search Engine Rankings with RSS

RSS is the latest craze in online publishing. But what exactly is RSS?RSS or Rich... Read More

What To Look For When Shopping For A SEO Specialist

When, shopping for a Search Engine Optimization (SEO) company/specialist you need to be aware of... Read More

Optimize your Search Engine Placement Five Easy Ways

If you are like me, you created web site metatags with the help of your... Read More

What is Google Pagerank?

PageRank is one of the factors that Google uses to evaluate your web site and... Read More

8 Essential SEO techniques

1) Title Tag - The title tag is the most powerful on-site SEO technique you... Read More

Banned from Google and Wondering Why?

There are those that get on the computer one night and find that all of... Read More

The Value of Search Engine Marketing

Search engine marketing vs traditional offline marketing. Most often, traditional offline marketing... Read More

How To Syndicate Your Content

Note: In order to preserve the proper format for the HTML code snippet in this... Read More

How to Develop a Successful ROI for Your Website

Are You Really Measuring for Return on Investment (ROI) with Search Engine Optimization?Puzzling studies and... Read More

Why Optimize Your Site For Search Engines?

Sometimes a search engine optimization company will miss that glaring question posed by potential clients... Read More

Monitor and Increase Your Search Engine Visibility with the DIY SEO Tools

In this three part article, you'll find many tools that any webmaster can use to... Read More

Why Fear? Now Change Your Filenames Without Disturbing Search Engines/Visitors And Rank Well

By now, you know that to rank well in the Search Engines, you need to... Read More

Buzzwords vs Effective SEO Keywords

Ever see a website that seems to speak a foreign language...in English? We encounter many... Read More

Does The Number Of Links On A Page Affect Ranking?

Lots of research has focused on inbound links to a site, but little has focused... Read More

10 Ways to Make Sure Your SEO Goes Out of Its Way for You

If they want to have success, companies should do everything they can to ensure that... Read More

Meta Tags Help Optimize Your Web Site Rankings

Are you working on getting your web site up onto the internet?There are a few... Read More

Youve Got The Power, Why Arent You Promoting It?

Like all things, theft is theft, misery loves company, negative people are everywhere, business is... Read More

SEO, the Simplified Version

Lets get things straight. SEO is a very competitive market. If you have the time... Read More

Search Engine Optimization - Free Tips and Help - Part One - The Title Tag

Search Engine Optimization is a widely misunderstood industry. Many webmasters, including myself, have been mislead... Read More

How Real SEO Analysis Works

If you're serious about SEO, you need to know how to analyze the information you... Read More

Site-Digest 7 Simple Steps To Place Your Site Map On Steroids

The Site Map is a too often overlooked piece of the Search engine Optimization Puzzle.... Read More

How Do I Submit My Site To Search Engines?

You've built up a good site, put in a lot of content, but none of... Read More

How to Verify and Monitor Your Search Engine Listing on Google?

Being listed in search engines and ranked high on searches is the overall goal a... Read More

How To Weather Google Algorithm Changes

If you spend much time reading the various webmaster forums, you're no doubt well aware... Read More

One Way Link Building Secures Long Term Ranking Results

One-way link building is a great way to improve your link popularity and ranking in... Read More

Google?s Siren Call ? Is It Crashing Your Search Engine Marketing?

It's difficult to dispute the rational behind the rant since Google continues to outpace its... Read More

Five Short Quick Tips on Web Page Titles

By Catherine FranzSearch engine spiders read and record page titles first. Here are five tips... Read More

How to Pick an SEO Firm

If you're looking for an SEO firm, we recommend that you weigh 10 factors that... Read More

META Tags Explained and How To Use Them For Ranking

The META tags are used to provide extra information about a web page. There was... Read More

Search Engine Keywords - What Do People Search For?

Do you ever wonder how people search for things on the Internet? What if you... Read More

Meta Tags - What Are They and Which Search Engines Use Them?

Defining Meta Tags is much easier than explaining how they are used, and by which... Read More

Tips On Optomizing Your Website For Search Engines

I wish to give a few little tips, about optimizing. You may find it quite... Read More

Keep Your Web Site Content Relevant

Visitors and search engines love content-rich web sites, but just having a lot of content... Read More

Search Engine Strategies for Affiliate Websites

The major search engines are always on the lookout to improve their search results and... Read More

Link Building for Hilltop

Hilltop is one of the major concepts underpinning Google's search algorithm, yet its workings and... Read More

Search Engines Are Going to Love You for This

What's the most potent free traffic source on the web?*** Search Engines! ***Surprised? Of course... Read More

Keywords: The First Step To Recognition

Open Wordtracker [ http://www.wordtracker.com/ ] and you'll see following proclamation by Brent Winters,... Read More

Link Popularity: Improve Your Search Engine Rankings

What is link popularity?Link Popularity is simply the total number of pages that link to... Read More

6 Ways To Attract Search Engines To Your Website More Often

Adding fresh, updated content to your website is the surest way to get search engines... Read More

Lets Make Your Website #1

Their is simple way of making your website rank top and search optimized.Step 1. Before... Read More

How To Start An Internet Business ? Meta Tags and Keyword Density

Okay, you have a domain name, layout and content. Now we get to a step... Read More