Just how important is a Robots.txt file to GoogleBot?

I have always been under the impression that a site does not necessarily require a robots.txt file. In fact, alot of sits I have built do not have a robots.txt file and yet there are very well indexed by GoogleBot and other search engine robots.

However, recently Google employee JohnMu, said in a Google Groups thread that if your robots.txt file is unreachable due to timing out or other issues, not including a 404 not found status, Google "tends not to crawl the site at all just to be safe."

Read more

Google Trends update: Numeric Scale and CSV Downloads

A popular tool since its launch in 2006, Google Trends is one of several free tools available that provides helpful insight into search query popularity and by extension search volumes.  Since its launch, Google has added features like "hot trends" and the country/city where the search queries originate.

Now Google has adding a relative "numeric scale" to Google Trends and the option the download CSV files.

Here's a search for "hd dvd, blu-ray" before:

And after:

The numeric scale can be seen on the vertical (Y) axis. According to Google:

"These numbers don't actually refer to exact search volume figures; instead, Trends scales the first entered term so that its average search traffic in the chosen time period is 1.00. Then, each additional term is scaled relative to the first."

Here's more information and color from the Google Blog.

Read more

Googles new "Skip Intro" link enables users to skip those annoying Splash Pages

If your anything like me, your probably sick of sites that force you to sit through a pointless Flash splash screen. Thankfully, now Google has provided a new "Skip Intro" link to save you that annoyance.

A search for yuasa france, for example, returns a result that adds "Skip Intro" link next to the listing title. Clicking on the link will bypass the splash screen and take the searcher directly to the home page. Here is a screen capture:

Read more

How Google deals with duplicate content issues caused by content scrapers

Sven, of the Google Serach Quality Team has addressed the underlying concerns webmasters have regarding how Google Handles scraped content. The article can be seen on the Google Webmaster Blog.

In the two scenarios discussed, one (duplicate content within your domain) can be controlled. The other cannot. However, he recommends that you offer a link back for syndicated content:

In cases when you are syndicating your content but also want to make sure your site is identified as the original source, it's useful to ask your syndication partners to include a link back to your original content

If scraped content ranks higher than the original content, it is probably a technical issue on your end (a "rare case," says Sven). You should check that the content is not blocked out by robots.txt, see the sitemap file for any changes, or check if the site is in line with the Google Webmaster Guidelines.

Read more

How much is a Top Level Domain (TLD) worth in terms of SEO?

For most people, registering a domain name is the first thing to do when starting a website and at this point, we have to choose which TLD to use. As webmasters or search engine marketers we do not want to make a mistake at the very start of a web campaign, so we should consider the following:

1. Does Google pay any attention to a top level domain when ranking websites?

If you are talking about regionally specific TLDs (such as .au domain names), the answer is definately yes of course. A country code top level domain is the most effective tool for geo targeting. But what about non-country-specific domains? How do they fit in with the Google Algorithm?

Historically, a generic top level domain shows which type of organization a website belongs to:

• .com = ‘commercial’ (an open TLD)
• .edu = ‘educational’ (limited)
• .gov = ‘governmental’ (limited)
• .mil = ‘military’ (limited)
• .org = ‘(non-profit) organization’ (an open TLD)
• .name = ‘individuals, by name’ (an open TLD)
• .net = ‘network’ (an open TLD)
• .biz = ‘business’ (an open TLD), etc

So ideally a TLD should mean something and it does. For example, an incoming link from a .gov or .edu link is worth more than a link from a .com site because Google places more trust in .edu and .gov sites.

• Google does seem to perceive a top level domain as a separate word: for example, searching for [fishing net] we get more .net domains on the first page than when searching for [fishing netting].
• Google seems to be biased against .info domains and the reason for that is a very low price for .info domains and hence its abuse by spammers.

Read more