Music on Rails

I mentioned this in an earlier post about how the site’s traffic from google has wild fluctuations without any significant change to the site.

Overall the traffic to the site has been growing as a result of additional content, but I have periods that the traffic dries to practically zero only to rebound again. Its not that the pages are not indexed - they are, but for some reason during the “dry spells” they are buried deep in the search results. Take a look at this report from google analytics:

As you can see every so often the traffic drops significantly - usually for a few days, but in December this lasted for about two weeks. In these periods the site “practically” vanishes from search results (though as I mentioned the pages are all indexed and can be found if searching for them specifically).

No particular keyword contributes more than 1% of total traffic to the site, but I do track a couple of keywords which warn me of this phenomena : “song list” is usually ranked in the #2 or #3 spot , but during the Google “mood swing” it is not in the 1st 1000 results.

If you visit Google’s webmaster central forums. Many complain about a drop in their ranking for a particular keyword from top page to “buried”. The response from the participants (mostly conjectures, since most of Google’s methods are not public) is that something is “wrong” with the site: bad linking, non valid markup, bad practices etc….

This does not seem to apply to song-list.net, since (a) I do not change anything substantial on the site apart from adding content and (b) If I have done something “bad” in Google’s eyes then the ranking/traffic would not rebound back.

Even if Google continuously tweaks the search algorithm continuously why would it cause such enormous fluctuations. I can understand that the ranking for a particular keyword could fluctuate considerably , but not all keyword would fluctuate.

Anyone care to postulate what is happening ?

Almost immediately after my last post, Google traffic plummeted by about 80% - was anyone reading the blog ??? ;-), so the majority of traffic is now coming from Yahoo even though it has indexed much less of the site. The Google bot still visits much more aggressively than Yahoo slurper. I am very curious as to the cause of the drop in traffic - most of the hits before and after are from pages in the supplemental index and I cannot imagine that the distribution of queries has changed. The different behavior is just about 4 weeks old - I am just working on other things, waiting to see what happens.

My traffic currently is split about evenly between Google and Yahoo (both bringing about 40%). The way they bring traffic is totally different. The Yahoo traffic is directed whereas Google’s traffic is much more random. Let me elaborate.

I track both the artist pages and the total site pages (they are mostly individual track pages) that are indexed by both engines. Here is today’s report:

• Google: 2,130 artist pages out of 89,900 total
• Yahoo: 457 artist pages out of 2,852 total

As you can see Google has indexed 30 times more pages than Yahoo ! It is easy to see that also from the logs - Google’s Bot is extremely aggressive - it spiders the site all day - last week it hit 20,000 pages in one day. Yahoo Slurps at a more casual rate - a few hundred to a thousand pages - dependent on its mood ;-).

One would think that Google would dominate the traffic, but this is not the case. Here are a few keyword hits from both engines:

Yahoo:

list of 50 cent songs
celine dion list of songs
list of ray charles song
list of eric clapton songs
list of jay z songs
lil jon songs

Google:

www.song play.com
theclarksisters
HECTOR LAVOE.COM
morcheeba songlist
“soft by boney james”
le notti di cabiria rota listen
“song of delilah”
THEHAPPYGOODMAN
Songlist Farewell Angelina
cunla song wikipedia
lil boosie distant lover

The Yahoo traffic comes primarily from searchers that are looking for song lists and the site ranks well for these searches as I would hope. Google’s traffic is coming from particular word combinations that match certain pages very well even if they are hidden in the supplemental index. I hope that the main artist pages will get indexed in the main index and ranked reasonably so I will get the same kind of hits from Google that I am getting from Yahoo. I am sure many people are searching for ” [artist name] song list”, but the site does not pop up in the first few pages.