Music on Rails

I mentioned this in an earlier post about how the site’s traffic from google has wild fluctuations without any significant change to the site.

Overall the traffic to the site has been growing as a result of additional content, but I have periods that the traffic dries to practically zero only to rebound again. Its not that the pages are not indexed - they are, but for some reason during the “dry spells” they are buried deep in the search results. Take a look at this report from google analytics:

As you can see every so often the traffic drops significantly - usually for a few days, but in December this lasted for about two weeks. In these periods the site “practically” vanishes from search results (though as I mentioned the pages are all indexed and can be found if searching for them specifically).

No particular keyword contributes more than 1% of total traffic to the site, but I do track a couple of keywords which warn me of this phenomena : “song list” is usually ranked in the #2 or #3 spot , but during the Google “mood swing” it is not in the 1st 1000 results.

If you visit Google’s webmaster central forums. Many complain about a drop in their ranking for a particular keyword from top page to “buried”. The response from the participants (mostly conjectures, since most of Google’s methods are not public) is that something is “wrong” with the site: bad linking, non valid markup, bad practices etc….

This does not seem to apply to song-list.net, since (a) I do not change anything substantial on the site apart from adding content and (b) If I have done something “bad” in Google’s eyes then the ranking/traffic would not rebound back.

Even if Google continuously tweaks the search algorithm continuously why would it cause such enormous fluctuations. I can understand that the ranking for a particular keyword could fluctuate considerably , but not all keyword would fluctuate.

Anyone care to postulate what is happening ?

A few weeks ago I installed the “Share This” Plugin by Alex King on this web site. I thought it maybe nice to have this work on the main site for the various song lists. What I really liked was that it was a javascript pop-up that also worked when javascript was off via a dedicated page.

I’ve done a fairly dirty conversion to rails. The original used the Wordpress internal code to call the mailer so I’ve added a controller to call ActionMailer, the same controller is responsible for the non-javascript page. As usual most of the time went into layout and CSS issues

Maybe sometime I’ll clean it up and re-release it as a rails generator, but we’ll see if I get to it - I am drowning with lots of things to do.

Finally I’ve upgraded to Rails 1.2.2 with a new database (now around 3100 artists) and added DVD and Book boxes to the song list page.

Almost immediately after my last post, Google traffic plummeted by about 80% - was anyone reading the blog ??? , so the majority of traffic is now coming from Yahoo even though it has indexed much less of the site. The Google bot still visits much more aggressively than Yahoo slurper. I am very curious as to the cause of the drop in traffic - most of the hits before and after are from pages in the supplemental index and I cannot imagine that the distribution of queries has changed. The different behavior is just about 4 weeks old - I am just working on other things, waiting to see what happens.

My traffic currently is split about evenly between Google and Yahoo (both bringing about 40%). The way they bring traffic is totally different. The Yahoo traffic is directed whereas Google’s traffic is much more random. Let me elaborate.

I track both the artist pages and the total site pages (they are mostly individual track pages) that are indexed by both engines. Here is today’s report:

• Google: 2,130 artist pages out of 89,900 total
• Yahoo: 457 artist pages out of 2,852 total

As you can see Google has indexed 30 times more pages than Yahoo ! It is easy to see that also from the logs - Google’s Bot is extremely aggressive - it spiders the site all day - last week it hit 20,000 pages in one day. Yahoo Slurps at a more casual rate - a few hundred to a thousand pages - dependent on its mood .

One would think that Google would dominate the traffic, but this is not the case. Here are a few keyword hits from both engines:

Yahoo:

list of 50 cent songs
celine dion list of songs
list of ray charles song
list of eric clapton songs
list of jay z songs
lil jon songs

Google:

www.song play.com
theclarksisters
HECTOR LAVOE.COM
morcheeba songlist
“soft by boney james”
le notti di cabiria rota listen
“song of delilah”
THEHAPPYGOODMAN
Songlist Farewell Angelina
cunla song wikipedia
lil boosie distant lover

The Yahoo traffic comes primarily from searchers that are looking for song lists and the site ranks well for these searches as I would hope. Google’s traffic is coming from particular word combinations that match certain pages very well even if they are hidden in the supplemental index. I hope that the main artist pages will get indexed in the main index and ranked reasonably so I will get the same kind of hits from Google that I am getting from Yahoo. I am sure many people are searching for ” [artist name] song list”, but the site does not pop up in the first few pages.

It’s official - Song List is part of web 2.0. I’ve been slashdotted here. The torrent was hard to withstand… a total of 10 hits! Nonetheless - a milestone.

I have interest in seeing how the site is being used. The logs and google analytics give a very nice snapshot of that. What I don’t get is visibility to the clicks to external sites. With Google Analytics you can instrument click-outs but you need to add Onclick javascript code to each link.
So.. I added a level of indirection. Now when the users click on an online music store icon - it comes back to the Song List controller. For example:

http://www.song-list.net/mp3/itunes/949650/949525

The MP3 action then redirects to the appropriate music library (in this case itunes). I keep all the neccesary information in the url so the controller doesn’t need to do a database call. I’ve seen some shopping sites do the same only the real link is transmitted via a key - requiring a DB call, and taking much more time.
Now I can see in the logs which music libraries get clicked more often (answer: so far it is Rhapsody).

I just released a new version of the database. There are now over 2,000 artists listed. This is of course just a drop in the sea of available music.

I’ve started a new job, so I have a limited amount of time to enhance the site and collect more data. At the moment I am trying to get the current content indexed a bit better by google and yahoo. This has been challenging - more in a future post.

I managed to avoid understanding anything related to UTF-8 until yesterday. I had a few hacks that allowed me to normalize song titles coming from Amazon so I can match them to the titles in other online libraries. For example:
“Danza de los Ñáñigos” would be normalized to “Danza de los Nanigos”.

My old hacks did not cover many of the Latin American characters with diacritics and limited my ability to collect Spanish speaking artists (Ferret crashes when faced with the UTF8 Multibyte characters). Unfortunately I couldn’t find a library that allowed me to do what I wanted (though this perl reference was a good explanation).

I ended up with an improved hack where I manually built a translation table for the extended latin characters, use STRING::unpack(’U*’) to translate U+00CO thru U+017E to the corresponding ASCII equivalent characters. Amazon Webservices do insert some multi-byte control characters in some track titles which I now just discard.

It seems that Rails 1.2 may have some UTF-8 libraries included - looking forward to that. Here are some other links that I found useful if you are trying to get a handle on UTF-8.

• Wikipedia Article that demystified the representation and what each bit is. The variety of representations in different articles (Octal, Hex or decimal) is very confusing.
• Great dynamic Unicode/UTF Table
• This article that has code and a discussion helped understand some of Ruby’s problems with Unicode/multi-byte representations.
• Unicode Hacks - Rails support for Unicode - seems this is what will be included in 1.2. Read the documentation.

I spent the weekend updating the database and cleaning up some visual issues. Some lessons learned:

• When redeploying - I must kill old fastcgi processes.
• I use caching for the top two levels of the website - the results are the same for every page and don’t change over time. The third level data is small and fast enough to let be. However for every cosmetic change I make, the cached pages need to be regenerated. I have placed cache pages in a separate directory for easier management, but don’t want the first user on each cached page to wait. One option is to generate them all on my staging site and then symlink. That doesn’t work because the internal links that rails generated are absolute not relative. I ended up writing a script that make requests for all the pages
• Not many have visited so far - the search engines have not indexed it yet. I am getting some traffic from The Real Book Listening Guide and got some in the last couple of days from Happycodr a website for Rails applications.
• I’ve added some more data to the site, the tables are getting quite big - I need to do some optimizing of database access.