Ambitious Query Indexer can now:

• Identify associations between ActiveRecord models, execute the queries used and show what indexes are required.
• Automatically generate the ActiveRecord::Migration syntax, so it’s now really easy to translate the plugin’s output into an actual improvement in the application’s performance.
• Move accurately parse the Ruby code, including no longer getting foxed by commented-out lines.
• Identify optimisation candidates within the Model layer itself.

Of course, I also took the opportunity to tidy up some of the code and leave it in a better state than when I opened it up. As ever, I’m gagging for feedback – especially as this plugin is quite difficult to test. It’s working on every Rails app I can get my hands on at the moment – let me know if you have any issues and I’ll do my best to address them.

Bookmark:

MySQL

As we all know, installing the MySQL gem on Snow Leopard required specifying the architecture under which to build – because the gem cannot work universally. Because the build is quite specific to the OS you’re working on, you have two choices when running with bundler.

Option A: bundle the gem and specify the architecture on the CLI

I’m storing (and versioning) the .gem binary files, and not files created when you run ‘gem bundle’. If you’ve got the MySQL gem cached as well, you’ll need to run ‘gem bundle’ passing a config file, something like this:

--build-options build_options.yml

Where the build_options.yml looks something like:

mysql:
  mysql-config: /usr/local/lib/mysql

And you’ll have to specify the architecture, bringing the whole command to:

$ env ARCHFLAGS="-arch x86_64" gem bundle --build-options build_options.yml

Instead, I chose to go with:

Option B: Use ruby’s MySQL gem

This might be your rvm ruby, or your system. Either way, if you want your app to be able to access the MySQL gem, but not to bundle it, add the following to your Gemfile:

gem 'mysql', :bundle => false

And run ‘gem bundle’ again to regenerate the environment file. It seems that, depending on your system, bundler will sometimes fall back to ruby’s own gems automatically, and sometimes it won’t. For me, I was getting ‘no such file to load – mysql’ until I added the config to the Gemfile.

Btw, adding ‘disable_system_gems’ to your Gemfile when you’re calling a system gem will not work. Perhaps this is obvious – I had assumed (perhaps mistakenly) that the behaviour would only disable the graceful ‘fallback’ to the system; in fact it seems to block even explicit calls to the system.

Changing the vendor directory with Rails

The current default for the bundler gem cache is vendor/gems – this is the same as Rails 2.x and can mean config.gem tries to deal with them. Do yourself a favour and add the following to your Gemfile:

bundle_path "vendor/bundler_gems"

Or similar… this helped me with a few weird Rails issues.

Helpful stuff on bundler:

• Main docs
• Gem Bundler is the Future
• A rails template for gem bundler

The peeps in #carlhuda on irc.freenode.net were also very helpful when I was struggling with this.

Bookmark:

RVM, Ruby Version Manager, allows you to run multiple ruby versions on the same machine. This is good if you want to test upgrades etc. For me, a pleasant side effect was being forced away from using the system version of ruby. Relying on the system version (and system gems… more on that in a bit!) seemed a little lazy, and was always a bit of a pain when unraveling dependencies and moving stuff live.

Another complementary tool, which I’ll hopefully blog about separately, is bundler. Do yourself a big favour and don’t try to install both at once; it can get a mind-boggling. Get to the point where you’re happy with one of the two, and then start on the other.

The RVM site is full of useful information, although it’s not always super easy to find what you need. The stuff on installation is vital (for the record, the gem version worked for me – in fact, rvm is the only gem I have installed on the system ruby).

Once installed, I installed ruby 1.8.7 with:

$ rvm install 1.8.7

The system was already running 1.8.7, so I figured this was probably the best place to start. The plan is run within this version by default, and to use other versions as appropriate.

Once installed, activate with:

$ rvm use 1.8.7

This will switch the ruby version for the current terminal. Only. This wasn’t immediately clear to me; if you want to switch for all terminals, add the default flag:

$ rvm use 1.8.7 --default

Fire up a new terminal, and it should now also be running the same version of ruby. Check with ‘ruby -v’:

$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0]


To be sure that you’re using the version of ruby you expect, there are a few things you can check:

$ which ruby
/Users/samphillips/.rvm/ruby-1.8.7-p248/bin/ruby


If you get ‘/usr/bin/ruby’ instead, you’re still using the system version.

RubyGems is installed separately for each ruby instance. To check that the ‘gem’ command is hitting the right place, again, break out ‘which’:

$ which gem
/Users/samphillips/.rvm/ruby-1.8.7-p248/bin/gem


Another good check is ‘gem env’:

$ gem env
RubyGems Environment:
- RUBYGEMS VERSION: 1.3.5
- RUBY VERSION: 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0]
- INSTALLATION DIRECTORY: /Users/samphillips/.rvm/gems/ruby-1.8.7-p248
- RUBY EXECUTABLE: /Users/samphillips/.rvm/ruby-1.8.7-p248/bin/ruby
- EXECUTABLE DIRECTORY: /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86-darwin-10
- GEM PATHS:
- /Users/samphillips/.rvm/gems/ruby-1.8.7-p248
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- :sources => ["http://gems.rubyforge.org/", "http://gemcutter.org/"]
- REMOTE SOURCES:
- http://gems.rubyforge.org/
- http://gemcutter.org/


Again, if you’re seeing references to directories like these:

- INSTALLATION DIRECTORY: /Library/Ruby/Gems/1.8
- RUBY EXECUTABLE: /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
- EXECUTABLE DIRECTORY: /usr/bin


Then your switch hasn’t worked.

Some reasons I’ve found why switching hasn’t worked:

• It’s apparently possible to drop the ‘use’ keyword when choosing ruby versions. I’ve learned not to do this; firstly it doesn’t give you a nice confirmation that it’s worked, and secondly it didn’t seem to work correctly (perhaps the two are related…)
• Similarly, although you’ve assigned the ‘default’ version of ruby, running ‘rvm use default’ doesn’t seem to work properly all the time.

If keeping to the full ‘rvm use xxx –default’ command doesn’t help you, it might be worth re-installing. This is easy; switch back to the system version of ruby (’rvm use system’), delete the .rvm directory in your home directory and uninstall the gem. It seems that my mistake with my original install was a classic one:

Too much sudo

We’ve got very used to installing gems, and doing anything to do with them, with sudo. With rvm, you don’t need to do this and it will lead to confusing results, as the docs point out at length. Every time you use sudo, you’re going to be messing with the system ruby version (as this is in root’s path), and weird results will follow.

To summarise: use sudo to install the gem on system ruby, and then leave the sudo well alone.

Passenger

If you’re using with passenger + apache, first install the passenger gem against your rvm ruby version:

$ rvm use 1.8.7
$ gem install passenger
[snip]
$ passenger-install-apache2-module


Note the lack of sudo. The module installer will suggest the apache config syntax. This needs changing slightly, as per the docs –

$ cat /etc/apache2/other/passenger.conf
LoadModule passenger_module /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/gems/passenger-2.2.8/ext/apache2/mod_passenger.so
PassengerRoot /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/gems/passenger-2.2.8
PassengerRuby /Users/samphillips/.rvm/bin/passenger_ruby


(restart apache…)

The last line will point apache at your rvm ruby version. When using passenger with rvm, switch the rvm version with the following syntax:

$ rvm use 1.9.1 --default --passenger

I was confused for a number of minutes when gems installed on my rvm version of ruby weren’t available to apache; not doing this was why.

RVM works nicely already, and has potential to do lots of good. If you’ve got any problems, hop on to #rvm on irc.freenode.net, where friendly folks will help you out.

Bookmark:

Twitter is the stage on which this story was played out. What is Twitter? The best explanation I’ve heard so far is ‘one-to-many text messaging’. You send a text – 140 characters – and instead of it being sent to just one friend, it’s sent to all of your ‘followers’. They receive the text on their phones, through the Twitter website, or through Twitter software installed on their computers.

The story is not set in stone, and it happens in real time. In this way, it’s almost more of a reality show than a fictional work with a predetermined path. The characters – Persephone, James, Leonie, Darren, Regina, Paul, Melody and Lee – live in our reality. When we put together the main site, we even gave it a feel of a Big Brother microsite. This is a reality show like no other.

Persephone and Leonie went to the christmas lights being turned on in Manchester city centre, for example. It was quite the live drama! Paul and James went to a football match at Old Trafford. Leonie’s in PR, so if there’s a party or networking event going on, she’s there.

The characters also respond to other people’s tweets – meaning that, just by tweeting them and getting a reply, anybody can become a character in the story. It’s been fascinating to watch people get involved with the story.

But what is most striking about these characters is that they are very almost real people, very believably existing on Twitter – in a way that, in reality, most people don’t. Although Facebook is a mass phenomenon, and MySpace/Bebo remain a modernised version of Geocities, Twitter isn’t quite there yet. It’s the domain of geeks, marketeers and companies doing innovative customer engagement. The November In Manchester characters aren’t any of these types – they’re normal mancunians trying to get from the start to the end of a day unscathed. You can believe that they are real, and reading their tweets you can believe that other, real, people could use Twitter in this way as well.

November In Manchester is now my first example when I explain to a client, colleague, friend or grandparent what Twitter is, what it does, and what it’s for. What is Twitter for? It’s for telling your story to the rest of us.

Interested in how the November In Manchester site works? Check out November In Manchester: Joining those technical dots.

Bookmark:

For my own notes, I thought I’d jot down a few details of the implementation for future reference, or for anyone who was interested. November In Manchester is built in Rails using HAML and SASS.

Scheduling

This was a nice opportunity to use the whenever gem. If you’ve not used this before, check it out – it’s a cracking gem and its example Capistrano recipe works a treat. Although I’m happy enough with cron syntax I was pleasantly surprised to have it so smoothly taken out of my hands, and means that our various scheduled tasks are managed straight from a config file in our application.

Twitter

The Twitter implementation was the most difficult part. The initial version used dancroak’s twitter search gem and worked a treat with the code. I don’t particularly like the query syntax, but this is down to twitter, not the gem. Also down to twitter is the job of updating the search results. Unlike the main site, the search site (and API) is not always real-time and can lag heavily behind. It’s no good being 8 hours on a story that’s supposed to be happening in real time.

We therefore switched the search gem out and ended up using the main API and gem. Initially we were perplexed about how to best get the information about various users’ tweets. The API is designed for a piece of software acting as a twitter user, and so can tweet, follow, etc – but it can’t search. In the end we settled on using the ‘user_timeline’ method and passing the username (screen_name) of the user we wanted to get the tweets for. We run this search every five minutes, and as long as we’re not more than 20 tweets behind (the method’s limit – although you may be able to override), we’re golden. It’s not ideal (better suggestions received) but it got us back to real-time.

Thankfully our architecture allowed this switch to be made in 10 lines or so of code. Good job – I had a matter of minutes before BarCamp Manchester 2 started to switch it over!

RSS/Atom

Ruby has built-in support for RSS, and we initially used this for the integration. However, it doesn’t support Atom and I didn’t want to be caught by surprise if a new blog turned up that used this format (I didn’t know what new characters and data sources Tom would introduce as the plot twisted and turned!), so we switched over to the excellent Simple RSS gem. This gem also supports reading from the content of the posts – which at one point we were using – which the built-in Ruby stuff doesn’t.

Flickr

Photos are a big part of the project, and we wanted to have the most recent photos displaying on the site, pulled from the flickr feed.

We used the flickr-fu (not ‘flickr_fu’, confusingly) gem for our integration here, and took inspiration from this blog post. I don’t think that helpers are the right approach here, though, so we instead import our flickr photos every 3 hours through a scheduled process and stick them in a database – hotlinking the actual images straight from flickr.

Deployment

We use Capistrano for deployment and git for version control.

Overall the tech stuff on the site was just the work of a weekend or two and a few evenings. It was nice to have a small project that was live quickly – something I’ll definitely try to bear in mind for the future. Hope you’re enjoying the story!

Bookmark:

A few well-known blog posts have dealt with this subject, giving excellent advice on how to index your tables for best performance. Jamis dealt with the subject a while ago, and recently Matt Jankowski has posted a fairly exhaustive guide. I felt that this excellent guides could be built upon, and the analysis that I would do, given a query to index, could be automated.

Enter Ambitious Query Indexer. Unlike other plugins in this space, AQI (I’m starting to shorten it to ‘aqute’; not sure if this will stick) doesn’t optimise queries on the fly by reducing eager loading, and it doesn’t look at tables in an attempt to guess what indexes might be needed to query it.

AQI’s approach is to analyse your Rails app, work out what queries it will run during its operation and spot any indexes that could be added for the MySQL engine to use. Why is this a good approach? Simply, because it focuses on the queries you run, not the tables they run upon. Imagine the following story:

1. You have an Article model containing a user_id field in an articles table
2. You have a User model that has_many articles
3. Calling User.first.articles will run a query like “SELECT * FROM articles WHERE user_id = ‘1′”.

How do you speed this query up? You would add an index to the articles table on the user_id field. Happy days.

Now imagine you install acts_as_paranoid on the Article model or something similar – a plugin that overrides the default finders – suddenly, your query will be something like:

SELECT * FROM articles WHERE user_id = '1' AND (deleted_at IS NULL or deleted_at < '2009-10-05 22:07:00')

Whoops - your index doesn't work any more because you've added a field to the query. Your database is about to go to the dogs, and only a cursory look at the table wouldn't reveal why because it's not immediately obvious that you now need a compound index on user_id and deleted_at.

Unfortunately, when it comes to indexing, looking at a table to work out what indexes you need is not enough. Adding indexes during your migrations won't always cover you, and best practice indexing won't fly on most real-world applications. You have to look at the queries that interrogate that table. AQI will do this; it will find, run and analyse your queries and give you a list of indexes that will improve performance. Yay!

To install, follow the easy instructions on GitHub - Ambitious Query Indexer. This plugin is still under active development and I'm adding new features all the time. Please let me know any feedback you have, or any issues you encounter.

Bookmark:

But £10 is too much, so for now I’m going to be listening to the following terrible adverts:

The Right Livelihood Award

I’ve never heard of this award, and I’m sure it’s all for a very good cause. But the advert is some shonky stuff. It starts with a vocal stumble when trying to say ‘Right Livelihood Award’ (never a good start to fluff the brand!), and then goes to tell me absolutely nothing about what the award is or does. But apparently there’s one thing I should remember – that we can all do something if we try hard enough, that wishing and dreaming make crazy things happen, and that you should never eat yellow snow. Or something like that – the point is, this ‘one’ thing I should remember is divided into three meaningless clichés, and there’s no chance I’m remembering them.

The whole thing sounds like it was recorded outside on a busy street. And still I’m none the wiser about what this doubtless very worthy award is for!

ShareMyPlaylist.com/ShareMyPlaylists.com

Nice idea; social playlists for spotify. Presumably a feature that spotify will make themselves eventually, but anyway, what I love about this advert is the girl talking gets the domain name wrong. It’s www.sharemyplaylists.com (playlists plural), but she twice calls it www.sharemyplaylist.com (playlist singular). The latter domain is a most certainly not a production ready website – in that it’s a file index with a couple of favicons and a robots.txt file. Fail!

Crucial Memory

Your computer has slowed down – do you need more memory, or just to be patronised by the ‘computer doctor’? Apparently, with crucial, both are possible. Joy.

This one seems to have buggered off for now, but it sure was odious when it was about – and for one 48 hour heavy work weekend it seemed to be the only advert available.

MTV safe sex ultra-liberal moralising

‘Everybody loves music… especially in the bedroom’. What?! I prefer music in my study. Or at my piano. Quite the impressive segue though. This MTV ad, featuring some beautiful Irish bird, suggests that I should refrain from unsafe sex and enjoy ’sexy playlists’ in the bedroom instead. Not entirely sure that this is the best way to reach ‘young people’.

That said, I still chuckle to myself when this ad comes on; just once I wish she’d say ‘Everybody loves sex… especially in the music room‘. Then I’d definitely never pay for the premium subscription.

Rhino Vinyl

‘Spotify is ace; streaming, high quality, distributed. At Rhino, we love it almost as much as vinyl records’. Say again?You like mp3s, but really you like vinyl and think mp3 is shit? Again, quite the segue and as the advert rolls on, it does seem to make sense. But I love the transition; it’s akin to local news shifting between a murder case and a human interest piece about puppies and their hilarious habits.

Other favourites include the advert for Scarlett Johansson’s single getting cut off half way through every single time, and Amy McDonald, who talks for what seems like hours about how, somewhere, live tracks and her whole album is available to listen. It might be on Spotify – it probably is – but she seems to neglect to mention it!

And just as I write, Adam from Spotify has used the Spotify recording microphone, bought for £7.99 at Argos, to record an ad telling me how good their ad targetting is. Touché!

Bookmark:

It’s fair to say that Twitter’s influence is, in some spheres, pretty large. TechCrunch gets more traffic from Twitter than from Digg. That’s pretty impressive, considering the amount of time that people still spend gaming Digg. But its influence is still pretty limited. As a rule, it’s still fairly restricted to the technology industry. It’s not that easy to use, and as replies, retweets and hashtags take over, it’s becoming more and more like a raw message format than something that’s human readable. URL shortening services are hardly accessible to the lay person.

I think we’ve all tried to explain what twitter is to a friend, even a tech-savvy one. Perhaps we’ve got as far as getting them signed up and an iPhone client downloaded. Three months later, they haven’t updated it. They don’t understand what it’s for, and they don’t have any followers. People who abandoned MySpace and Bebo for Facebook  in a heartbeat are not migrating to Twitter in the same way.

Twitter is still a niche product. So, if it does become the ‘pulse of the planet‘, is its current utility as a community management/traffic driving/networking/brand management/corporate communications/customer service tool going to be limited? I think it might be.

I follow 119 people – and something like 20 of these tweet on any sort of regular basis. Even that is a bit much. I had to unfollow Jonathan Ross and Zeldman; I couldn’t hack it. More than 20 new tweets in an hour and I’m screwed. How am I going to cope if every tweet that mentions a company has an @reply from that company’s community management being dead helpful? I’m not. If there a billion people on twitter, I’ll be swamped and I’ll have to leave.

More to the point, how am I going to get that personalised service if said community managers are inundated because there are a billion people on twitter? Think of the companies with REALLY dreadful customer service, like O2 and BT. How are they going to keep up with the moaning? They couldn’t, unless the metric/filtering stuff gets really clever. So there goes my personal service.

So perhaps we should view the current state of Twitter as a VIP customer service and get-satisfaction-esque issue resolution tool available to geeks who can understand it. Will every moan and question get dealt with if the user base multiplies? No, unless the very basis of the whole idea changes. There’s a reason that all of these companies have taken their phone numbers of their sites and instead installed automated customer service systems/mantraps – individual interaction with the whole customer base just isn’t possible.

Will Dell be able to post and get a conversion on real-time offers of limited product lines/refurb products, if the user base was massively bigger? Not under the current model – the offers would be snapped up so quickly that people would stop bothering to click on the links.

What of real time web? What of it. Whilst aggregating tweets is a good way to find out news really quickly, it’s a job for a computer, not a person. I can’t have more than one or two ‘news’ (be it world affairs, the BBC or something on formula 1) users in my follow list; anything more is information overload. I rely on power users to filter the real time web down for me, again in real time, so that it’s just enough information to digest.

So perhaps the future of Twitter under the current model is stronger groups/networks that are reliant on real-world connections, mixed with a lot less business interaction and a selection of a very few, very important, power users.

Sounds like a pretty shaky business idea to me. No doubt, I’ll be proven wrong.

Bookmark:

Previously I have spurned pen and paper because it feels very messy and can be difficult to keep track of. All of a sudden your centralised task list is split over various scraps of whatever you could lay your hands on at a particular moment, and my Backpack isn’t seeing the stuff it should be.

My new solution is unbelievably simple but still seems to have turned a few heads.

1. On any piece of paper, anywhere (spec document, chart, graph, list of tasks), draw an empty circle next to any ‘actionable’ item. What do I mean by ‘actionable’? I mean anything that you would miss if this piece of paper was lost. A lot of notes you make you will never read again; they are just to help you think. For everything that you need to do something on, draw a circle.
2. Once the item next to the circle has either been completed or graduated to another list, fill in the cirle.

Yep, that’s it. This means that any piece of paper I may have on my person may have these little circles on them. Once I’m done reading a particular document, or I’m back at my desk, I complete the tasks I can (in five minutes or less, GTD style), and graduate any items that can’t to Backpack. Once the paper has no empty circles, file it or (better) throw it.

This idea is not new by any stretch, and it’s certainly not anything clever, but it does help me keep things in order. The key to it really is the graduation. If you force yourself to keep all pieces of paper with a pending task or piece of information floating around (most likely on your desk), you will quickly go mad.

The other key is to have a ‘buck stops here’ list – that’s what Backpack is perfect for. Its email dropboxes also mean you can add stuff directly to this last-stop list easily. For me, the best thing about the dropboxes is that I can send tasks straight from my phone. This means that whilst I lay awake at night remembering all the stuff I need to do, I can send off a series of emails and turn my mind to other things like going to sleep.

Who knows, maybe Backpack is helping my insomnia!

Oh, and do yourself a favour when trying the pen and paper system – buy a nice pen, nice paper and instigate a recycling bin under your desk.

Bookmark:

1. Log in to Facebook
2. Add the Twitter Facebook App
3. Register your Twitter account with said app
4. Click ‘Allow Twitter to update your Facebook status’
5. Click yes to Vista-esque security settings to allow Twitter access to your Facebook status

Done!

Should you later want to stop Twitter updating your Facebook status:

1. In Facebook, hover over ‘Settings’ and choose ‘Twitter Settings’ from the menu
2. Click on ‘Additional Permissions’
3. Uncheck ‘Update my status without notifying me.’

NB: Apparently @reply tweets won’t turn up. Probably for the best; they wouldn’t make a whole lot of sense!

Anybody else think that with all the @, # and RT: syntax, Twitter is turning into an obfuscated language of its own? It reminds me of the weird format we used to have to send to roaming GPS boxes, back when I worked on a project that involved involved such things, and debugging problems with the message syntax got really quite difficult!

Bookmark: