Sam's World Of No

Last night, I took the opportunity of a free evening to add a few new features to Ambitious Query Indexer. My inspiration was a spot of welcome coverage from a few blogs, and also the changes that have been made to Elad Meidar’s RailsIndexes plugin. That project interests me a great deal because it is aimed at the same problem, but our implementations are quite different. There’s nothing wrong with a little friendly competition, and AQI was lacking a few features (especially around adding indexes for rails associations) that RailsIndexes had.

Ambitious Query Indexer can now:

• Identify associations between ActiveRecord models, execute the queries used and show what indexes are required.
• Automatically generate the ActiveRecord::Migration syntax, so it’s now really easy to translate the plugin’s output into an actual improvement in the application’s performance.
• Move accurately parse the Ruby code, including no longer getting foxed by commented-out lines.
• Identify optimisation candidates within the Model layer itself.

Of course, I also took the opportunity to tidy up some of the code and leave it in a better state than when I opened it up. As ever, I’m gagging for feedback – especially as this plugin is quite difficult to test. It’s working on every Rails app I can get my hands on at the moment – let me know if you have any issues and I’ll do my best to address them.

I’ve been feeling the love for bundler over the last week or so – if you’ve not used it before, it’s a new way to vendor gems within a ruby project that a distinct advantage over config.gem – it doesn’t suck. And it’s much better suited to deployment. I’ve already added it to a hobby rails project, a work gem project and have been working with one of our teams to get it added to a major project today. Here’s some stuff I’ve learned:

MySQL

As we all know, installing the MySQL gem on Snow Leopard required specifying the architecture under which to build – because the gem cannot work universally. Because the build is quite specific to the OS you’re working on, you have two choices when running with bundler.

Option A: bundle the gem and specify the architecture on the CLI

I’m storing (and versioning) the .gem binary files, and not files created when you run ‘gem bundle’. If you’ve got the MySQL gem cached as well, you’ll need to run ‘gem bundle’ passing a config file, something like this:

--build-options build_options.yml

Where the build_options.yml looks something like:

mysql:
  mysql-config: /usr/local/lib/mysql

And you’ll have to specify the architecture, bringing the whole command to:

$ env ARCHFLAGS="-arch x86_64" gem bundle --build-options build_options.yml

Instead, I chose to go with:

Option B: Use ruby’s MySQL gem

This might be your rvm ruby, or your system. Either way, if you want your app to be able to access the MySQL gem, but not to bundle it, add the following to your Gemfile:

gem 'mysql', :bundle => false

And run ‘gem bundle’ again to regenerate the environment file. It seems that, depending on your system, bundler will sometimes fall back to ruby’s own gems automatically, and sometimes it won’t. For me, I was getting ‘no such file to load – mysql’ until I added the config to the Gemfile.

Btw, adding ‘disable_system_gems’ to your Gemfile when you’re calling a system gem will not work. Perhaps this is obvious – I had assumed (perhaps mistakenly) that the behaviour would only disable the graceful ‘fallback’ to the system; in fact it seems to block even explicit calls to the system.

Changing the vendor directory with Rails

The current default for the bundler gem cache is vendor/gems – this is the same as Rails 2.x and can mean config.gem tries to deal with them. Do yourself a favour and add the following to your Gemfile:

bundle_path "vendor/bundler_gems"

Or similar… this helped me with a few weird Rails issues.

Helpful stuff on bundler:

• Main docs
• Gem Bundler is the Future
• A rails template for gem bundler

The peeps in #carlhuda on irc.freenode.net were also very helpful when I was struggling with this.

Upon installing a new Mac for home development, I decided to try out a few new technologies – RVM was one of them. It was relatively painless in terms of commands; but it did require a bit of a mindset shift to understand how to debug it. Hopefully this will help someone; the steps below should work for linux machines too, albeit with some slight changes to paths.

RVM, Ruby Version Manager, allows you to run multiple ruby versions on the same machine. This is good if you want to test upgrades etc. For me, a pleasant side effect was being forced away from using the system version of ruby. Relying on the system version (and system gems… more on that in a bit!) seemed a little lazy, and was always a bit of a pain when unraveling dependencies and moving stuff live.

Another complementary tool, which I’ll hopefully blog about separately, is bundler. Do yourself a big favour and don’t try to install both at once; it can get a mind-boggling. Get to the point where you’re happy with one of the two, and then start on the other.

The RVM site is full of useful information, although it’s not always super easy to find what you need. The stuff on installation is vital (for the record, the gem version worked for me – in fact, rvm is the only gem I have installed on the system ruby).

Once installed, I installed ruby 1.8.7 with:

$ rvm install 1.8.7

The system was already running 1.8.7, so I figured this was probably the best place to start. The plan is run within this version by default, and to use other versions as appropriate.

Once installed, activate with:

$ rvm use 1.8.7

This will switch the ruby version for the current terminal. Only. This wasn’t immediately clear to me; if you want to switch for all terminals, add the default flag:

$ rvm use 1.8.7 --default

Fire up a new terminal, and it should now also be running the same version of ruby. Check with ‘ruby -v’:

$ ruby -v
ruby 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0]


To be sure that you’re using the version of ruby you expect, there are a few things you can check:

$ which ruby
/Users/samphillips/.rvm/ruby-1.8.7-p248/bin/ruby


If you get ‘/usr/bin/ruby’ instead, you’re still using the system version.

RubyGems is installed separately for each ruby instance. To check that the ‘gem’ command is hitting the right place, again, break out ‘which’:

$ which gem
/Users/samphillips/.rvm/ruby-1.8.7-p248/bin/gem


Another good check is ‘gem env’:

$ gem env
RubyGems Environment:
- RUBYGEMS VERSION: 1.3.5
- RUBY VERSION: 1.8.7 (2009-12-24 patchlevel 248) [i686-darwin10.2.0]
- INSTALLATION DIRECTORY: /Users/samphillips/.rvm/gems/ruby-1.8.7-p248
- RUBY EXECUTABLE: /Users/samphillips/.rvm/ruby-1.8.7-p248/bin/ruby
- EXECUTABLE DIRECTORY: /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86-darwin-10
- GEM PATHS:
- /Users/samphillips/.rvm/gems/ruby-1.8.7-p248
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- :sources => ["http://gems.rubyforge.org/", "http://gemcutter.org/"]
- REMOTE SOURCES:
- http://gems.rubyforge.org/
- http://gemcutter.org/


Again, if you’re seeing references to directories like these:

- INSTALLATION DIRECTORY: /Library/Ruby/Gems/1.8
- RUBY EXECUTABLE: /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
- EXECUTABLE DIRECTORY: /usr/bin


Then your switch hasn’t worked.

Some reasons I’ve found why switching hasn’t worked:

• It’s apparently possible to drop the ‘use’ keyword when choosing ruby versions. I’ve learned not to do this; firstly it doesn’t give you a nice confirmation that it’s worked, and secondly it didn’t seem to work correctly (perhaps the two are related…)
• Similarly, although you’ve assigned the ‘default’ version of ruby, running ‘rvm use default’ doesn’t seem to work properly all the time.

If keeping to the full ‘rvm use xxx –default’ command doesn’t help you, it might be worth re-installing. This is easy; switch back to the system version of ruby (’rvm use system’), delete the .rvm directory in your home directory and uninstall the gem. It seems that my mistake with my original install was a classic one:

Too much sudo

We’ve got very used to installing gems, and doing anything to do with them, with sudo. With rvm, you don’t need to do this and it will lead to confusing results, as the docs point out at length. Every time you use sudo, you’re going to be messing with the system ruby version (as this is in root’s path), and weird results will follow.

To summarise: use sudo to install the gem on system ruby, and then leave the sudo well alone.

Passenger

If you’re using with passenger + apache, first install the passenger gem against your rvm ruby version:

$ rvm use 1.8.7
$ gem install passenger
[snip]
$ passenger-install-apache2-module


Note the lack of sudo. The module installer will suggest the apache config syntax. This needs changing slightly, as per the docs –

$ cat /etc/apache2/other/passenger.conf
LoadModule passenger_module /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/gems/passenger-2.2.8/ext/apache2/mod_passenger.so
PassengerRoot /Users/samphillips/.rvm/gems/ruby-1.8.7-p248/gems/passenger-2.2.8
PassengerRuby /Users/samphillips/.rvm/bin/passenger_ruby


(restart apache…)

The last line will point apache at your rvm ruby version. When using passenger with rvm, switch the rvm version with the following syntax:

$ rvm use 1.9.1 --default --passenger

I was confused for a number of minutes when gems installed on my rvm version of ruby weren’t available to apache; not doing this was why.

RVM works nicely already, and has potential to do lots of good. If you’ve got any problems, hop on to #rvm on irc.freenode.net, where friendly folks will help you out.

November In Manchester, the brainchild of my good friend Tom Mason, is a social media love story. What does this mean? It means that the eight characters have played out their story in real time, over social media. But what is ’social media’? Well, for this project, it meant Flickr, Blogger, and, fundamentally, Twitter.

Twitter is the stage on which this story was played out. What is Twitter? The best explanation I’ve heard so far is ‘one-to-many text messaging’. You send a text – 140 characters – and instead of it being sent to just one friend, it’s sent to all of your ‘followers’. They receive the text on their phones, through the Twitter website, or through Twitter software installed on their computers.

The story is not set in stone, and it happens in real time. In this way, it’s almost more of a reality show than a fictional work with a predetermined path. The characters – Persephone, James, Leonie, Darren, Regina, Paul, Melody and Lee – live in our reality. When we put together the main site, we even gave it a feel of a Big Brother microsite. This is a reality show like no other.

Persephone and Leonie went to the christmas lights being turned on in Manchester city centre, for example. It was quite the live drama! Paul and James went to a football match at Old Trafford. Leonie’s in PR, so if there’s a party or networking event going on, she’s there.

The characters also respond to other people’s tweets – meaning that, just by tweeting them and getting a reply, anybody can become a character in the story. It’s been fascinating to watch people get involved with the story.

But what is most striking about these characters is that they are very almost real people, very believably existing on Twitter – in a way that, in reality, most people don’t. Although Facebook is a mass phenomenon, and MySpace/Bebo remain a modernised version of Geocities, Twitter isn’t quite there yet. It’s the domain of geeks, marketeers and companies doing innovative customer engagement. The November In Manchester characters aren’t any of these types – they’re normal mancunians trying to get from the start to the end of a day unscathed. You can believe that they are real, and reading their tweets you can believe that other, real, people could use Twitter in this way as well.

November In Manchester is now my first example when I explain to a client, colleague, friend or grandparent what Twitter is, what it does, and what it’s for. What is Twitter for? It’s for telling your story to the rest of us.

Interested in how the November In Manchester site works? Check out November In Manchester: Joining those technical dots.

November In Manchester is a social media love story. It focuses on the lives of eight fictional mancunians over the course of November – their story is played out over the characters’ tweets and blog posts; all of which are disguised as real people. They even interact with real people; they attend real events and they take real photos of the city. The main November In Manchester site was to bring their story together in to a single timeline, and when Tom approached me to build it, I was happy to oblige.

For my own notes, I thought I’d jot down a few details of the implementation for future reference, or for anyone who was interested. November In Manchester is built in Rails using HAML and SASS.

Scheduling

This was a nice opportunity to use the whenever gem. If you’ve not used this before, check it out – it’s a cracking gem and its example Capistrano recipe works a treat. Although I’m happy enough with cron syntax I was pleasantly surprised to have it so smoothly taken out of my hands, and means that our various scheduled tasks are managed straight from a config file in our application.

Twitter

The Twitter implementation was the most difficult part. The initial version used dancroak’s twitter search gem and worked a treat with the code. I don’t particularly like the query syntax, but this is down to twitter, not the gem. Also down to twitter is the job of updating the search results. Unlike the main site, the search site (and API) is not always real-time and can lag heavily behind. It’s no good being 8 hours on a story that’s supposed to be happening in real time.

We therefore switched the search gem out and ended up using the main API and gem. Initially we were perplexed about how to best get the information about various users’ tweets. The API is designed for a piece of software acting as a twitter user, and so can tweet, follow, etc – but it can’t search. In the end we settled on using the ‘user_timeline’ method and passing the username (screen_name) of the user we wanted to get the tweets for. We run this search every five minutes, and as long as we’re not more than 20 tweets behind (the method’s limit – although you may be able to override), we’re golden. It’s not ideal (better suggestions received) but it got us back to real-time.

Thankfully our architecture allowed this switch to be made in 10 lines or so of code. Good job – I had a matter of minutes before BarCamp Manchester 2 started to switch it over!

RSS/Atom

Ruby has built-in support for RSS, and we initially used this for the integration. However, it doesn’t support Atom and I didn’t want to be caught by surprise if a new blog turned up that used this format (I didn’t know what new characters and data sources Tom would introduce as the plot twisted and turned!), so we switched over to the excellent Simple RSS gem. This gem also supports reading from the content of the posts – which at one point we were using – which the built-in Ruby stuff doesn’t.

Flickr

Photos are a big part of the project, and we wanted to have the most recent photos displaying on the site, pulled from the flickr feed.

We used the flickr-fu (not ‘flickr_fu’, confusingly) gem for our integration here, and took inspiration from this blog post. I don’t think that helpers are the right approach here, though, so we instead import our flickr photos every 3 hours through a scheduled process and stick them in a database – hotlinking the actual images straight from flickr.

Deployment

We use Capistrano for deployment and git for version control.

Overall the tech stuff on the site was just the work of a weekend or two and a few evenings. It was nice to have a small project that was live quickly – something I’ll definitely try to bear in mind for the future. Hope you’re enjoying the story!