This screeching noise

This noise, this screeching, whining noise, has forever embedded itself in my auditory system as the sound of amazing things happening.

First, the long screech of the handshake; two machines sizing each other up. Questions asked and answered. Decisions made.

With both satisfied, the stuttering chatter. Quick as you can, what we have to say is too important to waste with frivolities.

In the silence — both machines process their exchange across the ether.

Somewhere, a bit flips. We repeat.

Update: Thanks to Chris for letting me know that, of course, Toshiba shamelessly ripped this advert off Simon Faithfull’s Escape Vehicle — which even has the original screeching noise.

Using Geoplanet Data in Ruby on Rails

The Unrendered City is Here for You to Use

Noticings is possibly one of the first services to integrate the Yahoo Geoplanet Data deeply, although it seems we can now add Twitter to the list. I imagine we’ll see a few more services begin to use it soon – Yahoo have released it under a Creative Commons Attribution license, and if Twitter are using it then a whole bunch of things are going to spring up around that.

It gives us the opportunity to use colloquial geography rather than bounding boxes and radial searches and the like. I banged on about this in my talk at the AGI conference recently. I am such a geography bore.

Anyway, we couldn’t have built Noticings without it.

However, it is a little bit difficult to get up and running at first, so I want to delve a bit into how we’re using it, hopefully helping others get rolling a bit quicker. Noticings is written in Ruby on Rails, but I’m sure the same principles apply to whatever you’re framework/language you’re using.

First, some background. Geoplanet is a database of 5.4 million places in a hierarchy. Each entry has a unique, permanent ID (WOEID), a name and a place type. For example, Homerton (20089379) is a Suburb in the London Borough of Hackney (12695808), which is a LocalAdmin in London (44418), which is a Town in Greater London (23416974), and so on.

Once you’ve discounted the place types that Flickr doesn’t use for associating photos with (there are huge numbers of zip codes and telephone dialling zones, for example), then there are about 1.4 million places that Noticings cares about.

The Geoplanet download contains three tab-separated files. Places, which does what it says. Aliases, which contains alternate language names for each place. Adjacencies, which contains info about which places are adjacent to each other (although not necessarily geographically continuous).

There are three tables in our database, one for each of these:

:::ruby
create_table "geoplanet_adjacencies", :force => true do |t|
  t.integer "woeid", :limit => 8
  t.string  "iso_code"
  t.integer "neighbour_woeid", :limit => 8
  t.string  "neighbour_iso_code"
end

add_index "geoplanet_adjacencies", ["woeid"], :name => "index_geoplanet_adjacencies_on_woeid"

create_table "geoplanet_aliases", :force => true do |t|
  t.integer "woeid", :limit => 8
  t.string  "name"
  t.string  "name_type"
  t.string  "language_code"
end

add_index "geoplanet_aliases", ["woeid"], :name => "index_geoplanet_aliases_on_woeid"

create_table "geoplanet_places", :force => true do |t|
  t.integer "woeid", :limit => 8
  t.integer "parent_woeid", :limit => 8
  t.string  "country_code"
  t.string  "name"
  t.string  "language"
  t.string  "place_type"
  t.string  "ancestry"
end

add_index "geoplanet_places", ["ancestry"], :name => "index_geoplanet_places_on_ancestry"
add_index "geoplanet_places", ["parent_woeid"], :name => "index_geoplanet_places_on_parent_woeid"
add_index "geoplanet_places", ["woeid"], :name => "index_geoplanet_places_on_woeid", :unique => true

And there’s rake task which handles the import. This takes ages. In addition to the 5.4 million places, there are about 2 million aliases and 8.4 million adjacencies. Go and make several cup of teas if you’re running this. Do the crossword too.

:::ruby
namespace :geoplanet do
  DATA_PATH = File.join(Rails.root, 'data', 'geoplanet', 'geoplanet_data_7.3.2')
  namespace :import do
    task :all => [:places, :aliases, :adjacencies]
    task :places => :environment do
      ActiveRecord::Base.connection.execute("TRUNCATE TABLE geoplanet_places")
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_places DISABLE KEYS")
      ActiveRecord::Base.connection.execute("LOAD DATA LOCAL INFILE '#{DATA_PATH}/geoplanet_places_7.3.2.tsv' REPLACE INTO TABLE geoplanet_places FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' IGNORE 1 LINES (woeid, country_code, name, language, place_type, parent_woeid);")
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_places ENABLE KEYS")
    end

    task :aliases => :environment do      
      ActiveRecord::Base.connection.execute("TRUNCATE TABLE geoplanet_aliases")      
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_aliases DISABLE KEYS")      
      ActiveRecord::Base.connection.execute("LOAD DATA LOCAL INFILE '#{DATA_PATH}/geoplanet_aliases_7.3.2.tsv' REPLACE INTO TABLE geoplanet_aliases FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' IGNORE 1 LINES (woeid, name, name_type, language_code);")
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_aliases ENABLE KEYS")
    end

    task :adjacencies => :environment do
      ActiveRecord::Base.connection.execute("TRUNCATE TABLE geoplanet_adjacencies")
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_adjacencies DISABLE KEYS")
      ActiveRecord::Base.connection.execute("LOAD DATA LOCAL INFILE '#{DATA_PATH}/geoplanet_adjacencies_7.3.2.tsv' REPLACE INTO TABLE geoplanet_adjacencies FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' IGNORE 1 LINES (woeid, iso_code, neighbour_woeid, neighbour_iso_code);")
      ActiveRecord::Base.connection.execute("ALTER TABLE geoplanet_adjacencies ENABLE KEYS")
    end
  end
end

We’ve also got three models — one for each of the tables.

:::ruby
class GeoplanetPlace < ActiveRecord::Base
  set_primary_key 'woeid'
  has_many :aliases, :class_name => 'GeoplanetAlias', :foreign_key => 'woeid'  has_many :adjacencies, :class_name => 'GeoplanetAdjacency', :foreign_key => 'woeid'  has_many :adjacent_places, :through => :adjacencies
end

class GeoplanetAlias < ActiveRecord::Base
  belongs_to :geoplanet_place, :foreign_key => 'woeid', :primary_key => 'woeid'
end

class GeoplanetAdjacency < ActiveRecord::Base
  belongs_to :place, :class_name => 'GeoplanetPlace', :foreign_key => 'woeid', :primary_key => 'woeid'  belongs_to :adjacent_place, :class_name => 'GeoplanetPlace', :foreign_key => 'neighbour_woeid', :primary_key => 'woeid'
end

Now we’ve got a usable copy of Geoplanet, and we can do things like:

:::ruby
london = GeoplanetPlace.find_by_name "London"
londons_children = GeoplanetPlace.find(:all, :conditions => { :parent_woeid => london.id })

Which is all well and good, but if you want to fetch the second level children you have to add a JOIN, and another for third level children. It quickly becomes slow and impossible.

We need to start caching the tree for each row somehow, making it easier and quicker to find children, siblings and ancestors.

Step forward Ancestry, a plugin by Stefan Kroes for organising ActiveRecord models in a tree structure. It store the ancestors for each row in a string, using the ‘LIKE‘ statement to SELECT them.

:::ruby
>> homerton = GeoplanetPlace.find_by_name "Homerton"
>> homerton.parent
>> homerton.parent.parent

Ancestry adds a class method called build_ancestry_from_parent_ids!, for transforming a more traditional parent_id tree structure into the format for Ancestry. That’s what we’ve got here, except the parent field is called parent_woeid. In our case Earth is the root of the tree, and has a parent_woeid of -1, not nil.

By overriding that method in GeoplanetPlace we can convert the Geoplanet tree structure into something usable by Ancestry.

:::ruby
def self.build_ancestry_from_parent_ids! parent_id = nil, ancestry = nil
  parent_id = parent_id || -1
  self.base_class.all(:conditions => {:parent_woeid => parent_id}).each do |node|
    node.without_ancestry_callbacks do 
      node.update_attribute ancestry_column, ancestry
    end
    build_ancestry_from_parent_ids! node.id, if ancestry.nil? then "#{node.id}" else "#{ancestry}/#{node.id}" end
  end
end

This takes bloody ages. About 4 hours on my laptop. You can go to bed now if you like.

To save you all that here’s a prebuilt SQL dump of all three tables (156MB gzipped SQL), ready for import. It’s built from version 7.4.0 of the Geoplanet Data, but you should check whether that’s the latest.

And once you’ve done all that you’ll be able to do things like this in a blink of an eye:

:::ruby
# in use is a named_scope in GeoplanetPlace with conditions on place_type
>> homerton.siblings.in_use.map(&:name)
["Shoreditch", "Upper Clapton", "Kingsland", "Lower Clapton", "Shacklewell", "Haggerston", "Clapton Park", "Homerton", "Hackney Wick", "South Hackney", "Dalston", "De Beauvoir Town", "Dalston Kingsland", "Brownswood Park", "Stoke Newington", "Stamford Hill", "Finsbury Park", "Clapton", "Hackney"]
>> homerton.ancestors.map(&:name)
["Earth", "United Kingdom", "England", "Greater London", "London", "London Borough of Hackney"]
>> homerton.parent.descendants.count
5038
>> homerton.parent.descendants.in_use.count
20

One thing is missing, which may or may not be an issue for you: bounding boxes and polylines for each place. You’ll have to use the Geoplanet live API or Flickr for that – the downloadable data provided by Yahoo doesn’t contain this information. But hopefully it soon will — Yahoo have said they will open up all their geodata by the end of 2010.

The Scruffying of Print

Printers

The books and newspapers I read are, on the whole, designed by people pushing boxes around in expensive pieces of software. They’ve been carefully nudged, tweaked and adjusted to look Just Right. Designed, in the first order sense of the word.

Lots of the traditional print businesses are looking to print-on-demand for new business models and approaches. And rightly so – it’s exciting stuff. But if you’re going to, for example, tailor a portion of the content in your newspaper to the reader’s local area then you need to be able to automate that completely, because you can’t afford to tweak 300,000 newspapers individually.

The job of the designer becomes a second-order one — of creating templates, workflows and systems to support that — not pushing around boxes of text by hand.

Web designers are used to this. You quickly learn when making a website that coding each page by hand is impossible, and that you’re going to have to use layouts and templates. And these layouts and templates need to support headlines which are slightly longer than you might expect, or content that wraps in an unexpected place. So you design things with high tolerances and gentle failure modes which still look OK when everything isn’t quite as expected.

This is something that print is going to have to get used to. Slightly messier, with a few more bad line breaks and unbalanced columns. I don’t think readers are going to have any trouble with this — if anything people are getting used to scruffy nature of the web, and they definitely don’t care as much about design as designers think they do. It’s the producers and editors that are going to have to get used to letting go, giving a little bit less time to perfecting visual design and a bit more time to making sure the content shines through.

Crowdsourcing the Shape of Neighbourhoods: The Motion Picture

I gave a talk at the AGI Geocommunity ’09 a few weeks ago, entitled “Crowdsourcing the Shape of Neighbourhoods”.

I treated it as a bit of an introduction to that psychogeography stuff, a fairly new topic for the AGI, and I was really pleased when there was such a positive response from people. It provoked some good discussion, and hopefully injected something a bit different into the proceedings.

So here it is. I’ve not had time to watch it all the way through, but I seemed to be mostly coherent, which is a bonus.

Clarke on Snow Leopard

I forgot to mention that Clarke now works on Snow Leopard, using that Core Location cleverness. Which might make it the first app in the wild to do so, although I’m sure someone will tell me it’s not.

People have asked me to add support for Google Latitude, mapme.at, or Brightkite and all those things. I don’t use them, but the source code is on Github, and I’m happy to merge changes back in. If you want to update your Twitter location automatically, EagleTweet will do the trick.

The only thing I still want to add is the ability to ping a URL of your choice when your location changes. For those home/office automation tricks.

And yes, I know the menu icon is still in colour. It’s on Ben’s voluminous todo list.

Bike Hero

It’s Autumn. The leaves are dropping, just like the temperature. Which means it’s time for Bike Hero. Somewhat inspired by this video, Bike Hero is a game to be played in your head whilst cycling along leafy paths and lanes.

The objective of the game is to run over as many leaves as possible. The faster the better. Every leaf that makes an audible crunch, soft or loud, is a point. You lose points for crashing into cars, pedestrians or park benches. As many as you feel is necessary.

A typical level might look something like this:

Bike Hero Level

And a good route through it would probably be something like this:

Bike Hero Level Augmented

Now, obviously it’s not as simple as that. There are traps to be avoided. You might think this is a good section to aim for. Lots of leaves, prime for crunching. But you’d be wrong. There’s a drain cover just under those leaves, perfect for wiping out on.

Bike Hero Trap

But if you do there’s always an opportunity to make up some points with a bonus section like this:

Bike Hero Bonus Section

Or an extra dangerous drunken night mission (double points):

Bike Hero Night Mission

There’s a limited window in which to play Bike Hero every year, so make the most of it while you still can. Happy crunching.

Paused

iPhone apps I use & love

I was helping a friend get set up on his iPhone, and he wanted some tips on apps that are worth getting. Rather than stick it in an email, here it is: what I use on a regular basis and have survived the test of time.

Noticings

For a while Tom Armitage and I have been kicking around the idea of a game called Noticings. Recently we managed to find the time to build a simple version of it, and a few of our friends have been playing it for a few weeks. It seems to be going OK, so yesterday I took the password off and opened it to the world.

From our about page:

Cities are wonderful places, and everybody finds different things in them. Some of us like to take pictures of interesting, unusual, or beautiful things we see, but many of use are moving so fast through the urban landscape we don’t take in the things around us.

Noticings is a game you play by going a bit slower, and having a look around you. It doesn’t require you change your behaviour significantly, or interrupt your routine: you just take photographs of things that you think are interesting, or things you see. You’ll get points for just noticing things, and you might get bonuses for interesting coincidences.

If you spot something interesting, take a photo of it and upload it to Flickr, tagged with ‘noticings’ and some geodata. Each day is a turn, and photos from the day get pulled in at 3pm GMT the following day, giving you time to upload them and tag them properly.

We’re starting very simple. There are two rules:

Slowly, as the game evolves we’ll add new, more exciting rules and tweak the old ones. Perhaps extra points for venturing out at lunchtime every day for a week, or for spotting things along lines of latitude or longitude. Who knows? It’s part art, part psychogeography, part city discovery engine. You can keep up with the changes on the blog.

We’d love it if you gave it a go, and even more if you let us know your thoughts (email is probably best).

Update: Tom has jotted down some his thoughts, expressing the goal much better than I.

London Bus iPhone and iPod Touch Application

London Bus iPhone and iPod Touch Application

All of this augmented reality stuff looks incredibly dated before its even launched. There must be something more interesting that can be done with the Little Box of Powerful Sensing that doesn’t involve stabbing your fingers at Big Logos floating across smudged glass screens. Something more like the Whispers of Angels.

Just a hint, a gentle nudge. Like a cheat in a videogame. Technology finessed to the point that you could easily convince yourself that if you were smarter, a more capable person, you wouldn’t need it.