CityCyclist 1.0

Untitled

tldr; I made an app for bike navigation, you can download it here.

For a few months, in slivers of spare time, I’ve been working on a little app for city bike navigation, called CityCyclist.

I’ve tried to build something clean and accessible, that gets a good bike route on the screen as quickly as possible. That’s glanceable while on a bike, and more useful when off.

KEY INNOVATIONS!

There’s a little scrubber on the elevation profile at the bottom to fly quickly along a route without zooming and panning around. My hypothesis was that might make it easier to consign a route to memory. I suspect that’s not true, but I still like it.

The search results use a combination of Foursquare and Apple’s address geocoder, and seem fairly good.

The routing is powered by CycleStreets (backed by OpenStreetMap) with a selection of three options: fast, balanced, quiet. (UK only for now.)

END KEY INNOVATIONS!

I think version 1.0 is a good base for future experiments. I’d like to try a take on turn-by-turn navigation that works when the phone is in your pocket or bag, and I’d like to find a feedback loop to improve routing/map data. But this’ll do for now.

It’s in the App Store now and it’s free. If you give it a try, I’d love to know what you think. Get it here.

(PS: If you work for the kind of organisation that would benefit from me supporting, improving, and getting this app to as many people as possible, and want to help out, do get in touch.)

What Do People Do All Day?

Everyone is a Worker

One of Sam’s favourite books at bedtime is Richard Scarry’s ‘What Do People Do All Day?’. I was going to write a post all about it, because it’s kind of brilliant (explaining economic and industrial processes for the under fives! lovely drawings!) and kind of terrible (gender roles! a society seen solely through the eyes of capitalism!).

So I started writing, but then I found this great post deconstructing it better than I could:

Which rather begs my usual question: so why is it that we no longer make these kinds of books? Why is it that we have shifted our focus to how things work or how people used to work as opposed to how people work now? Is it that work is too elusive, that new economy jobs are harder to draw? Can we not deal with the fact that Alfalfa has become a derivatives trader? But work of course is far from invisible. It’s not just that we do so many of the occupations lovingly drawn by Scarry, and in more or less the same way. It’s also that people still work in manufacturing, only mostly elsewhere. We could teach our children about that, just like we teach them that everybody poops.

(Obviously, someone has done a ‘funny’ 21st century version, but that doesn’t count.)

So I read it, and I looked at the news, and it became pretty clear to me while I’d like to see more of this sort of book, the kids will be just fine. But we could do with a bit more Richard Scarry, and a bit more How Things Work, and a bit more Secret Life of Machines for everyone else.

New year, new job!

Me at Desk

This is my last fortnight at Newspaper Club, after a brilliant five and a half years. We’ve built a great little company that’s growing fast, but it’s time for me to move on to something new. I’m pleased to be leaving it in very capable hands, and I can’t wait to see all the 2015 plans come to fruition. I’m going to miss all my colleagues there, but a special mention to Anne, who has worked tirelessly to turn our silly idea into, well, a non-silly idea. One that employs people, pays taxes and has happy customers all over the world.

This has been in the works for a little while, and I’d planned on going freelance again, until the next meaty long-term thing came along. So it was a surprise to find that thing before I’d even got started.

I’m joining Moo, as Head of Technology of a new multidisciplinary team working on scalable digital products. We’re starting from scratch, with the Moo engine behind us, and a blank whiteboard in front of us. Exciting! (Also, in an odd, but nice twist, we’ll be occupying the old RIG/BERG/MakieLab space on Scrutton St! Come visit!)

Happy 2015 everyone!

Custom EC2 hostnames and DNS entries

I’ve been doing some work with EC2 recently. I wanted to be able to bring up a server using Ansible, pre-configured with a hostname and valid, working FQDN.

There’s a few complexities to this. Unless you’re using Elastic IP addresses, EC2 instances will change public IP address on reboot, so I needed to ensure that the DNS entry of the FQDN will update if the host changes.

A common way of doing this is to bake out an AMI, pre-configured with a script that runs on boot to talk to the DNS server and create/update the entry. But you still need a way of passing the desired hostname in when you launch the instance for the first time, and you end up with your security keys baked onto a AMI, making it difficult to rotate them. And custom AMIs are fiddly – I’d prefer to use the official ones from Ubuntu or Amazon so I don’t have to bake out a new AMI on every OS point release.

I ended up with an approach that uses a combination of cloud-init, an IAM instance role, and Route 53, to set the hostname, and write a boot script to grab temporary credentials and set the DNS entry.

EC2 supports a thing called IAM Instance Roles, allowing an EC2 instance to grab temporary credentials for a role, letting it access AWS resources without hardcoding the access tokens. It does this by fetching credentials from an internal HTTP server, but if you use awscli or other official libraries, they’ll do this for you, unless you provide credentials explicitly.

In this case, we grant just enough permission to be able to update a specific zone on Route 53. I chose to put all my server DNS entries in their own zone to isolate them, but you don’t have to do that. I made a role called ‘set-internal-dns’ and gave it a policy document like this:

{
  "Statement":[
    {
      "Action":[
        "route53:ChangeResourceRecordSets",
        "route53:GetHostedZone",
        "route53:ListResourceRecordSets"
      ],
      "Effect":"Allow",
      "Resource":[
        "arn:aws:route53:::hostedzone/<ZONE_ID_HERE>"
      ]
    }
  ]
}

Next, I wrote an Ansible task to boot a machine set to that role, with a user-data string containing cloud-init config.

- name: Launch instance
  ec2:
    keypair: "{{ keypair }}"
    region: "{{ region }}"
    zone: "{{ az }}"
    image: "{{ image }}"
    instance_type: "m3.medium"
    vpc_subnet_id: "{{ vpc_subnet_id }}"
    assign_public_ip: true
    group: ['ssh_external']
    exact_count: 1
    count_tag:
      Name: "{{ item.hostname }}"
    instance_tags:
      Name: "{{ item.hostname }}"
      role: "{{ item.role }}"
      environment: "{{ item.environment }}"
    volumes:
      - device_name: /dev/sda1
        volume_size: 30
        device_type: gp2
        delete_on_termination: true
    wait: true
    instance_profile_name: set-internal-dns
    user_data: "{{ lookup('template', 'templates/user_data_route53_dns.yml.j2') }}"
  with_items:
    - hostname: "computer1", 
      fqdn: "computer1.{{ domain }}"
      role: "computation"
      environment: "production"

Ansible expects the user_data property as a string, so we load a template as a string using lookup.

cloud-init has the lowest documentation quality to software usefulness ratio I think I’ve ever seen. In combination with EC2 (and presumably other cloud services?), it allows you to pass in configuration settings, packages to install, files to upload and much more, all through a handy YAML file. But all the useful documentation about the supported settings is completely hidden away or just placeholder text, except for a huge example config.

Our user_data_route53_dns.yml.j2 template file is below. If you’re not using Ansible, the bits in the curly brackets are templated variables being set by the task above.

#cloud-config

# Set the hostname and FQDN
hostname: "{{ item.hostname }}"
fqdn: "{{ item.fqdn }}"
# Set our hostname in /etc/hosts too
manage_etc_hosts: true

# Our script below depends on this:
packages:
  - awscli

# Write a script that executes on every boot and sets a DNS entry pointing to
# this instance. This requires the instance having an appropriate IAM role set,
# so it has permission to perform the changes to Route53.
write_files:
  - content: |
      #!/bin/sh
      FQDN=`hostname -f`
      ZONE_ID="{{ zone_id }}"
      TTL=300
      SELF_META_URL="http://169.254.169.254/latest/meta-data"
      PUBLIC_DNS=$(curl ${SELF_META_URL}/public-hostname 2>/dev/null)

      cat << EOT > /tmp/aws_r53_batch.json
      {
        "Comment": "Assign AWS Public DNS as a CNAME of hostname",
        "Changes": [
          {
            "Action": "UPSERT",
            "ResourceRecordSet": {
              "Name": "${FQDN}.",
              "Type": "CNAME",
              "TTL": ${TTL},
              "ResourceRecords": [
                {
                  "Value": "${PUBLIC_DNS}"
                }
              ]
            }
          }
        ]
      }
      EOT

      aws route53 change-resource-record-sets --hosted-zone-id ${ZONE_ID} --change-batch file:///tmp/aws_r53_batch.json
      rm -f /tmp/aws_r53_batch.json
    path: /var/lib/cloud/scripts/per-boot/set_route53_dns.sh
    permissions: 0755

We’re installing our script into cloud-init’s per-boot scripts rather than anywhere else because I know cloud-init will run it on first boot, after it has been installed. If we put it in rc.d, for example, we’d still have to tell cloud-init to go and run it on first boot, so this is just one less thing to mess up. I’m already feeling pretty bad about writing JSON in a shell script in a YAML file.

When you boot the instance you should be able to tail /var/log/cloud-init-output.log and see a confirmation from the awscli script that the DNS change is pending. It can take 10-60 seconds to become available.

We’re using a CNAME to the EC2 public DNS entry because I still want to use split horizon DNS – if you look that entry up from inside your EC2/VPC network you’ll get the internal IP address.

Computers.

Surprises, small and big

OK, look, I know think pieces about Apple belong on Medium, but just watch video of the iPhone launch in 2007. The bit where Steve demos the ‘slide to unlock’ (15:14), and there’s an audible intake of breath from the audience. And then again, later, with the ‘pinch to zoom’ (33:22).

(It’s also a really funny presentation! I forgot how light hearted they used to be.)

The first iPhone astounded me, because it felt like something from 2-3 years in future. At the time, multitouch screens were reserved for table top projection surfaces (Microsoft’s original Surface project, for example), and while it seemed clear that it was going to be an important form of interaction in the future, every other device on the market had a physical keyboard or a stylus.

I didn’t even know multitouch technology was capable of being shrunk to that size, for that price. Apple had managed to not only make it happen, but had orchestrated their supply chain into producing millions of them, leaving the rest of the phone industry dumbfounded. And due to their exclusivity contracts, it would take years before they truly caught up. Incredible.

Satellite View of GCHQ Bude

It’s summer 2013, and the Snowden leaks are in full swing. All the geeks will tell you that they expected it all along, but they’re lying. No-one expected it to be that fierce, that pervasive, that explicit.

The culmination of it, for me, was when we learnt that GCHQ intercepts and stores all transatlantic network communications through their intercept station in Bude, Cornwall.

The story isn’t clear at first. All of it? That can’t possibly be right. But yes, that turns out to be 21 petabytes a day, over a rolling 3 day buffer. Around 60+ petabytes at any one time.

Storing it I can fathom – just about. You need lots of space, lots of disks, lots of money, and I guess they get a volume discount. But searching it in real-time too? Huh.

This is how people make things now

It’s always good seeing behind the scenes of stuff you love. More so if there’s good engineering involved, so I enjoyed this pair of videos from Spotify. Lots of overlap with Creativity, Inc. too, unsurprisingly. This is how people make things now.

(part 1, part 2)

PaperLater

PaperLater

This week we launched a thing I’ve been working on for the last few months, alongside a brilliant little team of colleagues and freelancers.

PaperLater lets you save the good bits of the web to print, so you can enjoy them away from the screen. If you’ve used something like Instapaper, Pocket or Readability before, it’s a bit like that, but in print.

It’s a really nice scale of product to have built. We solved some gnarly technical problems (automated layout/typography, single copy print production, content extraction), but it’s distilled into what’s really quite a small web app. There’s only a handful of pages, and we’ve tried to make the whole experience feel really light and easy. Time will tell us whether we’ve got that right, but I’m proud of that.

It’s nice to realise that we’re getting better at launching things. Little things make it easier: knowing to get nice photos shot before launch; having a customer support system to bolt into; having an existing framework for legal documents; and so on.

I’m also getting comfortable with patterns and tools that reduce the numbers of things I need to think about, and let me concentrate on building the thing at hand. I’m never changing my text editor again, for example. That’s a good feeling, and only taken a decade.

I think of PaperLater a bit like podcasts. I don’t really listen to podcasts, not because I don’t like them, but there just isn’t a podcast shaped hole in my life. But there is a PaperLater shaped hole, and we built it because our hunch is there’s one in other people’s lives too. If there’s one in yours, I hope you enjoy it.