Internet Research #1

1. I've been enjoying noodling around with some electronics recently. One of the things I'm working on is converting a pair of beautiful old Bang and Olufsen CX50 speakers into an open source Sonos-esque multiroom audio system. I'm following HifiBerry's guide for the hardware part, and in it there's two 3D printed parts you need.

So I go on 3D Hubs, upload the two .stl files provided, get a quote, credit card, and now it's printing somewhere in Farringdon. You still need to know whether you want PLA or ABS or whatever, but I was very impressed at how easy the whole thing was. Just another mundane e-commerce transaction. We're out the other side of the hype cycle.

2. Simon Willison linked to libpostal on Twitter. On the surface, a useful library for turning unformatted addresses into structured data. Useful, but nothing out of the ordinary. But on a closer look:

libpostal's international address parser uses machine learning (Conditional Random Fields) and is trained on over 1 billion addresses in every inhabited country on Earth.

Increasingly, just chunking loads and loads of data at a problem becomes an option. Why bother writing a parser to handle every condition and edge case if you can just throw all the known address data in the world at it and let the computer work it out? The writeups on how it works are interesting and thorough.

Statistical NLP on OpenStreetMap: Part 1, Part 2

libpostal screenshot

3. Mapzen funded libpostal, and loads of other brilliant geodata projects. Sadly they're shutting down, but because it's all open source the tens of £millions of value won't be lost. Their migration guide walks through the alternatives to all their hosted services.

Aaron's post about Who's On First, the gazetteer of places, is a brilliant example of how to ensure an important project lasts beyond the lifespan of a single organisation or corporate strategy. Lots of loosely joined pieces, many easy to self or community host. Designing your project to be defended against the organisation that hosts it means designing that in from day one.

In many ways everything about the way Who’s On First has been designed has been done with this day in mind. We all endeavour to achieve the sort of “escape velocity” that immunizes us from circumstance but that is rare indeed and there was always a chance this day would come. So while “success” was the goal in many ways preventing what I call “the reset to zero” has always been of equal importance.

Who's On First, Chapter 2

DRAWING, THE CHROMO CUBE, 1980

4. I think spotted this on FaveJet probably via Russell.

David Rudnick's beautiful drawings of MiniDiscs. Drawings. In Photoshop.

An endless journey through an imaginary England

I turned my silly post about generating village names using a neural network into @urnowentering, a silly Twitter bot. It's the gift that keeps on giving, procedurally, forever.

Generating English village names with neural networks

Roll 13/02

I'm trying to wrap my head around the new generation machine learning tools: deep neural networks and the like. It feels like this technology is approaching where databases were 20-30 years ago: the tooling is getting easy enough that an idiot like me can have a stab at wiring something up, even if I don't quite understand all the magic incantations that I need to type. And it's pretty clear it's going to be important.

The world seems to be settling on Tensorflow, for now, so I had a go at getting something stupid up and running. I ended up making an English village name generator, using a corpus from OS Open Names (with a healthy amount of awk and grep), and a character level recurrent neural network written in Tensorflow.

Like I was with SQL many moons ago, I think I understand some of the principles, what's possible and what's not, and I can make sense of someone else's code - but it's a bit of struggle getting all the words in the right order when I have to change anything.

Anyway, the results are quite fun - here's 20 of them:

Allers
Bottom
Culack
Swrarby
Fenwall
St Eastake
Anbarth
St Ninhope
Thawkanham Water
Green Mige Lane
Up Maling
Firley
Dinch
Lindlemere
Stan Hill
Hiddlesley
Pibley
Hunmastreet
Shenworth
Strough
Hendrelds Hill
Scottedane
Crickines
Stranal
Footh

And here's another 980.

Update: I turned this into a Twitter bot, @urnowentering.

FaveJet

I've used Jason Kottke's Stellar for a couple of years now, and love it. If you've never tried it, it's a stream of things your friends are favouriting, mostly on Twitter, but YouTube, Vimeo and other stuff too.

Favourites are quieter than retweets, and more nuanced. A little nod of acknowledgement, a gentle arm around a shoulder, or sometimes just a smile.

Stellar surfaced those in a way that always felt like a superpower: the ability to see a little further, beyond my immediate network, slicing through the fog.

And so when it went offline a couple of months ago, Twitter got a little less enjoyable for me.

To plug the gap, I've made a little thing called FaveJet, and you can have a go if you like.

FaveJet

Running this type of thing is hard. Pulling down everyone's favourites every 30 mins and reconciling them into an activity stream is intensive as websites go. It seems fine for the 40 people using it at the moment, but it'll look quite different at a thousand or five thousand.

But I'd like anyone to use it, and I don't want it to be beholden to the whims of me and my rare free time. I'd like to find a model to support it, that doesn't involve emptying my pockets alone, or losing precious family time.

Maybe that looks like a company with some people paying and using those proceeds to pay for work on it, or maybe that looks like some kind of co-ownership/co-op model, splitting the work between a few people. Or maybe a mix.

I'm not sure, but I'd like to find something that works, and hopefully an approach I can use to support other niche products in the future.

If you've got any ideas, you'd like to help or you've tried this in the past, please drop me a line.

Anyway, in the meantime, feel free to give it a go. It only works for Twitter at the moment, and I've capped sign ups to 20 per day for now. Happy faveing.

Galloway

Went to Galloway a few weekends ago. Stayed in a treehouse. Typical. Amazing light though.

08:17

08:19

08:29

09:57

VEL-O-DROME

I went to the Six Day at the Olympic Velodrome this week. Great atmosphere, lots of fun.

They've gone full Wipeout with it all. A DJ from Ministry of Sound watching the action and keeping tempo accordingly. Coloured flood lighting. Introducing all the riders one by one as they do laps. Very little hanging around between events, and you mostly knew what was going on.

And whoever had the Derny riders come out to Ride of the Valkyries deserves a bonus.

Velodrome

Derny Racing

Camera

Hey Thermostat

Alright then, that works.

Siri

I promise I'm never blogging about my thermostat ever again.

Not a Nest

Raspberry Pi Wireless Thermostat

As I wrote before, I've been hacking around with my wireless thermostat. Over the weekend I spent a couple of hours throwing together some code to act as a thermostat control loop, running from a Raspberry Pi. I've shared the code on GitHub.

There's two bits: a little C executable using Wiring Pi, that sends the commands to the boiler, and a Go executable that watches the temperature from an attached digital thermometer (a DS18B20) and triggers the thing that sends the commands. (The whole thing was Go for a while, but Go is too unreliable for timing the transmission, so it seemed easiest to move that bit to Wiring Pi in C.)

To put all this on a schedule, I'm using cron to write the desired temperature into a file that's being read every few seconds. If the monitored temperature goes 0.5°C over the desired temperature, the boiler turns off. And if it goes 0.5°C under, it turns on. It's a simple little system, but it's been running for a couple of days now, and hasn't set the house on fire yet.

The thermostat is now marginally less convenient than it was when I started, so the next step is to do what I promised with a shared calendar (not everyone in the family wants to SSH in to turn the heating on).

And it's not done until you've put it in a nice box and it doesn't look like an improvised explosive device. I've always enjoyed watching Tom put things in boxes, so it might be time to get some advice.

For a stretch goal, I'd like to poke around with HomeKit integration: "Hey Siri, turn the heating on", etc. And actually, thinking about it, that's something you can't do with a Nest.

Electric Motors

Electric Motors

For Matt, from The Secret Life of the Home at the Science Museum. More about the making of the exhibition.

Reverse engineering a wireless thermostat

We moved the boiler in our house a few months ago, and replaced the wired, rotary thermostat in the process, with a 7 day digital wireless one. Or rather, the builders did, and I didn't really pay attention.

The timers are pretty simple - a morning schedule, an early evening schedule and a late evening schedule - but we're often in and out at different times, and the UI is so terrible and the buttons so horrible to press, it's preferable to just sit in the cold and shiver.

Anyway, I wanted to see if I could control the heating from something smarter. I've not quite done that yet, but I've managed to reverse engineer the signalling, and I thought it might be useful to write down how I did it.

We've got a Tower RFWRT thermostat - it's pretty cheap. There's two parts: the heating control unit, connected to the boiler, and a wireless thermostat you can place anywhere within range. These communicate over 433MHz, an unlicensed band which is pretty common for household devices.

The thermostat sends one of two command: turn on or turn off. It sends the current command on a 30s cycle, and when the state is changed. If the on command isn't received for a certain period (the thermostat has gone out of range, run out of battery, etc.) the heating control unit will switch the boiler off as a safety measure.

If we can find out what these signals are we can send them ourselves, and eventually replace the wireless thermostat with something smarter. (Or, y'know, you can just buy a Nest or something. You can stop reading here.)

There's a few ways to receive 433MHz signals, but the easiest for me was to use an RTL-SDR USB digital TV receiver. These are about £10, and let you listen across a wide range of frequencies (not just TV/radio), up to about 1700MHz.

I used CubicSDR, a Mac SDR application, and pressed the temperature controls up and down watching for the on/off commands. Sure enough, I spotted a short blip whenever the desired temperature is set.

CubicSDR

Using Audio Hijack (Soundflower works too), I recorded both commands to an audio file, trimmed the silence, and put the two segments side by side to compare.

Waveform

It looks like the signal is repeated twice, probably to reduce the likelihood of interference.

You can see a preamble at the beginning - a long high and long low, four times - probably designed to let the receiver calibrate its automatic gain control. These receivers crank up the volume (and the noise) during silence, and then adjust it when they hear a signal. The preamble gives the gain control a chance to respond so the beginning of the data isn't lost in the noise.

After that we're into the command data. I spent a long time staring at this, initially thinking it was Manchester encoded, before realising I needed to switch Audacity into ‘Waveform (dB)’ mode to see what's really going on. This is actually pulse width modulation, with a 0 sent as a short high and a long low, and a 1 sent as a long high and a short low.

Waveform

I measured each short section as 250μs long, and each long section as 500μs, and each preamble section as 1000μs.

The off command is just a run of 25 zeros, while the on command sends a few 1s too. It doesn't really matter what it means — I just need to send the same thing.

I took a cheap 433MHz transmitter and soldered a coiled 17cm antenna on. I attached this to the digital out of an Ardiuno, and wrote a little script to perform the transmission.

#define transmitPin 4 // Digital Pin 4

// on
bool data[] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0 };
// off
// bool data[] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };

int shortDelay = 250;
int longDelay = 500;
int preambleDelay = 1000;

void setup() {
Serial.begin(9600);
pinMode(transmitPin, OUTPUT);
delay(3000);
}

void loop() {
sendPreamble();
sendData();
sendPreamble();
sendData();
delay(30000);
}

void sendPreamble() {
for (int i = 0; i < 4; i++) {
digitalWrite(transmitPin, HIGH);
delayMicroseconds(preambleDelay);
digitalWrite(transmitPin, LOW);
delayMicroseconds(preambleDelay);
}
}

void sendData() {
for (int i = 0; i < sizeof(data); i++) {
bool b = data[i];
if (b == 1) {
digitalWrite(transmitPin, HIGH);
delayMicroseconds(longDelay);
digitalWrite(transmitPin, LOW);
delayMicroseconds(shortDelay);
} else {
digitalWrite(transmitPin, HIGH);
delayMicroseconds(shortDelay);
digitalWrite(transmitPin, LOW);
delayMicroseconds(longDelay);
}
}
}

A little click, the boiler goes on, and we're in business. To start with, I'd like to schedule the timing from a shared family calendar, so now I need to make a thermostat control loop, and drive the whole thing from something web connected, like a Raspberry Pi.