Example 02 · RNN

Gesture Recognition

Gesture control for wearables and remotes, running on the device’s own microcontroller. Our 388-parameter network reads the accelerometer one sample at a time and tells a shake from a tilt, keeping its whole memory of the motion in just 16 numbers.

Read the write-up on GitHub

A watch on a wrist being shaken or tilted, and a detail of the board inside: a microcontroller wired to a 3-axis accelerometer whose x, y and z axes are drawn as arrows. — Inside a wrist-worn device: each new x, y, z reading from the accelerometer goes straight into the network.

parameters in the whole network: 388
freshly generated test gestures classified correctly: 200 of 200
to call every test gesture correctly, sampling at 100 Hz: 80 ms
for the whole model in 8-bit form, still 200 of 200 correct: 528 bytes

Where it's used

Gesture control, built into the product

Small devices use gestures all the time, and each one needs a tiny classifier that runs nonstop on the device’s microcontroller, reading a live stream from its accelerometer.

A single reading can’t tell a shake from a tilt, or from a hand holding still. What sets them apart is how the motion unfolds over time, so this network keeps a running memory as each reading arrives, and it uses the same small amount of memory however long it listens.

A smartwatch that wakes its display at a flick of the wrist
A TV remote that turns a shake or a tilt into a command
A fitness band that starts counting reps when it spots a move

The data

Every gesture is generated fresh

There’s no dataset to download. Each training window is generated on the spot, 32 readings of x, y and z, so the network never sees the same one twice. Every reading gets a little random noise and every shake gets its own speed and timing, so no two examples of a gesture are alike.

Still: just sensor noise, with gravity reading about 1 g on z
Shake: a fast back-and-forth on x and y, about three to five swings per window
Tilt left: x ramps steadily down to −1.2 g as the device rotates
Tilt right: x ramps steadily up to +1.2 g

Four panels of 32 timesteps each. Still: x and y near 0 and z near 1 g. Shake: x and y oscillating between about −1 and +1 g, offset in phase. Tilt left: x falling steadily to about −1.2 g. Tilt right: x rising to about +1.2 g. z stays near 1 g throughout. — One generated window of each gesture: x, y and z across 32 readings.

How we built it

A network with a memory

This network is recurrent. It takes one accelerometer reading per call and carries a hidden state from one call to the next, so it builds up a picture of the gesture while it’s happening. We clear that state at the start of each new gesture, so one never bleeds into the next.

RNN architecture: input of 3 values (x, y, z), into a 16-unit tanh RNN layer with a recurrent loop carrying its hidden state from the previous timestep, into a 4-way softmax output; one timestep per call, and nn_reset_state() zeroes the hidden state between independent gesture windows. — The recurrent network. The loop carries the hidden state from one reading to the next.

Sixteen units, 388 parameters. That’s 64 for the incoming readings, 256 for the loop that carries the hidden state, and 68 for the four-way answer at the end.

Trained in under a second. The network learns from every reading, not just the last, and the training method keeps memory use flat however long a gesture runs. That matters on a microcontroller, where storing a whole sequence to learn from isn’t an option.

Results

Every gesture, within 80 milliseconds

We tested on 200 freshly generated gestures, 50 of each, judged on the network’s answer after the final reading. It got every one right.

Still
50 of 50
Shake
50 of 50
Tilt left
50 of 50
Tilt right
50 of 50

How fast it makes up its mind

With a recurrent network, the interesting question is how soon it knows. Shake is spotted almost from the first reading, since it starts mid-swing, far from rest. Still and the two tilts all begin at rest and only pull apart as the tilt builds, so they take a few readings. Across four separately trained models, every test gesture was called correctly by reading 8 of 32 and stayed that way. At 100 readings a second, that’s 80 ms into the gesture.

Share of test gestures already classified correctly after each reading, for one trained model.

On the device

Room to spare on the smallest microcontrollers

The whole model is 388 parameters, about 1.5 KB, and each reading costs about 370 multiply-adds. That’s light enough to keep up with the accelerometer’s own sample rate, right beside the rest of the firmware.

The firmware loop is simple. Reset the network when a gesture begins, feed it each new reading as the accelerometer delivers it, and act when the gesture ends or as soon as one answer is confident enough. Nothing gets buffered, because the hidden state is the network’s whole memory of the gesture so far.

in the zero-copy format that runs straight out of flash: 1,604 bytes
quantized to 8 bits, and still 200 of 200 correct: 528 bytes
multiply-adds per accelerometer reading: ~370
of state, the network’s entire memory of the gesture: 16 numbers

Get the code and the full write-up on GitHub

Ready to build your next intelligent device?

Let's discuss your project and how Cole Design & Development can bring your idea to life.

Talk to Us Today Explore Our Services

Founded in 2008 · Hudson, Ohio · info@coledd.com