Example 01 · CNN

Character Recognition

Handwriting recognition that runs entirely on a microcontroller. No cloud, no network connection. We trained two neural networks on 70,000 handwritten digits, compared them head to head, and our library runs this same task live on an STM32H7.

Read the write-up on GitHub

Digits 0 through 9 drawn one after another on an STM32H7 touchscreen, each recognized and lit up on the on-screen keypad. — Live on an STM32H7

accuracy on 7,000 handwritten digits it had never seen: 98.24%
parameters, small enough for a microcontroller: 100,254
of weights, and about a quarter of that once quantized to 8 bits: 392 KB
the microcontroller in our live demo: STM32H7

Where it's used

A small image in, an answer out, all on the device

Lots of products need to turn a small picture into an answer on the spot. When the network runs on the device itself, it works without a connection, it answers right away, and the data never has to leave the product.

Reading handwritten digits is a great example of that kind of model, and it’s the same approach behind products like these.

A smart pen digitizing handwritten forms
A point-of-sale device reading a handwritten amount
A meter reader recognizing a dial
A postal sorter reading a ZIP code

A handwriting pad with a stylus and a handwritten 8; the image is scaled and thresholded to a 28 by 28 one-bit grid and passed to an STM32H7 microcontroller running the 100K-parameter CNN, which outputs the digit 8 with its confidence. — How a device puts it to work: the handwriting is scaled down to a 28 × 28 black-and-white image, and the microcontroller works out which digit it is.

The data

70,000 handwritten digits

We trained on MNIST, the classic handwriting benchmark: 70,000 digits written by about 500 different people. Each one is a 28 × 28 image, and we cut every pixel down to a single bit, black or white, to keep the input as small as possible for the microcontroller.

Training · 80%, 56,000 digits: what the networks learn from
Validation · 10%, 7,000 digits: checked after each round of training, so we keep the best model
Test · 10%, 7,000 digits: kept aside until the very end. The 98.24% comes from these.

A grid of real MNIST test digits, fourteen of each digit from 0 to 9, shown binarized in black on white; handwriting styles vary widely, for example open and closed 4s and slanted 1s. — Real digits from the dataset, fourteen of each, exactly as the networks see them.

How we built it

Two networks, one test

Instead of training one network and reporting a number, we built two different kinds, trained them on identical data and graded them with the same test, so the comparison is a fair one.

Convolutional network train_cnn.c

This one scans the image with small 5 × 5 filters that pick out strokes, combines those into shapes, and then hands the result to a few final layers that decide which digit it is. Only 3,424 of its 100,254 parameters sit in the filter layers, but that’s where most of the arithmetic happens, about 880,000 multiply-adds per digit in total.

CNN architecture: input 28 by 28 by 1; 5 by 5 convolution with 8 filters and same padding to 28 by 28 by 8; 2 by 2 max pool to 14 by 14 by 8; 5 by 5 convolution with 16 filters to 14 by 14 by 16; 2 by 2 max pool to 7 by 7 by 16; flatten into a 120-unit GELU layer; 30% dropout; a 20-unit GELU layer; 10-way softmax output. — The convolutional network. Two filtering stages turn the 28 × 28 image into a stack of small feature maps before the final layers make the call.

Fully-connected network train_fc.c

The classic alternative wires every one of the 784 pixels straight into a layer of 128 units, then 64, then the answer. It’s a similar size at 109,386 parameters and needs only about 109,000 multiply-adds per digit, but it has no sense of which pixels sit next to each other. It’s the network a convolutional design has to beat to earn its place.

Fully-connected architecture: 784 flattened input pixels, dense all-to-all into a 128-unit GELU layer, then a 64-unit GELU layer, then a 10-way softmax output. — The fully-connected network. Every pixel connects to every unit in the first layer.

We graded both strictly. A digit only counts as right if the network gives the correct answer at least 50% confidence and no other digit gets that much. That’s a tougher bar than simply taking the network’s top guess.

Results

Fewer mistakes, same memory

Network	Parameters	Multiply-adds per digit	Epochs	Training time	Train accuracy	Test accuracy
Convolutional train_cnn	100,254	~880K	11	8 min 25 s	99.17%	98.24%
Fully connected train_fc	109,386	~109K	10	15 s	99.44%	97.26%

Training times are on a single desktop CPU core. The convolutional network came out ahead in both pairs of runs we measured: 98.24% against 97.26% here, and 98.49% against 96.13% in an earlier pair.

Why it wins

With slightly fewer parameters, the convolutional network makes about a third fewer mistakes on digits it’s never seen: 123 out of 7,000, against 192. The fully-connected network actually scores higher on its own training data, but that’s memorization, and its error on new digits starts climbing after the fifth round of training. Convolutional filters look for strokes wherever they appear, so what the network learns carries over to handwriting it hasn’t met before.

What it takes

The extra accuracy comes from arithmetic, not memory. Each digit takes about eight times the multiply-adds, and training took about 8½ minutes instead of 15 seconds. On a device reading one digit at a time, it still works out to just milliseconds.

Error on the training and validation sets after each round of training. The dot marks the model we kept, the one with the lowest validation error.

Mistakes a person could make

Many of the digits it gets wrong are hard to read for anyone. Think 4s closed at the top that look like 9s, 8s and 9s whose loops run together, and 2s and 3s with a straight tail that pass for 7s.

Twenty-four misclassified test digits, each labeled with its true and predicted class, for example a 4 read as 9 and a 2 read as 4; many are ambiguous or unusually written. — The first 24 test digits it got wrong, labeled actual → predicted. Would you have read them all correctly?

On the device

Small enough for a microcontroller

The trained network needs 392 KB for its weights. An STM32H743 has 2 MB of flash and 1 MB of RAM, so there’s plenty of room left for the rest of the product’s firmware. Our library can also convert the model to 8-bit integers, which cuts it to about a quarter of the size, and store it in a format that runs straight out of flash without copying anything into RAM.

The code on the device is short. It takes the image from the touch panel or camera, scales it to 28 × 28, hands it to the network and reads off the answer. At about 880,000 multiply-adds per digit, that works out to a few milliseconds on a Cortex-M7 running at several hundred megahertz.

The built-in 28 by 28 sample, a blocky 8, beside the CNN's softmax output: 0.989 for 8 and under 0.004 for every other digit. — Shown a sample 8, the network is 98.9% sure it's an 8 and gives every other digit less than half a percent.

Get the code and the full write-up on GitHub

Ready to build your next intelligent device?

Let's discuss your project and how Cole Design & Development can bring your idea to life.

Talk to Us Today Explore Our Services

Founded in 2008 · Hudson, Ohio · info@coledd.com