Raspberry Pi Multitool

it can do night vision! and thermal!

<picture of night vision regular, night vision, and thermal overlay>

Read on if you want to know more about how (and what) I built in this project!

Background

The inspiration for this project actually started quite modestly and without any custom hardware aspirations - I found this tutorial¹ online for building a thermal camera overlay with a Raspberry Pi and I wanted to try it out for myself.

The more I got into it though, the more ambitious my aims became.

What if I could have it do license plate recognition too?

What if I wanted to add night vision?

Well, if I’m adding IR LEDs, why not add an IR receiver too?

The scope of the project and my latent desire to see how far frontier LLMs could take me on hardware design/builds helped me land on this:

A Raspberry Pi custom headerboard containing IR LEDs (for night vision and remote replay), an IR receiver (for capture remote codes), and a StemmaQT type connector (for more easily attaching the thermal sensor). These components, combined with both a standard and NoIR camera (the Pi 5 supports dual cameras) would then round out the sensor array for my multitool.

Getting Started with Hardware

I’ll be honest, the last time I designed a circuit board was in 2017. The entire landscape has changed dramatically (not to mention my skills have atrophied significantly). I was surprised though at how readily several of the concepts came back to me - particularly with the in-depth explanations from Claude. PWM. Pull-down resistors. Decoupling capacitors. EEPROM. All terms that I’ve given very little headspace in the almost-decade since I left college. I was surprised at the level of joy that reintroduction brought.

I started by having Claude generate a Kicad schematic. It was…not great, but it had several of the necessary components and the circuit was coherent. That was the first general lesson: it’s great at theory but struggled with raw implementation.

After cleaning up the schematic quite a bit and running a few simulations, here’s what I came up with:

Then it was on to PCB design and routing. There is an existing Pi header PCB outline already published, which was quite helpful. In reading up on PCB design I saw several folks reiterate the idea that it’s 90% component placement and 10% routing.

Being an impatient lad though, I clipped through the component placement rather quickly and paid for it later by having to constantly adjust footprints as well as doing some wacky routing right at the end.

I wish I’d taken more “work in progress” pictures. It was a labor of love in many ways, but routing is one big puzzle, and that aspect really appealed to me.

I also challenged myself a bit by having a fairly decent sized cutout in the board to allow for the camera ribbon cables to feed through, as well as to help with cooling. This reduction in real estate forced me to redesign the board a few times but oftentimes those types of constraints can force you to get creative in ways that you wouldn’t normally otherwise have to. In the end I was able to get it all to fit.

Once all the pieces were in place it was a constant back-and-forth of running the Design Rules Check and cleaning up the errors and warnings. I had forgotten to do a groundpour which solved a lot of my problems and there were a few issues here and there with footprints being too close together and other stuff.

Kicad has a cool 3D renderer at this stage:

Then, it was ready fabrication! I had zero desire to hand-solder tiny surface mount components, so I opted for the full service fabrication (still rather cheap all things considered). This is the part where Claude really shined: I told it to build me the component BOM and cross check it with real suppliers and make a unified list of component part numbers with their suppliers. BAM. First shot it gave me a tidy list with all the component numbers from actual suppliers that was accepted by the vendor. This would have taken me at least an hour or two of muddling through things had I gone at it alone but it was done in 5 min.

As an aside, does anyone know of a US-based company that does full PCB prototyping and fabrication? Seems like all the major players are in China.

Anyway, away it went for fabrication and I sat back and hoped that I hadn’t made some dumb mistake that would render my board useless upon arrival.

One final rabbit trail on the hardware: the board includes 4 slots for IR LEDs to provide night vision illumination (and remote code replay). For IR LEDs you have a choice of two wavelengths: 850nm and 940nm. 850 has better range and brightness for a night vision camera but there’s a tradeoff: it lets off a faint red glow in the dark. It’s kinda creepy looking and alerts you to their presence ². For that reason I chose 940s for this project so that it would truly be undetectable.

Software and General Project Details

While the PCB was being fabricated, I started working on the software side of things.

At a high level, the Raspberry Pi runs a FastAPI server which the user connects to via websockets. The server registers all of the various sensors and cameras as singleton instances upon startup, allowing them to be individually added to the sensor registry.

At this point, the only sensor I had was the NoIR camera. I got it wired up and registered and started to play around with basic object detection, followed by license plate detection. However when I took the Pi over to the window to test it on my car’s plate…the screen was utterly blinding. Totally whited out. Which makes sense given that the entire point of this camera is that it has no IR filter…meaning it gets completely blinded by ambient daylight.

When I was first working through the design of this project I thought about trying to get a camera with a motorized IR cut filter (so that I could toggle between day/night mode with a single camera³), but I already had a standard Pi camera, so just upgrading to a Pi 5 which supports dual cameras felt like the smarter move. In hindsight this was a serendipitous decision because the price of a 16GB RPi 5 went from $200 to $300 a few weeks later due to the exploding cost of RAM. Ooof.

At this point I’d also gotten my hands on a thermal sensor, so I was able to being experiment with not only using multiple cameras, but also layering on the thermal sensor visualization.

I used the widely available MLX90640 thermal sensor for this project. The thing about thermal sensors is due to the underlying physics and their implications on cost, the sensors are rather low resolution (this one is 32x24 for a grand total of 768 pixels). However, they can be quite sensitive - registering to within .1 degree C per pixel. Here’s the actual raw output of the sensor:

Laughably small right? The aforementioned sensitivity of the pixels allows us to get creative on the software side though. First, we can “expand” the image to achieve much larger resolution. OpenCV has several algorithms for doing this - here’s a side-by-side comparison of some popular ones:

The tradeoff here is computational complexity, which roughly increases as you move from left-to-right. Now that we have a larger image we can run a smoothing algorithm to that the image looks “cleaner” and adjust any block-i-ness of the pixels.

In reality, most of the heavy lifting is done in the interpolation phase as the expansion smooths the gradients to begin with. So a 5x5 Gaussian blur for example is just mostly nudging pixels that already more or less look the same.

At this point, the UI buildout is highly customizable relative to the specific needs of the operator and the ways in which they would like to visualize this data. For example, my home screen provides both the camera inputs as options, along with any overlays the user desires (thermal, basic objects, license plates etc.)