This is a story from the past. A story of excitement, joy, frustration, denial, and acceptance. A story pulled from the depths of my notes archive, a draft left unfinished almost two years ago, discovered through my journey in notetaking. This is a true story, slightly skewed by time.

The story ends about two years ago, and begins far before that. It informed my decisions about my thoughts around my homelab, my hobbies, my time, and myself. It is not a deep story, and indeed it is technical in nature, but the conclusions have ripple effects which reached my entire homelab and beyond. The morale of this story resonates deeply with me, and while I have always believed what this story has taught me, it gives a new appreciation to the way I choose to run my home network.

This is the story of my experience with OPNSense.

A Sense for Open Source

OPNSense is firewall and router software built on FreeBSD. It’s open source, of course. It is designed primarily as a firewall and has features such as an IDS - Intrustion Detection System - built right in. It can be used as your router, provided you have the hardware to install it on, and it isn’t too hard to configure.

I had decided I needed better software support at the edge of my network, and OPNSense seemed to fit the bill. I had a few desires for what I was looking for, all of which OPNSense seemed to promise on:

  • A way for my then-Kubernetes ingress to communicate with the router to generate DNS entries.
  • A network-wide Wireguard VPN egress set up.
  • Better control over my network VLANs and exactly where network traffic was flowing from and to.
  • Network-wide DNS-level adblocking, with the ability to prevent devices from using their own, hard-coded DNS servers.

I had debated between pfSense and OPNSense for a while, but after seeing the direction the pfSense project was going - not good - and the drama in the community on both sides, I decided OPNSense was the better choice. pfSense was still the community favorite at the time, but it looked to me like tides were shifting. To my credit, that is true–OPNSense remains the preferred router software today.

Having researched my options and having selected my choice, I looked around for hardware I already owned. I found Nadeko - an Intel NUC, just a few years old at the time, which was decently small but powerful enough for what I needed. The only problem is that it only had one NIC. A few web searches later, and I discovered I could add a second 10/100/1000 NIC through a little pigtail attachment that slotted into the M.2 slot that the WiFi card was slotted into. A quick prime-delivery later, and we would good to go.

As I was installing the card, I should have known from the start this was not going to be a good choice. There is no where to pull through the pigtail on the system, and I don’t want to drill into it–there’s nowhere to do that safely! After an hour or two of messing with it, trying this and that, and just general frustration, I decided in the end to just…have the system live upside-down with the pigtail sticking out from the underside–now the topside–of the system for eternity. Little did I know what this little pigtail had in store for me.

Let’s take a step back and consider why I chose the NUC. If you’ve been keeping up with earlier posts, you’ll know I had a relatively beefy (for the time) Intel Xeon-based server, and indeed it had two gigabit ethernet ports on it. Why did I not just use this? Simple: separation of domains. I did not want a single machine to become the end-all be-all of my network, instead spreading out the failure domain. I did not want to deal with the software concerns of separating the router–which has a direct, unguarded connection to the outside internet–from the sensitive Kubernetes services running on the machine. These are not insurmountable problems, but I was already sensing the frustration that would come from combining it all, so I decided instead to keep things separate, to make my life easier. Alas, life is not always so kind.

A Sense for Frustration

Things were fine at first. Great, even. I was able to set things up exactly how I wanted them to be, or at least as close as I could reasonably get. DNS was blocked network-wide, including DoT, so anything that wanted to hit any DNS server had to go through Nadeko. It had the source of truth for all of my DNS names - I had to update it manually and it didn’t handle wildcards, but that’s okay, something I can deal with. I could set up specific routes for specific interfaces, and have certain machines running through an always-on network-wide VPN, and allow other systems, like my work machine to bypass it! It was great!

Over time, it started to get a little annoying to keep up with. I was using Ansible to maintain everything, but OPNSense can only be configured through the UI–technically everything is stored in config, but it’s a single giant XML file which is difficult to parse, not exactly config-management friendly. Every time I wanted to make a minor change I needed to log into the interface and tweak something around. The interface isn’t exactly straightforward, either–clearly this is a firewall first and a router a distant second. I hope you understand the fine details of everything you’re doing too, including exactly what the developers meant for every single label in the UI, because documentation was not grand.

Then the troubles started. My network would stop working every once in a while. I would, of course, blame Comcast because Comcast is almost always to blame–and to be fair, they were the problem about 50% of the time.

But the other 50%?

Might be one interface deciding it doesn’t want to be the internet-facing one anymore, so I need to swap them around. This happened more often then I would like and took at least a couple of hours to do. OPNSense isn’t great at dealing with moving interfaces, but it turns out that little pigtail wasn’t the best of ethernet adapters either. It lied and was only 10/100, and was super unreliable.

Or maybe there’s a weird issue with the software. Try updating! Oh, that made it worse. Can’t roll back now, gotta figure out how to reconfigure things. Again. Oh, a feature I depended on was moved and the configuration I have doesn’t work anymore? Fun, guess I’ll just never update my internet-facing, highly-suceptiable gateway to the world. Surely that’s not a security risk, right?

There’s gotta be a better way.

A Sense for The Past

Interestingly, there was one component of my network that was always rock-stable throughout all of this. My WiFi. It was running on my old ASUS router, I forget exactly which model now. I, of course, had it running Merlin and it was in a strict AP-mode: no DNS, no DHCP, no routing, nothing but a dumb WiFi access point (and sometimes wired switch). I looked at that router. I thought about my frustrations with OPNSense.

I bought an ASUS mesh router, which has been perfectly stable ever sense, and I haven’t looked back.

The new ASUS mesh works with the old router, so I didn’t need to replace it right away–it just became part of the mesh, though it was eventually retired. But what about my requirements from before? Well, it turns out that ASUS hadn’t been sleeping in the meantime, and I had other, better, approaches I could take where it still wouldn’t fit the bill:

  • DNS would be handled on a Pi-Hole running on an actual Raspberry Pi. Separate DNS from the router, and you get better ad-blocking and actual statistics, along with ways to turn on and off the adblocking easily, even for specific clients.
  • ASUS supports Wireguard VPN client mode now, for the whole network, with specific clients being able to bypass it. It’s way easier to configure then OPNSense too, considering wireguard is a third-party plugin there.
  • If I really want to, I can set up VLANs. Turns out, for me, they’re not really necessary. It’s good to sequester smart stuff on it’s own network, but it turns out that causes a lot of them to be unreliable, so I stopped doing that fairly quickly. But if I ever change my mind, ASUS supports that, anyway.
  • The only thing I’m missing is preventing clients from using their own DNS. While this is truly annoying, it’s not actually that big of a deal anymore, anyway. Any device that absolutely wants to use it’s own DNS is going to use hard-coded DoH by now, which is indistinguishable from HTTPS traffic. Is that something I can block? Technically, yes. Do I want to go through that much effort? Absolutely not.

When I made that final realization–I don’t want to go through that much effort–I had a realization about my entire homelab. I had forgotten a very, very important rule.

A Sense for The Simple

There’s a famous acronym out there: KISS. “Keep it simple, stupid.” Indeed, there is a limit to this rule, but “the simplest solution is usually the best,” “don’t add complexity where it isn’t needed,” “the most reliable things, and the most complicated, are often the simplest” and other variations get similar ideas across in better way. That is the rule I had forgotten, or rather, thought I was following but was not: make things easy to work with.

There is a balance to be had between doing everything yourself, the purist form of self-hosting, and understanding that time is limited and the more things you do yourself the greater chance of things breaking.

OPNSense is a complicated mess of a system. It isn’t configurable via config management, at least not really, and that presents an issue for long-term management of a complicated system. It does not integrate well with the greater self-hosted landscape. It’s open source, which means things move and break. A lot.

Meanwhile, the ASUS mesh is built by a company in China and runs an operating system that I don’t have much control over (though it turns out it’s mostly just Tomato under the hood). But it works, and it’s reliable, and I haven’t had to mess with it too much since I put it in. It’s just there, and that’s what I really want from my network–I want to run services that are useful and improve my life, I don’t want to work on those services all the time. I do that enough in my day job, that isn’t what my hobby is. My homelab exists to enhance my hobbies, and to provide me some level of control of my digital life and my data.

But if I can buy or download something off the shelf that does that, reasonably well, for a reasonable amount of money (which means no subscriptions), that doesn’t obviously invade too much on my privacy? Well, I think that’s something I can pick up and use.

A Sense for The Future

Just as a minor postscript, I don’t believe I’ll be living in ASUS land forever. To be honest, their hardware is overpriced for the power it provides, and while it does everything I need it’s still a bit painful to use. The mesh network works fine, but is much better if you can wire each node, at which point you don’t need a fancy mesh system–just a bunch of access points and some good configuration.

My next step is probably going to be into the Ubiquiti range of goods, which very much targets a prosumer target demographic. While UniFI gives me the heebie-jeebies, it seems most folks in the space today still swear by their products. I haven’t made a commitment yet–I’m still several years away from even needing to care–so I can easily change my mind if I need to when I start doing actual product research.

But suffice to say I think I’m done running my own stack. It works, but the amount of time, effort, and frustration put into it just doesn’t really pay out. That’s something I’m finding about my homelab in general–and that’s why I’m moving the whole thing into a tiny little Apple Mac Mini.

The moral of the story, as it turns out, is that compromise is important and simple is better then complex. Who knew.