My homelab is 10 years old. Almost. Well, that’s not true. I’ve had a homelab for much longer. My current server is nearly 10 years old, or at least one of them is.

Whatever, the point is that it’s time for me to migrate my homelab onto more modern hardware, and with M4 Macs coming down the pipe soon, I thought it might be a good time to start to consider what I have and where I’m going.

And hey, making this a multipart series might kick me into writing on this blog more than once a year. Who knows!

As a side game, bonus points if you can identify every character and every show I named each of my systems after. There are obscure ones in there! If I remember, I’ll include the answers in Part One.

The Tech Stack of Yesterday, Today

My current homelab isn’t as exciting as others you may see online. I currently run 2 main servers, with a couple of small other systems lying around here and there. Nothing is open to the internet, as I run everything through Tailscale.

The key thing that needs to be replaced is the main server, Chiyuki. Chiyuki is a home-built server with a Supermicro X9SRA motherboard holding an Intel Xeon E5-2670 CPU and 32 GB of RAM. The processor has been discontinued since 2015 and is out of support since 2020, but it continues to work perfectly fine. I saw no need to update until now.

Chiyuki has 8 drives acting as a NAS with an additional SSD as the boot drive. I literally just replaced most of these drives around a month ago, so I’ll be keeping them around — we’ll talk about that later. The NAS drives are 8TB 7200 RPM spinning disks, configured in a BTRFS RAID 10. It works…fine. I’d like to use RAID 6 here, but in BTRFS that’s still not considered production-ready, so RAID 10 it is.

Why not ZFS? When I built the precursor to this system, Claudia, it was unclear if ZFS or BTRFS would become the next-generation filesystem of choice on Linux, mostly due to the licensing surrounding ZFS. I chose BTRFS, which is a fine filesystem, but ultimately ended up being far less useful than ZFS would have been long-term. I don’t even make full use of BTRFS (or ZFS) features, so really I should have gone with something simpler. Even so, I wanted to learn BTRFS, so I did.

Chiyuki is the star of the show for this upgrade, but let’s talk about other components that could potentially be replaced here, starting with Tuuli. Tuuli is a system that was gifted to me when a friend moved to Australia and needed to lighten their load as much as possible. Tuuli is built with a Supermicro X7DCA holding two Intel Xeon E5450 processors and 11 GB of RAM. This machine is basically an antique at this point, and really is only running to provide additional compute for my Kubernetes cluster.

Potentially up for retirement also include a Raspberry Pi named Ro which does nothing but run Pi-hole and DNS for my network, an ASUS Tinker Board named Nano which only runs Homebridge, and an ancient Intel NUC, the exact model of which I’m not going to bother finding, which only runs AirPrint. That’s it, that’s all it does, nothing else. On top of that, there’s a VPS in the mix which runs my Mastodon instance (and is almost always down) and this blog, along with a couple of other minor things. The VPS might end up being retired, but this blog won’t migrate to my server. I’ll probably find a different, serverless, solution for it instead. Maybe in a future installment, though it’s a bit outside the scope of the homelab.

That’s a lot of old hardware, which means a lot of power. The biggest goal of this project is to reduce my power consumption dramatically, even if the financial difference in the end doesn’t actually work out. The second-biggest goal is to simplify the software setup.

The Softer Side of My Homelab

My software stack is kind of all over the place, owing largely to years upon years of cruft and experimentation. Let’s explore!

Starting with the main servers, Chiyuki and Tuuli, these are both running Ubuntu Server 22.04 LTS. It would be time to upgrade soon if I wasn’t replacing them anyway. They’re in a Kubernetes cluster built using k3s, and run a lot of software as containers including GitLab, Plex, Homebox, and a few other things. One server wasn’t enough (mostly because of GitLab and Plex), so Tuuli was brought in to help take some of the load. I shared the storage via NFS, because when I built my original Kubernetes cluster there was really a limited amount of ways to freely and easily make storage available to pods in Kubernetes. I built my original Kubernetes cluster from scratch, but dealing with the project’s updates and breakage got quite time-consuming, so I opted to use k3s instead.

This setup works perfectly fine until it doesn’t, and fails catastrophically. When a drive fails, for example, the system will start to get exceedingly slow with no real indication that a drive has failed until a SMART test runs and notifies me. This causes all sorts of things to behave strangely, especially anything using NFS. Eventually, when I reboot the system out of frustration, it inevitably fails to come up, and I have to physically attach a keyboard and mouse to the machine to mount the BTRFS partition in a degraded way to boot. This causes any containers that were running on Tuuli to freak out, and so I reboot Tuuli, but that has a chance of just causing all the containers to enter an “Unknown” state. Solving this is a frustrating process of manually killing pods and restarting the k3s process.

Basically, this setup kinda sucks.

What’s Mac Got to Do With It, Got to Do With It

My targeted solution for this is (drumroll please)…a new Mac Mini. Wait, wait, wait! Don’t leave!

Apple should be releasing a new M4 Mac Mini next week, if the rumors are to be believed. It should have a smaller footprint and a bunch of Thunderbolt ports, which is really all I care about (plus 2.5GbE minimum). Provided it does release as rumored, this should be a good system to replace everything I do with my current machines.

Now, it is still a Mac, not a server. Even so, the Unix-like environment of Darwin should be more than enough to facilitate everything I need to do. For Linux, I can use something like UTM (or maybe Proxmox, pretty please?) to run VMs whenever I need them. I would love to actually pull the software out into micro VMs running on the Mac Mini as a hypervisor.

The actual software situation I’m still researching. However, the key points are:

  • Apple Silicon Macs use very little power for a good bit of performance
  • All of my workloads should work on Apple Silicon, potentially in containers, with no problem
  • One decently-powered Mac Mini (M4 Pro maybe?) could easily take every workload I have running on 5 different systems now simultaneously
  • As I run a MacBook Pro as my daily driver, administering a Mac Mini will be pretty easy.

Can I simplify many of my problems by going with a new, basic, Linux server rig? Yes, but it would be challenging to get the same performance in a similar power budget. I also foresee a Linux system turning into the bloated mess my current environment has become, as the past always repeats itself. I like my homelab hobby, but I want to simplify it significantly, and a change in architecture seems like a good point to do so.

NASty Problems to Solve

There’s one key point that’s still a challenge to solve, however.

The disks.

This is going to be a particular challenge, because I plan to reuse the 8 TB drives that are currently running in Chiyuki. There are two problems with this:

  1. Mac Minis are, famously, mini. They do not have room for additional spinning drives, let alone 8 of them. Especially if, as rumored, the M4 minis will be about the size of an Apple TV.
  2. macOS does not have any BTRFS support. There’s no third-party support for the filesystem, either, as far as I can tell. This means I can’t just drop the drives into a new machine, but with the same filesystem. I have to start over.

This is a surprisingly challenging thing to solve, because I have 14 TB of data. For problem two, my current plan is to find a very, very large external hard drive and copy the entirety of the file system to it, reuse the drives, then copy it back. This is concerning because it now creates a dangerous single point of failure. I do have offsite backups, but using those are a pain. I would like to avoid it if possible. Perhaps I’ll purchase two USB Hard Drives just to avoid any issues, but that gets rather expensive, so I would like to avoid it. I’m continuing to research this problem, so I’ll explain my solution in Part 1.

That’s problem two. Problem one is more pressing though — it doesn’t matter what drives I want to use if I can’t physically use them. I have two possible solutions here, and I don’t like either one.

The first is to find an 8-bay Thunderbolt RAID-compatible DAS box. These are basically USB hard drive enclosures that support RAID and attach to the host machine via Thunderbolt. My key requirements — and some that are really hard to find — are SMART scanning of all the disks, and reporting the serial numbers on each one. I don’t really care if the DAS presents as a single filesystem or as individual drives. I can figure out RAID on the Mac if I need to, though by default I think Disk Utility only supports RAID 0, 1, and JBOD. The only things I really need are SMART and serials.

The second is to just…buy a separate 8-bay NAS, dedicated to serving files. I don’t love this solution because some services, like Plex or Nextcloud, really want to have everything on local storage. It also means I will be using some extra power that I would otherwise not need, if only on the order of 20 watts or so. Still, this is an easier (though still not easy) solution, so it may be the way I ultimately end up going. We’ll see.

I suppose there is a third option, which is to just give up on my otherwise brand-new disks entirely. I could spend an excessive amount of money, and buy a series of NVMe drives and an external NVMe enclosure to hold them, then configure them in a RAID. MCTK has already done this, and it seems to work fairly well, but I’d like to continue using the disks I have already purchased. I don’t have infinite money, after all.

In any case, there are still more unanswered questions.

The Questions Unanswered Are The Devilish Details

The biggest questions I still need to solve, outside how to attach the drives, surround what software I am actually going to run on this thing. I’m not going to migrate my entire stack. This is what I know:

  • I want to run Linux VMs if possible.
  • I will not be running Kubernetes. It was great to learn with, but the complexity is wholly unnecessary.
  • I will be migrating GitLab to Forgejo, Gitea, or something else. I’ll also be migrating from GitLab-CI to Woodpecker, probably. I may also consider moving my code to GitHub instead.
  • Containers are still how I want to deploy most services, this might be on the Mac itself or through VMs.
  • I haven’t decided on what config management to use yet, but I will be setting one up. Again. I’m thinking Ansible, Nix, or maybe I go way out of the box and do Terraform/OpenTofu.
  • I would like to move from bare rsync to a proper system for backups. Probably Restic? Time Machine is not an option here.

Everything else is currently up in the air, and I’m working out a plan. My next post will hopefully include both the basic software plan and my solution to attaching disks to the Mac.

For now, though, it’s time to get back to the drawing board.