For a few years now, I've tried as hard as I can to run everything on servers I control, ideally servers I physically control. Partly in an attempt to regain my privacy, party because it's much more fun this way.
Over the last year, things have stayed fairly stable hardware-wise - very little has changed. The largest change was to the CPU, which nicely reduced my power consumption and gained me a GPU. There are a few more smaller changes, and more changes I'd like to be able to do, but that's what this post is here to talk about.
Many of you will have noticed a slow down in content over the last year. There are a few reasons for that, but I'm hoping to improve. In just the last few weeks, I moved house, and that's sparking off a lot of new project ideas and post ideas, so expect the dry spell to be over fairly soon. Therefore here I'm not only spending time looking back at how my server is setup at the end of 2022, but also looking forward to 2023 and how I can improve the gripes I have and improve the stability and reliability of my setup.
If you're here, you're probably nosy about what applications I run to make my life function. Well here they are:
There are a couple other thing things lying around, but none that are worth mentioning, nor will be around for much longer. Most of the applications are run through Docker, but many aren't. Jellyfin for example lives in its own LXC with the iGPU passed through to it because it's easier to manage than a full VM or another containerisation layer. Qbittorrent used to run in a VM for easier VPN connection, but I switched that to an LXC too 6 months ago and haven't noticed a difference (besides the fact that passing through a directory is much much simpler).
As much as it pains me to say, there are also some applications I use which I don't host myself yet. For more about those, take a look at my State of the Apps post.
My storage setup hasn't changed in a while. It's taken the shape of 3 tiers:
tank)bulk)This separation gives me the benefits of ZFS for important data, but the flexibility of snapraid for future expansion. In reality though, I don't need the expansion benefits, and I don't have that much data to benefit from having all the different tiers. By running fewer drives and having only a single "data" pool, I'd only have a single pool to have to deal with, and improve the redundancy on my bulk media pool. The media pool, whilst snapraid, only reads/writes data to a single drive, meaning only a single drive is being degraded.
If data isn't backed up, it may as well not exist. Whilst I'm not quite following the 3-2-1 rule for backups, I'm pretty confident on them.
Whilst RAID shouldn't be considered a backup, as it's still tightly coupled to the original data and system, it's still the first part of my backup strategy. Using sanoid, I capture regular snapshots of my data in ZFS. This means should I mess something up, which is quite likely, atomically rolling back to a previous point is pretty straightforward.
Restic has been absolutely rock solid for the few years I've been running it, and since I've rolled it out in a few more places professionally. I have Restic running in its own dedicated LXC (for permissions and isolation reasons), and backs up everything important to Backblaze. Recently, Restic added support for compression, which given a fair amount of my data is text files, could make a nice difference to backup efficiency. It's going to need some testing, as porting over isn't completely straightforward, but it's something which shows promise.
As currently configured, all my servers point to a single restic repository. This means all my servers have the ability to read and write all of my backups. There's nothing wrong with this necessarily, but it means in the event of a breach, all my backups are at risk of being leaked. In the future, I'd like to update this to use separate repositories, and move all the forget executions onto the relevant servers. This does reduce the efficiency of the backups, as the same file on different servers would be backed up multiple times, but I doubt that makes much practical difference.
The core of my home server is still Proxmox, however reluctant I may be. For a while, I've not been very happy with Proxmox. It's appliance-like nature is great for some, but makes tinkering under the hood quite a pain - it's either the Proxmox way or the highway. Proxmox has served me well so far, but it's time to try something different.
I want something simpler, easier to customize, more lightweight and much easier to debug for me. I want to try something different, but switching is quite a costly process, as I'd have to rebuild most things from scratch (well, semi-scratch). Rebuilding my server last time took me almost 3 days of stress and frantic searching, and I'd rather not go through that again! I think I know where I want to go: LXD. But I need to get more confident with it before I can switch some critical parts of my server to it. Switching to LXD may also force me to run Ubuntu on the host (as LXD is most commonly shipped and supported as a snap), which I'd rather not do, but we'll see.
My servers run a lot of different things, and sometimes they do things I need to know about. Sadly, for a lot of those things, that's happening through Email. It's not ideal, but it's good enough. More on that in a second.
A few notifications I've moved to Matrix. Matrix is definitely overkill, but it's reliable for my needs and notifies pretty quick, and gives me a few more reasons to keep my matrix server running.
Recently, KDE announced a self-hosted system notification service, which could be backed by Gotify. I used to run Gotify for push notifications, but ended up dropping it as its app was a little annoying. Given all I need is push notifications, Gotify could be something interesting to add to my deployment catalogue.
If an application can send notifications, there's a pretty strong chance it either only does this by email, or at least supports sending by email. Some support sending email directly, but that's insecure and often gets caught by spam. Generally, the best practice is to connect the application to a MTA (Mail Transfer Agent) using SMTP.
Previously, I was handling this through my main email service: Fastmail. This works great, as Fastmail allows creating SMTP-only credentials. This prevents them from reading my emails but does allow my applications to impersonate my address and send emails as me. For some, this may be fine, but I'd rather do things properly. For this task, lots of people go to Sendgrid. Sendgrid is a great service from Twilio, but I just didn't quite get on with it and its interface with the albeit little playing around with it I did.
Instead, I settled on Mailgun. I've used Mailgun professionally for a few years, so I already know how to use it and trust it works just fine. It's not obvious, and took me a few emails with their support to work out, but Mailgun does actually have a free tier (although it does require payment details), it's just very poorly documented. Mailgun allows me to provision separate sets of credentials for each application, so rotating becomes nice and simple. Mailgun is setup to only send using a subdomain (mg). If it tries to send email from the root domain, it won't be correctly signed by DKIM, and thanks to my DMARC settings, it will be flagged as spam.
DNS is pretty important on the internet - It's how your browser knows theorangeone.net should be handled by my server. Currently, my domains are registered with Namecheap with DNS handled by Cloudflare.
For the last few months, I've wanted to leave Cloudflare. Not because I've had any issues with their service, Cloudflare has actually been rock solid for the few years I've used it. But Cloudflare makes up a huge portion of the internet and I don't really like the centralisation (among other things). Whilst I use Cloudflare, I use exactly 0 of their primary features: No workers, no CDN, no firewalls, nothing. That means I have little reason to stay with Cloudflare, but also that switching should be trivial.
My top runner right now is Gandi. Gandi are a well-respected company with a feature-complete web dashboard for DNS, an API supported by Lego (the ACME TLS-01 library used by Traefik) and a terraform provider (it's hard to tell how official it is, but it's endorsed by Gandi so it's good enough for me). I'm sure there are people yelling "Porkbun" at their monitors right now, but I trust Gandi, as a company they do good things, and I've used them in the past.
It's important to me that my applications are up and available as much as possible. For that, I monitor them all.
Previously, I ran a dedicated server in Vultr's Germany datacenter for monitoring. For the monitoring itself, I used uptime-kuma, a self-hosted version of UptimeRobot for monitoring different applications. If there were problems with an application, uptime-kuma would send me a notification. If you want monitoring, I highly recommend it. But, I wanted a change.
I've been running Prometheus for a couple years now, mostly out of interest for grabbing metrics for a few services and piping them into Grafana. I've been playing around with it even more professionally, and want to do something more with it. Prometheus is incredibly powerful and flexible, and supports querying and alerting - so I might as well use it.
To save some complexity, I also don't need a dedicated server. the dedicated server meant that if there was a problem with my home server, I would still be alerted. However, I still needed some external monitoring to make sure the monitoring server was working (who monitors the monitors after all). Given I needed to monitor the monitoring server anyway, might as well use my home server for it. Because of my networking setup, I need to make sure the monitoring probes use the public routes, but that's much easier (and cheaper) than a whole server.
Prometheus gets its data using "exporters", a popular one of which is blackbox which monitors external sites and reports on its availability. If a site goes down, Promtheus tells Alertmanager to tell me something's wrong.
For the public monitoring, I'm still using UptimeRobot. I've been using UptimeRobot for several years at this point and I trust their reliability in telling me my sites are down. I mostly use UptimeRobot to monitor the core parts of my network. I can't use UptimeRobot to monitor prometheus itself, for that I do something else: Healthchecks. I use healthchecks to monitor the critical scheduled tasks like backups and drive scrubs and tell me if something either goes wrong or it doesn't run at all.
For a few years I've been a big advocate of, and happy user of, my "VPN gateway" approach to exposing your home server. The idea being that rather than port forwarding and dynamic DNS, you run your own server in the cloud with a static IP, and forward traffic from that down a VPN to your home. I originally designed it a few years back, and it's been working fantastically ever since.
Currently, the way I avoid having traffic from my house hitting said server is by using a tiny DNS server to override the IP of records from the VPS to my server's local IP. Internal-only applications though still just use IP addresses, which doesn't feel very modern. I'd like to improve this, but it's a fairly complex problem which requires me learning some actual networking things. It's likely the solution lies somewhere around IPv6 and a more complex cloud server.
The gateway itself hasn't changed very much. Last year I swapped out HAProxy with nginx as I'm more comfortable configuring it, but also because it's easier to integrate some more interesting features. It's still just pushing packets and adding some PROXY protocol magic, but in future being able to run some extra services right on the server, or adding some edge caching either with nginx or varnish would be another fun rabbit hole to dive down.
I'm also sorry to all you networking nerds to say that my entire network is fully IPv4. And not just IPv4, IPv4 only. Even the public VPS doesn't actually have any AAAA records. It's a trivial fix of course, but as I want to explore deploying IPv6 more widely, it's not something I've done yet. I'm very sorry, but it'll happen eventually!
Whilst I'm a big proponent of "infrastructure as code", there are some little tools and scripts which I often can't be bothered to fully provision with Ansible, or people reading my infrasturcture repo don't need to know about. For this, I've been using Dokku, which provides much the same functionality as something like Heroku (simple configuration, git-based deployment), such that I don't bother configuring it with Ansible (it still can be though). If you're after getting started with running your own applications on a server, Dokku is a really good way to go. I wouldn't use it for running other people's applications, nor for something I depended on commercially, but for hobbyists it's great!
My biggest gripe with Dokku may actually be considered a strength to others: It's a very "appliance"-like piece of software. Sure it's highly configurable, but really it takes over quite a lot of your system, to the point you probably don't want to run anything else major on it. It also means that in the event of weird issues, it can be difficult to diagnose. Dokku itself relies heavily on Bash and file templating. For example, in the past, I've had issues with it serving the wrong site for a given domain, or just weirdly crashing for no obvious reason (that last one may be my fault). Dokku also does interesting things around the default host - if it receives traffic for a domain it doesn't know about, it just serves the first app alphabetically (I wrote a small application to deal with this in a former life, don't judge it too hard).
My Dokku server's days are numbered. I'm slowly removing the apps from it and trying to move them to other places. I don't entirely know what I'll be switching to. It might be that Dokku stays but in a slightly different format, it might be that I try something completely different as part of a larger shift like Kubernetes. Either way, I know I need something as an application server like this which is quick and simple to deploy things to, without needing to tell everyone what and how I'm running.
My website server is very important to me. If you're reading this content, then chances are you got it from this server (if you didn't, where did you get it?). Previously, my website was just a static site, so serving it was trivially simple - but now it's its own custom Wagtail application backed by a database. The same server also runs my analytics platform (Plausible) as well as the comments you see underneath my posts (Commento++), so there's a lot going on on it.
Currently, the server is small VM from Vultr, with a slightly beefed up CPU and NVMe storage compared to their other tiers. Given the size of the VM, I think site response times are doing very well (thanks in part to lots of performance optimisation and Redis) As most of these tools are CPU-bound, having a faster CPU was more important than a little more RAM. Now that my website is much heavier, a little more resource would definitely be nice. For that, I intend to jump up the Vultr plans just a little to something with a little more RAM. The CPU usage on the VM is pretty good, but RAM usage idles around 85%, and frequently puts things in SWAP, so I'd like to avoid that.
As a developer (ish), I build a few applications which only really make sense for my own consumption (eg this website, my infrastructure repos etc). All these applications have dependencies of some kind, and it would be nice to make sure they're up-to-date. On GitHub, there are tools like Dependabot which can do this, but on my self-hosted GitLab, those aren't available.
Instead, I started running my own instance of Renovate, an open source and completely self-hostable dependency monitoring tool. Not only was it pretty simple to setup, but using it is simple too. Since getting it running, I've not had to mess with the configuration file at all (as can be seen by its history). Renovate just logs in to my GitLab, clones projects, checks for updates, then opens MRs for each change - exactly like Dependabot.
Renovate isn't perfect by any stretch. I love the fact that I can run it myself and that it's entirely open source, but it's not quite as polished as Dependabot. Notably, it had issues when trying to update psycopg2, a Python PostgreSQL client, due to the lack of pg_config in its update container. It also doesn't support newer Python package managers like hatch, which would have given me nicer dependency management features without having to go near the horrors that are poetry.
For the setup itself, Renovate runs in its own LXC container, with its own instance of docker. This ensures it's confined to a single machine, and can do really whatever it wants on it. As Renovate needs full access to Docker, there's also no risk of it trying to pivot and mess around with my other applications, whether intentionally or not. Renovate itself is also running under docker, mostly for the much simpler update strategy. There is a package on the AUR, but it's fairly out of date, and being able to use my existing docker update scripts makes my life easier when it comes to installing updates.
For the few cloud servers I have, I need a VPS provider I trust. I've bounced between Linode and Vultr a few times, and am currently straddled between the 2.
I first tried Linode after their Jupiter Broadcasting sponsorship, as I value a company's commitment to the community. Since however, they've been bought by Akami (not a bad thing), and no matter how many times I try and tell myself otherwise, they just seem slower than Vultr. Now that could just be that Vultr is faster than average rather than Linode being slower.
For 2023, I plan on switching to Vultr fully. Vultr has the speed and features I need for the price my wallet needs.
View all → The cloud is just someone else’s computer. In my case, the cloud is usually just my computer. As an avid self-hoster, I like to run a lot of things myself, for learning, privacy, and of course fun. And it’s hard to host anything without servers. I have quite a few… Back in December, I fully rebuilt my home server from the ground up based on Proxmox. Being a hypervisor OS, it makes sense to run everything in VMs or LXC containers, not on the host. Therefore, there’s a huge amount of scope for opinions, lessons and customizations. I’ve had quite… For the last 7 years or so, I’ve had some form of server sitting in my house. Its size and utility have changed massively in this time, but I couldn’t imagine life without it. Both prefer to and enjoy running a number of applications for myself, in a (possibly futile)…
Server Setup 2022
Server build 2020 - Proxmox setup
Home Server 2022