Moving my website to Hetzner

If you're reading this, you read it from a server hosted by Hetzner - congratulations!

My website is quite important to me. I want it to be reliable, stable, and yet still be something I can tinker with easily if I want to. Almost 2 years ago, I rewrote it to use Wagtail, and it's been running on a single VPS in London ever since. However, for a few reasons, I want to move it to Hetzner.

#Where was it before?

For many years, I've been a fan of Vultr. Their servers are well priced, they support Terraform, and generally seem like decent people.

In preparation for the move from Hugo to an actual web server, I provisioned a "High Frequency" VM. As the server would be doing actual work, as opposed to just reading static files from the file system, the ~60% boost ought to serve me nicely, and it was only an extra $1/mo for that plus an NVMe disk.

A year or so in, I started facing memory exhaustion issues. This server runs my website, analytics and comments. My analytics tool of choice, Plausible, is sadly not a lightweight platform, notably because it uses Clickhouse which can be a bit of a beast. This jumped the cost from $6/mo to $12/mo. Still not huge, but something I wanted to keep an eye on.

Around a year ago, I tried to get IPv6 working for my website - it ought not to be too difficult, I thought. In the end, it wasn't, and it was interesting getting IPv6 configured (I hadn't really touched IPv6 before). After deploying it for a few months, I started having odd monitoring blips saying my website was down, but only over IPv6. A friend of mine has also had odd issues with IPv6 in Vultr, so it's likely not an isolated incident. It's been much better more recently, but it was an annoying few months.

In March 2024, Vultr had some issues with the wording of their terms and conditions, which the internet really didn't like. I was intending to move from Vultr anyway, but this just accelerated my plans a little. Vultr responded to the issue pretty well and pretty quick.

Whilst I don't have any servers there anymore (at least once this VM is actually deleted), Vultr will still likely be my go-to VM provider for certain use cases, especially managed kubernetes (if the need arises).

#Where's it going

Hetzner are a German hosting company, well known for their great service and even greater pricing. They've definitely had some issues in the past, but as a company I trust them and like the mission they seem to be on.

#Pricing

One of the big benefits of Hetzner is price to performance. Hetzner use AMD EPYC or Intel Xeon CPUs, which Vultr refers to as their "High Performance" tier.

Hetzner's cheapest server, the CX11, is €3.79 / mo (~$4.07), which gives a single Intel CPU and 2GB RAM. Vultr's similar offering is triple this, at $12/mo.

The pricing gets even better for Hetzner as the resources grow. Being able to run multiple processes concurrently is very beneficial to Django, so I went with the 2-core CPX11, for a mere €0.56/mo more (totalling €4.35/mo, or $4.67). Vultr in contrast is again much more expensive, at $18/mo.

Saving between $8 - $12/mo (depending on how you measure), and getting more for what I do spend is one of the main reasons i'm here.

#Location

Servers in their Nuremberg datacenter are around 27ms from me, vs the 4ms of Vultr's London site. For a latency-sensitive application, this might be an issue - But for my website, I'm not too bothered. For my gateway server, I'd like that to be as fast as possible, so Hetzner isn't an option (it's not like they'll open a UK datacenter anytime soon).

I'm also a fan of moving my server hosting away from US-based companies.

#Energy

When we talk about computing, we have to talk about energy usage. I try and make my projects as efficient as possible, and with speed and performance improvements also come energy efficiency improvements. My website is a drop in the ocean of global energy usage, but I'd still like to do my bit if I can.

Whilst Vultr have made strides in this area, Hetzner run 100% hydropower, and have done for a while. This means, my website is now "hosted green"!

#How'd the move go?

I've been putting off the move for a while. Last weekend, I decided to provision and bootstrap the basics onto a VM, and this weekend I sat down and just did it. I could spend time thinking of the best plan possible, or I could just get on with it.

I knew the process would be fiddly, mostly because I wanted to reuse the hostname for the 2 servers, and neither Ansible nor DNS really like that.

My website is exactly that: a website. It's not mission critical, it's not earning me (much) money. It's not the end of the world if it's not available 100% of the time. So I decided to accept downtime knowing the transfer would be simpler, cleaner, and likely much less stressful.

According to Grafana Cloud, my website was down for 59 minutes, which is perfectly acceptable to me. The time would have been much less, but I did a lot of prep and tinkering as I went, rather than ensuring everything was ready before. Along the way, I hit some wonderful issues with IPv6, DNS and memory usage (the usual).

#Migration process

Before I started, like any good sysadmin, I noted down the steps I needed to do to get things transferred:

1. Stop containers: Everything notably running on that server runs in Docker, so copying the docker-compose.yml and volumes is all that's needed.
2. Sync files: To copy these files between the servers, I used tar to combine and compress them into a single file, and then scp -3 to sync them between the 2 servers via my local machine (to avoid any agent issues).
3. Update DNS: I manually updated the single CNAME record for the server to point to the new IP addresses
4. Run ansible: Ansible would then take care of everything which is fully versioned, and issue certificates
5. Update Terraform: Import the server and DNS changes into Terraform to make management in future easier.

In the original draft, step 4 had the certificate step extracted for convenience. I ended up not doing that, as there was no real benefit, and it would have required more changes to my Ansible configuration than I expected or wanted (dependency management is a hard problem).

#IPv6

Hetzner give you a few options for base OS: Ubuntu, Fedora, Debian, CentOS Stream, Rocky Linux and AlmaLinux, with the option of booting into a custom ISO after setup if you need to. All my infrastructure runs Arch (btw), because I'm mad, and it's the distro I'm most comfortable managing.

When I set up the server, I chose Debian 12, and then booted into their kindly-provided Arch ISO to do a proper installation. This worked perfectly, and I ended up with a nicely-provisioned Arch machine.

What I hadn't noticed though, was something missing: IPv6. The server had an IPv6 address, but it wasn't added to the interface properly.

The cause of the problem was systemd-networkd. During the Arch install, the interface was eth0, which was encoded in the cloud-init created interface configuration. Now the OS was installed, the interface was enp3s0 (because reasons). systemd-networkd was already matching the correct interface by MAC address, so removing the interface name match and rebooting gave me lovely working IPv6!

#DNS

Repeat after me: It's always DNS. Although this time, it wasn't technically DNS - just DNS adjacent.

Recently, I made a change to use CoreDNS and Nginx to replace Traefik (that's an interesting journey, which I'll talk about soon). As part of this, CoreDNS was listening on port 5353. On my Vultr VM, this seemed fine. However this Arch install had systemd-resolved on that port instead, likely to service mDNS. 5353 is reserved for mDNS, so I just changed the port to 53053 and continued on with my day.

Conveniently, I'd had to solve a similar issue with systemd-resolved being too powerful on my AdGuardHome installation. They have some great documentation on how to tell systemd-resolved to get out the way of system DNS, and let DHCP and the static DNS configuration deal with things natively. If you have a similar issue, that's a valid solution.

#Memory usage

Even though the server had the same amount of memory, as soon as I started Plausible, it immediately consumed every last byte of memory and the server froze, requiring a reboot.

What I'd forgotten about, in the migration, was the increased Swap. To help that server deal with the short memory spikes Plausible and Clickhouse like to do, among other things, I added 8GB of swap. After adding 6GB (to be a little more conservative), everything started up just fine. If I want to change swap in future, it's not that difficult.

Yes, I'm aware, I could solve this with proper memory constraints on the container, but I like to allow applications to use as much as they need to do their best work.

#Result

As I write this, it's only been a few hours since everything switched over, but I'm pretty happy with it. A good measure of that is in the fact that everything is working, there's no notable change in functionally, and in fact feels like page loads might be even faster!

With the jump from 1 to 2 cores, I can scale up my website, too. Gunicorn, the web server I'm using, suggests around 2n+1 workers, where n is the number of CPU cores. This means I can comfortable jump from 3 to 5 processes, which should drastically reduce latency under load.

Grafana Cloud is strangely showing a global average latency increase, of around 50ms, but that's more than within a margin of error. London has naturally increased from ~40ms to ~120ms for a homepage load, and Frankfurt has reduced from ~110ms to ~65ms. Hetzner peer with a lot of other ISPs, but perhaps Vultr's were a little better for certain paths. From here, pages seem to load snappier, although that could just be me being optimistic.

This migration has definitely been a success, but I have one more cloud VPS to move, this time from Linode. I'd love to use Hetzner, but I really want a UK-based server for this, to help reduce latency. I have some ideas, but you'll have to wait and see to find out what they are...