Running Kubernetes Cluster for Less Than $5

25 Jan 25

Ever since switching career from programmer to an infrastructure engineer, I have been exposed to many new interesting topics. Some of which are kubernetes and homelabbing. The idea of having a physical server running at home for the purpose of general service hosting and also as a literal “lab” has been lingering on my head for quite some time now. And that is how I got the idea of running a kubernetes cluster on premise in my house using my own hardwares. So that’s what I did, I repurposed some of my retired computers to be used as servers and they have been the backbone of my homelab kubernetes cluster for the past 3 months.

Table of Content

  1. Hardware
  2. Software
  3. Cost
  4. Future Work

(Old) Hardwares

Let’s start with the hardware. I am using some old computers that I have lying around, and what I mean by “some”, to be precise, is two computers consisting of 1 old laptop (Thinkpad T430) and 1 PC. When I say “old”, it means literally old. The laptop is around 12 years old while the PC is about half that age. Not super exciting and they don’t have a lot of compute power but they are way more than enough for my current use case with tolerable power consumption.

Compute

Below is the general idea of the computers’ specifications:

  • Laptop
    • I5 3320M 2 cores 4 threads
    • 16G DDR3L Memory
    • 128G Sata SSD
  • PC
    • Athlon 3000G 2 cores 4 threads
    • 16G DDR4 Memory
    • 240 + 128G Sata SSD

Both machines have some “conditions”, as with many old computers. First, for the laptop, it has broken hinges, dead battery, non-functional keyboard & touchpad, and also dead CMOS battery. Imperfect as it may, but that laptop is way more reliable, in a way, than the slightly more modern Athlon PC. That PC would just randomly be unable to boot. Sometimes I can’t even go into the BIOS setting, sometimes resetting the BIOS works, sometimes just removing and reinserting the CMOS battery seems to work, sometimes I just let it sleep for a day and be hopeful that the next day it will magically boot. Moreover, this variant of AMD CPU combined with my motherboard won’t go deeper than C2 state (ref), which is not really ideal for running server in which maybe 99% of its uptime is spent on idle.

powertop result in PC
powertop result in PC
powertop result in Thinkpad
powertop result in Thinkpad
foxxy the security engineer
foxxy the security engineer

Networking

There is nothing fancy going on in the networking department. I’ve subscribed to a 50Mbps internet bandwidth from a well known ISP here for a relatively fair price + additional cost for their leased ONT router. I don’t have any outage as far as I remember and it’s been running well. Two things to note though:

  1. It is behind a CGNAT
  2. It has some sort of LAN-to-WLAN isolation, meaning my devices connected through the LAN interface cannot communicate with devices connected to WLAN interface

The second point forces me to buy an additional router and run it in Double-NAT configuration.

router abomination
router abomination

Software

Virtualization

I decided to use proxmox as my Virtualization OS. It is free, easy to use, and is recommended a lot by some folks online. Here are most of what I’m currently running on my infrastructure:

  1. PiHole on an LXC Container as DNS server
  2. Private Image Registry
  3. Debian VMs for k8s nodes
  4. An LXC Container functioning as a “gate” for me to access my entire homelab via tailscale

Remote Access

My home network is running behind a CGNAT and my ISP doesn’t allow port forwarding. So, I don’t actually have an out of the box solution for accessing my home network from public internet. Luckily, tailscale has an amazing service for allowing several machines to be interconnected via what they called a “tailnet”. Their free tier is more than enough for simple homelab architecture like mine for allowing up to 3 users and 100 devices to be interconnected. I installed tailscale on one dedicated LXC container called “gate” and I use that container as some sort of reverse proxy to access other services in my home network.

About Back Up

I don’t currently run any automated backup in my infrastructure. The reasons are simple:

  1. I don’t have a lot of storage
  2. I am cheap so I don’t really want to pay for cloud storage just to store my backup files
  3. I am not running mission critical software or anything. If all the data is gone, then so be it.

I do, however, occasionally backup my entire VM or LXC container directly from proxmox. Proxmox supports backing up entire VM or LXC container using vzdump but honestly I only ever use the backup feature via their web UI. By default on the web UI it will compress the back up file using zstd algorithm compression and store the file in /var/lib/vz/dump/ directory.

Kubernetes

I tried to run most of my application in a kubernetes cluster. Except for the ones that I want easy access to backup and restore like DNS server and image registry. I chose K3s as my Kubernetes distribution since they support HA and pretty lightweight. The Kubernetes cluster is setup in a High Availability mode consisting of 3 nodes (VM) with etcd as the cluster datastore. I set it up using Ansible mostly by copy-pasting code from k3s repo with slight modifications (mostly removing stuffs I don’t use).

list of my kubernetes nodes
list of my kubernetes nodes

As for the day to day operations on the kubernetes cluster itself, I rely on GitOps, specifically using fluxcd. The applications running on top of the cluster itself is not that many, consisting of:

  • Monitoring stack
    • Grafana
    • Prometheus
    • Loki
  • Github Action Runner Controller (and its runners)
  • Apps

I might write about those individual tech more detail in the future. And yes, yes, yes, I know, I have more workloads running infrastructure automation and monitoring than the actual applications. Hopefully I get to finish many more project in the future and will be able to update that list.

Cost

After watching Wolfgang’c Channel’s Video about building an efficient home server I became obsessed with observing how much power my entire homelab requires. And whether it is worth it to keep running them or should I just rent a cheap VPS server somewhere. I mentioned before that I’m cheap, and I want to spend as little as possible for this hobby project. So I decided to buy one of those wattmeter. And do some rough calculation on how much this entire setup cost me. For reference, Contabo has an offering of $5.5 for 4 vCPU and 6GB of RAM so I set a limit of the same $5.5 for running the entire homelab.

As seen on the image below, the total wattage for 46.5 days is around 53.54 kWh (sometimes I run my washing machine on the same outlet which adds some additional wattage but let’s ignore that). That brings the monthly electric consumption to be around 34.5 kWh. With the current price of electricity of around $0.089 ( Rp 1444.7 ) per kWh, that result in a cost of about $3.1 (IDR 49,842.15) per month which is well within my budget.

electricity consumption
electricity consumption

Future Work

This setup is obviously not permanent. I might change a couple of things here and there along the way to adjust my needs. I might even ended up taking down the entire homelab should that no longer be interesting and/or give minimum to no value anymore (who knows). But as of right now there are a couple things I want try out:

  1. HA Proxmox
  2. Setting up NAS
  3. Having off-site backup server
  4. Setup terraform project for managing Proxmox VM / LXC.
  5. Setup new servers using more modern CPU
  6. Setup a server rack (this one has a very low wife-approval factor)