# Home lab with NanoBSD, APU2 and Kubernetes on Raspberry Pi - 0



## Home lab.

{{< style "text-align:justify;  margin-bottom:2em;" >}}
For some time I had been thinking about building a home lab (Kubernetes cluster) with access
to and from the internet so I could research and practice with the technologies I care about, using
low-power components that are as resilient as possible to power outages (to avoid corruption
of storage devices). To get there I reused hardware I had bought at some point simply because it
interested me. Without further delay, here is a network diagram of how this lab ended up:

{{< image src="lab-schema-1.png" alt="Diagram" caption="Lab network diagram" >}}
{{< /style >}}

The main idea is to build a home lab using low-power devices and to configure/build all the
hardware and software components needed to consume and expose personal services to and from the internet.

{{< admonition note "Note" >}}
**APU2 ENV** refers to the virtual environment inside an embedded device using FreeBSD Jails.
{{< /admonition >}}

<br>
<br>

## Hardware.

| Label   | Device   | Role           |
|:-------|:---------|:-------------|
| ISP Wireless router | ISP default       | Internet access / port forwarding |
| APU2                | pcengines.ch      | Internal router/firewall + virtual servers  |
| Virtual Server      | FreeBSD Jail      | Virtual servers inside the APU2        |
| NetGear POE+ switch | ProSAFE MS510TXPP | Switch + Power over Ethernet for Raspberry Pi nodes |
| Raspberry           | Raspberry Pi 5    | Kubernetes node (kubelet) |

<br>
<br>

### Virtual servers

{{< style "text-align:justify;" >}}
The internal router (APU2) runs [NanoBSD](https://docs.freebsd.org/en/articles/nanobsd/). NanoBSD is a
[FreeBSD](https://www.freebsd.org/) project that provides a FreeBSD image for embedded systems. Inside NanoBSD
it is possible to create confined processes ([jails](https://wiki.freebsd.org/Jails)) that behave like independent
servers, isolated from each other and from the base operating system. To achieve this isolation FreeBSD has
implemented interesting technologies such as network stack virtualization ([VNET](https://wiki.freebsd.org/VIMAGE)),
resource limits ([rctl](https://wiki.freebsd.org/Hierarchical_Resource_Limits)), among others.
<br>
<br>
You can think of FreeBSD Jails as the analogue of Docker containers on Linux.
{{< /style >}}

### APU2

{{< style "text-align:justify;" >}}
To implement the internal router/firewall I used an [APU2](https://openwrt.org/toh/pcengines/apu2) motherboard
from [pcengines](https://www.pcengines.ch/). The main trait of this device is that it is designed as an
embedded system: it has no moving parts such as mechanical hard drives, no sockets for plugging in RAM modules,
and passive cooling (no fan is required because it is a low-power device). Combined with the advantages of the
[NanoBSD](https://docs.freebsd.org/en/articles/nanobsd/) project, that makes it ideal for this setup.

The APU2 motherboard includes the following features (among others):

* 1 AMD Embedded G series GX-412TC processor, 1 GHz quad Jaguar core with 64 bit  2MB L2 cache.
* 4 GB DRAM (soldered onto the board).
* 1 m-SATA slot.
* 3 Gigabit Ethernet ports.

{{< image src="apu2c2_1.jpg" alt="APU2" caption="APU2 motherboard" >}}

As storage, a solid-state mSATA drive similar to the one shown below was used.

{{< image src="msata16d2.jpg" alt="mSATA SSD" caption="Solid-state drive." >}}

{{< /style >}}

{{< admonition note "Note" >}}
Unfortunately, at the time of writing this article, [pcengines](https://www.pcengines.ch/) has ended development
and production of APU2 boards ([here](https://www.pcengines.ch/eol.htm) is their announcement and rationale). Hopefully
in the not-too-distant future they will surprise us again with a similar or better design.
{{< /admonition >}}

<br>
<br>


### NetGear switch

{{< style "text-align:justify;" >}}
The switch used is the ProSAFE MS510TXPP. I chose this model mainly because of its PoE+
(Power over Ethernet) capabilities, which let me power the Raspberry Pi 5 devices over the network cable
(a PoE HAT must be purchased for the Raspberry Pi boards).

<br>

The switch has the following features:
<br>
<br>

Web-based management with the following capabilities:

* **VLANs**: Supports up to 256 port-based VLANs, IEEE 802.1Q, Auto Voice VLAN, and Auto Video VLAN.
* **Static routing (Layer 3 Lite)**: Supports up to 32 static IPv4 routes and 32 static IPv6 routes, plus an ARP table of up to 512 entries.
* **Link aggregation / port trunking:** Compatible with IEEE 802.3ad LACP (Link Aggregation Control Protocol).
* **Quality of Service (QoS)**: Traffic prioritization based on ports, IEEE 802.1p, DSCP, TCP/UDP, and DiffServ with WRR (Weighted Round Robin) queues or strict priority.
* **Monitoring**: SNMP v1, v2c, and v3, RMON (groups 1, 2, 3, and 9), and remote syslog.

<br>
<br>

Port configuration (10 ports total):

* **Standard Gigabit ports (1G)**: 4 RJ-45 ports (10M/100M/1G) with PoE+ support.
* **Multi-Gigabit ports (2.5G)**: 2 RJ-45 ports (100M/1G/2.5G) with PoE+ support (IEEE 802.3bz / NBASE-T).
* **Multi-Gigabit ports (5G)**: 2 RJ-45 ports (100M/1G/2.5G/5G) with PoE+ support (IEEE 802.3bz / NBASE-T).
* **10G copper uplink**: 1 dedicated RJ-45 port (100M/1G/2.5G/5G/10GBASE-T).
* **10G SFP+ fiber uplink**: 1 dedicated SFP+ port (1G/10GBASE-X SFP+).

{{< image src="switch-prosafe-ms510txpp.jpg" alt="MS510TXPP" caption="MS510TXPP" >}}
{{< /style >}}

### Raspberry Pi 5

Not much to say here: four Raspberry Pi 5 boards were used for the Kubernetes nodes. At the time of writing this
article, the official [Raspberry Pi](https://www.raspberrypi.com/products/raspberry-pi-5/) product page offers
a 16 GB RAM version; the devices used here have half of that, 8 GB of RAM.

{{< image src="rbp-5.jpg" alt="rbp-5" caption="Raspberry Pi 5" >}}

<br>
<br>

microSD cards are relatively fragile under power outages, so the Kubernetes nodes were fitted with PoE HATs
(Power over Ethernet) to achieve two goals: first, replace the microSD card with an m.2 solid-state drive that is
more resilient to power cuts; and second, power the Raspberry Pi boards over the network cable. As mentioned earlier,
the NetGear MS510TXPP switch can power network devices over the Ethernet connection.

{{< image src="poe-hat.png" alt="poe-hat" caption="Power over Ethernet HAT" >}}

### Rack for Kubernetes

One problem when building a Kubernetes cluster with Raspberry Pi boards is how to organize the devices—both the compute
nodes and the router. [GeeekPi](https://thegeeekpi.com/) manufactures racks for Raspberry Pi boards and other accessories
to organize this kind of project. The mini-rack chosen for this home lab is the
[T0](https://www.amazon.com.mx/GeeekPi-Armario-servidores-equipos-RackMate/dp/B0FPF8CRYG/ref=asc_df_B0FPF8CRYG?mcid=5b3116fae09b37a6adf5d0d3bd2ab0ba&tag=gledskshopmx-20&linkCode=df0&hvadid=782828334665&hvpos=&hvnetw=g&hvrand=1425340664839516678&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9047091&hvtargid=pla-2441132577982&hvocijid=1425340664839516678-B0FPF8CRYG-&hvexpln=0&language=es_MX&th=1) model.

{{< image src="t0.png" alt="rack" caption="GeeekPi rack" >}}

### Final setup

<video controls width="100%" playsinline>
  <source src="/videos/home-lab.mp4" type="video/mp4">
</video>


### Next steps

1. Install the NanoBSD image.
2. Install Raspberry Pi OS on the Raspberry Pi nodes.
3. Create the Kubernetes cluster.