I have been experimenting and researching how to approach my Home Assistant deployment to ensure I will have a reliable install, low maintenance and room to expand. There were a few questions I needed to answer before I could go forward with my migration. Was my hardware detection script portable across devices? How should I handle storage? How does internal DNS on k8s work? What is the right way to deploy updates? All these questions and then some had answers I would need to discover in order to complete this deployment.
I was able to reach a point in the deployment where I am pleased with the overall results. I have the ability to easily roll out updates, plenty of fast storage, and new coordinator hardware for both my Zigbee and Zwave networks. There are some plans for future expansion, but I will need to invest in more hardware if I am going to explore distributed storage or other interesting projects. Future growth could also include writing up the deployment as a helm chart to make it easier to manage.
Expanding the Storage
The Dell thin-client I have has a Gen 8 i7 with 16Gb of RAM and came with a "Kingfast" 512Gb SSD. This is a great setup for a home server, but is light on storage for a longer running cluster, downloading all those container layers adds up. Cracking open the case I was greeted with an m.2 M key slot for an NVME drive, and an m.2 E key slot for a WLan card.
I ordered 2 NVME drives (one 2Tb and one 500Gb) hoping I could get both to work. I first tested the smaller NVME drive using the m.2 adapter as the only drive plugged in. Booting SystemRescue I was able to see and format the drive as ext4. After powering down my system, I plugged both the original drive and the other new NVME drive into the system and booted. This is where I ran into my first set of issues.
I kept getting warnings at boot about the power-supply not being properly detected. This was causing a POST error and preventing boot from continuing. I tried clearing the error but it came back. I decided to check the battery on the motherboard with a multimeter and confirmed it was dead. Slotting a new one in I had on hand for all my Zigbee sensors, I was able to ensure that the bios/EUFI settings persist.
However once I got booted with the other drives plugged in, I could no longer see the smaller NVME drive plugged into the WLAN port, despite having just formatted it on the prior boot. I rebooted several times with various drive combinations plugged in, and out of the near 2 dozen boots, the drive was only detected once more, this time with all drives plugged in. Again however it was gone after reboot. I decided that this lack of stability wasn't worth the risk but it was an interesting idea and I hope there is an opportunity to use the adapter in the future.
Formatting the 2Tb drive as ext4, I then updated /etc/fstab to mount the drive for storage using the UUID format.
UUID=4964ac3a-a0db-4213-865a-180539a81703 /mnt/storage ext4 defaults 0 1
Now I could fix my k3s install, because the default storage path for the Local provisioner is /var/lib/rancher/k3s/storage/ and I now have a drive mounted at /mnt/storage instead. I could have used symlinks but k3s provides an easy way to configure this on install using the --default-local-storage-path flag.
curl -sfL https://get.k3s.io | sh -s - --default-local-storage-path /mnt/storage
Prometheus and Bob
I am using Lens Desktop to have an easier way to manage and monitor my cluster. Lense uses Prometheus installed on the cluster to help visualize your cluster on your local machine, while also giving a way to easily scale deployments, clean up cruft or browse and install helm charts.
I added the Prometheus helm repo and tried to install the prometheus stack. Everything but the metrics monitoring server came up fine. That server however kept crashing with the error message
(combined from similar events): Error: failed to generate container "ec45dc07a178af82d1499ab9b1a82f5292a85f1ba463367fb81d8fe3552164cf" spec: failed to generate spec: path "/" is mounted on "/" but it is not a shared or slave mount
Searching around the internet, I came across a similar enough error in an issue on Gitlab. The recommended solution was to create a values.yaml file with the following contents
prometheus-node-exporter:
hostRootFsMount: false
Additionally the default namespace is used if none is provided, I would like to move Prometheus to it's own namespace to help focus on the cluster workloads in the default namespace. So in order to fix both of these we need to uninstall Prometheus, create a new namespace, then re-install into the new namespace with the new values applied.
helm uninstall prometheus
kubectl create namespace monitoring
helm install prometheus -n monitoring prometheus-community/prometheus --values prometheus-values.yaml
Now prometheus finishes it's deployment with out errors, and the default namespace is less cluttered.
Insect Broker
During the redesign of the home automation stacks, I had been reading about different ways folks like to setup and deploy home assistant. Something that came up multiple times is the benefits of running zigbee2mqtt over zha. Also having an mqtt broker on the network could be useful for additional future projects.
I quickly decided to try out mosquitto as my MQTT broker as it's the same one that is used for home-assistant's add-ons. Moqsuitto is FOSS software from the Elcipse foundation, and supports 5.0, 3.1.1, and 3.1 protocol versions for great compatibility with clients.
Following the docs on the official docker hub image page, I was able to craft the required deployment, service, and PVC for mosquitto.
Once deployed, I updated the configuration file in the PVC to have the right security credentials and network settings. Then I used MQTTExplorer to connect to the broker and confirm my configuration settings worked.
Device Dynamism
Normally in a k8s cluster, there is just a big slush-fund of CPU, memory and storage. This allows the control plane to schedule jobs on which ever worker nodes have capacity. This is really useful when you have spare resources distributed across several physical nodes. A problem rises when some of those nodes have specific hardware attached that certain pods need. Think GPUs, ASICs, or in my case USB serial devices.
One approach is to use node selectors or node affinity, but this requires setting up each node as I expand the cluster, and requires reconfiguration if I were to move the usb device to a different physical node. Fortunately k8s supports a concept called Device Plugins, which allows hardware vendors to write a wrapper around their hardware in order to expose it to the cluster as a resource for scheduling. This is better than node-affinity as it can be scaled across the cluster more easily. Unfortunately k8s expects every vendor to just provide a good working Device Plugin for us to use, and for some vendors like Nvidia, they do. However I just want a generic serial device exposed, and everything in *nix is just a fle any way.
Enter the Kubenetes Generic Device Plugin. A vendor agnostic, simple way of passing through devices to pods, using a DeamonSet that runs across the cluster to dynamically detect the hardware. This way if I move a USB dongle to another node, the cluster can just reschedule that pod. It also prevents us from doing something silly like trying to hand out the same hardware device to two containers at the same time.
When I first came across this plugin, I did not understand daemon sets or the device plugin idea k8s was trying to convey. It took a few attempts at reading the docs and playing with my deployment to fully understand exactly what was going on. I only needed to make a few minor tweaks to the manifest to best support my Zigbee and Zwave devices.
spec:
containers:
- image: squat/generic-device-plugin:11fa4f8a9655c253f01b3291d34588b3270d73dc
args:
- --domain
- "home-lab"
- --device
- '{"name": "zwave", "groups": [{"paths": [{"path": "/dev/zwave"}]}]}'
- --device
- '{"name": "zigbee", "groups": [{"paths": [{"path": "/dev/zigbee"}]}]}'
Hardware Havoc
About this time however I ran into an issue with test hardware I was using for my Zwave and Zigbee stick. I had borrowed a Nortek HUSBZB-1 from a friend to test out my deployment. This device is interesting as on the same stick it has both a Zigbee and Zwave network support. The device ends up being mounted on /dev/ttyUSB0 and /dev/ttyUSB1 where USB0 is always the Zwave coordinator.
However when I tried to use my previous bash script to detect the hardware vendor to sys path mapping, I was not getting the right results. The output to dmesage was hardly similar and didn't provide enough information to even link it.
I switched tactics and tried to use hwinfo as it's output uses a consistent format. There was in issue still however with this particular device, because of the way it shows up on the USB hub. There wasn't a good way to do this dynamically with this device unfortunately, but since we know the Zwave is always USB0 then it is possible at least to setup the symlinks.
I decided to bite the upgrade bullet, and swap my coordinator dongles for some new ones. Not only would this give me a better opportunity to test the hardware I wanted to use, but I was able to upgrade to some better supported hardware with firmware updates easily available.
For Zigbee I went with the Sonoff Zigbee 3.0 P (not E), which has an aluminum enclosure for heat dissipation and RF shielding, as well as an external antenna for better gain. I considered swapping the stock dipole with a circular polarized one I have for my drones, but the SMA type was wrong. If I have issues with devices on the Zigbee network I will keep this modification in my pocket as a possible solution. I already have my network on the Zigbee channel 15 which should keep me just out of WiFi channels 1 and 6.
For my Zwave hub I was fortunate in that I could skip past the series 7 hardware entirely as some series 8 had just been released. Zooz has an 800 series dongle that is already supported by zwave-js, the ZST39. It works with OTW firmware updates as well making it an easy choice. The smartest home was running a sale at the time and I grabbed one for less than $30.
With my new hardware in hand, hwinfo installed into alpine I could proceed with tweaking my script for symlink creation.
the main line at issue was
PCI_ID=`sleep 1 && dmesg | grep $DEV_ID | grep 'usb-' | tail -n 1 | cut -d - -f 3`
This no longer worked as I learned that the output of dmesg is not standard. Playing around with hwinfo however I was able to use the consistent output format and write the following script.
#!/bin/ash
# Dynamically grab the device ID
# Give dmesg enough time to have the logs we need, by calling sleep 1
VENDOR_ID=`echo $USB_ID | cut -d : -f 1`
DEV_ID=`echo $USB_ID | cut -d : -f 2`
DEV_PATH=`hwinfo --usb | grep -A 8 $VENDOR_ID | grep -B 1 -A 5 $DEV_ID | grep "Device File" | awk '{print $3}'`
LINK_PATH=/dev/$DEV_TYPE
TTY_DEV=/dev/$MDEV
# Test if we found a match for our device in mdev event
if [[ "$DEV_PATH" == "$TTY_DEV" ]];
then
logger "Matched $DEV_TYPE device at $TTY_DEV to $VENDOR_ID:$DEV_ID"
# Test if the tty device was added or removed
if [ "$ACTION" = "add" ];
then
ln -sf $TTY_DEV $LINK_PATH
# Confirm success or failure on creation of symlink
if [ $? == 0 ];
then
logger "Created soft-link from $LINK_PATH to $TTY_DEV"
else
logger "Soft-link creation between $TTY_DEV and $LINK_PATH failed"
fi
fi
if [ "$ACTION" = "remove" ];
then
rm $LINK_PATH
# Confirm success or failure on removal of symlink
if [ $? == 0 ];
then
logger "Removed soft link from $LINK_PATH to $TTY_DEV"
else
logger "Soft-link removal between $TTY_DEV and $LINK_PATH failed"
fi
fi
fi
Providing the combined vendor and device IDs as the USB_ID I was able to directly discover the device path, and compare it with the one provided by the mdev event to ensure we got a match.
With the new script for setting up symlinks now working and integrated on boot, I was able to easily link /dev/zwave and /dev/zigbee back to the associated /dev/tty* devices with hot-plug events.
Supporting Cast and Crew
At this point I felt I had a working pattern I could use to copy-paste my way to deployment success through iterative practice. The next task in front of me was to address the zigbee and zwave services that home-assistant would be speaking to.
I already had some experience with zwave-js from making the switch on my previous instance, so I decided to start there. I copied the same format for my PVC and deploy YAML files, and only adjusted the relevant sections like container name, deployment name, image, ports, environment variables, and mount paths.
The zwave-js-deploy.yaml file I created has the following contents
apiVersion: apps/v1
kind: Deployment
metadata:
name: zwave-js
labels:
app: zwave-js
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: zwave-js
template:
metadata:
labels:
app: zwave-js
spec:
volumes:
- name: zwave-js-store
persistentVolumeClaim:
claimName: zwavejs-store-pvc
containers:
- name: zwave-js-ui
image: zwavejs/zwave-js-ui:8.11.0
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
home-lab/zwave: 1
ports:
- containerPort: 8091
- containerPort: 3000
volumeMounts:
- name: zwave-js-store
mountPath: /usr/src/app/store
env:
- name: SESSION_SECRET
value: "A59E88FBDD4A10D7E1925C755F3237B1"
- name: TZ
value: EST5EDT
- name: ZWAVEJS_EXTERNAL_CONFIG
value: /usr/src/app/store/.config-db
---
apiVersion: v1
kind: Service
metadata:
name: zwave-js
spec:
selector:
app: zwave-js
type: LoadBalancer
ports:
- name: http
port: 8091
targetPort: 8091
- name: websockets
port: 3000
targetPort: 3000
The following is the contents of my zwave-js-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: zwavejs-store-pvc
namespace: default
spec:
resources:
requests:
storage: 256Mi
storageClassName: local-path
accessModes:
- ReadWriteOnce
In order for the deployment to succeed, the PVC need to be in place first.
kubectl apply -f zwave-js-pvc.yaml
kubectl apply -f zwave-js-deploy.yaml
Once the container was up I was able to set the configuration values for the device so that /dev/zwave was used as the device path and mqtt (the name of the container) was my hostname for the mqtt server. As far as I could tell, everything was up and running but with out a home assistant server or any hardware to test, I wasn't sure.
I took a spare Zwave smart plug and tried to pair it to the new dongle and it worked! I was able to control it's state just fine. From here I was satisfied that my zwave-js deployment was working just fine and all I needed to do was setup home-assistant to speak to zwavejs.
Next on my task list was getting zigbee2mqtt setup, which did not appear all that daunting due to the large amount of clear documentation provided by the project.
Copying over the same structure from the zwave and mosquitto deployments, I quickly had something ready to deploy on my cluster. However this is where I hit a snag with my deployment. Turned out it was a gap in my understanding of internal DNS on a k8s cluster.
What's in a Name?
Domain Name Service is the system our computers use to resolve hostnames to IP addresses. By default, k3s comes with and deploys CoreDNS as an internal DNS server. This means services (not containers), get a DNS entry that can be resolved based on the service name (or internal FQDN). This threw me off because I had been focused on container names in docker-compose not recognizing I was using the same service name as container name. I had not carried over the pattern of a 1:1 service name and container name when I wrote my k8s deployment files.
After reading the k8s docs on networking and DNS, I realized my assumption and corrected my configuration pointing to the mosquitto service, which all the other pods could find. zigbee2mqtt finished deploying successfully and we were ready for the next deployment.
The important take-away for me is to keep in mind the difference between a deployment, service, and container and which layer is doing which job, and to check my assumptions at the door.
Main Character Syndrome
With the supporting cast and crew deployed, we were finally ready to tackle the home-assistant deployment and PVC. Again I just followed my existing patterns, changing names, mount points, containers, etc where appropriate.
At this point I was expecting trouble, but pleased to find, it just worked! At this point I started configuring home-assistant to speak to mosquitto for zigbee devices, and I decided to have it speak directly to zwave-js for zwave devices. I used websockets for both service connections, due to the lower amount of resource consumption websockets generally has compared to polling.
With the stacks deployed, I added a few of my new sensors and plugs to the network to confirmed everything was working im homeassistant. At this point, I was ready to begin moving the devices to the new dongles but was daunted at how many I would have to do and how much work it might be getting everything setup again. Waiting was not going to solve anything so it was time to pick a Saturday and just start in the morning and hope I am done by Sunset when most of my automations start to kick in more.
The Great Migration
Zwave
It was time, hardware was ready, software configured, and mentally I was prepped to make it happen. Starting early in the day, I decided to migrate the Zwave network first as I had fewer devices, and many of them were easy enough to control manually if needed (like light and power switches).
The general workflow was simple enough, set zwave-js into exclude mode, and then activate the exclusion method on the device. Once exclcuded, activate the inclusion mode on the new network, and activate the inclusion on the device. If it had S2 security, I also needed to have the Zwave DSK pin code to securely pair the device or be required to use a lower security level. I have long since disposed of all the paper and packaging for my existing hardware, so I had to grab the DSK from the physical devices. This was as simple as just locating it on the devices and the only devices that required a little extra effort were the in wall switches, where I had to pull the wall plate off first.
There were a few devices that failed to exclude, for these I reset the device entirely, and then removed the dead device from zwave-js, before putting the device back into inclusion mode and adding it to the new Zwave network, and provide the DSK if there was one.
During the inclusion, zwave-js asks you for a device name. You can set this later but setting it up front worked well for me so I could be methodical about the approach. Either way I think it is a good idea to name the device at the zwave-js level because this name is propagated down to home-assistant, keeping things consistent for device identification.
I also tested the OTW firmware update for the 800 series stick from zooz. It worked first try with no issues. One of the reasons I chose the zooz stick was because they had firmware updates available for their devices and seem to be a good vendor in the zwave ecosystem.
Zigbee
The zigbee network was up next, and while zha has added migration capabilities, I was also going to migrate to zigbee2mqtt and I felt starting fresh might be the best move.
A lot of the zigbee devices that run on mains power immediately go into inclusion mode after you exclude them from the zigbee network. This meant I could exclude devices on the old system, then run the inclusion on the new system and they would show up right away with no physical interaction. For devices that were running on battery, I had to physically reset them after exclusion and put them back into pairing mode. A few devices did not auto switch into pairing mode or failed to remove. For them, I just reset them manually and included them into the new network.
As I added each device, I also updated the device name, and checked to update the name on home-assistant, again keeping everything in sync. Browsing the interface I saw the OTA tab tried updating some of my devices, and several of my lightbulbs were able to update but not all of them. I had to power cycle the bulbs a few times as well as the system, but eventually I tried again and was able to compete all the available OTA updates for my zigbee devices.
Finally I decided to update the firmware on the zigbee coordinator stick itself. The process is well documented and setup to use a docker container to make things easy. I had to install docker on the alpine host, scale the deployment down to 0 replicas, run the container, then scale up the deployment. This was also an easy process thanks to the amount of documentation available.
Burrito Wrap It
Letting the system run for a couple of weeks let me get a sense of the load. I realized setting what I thought were modest limits, I had over-provisioned CPU and memory for all the services. Lowering the requests and limits for the services, I think I have a lot more room for expansion than I initially suspected.
There is still some room to expand the cluster. For one, it's still only one physical node, so calling it a cluster really isn't appropriate just yet. Also I need to spend more time investigating how I will handle distributed storage once I add more nodes. There are a few solutions in this space I have been looking into, including rook ceph, longhorn, and kadalu. Next up though is going to be secret management. I have one Zwave device that refuses to update, not sure what the best solution is but it's something I can look into.
Share and Enjoy!