That wouldn’t be so bad if a lot of the home automations didn’t factor in presence status which gets synced through that, so this has been a source of minor annoyance whenever his status refused to mirror his presence or absence. It just happened again and because every single time now we’ve had to try to remember how to fix it, here’s a quick TIL to encourage my memory 😅

And it’s simple really. Open the iOS settings, scroll down to the Home Assistant settings and make sure that location sharing is set to “Always”.

First of all, in your unifi controller you should create a new user that Home Assistant will act as to manage your port forward(s) for you. So, log into the controller, go into Settings > Administrators and add a new Administrator user¹.

Then create your port forward in Settings > Routing & Firewall > Port Forwarding. Take note of the id of the port forward you have created - you can find it by clicking edit on it again, it will be the number at the end of the URL of the edit page. E.g. if the URL looks like this: https://my.unifi.controller/manage/site/default/settings/portforward/edit/1234567890 then this is the id of the port forward: 1234567890.

Next, copy this shell script to /config/scripts/unifi.sh in your Home Assistant. Make sure to adjust https://my.unifi.controller (and, if necessary, the site default) to your own values.

#!/bin/sh

set -e

# based on https://community.home-assistant.io/t/automating-unifi-port-forwarding-based-upon-presence-detection/168185

cookie=$(mktemp)
headers=$(mktemp)
curl_cmd="curl --silent --cookie ${cookie} --cookie-jar ${cookie} -D ${headers} --insecure"

BASEURL="https://my.unifi.controller"
SITE="default"

auth() {
  USERNAME=$1
  PASSWORD=$2

  # authenticate against unifi controller
  ${curl_cmd} --output /dev/null -d "{\"username\":\"$USERNAME\", \"password\":\"$PASSWORD\"}" $BASEURL/api/login

  # grab the `x-csrf-token` and strip the newline (added when upgraded to controller 6.1.26)
  csrf="$(awk -v FS=': ' '/^x-csrf-token/{print $2}' "${headers}" | tr -d '\r')"
  echo $csrf
}

portfwd() {
  USERNAME=$1
  PASSWORD=$2
  FORWARD_ID=$3
  FORWARD_ENABLED=$4

  # authenticate against unifi controller
  csrf=$(auth $USERNAME $PASSWORD)

  # enable/disable firewall rule
  body=$(${curl_cmd} -X GET $BASEURL/api/s/default/rest/portforward/$FORWARD_ID | jq '.data[0] | .enabled='$FORWARD_ENABLED'')
  ${curl_cmd} -X PUT $BASEURL/api/s/default/rest/portforward/$FORWARD_ID -H "Content-Type: application/json" -H "x-csrf-token: ${csrf}" -d @<(echo "$body")
}

isportfwd() {
  USERNAME=$1
  PASSWORD=$2
  FORWARD_ID=$3

  # authenticate against unifi controller
  csrf=$(auth $USERNAME $PASSWORD)

  ${curl_cmd} -X GET $BASEURL/api/s/default/rest/portforward/$FORWARD_ID | jq '.data[0].enabled'
}

"$@"

Now, let’s imagine you want to add a switch for an SFTP port forward that you’ve just created. Then, in your secrets.yaml file, add the following:

unifi_forward_sftp_check: '/bin/bash /config/scripts/unifi.sh isportfwd <user> <password> <forward_id>'
unifi_forward_sftp_enable: '/bin/bash /config/scripts/unifi.sh portfwd <user> <password> <forward_id> true'
unifi_forward_sftp_disable: '/bin/bash /config/scripts/unifi.sh portfwd <user> <password> <forward_id> false'

Replace <user>, <password> and <forward_id> with the login credentials and id of the forward you just created.

Next, add a command line switch definition to your configuration.yaml (or in my case to my packages/network.yaml file):

switch:
  - platform: command_line
    switches: 
      sftp_port_forward:
        friendly_name: "SFTP Port Forward"
        command_state: !secret unifi_forward_sftp_check
        command_on: !secret unifi_forward_sftp_enable
        command_off: !secret unifi_forward_sftp_disable
        value_template: '{{ bool(value, false) }}'

Throw that somewhere on your dashboard, or alternatively tie it into some automation, and you’re good to go!

I attended RC3 from December 27th until December 30th. While it was (once again) only a virtual edition of the Chaos Communication Congress, at least this time around I managed to have a similar experience to 36c3, as in, I spent the last two days mostly hanging out with a bunch of fellow geeks in a fun location (a jitsi conference that also included a camera pointing at an aquarium full of fish) and nerding out while tinkering around with electronics.

And thus I finally integrated the CO2 sensor unit I bought a couple weeks ago into my Home Automation setup, with the help of a Wemos D1 Mini and ESPHome. At first I went with an ESP12, a voltage regulator and a different firmware, but that didn’t work out due to the ESP not wanting to behave (my guess is I didn’t wire the barebone module up correctly or the voltage regulator was causing issues) and I also got some weird readings reported by the firmware (20k ppm CO2 - I know the air in my office can get bad after a couple of hours of coding, but not THAT bad).

The CO2 sensor is an “AIRCO2NTROL MINI” from TFA Dostmann, but it is also available under other names with a very similar case and more or less the same internals, as I learned from the ESPHome docs. Where my Dostmann edition seems to differ from the majority is the pin order of the internal debug port, which turned out to have CLK and DATA swapped in my case, which caused me quite the headache and a bit of frustration. So just for future reference, the pin order I found in my device is 5V - Data - CLK - Gnd from left to right with the hose to the left:

I hooked these up to the Wemos D1 Mini (clone) like this:

So 5V to 5V, Gnd to Gnd, Data to D1 and CLK to D2.

The ESPHome config I then flashed to the D1 is the following:

esphome:
  name: dostmann-office
  platform: ESP8266
  board: d1_mini

logger:

mqtt:
  broker: !secret mqtt_iot_broker

ota:
  password: !secret ota_pass

wifi:
  ssid: !secret wifi_iot_ssid
  password: !secret wifi_iot_pass
  ap:
    ssid: "Dostmann Office Fallback"
    password: !secret fallback_pass
  power_save_mode: high

captive_portal:

sensor:
  - platform: zyaura
    clock_pin: D2
    data_pin: D1
    co2:
      name: "Office Dostmann CO2"
    temperature:
      name: "Office Dostmann Temperature"

  - platform: wifi_signal
    name: "Office Dostmann WiFi Signal"
    update_interval: 60s

(If you are wondering about the !secret stuff, those values are contained in a secret.yaml file in my esphome folder, and you can read all about that in the ESPHome docs here.)

And with that I could now see the sensor in my Home Assistant instance and forward the data easily to my InfluxDB & Grafana monitoring stack.

To put everything together physically, fully contained, I used this alternative backplate by Stefan Kern.

However, I’ve noticed a temperature increase of around 3°C with it and the ESP in place, and I fear this might be screwing with the CO2 sensor’s calibration (as the measurement is temperature sensitive). I already mitigated this a bit by setting the chip to power save and adding some strategically placed aluminium tape, but that’s only improved things slightly. Due to that I plan to redesign the backplate to have the ESP outside the sensor case, in its own compartment. I hope that will solve the “running hot” issue for good then, but we’ll see.

Update from January 27th 2022 I redesigned the backplate and now have a solution that seems to work better, based on the reported temperature and CO2. I’ve published it here.

In any case, for now I at least got a reliable indicator of my office’s CO2 levels that also now are trackable long term, and I have also forwarded the current values to my AWTRIX mini via some NodeRED flow that also takes care of color coding. Further possibilities include flashing the office lights or some audio cues, should just the visual warning turn out to be insufficient in the long term 😉

For a bit more than two years now I’ve been closely monitoring my network uplink. In the past I had a ton of issues with up- or download speeds not being what I paid for, packet loss issues and outright full blown outages. In order to put myself into a better position when reaching out to the ISP’s support hotline I figured it would be good to be able to proof not only the existence of these issues but to also be able to determine the exact times they happened at and also to verify and show that in fact it was only external connections that were suffering and it was not an issue with my own internal network. Given that I don’t trust the cable modem/router they force on me to be my edge router and instead have my own Unifi gear set up behind it (considering anything not exclusively under my control to be part of the hostile public internet) this otherwise will usually lead to endless attempts to blame my LAN when in fact the issue lies outside of my reach.

I already had an InfluxDB and Grafana setup running anyhow for my Home Assistant instance to dump values from my home climate sensors into, so it was a logical next step to simply add some additional sensors to the mix.

Throughput

I currently run a speed test of the network throughput every 20min and log the results via MQTT into InfluxDB. I had to find out that neither the speed test integration in Home Assistant nor the official speedtest-cli tool were performing reliably enough for this – I was constantly getting dips in measured throughput and thus alerts, even when everything was completely fine with my uplink.

I solved this by turning to speedtest-rs and a small shell script that parses the output and pushes it into MQTT to Home Assistant, which then processes it further for some visualization right on my dashboard but also forwards it further into InfluxDB. You can find the Dockerfile and the script plus some further info in this gist.

In Grafana I then use this data to provide me with some single stat panels for the current downstream, upstream and ping values as well as the averages over the selected time range:

Additionally, I also plot the down- and upstream speed in a timeline, together with the current bandwidth consumption as extracted by Home Assistant from my ISP’s cable modem/router (thanks to the Fritzbox NetMonitor integration). Together, this gives me a good picture of whether there is actually an issue when I see a dip in the measured values, or if it’s just too high bandwidth utilization:

You can see in these screenshots that I recently upgraded my plan with my ISP – from 200/20 to 500/50 MBit. The problem: The speedtest run by my monitoring setup doesn’t hit the 500 mark, whereas running a manual test on speedtest.net works just fine. Looking at the speedtest-rs README it becomes apparent that this is a known issue with the legacy (open) Speedtest.net API:

This tool currently only supports HTTP Legacy Fallback for testing.

High bandwidth connections higher than ~200Mbps may return incorrect results!

The testing operations are different from socket versions of tools connecting to speedtest.net infrastructure. In the many FOSS Go versions, tests are done to find an amount of data that can run for a default of 3 seconds over some TCP connection. In particular, speedtest-cli and speedtest-rs tests with what Ookla calls the “HTTP Legacy Fallback” for hosts that cannot establish a direct TCP connection.

I fear I might have to look into reimplementing the current speedtest-to-mqtt setup with another container utilizing the official (and sadly proprietary) Speedtest CLI tool to mitigate this issue. Thankfully, it should be quite easy to build a drop-in replacement thanks to the modularization in effect.

Update from March 30th 2021 I’ve now done that and here’s an updated gist that works identically to the speedtest-rs approach, but instead utilizes Ookla’s official command line tool. The results are stable numbers that reflect the expected bandwidth and also match the web based test results.

Update from March 31st, 2021 I wasn’t too happy with running a proprietary tool for my speed testing, went looking for an OSS alternative, came across librespeed and therefore have now replicated the setup again using that. You might want to experiment a bit to find a server close to you and define that via --server <id>, the auto discovery appears to be a bit wonky. Or just use your own server list via --server-json or --local-json.

Latency and packet loss

In addition to the available up- and downstream speeds, I constantly monitor latency and packet loss to a selected number of hosts both external and internal to my network as well. For this I ping some public DNS servers (Google, Cloudflare and Quadnine) and some of my own vservers for the remote side, and the ISP’s Fritzbox, my managed network gear and internal servers for the LAN side. I used to do this via Smokeping, but when I set up my InfluxDB/Grafana stack I wanted to find a solution to have everything together in one place.

Thankfully I almost immediately found this post by Tor Hveem who solved this with a little custom Go tool to run fping against a number of configurable hosts and push the results right into InfluxDB. This was exactly what I wanted and thus I replicated the outlined setup, albeit with a slightly different color scheme.

I use a modified version of Tor’s infping tool maintained by Nick Van Wiggeren and run that in a Docker container on my NAS. You can find everything needed to run this on your own in this gist.

As a result I get ping output for all hosts every 60 sec with times and packet loss information pushed right into InfluxDB. This is easily queried by Grafana and looks quite nice when visualized:

And on my network dashboard, I plot only the avg values across all hosts and a mean loss value into one single graph each for external and internal hosts:

This allows me a good overview of the current state of uplink and internal network at one glance.

Alerts

Since just graphs won’t give me an immediate heads-up when something goes wrong, I have a bunch of alerts set up in Grafana:

• Measured download speed falls beneath 250MBit for more than one hour
• Measured upload speed falls beneath 35MBit for more than one hour
• Mean packet loss across all external hosts rises above 25% for more than ten minutes

All of those trigger a notification to a private Discord server (via Grafana’s own notification mechanism). In theory this notification should even include a screenshot of the panel for which the alert was triggered for, but I’m having some problems with that still that I need to investigate.

This notification channel has an obvious problem: When the uplink goes out completely, I won’t get the notification if my phone is in my LAN. I really need to add a local alert as well at some point 😅

Still, it usually will give me a heads-up in time for me to reach out to my ISP on short notice and request they start troubleshooting.

Conclusion

This monitoring setup has proven valuable in debugging network performance issues and also getting an early heads-up about current ISP issues. I have successfully used screenshots for proving ongoing issues to my ISP, and also sped up the one or other troubleshooting session when there was in fact an issue with my LAN. In my book, that makes it absolutely worth the time it took me to set this up and maintain it. And: it kinda looks cool 😎

If you want to give this a go yourself, this might be of interest to you:

• Dockerfile, compose and instructions for speedtest container
  • Ookla speedtest based version
  • speedtest-rs based version
• Dockerfile, compose and instructions for infping container
• Panel JSON for the mentioned visualizations