The websites in question are actually on GitHub Pages and my servers are just acting as reverse proxy to resolve domain and TLS, and a look into the web server’s error.log showed that the issue in serving was that the server could no longer reach its upstream at github.io when Tailscale was active. It wasn’t a general loss of external connectivity though - IPv4 addresses still worked great, the webserver however was trying to connect to the upstream via IPv6 and this is where things failed.

I did some quick tests, pinging Google’s DNS on both IPv4 (8.8.8.8) and IPv6 (2001:4860:4860::8888) with Tailscale running, and that showed that while Tailscale was running, IPv6 connectivity just broke down completely while IPv4 continued to work:

$ sudo tailscale up && ping -c 3 8.8.8.8 && sudo tailscale down
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=3.98 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=3.69 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=3.63 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 3.631/3.771/3.986/0.162 ms

$ sudo tailscale up && ping -c 3 2001:4860:4860::8888 && sudo tailscale down
PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes

--- 2001:4860:4860::8888 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2051ms

As a next step, I checked the route that would be taken for resolving 2001:4860:4860::8888 with and without Tailscale:

$ ip route get 2001:4860:4860::8888
2001:4860:4860::8888 from :: via fe80::1 dev eth0 src aaaa:bbbb:cccc:dddd::1 metric 1024 pref medium

$ sudo tailscale up && ip route get 2001:4860:4860::8888 && sudo tailscale down
2001:4860:4860::8888 from :: via fe80::1 dev eth0 src fd7a:115c:a1e0::aaaa:bbbb metric 1024 pref medium

So, the issue was that for some reason, while Tailscale was running the system decided to use Tailscale’s internal IPv6 fd7a:115c:a1e0::aaaa:bbbb set on tailscale0 as the source IP but send the packet through the default route and eth0, and that didn’t work. It basically hijacked the IPv6 traffic, even when the Tailnet wasn’t even involved.

The routes looked fine to me and some buddies I asked also didn’t spot anything amiss:

$ ip -6 rule show
0:      from all lookup local
5210:   from all fwmark 0x80000/0xff0000 lookup main
5230:   from all fwmark 0x80000/0xff0000 lookup default
5250:   from all fwmark 0x80000/0xff0000 unreachable
5270:   from all lookup 52
32766:  from all lookup main

$ ip -6 route show table local
local ::1 dev lo proto kernel metric 0 pref medium
local aaaa:bbbb:cccc:dddd::1 dev eth0 proto kernel metric 0 pref medium
local fd7a:115c:a1e0::aaaa:bbbb dev tailscale0 proto kernel metric 0 pref medium
local fe80::5fdc:58a7:1a93:da5a dev tailscale0 proto kernel metric 0 pref medium
local fe80::9400:ff:fe0d:61a1 dev eth0 proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 proto kernel metric 256 pref medium
multicast ff00::/8 dev tailscale0 proto kernel metric 256 pref medium

$ ip -6 route show table 52
fd7a:115c:a1e0::53 dev tailscale0 metric 1024 pref medium
fd7a:115c:a1e0::/48 dev tailscale0 metric 1024 pref medium

$ ip -6 route show table main
aaaa:bbbb:cccc:dddd::/64 dev eth0 proto kernel metric 256 pref medium
fd7a:115c:a1e0::aaaa:bbbb dev tailscale0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::1 dev eth0 metric 1024 pref medium

From what I could see, firing up Tailscale would add the following new rules to the routing table:

$ ip route show table all > ts-off.txt
$ sudo tailscale up && ip route show table all > ts-on.txt && sudo tailscale down
$ diff ts-off.txt ts-on.txt
0a1,2
> 100.a.b.c dev tailscale0 table 52
> 100.100.100.100 dev tailscale0 table 52
3a6
> local 100.x.y.z dev tailscale0 table local proto kernel scope host src 100.x.y.z
12a16,17
> fd7a:115c:a1e0::53 dev tailscale0 table 52 metric 1024 pref medium
> fd7a:115c:a1e0::/48 dev tailscale0 table 52 metric 1024 pref medium
13a19
> fd7a:115c:a1e0::aaaa:bbbb dev tailscale0 proto kernel metric 256 pref medium
17a24
> local fd7a:115c:a1e0::aaaa:bbbb dev tailscale0 table local proto kernel metric 0 pref medium

I still have not figured out what is actually going on there, and a reproduction on a fresh server so far also wasn’t successful. The problem is that packets are being sent with the wrong source IPv6, but that’s just a symptom of the underlying cause.

Thankfully, my buddy Jub came up with the workaround to change the default route to use a fixed IPv6 source address - the correct one - and that solved the issue (by fixing the symptom):

ip -6 route replace default via fe80::1 dev eth0 src aaaa:bbbb:cccc:dddd::1

I put that on a new post-up line into the network setup in /etc/network/interface.d/50-cloud-init.cfg

auto eth0:0
iface eth0:0 inet6 static
    address aaaa:bbbb:cccc:dddd::/64
    gateway fe80::1
    post-up route add -net :: netmask 0 gw fe80::1%eth0 || true
    post-up ip -6 route replace default via fe80::1 dev eth0 src aaaa:bbbb:cccc:dddd::1 || true
    pre-down route del -net :: netmask 0 gw fe80::1%eth0 || true

A reboot confirmed that this works as a workaround.

But as I mentioned, I still can’t make any sense of the underlying issue. I found one open bug report in Tailscale’s bug tracker that sounded familiar, but it didn’t fully match my situation. I also have to admit that my administration skills kinda get a bit fuzzy when it comes to full blown route analysis & debugging - so should you have any ideas at all what is actually causing this behaviour here, please get in touch on Mastodon - I’d love to see this mystery solved, but am out of my depth here 😅

ssh admin@192.168.1.1 'sudo tcpdump -f -i eth1 -w - src 192.168.1.12' | wireshark -k -i - 

I could confirm that this works and created a small script to make it easier to use by throwing this into ~/.local/bin/gatedump²:

#!/bin/bash

ARGS="$@"
TCPDUMP="sudo tcpdump -f -w - $ARGS"

ssh usg "$TCPDUMP" | wireshark -k -i -

This now allows me to easily run tcpdump remotely with custom arguments, e.g. gatedump -i eth1 host 192.168.1.123, and have it fire up Wireshark automatically. Wish me luck I’ll now be able to figure out what’s going on on my network, because it’s driving me up the wall.

Update from 2023-12-06: In case you are wondering how this story ended, the issue resolved itself with the next OS update of my partner’s iPhone. So whatever caused it, it’s gone now, and I hope for good.

I already had an InfluxDB and Grafana setup running anyhow for my Home Assistant instance to dump values from my home climate sensors into, so it was a logical next step to simply add some additional sensors to the mix.

Throughput

I currently run a speed test of the network throughput every 20min and log the results via MQTT into InfluxDB. I had to find out that neither the speed test integration in Home Assistant nor the official speedtest-cli tool were performing reliably enough for this – I was constantly getting dips in measured throughput and thus alerts, even when everything was completely fine with my uplink.

I solved this by turning to speedtest-rs and a small shell script that parses the output and pushes it into MQTT to Home Assistant, which then processes it further for some visualization right on my dashboard but also forwards it further into InfluxDB. You can find the Dockerfile and the script plus some further info in this gist.

In Grafana I then use this data to provide me with some single stat panels for the current downstream, upstream and ping values as well as the averages over the selected time range:

Additionally, I also plot the down- and upstream speed in a timeline, together with the current bandwidth consumption as extracted by Home Assistant from my ISP’s cable modem/router (thanks to the Fritzbox NetMonitor integration). Together, this gives me a good picture of whether there is actually an issue when I see a dip in the measured values, or if it’s just too high bandwidth utilization:

You can see in these screenshots that I recently upgraded my plan with my ISP – from 200/20 to 500/50 MBit. The problem: The speedtest run by my monitoring setup doesn’t hit the 500 mark, whereas running a manual test on speedtest.net works just fine. Looking at the speedtest-rs README it becomes apparent that this is a known issue with the legacy (open) Speedtest.net API:

This tool currently only supports HTTP Legacy Fallback for testing.

High bandwidth connections higher than ~200Mbps may return incorrect results!

The testing operations are different from socket versions of tools connecting to speedtest.net infrastructure. In the many FOSS Go versions, tests are done to find an amount of data that can run for a default of 3 seconds over some TCP connection. In particular, speedtest-cli and speedtest-rs tests with what Ookla calls the “HTTP Legacy Fallback” for hosts that cannot establish a direct TCP connection.

I fear I might have to look into reimplementing the current speedtest-to-mqtt setup with another container utilizing the official (and sadly proprietary) Speedtest CLI tool to mitigate this issue. Thankfully, it should be quite easy to build a drop-in replacement thanks to the modularization in effect.

Update from March 30th 2021 I’ve now done that and here’s an updated gist that works identically to the speedtest-rs approach, but instead utilizes Ookla’s official command line tool. The results are stable numbers that reflect the expected bandwidth and also match the web based test results.

Update from March 31st, 2021 I wasn’t too happy with running a proprietary tool for my speed testing, went looking for an OSS alternative, came across librespeed and therefore have now replicated the setup again using that. You might want to experiment a bit to find a server close to you and define that via --server <id>, the auto discovery appears to be a bit wonky. Or just use your own server list via --server-json or --local-json.

Latency and packet loss

In addition to the available up- and downstream speeds, I constantly monitor latency and packet loss to a selected number of hosts both external and internal to my network as well. For this I ping some public DNS servers (Google, Cloudflare and Quadnine) and some of my own vservers for the remote side, and the ISP’s Fritzbox, my managed network gear and internal servers for the LAN side. I used to do this via Smokeping, but when I set up my InfluxDB/Grafana stack I wanted to find a solution to have everything together in one place.

Thankfully I almost immediately found this post by Tor Hveem who solved this with a little custom Go tool to run fping against a number of configurable hosts and push the results right into InfluxDB. This was exactly what I wanted and thus I replicated the outlined setup, albeit with a slightly different color scheme.

I use a modified version of Tor’s infping tool maintained by Nick Van Wiggeren and run that in a Docker container on my NAS. You can find everything needed to run this on your own in this gist.

As a result I get ping output for all hosts every 60 sec with times and packet loss information pushed right into InfluxDB. This is easily queried by Grafana and looks quite nice when visualized:

And on my network dashboard, I plot only the avg values across all hosts and a mean loss value into one single graph each for external and internal hosts:

This allows me a good overview of the current state of uplink and internal network at one glance.

Alerts

Since just graphs won’t give me an immediate heads-up when something goes wrong, I have a bunch of alerts set up in Grafana:

• Measured download speed falls beneath 250MBit for more than one hour
• Measured upload speed falls beneath 35MBit for more than one hour
• Mean packet loss across all external hosts rises above 25% for more than ten minutes

All of those trigger a notification to a private Discord server (via Grafana’s own notification mechanism). In theory this notification should even include a screenshot of the panel for which the alert was triggered for, but I’m having some problems with that still that I need to investigate.

This notification channel has an obvious problem: When the uplink goes out completely, I won’t get the notification if my phone is in my LAN. I really need to add a local alert as well at some point 😅

Still, it usually will give me a heads-up in time for me to reach out to my ISP on short notice and request they start troubleshooting.

Conclusion

This monitoring setup has proven valuable in debugging network performance issues and also getting an early heads-up about current ISP issues. I have successfully used screenshots for proving ongoing issues to my ISP, and also sped up the one or other troubleshooting session when there was in fact an issue with my LAN. In my book, that makes it absolutely worth the time it took me to set this up and maintain it. And: it kinda looks cool 😎

If you want to give this a go yourself, this might be of interest to you:

• Dockerfile, compose and instructions for speedtest container
  • Ookla speedtest based version
  • speedtest-rs based version
• Dockerfile, compose and instructions for infping container
• Panel JSON for the mentioned visualizations