Mystery IPv6 woes!
During my work day, Sabriena came in and mentioned off-hand that sometimes her Android phone’s apps like Instagram etc would cease working. She’d put it onto 4G (ie, off the Wifi) and back on and they’d be working again.
This sounded interesting, so I started digging into it: it looked like it might be MTU related as I was having the same grief on Twitter at times - images would cease to load for a few minutes, then start up again. Always with images!
So I spent a little time after work trying to diagnose it, and ended up catching the problematic host in action, and it turned out to be IPv6 related! My phone would spend some time waiting for the IPv6 connection to time out, then switch to IPv4 and it would continue working just fine. Eventually something would cause it to try v6 again and it’d stop working briefly once more.
Indeed, here’s the IPv6 host for pbs.twimg.com seemingly ignoring my SYN packets from my Android phone (local IPv6 addresses munged to protect the guilty):
15:01:02.857099 IP6 fe80:b33f:00:2:5cc3:f93b:d957:fd62.49294 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [S], seq 3343625008, win 65535, options [mss 1440,sackOK,TS val 2973328381 ecr 0,nop,wscale 8], length 0
15:01:03.869739 IP6 fe80:b33f:00:2:5cc3:f93b:d957:fd62.49294 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [S], seq 3343625008, win 65535, options [mss 1440,sackOK,TS val 2973329393 ecr 0,nop,wscale 8], length 0
15:01:04.416376 IP6 fe80:b33f:00:2:5cc3:f93b:d957:fd62.49306 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [S], seq 1853496736, win 65535, options [mss 1440,sackOK,TS val 2973329940 ecr 0,nop,wscale 8], length 0
15:01:04.425905 IP6 fe80:b33f:00:2:5cc3:f93b:d957:fd62.49308 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [S], seq 146369347, win 65535, options [mss 1440,sackOK,TS val 2973329949 ecr 0,nop,wscale 8], length 0
and here’s the exact same server, happily chatting with my work Macbook:
15:03:21.484743 IP6 fe80:b33f:00:1:4953:02c0:2dbb:adc0.60072 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [S], seq 4213609127, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 601477456 ecr 0,sackOK,eol], length 0
15:03:21.501038 IP6 2606:2800:248:1347:709:24f:182c:618.443 > fe80:b33f:00:1:4953:02c0:2dbb:adc0.60072: Flags [S.], seq 1258438470, ack 4213609128, win 65535, options [mss 1220,sackOK,TS val 3124852147 ecr 601477456,nop,wscale 9], length 0
15:03:21.502155 IP6 fe80:b33f:00:1:4953:02c0:2dbb:adc0.60072 > 2606:2800:248:1347:709:24f:182c:618.443: Flags [.], ack 1, win 2057, options [nop,nop,TS val 601477474 ecr 3124852147], length 0
15:03:21.532010 IP6 2606:2800:248:1347:709:24f:182c:618.443 > fe80:b33f:00:1:4953:02c0:2dbb:adc0.60072: Flags [.], ack 233, win 131, options [nop,nop,TS val 3124852178 ecr 601477481], length 0
So what’s different here? Well right off the bat we can see that the window scaling is different, but the MSS is the same size. And I verified that packets up to about 1460 bytes can easily make it through my firewall, so it’s probably not MTU-related. But then in the process of uploading some .pcaps for anyone else to take a look at (and to probably pass along to my ISP as part of a ticket), I noted that my shell server couldn’t connect via IPv6 either!
This was a Linux device also, but I was not expecting to find an IPv6 bug in Linux that affected both Ubuntu and Android. Then the real kicker was my container host was able to connect fine, which eventually narrowed down the difference: the VLAN (see above in the packet captures, fe80:b33f:00:1 represents the work VLAN, while fe80:b33f:00:2 is the home VLAN).
So then I checked on my other Macbook, which is on the home VLAN - sure enough, I cannot SSH via IPv6, but pings of all sizes worked! I mentioned this on IRC and juha suggested perhaps it was a firewall issue…
Of course! Some time ago I noticed that my firewall to keep my kid’s Windows machine from seeing my work stuff allowed IPv6 through, so I blocked that… allowing ICMP though. That explains it, and sure enough, I mucked up on those rules and that was the cause of these periodic problems.
God damn it.
fwagglePublished:
Modified:
Filed under:
Location:
Horsham, VIC, AustraliaNavigation: Older Entry Newer Entry