177 lines
9.4 KiB
Markdown
177 lines
9.4 KiB
Markdown
---
|
|
title: "IPv6-native networking: a project report"
|
|
date: 2021-08-23
|
|
...
|
|
If you have reached this post, chances are you already know my [AS211579](https://zotan.network){target="_blank"}
|
|
project.
|
|
This post serves as a summary of the things I learnt and the roadblocks
|
|
I had to overcome on the way to get the network to its current state.
|
|
|
|
### Goals and setup
|
|
|
|
For those who don't know the project, here's a quick recap. After having
|
|
dabbled a bit with [DN42](https://dn42.eu){target="_blank"} last year during the first lockdown, I wanted to do the real
|
|
thing, in
|
|
the same global routing system your ISP is using to reach this very web
|
|
server. While this sounds unnecessarily convoluted and complicated to
|
|
accomplish, it was actually pretty easy and not that expensive. The easy
|
|
part is almost completely due to my knowledge gained by interacting with
|
|
DN42 and the awesome people in #dn42 on the [hackint](https://hackint.org){target="_blank"} IRC, who have
|
|
helped me with debugging the stupidest of mistakes. If you're interested
|
|
in any of the things I'm about to talk about, be sure to check out the
|
|
community behind it, it's seriously amazing.
|
|
|
|
Alright, we've established the goals, where do we go from here? I
|
|
quickly found a sponsoring AS (something you need for the *cheap* part)
|
|
and had all the documents on the way to RIPE (the regional internet
|
|
registry responsible for, among others, Europe and Asia). Once all that
|
|
was processed, a nice person from one of the many network group chats
|
|
I'm in (Wim, if you are reading this, thank you so much), hooked me up
|
|
with a free /40 IPv6 subnet for all my routing needs. Next, I got BGP
|
|
VMs. Good places to get them are either
|
|
[Vultr](https://vultr.com){target="_blank"} (cloud provider)
|
|
or various smaller providers like
|
|
[iFog](https://ifog.ch){target="_blank"}. Peering mostly
|
|
happened on [LocIX](https://locix.online){target="_blank"}.
|
|
From there, I provided connectivity to home routers, laptops and other
|
|
computer-y devices via WireGuard.
|
|
|
|
### Going a step further
|
|
|
|
Once my projects (mostly RPKI) eclipsed the performance level provided
|
|
by the VMs, I contacted the nice people at
|
|
[Meerfarbig](https://meerfarbig.net){target="_blank"} for a
|
|
dedicated machine. That one is the primary server running the network to
|
|
this day. If you are trying to set up a similar thing and are looking
|
|
for specifics, feel free to [contact me](/#contact) and I will give you
|
|
appropriate resources.
|
|
|
|
### The trials and tribulations of networking without IPv4
|
|
|
|
Now for the fun part, and the likely reason you are here in the first
|
|
place: all the things that broke along the way. You see, as you might be
|
|
able to tell from the title, my project goal was to set up an
|
|
IPv6-native network. That implies NAT64, DNS64, 464XLAT and a whole bag
|
|
of other fun things. But let's start at the beginning.
|
|
|
|
When you have IPv6-only networks, especially when talking about eyeball
|
|
networks (the kind mostly used for content consumption, e.g. viewing
|
|
webpages and their content), you will want a way to reach IPv4-only
|
|
servers. Many popular websites still presently don't support IPv6. At
|
|
time of writing, this includes GitHub, which is fairly important for
|
|
developing things. A less important (but still relevant) example is
|
|
Reddit. For now, we can't reach those websites. What do we do from here?
|
|
|
|
Our (first) solution is called NAT64, which translates packets between
|
|
IPv6 and IPv4, hence the name. The software I chose for this task is
|
|
[Jool](https://jool.mx){target="_blank"}. Setting it up was
|
|
fairly trivial, and I quickly set up three redundant NAT64-gateways that
|
|
announce the NAT64-WKP (well-known prefix, <span
|
|
class="highlight-bg">64:ff9b::/96</span>). So far so good, but how do we
|
|
get our systems to actually *use* those gateways?
|
|
|
|
For that, we need to look at DNS, which is responsible for translating
|
|
our domain names (e.g. <span class="highlight-bg">github.com</span>) to
|
|
an address we can connect to (e.g. <span
|
|
class="highlight-bg">140.82.121.3</span>). If we configure the resolver
|
|
that does that lookup to synthesize an AAAA record for our IPv4-only
|
|
domain, we can connect to it! And that's what I'm doing. Using
|
|
<a href="https://www.nlnetlabs.nl/projects/unbound/about/"
|
|
target="_blank">unbound</a>, the address <span
|
|
class="highlight-bg">140.82.121.3</span> (from the real A record) is
|
|
translated to <span class="highlight-bg">64:ff9b::140.82.121.3</span>,
|
|
or <span class="highlight-bg">64:ff9b::8c52:7903</span> in encoded form,
|
|
and returned as a synthesized AAAA record. Once we configure our system
|
|
to use this resolver, *most* IPv4-only sites work perfectly! If you are
|
|
asking why I said most there, I hope you are in for a ride.
|
|
|
|
### 464XLAT and questioning whether connecting computers together was a good idea
|
|
|
|
The answer to that question is no, obviously. And it keeps chipping away
|
|
at [my sanity](https://xkcd.com/2259/){target="_blank"}. But
|
|
since we're here, I might as well roll with it. So, what does 464XLAT
|
|
even mean? It's a specific combination of systems in place to ensure
|
|
connectivity to IPv4 hosts from IPv6-only networks. Namely, in addition
|
|
to DNS64 and the NAT64 gateway (called PLAT in this setup, provider-side
|
|
address translator), we need a second piece of software, the CLAT
|
|
(client/customer-side address translator).
|
|
|
|
Why, you ask? Because lots of rather popular software (for example,
|
|
Skype and Spotify) not only use IPv4, but hardcoded IPv4 literals. That
|
|
means that instead of connecting to <span
|
|
class="highlight-bg">somedomain.tld</span>, the software tries to
|
|
connect to <span class="highlight-bg">192.0.2.255</span>, which will
|
|
fail without a CLAT, since we are only capturing (and synthesizing AAAA
|
|
records for) DNS queries. A CLAT will take that packet and translate it
|
|
to an IPv6 packet destined for the DNS64-synthesized address of the
|
|
target host, the same one the resolver would have synthesized, had we
|
|
not used literals.
|
|
|
|
Okay, sounds simple enough, how do we do this? Most mobile operating
|
|
systems (Android and iOS) and some desktop operating systems (Windows)
|
|
support this natively, though support outside of cellular connections is
|
|
limited to non-existent. Since we are working with WireGuard here, this
|
|
won't help us. The solution I used here is giving all clients a private
|
|
IPv4 address, and instead running the CLAT on the router the tunnel
|
|
terminates on. For compatibility reasons I use addresses from the prefix
|
|
<span class="highlight-bg">100.64.0.0/10</span> meant for CGNAT (which
|
|
is almost what we are doing here) for this purpose.
|
|
|
|
### Trying to get it all to work
|
|
|
|
After the configuration part, getting NAT64 to run was fairly easy,
|
|
despite some initial issues with routing the NAT64-WKP. Once I turned on
|
|
464XLAT however, everything broke. <span class="highlight-bg">::1</span>
|
|
(the IPv6 loopback address) was unreachable. Traceroutes that shouldn't
|
|
even have gone through the 464XLAT stopped working. I ran into bugs in
|
|
Jool. Two fairly major ones, to be exact. However, the developers were
|
|
very helpful in debugging the problems, and got both of them resolved
|
|
within about two months (shoutouts to ydahhrk!). If you intend on
|
|
deploying a similar setup, I recommend going for the -git version, since
|
|
those bugs are fixed there already.
|
|
|
|
### One more thing
|
|
|
|
One last roadblock was wireguard-quick for macOS. For those unfamiliar
|
|
with these issues, I'm glad you didn't go through that debugging rabbit
|
|
hole. To start with, if you are tunneling all of your traffic and your
|
|
WiFi or Ethernet connection doesn't support IPv6 while your tunnel does,
|
|
macOS will sometimes decide to be smart and not attempt to request AAAA
|
|
records at all, thereby making the IPv6 connectivity of the tunnel
|
|
redundant. To work around this issue I created a very dodgy-looking
|
|
script that is run post-up by wireguard-quick, which creates a custom
|
|
network service for the tunnel interface. If you have this problem and
|
|
want this script, please [contact me](/#contact) directly as I don't
|
|
feel comfortable publishing something that terrible on my website.
|
|
|
|
Back to wireguard-quick, though. It turns out that the macOS version is
|
|
particularly dodgy, since the bypass for the "we only have one routing
|
|
table to work with" problem the devs went for is adding a more specific
|
|
override route for the tunnel endpoint, which depends on parsing
|
|
unchecked output of commands that print the routing table to determine
|
|
the default gateway. For reasons I can't explain, this breaks when you
|
|
create a custom network service as mentioned above, and it tries to set
|
|
the default gateway to <span class="highlight-bg">link#32</span> or
|
|
similar. This fails, and therefore the tunnel breaks, as tunnel traffic
|
|
is then routed back through the tunnel in an infinite loop. After
|
|
contacting the devs in the #wireguard channel on
|
|
[libera.chat](https://libera.chat){target="_blank"}, my patch
|
|
was accepted and this specific problem shouldn't occur anymore, though
|
|
that doesn't change the bodge that is the route monitor code of
|
|
wg-quick.
|
|
|
|
### Epilogue
|
|
|
|
That concludes the network setup. Everything is running smoothly and
|
|
thus far, no further bugs were found. I provide IPv6-tunnels for a few
|
|
friends who haven't reported any problems either, so at least for now I
|
|
think that this project is complete.
|
|
|
|
I have already removed IPv4 addresses from a few services I run, and I
|
|
hope to do so for the entirety of my online presence by the end of 2021,
|
|
maybe with a few exceptions for critical services used by friends from
|
|
Austria where major telcos still don't support IPv6.
|
|
|
|
Maybe the end of IPv4 is actually near, at least in my small corner of
|
|
the internet. Thanks for reading, and have a wonderful day.
|