92 lines
11 KiB
PHP
92 lines
11 KiB
PHP
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset='utf-8'>
|
|
<meta http-equiv="X-UA-Compatible" content="chrome=1">
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="zotan's blog"/>
|
|
<meta name="twitter:description" content="IPv6-native networking: a project report"/>
|
|
<link rel="stylesheet" href="/style.css">
|
|
<title>zotan's blog</title>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<div class="container">
|
|
<h1><a href="/blog">zotan's blog</a></h1>
|
|
<?php if($_SERVER['HTTP_HOST'] === "zotan.pw" || $_SERVER['HTTP_HOST'] === "zotan.pw.dev"): ?>
|
|
<h2 style="margin-bottom: 0px;">Welcome internet user.</h2>
|
|
<?php else: ?>
|
|
<h2 style="margin-bottom: 0px;">Welcome <?php echo($_SERVER['SERVER_PROTOCOL'])?> internet user.</h2>
|
|
<?php endif; ?>
|
|
<small><a href="/blog">Back to blog overview</a></small>
|
|
</div>
|
|
</header>
|
|
<div class="container">
|
|
<section id="main_content">
|
|
<b>2021-08-23</b> - a 10 minute read (150 wpm)
|
|
<h1 id="post">IPv6-native networking: a project report</h1>
|
|
<div align="justify">
|
|
<p>
|
|
If you have reached this post, chances are you already know my <a href="https://as211579.net" target="_blank">AS211579</a> project. This post serves as a summary of the things I learnt and the roadblocks I had to overcome on the way to get the network to its current state.
|
|
</p>
|
|
<p>
|
|
<h3>Goals and setup</h3>
|
|
For those who don't know the project, here's a quick recap. After having dabbled a bit with <a href="https://dn42.eu" target="_blank">DN42</a> last year during the first lockdown, I wanted to do the real thing, in the same global routing system your ISP is using to reach this very web server. While this sounds unnecessarily convoluted and complicated to accomplish, it was actually pretty easy and not that expensive. The easy part is almost completely due to my knowledge gained by interacting with DN42 and the awesome people in #dn42 on the <a href="https://hackint.org" target="_blank">hackint</a> IRC, who have helped me with debugging the stupidest of mistakes. If you're interested in any of the things I'm about to talk about, be sure to check out the community behind it, it's seriously amazing.
|
|
</p>
|
|
<p>
|
|
Alright, we've established the goals, where do we go from here? I quickly found a sponsoring AS (something you need for the <i>cheap</i> part) and had all the documents on the way to RIPE (the regional internet registry responsible for, among others, Europe and Asia). Once all that was processed, a nice person from one of the many network group chats I'm in (Wim, if you are reading this, thank you so much), hooked me up with a free /40 IPv6 subnet for all my routing needs. Next, I got BGP VMs. Good places to get them are either <a href="https://vultr.com" target="_blank">Vultr</a> (cloud provider) or various smaller providers like <a href="https://ifog.ch" target="_blank">iFog</a>. Peering mostly happened on <a href="https://locix.online" target="_blank">LocIX</a>. From there, I provided connectivity to home routers, laptops and other computer-y devices via WireGuard.
|
|
</p>
|
|
<p>
|
|
<h3>Going a step further</h3>
|
|
Once my projects (mostly RPKI) eclipsed the performance level provided by the VMs, I contacted the nice people at <a href="https://meerfarbig.net" target="_blank">Meerfarbig</a> for a dedicated machine. That one is the primary server running the network to this day. If you are trying to set up a similar thing and are looking for specifics, feel free to <a href="/#contact">contact me</a> and I will give you appropriate resources.
|
|
</p>
|
|
<p>
|
|
<h3>The trials and tribulations of networking without IPv4</h3>
|
|
Now for the fun part, and the likely reason you are here in the first place: all the things that broke along the way. You see, as you might be able to tell from the title, my project goal was to set up an IPv6-native network. That implies NAT64, DNS64, 464XLAT and a whole bag of other fun things. But let's start at the beginning.
|
|
</p>
|
|
<p>
|
|
When you have IPv6-only networks, especially when talking about eyeball networks (the kind mostly used for content consumption, e.g. viewing webpages and their content), you will want a way to reach IPv4-only servers. Many popular websites still presently don't support IPv6. At time of writing, this includes GitHub, which is fairly important for developing things. A less important (but still relevant) example is Reddit. For now, we can't reach those websites. What do we do from here?
|
|
</p>
|
|
<p>
|
|
Our (first) solution is called NAT64, which translates packets between IPv6 and IPv4, hence the name. The software I chose for this task is <a href="https://jool.mx" target="_blank">Jool</a>. Setting it up was fairly trivial, and I quickly set up three redundant NAT64-gateways that announce the NAT64-WKP (well-known prefix, <span class="highlight-bg">64:ff9b::/96</span>). So far so good, but how do we get our systems to actually <i>use</i> those gateways?
|
|
</p>
|
|
<p>
|
|
For that, we need to look at DNS, which is responsible for translating our domain names (e.g. <span class="highlight-bg">github.com</span>) to an address we can connect to (e.g. <span class="highlight-bg">140.82.121.3</span>). If we configure the resolver that does that lookup to synthesize an AAAA record for our IPv4-only domain, we can connect to it! And that's what I'm doing. Using <a href="https://www.nlnetlabs.nl/projects/unbound/about/" target="_blank">unbound</a>, the address <span class="highlight-bg">140.82.121.3</span> (from the real A record) is translated to <span class="highlight-bg">64:ff9b::140.82.121.3</span>, or <span class="highlight-bg">64:ff9b::8c52:7903</span> in encoded form, and returned as a synthesized AAAA record. Once we configure our system to use this resolver, <i>most</i> IPv4-only sites work perfectly! If you are asking why I said most there, I hope you are in for a ride.
|
|
</p>
|
|
<p>
|
|
<h3>464XLAT and questioning whether connecting computers together was a good idea</h3>
|
|
The answer to that question is no, obviously. And it keeps chipping away at <a href="https://xkcd.com/2259/" target="_blank">my sanity</a>. But since we're here, I might as well roll with it. So, what does 464XLAT even mean? It's a specific combination of systems in place to ensure connectivity to IPv4 hosts from IPv6-only networks. Namely, in addition to DNS64 and the NAT64 gateway (called PLAT in this setup, provider-side address translator), we need a second piece of software, the CLAT (client/customer-side address translator).
|
|
</p>
|
|
<p>
|
|
Why, you ask? Because lots of rather popular software (for example, Skype and Spotify) not only use IPv4, but hardcoded IPv4 literals. That means that instead of connecting to <span class="highlight-bg">somedomain.tld</span>, the software tries to connect to <span class="highlight-bg">192.0.2.255</span>, which will fail without a CLAT, since we are only capturing (and synthesizing AAAA records for) DNS queries. A CLAT will take that packet and translate it to an IPv6 packet destined for the DNS64-synthesized address of the target host, the same one the resolver would have synthesized, had we not used literals.
|
|
</p>
|
|
<p>
|
|
Okay, sounds simple enough, how do we do this? Most mobile operating systems (Android and iOS) and some desktop operating systems (Windows) support this natively, though support outside of cellular connections is limited to non-existent. Since we are working with WireGuard here, this won't help us. The solution I used here is giving all clients a private IPv4 address, and instead running the CLAT on the router the tunnel terminates on. For compatibility reasons I use addresses from the prefix <span class="highlight-bg">100.64.0.0/10</span> meant for CGNAT (which is almost what we are doing here) for this purpose.
|
|
</p>
|
|
<p>
|
|
<h3>Trying to get it all to work</h3>
|
|
After the configuration part, getting NAT64 to run was fairly easy, despite some initial issues with routing the NAT64-WKP. Once I turned on 464XLAT however, everything broke. <span class="highlight-bg">::1</span> (the IPv6 loopback address) was unreachable. Traceroutes that shouldn't even have gone through the 464XLAT stopped working. I ran into bugs in Jool. Two fairly major ones, to be exact. However, the developers were very helpful in debugging the problems, and got both of them resolved within about two months (shoutouts to ydahhrk!). If you intend on deploying a similar setup, I recommend going for the -git version, since those bugs are fixed there already.
|
|
</p>
|
|
<p>
|
|
<h3>One more thing</h3>
|
|
One last roadblock was wireguard-quick for macOS. For those unfamiliar with these issues, I'm glad you didn't go through that debugging rabbit hole. To start with, if you are tunneling all of your traffic and your WiFi or Ethernet connection doesn't support IPv6 while your tunnel does, macOS will sometimes decide to be smart and not attempt to request AAAA records at all, thereby making the IPv6 connectivity of the tunnel redundant. To work around this issue I created a very dodgy-looking script that is run post-up by wireguard-quick, which creates a custom network service for the tunnel interface. If you have this problem and want this script, please <a href="/#contact">contact me</a> directly as I don't feel comfortable publishing something that terrible on my website.
|
|
</p>
|
|
<p>
|
|
Back to wireguard-quick, though. It turns out that the macOS version is particularly dodgy, since the bypass for the "we only have one routing table to work with" problem the devs went for is adding a more specific override route for the tunnel endpoint, which depends on parsing unchecked output of commands that print the routing table to determine the default gateway. For reasons I can't explain, this breaks when you create a custom network service as mentioned above, and it tries to set the default gateway to <span class="highlight-bg">link#32</span> or similar. This fails, and therefore the tunnel breaks, as tunnel traffic is then routed back through the tunnel in an infinite loop. After contacting the devs in the #wireguard channel on <a href="https://libera.chat" target="_blank">libera.chat</a>, my patch was accepted and this specific problem shouldn't occur anymore, though that doesn't change the bodge that is the route monitor code of wg-quick.
|
|
</p>
|
|
<p>
|
|
<h3>Epilogue</h3>
|
|
That concludes the network setup. Everything is running smoothly and thus far, no further bugs were found. I provide IPv6-tunnels for a few friends who haven't reported any problems either, so at least for now I think that this project is complete.
|
|
</p>
|
|
<p>
|
|
I have already removed IPv4 addresses from a few services I run, and I hope to do so for the entirety of my online presence by the end of 2021, maybe with a few exceptions for critical services used by friends from Austria where major telcos still don't support IPv6.
|
|
|
|
</p>
|
|
<p>
|
|
Maybe the end of IPv4 is actually near, at least in my small corner of the internet. Thanks for reading, and have a wonderful day.
|
|
</p>
|
|
</div>
|
|
<p style="color: #666666"><i>This request was served by <?php echo gethostname() ?> in <?php echo(round((microtime(true) - $_SERVER["REQUEST_TIME_FLOAT"])*1000, 2)) ?> ms<i></p>
|
|
</div>
|
|
</body>
|
|
</html>
|