This post will explore some lesser-known nuances of networking on AWS EC2:
- There is only unicast communication, therefore…
- There is no multicast or broadcast whatsoever!
- ARP and Neighbor Discovery replies are synthesized by AWS
- Manual IP configuration is possible – but don’t do this
- Alternatives for IP address failover
- IPv6 link-local addresses must not be randomized (no private addresses)
Should We Disable IPv6 DAD?
I started looking into AWS networking details because of an interesting question on Twitter:
Can anyone imagine a situation where they would want IPv6 Duplicate Address Detection to be operational in EC2?
Several questions arose in that thread:
- Is disabling IPv6 Duplicate Address Detection (DAD) on EC2 by default a good idea?
- Is it even possible to “have” a duplicate address on EC2?
- How do SLAAC and DHCPv6 interact, anyway?
- How does enabling DAD seem to help with slow failovers?
We’ll happily ignore the core question of why ISC dhclient fails on FreeBSD.
Instead we’ll play around and explore the lack of multicast on AWS, with no specific goal.
Let’s do a quick recap of networking basics. I’m guessing that lots of EC2 users are aware of IP basics, but less so of the magic glue below that. Feel free to skip if you know the “Ethernet things”.
For all our practical purposes here, our network layer – the IP protocols – runs on top of the IEEE 802 data link layer, used for pretty much every relevant physical layer from classic cable-bound Ethernet to Wi-Fi.
IEEE 802 originated from Ethernet, which is designed for a shared medium, meaning any machine can talk to any machine. Originally, all machines where hooked up to the same cable.
IEEE 802 Media Access Control uses 48-bit addresses (MAC addresses for short), usually displayed in hex format, e.g.
66:72:6e:45:12:e1. MAC addresses are literally everywhere – on your fancy Watch, on your TV, on your WiFi router, on your EC2 instances, and probably a bunch of them in your car.
Broadcasting means sending to all machines in the network, accomplished by sending to MAC address
ff:ff:ff:ff:ff:ff. Every machine on the local network listens to that address.
Multicasting means sending to a subset of machines, those that subscribed to a specific address. This concept exists only in IP protocols and is treated as a broadcast on Ethernet, but it does use different MAC addresses (
01:00:0e:... for IPv4 and
33:33:... for IPv6).
Recap: Address Resolution and IPv6 DAD
Alright, so IPv4 and IPv6 sit on top of that data link layer.
To send IP packets to another machine on the local network, MAC address resolution has to happen first.
To figure out which MAC address to send to, the sender will broadcast to all machines: Who has IP address XXX?, to which the receiver will reply with its MAC address. Only then can the sender start to send IP packets to the receiver’s MAC address.
This process is basically the same for IPv4 (ARP, Address Resolution Protocol) and IPv6 (Neighbor Discovery), IPv6 just uses a special multicast address instead of broadcast.
One of the features of IPv6 is mandatory Duplicate Address Detection (DAD). Whenever a new address is configured, an IPv6 machine (a node in IPv6 lingo) has to check if that address is already in use by another node.
It does so by using that same address resolution process: It asks this Who has? question for the IP address being configured – if some node responds to that, the address is obviously already in use.
No broadcast, no multicast!
As we’ve seen, multicast / broadcast are essential for any IP communication to take place.
Here’s the kicker about AWS networking: There is no broadcasting. There is no multicasting.
I’ve just recently discovered that. Spontaneously I cannot even find a clear piece of official documentation on that, the VPC FAQ doesn’t mention it either.
What you can find is an excellent piece on the AWS APN blog: Amazon VPC for On-Premises Network Engineers:
Routing and switching: This is the fun part. The re:Invent session Another Day, Another Billion Packets, which I mentioned at the beginning of this blog post, covers this topic in depth.
There are a few deviations network engineers will want to know about routing and switching in a VPC:
All traffic is unicast (the Address Resolution Protocol, or ARP, is gone too). The VPC forwarding design knows where all IP addresses are, and proxies ARP responses for all ARP requests locally at the hypervisor.
All subnets in a VPC have direct access to one another, unless they are filtered in the host or in security groups or network ACLs.
The re:Invent session video is really interesting. It also explains how ARP (and Neighbor Discovery) requests are handled by a Mapping Service.
IPv6 relies on several multicast mechanisms, so let’s see what happens.
IPv6 defines several multicast addresses, e.g.
ff02::1 for all hosts and
ff02::2 for all routers. Let’s try those.
In a conventional network, this works as expected:
Pinging all hosts yields many replies:
root@pooch:~ # ping6 ff02::1%wlan0 PING6(56=40+8+8 bytes) fe80::76da:38ff:fe6a:d72d%wlan0 --> ff02::1%wlan0 16 bytes from fe80::7[...]%wlan0, icmp_seq=0 hlim=64 time=0.665 ms 16 bytes from fe80::6[...]%wlan0, icmp_seq=0 hlim=64 time=13.595 ms(DUP!) 16 bytes from fe80::2[...]%wlan0, icmp_seq=0 hlim=64 time=17.098 ms(DUP!) 16 bytes from fe80::1[...]%wlan0, icmp_seq=0 hlim=64 time=108.220 ms(DUP!) 16 bytes from fe80::1[...]%wlan0, icmp_seq=0 hlim=64 time=240.956 ms(DUP!) 16 bytes from fe80::8[...]%wlan0, icmp_seq=0 hlim=64 time=262.192 ms(DUP!)
When pinging all routers, the local router dutifully responds:
root@pooch:~ # ping6 ff02::2%wlan0 PING6(56=40+8+8 bytes) fe80::76da:38ff:fe6a:d72d%wlan0 --> ff02::2%wlan0 16 bytes from fe80::[...]%wlan0, icmp_seq=0 hlim=64 time=11.548 ms
Pinging all hosts on AWS yields nothing, except for the pinging node that responds to itself:
root@n0-1c:~ # ping6 ff02::1%xn0 PING6(56=40+8+8 bytes) fe80::8ca:5ff:fe9c:fc14%xn0 --> ff02::1%xn0 16 bytes from fe80::8ca:5ff:fe9c:fc14%xn0, icmp_seq=0 hlim=64 time=0.066 ms
And also nothing for all routers, where you might expect the AWS VPC router to respond:
root@n0-1c:~ # ping6 ff02::2%xn0 PING6(56=40+8+8 bytes) fe80::8ca:5ff:fe9c:fc14%xn0 --> ff02::2%xn0 ^C --- ff02::2%xn0 ping6 statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss
So indeed, there is no multicast traffic between nodes.
Interestingly, sending a Router Solicitation to all routers, the VPC router replies, as seen in
root@n0-1c:~ # rtsol xn0 [...] fe80::8ca:5ff:fe9c:fc14 > ff02::2: ICMP6, router solicitation, length 16 [...] fe80::8a1:79ff:fedb:2106 > 2a05:d014:631:380c:7022:b490:22d9:95b8: ICMP6, router advertisement, length 56
… which leads me to believe that the VPC router does, in fact, receive packets on the all routers address (and maybe others), but simply drops most request types. I haven’t digged into that though.
Also noteworthy: In response to a Router Solicitation, the reply is sent to the assigned address (here:
2a05:d014:631:380c:7022:b490:22d9:95b8). That’s not even configured on this node (yet)!
The periodic unsolicited Router Advertisements go to the multicast all hosts instead:
fe80::8a1:79ff:fedb:2106 > ff02::1: ICMP6, router advertisement, length 56.
Duplicate Address Detection
If there is no multicast, then there’s no way that DAD can work, right? Other nodes will never see the Neighbor Solicitation for the new IP address. Let’s confirm this.
First, we’ll configure some address on one node:
root@n1-1c:~ # ifconfig xn0 inet6 2001:db8::2/64
Then we’ll configure the same address on another node (same Subnet) and see silence in tcpdump:
root@n0-1c:~ # tcpdump -enixn0 'ip6 and not icmp6[icmp6type]=icmp6-routeradvert' &  1126 root@n0-1c:~ # ifconfig xn0 inet6 2001:db8::2/64 [...] :: > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has 2001:db8::2, length 32
… and no reply. So indeed, DAD is absolutely pointless on EC2.
Manually configuring IP addresses
But is it possible to manually configure IP addresses, anyway? Well, of course you can configure them, but do they work? Because only if this works could a duplicate address ever become a problem at all.
Spoiler: It does work! There’s two hoops we need to jump through:
Since the Mapping Service does not know of our secret little IP addresses, it will not resolve them. We need to tell our machines about the others’ MAC addresses manually. That’s annoying, but it’s possible. For ad-hoc configurations, we can just add them on the fly using
ip neighbor). For permanent configuration, there’s the ancient
By default, the EC2 hypervisor only allows IP packets to be sent/received by instances when they match the addresses that the Mapping Service knows for them. In other words, it will block packets when it thinks those IP addresses are wrong. Fortunately, there’s many scenarios where you need to do this, e.g. running your own proxy / router / NAT instance, so EC2 provices the option to Disable Src/Dest Check (for both sending and receiving instances in our case).
Also keep in mind that the Security Groups need to allow traffic, as always.
So let’s configure an address on our first node, add some other nodes’ ether addresses and start pinging:
root@n0-1c:~ # ifconfig xn0 inet6 2001:db8::0/64 root@n0-1c:~ # ndp -s 2001:db8::1 0a:7b:67:a0:20:6a root@n0-1c:~ # ndp -s 2001:db8::2 06:48:58:f5:f1:8a root@n0-1c:~ # ping6 2001:db8::1 PING6(56=40+8+8 bytes) 2001:db8:: --> 2001:db8::1 [...] 2001:db8:: > 2001:db8::1: ICMP6, echo request, seq 0, length 16 [...] 2001:db8:: > 2001:db8::1: ICMP6, echo request, seq 1, length 16
… we’re sending pings to the correct MAC address, but no replies yet. We can even see those packets arriving in tcpdump on the target node, once we disable the Src/Dest Check.
We need to add the IP address as well as the MAC addresses (for the return packets!) on the second node:
root@n1-1c:~ # ifconfig xn0 inet6 2001:db8::1/64 root@n1-1c:~ # ndp -s 2001:db8::0 0a:ca:05:9c:fc:14 root@n1-1c:~ # ndp -s 2001:db8::2 06:48:58:f5:f1:8a
Et voilà! The first node successfully receives ping replies:
[...] 2001:db8:: > 2001:db8::1: ICMP6, echo request, seq 47, length 16 [...] 2001:db8::1 > 2001:db8::: ICMP6, echo reply, seq 47, length 16 16 bytes from 2001:db8::1, icmp_seq=47 hlim=64 time=0.352 ms
It’s always such a nice feeling when ping finally works…
… until you notice that something isn’t quite right yet. Let’s configure the third node, in a different Subnet:
root@n2-1b:~ # ifconfig xn0 inet6 2001:db8::2/64 root@n2-1b:~ # ndp -s 2001:db8::0 0a:ca:05:9c:fc:14 root@n2-1b:~ # ndp -s 2001:db8::1 0a:7b:67:a0:20:6a
root@n0-1c:~ # ping6 2001:db8::2 PING6(56=40+8+8 bytes) 2001:db8:: --> 2001:db8::2 [...]2001:db8:: > 2001:db8::2: ICMP6, echo request, seq 0, length 16 [...]2001:db8:: > 2001:db8::2: ICMP6, echo request, seq 1, length 16
So it seems that another reason for having different IP ranges on different Subnets is that you cannot have “direct” connections between machines in different Subnets (because for non-local destination IPs, the node will send to a router instead, in this case our VPC router). Interesting! By the way, it makes no difference if those Subnets are on the same Availability Zone or not.
Either way: Do not do this. Do not manually configure MAC addresses. It might be years from now, but someone will stab you if you do this. It’s like using
/etc/hosts, but a hundred times more nasty.
Mapping Service: Link-local addresses
The Mapping Service, which translates your IP address to the correct MAC address, works for global IPv6 addresses (e.g.
2a05:...) just as it does for IPv4 – it responds only to addresses it knows about, i.e. addresses that have been configured in EC2.
But here’s a cool thing: The Mapping Service will respond to any valid link-local address (as long as it’s EUI-64 based).
The link-local address (e.g.
fe80::...) is usually constructed with an Interface Identifier based on the MAC address. That means the MAC address for a given IPv6 link-local address can be easily calculated.
ndisc6 to generate some Neighbor Solicitations for made-up IP addresses:
root@n0-1c:~ # ndisc6 -q fe80::8ca:5ff:fe9c:fc02 xn0 0A:CA:05:9C:FC:02 root@n0-1c:~ # ndisc6 -q fe80::8ca:5ff:fe9c:fc03 xn0 0A:CA:05:9C:FC:03 root@n0-1c:~ # ndisc6 -q fe80::8ca:5ff:fe9c:fc04 xn0 0A:CA:05:9C:FC:04
As we can see, the Mapping Services generates the right MAC address for each address.
You may have noticed that this would cause DAD to malfunction: If the Mapping Service would respond to the node’s own address, the address would appear to be already in-use. Luckily, the Mapping Service does not respond for the sender’s own address:
root@n0-1c:~ # ndisc6 fe80::8ca:5ff:fe9c:fc14 xn0 Soliciting fe80::8ca:5ff:fe9c:fc14 (fe80::8ca:5ff:fe9c:fc14) on xn0... Timed out. Timed out. Timed out. No response.
If there’s only unicast traffic, it’s not possible to use some old friends – it’s mostly clustering and failover solutions that use multicast and broadcast to discover their peers. The Virtual Router Redundancy Protocol (VRRP) is popular – not only for routers, but for sets (usually pairs) of redundant systems as well, e.g. loadbalancers or reverse proxies.
You can’t use all that.
Alternatives for Failover
Some software might support unicast addressing. For example, you can run
unicast_peers (see this example).
You could also hack something up to re-assign an Elastic IP to the active instance.
Another interesting idea is to configure a dedicated Failover Virtual IP using the AWS API to control a Route Table entry
Interaction of SLAAC and DHCPv6
On AWS, SLAAC is used with the Router Advertisement Flag “Managed”, which tells hosts to retrieve their address and other configuration via DHCPv6. The Router Advertisement is also important to know about the default router to be used.
How Does DAD Help with Faster Failover?
I have no idea. The tweet mentions “certain switches/routers”, so I’ll assume the experience was not on AWS. And on non-AWS I can at least remotely imagine that DAD causes recipients (or switches along the way) to unclog their MAC address table, but not really.
As we’ve seen, this cannot make a difference on AWS.
But What’s the Issue with dhclient?
I can reproduce the described effect. Sometimes.
It’s quite weird.
But I removed the
isc-dhcp44-client package and installed the
dhcpcd package instead. Then I’ve changed
/etc/rc.conf to 1)
remove the (we don’t want the system to start
dhclient) 2) remove the
dual-dhclient hack 3) add
dhcpcd_enable=YES. That’s the dhcpcd by Roy Marples, as suggested in one of the replies. It takes good care of both IPv4 and IPv6.
dhcpcd, getting the proper address via DHCPv6 just works. Always.
It’s not an answer, but it is a solution.
Bonus: Link-local only with EUI-64 addresses
You’d think the story ends here. But oh no! After setting up
dhcpcd, suddenly the nodes can’t ping each others’ link-local addresses anymore. What happened?
After some thorough head-scratching, I noticed that the link-local addresses are different now:
inet6 fe80::5562:63fd:691b:3c92%xn0 prefixlen 64 scopeid 0x2
This is not an EUI-64 address (those are easily identified by having
ff:fe as middle part).
Usually this wouldn’t be a problem, but on AWS, as we’ve learned, this doesn’t work: The Mapping Service does not know how to handle those. It can only answer for EUI-64 based addresses. Therefore our Neighbor Solicitation for other addresses goes unanswered.
As it turns out, dhcpcd is configured by default to generate a private address, i.e. one that does not “leak” the system’s MAC address. I’d argue this makes sense for globally routed addresses, but not for link-local addresses. The culprit is
slaac private in
/usr/local/etc/dhcpcd.conf. After removing this option,
everything works as expected.
Bonus: About DHCPv6 and prefix lengths and on-link prefixes
You’d think the story ends here. But oh no!
DHCPv6 was working correctly now to assign the global address (e.g.
2a05:...). What I didn’t test was if the nodes can ping each other with those addresses (I had only tested the link-local addresses).
Turns out, they can’t:
root@n0-1c:~ # ping6 2a05:d014:631:380c:8114:7e93:4c0f:4488 PING6(56=40+8+8 bytes) 2a05:d014:631:380c:7022:b490:22d9:95b8 --> 2a05:d014:631:380c:8114:7e93:4c0f:4488 ping6: sendmsg: No buffer space available ping6: wrote 2a05:d014:631:380c:8114:7e93:4c0f:4488 16 chars, ret=-1
As it turns out, DHCPv6, by design, really just assigns single addresses, without a prefix len. So the address gets configured as a /128. The system doesn’t know anything about a local /64 that’s supposed to be there.
This is another function of SLAAC / Router Advertisements: It tells nodes which networks (prefixes) are on-link. The nodes then install a route entry, so they know this prefix is directly reachable.
What I had missed, when removing the
ifconfig_DEFAULT lines from
rc.conf, is that it also contained
accept_rtadv. Without this interface flag, the kernel will not process those advertisements and will not know about the on-link prefixes.
I’d thought that re-adding
ifconfig_DEFAULT="inet6 accept_rtadv" would be enough, but for whatever reason, this doesn’t take effect. That flag is not activated.
So instead I returned to using
#dhcpcd_enable=NO dhclient_program="/usr/local/sbin/dhcpcd" ifconfig_DEFAULT="SYNCDHCP inet6 accept_rtadv"
NOW everything works!
… except for detaching an additional NIC, but I’m done for today.
We’ve learned a few things:
- There’s no broadcast or multicast on AWS
- Therefore Duplicate Address Detection is pointless
- No direct communication between different Subnets (contrary to what the Networking Blog claimed)
- Some failover and HA solutions might not work as they used to
- The Mapping Service resolves MAC addresses known to AWS, plus EUI-64 based link-local addresses
- private link-local addresses do not work
- For FreeBSD, dhcpcd seems to work more smoothly than separately running dhclient for IPv4 and isc-dhclient for IPv6
- Also for FreeBSD, DHCPv6 should really just work instead of… this.
You’re still here? Wow. Next time I’ll make it two parts. Pinky swear!