neveragain.de teletype

AWS: Cannot Escape IPv4

2023-09-21

This is part of a blog series:

trying to get home

Introduction

AWS has announced that they will charge for public IPv4 addresses soon. Surprisingly, most reactions to this announcement were welcoming, as it’s believed to bring a big push for IPv6 adoption.

Their blog had this to say:

This change … is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization and conservation measure.

I will explain why this charge will not affect IPv6 adoption in any meaningful way.

I will also show that it is pretty bold to use IPv6 in trying to soften the blow of this charge: Their own IPv6 support has huge gaps. They make it impossible for customers to move on from IPv4.

But first, a quick summary, and some costly anti-patterns to watch out for.

What’s Going to Change

Every dedicated public IPv4 address1 will be charged with $0.005 per hour (almost $4 per month / more than $40 per year), starting February 2024.

Some examples of affected services:

So for a public Elastic Load Balancer configured in three availability zones (as recommended), this is a base price increase of over 50%.

For Managed NAT Gateway, this is a base price increase of ~10%. I didn’t think those could become more expensive, but here we are. By the way, this is a good time to remember that a NAT Gateway does not provide zonal redundancy – a NAT Gateway in every availability zone is required when outbound connections are critical.

All shared IPv4 addresses, like those used for Cloudfront distributions or API Gateway endpoints, are not affected. Public IPv4 addresses used by non-VPC Lambda functions aren’t affected either.

Wasteful Patterns

I have observed two patterns that waste quite a lot of addresses and are heavily impacted by the upcoming charges.

The first pattern is having multiple Load Balancers (per VPC); this is often the result of using several readily available Cloudformation templates / Terraform modules, or somehow using Kubernetes ingress controllers that create a Load Balancer for every service. This is fixed by not doing that! A single Load Balancer can handle many URLs and services.

The second pattern is intentionally configuring public IPv4 on EC2 instances and Fargate tasks just to avoid Managed NAT Gateway (commendable until now!). Ironically, this will still be cheaper until hitting around ten public IPv4 addresses per availability zone.

How to Check

The usage data for public IPv4 addresses is already being accounted, so it’s easy to check. Usage is currently listed with $0 but will start to cost $0.005 per hour starting in February 2024.

Accelerating Your IPv6 Adoption

So it does sound pretty obvious: IPv4 addresses will soon cost a few bucks, while IPv6 addresses are free of charge. Even better: There is no concept of private addresses in IPv6, which means farewell to the Managed NAT Gateway and its magnificent pricing. It seems like the perfect time to finally adopt IPv6.

There are two approaches to adopting IPv6. Either by going dual-stack, which means configuring IPv4 and IPv6 addresses on all systems. Or by going IPv6-only internally, with IPv4 connectivity only where facing the general public – for example at the very edge, on the CDN, which takes care of serving IPv4 clients.

Running IPv6-only is the optimal solution for a future-proof network. Even the U.S. Government, of all places, recognizes this. A memorandum from the Executive Office of the President states:

In recent years it has become clear that [running dual-stack] is overly complex to maintain and unnecessary. As a result, standards bodies and leading technology companies began migrating toward IPv6-only deployments, thereby eliminating complexity, operational cost, and threat vectors associated with operating two network protocols.

That memorandum was issued three years ago.

The underlying IPv6 support is excellent. It’s possible to configure IPv6-only EC2 instances in IPv6-only subnets and everything works fine, including local DNS, time service, host configuration, security and all the stuff to be expected in an IPv6-enabled network. Components like VPN, Transit Gateway, Direct Connect and Network Firewall also support IPv6.

Cannot Escape IPv43

The problem is actually using all the (other) Amazon Web Services.

It’s practically impossible to run IPv6-only. Trying to configure core services like API Gateway, Lambda, ECS or App Runner into IPv6-only subnets makes them either sternly refuse those subnets or throw comically sad error messages like Not enough IP space available (Elastic Load Balancer).

And it gets worse: Not even running dual-stack internally helps. Most services, when configured into a dual-stack subnet, happily ignore that the subnet has IPv6 configured; they can connect to IPv4 targets only. This actually hurts customers.

Almost no service API can be used from a VPC without public IPv4 addresses4. Running an innocent aws s3 ls from an EC2 instance will fail. Using Systems Manager – the recommended way to connect to and manage an instance – does not work. Accessing SQS from an ECS Task is impossible. And so on. This is because more than 90% of all service API endpoints do not support IPv6. Other endpoints like SES SMTP or ECR repository endpoints5 don’t work either. Cloudfront cannot connect to IPv6 origins.

And then AWS frames this new charge as an “encouragement” to adopt IPv6. This is what ticked me off: There is no escape, no way to avoid these charges – because they have significantly neglected IPv6 for years.

Charges Without Changes

For most customers, nothing will change. Which is exactly my point.

For customers of any relevant size, this IPv4 charge is a drop in the bucket; they won’t even notice it on their bill. In my experience so far, this will usually be very well below 1%. Nobody will invest significant engineering effort for that alone.

For many SMBs, hobbyists and startups though, this charge can easily amount to 10-30% of the bill. These customers certainly would adopt IPv6 to avoid such a price increase.

So that’s where we are: The larger customers do not care about this charge; and those that will feel the impact have no way to avoid it.

Conclusion So Far

I think the IPv4 charge by itself is fine – maybe even necessary, given the IPv4 acquisition costs lately.

I would like AWS to be a role model in IPv6 adoption, like they are in many other areas. If their service landscape had mature IPv6 support, many customers could indeed migrate to IPv6-only environments. Larger customers still would not care – but it would create a lot of first movers within the community, setting examples to follow.

But IPv6 is clearly not a first-class citizen today.

Given those circumstances, this charge will do nothing to drive IPv6 adoption. It will stiffle innovation and make it even less attractive to have a personal account – for example, for learning, to study for certifications (to invest in AWS as a career path) or to support open-source work. AWS talks big about democratizing cloud; this isn’t it.

Other links to this article: Twitter, Hacker News

Update 2023-09-21 – added “Links” section

Update 2023-09-21 – affected services: added RDS; made it clear that those are examples (list is not exhaustive)

Update 2023-10-12 – added picture

Update 2023-10-24 – added links to Part Two

Update 2023-10-31 – minor edit for readability

  1. IPv4 address charges only apply to AWS-owned IP addresses. It’s possible to “bring your own IP” (BYOIP), so if a customer happens to “own” a routable block of public IPv4 addresses (many large organizations do), they can use those addresses. Public “BYOIP” addresses will not be charged. ↩

  2. Interestingly, while EKS supports Fargate, it’s not possible to configure public IP addresses. And Fargate’s awsvpc networking mode is available on EC2-backed ECS as well, but doesn’t support public IP addresses there either. ECS/Fargate seems to be the only combination to allow this. ↩

  3. Thanks to u/slfyst for this reddit post that I shamelessly stole the title from. ↩

  4. So it’s not possible to launch ECS tasks from Elastic Container Registry, but it is possible from Dockerhub, which does support IPv6. Dockerhub also does rate limiting on pulls. ↩