Introduction

In February 2024, AWS started charging customers for public IPv4 addresses. I recommend reading my rant analysis of the AWS announcement first: Cannot Escape IPv4. The second part of this series explored the situation for ingress traffic.

This final part will explore options for egress traffic from VPCs, with a focus on using IPv6 to avoid public IPv4 addresses entirely (spoiler: not feasible).

Egress connectivity “to the internet” is required not only for downloading software and container images, but also for accessing other services like Github, payment providers, sending telemetry, etc. – and for using AWS services: EC2 instances might access objects on S3, containers might send messages to SQS queues, and so on.

And as connecting to AWS service endpoints via IPv6 is surprisingly involved, I will take a look at the different kinds of endpoints and the additional configuration required for the AWS CLI and SDKs.

Egress from VPC Resources

Options for Egress

With IPv4, the common patterns for enabling egress traffic from VPCs are:

• Public subnet: Route to an Internet Gateway; direct assignment of public IP addresses to VPC resources like EC2 instances and containers
• Private subnet: Using only private addresses on resources, but adding a Managed NAT Gateway (or a cheaper self-managed NAT Instance¹) that translates all egress connections to a single public IPv4 source address
• PrivateLink: Even when a subnet does not have any internet connectivity (internal subnet), it’s possible to enable egress to specific AWS services by adding Interface Endpoints (PrivateLink) or Gateway Endpoints² to the VPC

This is mostly the same with IPv6, but there is one significant difference.

No More NAT Gateway

With IPv6, there is no NAT³. Every IPv6 address used in a VPC is from a public address block. The difference between a public and a private subnet would no longer exist.

To mimic the familiar security characteristics of private subnets with NAT, AWS offers the Egress-only Internet Gateway for IPv6: It works like the regular Internet Gateway, but it only allows egress connections, i.e. it will never allow connections from the internet to VPC resources.

And just like Internet Gateway, it doesn’t cost anything. It has no fixed hourly price, it doesn’t have additional traffic-based charges, and it doesn’t require a public IP address that adds to its price – all of which a NAT Gateway does. IPv6 can be a cost optimization measure!

Dual-Stack and IPv6-Only Subnets

IPv6-enabled VPC subnets will usually be dual-stack, meaning IPv6 will be configured in addition to IPv4 (running two network protocol stacks). Any resource running in dual-stack subnets will pick up an IPv4 address as always, and can additionally pick up an IPv6 address.

It’s possible to create IPv6-only subnets that have no IPv4 configuration at all, avoiding the complexity of running two network protocols. I love that this is possible, and I hope this will be the default mode of operation at some point. And if the VPC also has public IPv4 and a Managed NAT Gateway, DNS64 and NAT64 support enables IPv6-only resources to seamlessly access to IPv4-only targets.

IPv6 Egress Support by Service

So far, IPv6-only subnets are only supported by EC2 instances built on Nitro. All other AWS services will reject deployment into an IPv6-only subnet, so it’s not possible to add a Lambda function or a Load Balancer into an IPv6-only subnet, for example.

But dual-stack subnets are supported by the major AWS compute services, so IPv6 egress connections are possible from EC2 instances, containers running on ECS⁴ and EKS (both via Fargate and EC2), and VPC-enabled Lambda functions.

Most other VPC-enabled services only support IPv4 – they simply ignore IPv6 when deployed into a dual-stack subnet; examples include App Runner, API Gateway, AppSync, and CodeBuild. In a VPC without any public IPv4 connectivity (or PrivateLink), they will not be able to connect to external resources, including AWS services.

Accessing AWS Services via IPv6

But even when a VPC resource can initiate connections via IPv6, some public IPv4 egress (or PrivateLink) is still required for many workloads, as most AWS services do not support requests from IPv6 clients. Here are some example requirements that cannot work otherwise:

• Pulling container images from the Elastic Container Registry
• Sending e-mails via SES (neither via API nor via SMTP)
• Sending metrics or X-Ray traces to CloudWatch
• Managing EC2 instances via Systems Manager (SSM), including Session Manager, Patch Manager, etc.
• Dynamic SSH keys for EC2 instance access via EC2 Instance Connect
• Connecting to any (public) API Gateway endpoint
• Using the ECS Exec feature to log into ECS tasks
• Using any AWS service that does not support IPv6 on its endpoints, like SQS, SNS, Eventbridge, Bedrock, and many others

This also applies to third-party services without IPv6 support like Github⁵.

Many AWS services deploy resources into the customer’s VPC, like RDS or ElastiCache. Connecting to those resources happens entirely within the VPC, so this works fine without public IPv4 connectivity (using the VPC’s private IPv4 addresses). That’s also how PrivateLink Interface Endpoints work.

Cloudwatch Logs

Many applications need to send log output to Cloudwatch Logs. This has been supported via IPv6 since February 2024, but not by default – so sending logs to Cloudwatch requires configuration of each application. For example, for the Cloudwatch Agent (EC2), this can be specified via the endpoint_override option in the configuration file.

Elastic Container Service

While ECS supports IPv6 for workloads, it often needs access to other AWS services as part of starting the task.

AWS’s Elastic Container Registry does not support IPv6, so ECS tasks cannot pull container images from ECR via IPv6. Dockerhub supports IPv6 though.

ECS tasks are commonly configured to send the containers’ log output to Cloudwatch, using the awslogs driver in the logging configuration. As mentioned above, extra configuration is required – but awslogs does not support endpoint configuration (Github issue #73, open since 2018). Tasks will fail to start unless configured without logging or using a different logs driver. There was a suggested workaround using the FireLens driver to send logs to Cloudwatch.

ECS supports retrieving secret values from either Secrets Manager or the SSM Parameter Store, but the latter does not support IPv6.

Apropos ECS: The ECS service team has published a pattern: Dual-stack IPv6 networking for Amazon ECS and AWS Fargate. It describes an ECS setup without public IPv4 in great technical detail and has some example code. The issues I mention here are solved with PrivateLink Interface Endpoints for ECR, Cloudwatch, etc. (for about $160/month). A great lab setup to gain experience with IPv6 on AWS!

I don’t know too much about EKS, but I assume the situation is similar there. AWS offers an IPv6 workshop for EKS, by the way.

Hard Truth

Clearly, it’s not feasible to fully avoid public IPv4 for egress on AWS, except for very specific use cases.

If some public IPv4 is required, the general advice for costs is this: A small self-managed NAT Instance is usually the cheapest option by far (in fact, the required public IPv4 address can cost more than the EC2 instance). For maximum quality of life, Managed NAT Gateway is the best choice – and for any enterprise environment, this is a no-brainer, despite its price tag.

But there are grey areas for small setups: Using public IPv4 addresses directly is cheaper than Managed NAT Gateway until around ten IPv4 addresses (supported for EC2 instances and ECS Fargate), and using PrivateLink is cheaper than Managed NAT Gateway until five Interface Endpoints. This is per Availability Zone for both, if AZ redundancy is required. In large environments, it’s common to use Resource Sharing and/or Transit Gateway to centralize egress – but this is out of scope for this post.

So if some public IPv4 is required anyway, it may seem pointless to spend time on IPv6 for egress at all, especially as it adds complexity. I recommend doing it anyway, as it’s important to gain operational experience with IPv6 – it’s coming slowly, but it’s coming. And every bit of egress traffic that doesn’t go through a NAT Gateway decreases costs.

AWS Service Endpoints and Client Applications (AWS CLI and SDK)

The remainder of this post shows how to make the AWS SDK use IPv6, so it’s only relevant when running entirely without public IPv4 connectivity (and PrivateLink).

On AWS Service Endpoints

When an AWS application – anything using the AWS SDK, including the AWS CLI – needs to send requests to an AWS service, it must know which service endpoint to connect to.

Endpoints are usually region-specific DNS names, like logs.eu-central-1.amazonaws.com for Cloudwatch Logs in Frankfurt. Some services are global⁶ instead, so they don’t contain a region name, like iam.amazonaws.com or cloudfront.amazonaws.com.

The SDK doesn’t actually care much about IPv4 vs. IPv6; it just connects to the service endpoint using the standard behavior of the operating system (usually: try IPv6 first, then try IPv4).

For the few services that support IPv6 endpoints, they usually do so only on separate dual-stack endpoints. This is an important and surprising detail – virtually everyone in the world implements IPv6 on websites and APIs by allowing requests via IPv6 and IPv4 on the same endpoint, so IPv6 “just works” when available on both ends of the connection.

But on AWS, IPv6 is usually not supported on the default service endpoint. For example, in Ireland (eu-west-1), EC2 supports IPv6 clients, but on a separate endpoint:

• default: ec2.eu-west-1.amazonaws.com (IPv4 only)
• dual-stack: ec2.eu-west-1.api.aws (IPv4 and IPv6)

Some AWS services need to discover endpoints dynamically – like Timestream, where the SDK needs to connect to the correct cell that handles the requested Timestream database. This mechanism isn’t aware of different endpoint flavors, so it could not support separate dual-stack endpoints, as far as I can tell.

Unfortunately, the AWS SDK will never attempt to use a service’s dual-stack endpoint by default. It must be configured to do so, or it will never use IPv6 for most services.

Service Endpoint Configuration

Using Dual-Stack Endpoints

By default, when no endpoint has been given explicitly, the SDK will look up the service endpoint using an internal ruleset. This supports a configuration option to make it select a dual-stack endpoint.

Using this option is preferred over explicitly specifying an endpoint, if possible.

The effect can be verified easily by using AWS CLI with --debug, as it will log the result of the rule-based lookup.

Standard behavior:

$ aws --debug ec2 describe-instances 2>&1 | grep "Endpoint provider result"
[...] Endpoint provider result: https://ec2.eu-west-1.amazonaws.com

Requesting the dual-stack endpoint via environment variable:

$ export AWS_USE_DUALSTACK_ENDPOINT=true
$ aws --debug ec2 describe-instances 2>&1 | grep "Endpoint provider result"
[...] Endpoint provider result: https://ec2.eu-west-1.api.aws

Configuring this via environment variables also works for other SDK applications, not just the CLI. It can also be enabled in code when initializing the SDK client. Python example:

import boto3
from botocore.config import Config

ec2 = boto3.client(
	service_name = 'ec2',
	config = Config(
		use_dualstack_endpoint = True,
	),
)

But if the SDK is configured to select a dual-stack endpoint, it will generate a dual-stack endpoint name even if that is not supported for the service. There is no fallback to the default endpoint. Requesting a dual-stack endpoint for EC2 in Frankfurt (eu-central-1), for example, would abort with a connection error because ec2.eu-central-1.api.aws does not exist. EC2 supports dual-stack endpoints only in some regions.

Therefore, it’s important to carefully check which service supports which endpoints in which regions. I have built a map of AWS Service Endpoints by Region and IPv6 Support to navigate the chaos.

Explicitly Specifying Endpoints

A service endpoint can also be specified explicitly. Python example:

import boto3

ec2 = boto3.client(
        service_name = "ec2",
        endpoint_url = "https://ec2.eu-west-1.api.aws",
)

AWS CLI example:

$ aws --endpoint https://ec2.eu-west-1.api.aws ec2 describe-instances

Or using environment variables:

$ export AWS_ENDPOINT_URL=https://ec2.eu-west-1.api.aws
$ aws ec2 describe-instances

Additionally, it’s possible to configure service-specific endpoints like this:

$ export AWS_ENDPOINT_URL_LAMBDA=https://lambda.eu-central-1.api.aws

This makes it easier to configure an application that needs to access several services, without modifying the code.

Both endpoint configuration options are also supported via the configuration files. The full documentation for specifying endpoints is here, but some SDK languages lack support for this (e.g. Java and Rust).

IPv6-Enabled Default Endpoints

A few services (like Secrets Manager) actually support IPv6 on their default service endpoint.

It’s indeed possible to set up an IPv6-only subnet, launch a new IPv6-only EC2 instance, and run this command:

$ aws secretsmanager list-secrets

This “just works”, without any configuration and in all supported regions. As it should be.

(But if the “use dual-stack endpoint” option is enabled, this command will assume a wrong service endpoint and fail.)

Conclusion

The basic building blocks are there. AWS has done all the hard work – the IPv6 support in VPC, EC2, Lambda, and ECS/EKS is good.

But the obstructive SDK behavior and the frugal IPv6 support of service endpoints make IPv6 complex and error-prone to implement on AWS. And that’s in addition to all the little things that crop up. It all feels like it has never been seriously used without public IPv4 outside a lab environment.

It’s pretty clear that IPv6 on AWS will not see widespread adoption this way. Just ignoring IPv6 for egress is so much easier than running without public IPv4 addresses. This way we will never escape IPv4. So I can only hope that AWS can rally service teams to add IPv6 support to their endpoints. Preferably to the default endpoints, as the rest of the world does – this would alleviate the SDK issues significantly.

In other words – referring to Jeff Barr’s initial blog post about the IPv4 charges –, I’d like to encourage AWS to think about accelerating adoption of IPv6 as a modernization and conservation measure.

Other links to this article: Twitter

This is part of a blog series:

• Part One: Cannot Escape IPv4
• Part Two: Ingress Traffic: Avoiding Public IPv4 Address Charges
• Part Three: Egress Traffic and Using AWS Services via IPv6 (this part)

That’s Not Supposed to Work!

Every now and then, I run a command that I expect to fail; usually to verify the system’s behavior before and after a change. So I was querying a secret from the Parameter Store on some throwaway EC2 instance with no special permissions – and it worked. Oops!

IAM Policies for the SSM Agent

The Systems Manager Agent is the recommended way to manage EC2 instances, e.g. for automated patching and as an alternative to SSH.

Over the years, different IAM Policies were recommended for the SSM Agent:

Since 2019, the recommended policy was AmazonSSMManagedInstanceCore (setup instructions archive link). Unfortunately, this policy allows to read all parameters from the Parameter Store (ssm:GetParameter for *).

This sounds rather harmless at first, but Parameter Store can also store secrets (type SecureString). It’s pretty common to use Parameter Store for secrets, as the Secrets Manager service charges $0.40 for every single secret (and its additional features are usually not needed). So my EC2 instance was able to access all secrets in the AWS account, just because it was running the SSM Agent.

The previous policy was even worse: AmazonEC2RoleforSSM was recommended until 2019. To this day, it includes full read and write access to all S3 buckets in the account (s3:PutObject and s3:GetObject for *) – and the mentioned permissions to Parameter Store as well. The role has not been deprecated yet, but the policy’s documentation says this is supposed to happen “soon” and suggests to use AmazonSSMManagedInstanceCore instead.

Since fall of 2022, there is a new policy AmazonSSMManagedEC2InstanceDefaultPolicy. The only difference to the AmazonSSMManagedInstanceCore role is that it doesn’t include permissions to the SSM Parameter Store. This policy is meant to be used with the new Default Host Management Configuration (a great overview of how this works under the hood can be found on Aidan Steele’s blog).

Impact

According to a recent scientific study¹, almost two in three customers use one of the older policies – likely unaware that it might allow unintended access to all secrets (and all S3 buckets, in case of the oldest policy). That’s not too surprising, as AmazonSSMManagedInstanceCore still is recommended as part of the Cloudwatch Agent setup and for the SSM Agent setup without DHMC.

Alternatives

The way forward is to adopt DHMC. If that’s not an option, permission boundaries can help, or simply replacing the policy with the newer AmazonSSMManagedEC2InstanceDefaultPolicy or a custom policy.

Another option is using a custom KMS key to encrypt the parameters. In that case, the parameter could still be read, but decrypting its value would fail unless the EC2 instance has permission to use the key.

Conclusion

The most flexible and – usually – most secure option is to write a custom IAM policy for everything. Most customers use the provided Managed IAM Policies instead, and the AWS service documentations usually recommend doing so. Given the amount of services and the complexity of IAM interactions, I’d even say it’s not feasible to write and maintain custom policies for every use-case, except for organizations with dedicated security teams.

Still, Managed Policies should not be trusted blindly. Those policies can change at any time², and it’s always easy to miss something – like I did in this case.

Other links to this article: Twitter

Update 2023-11-01 – updated to reflect that the issue only applies to parameters encrypted with the default KMS key (thanks to Aidan Steele for keeping me on my toes!)

Introduction

AWS will soon start to charge customers for public IPv4 addresses. I recommend reading my rant analysis of the AWS announcement first: Cannot Escape IPv4.

This post will take a quick look at available options specifically for ingress traffic – how to get requests from end users to an application built on AWS.

The first goal is providing access to end users on both IPv4 and IPv6. Global IPv6 adoption is around 50% for end users¹, so it’s imperative to provide the best connectivity. At the same time, it’s still necessary to serve end users that are limited to IPv4.

The second goal is avoiding the AWS public IPv4 address charges – for reasons outlined in my previous post, I want to show that there is often no good way to avoid those charges. And because it helps IPv4 address conservation, of course. To be clear, I do not recommend actually doing this, as these charges will have a very small effect on the AWS bill; it would be silly to change architecture for a few bucks. For most customers.

Dedicated and Shared IPv4 Addresses

The new AWS IPv4 charges only apply to public IPv4 addresses that are dedicated, meaning that they are used exclusively by a single AWS customer.

For web (HTTP-based) applications, it’s common to use shared IP addresses, meaning many customers can be hosted on the same IP address (using HTTP Virtual Hosting and TLS SNI). Some AWS services provide centralized ingress using shared IP addresses – consequently, there are no IPv4 address charges with those services.

Ingress Services in Detail

The following table lists common AWS services used for ingress and their capabilities regarding the two goals.

All services support IPv4 ingress; the first column shows whether the service uses a shared IPv4 address (no charge) or a dedicated IPv4 address (incurs public IPv4 charges).

The last column is included for completeness; it’s not relevant to the two goals.

Elastic Load Balancer and Global Accelerator

Probably the most popular choice of ingress is Application Load Balancer, which routes HTTP requests to backend applications. For other non-HTTP applications, Network Load Balancer is a common choice. Global Accelerator can be used to provide public ingress IPs for (internal) Load Balancers and EC2 instances.

Fortunately, these AWS services do support IPv6 for ingress. Unfortunately, they do not allow to opt out of IPv4, so even if an application would only need IPv6, charges for multiple IPv4 addresses will apply. For a Load Balancer in three Availability Zones, this will be a base price increase of more than 50%.

As Load Balancers and Global Accelerator violate the second goal, we need to look at other AWS services to provide ingress.

API Gateway

API Gateway supports VPC Links on HTTP APIs. Together with proxy integrations, this allows some interesting combinations.

API Gateway can be combined with the Cloud Map service discovery to provide ingress for ECS Services. See this description on the AWS blog (2020) or the entry on “Containers on AWS” (2023, with SAM / Cloudformation).

API Gateway can also be the entry point for an internal Load Balancer which doesn’t use public IPv4 addresses.

API Gateway has some noteworthy limitations, like a maximum request duration of 30 seconds and no support for unencrypted HTTP. And no IPv6 support.

Elastic Beanstalk

Elastic Beanstalk orchestrates Elastic Load Balancer and compute resources. While the underlying resources do support IPv6, Elastic Beanstalk offers no way of configuring it. There are Github issues on the EB Roadmap that have been open and unanswered since 2021 for both ingress and VPC connectivity.

Elastic Beanstalk seemed somewhat abandoned for years, and this shows in its IPv6 support as well.

Others

For simple container-based applications, migrating to App Runner could be an interesting option. It does not support IPv6 though, and has its own set of limitations.

EC2 instances can run in IPv6-only subnets, so if no additional AWS services are required, this would be a good option.

Lambda Function URLs are a nice surprise, as they support IPv6 not only for ingress, but now also for egress (to VPC and to internet) – a feature added just a few days ago. Many existing applications are not easily migrated to Lambda though³.

Content Delivery Networks

When an AWS service has missing IPv6 support, this can usually be fixed by adding a Content Delivery Network (CDN) with support for IPv6 at the edge, like Cloudfront; many applications already use Cloudfront anyway. Cloudfront can serve IPv4 and IPv6 to end users, while talking to origin services over IPv4.

Cloudfront does not support IPv6 to origins though, so it would not help with IPv6-only EC2 instances. CDN competitor Cloudflare seized the opportunity to make fun of AWS help AWS customers out by offering a discount⁴ for using IPv6 origins on AWS. This allows to serve IPv4 and IPv6 end users while running IPv6-only EC2 instances. In many cases, a CDN can even be used for simple load balancing between EC2 instances.

One big drawback of not using Application Load Balancer in front of EC2 instances is the overhead for installing and managing TLS certificates.

Conclusion

When avoiding Elastic Load Balancers, which incur IPv4 address charges, the support for both IP protocols can usually be achieved by using a CDN. On the other hand, most workarounds to avoid Elastic Load Balancers are neither technically nor financially good ideas. If anything, this serves to show the weird state of IPv6 support on AWS.

I would love to see Elastic Load Balancing add the option to run without IPv4 addresses, combined with Cloudfront support for IPv6 origins. This combination would be a relatively easy fix for many setups, and it would be a big step towards making IPv6-only setups feasible.

But until AWS IPv6 support improves, the only sensible option for customers using Elastic Load Balancers is to just accept the upcoming IPv4 charges.

This is part of a blog series:

• Part One: Cannot Escape IPv4
• Part Two: Ingress Traffic: Avoiding Public IPv4 Address Charges (this part)
• Part Three: Egress Traffic and Using AWS Services via IPv6

Other links to this article: Twitter

Addendum 2024-05-20 (1) – As of November 2023, App Runner supports IPv6 ingress, making it a solid choice for quick and easy IPv6-enabled ingress without extra IPv4 address charges.

Addendum 2024-05-20 (2) – As of May 2024, Application Load Balancer supports IPv6-only ingress, allowing to run ALBs without public IPv4 addresses. This feature was on my conclusion’s wish list! Now if Cloudfrount adds IPv6 origin support as well, this would be great.

Update 2023-10-25 – added link to “Containers on AWS” for API Gateway / Cloud Map / ECS

Introduction

AWS has announced that they will charge for public IPv4 addresses soon. Surprisingly, most reactions to this announcement were welcoming, as it’s believed to bring a big push for IPv6 adoption.

Their blog had this to say:

This change … is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization and conservation measure.

I will explain why this charge will not affect IPv6 adoption in any meaningful way.

I will also show that it is pretty bold to use IPv6 in trying to soften the blow of this charge: Their own IPv6 support has huge gaps. They make it impossible for customers to move on from IPv4.

But first, a quick summary, and some costly anti-patterns to watch out for.

What’s Going to Change

Every dedicated public IPv4 address¹ will be charged with $0.005 per hour (almost $4 per month / more than $40 per year), starting February 2024.

Some examples of affected services:

• Elastic Load Balancers (one address per availability zone, at least two – and IPv4 cannot be turned off)
• EC2 instances & Elastic IPs
• ECS Fargate² tasks configured with public IPs
• Global Accelerator IPs
• Site-to-site VPN IPs
• RDS (nine in ten customers have public IP on RDS by accident)
• Managed NAT Gateways

So for a public Elastic Load Balancer configured in three availability zones (as recommended), this is a base price increase of over 50%.

For Managed NAT Gateway, this is a base price increase of ~10%. I didn’t think those could become more expensive, but here we are. By the way, this is a good time to remember that a NAT Gateway does not provide zonal redundancy – a NAT Gateway in every availability zone is required when outbound connections are critical.

All shared IPv4 addresses, like those used for Cloudfront distributions or API Gateway endpoints, are not affected. Public IPv4 addresses used by non-VPC Lambda functions aren’t affected either.

Wasteful Patterns

I have observed two patterns that waste quite a lot of addresses and are heavily impacted by the upcoming charges.

The first pattern is having multiple Load Balancers (per VPC); this is often the result of using several readily available Cloudformation templates / Terraform modules, or somehow using Kubernetes ingress controllers that create a Load Balancer for every service. This is fixed by not doing that! A single Load Balancer can handle many URLs and services.

The second pattern is intentionally configuring public IPv4 on EC2 instances and Fargate tasks just to avoid Managed NAT Gateway (commendable until now!). Ironically, this will still be cheaper until hitting around ten public IPv4 addresses per availability zone.

How to Check

The usage data for public IPv4 addresses is already being accounted, so it’s easy to check. Usage is currently listed with $0 but will start to cost $0.005 per hour starting in February 2024.

• In Billing Dashboard > Bills, usage is displayed as PublicIPv4:InUseAddress in the Virtual Private Cloud section
• In Cost Explorer, it’s easy to see usage hours by filtering to only include PublicIPv4:InUseAddress; this is nice because Cost Explorer shows usage per day/hour and it’s easy to group e.g. by member account
• VPC > VPC IP Address Manager (at the very bottom) > Public IP insights is a nice way to get an overview of public IPv4 address usage; this part of IPAM is free, but it shows data only per account and region, and it’s incomplete data – it doesn’t capture transient address usage.

Accelerating Your IPv6 Adoption

So it does sound pretty obvious: IPv4 addresses will soon cost a few bucks, while IPv6 addresses are free of charge. Even better: There is no concept of private addresses in IPv6, which means farewell to the Managed NAT Gateway and its magnificent pricing. It seems like the perfect time to finally adopt IPv6.

There are two approaches to adopting IPv6. Either by going dual-stack, which means configuring IPv4 and IPv6 addresses on all systems. Or by going IPv6-only internally, with IPv4 connectivity only where facing the general public – for example at the very edge, on the CDN, which takes care of serving IPv4 clients.

Running IPv6-only is the optimal solution for a future-proof network. Even the U.S. Government, of all places, recognizes this. A memorandum from the Executive Office of the President states:

In recent years it has become clear that [running dual-stack] is overly complex to maintain and unnecessary. As a result, standards bodies and leading technology companies began migrating toward IPv6-only deployments, thereby eliminating complexity, operational cost, and threat vectors associated with operating two network protocols.

That memorandum was issued three years ago.

The underlying IPv6 support is excellent. It’s possible to configure IPv6-only EC2 instances in IPv6-only subnets and everything works fine, including local DNS, time service, host configuration, security and all the stuff to be expected in an IPv6-enabled network. Components like VPN, Transit Gateway, Direct Connect and Network Firewall also support IPv6.

Cannot Escape IPv4³

The problem is actually using all the (other) Amazon Web Services.

It’s practically impossible to run IPv6-only. Trying to configure core services like API Gateway, Lambda, ECS or App Runner into IPv6-only subnets makes them either sternly refuse those subnets or throw comically sad error messages like Not enough IP space available (Elastic Load Balancer).

And it gets worse: Not even running dual-stack internally helps. Most services, when configured into a dual-stack subnet, happily ignore that the subnet has IPv6 configured; they can connect to IPv4 targets only. This actually hurts customers.

Almost no service API can be used from a VPC without public IPv4 addresses⁴. Running an innocent aws s3 ls from an EC2 instance will fail. Using Systems Manager – the recommended way to connect to and manage an instance – does not work. Accessing SQS from an ECS Task is impossible. And so on. This is because more than 90% of all service API endpoints do not support IPv6. Other endpoints like SES SMTP or ECR repository endpoints⁵ don’t work either. Cloudfront cannot connect to IPv6 origins.

And then AWS frames this new charge as an “encouragement” to adopt IPv6. This is what ticked me off: There is no escape, no way to avoid these charges – because they have significantly neglected IPv6 for years.

Charges Without Changes

For most customers, nothing will change. Which is exactly my point.

For customers of any relevant size, this IPv4 charge is a drop in the bucket; they won’t even notice it on their bill. In my experience so far, this will usually be very well below 1%. Nobody will invest significant engineering effort for that alone.

For many SMBs, hobbyists and startups though, this charge can easily amount to 10-30% of the bill. These customers certainly would adopt IPv6 to avoid such a price increase.

So that’s where we are: The larger customers do not care about this charge; and those that will feel the impact have no way to avoid it.

Conclusion So Far

I think the IPv4 charge by itself is fine – maybe even necessary, given the IPv4 acquisition costs lately.

I would like AWS to be a role model in IPv6 adoption, like they are in many other areas. If their service landscape had mature IPv6 support, many customers could indeed migrate to IPv6-only environments. Larger customers still would not care – but it would create a lot of first movers within the community, setting examples to follow.

But IPv6 is clearly not a first-class citizen today.

Given those circumstances, this charge will do nothing to drive IPv6 adoption. It will stiffle innovation and make it even less attractive to have a personal account – for example, for learning, to study for certifications (to invest in AWS as a career path) or to support open-source work. AWS talks big about democratizing cloud; this isn’t it.

This is part of a blog series:

• Part One: Cannot Escape IPv4 (this part)
• Part Two: Ingress Traffic: Avoiding Public IPv4 Address Charges
• Part Three: Egress Traffic and Using AWS Services via IPv6

Links

• Dual Stack and IPv6-only Amazon VPC Reference Architectures (PDF)
• Workshop: Get hands-on with IPv6
• Blog
  • Public IPv4 Address Charge + Public IP Insights
  • Identify and optimize public IPv4 address usage
  • Dual-stack IPv6 architectures and hybrid networks Part 1 and Part 2
• Whitepapers
  • Designing an IPv6 Cloud network
  • IPv6
• VPC documentation
  • Services that support IPv6 (includes a list of ~30 services that have at least some IPv6 support; not listed at all are the ~200 other services)
  • Migrate your VPC from IPv4 to IPv6
  • DNS64 and NAT64

Other links to this article: Twitter, Hacker News

Update 2023-09-21 – added “Links” section

Update 2023-09-21 – affected services: added RDS; made it clear that those are examples (list is not exhaustive)

Update 2023-10-12 – added picture

Update 2023-10-24 – added links to Part Two

Update 2023-10-31 – minor edit for readability

If you’ve been around in networking for a while, you probably know this haiku:

Recently I’ve had one more experience like that, where enabling “VPC Hostnames” on AWS caused an outage. I’d have sworn on my love of beer that it cannot be related to DNS.

Time for a closer look at DNS support in VPCs.

Overview: The VPC DNS Resolver

Amazon provides standard DNS resolvers in each VPC, free of charge. You can find the resolver service at several IP addresses:

• IPv6: At the local address fd00:ec2::253
• Legacy IP:
  • Local address 169.254.169.253
  • VPC’s base network address plus 2 (e.g. 172.31.0.2 or 10.0.0.2) – therefore sometimes called the “dot-2 resolver”

This resolver is announced as part of the default DHCP options.

It can resolve both public internet DNS names (like neveragain.de) and VPC-specific hostnames.

The documentation of DNS support for VPCs fits on a single page.

Overview: VPC-Specific DNS Hostnames

Each Elastic network interface (ENI) in a VPC can have DNS hostnames pointing to it.

For EC2 instances, this would be ec2-$IPADDRESS.$REGION.compute.amazonaws.com (legacy IP only).

It works the same way for other in-VPC AWS services: They create ENIs in the VPC and some corresponding DNS records. With RDS, for example, this is something like mysql1234.cluster-oisdhosdahsj.eu-central-1.rds.amazonaws.com.

RDS demonstrates an important distinction:

• For private-only databases, they always (globally!) resolve to the private IP address
• For databases with public IPs:
  • queries from “outside” resolve to the public IP address
  • queries from “inside” resolve to the private IP address

Here, “inside” usually means “from within the same VPC”, but this line is blurred by VPC Peerings and other connections like Transit Gateway.

VPC-Specific DNS settings

There’s two settings to control the DNS resolver’s behavior. In the documentation’s own words:

• enableDnsHostnames: Indicates whether instances with public IP addresses get corresponding public DNS hostnames¹

• enableDnsSupport: Indicates whether the DNS resolution is supported

Confusingly, the default settings vary a bit: “By default, both attributes are set to true in a default VPC or a VPC created by the VPC wizard” – but any other VPC will have enableDnsHostnames disabled by default.

Reading Naïvely skimming the VPC DNS documentation, the following assumption seemed sound: enableDnsHostnames just creates additional DNS entries for IP addresses (authoritative DNS), and enableDnsSupport controls whether the VPC’s resolver is enabled at all.

But, as we all know: Assumption is the mother of all fuck-ups.

enableDnsSupport

This works pretty much as expected: If this is enabled, the VPC resolver IP addresses (see above) respond to queries.

Disabling it immediately casues those resolvers to go dark, i.e. any DNS query will not be answered (you’ll see a timeout).

It is also required for the next setting to have any effect.

enableDnsHostnames

As expected, this creates additional DNS records. That’s why this option is also required if you want to have public IPs for RDS databases, for example.

Entirely not expected – for me, at least – is that it sythesizes forward and reverse responses for any private IP address. That’s right: Not just the private IP addresses within your VPC CIDR range, but any private IP address. Like, you know, those private IP addresses that you’re using on-premises.

This seems reasonable for VPC addresses:

[ec2-user@ip-10-0-0-34 ~]$ dig +noall +ans -x 10.0.0.34
34.0.0.10.in-addr.arpa.	285	IN	PTR	ip-10-0-0-34.eu-central-1.compute.internal.

But now your on-premises servers have a reverse lookup, too! From the VPC’s perspective, at least:

[ec2-user@ip-10-0-0-34 ~]$ dig +noall +ans -x 192.168.123.234
234.123.168.192.in-addr.arpa. 600 IN	PTR	ip-192-168-123-234.eu-central-1.compute.internal.

[ec2-user@ip-10-0-0-34 ~]$ dig +noall +ans -x 172.16.0.3
3.0.16.172.in-addr.arpa. 600	IN	PTR	ip-172-16-0-3.eu-central-1.compute.internal.

If this option is disabled, those queries return NXDOMAIN (entry does not exist), as they should.

This is documented, but rather in the fine print instead of obviously with the options’ descriptions.

Resolution (Pun Inteded)

So, after all: It was DNS. The client in question performs a reverse lookup of the server’s IP address and then uses that information to see which credentials it should present. After enabling VPC DNS hostnames, that lookup was no longer answered with “no such entry” but with the synthesized ip-192-168-0-123.eu-central-1.compute.internal. Subsequently, the credentials lookup failed to find a match.

Additional Trivia

Version Query

This is something I’ve observed often on AWS: They seem to like easter eggs and/or proper implementations.

From ancient times, when ISC BIND was the de-facto authoritative DNS server software, you’d query a nameserver’s version information from the version.bind hostname in the Chaos (CH) class (as opposed to the default Internet class IN). The nameserver would answer with the software version, e.g. "9.11.0".

Most modern nameservers simply don’t answer this query. The VPC resolver, however, does:

[ec2-user@ip-10-0-0-34 ~]$ dig +noall +ans version.bind txt ch @10.0.0.2
version.bind.           0       CH      TXT     "EC2 DNS"

Caching

The resolver IP address seems to employ some caching of its own, apparently for a few minutes per record.

This is important when changing the VPC Hostnames setting: It will not take effect immediately, and not consistently!

Fleet

I can observe fluctuating time-to-live values returned by the resolver IP address, which leads me to believe that there’s actually a handful of different resolvers answering the queries:

[ec2-user@ip-10-0-0-34 ~]$ while true; do dig +noall +ans neveragain.de soa @10.0.0.2; sleep 1; done
neveragain.de.          243     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
neveragain.de.          242     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
neveragain.de.          241     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
neveragain.de.          247     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
neveragain.de.          245     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
neveragain.de.          246     IN      SOA     squigley.hq.neveragain.de. hostmaster.neveragain.de. 2021080901 86400 3600 2419200 10800
[ ... and so on ...]

Additional Reading

For advanced DNS resolution topics, see also:

• Route53 Resolver Endpoints to forward DNS queries between external (e.g. on-premises) DNS and AWS (beware: while this sounds like three lines of BIND configuration, it actually starts at almost $190/month!)
• DNSSEC validation (not enabled by default, unfortunately)
• DNS Firewall, which is a corny name for hostname-specific override rules in the resolver (you can fake NODATA, NXDOMAIN and arbitrary CNAME responses)
• For VPC Peerings, there’s separate settings for DNS

Conclusion

It was DNS.

Discuss and/or follow on Twitter!

This post is part of a blog series: Road to re:Web.

An Anti-Cloud Pattern

Idle resources are an anti-cloud pattern. As I see it, any service that charges by provisioned time (per second / hour / month) – regardless of actual workload – isn’t truly serverless. Resources are provisioned exclusively for a single tenant. For all practical purposes, they are always used at full capacity. But most of the time, they are idle.

It’s like leaving my car running when I get home.

Not being time-based – not blocking resources that are not actively used – should be a property of what’s called serverless. But much of the world already has another definition (e.g. Fargate is considered serverless, just because there are no servers to manage).

Therefore I’ll be calling those services timeless. Until I come up with a better name.

This post will focus on AWS, but the principles are the same for all cloud providers.

Time-Based Services Hurt Everyone

As I laid out before, cloud cost is problematic for many users. With the typical services that are time-based, like EC2 or Fargate, costs are significantly higher for almost all usage patterns. This is especially true for any non-production environment, as well as individuals (students, developers, freelancers etc.) and many small or start-up organizations.

Also I cannot “forget” to shut down a timeless service. Many people have received panic-inducing bills – simply because some resources were left running, but idle (here is one recent example of many). While this isn’t the only cloud billing trap, it’s a big one. And certainly not just for novices – especially larger companies tend to have quite a few resources running 24x7 that they don’t need.

Most importantly: Idle resources are waste. It’s simply not necessary. Besides wasting money, it has significant environmental impact. This impact is improved by being “in the cloud” instead of your grandma’s on-premises data center, because large cloud providers are really good at reducing excess waste (e.g. from cooling and power units) – but overall, it’s not much better.

Ain’t Easy Bein’ Green

The sad thing is: It’s really not easy to architect exclusively using timeless services.

Often it’s due to the application’s requirements, for example, when it needs a classic relational database like MySQL or PostgreSQL. Aurora Serverless v1 helps a bit, but isn’t quite there yet (v2 looks promising!) – and doesn’t fit for all use-cases.

And sometimes it feels like AWS is making this difficult on purpose – most notably with VPC design. Let’s take a closer look at that.

NAT Gateway

On AWS, the arch enemy, of course, is the Managed NAT Gateway.

This is required¹ if I want to have a non-public network (a very basic security requirement in many scenarios) that can do outbound connections to the internet – like, you know, for downloading security updates, or using AWS APIs.

This hits home for almost $40 per month without having moved a single network packet. Those cost extra, of course. Not to mention that you need one of these fuckers for each availability zone, if they need to be redundant. A laughable cost for most businesses – but absolutely out of the question for any individual.

Domino Effect

So a key desire is to not put myself in a situation where I need a NAT Gateway.

Take the Lambda / RDS pair as an example.

If both can exist on a private network and do not need outbound connectivity, everything is fine.

But as soon as my Lambda needs something from the internet or from an AWS API, it gets complicated:

• I could remove the Lambda from my VPC. Then it will have a public IP address, and it will happily connect to the world. Unfortunately, it will no longer be able to talk to my private network, where RDS is living.

• I could remove the Lambda from my VPC and place RDS in a public subnet, so it will have a public IP. The Lambda could connect to that. But a database with a public IP address is considered a rather strong red flag in IPv4-land, and I couldn’t even use a Security Group to limit access properly. Not to mention that while vanilla RDS can have a public IP address, Aurora databases can not – annihilating any dreams about the timeless Aurora Serverless v2 right away.

• I could move Lambda to a public subnet within my VPC. This would solve all my problems. It could get a public IP and could connect to any private subnet I have. Unfortunately, this isn’t supported: Lambda will not pick up a public IP address in a public subnet – probably because that would be too easy.

• If I just need access to an AWS API, I could use PrivateLink to map endpoints into my private subnet – but those are time-based as well: $9 per month, per availability zone, per AWS service. Right.

There are similarly weird examples. To name one, the network mode awsvpc in ECS will happily grab a public IP address when used with ECS-Fargate. If I use the same awsvpc network mode with ECS-EC2 on a VPC’s public subnet, it will refuse to pick up a public IP address.

Yesterday’s Internet

Just to add insult to injury: All this drama completely goes away with IPv6. There is no NAT in IPv6, and the VPC would use a regular Internet Gateway, or to mimic the NAT Gateway’s outbound-only characteristic, an Egress-only Internet Gateway.

Both are free!

Unfortunately, in 2021, it still is virtually impossible to use a IPv6-only VPC².

Defining Cloud

Everyone has their own definiton of cloud. To me, it’s using abstractions as much as possible. This, in turn, means that I can focus on what is interesting to me. All the infrastructure, all the software stack and runtimes below, making that secure and highly available – those are solved problems. I don’t want to do that myself.

Following that route of abstraction, we end up at truly serverless – timeless – services.

To me, those are inconsequential configuration items. Take S3 buckets, for example. I can have one or ten of those. It’s just a thing that’s there. I pay for actual data storage – not for merely having an empty bucket around.

Here’s some examples of time-based services:

• NAT Gateway, oh yes
• RDS and Aurora (at least in their non-Serverless flavors)
• EC2
• Fargate
• Elastic Loadbalancer
• PrivateLink
• EKS
• VPN

Some wonderful examples of timeless services:

• VPC itself (often overlooked!)
• S3
• Lambda
• EFS
• Timestream
• Infinidash
• API Gateway
• DynamoDB
• SQS

Noteworthy is the new App Runner service: it lands somewhere inbetween, because it uses a hybrid approach (CPU is timeless, RAM is not) – a very interesting step in the right direction.

Prototyping Without Regrets

Note that I can configure all of these timeless services and build an architecture that’s highly available across multiple availability zones, needs zero maintenance, scales automatically and costs literally nothing until I really use it. And even then it’s so much cheaper.

As an example: A while back, I built a backend for an app. The app never went anywhere, but this highly scalable timeless infrastructure was available for more than a year, and still is – and if the app would now be an overnight success, this architecture would scale with it, without me doing anything. I payed a few cents for all that, in total.

Try that kind of worry-free prototyping with time-based services…

Conclusion

Time-based services are an anti-cloud pattern. Yes, you could turn them on and off as needed, or have some pilot light with auto-scaling that would eventually react after a few minutes.

But that’s not what cloud is about. As cloud services evolve and time goes on, we’ll see more architectures that are fully built on timeless services, allowing everyone to worry less about costs and to focus on their actual work.

In a sense, re:Web is about building a bridge to timeless services: It makes it possible to run common web applications on Lambda instead of EC2 or Fargate (or App Runner).

Discuss and/or follow on Twitter!

This post is part of a blog series: Road to re:Web.

There’s also a new mode of billing, which is very exciting to me.

Let’s take a look, especially at the technical details!

Overview

The concept is awesome: App Runner is what happens when you mix CodeBuild, CodeDeploy, Beanstalk, Lambda and Fargate in a vial, shake it thoroughly, and then distill it into something beautifully simple.

It aims to provide the whole shebang from source code to production, while taking care of seamless deployments. I just plug in my web application, hit play, and I have a TLS-secured load-balanced application running in multiple availability zones. And it autoscales out of the box! That’s pretty cool.

To say this again: Virtually no prior AWS-specific knowledge is required. I don’t need to know anything about VPC networking. I don’t have to fumble with Elastic Loadbalancing for half an hour. I don’t need to know who ACM even is, or Fargate, or ECS, or – heaven forbid – EKS.

Note that I said web application: App Runner is explicitly designed for something that speaks HTTP, is health-checkable by HTTP and is used from the outside world via HTTP. And, needless to say, the application needs to be stateless – just like any web application that wants to be loadbalanced and scaled properly.

App Runner operates on containers. I can either simply use some container from a registry.

Or I can have App Runner build a container for me. But we’re not talking about the classic mode of “here’s my Dockerfile, go build that”. Instead, App Runner provides base images for some programming languages. It connects to a Github repository, places the actual source code in the base image, and builds a container from that. This is actually nice, because I don’t have to worry about maintaining and security-patching the base image. And this “just add source code” concept is already known from AWS Lambda. It’s magic.

Exciting New Billing Model

I’m really excited about a feature that got little attention: CPU pricing is only charged when the application is actually handling requests. Repeat: An idle application will not incur any CPU charges.

This brings the whole model much closer to what AWS Lambda does. For almost all applications, that’s a significant reduction. Not only for development and QA environments, but also for most production applications. While Lambda can do this per request with millisecond precision, App Runner does this by the second, which already is great.

Keep in mind that on AWS, paying less translates pretty well into less waste of resources – and therefore to my application’s carbon footprint. Less waste is good for everyone, not just for the CFO.

App Runner can do this easily because it’s in control of the loadbalancing. It knows when there are no active requests.

In contrast to Lambda, App Runner cannot fully stop an application when there are no requests. There may be background threads, cron jobs and so on. Also, the container’s startup time might be way too high to have the next request wait for it.

The secret here is that App Runner keeps the application running, but it throttles the CPU massively. This is a brilliant solution for containerized applications. And it’s a huge step towards the vision that I’m driving with re:Web.

This is also why only CPU charges are dynamic; memory is charged for the whole time the application is available – but memory pricing isn’t too significant in comparison.

An awesome next step would be if App Runner allowed to opt-in to fully stopping the application, so there is actually no instance running while there are no requests – like Lambda does.

Technical Deep Dive

Enough chitchat. You can get all the usual details from the AWS blog or the AWS blog.

Let’s take a look under the hood, as far as we can do that from a customer’s perspective.

Building

Building from Github is actually the only process where I get proper log output. It looks very much like I’d expect a Docker build to look:

[AppRunner] Starting to build your application source code.
[Build] foo
[Build] Sending build context to Docker daemon  9.216kB
[Build] Step 1/5 : FROM 172267607524.dkr.ecr.eu-west-1.amazonaws.com/awsfusionruntime-python3:3.8.5
[Build] 3.8.5: Pulling from awsfusionruntime-python3
[...]
[Build] Status: Downloaded newer image for 172267607524.dkr.ecr.eu-west-1.amazonaws.com/awsfusionruntime-python3:3.8.5
[Build]  ---> cca0c7fe0d48
[Build] Step 2/5 : COPY . /app
[Build]  ---> 1809bc3e008b
[Build] Step 3/5 : WORKDIR /app
[Build]  ---> Running in 1c8671d6366f
[Build] Removing intermediate container 1c8671d6366f
[Build]  ---> 5a491f6a232a
[Build] Step 4/5 : RUN echo foo
[Build]  ---> Running in 33150a894724
[Build] foo
[Build] Removing intermediate container 33150a894724
[Build]  ---> 23df2b9f7940
[Build] Step 5/5 : EXPOSE 8080
[Build]  ---> Running in 0d6f001a39d7
[Build] Removing intermediate container 0d6f001a39d7
[Build]  ---> e0d9f18a3c50
[Build] Successfully built e0d9f18a3c50
[Build] Successfully tagged application-image:latest

In case you’re wondering: echo foo is my build command, because I was forced to give one.

When not using the Console, the build configuration is maintained in apprunner.yml, as part of the repository. A simple example looks like this:

version: 1.0
runtime: python3 
build:
  commands:
    build:
      - pip install pipenv
      - pipenv install 
run: 
  command: python app.py

(taken from the documentation)

One small weirdness: The resulting container will have the generated Dockerfile in /app. Nothing to see there, though.

Sadly, I was not allowed to pull the awsfusionruntime-python3 image myself.

Architecture

From what I can tell, App Runner isn’t built on Application Loadbalancer. There seems to be a huge fleet of envoy proxies handling the requests from the internet.

The application container itself runs on an AWS-owned ECS cluster.

App Runner isn’t orchestrating other AWS services for me – it encapsulates them. This means that any resources used by App Runner will not show up in my account. I won’t see the ACM certificate or the ECS service, for example. Luckily, this also means that I will not have the usual IAM headaches to grant carefully scoped access to other AWS services.

DNS

The DNS record returns three IPs (even when only one instance is available), which I assume is one per availability zone. But for reasons unknown, App Runner takes a quick CNAME detour to explore the length limits of DNS names:

ysvp2d42qc.eu-west-1.awsapprunner.com. 60 IN CNAME 8a3b877e0fff4ad5abcbb491626a7531.5ysp12zfctly24v37dziw9tv9vgyog35m8cxbz6vyafmg15s70.6dd878t7dna7cgqgxld7zry5aq57tlrwg7uq14fk6g3sr656w4.eu-west-1.awsapprunner.com.
8a3b877e0fff4ad5abcbb491626a7531.5ysp12zfctly24v37dziw9tv9vgyog35m8cxbz6vyafmg15s70.6dd878t7dna7cgqgxld7zry5aq57tlrwg7uq14fk6g3sr656w4.eu-west-1.awsapprunner.com. 60 IN A 54.75.254.235
8a3b877e0fff4ad5abcbb491626a7531.5ysp12zfctly24v37dziw9tv9vgyog35m8cxbz6vyafmg15s70.6dd878t7dna7cgqgxld7zry5aq57tlrwg7uq14fk6g3sr656w4.eu-west-1.awsapprunner.com. 60 IN A 34.255.134.39
8a3b877e0fff4ad5abcbb491626a7531.5ysp12zfctly24v37dziw9tv9vgyog35m8cxbz6vyafmg15s70.6dd878t7dna7cgqgxld7zry5aq57tlrwg7uq14fk6g3sr656w4.eu-west-1.awsapprunner.com. 60 IN A 52.210.161.22

I don’t know how I feel about the 60 seconds TTL – seems like introducing a lot of delay for repeated lookups.

App Runner’s DNS names do not use DNSSEC.

TLS

There is no support for TLS 1.3 (bad). Only version 1.2 is supported, i.e. no older TLS versions (good).

Certificates for custom domain names are created with Certificate Transparency.

The Qualys SSL Labs server test grades an App Runner-hosted site as A (I’d assume that A+ were possible if the application emits HSTS headers).

Frontend

Most notably, App Runner-hosted applications will let everyone know they’re running behind envoy in the HTTP headers:

server: envoy
x-envoy-upstream-service-time: 2005

This even happens for successful requests. It’s rather irritating that a proxy overwrites my Server header.

Also there is no support for HTTP/1.0 (envoy limitation). While this isn’t an issue for modern browsers, some tools still default to HTTP/1.0 (Python’s http.server and ApacheBench, for example).

During my tests, I could see about 30 different proxy IPs, all from the 10.0.0.0/16 range. The TCP connection comes from some link-local address.

In the access log, it looks like this:

169.254.175.249 - - [18/Jun/2021 14:07:50] "GET / HTTP/1.1" 200 -

And X-Forwarded-For looks like this (where the first one is my obfuscated client IP):

X-Forwarded-For: 37.123.234.345, 10.0.187.62

Loadbalancing seems to be simple round-robin or connection count or something – I did not observe any kind of persistence, neither with a cookie nor by client IP. I consider this a good thing.

I tried three instances concurrently, and they were placed in eu-west-1{a,b,c}, so I’m guessing App Runner always spreads them evenly.

By the way: When the service is paused, envoy returns HTTP 404 (Not found). I’m not sure if that’s a smart move, especially with regard to search engine crawlers etc.

Container execution / ECS

The container will find itself in an ECS environment. This comes with the expected environment variables being injected:

AWS_EXECUTION_ENV=AWS_ECS_FARGATE
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/58f40ddf9c4b42fc83bfc6c7fe1a1cd8-193386898
ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/58f40ddf9c4b42fc83bfc6c7fe1a1cd8-193386898
[...]

Tickling $ECS_CONTAINER_METADATA_URI_V4/task reveals some interesting bits:

{
 "Cluster": "arn:aws:ecs:eu-west-1:172068907549:cluster/bullet-srv-999999999999",
 "TaskARN": "arn:aws:ecs:eu-west-1:172068907549:task/bullet-srv-999999999999/58f40ddf9c4b42fc83bfc6c7fe1a1cd8",
 "Family": "bullet-td-6b4120addb68419590ea47b94506ed43-8777b77689784d7a9cd6815f70cf4bc5",
 "Revision": "1",
 "Containers": [
   {
     "DockerId": "58f40ddf9c4b42fc83bfc6c7fe1a1cd8-193386898",
     "Name": "instance",
     "DockerName": "instance",
     "Image": "736217495344.dkr.ecr.eu-west-1.amazonaws.com/image-repo-6b4120addb68419590ea47b94506ed43@sha256:02b8e350b106f837b1346b52e64590281ed953fd506737779f6a59072d4c5cdb",
     "ImageID": "sha256:02b8e350b106f837b1346b52e64590281ed953fd506737779f6a59072d4c5cdb",
     "Labels": {
       "com.amazonaws.ecs.cluster": "arn:aws:ecs:eu-west-1:172068907549:cluster/bullet-srv-999999999999",
       "com.amazonaws.ecs.container-name": "instance",
       "com.amazonaws.ecs.task-arn": "arn:aws:ecs:eu-west-1:172068907549:task/bullet-srv-999999999999/58f40ddf9c4b42fc83bfc6c7fe1a1cd8",
       "com.amazonaws.ecs.task-definition-family": "bullet-td-6b4120addb68419590ea47b94506ed43-8777b77689784d7a9cd6815f70cf4bc5",
       "com.amazonaws.ecs.task-definition-version": "1"
     },
     "Limits": {
       "CPU": 960
     },
   },
   {
     "DockerId": "58f40ddf9c4b42fc83bfc6c7fe1a1cd8-329321356",
     "Name": "aws-fargate-request-proxy",
     "DockerName": "aws-fargate-request-proxy",
     "Image": "172068907549.dkr.ecr.eu-west-1.amazonaws.com/aws-fargate-request-proxy:cell9",
     "ImageID": "sha256:71bb81a8e1a15726cc5711f7334e11ccf27f53504724128a75bf3915ace1960b",
     "Labels": {
       "com.amazonaws.ecs.cluster": "arn:aws:ecs:eu-west-1:172068907549:cluster/bullet-srv-999999999999",
       "com.amazonaws.ecs.container-name": "aws-fargate-request-proxy",
       "com.amazonaws.ecs.task-arn": "arn:aws:ecs:eu-west-1:172068907549:task/bullet-srv-999999999999/58f40ddf9c4b42fc83bfc6c7fe1a1cd8",
       "com.amazonaws.ecs.task-definition-family": "bullet-td-6b4120addb68419590ea47b94506ed43-8777b77689784d7a9cd6815f70cf4bc5",
       "com.amazonaws.ecs.task-definition-version": "1"
     },
     "Limits": {
       "CPU": 64,
       "Memory": 40
     }
   }
 ],
 "Limits": {
   "CPU": 1,
   "Memory": 2048
 },
 "AvailabilityZone": "eu-west-1c"
}

(boring parts removed; and 999999999999 is where my account-id appeared)

I wonder what the story behind bullet is. It seems to be the internal name of App Runner, or they changed the name last-minute. Additional data point: At some point, the Console used to link to bullet.yml instead of apprunner.yml.

That fargate-request-proxy sounds interesting, so I tried pulling those images, of course. I wasn’t allowed to.

Other than that, I didn’t notice anything out of the ordinary.

Auto-Scaling

Scaling happens quite quickly.

I’ve configured that each container instance can stomach ten concurrent connections.

Then I overwhelmed the service by going immediately from 0 connections (idle container instance) to 40 concurrent connections. The available container instance responded immediately, as it should. A second container instance started serving requests just 22 seconds later, and the third instance another 28 seconds later.

The maximum number of instances was configured at three, so a fourth instance was never started.

The important part is to configure the concurrency per container instance lower than what it actually can handle. That leaves some headroom until scaling kicks in – or in case one container instance (or even a whole availability zone) fails.

I’d say scaling reacts fast enough for pretty much all use-cases (unless you’re expecting Advent of Code-style traffic peaks).

Throttling

Now this is super interesting.

As mentioned earlier, the application gets throttled almost immediately when there are no active requests.

Let’s put a number on that. Running some python code that does 10,000 iterations of random math stuff, here’s how long that takes:

• ~0.9 seconds on my 2017 iMac
• ~2.5 seconds on App Runner with 1 CPU configured, while active
• ~150 seconds while inactive

Yes, 2.5 minutes. The task is throttled by a factor of 60. While that throttled CPU time is actually free, I’ll assume that mining coins still isn’t worth it, even if memory pricing is small.

Limitations

• Source code: Github only
• Images: ECR only, but ECR Public is supported
• Managed runtimes: Python and Node only
• No VPC support (therefore no EFS support either)
• No support for IPv6, of course
• No support for secrets (Secrets Manager / Parameter Store)
• No support for ECS Exec
• No support for WAF
• No support for ARM (amd64 / “Intel” only)
• No support for HTTP/2
• HTTP/1.0 is not supported
• Backend: no HTTPS support between App Runner proxies and the application – which is perfectly fine with me, because nobody needs that within a VPC, really (even if it’s a VPC in an AWS-owned account)
• Frontend: TLS version 1.2 only
• Frontend: plain HTTP is not supported: port 80 is blocked (will time out) – I think a forced redirect to HTTPS would have made more sense
• Scaling based on concurrent connections only

Many of those items are already on the roadmap.

Premature Launch

What follows is a “grab bag” of nuisances. App Runner is a mere month old now, but still, it feels surprisingly unpolished. Maybe it would have been worth delaying the launch a bit.

That being said, this is all minor stuff and I’m sure they will be resolved quickly.

• Initial setup of Github causes a pop-up window – it took me a minute or two until I realized this (Safari is very good at hiding blocked pop-ups)
• Maybe it’s because I’m a Github noob, but this sentence confused me just as much: “Choose from locations where a GitHub app is already installed or install another elsewhere in your account.” (I didn’t even know Github had “apps”)
• The lack of CodeCommit support at launch is… an interesting statement
• Completing the Github connection gave me this meaningful error: null is not an object (evaluating 'Ke.postMessage') – but, as I figured out later, it was successful anyway
• There is a message “Deleting foo” when you click refresh (that was a new one for me)
• The application setup page needs a clear hint that auto-deploy costs a very silly $1/month – or rather, that silly $1 needs to go away entirely
• A build command is required by the form, even though it’s not needed in many scenarios
• All names need to be at least four characters long – I get upset when I can’t name my things “foo”
• Healthcheck claims to be TCP, when it’s actually HTTP
• Healthcheck interval has a vague definition – description: “interval can’t be less than the timeout”, yet the error message says “must be greater than timeout”
• Healthcheck timeout has some funkyness too – my test application does sleep(2), but a timeout of 3 caused it to fail; setting it to 4 worked
• The log is displayed upside down (newest lines first)… I mean… what?
• When creating the service fails deployment, the log claims to roll back – to what?
• When creating the service fails deployment, it cannot be updated and has to be deleted
• An obvious no-win-situation like command not found when starting the container should fail immediately, but App Runner keeps trying for minutes
• Generally, any kind of screwed up deployment will incur a five to 20-minute penalty, often with no helpful log output whatsoever – having to wait “forever” and having virtually no insight into what’s happening is highly frustrating
• Even better is this:

  12:05:27 [AppRunner] Service resume started.
  12:05:27 [AppRunner] Service status is set to OPERATION_IN_PROGRESS.
  12:22:34 [AppRunner] Service resume failed.
  12:22:35 [AppRunner] Service status is set to PAUSED.
  

  When the application is paused, and it won’t come up healthy, you can only delete it – this means a new URL is generated and updating the custom domain’s DNS records, potentially running into caching issues (depending on the DNS TTL). And yes, that’s all the log there is. Note the timestamps…

• Updating a service that is in the PAUSED state will give this error: Service status must be RUNNING or PAUSED to be updated
• Deployment is slooooow, even when successful
• Initially my application’s stdout wouldn’t show up as log for an hour or so, but later, everything worked as expected – and I couldn’t reproduce that, so maybe it was just my fat fingers
• Feature request: Make the built container available in my ECR (as an export-style option)
• Feature request: Generate apprunner.yml for a manually configured service
• When a healthcheck suddenly fails, the container is correctly restarted – but there is no logging about that whatsoever
• Custom domain: Only CNAME is supported, no Alias (just give me the Hosted Zone ID…?)
• Custom domain: This needs an ACM-style button “this zone lives in Route53, just create the records for me”
• The very last log entry of a deleted service is Service status is set to OPERATION_IN_PROGRESS (no log output to confirm deletion).
• When re-creating a service with the same name, Cloudwatch metrics will from then on be labelled with the new service-id for ActiveInstances and labelled with the old service-id for all request metrics
• Apparently, even pausing a service can fail:

  06-19-2021 03:54 PM [AppRunner] Service pause started.
  06-19-2021 03:54 PM [AppRunner] Service status is set to OPERATION_IN_PROGRESS.
  06-19-2021 04:12 PM [AppRunner] Service pause failed.
  

  And again note the timestamps – 18 minutes and no reason given. Also, here’s the obligatory xkcd about ISO 8601.

Pricing

Official pricing infos here.

Sadly, the smallest possible configuration is 2 GB RAM / 1 vCPU, which puts the monthly cost between $10.5 and $58.50 (depending on the amount of idle time).

There really need to be configurations with less RAM.

Saturday Morning Coffee Hacking

And now for a small detour that happened after finishing this article:

If you payed close attention to App Runner’s build output, you’ll have noticed that the build command is executed twice: In the build environment and as part of the Dockerfile. I can only assume that this maps to the pre-build and build commands of the configuration file. Anyway, this reminded me that I should poke around in the build environment, too.

This is the build environment’s STS get-caller-identity:

[Build] {
[Build]     "UserId": "AROA2W2P3O4YHNMOKQJK7:AWSCodeBuild-2834f7f3-ca90-4f78-b630-0183887718a0",
[Build]     "Account": "736217495344",
[Build]     "Arn": "arn:aws:sts::736217495344:assumed-role/bullet-system-build-role-6b4120addb68419590ea47b94506ed43/AWSCodeBuild-2834f7f3-ca90-4f78-b630-0183887718a0"
[Build] }

Some interesting environment variables:

[Build] BULLET_IMAGE_REPO=736217495344.dkr.ecr.eu-west-1.amazonaws.com/image-repo-6b4120addb68419590ea47b94506ed43:latest
[Build] AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/739b1a1e-ff7d-42b0-9b8c-05b07d3cc414
[Build] N_SRC_DIR=/n
[Build] CODEBUILD_BUILD_URL=https://eu-west-1.console.aws.amazon.com/codebuild/home?region=eu-west-1#/builds/Bufkpuli2wgb7b:96d122fd-7ae9-443f-b021-4a439e4a62fb/view/new
[Build] AWS_EXECUTION_ENV=AWS_ECS_EC2
[Build] BULLET_ASSETS_ACCOUNT_ID=172267607524
[Build] BULLET_AWS_ACCOUNT_ID=736217495344
[Build] CODEBUILD_SOURCE_REPO_URL=fusion-source-prod-eu-west-1-cell18/ufkpuli2wgb7b/d173d836d33a414595e5e5f316ad5e88/app

The ECS /task metadata didn’t reveal anything interesting; as far as I can tell, it’s just a standard CodeBuild environment. Though the image’s tag caught my eye:

      "Image": "570169269855.dkr.ecr.eu-west-1.amazonaws.com/codefactory-eu-west-1-prod-default-amazonlinux2-x86_64-standard:3.0-thirdwave"`.

Interesting that they are using ECS-EC2 instead of Fargate.

Retrieving the container credentials works too, as documented, but refuses to give me the complete credentials: "SecretAccessKey":"***". As it turns out, this is just a clever filter in the log output (in Cloudwatch maybe?) – piping it through base64 makes it possible to liberate the full credentials and use them on any EC2 instance.

This allowed me to conveniently get an ECR authorization token and docker-pull the base image, as well as the image that App Runner creates for me:

[ec2-user@ip-10-0-0-230 ~]$ docker pull 172267607524.dkr.ecr.eu-west-1.amazonaws.com/awsfusionruntime-python3:3.8.5
3.8.5: Pulling from awsfusionruntime-python3
a1534962cb01: Pull complete 
30101d64442c: Pull complete 
Digest: sha256:06cffb6a687d8ca78c477e3de301b2a57650080fefa43259cdbf74308e5ce87f
Status: Downloaded newer image for 172267607524.dkr.ecr.eu-west-1.amazonaws.com/awsfusionruntime-python3:3.8.5
172267607524.dkr.ecr.eu-west-1.amazonaws.com/awsfusionruntime-python3:3.8.5

(same for 736217495344.dkr.ecr.eu-west-1.amazonaws.com/image-repo-6b4120addb68419590ea47b94506ed43)

I couldn’t find much else I can do with this role’s credentials. Neither was it possible to pull aws-fargate-request-proxy nor did the role have any permissions that it shouldn’t have, as far as I could tell from trying a few harmless things. So, while interesting, everything looks sane here.

Conclusion

Conceptually, App Runner is great – it’s very simple, yet fits a lot of simple web applications perfectly. Also the billing model is a huge step in the right direction to reduce CPU time waste.

I’m really curious to see how App Runner will evolve – especially how the team will fight off feature creep. Some additions will be necessary (VPC support for example), but keeping the service simple will be very tough. If they add everything someone might be wishing for, we’ll end up where we started before App Runner came along.

Discuss and/or follow on Twitter!

This post will explore a CodeBuild-based solution for periodic mirroring of images.

Background

Container images need to be pulled from somewhere. By default, that’s Docker Hub. So when I’m using simply FROM nginx in my Dockerfile, that’s actually docker.io/nginx.

This is fine for an occasional build process.

But it’s quite problematic for automated use. For example, testing that runs at high frequency can cause a significant amount of pulls. The same goes for frequent deployments or ECS tasks, or any trigger that’s based on a public docker image. Especially problematic are misconfigured / failing services, which will perpetually try to start a service, pulling the image each time.

Understandably, Docker Hub is unwilling to serve massive amounts of – basically unnecessary – pulls for free. So I can’t really blame them for introducing those rate limits in November 2020, and I think their rate limit for free usage is reasonable. Also keep in mind that many images are rather large – sizes beyond one gigabyte are common.

This rate limiting becomes a significant issue on AWS though: In almost all scenarios, the pull request will be issued from an AWS IP address that’s effectively random. There’s no way of knowing how many pulls have already been made from that address today. Therefore, any pull from AWS has a good chance of failing with this error: Error response from daemon: toomanyrequests: You have reached your pull rate limit.

Other Issues

Using a frequently-pulled image from an external source has more issues besides rate-limiting errors:

• It’s wasteful: This is a lot of data transfer over the public internet – ingress traffic is free on AWS, but that’s no excuse for completely avoidable significant traffic

• It’s slow: Pulling an image from the public internet takes significantly longer than using an image hosted in my AWS region.

• It’s unreliable: As with any external service, there’s always an increased chance that it’s unreachable (for network or other issues – not to mention that there’s no SLA for a free service)

Generally speaking, not pulling the same image hundreds of times is just good manners.

Workarounds

AWS responded by publishing advice on the AWS blog.

Basically, the options are:

• Paying for a Docker Hub account. This might seem like the right thing to do, but doesn’t address any of the other issues.

• Using the AWS Public ECR. AWS managed to launch this incredibly fast after Docker’s announcement, and I believe it’s the best long-term solution. But as of today, many common images are not available there. I think a marketing push would be in order…

• Manually copying the required images from Docker Hub to my AWS account’s ECR.

In most cases, using the account’s private ECR is the most reasonable path out of this mess. But now I have a classic Day 2 problem: How do I make sure my mirrored image is up to date?

As with anything in life, the answer is automation.

Well, once again I’m writing too much. So let’s get to it.

Overview

My solution will be based on these components:

• CodeBuild provides the environment to pull an image from Docker Hub and then push it to my ECR

• EventBridge to trigger CodeBuild on a fixed schedule

• ECR to store the copied images

An important goal is to easily do this for multiple images, so the CodeBuild project has to accept the image name as a parameter.

I will use the public images of nginx and logstash as examples.

ECR

I need to create the repositories for the images to be mirrored.

• “Create repository” in the ECR Console
• Use nginx as repository name
• For everything else, defaults are fine
• Repeat for logstash and any others as necessary

Note the registry URL of the private ECR repositories – something like 123456789012.dkr.ecr.eu-central-1.amazonaws.com (without the trailing /nginx).

CodeBuild Project

Then I’ll create the CodeBuild project that takes care of mirroring:

• “Create build project” in the CodeBuild Console
• Use some beautiful name like foo or ecr-mirror
• As Source, I select No source
• Under Environment,
  • I’ll use any next-best-latest Linux image; currently, that is
    • Operating system: Amazon Linux 2
    • Runtime: Standard
    • Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
  • Enable Privileged
  • Use a New service role (note the name!)
  • In Additional configuration,
    • add an Environment variable named REGISTRY, using the registry URL from the ECR step
    • leave everything else as it is
• In Buildspec, I’ll Switch to editor, remove all suggested lines, and paste the build specification given below
• Finally, Create build project

Here’s the build specification to use:

version: 0.2

phases:
  build:
    commands:
       - aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
       - docker pull $PREFIX$IMAGE
       - docker tag $PREFIX$IMAGE $REGISTRY/$IMAGE
       - docker push $REGISTRY/$IMAGE

Finally, hidden in the project’s Build details tab, I’ll note the Project ARN.

Adjust IAM Permissions for CodeBuild

CodeBuild created a basic IAM Role for my project, but this role is not allowed to access my ECR repositories yet. To fix that:

• In the IAM Console, I locate the Role that has been created by CodeBuild (named as noted above; should be codebuild-ecr-mirror-service-role)
• Attach policies: AmazonEC2ContainerRegistryPowerUser

EventBridge Rule

I’ll use EventBridge to periodically update the images.

• In the EventBridge Console, Create rule
• name it ecr-mirror
• As pattern, I’ll select Schedule
• I could use a fixed rate, but to have better control, I’ll use a Cron expression: 10 0 * * ? * (that’s 00:10 UTC every day)
• As Target, I select a CodeBuild project and use the Project ARN from above
• Expand Configure input, and there
  • select Constant (JSON text)
  • paste the JSON given below (for nginx)
• Add target (for logstash)
  • using the same CodeBuild target and ARN
  • using the same JSON text, but replacing nginx:stable with logstash:latest
• Leave the rest as is and Create

{
	"environmentVariablesOverride": [
		{
			"name": "IMAGE",
			"type": "PLAINTEXT",
			"value": "nginx:stable"
		}
	]
}

It will be a single line after pasting it – that’s fine.

Testing

Scheduled jobs are annoying to test. I could adjust the Cron specification to trigger a few minutes from now.

A faster (but less thorough) way to test is the Start build with overrides button in the CodeBuild project overview. There under Environment variables override, I’ll Add environment variable named IMAGE with a value of, for example, logstash:7.11.2.

In the resulting build process, I like to click Tail logs.

By the way, CodeBuild will log to CloudWatch by default (in the /aws/codebuild/ecr-mirror log group, unless changed).

Next Steps

I now have a mirror of the images I need, updated daily.

What’s left is changing all references to the image, e.g. in my ECS task definitions and Dockerfiles. Instead of nginx, I’ll change those to use 123456789012.dkr.ecr.eu-central-1.amazonaws.com/nginx. This requires a login to that registry first, of course. ECS handles this transparently, but it needs the appropriate IAM permissions. Same for EKS.

Further improvements could be made:

For one, narrowing down CodeBuild’s IAM permissions to the appropriate repositories might be a good idea.

And for mirroring more than a few images, the CodeBuild job could be adjusted to retrieve the list of images from somewhere else, for example a StringList in the SSM Parameter Store.

The Docker in the Room

You might have noticed that this CodeBuild solution is plagued by the same problem initially described here: There’s a good chance that it will fail because of Docker Hub’s rate limiting.

Honestly, I don’t have a good idea for that. Most of the time it works. When it does not, a manual Retry build usually helps. So yes, a daily update could fail for one or two days – but usually, that’s not an issue. And if it is, you can always intervene manually and/or increase the mirroring frequency.

All in all, the whole situation is a mess without a good solution. But this one is better than nothing / mirroring by hand.

Let me know how you handle images!

Discuss and/or follow on Twitter!

This post is part of a blog series: Road to re:Web.

The AWS sign-up is a pretty scary experience for individuals without prior AWS experience.

But it’s an experience that became a motivation for me.

A Leap of Faith

For individual users, every dollar counts. Every dollar I save is a dollar I got. This is different from companies, where expected savings must be significant to justify expensive humans spending time on cost optimization.

And individual users isn’t just hobbyists – it’s students, freelancers, FOSS developers, small non-profits, people eager to “learn Cloud” and many others.

As an individual user, signing up for AWS is scary. I remember that vividly. It took me weeks to get over that paranoia. That’s my credit card, dammit. If I misunderstand something, or I forget to turn something off, or I don’t pay attention and click some super-expensive option instead of the covered-by-free-tier option, it’s my ass that gets fucked – this can go into four-digits territory fascinatingly fast. This gets less intimidating quickly with some AWS experience, but that’s kinda Catch 22 for completely novice users.

And it’s not unjustified fear: There’s a lot of such horror stories out there. To be fair, most of them ended well, because AWS seems to be very understanding with rookie mistakes. This is good to hear, but a far cry from any guarantee that I won’t be broke tomorrow.

In the end, I took a leap of faith and signed up.

Cost Optimization as a Mindset

So yeah, the whole sign-up billing fear is off-putting – and I bet AWS is losing tons of future power users and decision makers this way.

I also bet a lot of services are never tried because it just feels too risky to an AWS novice, or the pricing information is just too confusing.

But the experience has taught me a very valuable lesson, once I made the jump: It forced me to go on a first-name basis with AWS billing. It makes me think about costs constantly. I’m always on high alert when I’m trying new things on my personal AWS account. I carefully read the pricing information of services. I check that Billing Dashboard religiously. That is, of course, in addition to the recommended safeguards like configuring Budgets.

I believe most AWS users act differently if they haven’t been through the experience themselves – if it’s not their own credit card that’s backing the charges. It’s just like security – either you actually care and consider this in your decisions, or you do not care. Neither security nor cost-awareness can be bolted onto projects easily once they’re “done”.

… and as a Motivation

With my personal account, I’m a hobbyist who has some nonsense projects running on AWS.

But designing and building things on AWS for customers is also my day job, so costs are always a topic.

It’s kinda weird, but cost optimization has now become a bit of a game for me. It’s a complex issue to think about, and what we would have considered a good and cost-efficient architecture back in the Olden Days can have completely different cost characteristics on AWS, especially with changing usage patterns.

It’s a fun challenge, and it’s part of my motivation to push serverless architectures with re:Web.

So that’s the reward for this leap of faith, I guess.

Discuss and/or follow on Twitter!

This post is part of a blog series: Road to re:Web.

The road that led me there shall henceforth be remembered through a series of blog posts.

This very post will service as table of contents, if you will.

• Part 1: AWS Billing: A Leap of Faith

• Part 2: Timeless Services

• Part 3: Amazon Timestream and the Grafana use-case

• Part 4: The Epiphany of Lambda Container Image Support

• Part 5: Lambda Deep Dive: Architecture, Runtime API

• Part 6: Lambda Deep Dive: Connecting to ALB / API Gateway

• Part 7: Putting It All Together

• Part 8: Packaging / Distribution Insane Asylum

This post will be updated as I complete the individual blog posts. Culmination can be expected in about two dozen months. Give or take.

The above list is highly likely to be subject to change.