AWS: On IAM Policies for the Systems Manager Agent

2023-10-31

sheep jumping over a fence

That’s Not Supposed to Work!

Every now and then, I run a command that I expect to fail; usually to verify the system’s behavior before and after a change. So I was querying a secret from the Parameter Store on some throwaway EC2 instance with no special permissions – and it worked. Oops!

IAM Policies for the SSM Agent

The Systems Manager Agent is the recommended way to manage EC2 instances, e.g. for automated patching and as an alternative to SSH.

Over the years, different IAM Policies were recommended for the SSM Agent:

Since 2019, the recommended policy was AmazonSSMManagedInstanceCore (setup instructions archive link). Unfortunately, this policy allows to read all parameters from the Parameter Store (ssm:GetParameter for *).

This sounds rather harmless at first, but Parameter Store can also store secrets (type SecureString). It’s pretty common to use Parameter Store for secrets, as the Secrets Manager service charges $0.40 for every single secret (and its additional features are usually not needed). So my EC2 instance was able to access all secrets in the AWS account, just because it was running the SSM Agent.

The previous policy was even worse: AmazonEC2RoleforSSM was recommended until 2019. To this day, it includes full read and write access to all S3 buckets in the account (s3:PutObject and s3:GetObject for *) – and the mentioned permissions to Parameter Store as well. The role has not been deprecated yet, but the policy’s documentation says this is supposed to happen “soon” and suggests to use AmazonSSMManagedInstanceCore instead.

Since fall of 2022, there is a new policy AmazonSSMManagedEC2InstanceDefaultPolicy. The only difference to the AmazonSSMManagedInstanceCore role is that it doesn’t include permissions to the SSM Parameter Store. This policy is meant to be used with the new Default Host Management Configuration (a great overview of how this works under the hood can be found on Aidan Steele’s blog).

Impact

According to a recent scientific study¹, almost two in three customers use one of the older policies – likely unaware that it might allow unintended access to all secrets (and all S3 buckets, in case of the oldest policy). That’s not too surprising, as AmazonSSMManagedInstanceCore still is recommended as part of the Cloudwatch Agent setup and for the SSM Agent setup without DHMC.

Alternatives

The way forward is to adopt DHMC. If that’s not an option, permission boundaries can help, or simply replacing the policy with the newer AmazonSSMManagedEC2InstanceDefaultPolicy or a custom policy.

Another option is using a custom KMS key to encrypt the parameters. In that case, the parameter could still be read, but decrypting its value would fail unless the EC2 instance has permission to use the key.

Conclusion

The most flexible and – usually – most secure option is to write a custom IAM policy for everything. Most customers use the provided Managed IAM Policies instead, and the AWS service documentations usually recommend doing so. Given the amount of services and the complexity of IAM interactions, I’d even say it’s not feasible to write and maintain custom policies for every use-case, except for organizations with dedicated security teams.

Still, Managed Policies should not be trusted blindly. Those policies can change at any time², and it’s always easy to miss something – like I did in this case.