AWS: On IAM Policies for the Systems Manager Agent2023-10-31
That’s Not Supposed to Work!
Every now and then, I run a command that I expect to fail; usually to verify the system’s behavior before and after a change. So I was querying a secret from the Parameter Store on some throwaway EC2 instance with no special permissions – and it worked. Oops!
IAM Policies for the SSM Agent
Over the years, different IAM Policies were recommended for the SSM Agent:
Since 2019, the recommended policy was
AmazonSSMManagedInstanceCore (setup instructions archive link). Unfortunately, this policy allows to read all parameters from the Parameter Store (
This sounds rather harmless
at first, but Parameter Store can also store secrets (type
SecureString). It’s pretty common to use Parameter Store for secrets, as the Secrets Manager
service charges $0.40 for every
single secret (and its additional features are usually not needed). So my EC2 instance was able to access all secrets in the AWS account, just because it was running the SSM Agent.
The previous policy was even worse:
was recommended until 2019. To this day, it includes full read and write access to all S3 buckets in the account (
*) – and the mentioned permissions to Parameter Store as well. The role has not been deprecated yet, but
the policy’s documentation says this is supposed to happen “soon” and suggests to use
Since fall of 2022, there is a new policy
AmazonSSMManagedEC2InstanceDefaultPolicy. The only difference to the
AmazonSSMManagedInstanceCore role is that it
doesn’t include permissions to the SSM Parameter Store. This policy is meant to be used with the new Default Host Management Configuration (a great overview of how this works under the
hood can be found on Aidan Steele’s blog).
According to a recent scientific study1, almost two in three customers use one of the older policies – likely unaware
that it might allow unintended access to all secrets (and all S3 buckets, in case of the oldest policy). That’s not too surprising, as
AmazonSSMManagedInstanceCore still is recommended as part of the Cloudwatch Agent setup and for the SSM Agent setup without DHMC.
The way forward is to adopt DHMC. If that’s not an option, permission boundaries can help, or simply replacing the policy with the newer
AmazonSSMManagedEC2InstanceDefaultPolicy or a custom policy.
Another option is using a custom KMS key to encrypt the parameters. In that case, the parameter could still be read, but decrypting its value would fail unless the EC2 instance has permission to use the key.
The most flexible and – usually – most secure option is to write a custom IAM policy for everything. Most customers use the provided Managed IAM Policies instead, and the AWS service documentations usually recommend doing so. Given the amount of services and the complexity of IAM interactions, I’d even say it’s not feasible to write and maintain custom policies for every use-case, except for organizations with dedicated security teams.
Still, Managed Policies should not be trusted blindly. Those policies can change at any time2, and it’s always easy to miss something – like I did in this case.
Other links to this article: Twitter
Update 2023-11-01 – updated to reflect that the issue only applies to parameters encrypted with the default KMS key (thanks to Aidan Steele for keeping me on my toes!)