Sign in Subscribe

If you squint really hard, AWS is just a very large, "mostly consistent" API for acquiring all sorts of compute, networking, and storage. Like a good API, credentials are required for authentication and authorization in order to execute actions.

TLDR: When working with the AWS API interactively at a terminal (via CLI or some other program that uses an AWS SDK), you should have one periodically-rotated credential (that is carefully guarded and secured) that you use to generate temporary credentials that allow access your AWS account(s).  Your one credential can be as enterprise-grade as a username password combination from an identity store (with MFA) that has federated access or as simple as an IAM user with an MFA token.

The Credential Provider Chain

There are numerous places the AWS CLI and SDKs look to retrieve API credentials for AWS.  To name a few-- on an EC2 instance, they can be retrieved from the instance metadata. In a Lambda function, they're available as environment variables.  In an ECS task, the ECS agent makes credentials available through a task-specific URL.  When using the AWS CLI or a supported AWS SDK (like boto3 for Python), the CLI or SDK automatically searches through a credential provider chain that looks in all those places for credentials.  The linked document is for the Java SDK, but all the SDKs and the AWS CLI do something similar.  As soon as credentials are found at a link in the chain, the CLI or SDK attempts to use them.

Forms of AWS API Credentials

AWS API credentials come in two forms- non-expiring access keys and temporary tokens.  Non-expiring access keys consist of an access key ID (which is not a secret) and a secret access key.  Temporary tokens expire according to the expiration period of the method and principal used to retrieve them.  Temporary tokens have an access key ID, a secret access key and a session token.

Only IAM users have non-expiring access keys.  It's a best practice to rotate IAM user access keys at least every 90 days.  All other credentials are temporary and created either by assuming an IAM role or by generating temporary credentials from a set of IAM user access keys.

The following APIs are used to assume IAM roles to generate temporary tokens:

  • sts:AssumeRole
  • sts:AssumeRoleWithWebIdentity
  • sts:AssumeRoleWithSAML

The following APIs are used to generate temporary tokens from an IAM user without assuming an IAM role:

  • sts:GetFederationToken
  • sts:GetSessionToken

Where to Store Credentials

Use Environment Variables

Keep the temporary tokens you are using to call AWS API keys in environment variables.  This is the simplest, cleanest way to allow all command-line tools to use those credentials.  Many tools will offer a way keep credentials in their own config files-- do not do this.  Under the hood, any tool for interacting with AWS should be using an AWS SDK the leverages the default credential chain and can retrieve credentials from environment variables.

Limit Non-expiring Access Keys

Keeping non-expiring access keys in your ~/.aws/credentials file is tempting.  However, this is not secure.  Instead, store one set of non-expiring IAM user access keys in your credentials file.  Ensure that file is secure (e.g., do not allow read permissions to other, only user and group).  These access keys should only allow assuming IAM roles (and that only with MFA using sts:GetSessionToken).

Even Better Security

Even better, store your IAM user access keys in an encrypted vault on your local disk.  For the best solution, use identity federation with SAML if you already have a central identity provider with those capabilities. However, don't let "best" be the enemy of "good."  Well-secured IAM user access keys are a solid solution.

So, which environment variables?

The specific environment variables that need to be set with the outputs of the API calls listed above are:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN

As a bonus, set the AWS_DEFAULT_REGION to ensure API calls to AWS are sent to the right regional endpoint.

An Example Implementation

Eric Hammond has a great post demonstrating implementation of this approach to run AWS CLI commands across multiple AWS accounts.  His github repository has shell scripts for:

  1. Using an IAM user access key in a credentials file plus an MFA token to generate MFA-protected temporary tokens
  2. Using those temporary tokens to assume an IAM role to generate temporary tokens in with actual privileges
  3. Using those temporary tokens to run AWS CLI commands

Incredibly, those shell scripts also perform this without exporting any of the environment variables I listed above.  Environment leakage between users is largely a thing of the past.  However, when you can improve security without impacting user experience, it's a win-win.

Let's walk through each of those steps above in detail to understand the configuration.  The purpose here is to illustrate.  If you're actually setting up your environment to do this, use Eric's scripts.  We'll use the AWS CLI and the us-east-1 region for these examples.

Creating the Initial IAM User

We first need to have an IAM user that will be our primary user.

There's a chicken and egg problem here because in order to create that initial IAM user we will already need some admin-level credentials.  This is a common scenario on AWS.  You will need highly privileged credentials (through IAM user access keys or a Console login) to get started.  That's OK.  The important thing to remember is that we are moving past using these long-lived highly privileged credentials for day-to-day usage.

So, let's assume you have IAM user access keys with admin privileges in a default credentials profile.  Your ~/.aws/credentials file will look something like this:

[default]
aws_access_key_id = ASDFWEGWEWERWDF
aws_secret_access_key = kkObg8Z40Ht8Md8Z7jh+bbGfFs2tIDY9HPZ6CFar

You can use the AWS CLI (version 2 only) to generate a JSON or YAML file with the skeleton of a CLI command's parmeters.  Here we generate a skeleton file for the iam:CreateUser API call:

$ aws iam create-user --generate-cli-skeleton yaml-input > user.yaml

After editing the skeleton file to add required and desired values and removing unwanted optional parameters, my user.yaml looks like this:

Path: /           #  The path for the user name.
UserName: primary # [REQUIRED] The name of the user to create.
Tags:             # A list of tags that you want to attach to the newly created user.
  - Key: owner
    Value: Jeremy Axmacher
  - Key: email
    Value: jeremy@obsoleter.com

Now, I can use that to create my IAM user:

$ aws iam create-user --cli-input-yaml file://user.yaml
{
    "User": {
        "Path": "/",
        "UserName": "primary",
        "UserId": "AIDA3TAJZULNEEEMOCAQT",
        "Arn": "arn:aws:iam::111122223333:user/primary",
        "CreateDate": "2021-01-08T22:33:21+00:00",
        "Tags": [
            {
                "Key": "owner",
                "Value": "Jeremy Axmacher"
            },
            {
                "Key": "email",
                "Value": "test@example.com"
            }
        ]
    }
}

Then we create a virtual MFA device and save the QR code as a PNG image to enable it for the IAM user (you can also get the Base32 encoded virtual MFA seed if you want a text-based seed).  Use a friendly device name because that will display in your virtual MFA app:

$ aws iam create-virtual-mfa-device --virtual-mfa-device-name primary-111122223333 --outfile qr.png --bootstrap-method QRCodePNG
{
    "VirtualMFADevice": {
        "SerialNumber": "arn:aws:iam::111122223333:mfa/primary-111122223333"
    }
}

I can then open the qr.png file and scan it with the Google Authenticator app (or some other Virtual MFA device application).  This will allow the Google Authenticator app to start generating time-based one time passwords.

Before it can be used the virtual MFA device needs to be verified and connected to the IAM user.  The mfa device serial number from the previous command's output is required.  We also need two sequential MFA codes from our Google Authenticator app.

$ aws iam enable-mfa-device --user primary --serial-number arn:aws:iam::111122223333:mfa/primary-111122223333 --authentication-code1 600699 --authentication-code2 039072

Now we create an IAM role with admin privileges that can be used from the IAM user.

$ aws iam create-role --generate-cli-skeleton yaml-input > role.yaml
... edit role.yaml ...
$ cat role.yaml
Path: /                   #  The path to the role.
RoleName: primary-role    # [REQUIRED] The name of the role to create.
Description: Primary role # A description of the role.
MaxSessionDuration: 3600  # The maximum session duration (in seconds) that you want to set for the specified role.
                          # One hour (3600 seconds) is a good default.
# [REQUIRED] The trust relationship policy document that grants an entity permission to assume the role.
AssumeRolePolicyDocument: |
  {
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Principal": {
        "AWS": [
          "arn:aws:iam::111122223333:user/primary"
        ]
      },
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuth": "true"
        }
      }
    }]
  }
Tags:                     # A list of tags that you want to attach to the newly created role.
- Key: owner
  Value: Jeremy Axmacher
- Key: email
  Value: test@examples.com
$ aws iam create-role --cli-input-yaml file://role.yaml
{
    "Role": {
        "Path": "/",
        "RoleName": "primary-role",
        "RoleId": "AROACTAJZTLIESHOFZGZO",
        "Arn": "arn:aws:iam::111122223333:role/primary-role",
        "CreateDate": "2021-01-08T18:27:34+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "AWS": [
                            "arn:aws:iam::111122223333:user/primary"
                        ]
                    },
                    "Condition": {
                        "Bool": {
                            "aws:MultiFactorAuth": "true"
                        }
                    }
                }
            ]
        },
        "Tags": [
            {
                "Key": "owner",
                "Value": "Jeremy Axmacher"
            },
            {
                "Key": "email",
                "Value": "test@examples.com"
            }
        ]
    }
}
$ aws iam attach-role-policy --role-name primary-role --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

You'll notice that the AssumeRolePolicyDocument allows sts:AssumeRole from the IAM user as long as MFA is present.  We also attach the AWS managed Administrator policy to grant admin privileges to the role.

Finally, we need to grant matching sts:AssumeRole permission to our IAM user to allow it to assume the role. However, we do not need the MFA condition on this policy otherwise we would have to enter an MFA token twice.

$ aws iam put-user-policy --generate-cli-skeleton > policy.yaml
... edit policy.yaml ...
$ cat policy.yaml
UserName: primary             # [REQUIRED] The name of the user to associate the policy with.
PolicyName: assume-primary-role # [REQUIRED] The name of the policy document.
# [REQUIRED] The policy document.
PolicyDocument: |
  {
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::111122223333:role/primary-role"
    }]
  }
$ aws iam put-user-policy --cli-input-yaml file://policy.yaml

Our final setup step is to create an access key for the IAM user:

$ aws iam create-access-key --user primary

Now we can remove the access key in the default profile of the ~/.aws/credentials file and replace them with the access key ID and secret access key from the iam:CreateAccessKey call we just ran.  After making that change, sts:GetCallerIdentity shows the IAM user we created:

$ aws sts get-caller-identity
{
    "UserId": "AIDA3TAJZULNEEEMOCAQT",
    "Account": "111122223333",
    "Arn": "arn:aws:iam::111122223333:user/primary"
}

Generating MFA-protected temporary tokens for an IAM user

Based on the inline IAM policy we've given our IAM user, there is only one thing it is permitted to do- assume the IAM role we created.  In order to do that, we must first generate temporary credentials from the IAM user with its virtual MFA device.

We need the serial number of MFA device we setup earlier to run the following command as well as the current virtual MFA device one-time password to supply as the token-code.  We also supply a credential duration of 900 seconds which is more than enough time to assume the IAM role.  The default duration of 12 hours is longer than needed.

$ aws sts get-session-token --serial-number arn:aws:iam::111122223333:mfa/primary-111122223333 --token-code 359323 --duration-seconds 900

This returns a new access key ID, secret access key, and session token.  A quick way to run this command and convert the credentials into environment variables is as follows:

$ read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<<$(aws sts get-session-token --serial-number arn:aws:iam::111122223333:mfa/primary-111122223333 --token-code 948152 --duration-seconds 900 --query 'Credentials.[AccessKeyId, SecretAccessKey, SessionToken]' --output text)

Now we have MFA-protected temporary tokens for the IAM user in environment variables.  This means that further AWS CLI calls that do not use the --profile flag to directly specify a credentials profile will use credentials found in those environment variables.

Assuming an IAM role with MFA-protected IAM user credentials

Based on the inline IAM policy we've given our IAM user, there is only one thing it is permitted to do- assume the IAM role we created.

We need the serial number of MFA device we setup earlier to run the following command as well as the current virtual MFA device one-time password to supply as the token-code.  We also supply a credential duration of 3600 seconds  (one hour) which gives us plenty of time to use our IAM role.  The default duration of 1 hour set on role creation will be used if duration is not specified.  If we try to specify a long duration than allowed on the role, our assume role call will fail.

Now we're ready to assume the IAM role.  The IAM role we created has admin privileges, so the temporary tokens returned by this API call will allow administrator privileges to the AWS account.

$ aws sts assume-role --role-arn arn:aws:iam::111122223333:role/primary-role --role-session-name Jeremy --duration-seconds 14400 --serial-number arn:aws:iam::111122223333:mfa/primary-111122223333 --token-code 123456

This returns a new access key ID, secret access key, and session token.  We can use a similar command as before to convert the credentials into environment variables:

$ read -r access_key secret_key session_token <<<$(aws sts assume-role --role-arn arn:aws:iam::111122223333:role/primary-role --role-session-name Jeremy --duration-seconds 14400 --serial-number arn:aws:iam::111122223333:mfa/primary-111122223333 --token-code 123456 --query 'Credentials.[AccessKeyId, SecretAccessKey, SessionToken]' --output text)
$ AWS_ACCESS_KEY_ID=$access_key AWS_SECRET_ACCESS_KEY=$secret_key AWS_SESSION_TOKEN=$session_token aws sts get-caller-identity
{
    "UserId": "AROA4TQXZZLNEAHKQRINO:Jeremy",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/primary-role/Jeremy"
}

After setting non-exported environment variables, we can run sts:GetCallerIdentity to show execution of an AWS CLI command with the IAM role credentials.

Using IAM role credentials

Now that we have IAM role credentials with admin privileges, we can use them to make API calls to AWS.  In the previous example, we explicitly set the credential environment variables at the start of the aws sts get-caller-identity command.  This would be cumbersome to do repeatedly.  Eric Hammond's scripts get around this by defining a Bash function.  Here is a simplified version using the variable names from the read command above:

$ role-exec() {
    AWS_ACCESS_KEY_ID=$access_key AWS_SECRET_ACCESS_KEY=$secret_key AWS_SESSION_TOKEN=$session_token "$@"
}
$ role-exec aws sts get-caller-identity
{
    "UserId": "AROA4TQXZZLNEAHKQRINO:Jeremy",
    "Account": "111122223333",
    "Arn": "arn:aws:sts::111122223333:assumed-role/primary-role/Jeremy"
}

Rotating IAM user access keys

Even though our IAM user only has permission to assume our admin IAM role (and only with MFA), we still need to remember to rotate those IAM user access keys at least every 90 days.  Because IAM user access keys do not expire, if they are leaked they grant access to all future permissions you may grant to that IAM user.

Summary

When working with the AWS API interactively at a terminal (via CLI or some other program that uses an AWS SDK), you should have one periodically-rotated credential (that is carefully guarded and secured) that you use to generate temporary credentials that allow access your AWS account(s). Your one credential can be as enterprise-grade as a username password combination from an identity store (with MFA) that has federated access or as simple as an IAM user with an MFA token.

My goal here was to show step-by-step how to configure, manage and use AWS API credentials for interactive access (i.e., shell/terminal sessions) in a secure way.  It is very simple to keep long-lived IAM user access keys with admin privileges in a credentials profile on disk. However, with a little bit more effort secure CLI/SDK usage can be achieved with only a minor decrease in usability.  It's a worthwhile tradeoff that can help keep you or your company from becoming the next big data breach headline.

Managing AWS API Credentials

When working with the AWS API interactively, each user should have one periodically-rotated credential that is used to generate temporary credentials that allow access your AWS account(s). In this post, I show you a sample implementation of this pattern for managing AWS API credentials.