Overview

One thing I HATE with a passion is the lack of examples for valid IAM permissions in conjunction with tools for AWS. The AWS docs and examples from IAM are always too simple and/or the project’s docs are hit or miss for covering most of the use cases for properly locking down your instances in a sane matter. This tutorial will help show how to correctly harden your instances when using Credstash.

For secrets in AWS its hard to beat Credstash. Credstash allows you to leverage AWS Dynamodb and KMS services to create a simple secret store. One of my favorite parts about Credstash is that it allows you to set different encryption contexts per secret. Coupling this with IAM instance profiles for specific KMS decryption contexts and you’ve got a nice way of isolating secrets between different EC2 instances.

The example below will cover the setup of KMS decryption in IAM to silo secrets. For instructions on how to setup Credstash take a look at their documentation.

Credstash encryption contexts are set when putting a secret:

credstash put 'secretkey' 'secretval' <contextkey>=<contextval>

If you want to retrieve that same key assuming you have access to decrypt the key in KMS:

credstash get 'secretkey' <contextkey>=<contextval>

Example

Now we will setup a secret that will only be available to machines set with a IAM instance profile to the dev KMS encryption context:

credstash put 'asecretkey' '12345supersecret' env=dev

Locking down an encryption context for IAM instance profile is simple. Create an IAM instance profile with that includes the following policy (Make sure to substitute your own region, aws account, kms key, and dynamodb table):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "kms:Decrypt"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:kms:us-east-1:111111111111:key/a12345678-b123-c123-d1234567890"
      "Condition": {
        "StringEqualsIfExists": { "kms:EncryptionContext:env": "dev" }
      }
    },
    {
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:Query",
        "dynamodb:Scan"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:us-east-1:111111111111:table/credential-store"
    }
  ]
}

Now attach that profile to an EC2 instance and run credstash from that instance. You should be able to decrypt the only secrets set with the context from above: dev or secrets with no context at all. The following will work if the above steps were completed correctly:

credstash get 'asecretkey' env=dev

That’s all there is to it, now you can lock down your secrets and be sure only the correct environments have access!

Things to note:

You can add on additional encryption contexts to a secret. But this will require IAM permissions for all contexts to be able to successfully decryption the secret with KMS.
If you omit the context during the put then get will still work with or without the condition in the kms:Decrypt section of the IAM policy. I refer to these kinds of secrets as globals.

Errors

If you receive a KMS ERROR then check your context and IAM policy. Incorrect decryption policy errors will look something similar to:

KMS ERROR: Decryption error An error occurred (AccessDeniedException) when
calling the Decrypt operation: The ciphertext references a key that either doe
not exist or you do not have access to.

Sarcastic Admin

Overview

Example

Errors