A while back we wanted to set up a single pipeline with deployments of the same application to multiple AWS accounts so we needed to figure out a good clean way to configure the credentials for that.

Most people with some experience with AWS and IAM know that you can define the credentials to be used with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but what if you need to switch between two different pairs of credentials? Gitlab CI for example will not let you redefine variables defined outside the .gitlab-ci.yml file inside the file itself. Luckily there are a few less known AWS CLI environment variables that you can use: AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE combined with AWS_PROFILE.

So what do they do exactly? From the docs:

AWS_CONFIG_FILE – Specifies the location of the file that the AWS CLI uses to store configuration profiles. The default path is ~/.aws/config.

AWS_SHARED_CREDENTIALS_FILE – Specifies the location of the file that the AWS CLI uses to store access keys. The default path is ~/.aws/credentials.

AWS_PROFILE – Specifies the name of the AWS CLI profile with the credentials and options to use. This can be the name of a profile stored in a credentials or config file, or the value default to use the default profile.

The usual AWS file locations we mostly know already can be modified, what does that give us? In Gitlab CI you can define two types of variables: Variable and File. Variable is a simple environment variable, but File is special: you define the key and value and what Gitlab CI does is that it will write the value somewhere as a file and will add an environment variable with the key where the value is the path of the file.

This fits our needs perfectly: we can define our own credentials file inline and then we just need to switch between different AWS profiles in the pipeline. So what does that look like?

.gitlab-ci.yml example with deployment to two different ECS clusters in different AWS accounts. Note the value of the AWS_PROFILE environment variable.

production:
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-ecs:latest
  stage: release
  script:
    - ecs update-task-definition
    - aws ecs wait services-stable --cluster $CI_AWS_ECS_CLUSTER --services $CI_AWS_ECS_SERVICE
  variables:
    CI_AWS_ECS_CLUSTER: services
    CI_AWS_ECS_SERVICE: backend
    CI_AWS_ECS_TASK_DEFINITION: backend
    CI_APPLICATION_REPOSITORY: ${ECR_REPOSITORY_URL}
    CI_APPLICATION_TAG: ${CI_COMMIT_SHORT_SHA}
    AWS_PROFILE: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

staging:
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-ecs:latest
  stage: release
  script:
    - ecs update-task-definition
    - aws ecs wait services-stable --cluster $CI_AWS_ECS_CLUSTER --services $CI_AWS_ECS_SERVICE
  variables:
    CI_AWS_ECS_CLUSTER: services
    CI_AWS_ECS_SERVICE: backend
    CI_AWS_ECS_TASK_DEFINITION: backend
    CI_APPLICATION_REPOSITORY: ${ECR_REPOSITORY_URL}
    CI_APPLICATION_TAG: ${CI_COMMIT_SHORT_SHA}
    AWS_PROFILE: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

There are probably a lot of different ways to do that, but this is what worked for us. Same system can be used to assume a role as well, you just need to specify the relevant role in your credentials file under the profile with role_arn attribute.