AWS Secret Manager Integration with EKS

In this post, let’s discuss how to create secrets using Secret Manager and how to mount the secrets to the EKS pods using the AWS Secrets CSI driver Provider.

We can make parameters stored in Parameter Store and secrets stored in Secrets Manager appear as files mounted in Kubernetes pods by using the AWS provider for the Secrets Store CSI Driver.

With the AWS Secrets and Configuration Provider (ASCP), we can store and manage the secrets in Secrets Manager and then retrieve them through the workloads running on Amazon EKS.

We can restrict access to the secrets from the EKS pods using IAM roles and policies. The pod identification will be retrieved by the ASCP and then exchanged for an IAM role. ASCP assumes the pod’s IAM role, and then it can obtain secrets from the Secrets Manager.

Follow the steps below to install the Secret Store CSI Driver and ASCP

1. Install the Secrets Store CSI Driver

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts 
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set enableSecretRotation=true --set syncSecret.enabled=true --version 1.2.4

2. Install ASCP

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Let’s see how to create and mount a secret in an EKS pod.

1. Set the AWS Region and the name of your cluster as shell variables so you can use them in bash commands.

REGION=us-west-2
CLUSTERNAME=test-cluster

2. Create a secret and note down the <SECRETARN>

aws --region "$REGION" secretsmanager  create-secret --name test-secret --secret-string '{"username":"rajith", "password":"Q7VRTDyM4qRZ"}'

3. Create a resource policy for the pod that limits its access to the secret you created in the previous step. For <SECRETARN>, use the ARN of the secret. Save the policy ARN in a shell variable.

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name eks-deployment-policy --policy-document '{     "Version": "2012-10-17",     "Statement": [ {         "Effect": "Allow",         "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],         "Resource": ["<SECRETARN>"]     } ] }')

4. Create an IAM OIDC provider for the cluster, skip this step if you already have one.

eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once

5. Create the service account that the pod uses and link the resource policy we have created in step 3. Here, I use the service account name eks-deployment-sa for this tutorial.

eksctl create iamserviceaccount --name eks-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

6. Create the SecretProviderClass manifest to specify which secret to mount in the pod.

Example-SecretProviderClass.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: eks-test-secret
spec:
  provider: aws
  secretObjects:
  - secretName: test-secret
    type: Opaque
    data: 
    - objectName: username 
      key: username
    - objectName: password 
      key: password
  parameters:
    objects: |
        - objectName: "test-secret"
          objectType: "secretsmanager"
          jmesPath:
            - path: "username"
              objectAlias: "username"            
            - path: "password"
              objectAlias: "password"

Note: The objectName and objectAlias should be matching.

7. Apply the SecretProviderClass manifest

kubectl apply -f Example-SecretProviderClass.yaml

8. Create a ExampleDeployment.yaml file to deploy a Nginx pod. The secret mount path within the pod is/mnt/secrets-store.

kind: Service
apiVersion: v1
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: eks-deployment-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "eks-test-secret"
      containers:
      - name: nginx-deployment
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
        env:
        - name: username
          valueFrom:
            secretKeyRef:
              name: test-secret 
              key: username
        - name: password
          valueFrom:
            secretKeyRef:
              name: test-secret
              key: password

9. Apply the deployment manifest

kubectl apply -f ExampleDeployment.yaml

10. After applying this manifest, we should be able to get our AWS Secret’s content directly from the env var username and password

kubectl exec -it <NAME OF THE POD> -- env | grep username
kubectl exec -it <NAME OF THE POD> -- env | grep password

Reference

Tutorial: Create and mount a secret in an Amazon EKS pod

Use AWS Secrets Manager secrets in Amazon Elastic Kubernetes Service

Sync as Kubernetes Secret