Connect CircleCI with Vault using OIDC

Goal: Use OIDC to authenticate to Vault, retreiving secrets or tokens specific to project, leaving 0 secrets in CircleCI.

• Runnning Vault
• Running CCI Org
• Vault secret(s) to access exist in Vault
• Vault Policy with read permission for secret(s) (we will create a role)

Although vault is not ready, starting with how we want to consume the vault interaction from CircleCI is a useful way to set our direction.

Our recommended approach is use of Vault Agent to abstract API interactions. However you may map the same concepts to CLI/API calls.

• Create CircleCI contexts to generate unique OIDC tokens per environment
• Configure config.yml to read secrets from vault with token
• Workload runs as normal (push, deploy, etc)

Each context get’s a unique claim with that context_id. This allows us to distinquish Production from non-Production requests. We recommend 1 context per “environment” usually 2-4 per org. i.e. oidc-prod and oidc-dev.

We can look at Project_ID directly in the claims, so do not need to reflect that in multiple contexts across the org for each project.

Naming your production context with prod in name does not itself restrict access, but allows us to use CircleCIs Config Policy to limit which projects, branches, or conditions get access.

From our pipeline we will need to install the Vault CLI, and point it to our vault URL. We will then authenticate and pull and needed secrets. The relevant portion is highlighed and described below.

• Set VAULT_ADDR to your private vault URL (line 78)
• Authenicate using configured bindings (below) and save the responding Vault VAULT_TOKEN (line 79)
• Read a secret path using the stored token (lines 80 & 82)
• Use jq to convert JSON secret payload to exported Environment Variables (lines 81 & 83)
  (this is optional and you may use alternate output from vault as it meets each use case)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

version: 2.1
          
executors:
  base:
    docker:
      - image: cimg/deploy:2022.08

jobs:
  # Sample "Build and Push" job is not concerned with environments, only 1 Docker repo is exposed in Vault.
  docker-push-via-vault:
      executor: base
      steps:
        - checkout
        - load-credentials    
        - setup_remote_docker:
            docker_layer_caching: true
        - run: |
                        echo "${NEXUS_PASSWORD}" | docker login --username ${NEXUS_USERNAME} --password-stdin <YOUR_REPO_URL>
        

  k8s-via-vault:
    executor: base
    parameters:
      app-environment:
        type: string
        default: dev
        description: Environment suffix used by namesapce and SA account name.
    environment:
      APP_ENV: <<parameters.app-environment>>
    steps:
      - checkout
      # Reusable command to encapsulate Vault Interaction
      - load-credentials
      - run: echo "Now using ${K8S_USER}@${K8S_NAMESPACE}"
      - run: |
          # This method uses a previuosly defined ServiceAccount's secret (token and cert) to create the necessary kubeconfig to talk to a cluster.
          echo ${K8S_CERT} | base64 -d > ca.crt
          kubectl config set-cluster ${K8S_CLUSTER} --server=${K8S_URL} --certificate-authority=ca.crt
          export DECODED_TOKEN=$(echo ${K8S_TOKEN} | base64 -d) #kubectl prints an encoded value, MUST decode it to work.
          kubectl config set-credentials ${K8S_USER} --token=${DECODED_TOKEN}
          kubectl config set-context default --user=${K8S_USER}  --cluster=${K8S_CLUSTER} --namespace ${K8S_NAMESPACE}
          # Run apply, skaffold, helm, etc..
          kubectl get serviceaccounts -n ${K8S_NAMESPACE}          
   
workflows:
  main:
    jobs:
      - docker-push-via-vault:
          context: cera-vault-oidc
      - k8s-via-vault:
          name: Deploy Dev
          requires: [ docker-push-via-vault ]
          context: [ oidc-dev ] # nonprod context
      - k8s-via-vault:
          name: Deploy Production
          app-environment: prod #override which secret paths to use
          requires: [ Deploy Dev ]
          context: [ oidc-prod ]  # change context for prod context


commands:
  load-credentials:
    steps:
      - run:
          name: install vault agent (if not present)
          command: |
            vault -h && exit 0 || echo "Installing vault"
            #only runs if vault command above failed
            cd /tmp
            wget https://releases.hashicorp.com/vault/1.12.2/vault_1.12.2_linux_amd64.zip
            unzip vault_1.12.2_linux_amd64.zip
            sudo mv vault /usr/local/bin               
      - run:
          name: Load Credentials from Vault
          command: |
            ROLE=${APP_ENV:-"dev"} #fallback to dev when empty
            echo "Environment (APP_ENV ): $APP_ENV"
            export VAULT_ADDR="https://vault.cera.circleci-fieldeng.com"
            export VAULT_TOKEN=`vault write -field=token auth/jwt/login role=boa-$ROLE-deploy jwt=$CIRCLE_OIDC_TOKEN`
            vault read -format=json secret/data/cluster/boa-pipeline-$ROLE > /tmp/cluster-secret.json
            jq -r '.data.data | to_entries[] | "export K8S_"+(.key | ascii_upcase)+"="+(.value | @sh) ' /tmp/cluster-secret.json >> $BASH_ENV
            vault read -format=json secret/data/nexus/boa-deployer > /tmp/nexus-secret.json
            jq -r '.data.data | to_entries[] | "export NEXUS_"+(.key | ascii_upcase)+"="+(.value | @sh) ' /tmp/nexus-secret.json >> $BASH_ENV            

          

# VS Code Extension Version: 1.1.1

Make sure to give each job the appropriate context!

We need a few key ingredients in Vault:

• A JWT auth endpoint configured to trust CircleCI tokens
• Policies that describe access based on project & environment
• JWT/Auth claims that alllow OIDC tokens to bind to the correct role

This step is real easy. Just add a JWT Auth Method (once per org/vault combo) that is supplied with CircleCIs OIDC dicovery URL, unique to each org. You only need to repeat if you have multiple orgs, or accesing multiple vault instances.

• Add JWT auth to Vault boundIssuer and discoveryUrl should be your org like https://oidc.circleci.com/org/<ORG ID FROM ORG SETTINGS>

Access policies can be configured in UI or API. We’re using 2 policies for prod and nonprod. Unlike contexts, you may want these to be app specific, and we’ve indicated so in the name.

path "secret/*" {
  capabilities = ["list"]
}
path "secret/data/cluster/appname-pipeline-prod" {
  capabilities = ["list","read"]
}
path "secret/data/nexus/*" {
  capabilities = ["list","read"]
}
path "secret/metadata/cluster/appname-pipeline-prod" {
  capabilities = ["list","read"]
}
path "secret/metadata/nexus/*" {
  capabilities = ["list","read"]
}

path "secret/*" {
  capabilities = ["list"]
}
path "secret/data/cluster/appname-pipeline-dev" {
  capabilities = ["list","read"]
}
path "secret/data/nexus/*" {
  capabilities = ["list","read"]
}
path "secret/metadata/cluster/appname-pipeline-dev" {
  capabilities = ["list","read"]
}
path "secret/metadata/nexus/*" {
  capabilities = ["list","read"]
}

How do we link an OIDC token to the policies above? You will need API or CLI access to create auth bindings like the following.

To keep example simple we’re only doing 2, 1 for prod * non-prod. But you can see how the project-id claim can be changed to create more!

Rather than creating a role valid to any project authenticating with OIDC, we can restrict the Vault JWT Role to assign policies based on various claims.

The token from CircleCI contains an evolving list of claims that can be used to limit which contexts or pojects get access. This means specific projects can be limited to specific Vault secrets based on claims.

• Project ID
• Context ID (can repressent environment when paired with CPM)

You can use a tool like jwt.io to see data in your CIRCLE_OIDC_TOKEN token and set bound_claims accordingly.

vault write auth/jwt/role/boa-prod-deploy -<<EOF
{
  "role_type": "jwt",
  "user_claim": "sub",
  "user_claim_json_pointer": "true",
  "bound_claims": {
    "oidc.circleci.com/context-ids": ["abcdef-1234567890"],
    "oidc.circleci.com/project-id" : "abcdef-1234567890"
  },
  "policies": ["appname-prod-deploy"],
  "ttl": "10m"
}
EOF

NOTE: We use the same project ID, but an alternate Context ID to assign non-production credentials.

vault write auth/jwt/role/boa-dev-deploy -<<EOF
{
  "role_type": "jwt",
  "user_claim": "sub",
  "user_claim_json_pointer": "true",
  "bound_claims": {
    "oidc.circleci.com/context-ids": ["uvwxyz-1234567890"],
    "oidc.circleci.com/project-id" : "abcdef-1234567890"
  },
  "policies": ["appname-dev-deploy"],
  "ttl": "10m"
}
EOF

The `policies` above should match your existing access policy granting read rights to secrets needed.

The sequence diagram below shows how the flow should now look.

sequenceDiagram

    rect rgb(191, 223, 255)
    CircleCI Platform ->> CircleCI Job: start_job(OIDC_TOKEN)<br/>*dev context
    CircleCI Job->>+Vault: auth/jwt(OIDC_TOKEN)
    note right of Vault:    Vault will consider which<br/>role to claim based<br/> on the job's context!
    Vault-->>CircleCI Job: <VAULT_TOKEN>
    CircleCI Job->>Vault: `vault read /secret/dev...`
    Vault-->>-CircleCI Job: <secret>
    CircleCI Job->>DevSystem: doStuff(secrets)
    CircleCI Job-xVault: `vault read /secret/prod...`
    note right of Vault:  Only paths authorized <br/> by role are allowed
    Vault-->>CircleCI Job: 403
    end
    
    rect rgb(230, 100, 100)
    CircleCI Platform ->> CircleCI Job: start_job(OIDC_TOKEN)<br/>*prod context
    CircleCI Job->>+Vault: auth/jwt(OIDC_TOKEN)
    Vault-->>CircleCI Job: <VAULT_TOKEN>
    CircleCI Job->>Vault: `vault read /secret/prod...`
    Vault-->>-CircleCI Job: <secret>
    CircleCI Job->>ProdSystem: doStuff(secrets)
    end