Goal: Use OIDC to authenticate to Vault, retreiving secrets or tokens specific to project, leaving 0 secrets in CircleCI.
Prereqs
Runnning Vault
Running CCI Org
Vault secret(s) to access exist in Vault
Vault Policy with read permission for secret(s) (we will create a role)
CircleCI Setup
Although vault is not ready, starting with how we want to consume the vault interaction from CircleCI is a useful way to set our direction.
Our recommended approach is use of Vault Agent to abstract API interactions. However you may map the same concepts to CLI/API calls.
Create CircleCI contexts to generate unique OIDC tokens per environment
Configure config.yml to read secrets from vault with token
Workload runs as normal (push, deploy, etc)
CircleCI: Context Creation (once per “Environment”)
Each context get’s a unique claim with that context_id. This allows us to distinquish Production from non-Production requests. We recommend 1 context per “environment” usually 2-4 per org. i.e. oidc-prod and oidc-dev.
We can look at Project_ID directly in the claims, so do not need to reflect that in multiple contexts across the org for each project.
Naming your production context with prod in name does not itself restrict access, but allows us to use CircleCIs Config Policy to limit which projects, branches, or conditions get access.
The Config
From our pipeline we will need to install the Vault CLI, and point it to our vault URL. We will then authenticate and pull and needed secrets. The relevant portion is highlighed and described below.
Set VAULT_ADDR to your private vault URL (line 78)
Authenicate using configured bindings (below) and save the responding Vault VAULT_TOKEN (line 79)
Read a secret path using the stored token (lines 80 & 82)
version:2.1executors:base:docker:- image:cimg/deploy:2022.08jobs:# Sample "Build and Push" job is not concerned with environments, only 1 Docker repo is exposed in Vault.docker-push-via-vault:executor:basesteps:- checkout- load-credentials - setup_remote_docker:docker_layer_caching:true- run:|echo "${NEXUS_PASSWORD}" | docker login --username ${NEXUS_USERNAME} --password-stdin <YOUR_REPO_URL>k8s-via-vault:executor:baseparameters:app-environment:type:stringdefault:devdescription:Environment suffix used by namesapce and SA account name.environment:APP_ENV:<<parameters.app-environment>>steps:- checkout# Reusable command to encapsulate Vault Interaction- load-credentials- run:echo "Now using ${K8S_USER}@${K8S_NAMESPACE}"- run:| # This method uses a previuosly defined ServiceAccount's secret (token and cert) to create the necessary kubeconfig to talk to a cluster.
echo ${K8S_CERT} | base64 -d > ca.crt
kubectl config set-cluster ${K8S_CLUSTER} --server=${K8S_URL} --certificate-authority=ca.crt
export DECODED_TOKEN=$(echo ${K8S_TOKEN} | base64 -d) #kubectl prints an encoded value, MUST decode it to work.
kubectl config set-credentials ${K8S_USER} --token=${DECODED_TOKEN}
kubectl config set-context default --user=${K8S_USER} --cluster=${K8S_CLUSTER} --namespace ${K8S_NAMESPACE}
# Run apply, skaffold, helm, etc..
kubectl get serviceaccounts -n ${K8S_NAMESPACE}workflows:main:jobs:- docker-push-via-vault:context:cera-vault-oidc- k8s-via-vault:name:Deploy Devrequires:[docker-push-via-vault ]context:[oidc-dev ]# nonprod context- k8s-via-vault:name:Deploy Productionapp-environment:prod#override which secret paths to userequires:[Deploy Dev ]context:[oidc-prod ] # change context for prod contextcommands:load-credentials:steps:- run:name:install vault agent (if not present)command:| vault -h && exit 0 || echo "Installing vault"
#only runs if vault command above failed
cd /tmp
wget https://releases.hashicorp.com/vault/1.12.2/vault_1.12.2_linux_amd64.zip
unzip vault_1.12.2_linux_amd64.zip
sudo mv vault /usr/local/bin - run:name:Load Credentials from Vaultcommand:| ROLE=${APP_ENV:-"dev"} #fallback to dev when empty
echo "Environment (APP_ENV ): $APP_ENV"
export VAULT_ADDR="https://vault.cera.circleci-fieldeng.com"
export VAULT_TOKEN=`vault write -field=token auth/jwt/login role=boa-$ROLE-deploy jwt=$CIRCLE_OIDC_TOKEN`
vault read -format=json secret/data/cluster/boa-pipeline-$ROLE > /tmp/cluster-secret.json
jq -r '.data.data | to_entries[] | "export K8S_"+(.key | ascii_upcase)+"="+(.value | @sh) ' /tmp/cluster-secret.json >> $BASH_ENV
vault read -format=json secret/data/nexus/boa-deployer > /tmp/nexus-secret.json
jq -r '.data.data | to_entries[] | "export NEXUS_"+(.key | ascii_upcase)+"="+(.value | @sh) ' /tmp/nexus-secret.json >> $BASH_ENV# VS Code Extension Version: 1.1.1
Make sure to give each job the appropriate context!
Vault Setup
We need a few key ingredients in Vault:
A JWT auth endpoint configured to trust CircleCI tokens
Policies that describe access based on project & environment
JWT/Auth claims that alllow OIDC tokens to bind to the correct role
JWT Auth
This step is real easy. Just add a JWT Auth Method (once per org/vault combo) that is supplied with CircleCIs OIDC dicovery URL, unique to each org. You only need to repeat if you have multiple orgs, or accesing multiple vault instances.
Add JWT auth to Vault
boundIssuer and discoveryUrl should be your org like https://oidc.circleci.com/org/<ORG ID FROM ORG SETTINGS>
Access Policies
Access policies can be configured in UI or API. We’re using 2 policies for prod and nonprod. Unlike contexts, you may want these to be app specific, and we’ve indicated so in the name.
How do we link an OIDC token to the policies above? You will need API or CLI access to create auth bindings like the following.
To keep example simple we’re only doing 2, 1 for prod * non-prod. But you can see how the project-id claim can be changed to create more!
Rather than creating a role valid to any project authenticating with OIDC, we can restrict the Vault JWT Role to assign policies based on various claims.
The token from CircleCI contains an evolving list of claims that can be used to limit which contexts or pojects get access. This means specific projects can be limited to specific Vault secrets based on claims.
Project ID
Context ID (can repressent environment when paired with CPM)
You can use a tool like jwt.io to see data in your CIRCLE_OIDC_TOKEN token and set bound_claims accordingly.
The `policies` above should match your existing access policy granting read rights to secrets needed.
Putting it together
The sequence diagram below shows how the flow should now look.
CircleCI PlatformCircleCI JobVaultDevSystemProdSystemVault will consider whichrole to claim based on the job's context!Only paths authorized by role are allowedstart_job(OIDC_TOKEN)*dev contextauth/jwt(OIDC_TOKEN)<VAULT_TOKEN>`vault read /secret/dev...`<secret>doStuff(secrets)`vault read /secret/prod...`403start_job(OIDC_TOKEN)*prod contextauth/jwt(OIDC_TOKEN)<VAULT_TOKEN>`vault read /secret/prod...`<secret>doStuff(secrets)CircleCI PlatformCircleCI JobVaultDevSystemProdSystem