Centralized Config Management

Intro

The config SDK, dynamic config, and config polices can be used to create a centralized config management system.

Prerequisites

A project with a CircleCI config.yml
A CircleCI scale plan (only needed for config policies)

Advantages

Locks down who can manipulate CircleCI configs
Allows for automated distribution of config updates
Can use CircleCI to test and deploy config changes

Creating the centralized config repository

Create a repository for your config SDK module(s). Install the config SDK and create a Javascript/Typescript module.

The module can be as dynamic as you’d like, generating configs for multiple environments and/or projects. To start, it may be beneficial to translate current config files into individual JS/TS modules to take advantage of centralized configuration sooner.

Once you are happy with the module, publish the package via NPM.

The following is an example of a static config that has been converted to using the config SDK:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


const CircleCI = require("@circleci/circleci-config-sdk");

module.exports.basicConfig = function () {

  var config = new CircleCI.Config() 

  // Define base executor
  var dockerBase = new CircleCI.executors.DockerExecutor(
    "cimg/base:2023.02",
    "small"
  );

  // Define hello job
  var helloJob = new CircleCI.Job("hello", dockerBase);
  helloJob.addStep(new CircleCI.commands.Run({command: "echo hello john"}));
  config.addJob(helloJob);

  // Define workflow
  var helloWorkflow = new CircleCI.Workflow("hello-workflow");
  helloWorkflow.addJob(helloJob);

  // Add workflow to config
  config.addWorkflow(helloWorkflow);
  
  // Generate YAML config
  config.writeFile('basic_workflow.yml');

  return config;

};

Which produces the following basic_workflow.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# This configuration has been automatically generated by the CircleCI Config SDK.
# For more information, see https://github.com/CircleCI-Public/circleci-config-sdk-ts
# SDK Version: 0.0.0-development

version: 2.1
setup: false
jobs:
  hello:
    docker:
      - image: cimg/base:2023.02
    resource_class: small
    steps:
      - run:
          command: echo hello john
workflows:
  hello-workflow:
    jobs:
      - hello

Using the config SDK

In the working project the config came from, dynamic config will need to be enabled. In CircleCI navigate to Project Settings > Advanced and, at the bottom of the page, toggle on Enable dynamic config using setup workflows.

Within the .circleci directory, create a subdirectory that will contain a script that uses the module created previously:

├── .circleci
│   ├── setup.yml
├── └── generated
│       ├── generated.index.js
│       ├── package.json
│       └── package-lock.json
├── src

generated.index.js calls the SDK module’s function to generate the config file:

1
2
3
4
5


const configGen = require('./node_modules/[insert_your_dir_structure]/index.js');

var config = configGen.basicConfig();

console.log(config)

setup.yml runs the script and then uses the continuation orb to trigger a continued workflow that uses the generated config file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


version: 2.1

setup: true

orbs:
  continuation: circleci/continuation@0.2.0

jobs:
  setup:
    docker: 
      - image: cimg/node:19.7.0
    resource_class: small
    steps:
      - checkout
      - run: node .circleci/generated/generated.index.js
      - continuation/continue:
          configuration_path: basic_workflow.yml

workflows:
  setup-workflow:
    jobs:
      - setup

Implementing a config policy

The SDK module can only be manipulated by those with write access to the centralized config repo, but setup.yml can still be changed by anyone with write access to the working project. Config policies can be used to lock down setup.yml, preventing any changes or additions.

The following policy ensures that the working project’s setup config cannot be changed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52


package org

policy_name["project_lock"]

# A locks the policy to a specific project via UUID
# UUID can be found in the CircleCI UI under Project Settings > Overview
# use care to avoid naming collisions as assignments are global across the entire policy bundle
project_id := "your-project-UUID"

# enables the rule only for the setup workflow
enable_hard["check_project_config"] { input.setup == true }

check_project_config[reason] {
  # Feed full setup.yml contents into expected
  expected := yaml.unmarshal(`
  version: 2.1

  setup: true

  orbs:
    continuation: circleci/continuation@0.2.0

  jobs:
    setup:
      docker: 
        - image: cimg/node:19.7.0
      resource_class: small
      steps:
        - checkout
        - run: node .circleci/generated/generated.index.js
        - continuation/continue:
            configuration_path: basic_workflow.yml

  workflows:
    setup-workflow:
      jobs:
        - setup
  `)

    # the _compiled_ key exists along side the other keys of the uncompiled config
    # so, the data will never match.
    # we can remove the compiled value at which point, we should be comparing the
    # unmarshaled expected value against the input map.
    uncompiled_input := object.remove(
        input,
        {"_compiled_"}, # remove the compiled key from the input
    )

  # uncompiled_input is the original input, with the compiled key removed
  uncompiled_input != expected
  # Prints the expected config in the UI to assist with correctig the policy violation
  reason := sprintf("config.yml cannot be changed\n\nFound:\n%s\n\nExpected:\n%s", [uncompiled_input, expected])

The following policy test can be used for the above policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


test_check_project_config:
  input:
    version: 2.1

    setup: true

    orbs:
      continuation: circleci/continuation@0.2.0

    jobs:
      setup:
        docker: 
          - image: cimg/node:19.7.0
        resource_class: small
        steps:
          - checkout
          - run: node .circleci/generated/generated.index.js
          - continuation/continue:
              configuration_path: basic_workflow.yml

    workflows:
      setup-workflow:
        jobs:
          - setup
    
  decision:
    status: PASS
    enabled_rules: 
      - check_project_config

Next steps

This is an implementation of a centralized config management system that can allow you to lock down a config quickly, but it’s not recommended to stop there. The following are some next steps that can be taken to improve the management system.

Setup a test and publish pipeline for central config repo
Incorporate setup job in an orb to simplify project configs
Create a dynamic SDK script to account for multiple environments or projects
Use a tool like dependabot to allow for automatic distribution of config updates
Contribute to the config SDK
Use another config storage system to pull configs into a working project’s setup workflow