Data Retention in Server (versions 4.3 and above)

Background

In Server versions prior to 4.3, customers were able to set lifecycle policies for their object storage S3 buckets. However, there were some limitations where customers were unable to clean up the action-logs/ and circle-workflows-configuration/ directories because there was no supported method to clean up data in Postgres and Mongo.

Starting from version 4.3, customers can now set up retention policies for both Mongo and Postgres to clean up data older than n days. The following sections outline the step-by-step process for implementing these retention policies in a Server 4.3+ environment.

Step 1: Setting a Retention Period for Postgres

Connect to a REPL session in one of the frontend pods by running the following commands:
```
kubectl exec -it <frontend-xxx> -- /bin/bash
```
Then, connect to the REPL:
```
lein repl :connect 6005
```

Once connected, the current setting can be verified using the following command:

(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-retention-period)

The retention period can be set as needed (the example below sets it to 90 days):

(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-retention-period 90)

The deletion interval can be verified by running:
```
(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-interval)
```
By default, the interval is set to 0. This value must be updated to a number greater than 0 for WFC deletion to run every n seconds. For example, the following command sets it to 60 seconds:
```
(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-interval 60)
```
In instances with significant data volumes, additional workflows_conductor_event_consumer replicas may be required to ensure deletion progresses smoothly until it aligns with the configured retention period.
The WFC event consumer pod logs can be checked to verify that deletion is progressing without errors.

The oldest created_at date for a job can be verified to ensure alignment with the retention period using the following command:

kubectl exec postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U "postgres" -d "conductor_production" -c "SELECT * FROM public.jobs ORDER BY created_at ASC LIMIT 2;"'

Step 2: Setting a Retention Period for MongoDB (Action Logs)

Retention limits for action logs can be configured in the same REPL session using the following commands:

(circle.http.api.admin-commands/set-setting :delete-old-builds.retention-limit-days 180)
(circle.http.api.admin-commands/set-setting :delete-old-action-logs.enabled true)

Step 3: Set Up Lifecycle Policies for the S3 Bucket

The following paths should be set to n+1 days, where n represents the retention limit configured for Mongo and Postgres:

action-logs/
artifacts/
circleci-workflows-configuration/
jobs/

Note: Setting a lifecycle policy for the entire object storage bucket is possible, but it may result in the unintended deletion of critical items, such as audit logs, required for future analysis.