5.4. Kubernetes, OpenShift & HelmCharts Based Migration

This guide details the steps to update your containers to the latest version if you are running S4DBs on a Kubernetes, OpenShift and HelmCharts based installation.

5.4.1. from v2.0.0 to v2.1.0

The migration from v2.0.0 to v2.1.0 is way less difficult than from v1.8.X to v2.0.0. All container images got an upgrade for improvements and security fixes. So please checkout the latest deployment YML files that can be found here:

Kubernetes: Speedgain Timescale deployment file s4dbs-postgres.yaml

OpenShift: Speedgain Timescale deployment file s4dbs-postgres.yaml

HelmCharts: Speedgain values configuration file values.yaml

Please compare your current deployment files to the new ones and apply the changes. Check the migration section of Docker to get an overview about the changes Docker Based Migration

5.4.2. from v1.8.0/1.8.X to V2.0.0

With Speedgain for Databases Version 2.0.0 we introduce an own TimescaleDB image. The new TimescaleDB image will automatically migrate from PostgreSQL 13 to PostgreSQL 16 and also upgrades the TimescaleDB plugin to the latest version 2.18.2!

Note
 The initial start of the s4dbs_postgres pod will take way longer because of the DBMS upgrade from PostgreSQL 13 to 16.
Database upgrade information

PostgreSQL 13 will be out of maintenance soon. So its necessary to upgrade it. The new Speedgain for Databases TimescaleDB image will do that automatically! For that the pg_upgrade utility will be used internally. To prevent unnecessary data copies or pg_dump operations, the so called "link mode" of pg_upgrade will be used.

The S4DBS TimescaleDB image will reuse and migrate the new data files on the existing storage volume. To use "link mode" its an requirement to use the same filesystem, so don’t change it! After the successful upgrade to PostgreSQL 16 there will be a new directory inside of the s4dbs_postgres container called "/var/lib/postgresql/data/16" which is the new database directory.

Note
 The old database directory inside the s4dbs_postgres container and its data under "/var/lib/postgresql/data" will remain in its place! Do not delete the old database files and don’t move it! pg_upgrade will create links between the old and new data files to prevent unnecessary file copy or correction operations. Another advantage is that the old database directory can be used to start again PostgreSQL 13 even after the run of pg_upgrade.
Special Switches & Configurations

Under certain conditions your PDB is eventually created with special parameters or configurations. For example these changes could be:

  • custom WAL segement size

  • custom Min WAL size

  • custom Max WAL size

  • etc.

Under these conditions the default S4DBs migration to V2.0.0 will run into errors. To address these issues its possible to override the defaults and use your individual parameters and configurations. This mechanism works via two environment variables that are passed to the migration process or the startup of the PostgreSQL server inside of the container. These two environment variables are:

  • POSTGRES_INITDB_ARGS

  • POSTGRES_OPTIONS

The following is a detailed explanation of each variable and their use cases.

1. POSTGRES_INITDB_ARGS

This variable derives from the official PostgreSQL Container image environment variable (see: https://hub.docker.com/_/postgres#postgres_initdb_args). The variable will be passed to the initial creation of the new data directory under PostgreSQL 16 were the new PDB will be serve and is target of the internal migration.

Thus this variable affects actions the init-container phase of the deplyoment.

Note
 This variable must be set before you start the s4dbs_postgres container with the new s4dbs_timescaledb:2.0.0 image the first time! For instance if the WAL segment size was changed in your V1.8.X PDB and you omit POSTGRES_INITDB_ARGS incl. configuration of the corresponding "wal-segsize" parameter, the migration will fail. In that case, you have to manually cleanup your new data directory under the path of environment variable NEW_DATA_DIR! Take care to not delete the old/source data directory (OLD_DATA_DIR). By default NEW_DATA_DIR is set to /var/lib/postgresql/data/16. You need to delete only the folder "16" in that case.

You only have to define the variable with the corresponding values. Please include the double hyphen before each initDB parameter and seperate them by a space. For example:

Kubernetes/OpenShift - s4dbs_postgres.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    ...
    spec:
      ...
      initContainers:
      - name: s4dbs-postgres-migration
        ...
        env:
        - ...
        - name: INIT_CONTAINER
          value: "true"
        - name: POSTGRES_INITDB_ARGS
          value: "--wal-segsize=256"
        ...
      containers:
      - name: s4dbs-postgres
        ...
        env:
        - ...
        - name: POSTGRES_INITDB_ARGS
          value: "--wal-segsize=256"
    ...

Helm - values.yaml Because of CI/CD or GitOps systems like ArgoCD, there is the option to override certain environment variables from the HelmChart values.yaml. If you don’t have the option to override the POSTGRES_INITDB_ARGS variable from the values.yaml, you have to change it. You will find the POSTGRES_INITDB_ARGS in the s4dbsPostgresDeployment section:

# s4dbsPostgresDeployment defines timescale image and varas
s4dbsPostgresDeployment:
  image: docker.io/...
  ...
  postgres_initdb_args: "--wal-segsize=256"    # set for example to "--wal-segsize=XYZ" if you changed WAL segment size in your S4DBs V1.8.2 PDB - important for migration to V2.0.0
  ...
2. POSTGRES_OPTIONS

With this variable you can override configuration parameters that are stored by default in the postgresql.conf file. POSTGRES_OPTIONS is an alternative way to override those parameters. The common way to do this is to set a new command within your Container configuration, for example:

Kubernetes/OpenShift - s4dbs_postgres.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
      - name: s4dbs-postgres
        ...
        command: ["postgres"]
        args: ["-c max_connections=200", "-c max_wal_size=1GB", "-c max_val_size=5GB", "-c max_locks_per_transaction=256"]
    ...

Thus this variable affects actions of the container phase of the deployment.

The following is the same example but using POSTGRES_OPTIONS. Please note that you can omit the "-c" before each parameter and seperate them by a single space.

Kubernetes/OpenShift - s4dbs_postgres.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
      - name: s4dbs-postgres
        ...
        env:
        - ...
        - name: POSTGRES_OPTIONS
          value: "max_connections=200 max_wal_size=1GB max_val_size=5GB max_locks_per_transaction=256"
    ...

Helm - values.yaml As explained above you can override environment variables using CI/CD or GitOps systems like ArgoCD or you have the option to override the POSTGRES_OPTIONS variable from the values.yaml. You will find the POSTGRES_OPTIONS in the s4dbsPostgresDeployment section:

# s4dbsPostgresDeployment defines timescale image and varas
s4dbsPostgresDeployment:
  image: docker.io/...
  ...
  postgres_options: "max_connections=200 max_wal_size=1GB max_val_size=5GB max_locks_per_transaction=256"        # override with your custom configuration parameters
  ...
1. Update container image and tag in spec YML files

You can either download the latest YML files or update the PostgreSQL image and tag in your existing files.

You can find the latest YML files here:

Kubernetes: Speedgain Timescale deployment file s4dbs-postgres.yaml

OpenShift: Speedgain Timescale deployment file s4dbs-postgres.yaml

HelmCharts: Speedgain values configuration file values.yaml

If you want to update your existing specification files, proceed with step 2. change PostgreSQL image and tag in spec files

2. change PostgreSQL image and tag in spec files

Change the image tags in Helms value.yaml from

    ...
    # s4dbsPostgresDeployment defines timescale image and varas
    s4dbsPostgresDeployment:
      image: docker.io/timescale/timescaledb:2.11.2-pg13
    ...

to

    ...
    # s4dbsPostgresDeployment defines timescale image and varas
    s4dbsPostgresDeployment:
      image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...

In case of or Kubernetes/OpenShifts, change the s4dbs-postgres.yaml from

    ...
    spec:
      volumes:
        ...
      containers:
      - name: s4dbs-postgres
        image: docker.io/timescale/timescaledb:2.11.2-pg13
        ...

to

    ...
    spec:
      volumes:
        ...
      containers:
      - name: s4dbs-postgres
        image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
        ...
3. add initContainer block in YAML spec file for migration

Because we use Readiness, Liveness and Statup Probes we have to introduce an initContainer to do the PostgreSQL release migration. We can’t predict exactly how long the migration process will take and its likely that the Probes will cause the PostgreSQL pod to be killed and restarted in the middle of the upgrade process. The initContainer uses the same container image but with a additional environment variable (INIT_CONTAINER="true") so only the migration will be done.

For Kubernetes and OpenShift edit the s4dbs-postgres.yaml file from:

...
template:
    metadata:
      labels:
        app: s4dbs-postgres
    spec:
      volumes:
      - name: postgres-pv-storage
        persistentVolumeClaim:
          claimName: s4dbs-postgres-pv-claim
      containers:
      - name: s4dbs-postgres
        image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
        ports:
        - containerPort: 5432
        imagePullPolicy: Always
        env:
        - name: POSTGRES_PASSWORD
        ...

Add the new initContainers block after the volumes block and before the containers block. Please make sure the new block has the same indention as the volumes and containers block!

...
    spec:
      volumes:
        ...

      initContainers:
      - name: s4dbs-postgres-migration
        image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
        imagePullPolicy: Always
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: s4dbs-secrets
              key: postgres-password
        - name: POSTGRES_DB
          value: speedgain
        - name: OLD_DATA_DIR
          value: /var/lib/postgresql/data/pgdata
        - name: NEW_DATA_DIR
          value: /var/lib/postgresql/data/pgdata/16
        - name: INIT_CONTAINER
          value: "true"
        resources:
          requests:
            memory: "1Gi"
            cpu: "1"
          limits:
            memory: "16Gi"
            cpu: "8"
        volumeMounts:
        - name: postgres-pv-storage
          mountPath: /var/lib/postgresql/data/

      containers:
        ...

For HelmCharts the changes are similar. You have to edit the templates/s4dbs-postgres.yaml file. Add the following initContainers block after the volumes and before the containers block:

...
    spec:
      volumes:
      - name: postgres-pv-storage
        persistentVolumeClaim:
          claimName: {{ .Values.s4dbsPostgresDeployment.persistentVolumeClaimName }}

      initContainers:
      - name: s4dbs-postgres-migration
        image: {{ .Values.s4dbsPostgresDeployment.image }}
        imagePullPolicy: Always
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: s4dbs-secrets
              key: postgres-password
        - name: POSTGRES_DB
          value: speedgain
        - name: OLD_DATA_DIR
          value: /var/lib/postgresql/data/pgdata
        - name: NEW_DATA_DIR
          value: /var/lib/postgresql/data/pgdata/16
        - name: INIT_CONTAINER
          value: "true"
        resources:
          requests:
            memory: {{ .Values.s4dbsPostgresDeployment.requests_memory | quote }}
            cpu: {{ .Values.s4dbsPostgresDeployment.requests_cpu | quote }}
          limits:
            memory: {{ .Values.s4dbsPostgresDeployment.limits_memory | quote }}
            cpu: {{ .Values.s4dbsPostgresDeployment.limits_cpu | quote }}
        volumeMounts:
        - name: postgres-pv-storage
          mountPath: /var/lib/postgresql/data/

      containers:
        ...
4. Setting up environment variable for migration

IMPORTANT Because the upgrade process uses pg_upgrade in "link mode" the new and old data directory have to be on the same filesystem. Previous versions of the s4dbs-postgres.yaml specification file had to setup the PGDATA environment variable to eventually address file access issues. Because PGDATA was set, its neccessary to specify a new environment variables NEW_DATA_DIR instead of PGDATA.

Note
 Because the migration is done by the initContainer the variable OLD_DATA_DIR is not needed to run the final container.

Modify the s4dbs-postgres.yaml from:

      containers:
      - name: s4dbs-postgres
        image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
        ...
        env:
        ...
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
      ...

to

      containers:
      - name: s4dbs-postgres
        image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
        ...
        env:
        ...
        - name: NEW_DATA_DIR
          value: /var/lib/postgresql/data/pgdata/16
      ...
5. OPTIONAL: PostgreSQL Configuration Files
Note
 This step can be ignored if no changes to the PostgreSQL configuration files (postgresql.conf, pg_hba.conf or pg_ident.conf) were done!

The automated database migration uses the PostgreSQL utility pg_upgrade. This utility converts if necessary the binary structure of all data files so the PostgreSQL 13 data files are compatable to PostgreSQL 16. How ever the utility doesn’t handle configuration files! Later PostgreSQL release have for instance new configuration parameters or changed values to adress a different behavior. An automatic merge of the old and new configuration files can result in an unwanted or even faulty behavior of PostgreSQL. Therefore manual intervention is requried!

If you changed something in your postgresql.conf, pg_hba.conf or pg_ident.conf file, you have to move these changes from the old configuration file in the new file! The following table lists the locations of these configuration files. The default location can variy and is configurable via an environment variable. The following table only lists the default paths.

Environement variable Docker & Podman - default path OpenShift, Helm & Kubernetes - default path Configuration file name

NEW_DATA_DIR

/var/lib/postgresql/data/16

/var/lib/postgresql/data/pgdata/16

postgresql.conf
pg_hba.conf
pg_ident.conf

OLD_DATA_DIR

/var/lib/postgresql/data

/var/lib/postgresql/data/pgdata

If you have to move a lot of changes from the old configuration file to the new one, it is maybe easier to extract the config files form the pod and do the merging in your favorite merge editor (for example, delta, WinWerge, VS-Code, meld, kdiff3, etc.). Depending if you used file mounts, ConfigMaps or neigther to manage the configuration files, you have a couple of options to make a copy of the configuration file. If you don’t use file mounts or ConfigMaps you could extract the configuration files as follows:

kubectl cp <some-namespace>/<s4dbs-postgres-pod>:/<new-path-see-above->/<config-file-see-above> <new-local-copy-of-config-file>
kubectl cp <some-namespace>/<s4dbs-postgres-pod>:/<old-path-see-above->/<config-file-see-above> <old-local-copy-of-config-file>

Now do your merges with your favorite merge utility. Is this step completed you have to copy the merged configuration files back into the s4dbs_postgres pod. This can be done with the following command:

kubectl cp <merged-local-copy-of-config-file> <some-namespace>/<s4dbs-postgres-pod>:/<new-path-see-above->/<config-file-see-above>

If you use ConfigMaps or file mounts the configuration files are already accessable from outside the pod. After starting the s4dbs_postgres pod the first time you can extract the newer version of the configuration file as shown above. Then merge the changes and save them into the existing file mount or ConfigMap. Keep in mind that you need a different target mount point (see default paths table above) so PostgreSQL 16 uses the config file.

You have to restart the s4dbs_postgres pod so the changes are applied. Because you used a deployment to setup the PostgreSQL pod it will be restarted after it’s deletion.

kubectl delete pod <some-namespace>/<s4dbs-postgres-pod>

← Previous: Docker Based Migration | ↑ Up: HowTo | ⌂ Home: Speedgain for Databases | Next: Create a Custom Dashboard →