Speedgain for Databases

5.3. Docker Based Migration

This guide details the steps to update your containers to the latest version if you are running S4DBs on a docker based installation (docker compose, podman-docker or podman-compose).

5.3.1. from v2.0.0 to v2.1.0

The migration from v2.0.0 to v2.1.0 is way less difficult than from v1.8.X to v2.0.0. All container images got an upgrade for improvements and security fixes. So please take care to upgrade the TimeScaleDB and NginX image too!

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current tag versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_collector:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_service:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:2.0.0
    ...
    image: docker.io/nginx:1.27
    ...

    ...
    image: docker.io/itgainis/s4dbs_timescaledb:2.1.0
    ...
    image: docker.io/itgainis/s4dbs_collector:2.1.0
    ...
    image: docker.io/itgainis/s4dbs_service:2.1.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:2.1.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:2.1.0
    ...
    image: docker.io/nginx:1.29.1-alpine
    ...

3. pull or load the images

docker compose pull

docker load --input s4dbs_2.1.0_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.3.2. from v1.8.0/1.8.X to V2.0.0

With Speedgain for Databases Version 2.0.0 we introduce an own TimescaleDB image. The new TimescaleDB image will automatically migrate from PostgreSQL 13 to PostgreSQL 16 and also upgrades the TimescaleDB plugin to the latest version 2.18.2! The upgrade steps are similar to the upgrade from V1.7.X to V1.8.0 but you have also modify the healtcheck for s4dbs_postgres service besides changing the image.

Note	The initial start of the s4dbs_postgres service will take way longer because of the DBMS upgrade from PostgreSQL 13 to 16.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

Database upgrade information

PostgreSQL 13 will be out of maintenance soon. So its necessary to upgrade it. The new Speedgain for Databases TimescaleDB image will do that automatically! For that the pg_upgrade utility will be used internally. To prevent unnecessary data copies or pg_dump operations, the so called "link mode" of pg_upgrade will be used.

The S4DBS TimescaleDB image will reuse and migrate the new data files on the existing storage volume. To use "link mode" its an requirement to use the same filesystem, so don’t change it! After the successful upgrade to PostgreSQL 16 there will be a new directory inside of the s4dbs_postgres container called "/var/lib/postgresql/data/16" which is the new database directory.

Note

The old database directory inside the s4dbs_postgres container and its data under "/var/lib/postgresql/data" will remain in its place! Do not delete the old database files and don’t move it! pg_upgrade will create links between the old and new data files to prevent unnecessary file copy or correction operations. Another advantage is that the old database directory can be used to start again PostgreSQL 13 even after the run of pg_upgrade.

Special Switches & Configurations

Under certain conditions your PDB is eventually created with special parameters or configurations. For example these changes could be:

custom WAL segement size
custom Min WAL size
custom Max WAL size
etc.

Under these conditions the default S4DBs migration to V2.0.0 will run into errors. To address these issues its possible to override the defaults and use your individual parameters and configurations. This mechanism works via two environment variables that are passed to the migration process or the startup of the PostgreSQL server inside of the container. These two environment variables are:

POSTGRES_INITDB_ARGS
POSTGRES_OPTIONS

The following is a detailed explanation of each variable and their use cases.

1. POSTGRES_INITDB_ARGS

This variable derives from the official PostgreSQL Container image environment variable (see: https://hub.docker.com/_/postgres#postgres_initdb_args). The variable will be passed to the initial creation of the new data directory under PostgreSQL 16 were the new PDB will be serve and is target of the internal migration.

Note

This variable must be set before you start the s4dbs_postgres container with the new s4dbs_timescaledb:2.0.0 image the first time! For instance if the WAL segment size was changed in your V1.8.X PDB and you omit POSTGRES_INITDB_ARGS incl. configuration of the corresponding "wal-segsize" parameter, the migration will fail. In that case, you have to manually cleanup your new data directory under the path of environment variable NEW_DATA_DIR! Take care to not delete the old/source data directory (OLD_DATA_DIR). By default NEW_DATA_DIR is set to /var/lib/postgresql/data/16. You need to delete only the folder "16" in that case.

You only have to define the variable with the corresponding values. Please include the double hyphen before each initDB parameter and seperate them by a space. For example:

Docker-Compose

  s4dbs_postgres:
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    environment:
      ...
      - POSTGRES_INITDB_ARGS="--wal-segsize=256"
    ...

Podman

podman run -d ... -e POSTGRES_INITDB_ARGS="--wal-segsize=256" docker.io/itgainis/s4dbs_timescaledb:2.0.0

2. POSTGRES_OPTIONS

With this variable you can override configuration parameters that are stored by default in the postgresql.conf file. POSTGRES_OPTIONS is an alternative way to override those parameters. The common way to do this is to set a new command within your Container configuration, for example:

Docker-Compose

  s4dbs_postgres:
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    command: postgres -c max_connections=200 -c max_wal_size=1GB -c max_val_size=5GB -c max_locks_per_transaction=256
    ...

Podman

podman run -d ... docker.io/itgainis/s4dbs_timescaledb:2.0.0 postgres -c max_connections=200 -c max_wal_size=1GB -c max_val_size=5GB -c max_locks_per_transaction=256

The following is the same example but using POSTGRES_OPTIONS. Please note that you can omit the "-c" before each parameter.

Docker-Compose

  s4dbs_postgres:
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    environment:
      ...
      - POSTGRES_OPTIONS="max_connections=200 max_wal_size=1GB max_val_size=5GB max_locks_per_transaction=256"
    ...

Podman

podman run -d ... -e POSTGRES_OPTIONS="max_connections=200 max_wal_size=1GB max_val_size=5GB max_locks_per_transaction=256" docker.io/itgainis/s4dbs_timescaledb:2.0.0

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/timescale/timescaledb:2.11.2-pg13
    ...
    image: docker.io/itgainis/s4dbs_collector:1.8.X
    ...
    image: docker.io/itgainis/s4dbs_service:1.8.X
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.8.X
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.8.X
    ...

    ...
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_collector:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_service:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:2.0.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:2.0.0
    ...

Note	NginX image version tag is not changing.

Look for the healthcheck-test configuration in the s4dbs_postgres service and change it from

    ...
    healthcheck:
      test: ["CMD-SHELL", "su postgres -c \"psql -h localhost -c '\\q'\""]
      interval: 10s
    ...

    ...
    healthcheck:
      test: ["CMD-SHELL", "psql -h localhost -d speedgain -c \"select relname from pg_class where relname = 'pg_database'\" | grep -i pg_database"]
      interval: 10s
    ...

3. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. OPTIONAL: If you changed PostgreSQLs data directy by using the PGDATA environment variable

IMPORTANT If you had changed the default PostgreSQL data directory by using the PGDATA environment variable, you have to provide this directory to the upgrade process by specifying it with the OLD_DATA_DIR environment variable. Because the upgrade process uses "link mode" the new and old data directory have to be on the same filesystem. Thus adjustment of NEW_DATA_DIR is also required in the docker-compose.yml file! The following example shows this case where the PGDATA directory was changed.

  s4dbs_postgres:
    image: docker.io/itgainis/s4dbs_timescaledb:2.0.0
    ...
    environment:
      - OLD_DATA_DIR=/path/to/your/postgres/data_dir
      - NEW_DATA_DIR=/path/to/your/postgres/data_dir/16
    volumes:
      - s4dbs_postgres:/path/to/your/postgres/data_dir

5. Automated PostgreSQL upgrade from release 13 to 16 and healthcheck probing time

IMPORTANT This process is fully automated but requires some time. Depending on the amount of monitored objects and your retention settings, the upgrade process can take a very long time.

We recommend to modify the healthcheck config in the docker-compose.yml file to taken it into account and start the other services later after the upgrade is done.

Look for the healthcheck probing configuration in the s4dbs_postgres service and change it from

  ...
  s4dbs_postgres:
    ...
    healthcheck:
      ...
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s
    ...

  ...
  s4dbs_postgres:
    ...
    healthcheck:
      ...
      interval: 20s
      timeout: 20s
      retries: 20
      start_period: 240s
    ...

This tells Docker Compose to don’t probe for the first 4 minutes and then retry 4 times longer to check the service health. When you have a very huge PostgreSQL database filesystem you may have to increase the "retries" count even more.

After the first successful start of the s4dbs_postgres service, you can revert the healthcheck probing timings back to the old values!

Note

If your "retries" setting was to low and your Docker Compose reports the s4dbs_postgres service as unhealthy, please let the service run! Check the task 7. Monitor the automated PostgreSQL upgrade for further instructions to monitor the PostgreSQL upgrade. When the upgrade is successfully finished just run

docker compose stop

then retry task 6. create s4dbs_postgres service for migration again.

6. create s4dbs_postgres service for migration

For migration purpose we want to start only the s4dbs_postgres service first, so the automated upgrade will be executed. We don’t need to start the other services until the migration is done. Otherwise these services will have errors and need to be restarted later anyways. To do that you execute the following command:

docker compose up -d s4dbs_postgres

Check the task 7. Monitor the automated PostgreSQL upgrade how to keep track of the migration process.

7. Monitor the automated PostgreSQL upgrade

As said in task 4. the upgrade is fully automated and after successfully upgrading the s4dbs_postgres service it will start and provide the Speedgain Performance Database to the other services.

However you may want to monitor the upgrade process. This can be done by watching the logs for the s4dbs_postgres service:

docker compose logs -f s4dbs_postgres

As said the DBMS upgrade will take a while! Look in the log for the following messages:

Upgrade TimeScaleDB plugin and then PostgreSQL data!
...
>> Migrate TimeScaleDB plugin in database speedgain
...
Performing Consistency Checks
...
Performing Upgrade
...
Upgrade Complete
...
ALTER EXTENSION
                                              List of installed extensions
    Name     | Version | Schema |                                      Description
-------------+---------+--------+---------------------------------------------------------------------------------------
 timescaledb | 2.18.1  | public | Enables scalable inserts and complex queries for time-series data (Community Edition)
...
Migration done! :-)

PostgreSQL Database directory appears to contain a database; Skipping initialization

<date> <time> UTC [<pid>] LOG:  starting PostgreSQL 16.7 on x86_64-pc-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit
<date> <time> UTC [<pid>] LOG:  listening on IPv4 address "0.0.0.0", port 5432
<date> <time> UTC [<pid>] LOG:  listening on IPv6 address "::", port 5432
<date> <time> UTC [<pid>] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
<date> <time> UTC [<pid>] LOG:  database system was shut down at 2025-03-21 06:50:14 UTC
<date> <time> UTC [<pid>] LOG:  database system is ready to accept connections
<date> <time> UTC [<pid>] LOG:  TimescaleDB background worker launcher connected to shared catalogs
...

When you find all these messages the PostgreSQL upgrade was successful and the other services should be already automatically started! If you get drastically different log messages, please export your s4dbs_postgres service log and contact support@itgain.de for further assistance!

docker compose logs s4dbs_postgres > s4dbs_postgres.log

Note	There are some expected errors in the log that can be ignored. For example

...
ERROR:  TimescaleDB background worker connected to template database, exiting
...
FATAL:  terminating background worker "TimescaleDB Background Worker Scheduler" due to administrator command
...
FATAL:  terminating connection due to administrator command
...
WARNING:  failed to launch job <number> "Compression Policy [<pid>]": out of background workers
...
FATAL:  terminating background worker "Compression Policy [<pid>]" due to administrator command

These error messages are triggered by the planned restarts of the PostgreSQL server and that connections are blocked during the upgrade process.

8. create/re-create all containers

The migration of the database is now successfully finished and you can start the remaining services. Please execute the following command:

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

The Migration to Speedgain V2.0.0 is now complete!

9. OPTIONAL: PostgreSQL Configuration Files

Note	This step can be ignored if no changes to the PostgreSQL configuration files (postgresql.conf, pg_hba.conf or pg_ident.conf) were done!

The automated database migration uses the PostgreSQL utility pg_upgrade. This utility converts if necessary the binary structure of all data files so the PostgreSQL 13 data files are compatable to PostgreSQL 16. How ever the utility doesn’t handle configuration files! Later PostgreSQL release have for instance new configuration parameters or changed values to adress a different behavior. An automatic merge of the old and new configuration files can result in an unwanted or even faulty behavior of PostgreSQL. Therefore manual intervention is requried!

If you changed something in your postgresql.conf, pg_hba.conf or pg_ident.conf file, you have to move these changes from the old configuration file in the new file! The following table lists the locations of these configuration files. The default location can variy and is configurable via an environment variable. The following table only lists the default paths.

Environement variable	Docker & Podman - default path	OpenShift, Helm & Kubernetes - default path	Configuration file name
NEW_DATA_DIR	/var/lib/postgresql/data/16	/var/lib/postgresql/data/pgdata/16	postgresql.conf pg_hba.conf pg_ident.conf
OLD_DATA_DIR	/var/lib/postgresql/data	/var/lib/postgresql/data/pgdata

Environement variable

Docker & Podman - default path

OpenShift, Helm & Kubernetes - default path

Configuration file name

NEW_DATA_DIR

/var/lib/postgresql/data/16

/var/lib/postgresql/data/pgdata/16

postgresql.conf
pg_hba.conf
pg_ident.conf

OLD_DATA_DIR

/var/lib/postgresql/data

/var/lib/postgresql/data/pgdata

If you have to move a lot of changes from the old configuration file to the new one, it is maybe easier to extract the config files form the container and do the merging in your favorite merge editor (for example, delta, WinWerge, VS-Code, meld, kdiff3, etc.). You can make a copy of the configuration file by using the docker command:

docker cp s4dbs_postgres:/<new-path-see-above->/<config-file-see-above> <new-local-copy-of-config-file>
docker cp s4dbs_postgres:/<old-path-see-above->/<config-file-see-above> <old-local-copy-of-config-file>

Now do your merges with your favorite merge utility. Is this step completed you have to copy the merged configuration files back into the s4dbs_postgres container. This can be done with the following command:

docker cp <merged-local-copy-of-config-file> s4dbs_postgres:/<new-path-see-above->/<config-file-see-above>

You have to restart the s4dbs_postgres container so the changes are applied.

docker restart s4dbs_postgres

5.3.3. from v1.7.0/v1.7.1 to v1.8.0

At first when migrating from version 1.7.0 or 1.7.1 to 1.8.0, the image tags in the Docker Compose file have to be changed (except timescale and nginx), new images pulled or loaded, and containers recreated. Step 5 is optional, you can pre-fill a persistent aggregate with more than 7 days of data if you want to.

Note

Database migration may take longer due to the integration and pre-filling of a new persistent aggregate with data from the past 7 days. It is used for the new Db2 Dashboard "SQL Heatmap". See optional step at the end to backfill with more than these 7 days. It will grow to the normal retention setting but won’t backfill automatically.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions & change Grafana repository database

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.7.0
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.8.2
    ...
    image: docker.io/itgainis/s4dbs_service:1.8.2
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.8.2
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.8.2
    ...

Note	Timescale and NginX image version tag is not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5. OPTIONAL: backfill aggregate

Run the following SQL Statements at your Speedgain Repository Database (if you are not sure what you are doing, ask support for help).

Note	Compression will start asynchronously. Look at disk consumption during execution.

-- backfill base aggregate with the last <dd> days - adjust <dd> in SQL text to somethin higher than 7
do '
declare
temprow record;
begin
FOR temprow IN
        SELECT	time_bucket(''1 hour'', snapshot_time) as bucket FROM 	spg_mon_db2.mon_get_database where  snapshot_time < current_timestamp - INTERVAL ''<dd> d'' GROUP by bucket order by 1 desc
    LOOP
        CALL public.refresh_continuous_aggregate(''spg_mon_db2.v_cont_agg_1h_sql_exec_time'', temprow.bucket - interval ''1 hour'', temprow.bucket);
    END LOOP;
end; ';

-- backfill top aggregate with the last <dd> days - adjust <dd> in SQL text to somethin higher than 7
do '
declare
temprow record;
begin
FOR temprow IN
        SELECT	time_bucket(''1 hour'', snapshot_time) as bucket FROM 	spg_mon_db2.mon_get_database where  snapshot_time < current_timestamp - INTERVAL ''<dd> d'' GROUP by bucket order by 1 desc
    LOOP
        CALL public.refresh_continuous_aggregate(''spg_mon_db2.v_cont_agg_1h_sql_exec_time_heat'', temprow.bucket - interval ''1 hour'', temprow.bucket);
    END LOOP;
end; ';

5.3.4. from v1.6.0 to v1.7.0

At first when migrating from version 1.6.0 to 1.7.0, the image tags in the Docker Compose file have to be changed (except timescale), new images pulled or loaded, and containers recreated. IMPORTANT: For helm chart users: backup your postgresql passwort secret to have it in place if it gets removed during upgrade.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.6.0
    ...
    image: docker.io/nginx:1.23
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.7.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.7.0
    ...
    image: docker.io/nginx:1.27
    ...

Note	Timescale image version tag is not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.3.5. from v1.5.0 to v1.6.0

At first when migrating from version 1.5.0 to 1.6.0, the image tags in the Docker Compose file have to be changed (except timescale and nginx), small adjustment on the nginx.conf, new images pulled or loaded, and containers recreated.

Next, you should check any custom boards and alerts for compatibility with the new grafana version 10.2.3. There are breaking changes introduced in grafana v10.

Important

For helm chart users: backup your postgresql passwort secret to have it in place if it gets removed during upgrade.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.5.0
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.6.0
    ...

Note	Timescale and Nginx image version tags are not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. adjust nginx.conf

Adjust the nginx.conf file in your configuration folder. The lines starting with "rewrite" have to be removed. From:

# grafana
    location /grafana/ {
      rewrite  ^/grafana/(.*)  /$1 break;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;

    }

    location /grafana/api/live/ {
      rewrite  ^/grafana/(.*)  /$1 break;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
    }

# grafana
    location /grafana/ {
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;

    }

    location /grafana/api/live/ {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
    }

5. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

6. verfiy any custom boards and alerts are still functional

In Speedgain for Databases v1.6.0 we migrated grafana to version 10.2.3 If you are using custom boards or alarms in your Speedgain for Database installation you will have to check if your boards and alerts are affected by breaking changes. Major changes are:

There is a new Pie Chart visualization. The old pie chart will no longer be rendered. To switch to the new pie chart just edit the existing panel and switch the visualization.
The Graph Panel has been deprecated and should be migrated to the new time series visualization.
The alerting system has been completely reworked. All existing alarms will be automatically migrated to the new engine. Check, if your alarms could be migrated without errors. Adjust the alarm manually if necessary.

For further information on the changes in grafana 10 refer to the official documentation at https://grafana.com/docs/grafana/latest/breaking-changes/breaking-changes-v10-0/

5.3.6. from v1.4.0 to v1.5.0

At first when migrating from version 1.4.0 to 1.5.0, the image tags in the Docker Compose file have to be changed (inlcuding timescale and except nginx), new images pulled or loaded, and containers recreated.

Next, in the container-configuration for grafana remove the variable GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH if you do not use your own default dashboard for speedgain. The path to the speedgain start dashboard has changed and is now set within the container itself.

The last step is to upgrade the timescale extension in postgresql container.

Warning

For helm chart users: we changed the default for grafana database from sqllite to postgresql. If you are mgirating and made changes to grafana you may want to set the parameter GF_DATABASE_TYPE to sqllite instead of postgres to keep your old grafana settings.

Important

For helm chart users: backup your postgresql passwort secret to have it in place if it gets removed during upgrade.

Tip	No need to worry if you lost your postgresql password or secret. It can be easily reset if you have admin access to the pods/containers. Ask our support for help.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/timescale/timescaledb:2.5.1-pg13
    ...
    image: docker.io/itgainis/s4dbs_collector:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.4.0
    ...

    ...
    image: docker.io/timescale/timescaledb:2.11.2-pg13
    ...
    image: docker.io/itgainis/s4dbs_collector:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.5.0
    ...

Note	Nginx image version tags are not changing.

3. remove grafana default board variable from docker-compose

In your docker-compose.yml at the grafana container remove the row for the variable GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH.

4. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

5. update timescale extension in postgresql container and restart postgres container

docker compose up -d --no-deps s4dbs_postgres
docker compose exec s4dbs_postgres bash
  su - postgres
    psql -X
      ALTER EXTENSION timescaledb UPDATE;
      exit
    psql -X -d speedgain
      ALTER EXTENSION timescaledb UPDATE;
      exit
    exit
  exit
docker compose restart s4dbs_postgres

5. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.3.7. from v1.3.5 to v1.4.0

When migrating from version 1.3.5 to 1.4.0, at first the image tags in the Docker Compose file have to be changed (except for timescale and nginx), new images pulled or loaded, and containers recreated.

Next step is to re-set grafana file and folder ownerships (was root (0) and is now grafana(472)).

An optional step is to add healthchecks to the containers and update the dependencies if you are using healthchecks. This optimizes the startup routines.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.5
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.4.0
    ...

Note	Timescale and nginx image version tags are not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_<version>_linux_<amd64/arm64>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. reset grafana ownerships

To reset grafana ownerships you can use one of these 3 options:

Option a) run a sidecar container to reset ownerships and restart grafana container

docker run --volumes-from $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1) -i -u 0 -t --rm nginx:1.23 chown -R 472:root /var/lib/grafana;

Option b) copy grafana.db from container, remove container, recreate container, copy grafana db in container, set ownership of database, run container

docker compose cp s4dbs_grafana:/var/lib/grafana/grafana.db grafana.db.backup
docker rm -v $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1)
docker volume prune
docker compose create s4dbs_grafana
docker compose cp grafana.db.backup s4dbs_grafana:/var/lib/grafana/grafana.db
docker run --volumes-from $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1) -i -u 0 -t --rm nginx:1.23 chown -R 472:root /var/lib/grafana/grafana.db

Option c) delete the grafana container and all of its volumes (will lose any user driven data like grafana user and passwords, additional dashboards, new datasources etc.) and recreate the container

IMPORTANT:

This removes all user driven data in Grafana and should only be used if you do not have any of that!
Be aware, this removes all unused local volumes. Unused local volumes are those which are not referenced by any containers!
If you are unsure about one of the points above, please don’t use this option!

docker rm -v $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1)
docker volume prune --all

5. add healtchecks and update dependencies

Every service in the compose file gets its own healthcheck command and an update in the dependencies section to enhance the startup routine. You can adjust the settings based on your environment. See latest version of a complete Docker Compose example at Docker Compose file

s4dbs_postgres:

    healthcheck:
      test: ["CMD-SHELL", "su postgres -c \"psql -h localhost -c '\\q'\""]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s

s4dbs_collector:

    healthcheck:
      test: ["CMD-SHELL", "ps -efa | grep \"java\" | grep \"Speedgain_for_Databases.jar\""]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_service:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:8080/Speedgain_for_Databases_Service"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 60s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_grafana:

    healthcheck:
      test: ["CMD-SHELL", "ps -efa  |  grep \"grafana-server\" | grep \"packaging=docker\" "]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 10s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_frontend:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:8080/s4dbs-ui/"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 60s
    depends_on:
      s4dbs_service:
        condition: service_healthy
      s4dbs_grafana:
        condition: service_healthy

s4dbs_reverse:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:80"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 10s
    depends_on:
      s4dbs_grafana:
        condition: service_healthy
      s4dbs_frontend:
        condition: service_healthy
      s4dbs_service:
        condition: service_healthy

6. Add new frontend environment variable

The frontend got a new environment variable called FRONTEND_CONFIG_MODE to control its behavior in Kubernetes and OpenShift clusters. This variable must be added to your compose file under the s4dbs_frontend service entry. Existing variables like "SERVICE_PORT" etc, stay in place as they are.

  s4dbs_frontend:
  ...
    environment:
      - FRONTEND_CONFIG_MODE=true
      ...

Setting the FRONTEND_CONFIG_MODE to the value "false" is only suggested for Kubernetes and OpenShift environments.

7. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.3.8. from v1.3.0 to v1.3.5

When migrating from version 1.3.0 to 1.3.5 just the image tags in the Docker Compose file have to be changed (except for timescale and nginx), new images pulled or loaded and containers recreated. You can always find the latest version of a complete Docker Compose example at Docker Compose file

1. change docker-compose.yml to meet current versions

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.0
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.5
    ...

Note	Timescale and nginx image version tags are not changing.

2. load/pull images and create/re-create containers

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.

5.3.9. from v1.2.2 to v1.3.0

When migrating from version 1.2.2 to version 1.3.0 the image tags in the Docker Compose file have to be changed at first. With version 1.3.0 a reverse proxy is introduced for single port access to all web services. This setting is optional! If you want to use it, some settings and files have to be added and settings have to be changed. You can always find the latest version of a complete Docker Compose example at Docker Compose file. After changing the configuration the images have to be pulled/loaded and the containers have to be created/re-created.

1. change image tags

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_service:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.2.2
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.0
    ...

Note	Timescale image version tag is not changing.

2. add service for reverse proxy in Docker Compose

  ...
  s4dbs_reverse:
    image: nginx:1.23
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: 10m
        max-file: "5"
    ports:
      - "80:80"     # Disable this line when SSL should be used
      #- "443:443"   # Enable this line to activate SSL
    depends_on:
      - s4dbs_grafana
      - s4dbs_frontend
      - s4dbs_service
    volumes:
      - ./configuration/nginx.conf:/etc/nginx/nginx.conf           # Enable SSL in this config file
      #- ./ssl/ssl_cert.key:/etc/ssl/ssl_cert.key    # Enable this line to activate SSL
      #- ./ssl/ssl_cert.crt:/etc/ssl/ssl_cert.crt    # Enable this line to activate SSL
  ...

Note	The public port does not have to 80, you can choose any free port available. Just change the line from '- "80:80"' to '- "MyPublicPort:80', same applies for 443 if ssl is used.

3. change frontend settings to match new public port

search in your docker-compose.yml for the lines starting with "- SERVICE_PORT=" and "- GRAFANA_PORT=" and change the ports to the public port of your reverse proxy and add "/grafana" on the GRAFANA_PORT setting. An example if port 80 ist used:

  ...
  s4dbs_frontend:
  ...
    environment:
      - SERVICE_PROTOCOL=http                              #use http or https
      - SERVICE_HOST=<enter container host hostname here>
      - SERVICE_PORT=80                                    #use 80 or 443
      - GRAFANA_PROTOCOL=http                              #use http or https
      - GRAFANA_HOST=<enter container host hostname here>
      - GRAFANA_PORT=80/grafana                            #use 80 or 443
  ...

4. change grafana settings to match hostname and sub path serving

  ...
  s4dbs_grafana:
  ...
    environment:
      - PDB_USER=postgres
      ...
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana/
      - GF_SERVER_DOMAIN=<enter container host hostname here>
      ...
    depends_on:
      - s4dbs_postgres

5. remove all port mappings except the reverse proxy mapping

As every http/https communication between your browser and Speedgain for Databases is now served via a nginx proxy (s4dbs_reverse), all other ports can removed or marked as a comment. Example:

  ...
  s4dbs_service:
    ...
    #ports:
    #  - "8088:8080"
    ...
  s4dbs_grafana:
    ...
    #ports:
    #  - "3000:3000"
    ...
  s4dbs_frontend:
    ...
    #ports:
    #  - "8080:8080"
  ...

6. copy nginx.conf to configuration directory

create a subfolder called "configuration" besides your docker-compose.yml and place the nginx.conf file (NginX Configuration file) inside of it.

7. load/pull and create/re-create containers

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.

5.3.10. from <v1.2.2 to v1.2.2

When migrating from a version lower than 1.2.2 just the image tags in the Docker Compose file have to be changed, new images pulled or loaded and containers recreated. You can always find the latest version of a complete Docker Compose example at Docker Compose file

1. change docker-compose.yml to meet current versions

docker.io/itgainis/s4dbs_service:1.2.0 -> docker.io/itgainis/s4dbs_service:1.2.2
docker.io/itgainis/s4dbs_grafana:1.2.0 -> docker.io/itgainis/s4dbs_grafana:1.2.2

2. pull or load the images

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.