Speedgain for Databases

5.1. Migrate to newest release

Speedgain for Databases migration is an easy task when a container based installation is used. In general terms: only the image tag numbers have to be changed to current release + some additional tasks have to be done. The following chapters describe which tasks these are when migrating from one version to another.

5.1.1. general

New collectors will not automatically be enabled in pre-existing policies. Have a look for new collectors in every release and activate them if needed - see chapter Change / Add monitoring policies for a how-to and look at Collectors for a list of all available collectors and the version of their availability.

5.1.2. from v1.5.0 to v1.6.0

At first when migrating from version 1.5.0 to 1.6.0, the image tags in the Docker Compose file have to be changed (except timescale and nginx), small adjustment on the nginx.conf, new images pulled or loaded, and containers recreated.

Next, you should check any custom boards and alerts for compatibility with the new grafana version 10.2.3. There are breaking changes introduced in grafana v10.

Important

For helm chart users: backup your postgresql passwort secret to have it in place if it gets removed during upgrade.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.5.0
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.6.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.6.0
    ...

Note	Timescale and Nginx image version tags are not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_v<version>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. adjust nginx.conf

Adjust the nginx.conf file in your configuration folder. The lines starting with "rewrite" have to be removed. From:

# grafana
    location /grafana/ {
      rewrite  ^/grafana/(.*)  /$1 break;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;

    }

    location /grafana/api/live/ {
      rewrite  ^/grafana/(.*)  /$1 break;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
    }

# grafana
    location /grafana/ {
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;

    }

    location /grafana/api/live/ {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
      proxy_pass http://grafana;
      proxy_read_timeout 300;
      proxy_connect_timeout 300;
      proxy_send_timeout 300;
    }

5. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

6. verfiy any custom boards and alerts are still functional

In Speedgain for Databases v1.6.0 we migrated grafana to version 10.2.3 If you are using custom boards or alarms in your Speedgain for Database installation you will have to check if your boards and alerts are affected by breaking changes. Major changes are:

There is a new Pie Chart visualization. The old pie chart will no longer be rendered. To switch to the new pie chart just edit the existing panel and switch the visualization.
The Graph Panel has been deprecated and should be migrated to the new time series visualization.
The alerting system has been completely reworked. All existing alarms will be automatically migrated to the new engine. Check, if your alarms could be migrated without errors. Adjust the alarm manually if necessary.

For further information on the changes in grafana 10 refer to the official documentation at https://grafana.com/docs/grafana/latest/breaking-changes/breaking-changes-v10-0/

5.1.3. from v1.4.0 to v1.5.0

At first when migrating from version 1.4.0 to 1.5.0, the image tags in the Docker Compose file have to be changed (inlcuding timescale and except nginx), new images pulled or loaded, and containers recreated.

Next, in the container-configuration for grafana remove the variable GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH if you do not use your own default dashboard for speedgain. The path to the speedgain start dashboard has changed and is now set within the container itself.

The last step is to upgrade the timescale extension in postgresql container.

Warning

For helm chart users: we changed the default for grafana database from sqllite to postgresql. If you are mgirating and made changes to grafana you may want to set the parameter GF_DATABASE_TYPE to sqllite instead of postgres to keep your old grafana settings.

Important

For helm chart users: backup your postgresql passwort secret to have it in place if it gets removed during upgrade.

Tip	No need to worry if you lost your postgresql password or secret. It can be easily reset if you have admin access to the pods/containers. Ask our support for help.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

Change the image tags in the docker-compose.yml from

    ...
    image: docker.io/timescale/timescaledb:2.5.1-pg13
    ...
    image: docker.io/itgainis/s4dbs_collector:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.4.0
    ...

    ...
    image: docker.io/timescale/timescaledb:2.11.2-pg13
    ...
    image: docker.io/itgainis/s4dbs_collector:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.5.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.5.0
    ...

Note	Nginx image version tags are not changing.

3. remove grafana default board variable from docker-compose

In your docker-compose.yml at the grafana container remove the row for the variable GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH.

4. pull or load the images

docker compose pull

docker load --input s4dbs_v<version>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

5. update timescale extension in postgresql container and restart postgres container

docker compose up -d --no-deps s4dbs_postgres
docker compose exec s4dbs_postgres bash
  su - postgres
    psql -X
      ALTER EXTENSION timescaledb UPDATE;
      exit
    psql -X -d speedgain
      ALTER EXTENSION timescaledb UPDATE;
      exit
    exit
  exit
docker compose restart s4dbs_postgres

5. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.1.4. from v1.3.5 to v1.4.0

When migrating from version 1.3.5 to 1.4.0, at first the image tags in the Docker Compose file have to be changed (except for timescale and nginx), new images pulled or loaded, and containers recreated.

Next step is to re-set grafana file and folder ownerships (was root (0) and is now grafana(472)).

An optional step is to add healthchecks to the containers and update the dependencies if you are using healthchecks. This optimizes the startup routines.

You can always find the latest version of a complete Docker Compose example at Docker Compose file.

1. stop running containers

docker compose stop

2. change docker-compose.yml to meet current versions

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.5
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.4.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.4.0
    ...

Note	Timescale and nginx image version tags are not changing.

3. pull or load the images

docker compose pull

docker load --input s4dbs_v<version>_all_images.tar.gz

pulling and loading container images is described in detail in Install Step-by-Step chapter.

4. reset grafana ownerships

To reset grafana ownerships you can use one of these 3 options:

Option a) run a sidecar container to reset ownerships and restart grafana container

docker run --volumes-from $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1) -i -u 0 -t --rm nginx:1.23 chown -R 472:root /var/lib/grafana;

Option b) copy grafana.db from container, remove container, recreate container, copy grafana db in container, set ownership of database, run container

docker compose cp s4dbs_grafana:/var/lib/grafana/grafana.db grafana.db.backup
docker rm -v $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1)
docker volume prune
docker compose create s4dbs_grafana
docker compose cp grafana.db.backup s4dbs_grafana:/var/lib/grafana/grafana.db
docker run --volumes-from $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1) -i -u 0 -t --rm nginx:1.23 chown -R 472:root /var/lib/grafana/grafana.db

Option c) delete the grafana container and all of its volumes (will lose any user driven data like grafana user and passwords, additional dashboards, new datasources etc.) and recreate the container

IMPORTANT:

This removes all user driven data in Grafana and should only be used if you do not have any of that!
Be aware, this removes all unused local volumes. Unused local volumes are those which are not referenced by any containers!
If you are unsure about one of the points above, please don’t use this option!

docker rm -v $(docker ps -a --format "{{.ID}} {{.Names}}" | grep 's4dbs_grafana_1\|s4dbs_grafana-1'| cut -d " " -f1)
docker volume prune --all

5. add healtchecks and update dependencies

Every service in the compose file gets its own healthcheck command and an update in the dependencies section to enhance the startup routine. You can adjust the settings based on your environment. See latest version of a complete Docker Compose example at Docker Compose file

s4dbs_postgres:

    healthcheck:
      test: ["CMD-SHELL", "su postgres -c \"psql -h localhost -c '\\q'\""]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s

s4dbs_collector:

    healthcheck:
      test: ["CMD-SHELL", "ps -efa | grep \"java\" | grep \"Speedgain_for_Databases.jar\""]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 30s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_service:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:8080/Speedgain_for_Databases_Service"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 60s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_grafana:

    healthcheck:
      test: ["CMD-SHELL", "ps -efa  |  grep \"grafana-server\" | grep \"packaging=docker\" "]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 10s
    depends_on:
      s4dbs_postgres:
        condition: service_healthy

s4dbs_frontend:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:8080/s4dbs-ui/"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 60s
    depends_on:
      s4dbs_service:
        condition: service_healthy
      s4dbs_grafana:
        condition: service_healthy

s4dbs_reverse:

    healthcheck:
      test: ["CMD-SHELL", "curl -f localhost:80"]
      interval: 10s
      timeout: 10s
      retries: 10
      start_period: 10s
    depends_on:
      s4dbs_grafana:
        condition: service_healthy
      s4dbs_frontend:
        condition: service_healthy
      s4dbs_service:
        condition: service_healthy

6. Add new frontend environment variable

The frontend got a new environment variable called FRONTEND_CONFIG_MODE to control its behavior in Kubernetes and OpenShift clusters. This variable must be added to your compose file under the s4dbs_frontend service entry. Existing variables like "SERVICE_PORT" etc, stay in place as they are.

  s4dbs_frontend:
  ...
    environment:
      - FRONTEND_CONFIG_MODE=true
      ...

Setting the FRONTEND_CONFIG_MODE to the value "false" is only suggested for Kubernetes and OpenShift environments.

7. create/re-create containers

docker compose up -d

re-creating containers and further steps are described in detail in Install Step-by-Step chapter.

5.1.5. from v1.3.0 to v1.3.5

When migrating from version 1.3.0 to 1.3.5 just the image tags in the Docker Compose file have to be changed (except for timescale and nginx), new images pulled or loaded and containers recreated. You can always find the latest version of a complete Docker Compose example at Docker Compose file

1. change docker-compose.yml to meet current versions

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.0
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.5
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.5
    ...

Note	Timescale and nginx image version tags are not changing.

2. load/pull images and create/re-create containers

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.

5.1.6. from v1.2.2 to v1.3.0

When migrating from version 1.2.2 to version 1.3.0 the image tags in the Docker Compose file have to be changed at first. With version 1.3.0 a reverse proxy is introduced for single port access to all web services. This setting is optional! If you want to use it, some settings and files have to be added and settings have to be changed. You can always find the latest version of a complete Docker Compose example at Docker Compose file. After changing the configuration the images have to be pulled/loaded and the containers have to be created/re-created.

1. change image tags

change the image tags in the docker-compose.yml from

    ...
    image: docker.io/itgainis/s4dbs_collector:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_service:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.2.2
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.2.2
    ...

    ...
    image: docker.io/itgainis/s4dbs_collector:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_service:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_grafana:1.3.0
    ...
    image: docker.io/itgainis/s4dbs_frontend:1.3.0
    ...

Note	Timescale image version tag is not changing.

2. add service for reverse proxy in Docker Compose

  ...
  s4dbs_reverse:
    image: nginx:1.23
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: 10m
        max-file: "5"
    ports:
      - "80:80"     # Disable this line when SSL should be used
      #- "443:443"   # Enable this line to activate SSL
    depends_on:
      - s4dbs_grafana
      - s4dbs_frontend
      - s4dbs_service
    volumes:
      - ./configuration/nginx.conf:/etc/nginx/nginx.conf           # Enable SSL in this config file
      #- ./ssl/ssl_cert.key:/etc/ssl/ssl_cert.key    # Enable this line to activate SSL
      #- ./ssl/ssl_cert.crt:/etc/ssl/ssl_cert.crt    # Enable this line to activate SSL
  ...

Note	The public port does not have to 80, you can choose any free port available. Just change the line from '- "80:80"' to '- "MyPublicPort:80', same applies for 443 if ssl is used.

3. change frontend settings to match new public port

search in your docker-compose.yml for the lines starting with "- SERVICE_PORT=" and "- GRAFANA_PORT=" and change the ports to the public port of your reverse proxy and add "/grafana" on the GRAFANA_PORT setting. An example if port 80 ist used:

  ...
  s4dbs_frontend:
  ...
    environment:
      - SERVICE_PROTOCOL=http                              #use http or https
      - SERVICE_HOST=<enter container host hostname here>
      - SERVICE_PORT=80                                    #use 80 or 443
      - GRAFANA_PROTOCOL=http                              #use http or https
      - GRAFANA_HOST=<enter container host hostname here>
      - GRAFANA_PORT=80/grafana                            #use 80 or 443
  ...

4. change grafana settings to match hostname and sub path serving

  ...
  s4dbs_grafana:
  ...
    environment:
      - PDB_USER=postgres
      ...
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana/
      - GF_SERVER_DOMAIN=<enter container host hostname here>
      ...
    depends_on:
      - s4dbs_postgres

5. remove all port mappings except the reverse proxy mapping

As every http/https communication between your browser and Speedgain for Databases is now served via a nginx proxy (s4dbs_reverse), all other ports can removed or marked as a comment. Example:

  ...
  s4dbs_service:
    ...
    #ports:
    #  - "8088:8080"
    ...
  s4dbs_grafana:
    ...
    #ports:
    #  - "3000:3000"
    ...
  s4dbs_frontend:
    ...
    #ports:
    #  - "8080:8080"
  ...

6. copy nginx.conf to configuration directory

create a subfolder called "configuration" besides your docker-compose.yml and place the nginx.conf file (NginX Configuration file) inside of it.

7. load/pull and create/re-create containers

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.

5.1.7. from <v1.2.2 to v1.2.2

When migrating from a version lower than 1.2.2 just the image tags in the Docker Compose file have to be changed, new images pulled or loaded and containers recreated. You can always find the latest version of a complete Docker Compose example at Docker Compose file

1. change docker-compose.yml to meet current versions

docker.io/itgainis/s4dbs_service:1.2.0 -> docker.io/itgainis/s4dbs_service:1.2.2
docker.io/itgainis/s4dbs_grafana:1.2.0 -> docker.io/itgainis/s4dbs_grafana:1.2.2

2. pull or load the images

loading, starting, re-creating and further steps are described in detail in Install Step-by-Step chapter.