Skip to main content

Upgrade Requirements

Carefully read this information before upgrading AtScale to the corresponding release.

Important

It is highly recommended that you upgrade container-based AtScale consecutively and do not skip upgrades for intermediate releases. AtScale is introducing many innovations and corrections at this time. Upgrading to a version while skipping an intermediate one could introduce uncertainties despite the documented requirements for specific upgrades. This recommendation is expected to change in late 2025; check back as upgrade requirements are minimized and this recommendation subsides.

Attention

It is recommended that you backup your AtScale PostgreSQL instance before performing an upgrade.

C2025.5.0​

Upgrade prerequisites and breaking changes​

Before upgrading to AtScale C2025.5.0, you need to prepare by saving some of your existing secrets to your values override file.

  1. Copy all Keycloak users and client secrets:

    1. Obtain the secrets for the Keycloak users atscale-kc-admin and kc-admin from the existing secret keycloak-secret:

      kubectl get secret -n <atscale-release-namespace> keycloak-secret -o json | jq -r .data.KC_ATSCALE_ADMIN_PASSWORD | base64 -d
      kubectl get secret -n <atscale-release-namespace> keycloak-secret -o json | jq -r .data.KEYCLOAK_ADMIN_PASSWORD | base64 -d

    2. Obtain the AtScale client secrets:

      kubectl get secret -n <atscale-release-namespace> keycloak-api-secret -o json | jq -r .data.clientSecret | base64 -d
      kubectl get secret -n <atscale-release-namespace> keycloak-engine-secret -o json | jq -r .data.clientSecret | base64 -d
      kubectl get secret -n <atscale-release-namespace> keycloak-entitlement-secret -o json | jq -r .data.clientSecret | base64 -d
      kubectl get secret -n <atscale-release-namespace> keycloak-api-secret -o json | jq -r .data.publicApiClientSecret | base64 -d
      kubectl get secret -n <atscale-release-namespace> keycloak-sml-secret -o json | jq -r .data.clientSecret | base64 -d

  2. Add the secrets you obtained above to your values override file:

    global:
      atscale:
        keycloak:
          users:
            atscale:
              # defaults to atscale-kc-admin
              username: "your-username"
              # kubectl get secret -n <atscale-release-namespace> keycloak-secret -o json | jq -r .data.KC_ATSCALE_ADMIN_PASSWORD | base64 -d
              password: "your-password"
            admin:
              # defaults to kc-admin
              username: "your-username"
              # kubectl get secret -n <atscale-release-namespace> keycloak-secret -o json | jq -r .data.KEYCLOAK_ADMIN_PASSWORD | base64 -d
              password: "your-password"

          clients:
            # kubectl get secret -n <atscale-release-namespace> keycloak-api-secret -o json | jq -r .data.clientSecret | base64 -d
            api:
              clientSecret: "your-client-secret"
            # kubectl get secret -n <atscale-release-namespace> keycloak-engine-secret -o json | jq -r .data.clientSecret | base64 -d
            engine:
              clientSecret: "your-client-secret"
            # kubectl get secret -n <atscale-release-namespace> keycloak-entitlement-secret -o json | jq -r .data.clientSecret | base64 -d
            entitlement:
              clientSecret: "your-client-secret"
            # kubectl get secret -n <atscale-release-namespace> keycloak-sml-secret -o json | jq -r .data.clientSecret | base64 -d
            modeler:
              clientSecret: "your-client-secret"
            # kubectl get secret -n <atscale-release-namespace> keycloak-api-secret -o json | jq -r .data.publicApiClientSecret | base64 -d
            publicApi:
              clientSecret: "your-client-secret"

  3. If you have a multi-node cluster:

    • If you are running on a multi-node cluster that doesn’t have zone redundant storage (ZFS, EFS, NFS, Regional PD, etc.):

      1. If you don't have a default storage class set, specify the one you use in the values override file. For example:

        # AN EBS Storage Example (Not zone redundant) 
          telemetry:
            persistence:
              storageClass: "gp2"
              accessModes:
                - ReadWriteOnce

      2. Ensure that your engine pod and OTel pods are running on the same nodes, then add their node selectors to the values override file. You can obtain the node labels by running kubectl get nodes --show-labels.

        atscale-engine:
          nodeSelector:
            node-label: node-label-value
        telemetry:
          nodeSelector:
            node-label: node-label-value

    • If you are running on a multi-node cluster that does have zone redundant storage, make sure that accessModes under global.atscale.telemetry.persistence is set to ReadWriteMany. For example:

      # AN EFS Storage (zone redundant) 
        telemetry:
          persistence:
            storageClass: "efs-sc"
            accessModes:
              - ReadWriteMany

      In this case, node selectors are not applicable because you can write to this PV from any node.

  4. Upgrade AtScale.

New commands for obtaining AtScale admin users and passwords​

In C2025.5.0, the AtScale admin users and clients have been moved to a new secret. After upgrading, you can use the following commands to obtain the default admin users and their passwords.

# AtScale Admin USER
kubectl get secret keycloak-users-secret -n <release-namespace> -o jsonpath="{.data.atscaleAdmin}" | base64 -d
# AtScale Admin Password
kubectl get secret keycloak-users-secret -n <release-namespace> -o jsonpath="{.data.atscaleAdminPassword}" | base64 -d
# Keycloak Admin User
kubectl get secret keycloak-users-secret -n <release-namespace> -o jsonpath="{.data.keycloakAdmin}" | base64 -d
# Keycloak Admin Password
kubectl get secret keycloak-users-secret -n <release-namespace> -o jsonpath="{.data.keycloakAdminPassword}" | base64 -d

Troubleshooting​

The following are common issues you may encounter when upgrading to C2025.5.0.

  • In this release, the StatefulSets for Redis and Keycloak changed. When upgrading, if you receive an error like Error: UPGRADE FAILED: cannot patch "redis-master" with kind StatefulSet, it is because the upgrade process is attempting to make changes to the StatefulSets and your system is preventing it from doing so.

    To fix the issue, you need to remove your StatefulSets:

    kubectl get statefulsets -n <your-namespace>
    # Grab the names from the list
    kubectl delete statefulset <name of statefulset>
    Note

    This is required for Redis, and may be needed for Keycloak, as well. It is safe to delete the StatefulSets and run the upgrade again.

  • If you are performing multiple upgrades for any reason, ensure the client secrets for the following have updated in your pods: atscale-engine, atscale-entitlement, atscale-sml, and atscale-api. These should match the values you specified in the values override file above.

    It is safe to terminate the pods so they pick up the current secrets.

C2025.3.1​

Breaking change: ATSCALE_CRYPTO_KEY must be a 64 character hexadecimal string​

Before upgrading to C2025.3.1 or later, verify that the ATSCALE_CRYPTO_KEY secret is a 64 character hexadecimal string. This environment variable is stored in either the tableau-secret or the global.atscale.encryption section of the values override file. If this variable is not a 64 character hexadecimal string, it should be replaced.

Note

Replacing the variable may cause connections to Tableau and LLM to be lost. If this occurs, they must be recreated.

Update existing Tableau Server definitions​

In C2025.3.1, the workflow for configuring Tableau Server definitions changed. Any existing Tableau Server definitions must be reconfigured using the new workflow after upgrading to C2025.3.1 or later. For instructions, see Configuring Connections to Tableau Server.

ATSCALE-25647

C2025.3.0​

Breaking change: Nginx Ingress Controller replaced with Nginx Proxy​

In C2025.3.0, the Nginx Ingress Controller was removed and replaced with Nginx Proxy. As part of this change, the nginxproxy section was removed from the AtScale values file, and all proxy configurations moved to the section atscale-proxy. This brings more flexibility to the AtScale deployment, mainly:

  • The ability to install AtScale in multiple namespaces.
  • A single point of entry for the AtScale application.
  • Automatic generation of self-signed TLS secrets.
  • Use of HTTP/1.1 or 2.0 by default.
  • Ability to expose the AtScale application as either a service or an ingress.
  • Exposure of Nginx Proxy parameters on the Helm chart.

If you currently use nginxproxy, before upgrading to AtScale C2025.3.0 from an earlier version, you must migrate all the settings in the nginxproxy section of the values override file to the atscale-proxy section. Notably:

  1. Migrate all annotations from nginxproxy.service.annotations to atscale-proxy.service.annotations.
  2. Redirect all routes created to the nginx service to the atscale-proxy service.
  3. Review OpenShift Routes, Istio VirtualServices, and other environment-specific routing Custom Resources, and point them to the atscale-proxy service.
Note

If you do not use nginxproxy, no action is needed.

For more information on configuring the new atscale-proxy service, see Configuring the AtScale Proxy Service.

Breaking change: caCerts in values override file must be Base64-encoded​

As of C2025.3.0, the value of the caCerts property in the values override file must be Base64-encoded.

Before upgrading to C2025.3.0 from an earlier version, you must ensure that the value of this property is in Base64 format. For instructions on configuring your values override file, see Creating a Values Override File. For instructions on updating the values override file, see Updating the Values Override File.

C2025.2.0​

(Optional) Configure an external Grafana instance for monitoring​

As of C2025.2.0, Grafana (used to provide monitoring for the AtScale services) has been deprecated, and is no longer included in the AtScale Helm chart. If you want to continue to use Grafana for monitoring, you must configure an external instance. For instructions, see Configuring an External Grafana Instance for Monitoring AtScale.

Note

This is an optional configuration, and is not required for upgrading to C2025.2.0 or later.

C2025.1.0​

Manual steps required before upgrading​

The atscale-api deployment now requires the AtScale Public API Keycloak client secret. In existing AtScale installs, before upgrading, you need to add this secret to keycloak-api-secret on your Kubernetes cluster.

note

The exact steps for doing this depend on the way secrets are managed on your cluster.

Do the following:

  1. In the Identity Broker, go to Clients and open the atscale-public-api client.
  2. Go to the Credentials tab and copy the value of Client Secret.
  3. On your Kubernetes cluster, edit keycloak-api-secret to add the value you copied to the publicApiClientSecret parameter. The value you set must be Base64 encoded, or the Identity Broker will fail.
  4. Upgrade AtScale as normal.

Manual steps required to keep monitoring active​

In C2025.1.0, Grafana, Mimir, and Tempo are deprecated in the AtScale helm chart, and their subcharts are disabled by default. If you need to keep monitoring active for development purposes, add the following to values.yaml (or your values override file, if you have one):

tempo:
  enabled: true
mimir:
  enabled: true
grafana:
  enabled: true
note

The deprecated components will be completely removed in a future release. After full deprecation, AtScale will provide a separate chart for telemetry.

Update dimension SML files​

In C2025.1.0, the use_input_measure_format SML property (under calculated_members in dimension files) was renamed to use_input_metric_format. After upgrading, you should update your dimension files to reflect these changes.

For more information on dimension files, see the AtScale SML Object Documentation on GitHub.

ATSCALE-24226

C2024.10.0​

Manual steps required for internal PostgreSQL update​

In C2024.10.0, AtScale updated the PostgreSQL database to 16.4.0 (PostgreSQL chart 12.0.5 to 14.3.5). This change requires some manual effort on behalf of the AtScale admin when upgrading to C2024.10.0. Should you require assistance from AtScale, reach out to AtScale Support.

  1. Before upgrading, backup PostgreSQL with the following command. Note that this requires pg_dumpall to be installed. The pg_dumpall install process may be different for each OS.

    export PGPASSWORD=$(kubectl get secret -n <NAMESPACE> postgres-default-user -o json | jq -r '.data.password' | base64 -d)
    kubectl port-forward -n <NAMESPACE> svc/postgres-pgpool <DESTINATION_PORT>:10518
    pg_dumpall -h 127.0.0.1 -p <DESTINATION_PORT> -U postgres --no-password > backup.dump

    Be sure to replace <NAMESPACE> with the namespace where AtScale is installed, and <DESTINATION_PORT> with a free port on your local environment. A commonly used port is 25432.

    note

    Some warnings might appear; this is expected.

  2. The replication manager update should be enabled, and the number of replicas should be set to 1. In order to this, update values.yaml with the following:

    postgres:
        postgresql:
            replicaCount: 1
            upgradeRepmgrExtension: true

  3. Run a helm upgrade normally.

  4. On postgres-postgresql-0, there should be COLLATION ERRORS. To resolve these, connect to postgres-postgresql-0 with your preferred Database Manager (DBeaver, DataGrip, HeidiSQL, etc.) and run the following commands:

    ALTER DATABASE atscale REFRESH COLLATION VERSION;
    ALTER DATABASE keycloak REFRESH COLLATION VERSION;
    ALTER DATABASE pgwire REFRESH COLLATION VERSION;
    ALTER DATABASE postgres REFRESH COLLATION VERSION;
    ALTER DATABASE repmgr REFRESH COLLATION VERSION;
    ALTER DATABASE template1 REFRESH COLLATION VERSION;
    note

    If you need assistance connecting to postgres-postgresql-0, reach out to AtScale support.

  5. Undo the changes made in Step 2, then run helm upgrade again.

    note

    It may take some time to recreate all replicas.

Delete Kong resources after upgrading​

In C2024.10.0, the technology behind AtScale’s ingress controller changed. If you deployed AtScale using a base deployment (OOTB), Nginx will supplant Kong as the ingress controller, and the upgrade will automatically switch from Kong to Nginx.

As a result, some Kong resources will be left over. AtScale recommends that you remove these resources post-upgrade (note, however, that this is optional).

note

This recommendation assumes you have not used Kong for any application other than AtScale.

To remove the remaining Kong resources, run the following after upgrading to C2024.10.0:

kubectl delete crd $(kubectl get crd | grep konghq | awk '{print $1}')

If you added any customizations to Kong, they will likely need to be migrated to Nginx.

If you use a third party load balancer in lieu of AtScale’s ingress controller, Nginx does not need to be deployed. AtScale recommends that any deployment mimic its official Helm chart as closely as possible.

DOC-1520

Manually transition engine settings to a global settings file​

In C2024.10.0, AtScale introduced a new system for managing global settings: global_settings.yml files managed in Git, as opposed to the Settings configuration page. Although the Settings page is still available in C2024.10.0, AtScale recommends that you begin manually transitioning your existing engine settings to a global settings file. For instructions on creating and updating global settings files, see Configuring Global Settings.

ATSCALE-21925

Update calculation groups in SML​

C2024.10.0 introduced updates to the calculation_groups property in SML dimension files. After upgrading to this release, you must update any calculation groups in your dimensions to use the new syntax. Make the following changes:

  • For the calculation_groups property:

    • Change the name property to unique_name.
    • Add the label property. This property is required.

  • For the calculation_groups > calculated_members property:

    • Change the name property to unique_name.

For more information on defining calculation groups in SML, refer to the SML Object Documentation on GitHub.

ATSCALE-22943

C2024.9.0​

Update the atscale-api and atscale-engine clients​

Before upgrading to AtScale C2024.9.0, you need to update the atscale-api and atscale-engine clients with additional service account roles.

  1. In the Identity Broker, click Clients in the left-hand navigation, then click the atscale-engine client.

  2. On the Settings tab, in the Capability config section, locate the Authentication flow field and enable Service accounts roles.

  3. Click Save.

  4. Go to the Service accounts roles tab.

  5. Click Assign role, then assign the atscale-engine client following roles:

    • admin realm role
    • uma_protection client role

  6. Click Clients in the left-hand navigation and select the atscale-api client.

  7. On the Settings tab, in the Capability config section, locate the Authentication flow field and enable Service accounts roles.

  8. Click Save.

  9. Go to the Service accounts roles tab.

  10. Click Assign role and assign the atscale-api client the admin realm role.

note

If these changes are made post-upgrade, the engine and API containers need to be restarted.

DOC-1446

Update semi-additive metrics​

C2024.9.0 introduced updates to the semi_additive property in SML metric files. After upgrading to this release, you must update any semi-additive metrics in your models to use the new syntax. For the full reference for the new properties, refer to the AtScale SML Object Documentation on GitHub.

Update atscale.yml files​

In C2024.9.0, the atscale.yml file was renamed to catalog.yml, and its as_version property was renamed to version. After upgrading to this release, AtScale recommends making the following changes to the atscale.yml files in any existing models you have:

  1. Rename the files to catalog.yml.
  2. Rename their as_version properties to version.

C2024.8.0​

Resize the PostgreSQL and MinIO PVCs​

In C2024.8.0, the default sizes required for the MinIO and PostgreSQL PVCs included with AtScale increased from 16Gi to 64Gi. You must manually resize these as part of the upgrade procedure.

Note

The following procedure uses the default names for the PVCs and StatefulSets. Be sure to use the names specific to your cluster when performing these steps.

  1. Before upgrading AtScale:

    1. Ensure resizing is enabled for the PVCs:

      1. Find the PVC storage class:

        kubectl get pvc -n <namespace>

      2. Enable resizing:

        kubectl patch sc <storage_class_name> -p '{"allowVolumeExpansion": true}'

    2. Resize the PostgreSQL PVCs:

      KUBE_EDITOR="sed -i s/16Gi/64Gi/g" kubectl edit pvc -n <namespace> data-postgres-postgresql-0
      KUBE_EDITOR="sed -i s/16Gi/64Gi/g" kubectl edit pvc -n <namespace> data-postgres-postgresql-1
      KUBE_EDITOR="sed -i s/16Gi/64Gi/g" kubectl edit pvc -n <namespace> data-postgres-postgresql-2

    3. Resize the MinIO PVCs:

      KUBE_EDITOR="sed -i s/16Gi/64Gi/g" kubectl edit pvc -n <namespace> minio

    4. Delete the PostgreSQL StatefulSet:

      kubectl delete sts -n <namespace> --cascade=orphan postgres-postgresql

      PostgreSQL is a High Availability cluster and is therefore deployed as a StatefulSet. Because these don't support changes, you must delete them. Note that --cascade=orphan leaves all pods running, so you should not experience any downtime.

  2. Upgrade the AtScale chart:

    helm upgrade atscale oci://docker.io/atscaleinc/atscale --version <version> -n <namespace> -f <override_file>

    Where:

    • <version> is the version of AtScale you want to upgrade to.
    • <namespace> is the namespace in which AtScale is installed.
    • <override_file> is your values override file.

  3. Restart the pods in order to resize the mounted volume:

    kubectl rollout restart sts -n <namespace> postgres-postgresql

C2024.7.0​

Aggregate Rebuild on Upgrade Warning for IRIS Data Warehouses​

If you use InterSystems IRIS, aggregate tables will be rebuilt when you upgrade from C2024.1.1 to C2024.7.0 or later. You should plan your upgrade so that the system has time to rebuild the affected aggregates outside of business operations.

If you want to perform the upgrade without immediately rebuilding the aggregates, do the following:

  1. Before upgrading, set the custom global setting aggregate.maintenance.job.invalid-physical-plans.enabled = False and restart the engine.
  2. Perform the upgrade.
  3. When you are ready to rebuild aggregates after the upgrade, set aggregate.maintenance.job.invalid-physical-plans.enabled = True and restart the engine.

ATSCALE-21956