Restore backups

This page applies to SUSE® Observability v2.7.0 or newer.

Overview

This page describes how to restore backups for SUSE® Observability data stores using the backup CLI.

Restoring from backup will purge or overwrite existing data. This operation cannot be undone.

Always verify the backup name and namespace before confirming a restore operation.

GitOps Workflow Impacted This backup tool modifies K8s resources directly in your cluster:

  • Scales StatefulSets/Deployments to 0 replicas

  • Adds annotations for restore tracking

These changes will conflict with your GitOps workflow as they bypass Git-based reconciliation. GitOps controllers may attempt to revert these changes during backup operations. Important: Automatic reconciliation for the SUSE Observability Helm chart deployment must be disabled during backup restore to prevent conflicts.

Before you use the CLI, ensure that:

  • You are using the latest version of the backup CLI.

  • The CLI expects a kubeconfig with a current context to the cluster where SUSE Observability is installed.

Download the backup CLI

Download the latest version of the backup CLI using the commands below for your platform. The binary is named sts-backup.

  • macOS (Apple Silicon)

  • macOS (Intel)

  • Linux (ARM64)

  • Linux (x86_64)

  • Windows (x86_64)

VERSION=$(curl -sL https://api.github.com/repos/StackVista/stackstate-backup-cli/releases/latest | grep '"tag_name":' | cut -d'"' -f4)
curl -LO "https://github.com/StackVista/stackstate-backup-cli/releases/download/${VERSION}/stackstate-backup-cli-${VERSION#v}.darwin-arm64.tar.gz"
tar -xzf stackstate-backup-cli-${VERSION#v}.darwin-arm64.tar.gz
VERSION=$(curl -sL https://api.github.com/repos/StackVista/stackstate-backup-cli/releases/latest | grep '"tag_name":' | cut -d'"' -f4)
curl -LO "https://github.com/StackVista/stackstate-backup-cli/releases/download/${VERSION}/stackstate-backup-cli-${VERSION#v}.darwin-x86_64.tar.gz"
tar -xzf stackstate-backup-cli-${VERSION#v}.darwin-x86_64.tar.gz
VERSION=$(curl -sL https://api.github.com/repos/StackVista/stackstate-backup-cli/releases/latest | grep '"tag_name":' | cut -d'"' -f4)
curl -LO "https://github.com/StackVista/stackstate-backup-cli/releases/download/${VERSION}/stackstate-backup-cli-${VERSION#v}.linux-arm64.tar.gz"
tar -xzf stackstate-backup-cli-${VERSION#v}.linux-arm64.tar.gz
VERSION=$(curl -sL https://api.github.com/repos/StackVista/stackstate-backup-cli/releases/latest | grep '"tag_name":' | cut -d'"' -f4)
curl -LO "https://github.com/StackVista/stackstate-backup-cli/releases/download/${VERSION}/stackstate-backup-cli-${VERSION#v}.linux-x86_64.tar.gz"
tar -xzf stackstate-backup-cli-${VERSION#v}.linux-x86_64.tar.gz
$VERSION = (Invoke-RestMethod -Uri "https://api.github.com/repos/StackVista/stackstate-backup-cli/releases/latest").tag_name
Invoke-WebRequest -Uri "https://github.com/StackVista/stackstate-backup-cli/releases/download/$VERSION/stackstate-backup-cli-$($VERSION.TrimStart('v')).windows-x86_64.zip" -OutFile "stackstate-backup-cli.zip"
Expand-Archive -Path "stackstate-backup-cli.zip" -DestinationPath "."

For convenience, copy the sts-backup binary to a directory in your $PATH (e.g., /usr/local/bin on Linux/macOS) so you can run it from anywhere without specifying the full path.

Configuration and topology data (StackGraph)

List StackGraph backups

To list the StackGraph backups, execute the following command:

sts-backup stackgraph list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE® Observability is installed.

The output should look like this:

Setting up port-forward to suse-observability-minio:9000 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing Stackgraph backups in bucket 'sts-stackgraph-backup'...
NAME                            LAST MODIFIED            SIZE
sts-backup-20251128-0300.graph  2025-11-28 03:08:42 UTC  2GiB

The timestamp when the backup is taken is part of the backup name.

Restore a StackGraph backup

To restore a StackGraph backup, use one of the following approaches:

Restore a specific backup

sts-backup stackgraph restore --namespace <NAMESPACE> --archive <BACKUP_NAME>

Restore the latest backup

sts-backup stackgraph restore --namespace <NAMESPACE> --latest

Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

  • --background - Run restore in background without waiting for completion

Background restore

When using --background, the restore runs asynchronously. After starting the restore, use the following command to check status and finalize:

sts-backup stackgraph check-and-finalize --job <JOB_NAME> --wait --namespace <NAMESPACE>

The check-and-finalize command:

  • Checks the restore job status

  • With --wait flag, waits for job completion

  • Automatically scales up deployments that were scaled down during restore

  • Cleans up resources after completion

If a restore running without --background is interrupted (for example, by Ctrl+C), you must run check-and-finalize to scale up deployments and clean up resources.

Configuration and topology data v2 (StackGraph)

Since SUSE Observability {next-release-version} we are rolling out v2 of StackGraph backup/restore. For now both v1 and v2 backups will be made, until v2 is deemed completely stable, at which point we’ll sunset v1 backups.

V2 backups offer significant improvements in backup and restore speed, as well as incremental backups. Also, the stackgraph v2 restore procedure allows for restoring historical data while the system is running, which means much quicker time to get the system back up.

List StackGraph v2 backups

To list the StackGraph V2 backups, execute the following command:

sts-backup stackgraph-v2 list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE® Observability is installed.

The output should look like this:

Setting up port-forward to suse-observability-minio:9000 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing Stackgraph backups in bucket 'sts-stackgraph-backup'...
NAME                               LAST MODIFIED
sts-backup-20251128-0300.graph.v2  2025-11-28 03:08:42 UTC

The timestamp when the backup is taken is part of the backup name.

Restore a StackGraph v2 backup

The restore has a 'live' and 'backfill' portion of the backup. The 'live' part takes down the system and will restore the latest data. After the 'live' part is done, the system will scale up and is accessible by end-users. The following message appears to signify when the system is accessible again:

The system is back up and accessible. Continuing with backfilling historical data, this will happen while the system is running.
During this process not all historical data might be available.

To restore a StackGraph v2 backup, use one of the following commands:

Restore a specific backup

sts-backup stackgraph-v2 restore --namespace <NAMESPACE> --archive <BACKUP_NAME>

Restore the latest backup

sts-backup stackgraph-v2 restore --namespace <NAMESPACE> --latest

Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

== Settings

Settings backups include installed StackPacks with their configuration and other customizations created by the user, such as monitors, custom views, and service tokens. Settings backups are lightweight (typically only several megabytes) and quick to restore with minimal downtime.

=== List Settings backups

To list the Settings backups, execute the following command:

sts-backup settings list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE Observability is installed.

The output should look like this:

Setting up port-forward to suse-observability-minio:9000 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing Settings backups in bucket 'sts-configuration-backup'...
NAME                          LAST MODIFIED            SIZE
sts-backup-20251128-1328.sty  2025-11-28 13:29:12 UTC  2MiB

The timestamp when the backup was taken is part of the backup name.

=== Restore a Settings backup

Restoring a settings backup will also remove all topology, including health states, alerts and the topology history.

To restore a Settings backup, use one of the following approaches:

==== Restore a specific backup

sts-backup settings restore --namespace <NAMESPACE> --archive <BACKUP_NAME>

==== Restore the latest backup

sts-backup settings restore --namespace <NAMESPACE> --latest

==== Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

  • --background - Run restore in background without waiting for completion

==== Background restore

When using --background, the restore runs asynchronously. After starting the restore, use the following command to check status and finalize:

sts-backup settings check-and-finalize --job <JOB_NAME> --wait --namespace <NAMESPACE>

The check-and-finalize command:

  • Checks the restore job status

  • With --wait flag, waits for job completion

  • Automatically scales up deployments that were scaled down during restore

  • Cleans up resources after completion

If a restore running without --background is interrupted (e.g., by Ctrl+C), you must run check-and-finalize to scale up deployments and clean up resources.

== Metrics (Victoria Metrics)

Depending on the profile, nonha or ha, Victoria Metrics is deployed in different modes:

  • nonha profile - Single-node mode with one Victoria Metrics instance (victoria-metrics-0)

  • ha profile - HA (mirror) mode with two Victoria Metrics instances (victoria-metrics-0 and victoria-metrics-1)

=== List Victoria Metrics backups

To list the Victoria Metrics backups, execute the following command:

sts-backup victoria-metrics list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE Observability is installed.

==== Single-node mode output (nonha profile)

Setting up port-forward to suse-observability-minio:9000 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing VictoriaMetrics backups in bucket ...
NAME ({bucket}/{instance}-{created})                           UPDATED
sts-victoria-metrics-backup/victoria-metrics-0-20251030152500  2025-11-28 09:25:05 UTC

==== HA mode output (ha profile)

Setting up port-forward to suse-observability-minio:9000 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing VictoriaMetrics backups in bucket ...
NAME ({bucket}/{instance}-{created})                           UPDATED
sts-victoria-metrics-backup/victoria-metrics-1-20251030152500  2025-11-28 09:35:08 UTC
sts-victoria-metrics-backup/victoria-metrics-0-20251030152500  2025-11-28 09:25:04 UTC

NOTE: In HA mode, backups from both instances (victoria-metrics-0 and victoria-metrics-1) are listed.
      The restore command accepts either backup to restore both instances.

In HA mode, backups are created for both instances with different prefixes (victoria-metrics-0 and victoria-metrics-1). When restoring, you can specify either backup - the restore operation will restore the selected backup to both instances.

=== Restore a Victoria Metrics backup

All new metrics will be cached by vmagent during the restore process. Ensure the vmagent has enough memory to cache metrics.

To restore a Victoria Metrics backup, use one of the following approaches:

==== Restore a specific backup

sts-backup victoria-metrics restore --namespace <NAMESPACE> --archive <BACKUP_NAME>

==== Restore the latest backup

sts-backup victoria-metrics restore --namespace <NAMESPACE> --latest

==== Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

  • --background - Run restore in background without waiting for completion

==== Background restore

When using --background, the restore runs asynchronously. After starting the restore, use the following command to check status and finalize:

sts-backup victoria-metrics check-and-finalize --job <JOB_NAME> --wait --namespace <NAMESPACE>

The check-and-finalize command:

  • Checks the restore job status

  • With --wait flag, waits for job completion

  • Automatically scales up StatefulSets that were scaled down during restore

  • Cleans up resources after completion

If a restore running without --background is interrupted (e.g., by Ctrl+C), you must run check-and-finalize to scale up StatefulSets and clean up resources.

== OpenTelemetry (ClickHouse)

=== List ClickHouse backups

To list ClickHouse backups, execute the following command:

sts-backup clickhouse list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE Observability is installed.

The output should look like this:

Setting up port-forward to suse-observability-clickhouse-backup:7171 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Listing Clickhouse backups...
NAME                             CREATED              SIZE
incremental_2025-11-28T09-45-00  2025-11-28 09:45:03  65MiB
incremental_2025-11-28T08-45-00  2025-11-28 08:45:03  223MiB
full_2025-11-28T00-45-00         2025-11-28 00:45:03  3GiB
incremental_2025-11-27T23-45-00  2025-11-27 23:45:03  118MiB

Backup names starting with full_ are full backups, while names starting with incremental_ are incremental backups.

=== Restore a ClickHouse backup

The restore process automatically scales down workloads that produce data (like OpenTelemetry exporters) to prevent data loss during restoration.

To restore a ClickHouse backup, use one of the following approaches:

==== Restore a specific backup

sts-backup clickhouse restore --namespace <NAMESPACE> --snapshot <BACKUP_NAME>

==== Restore the latest backup

sts-backup clickhouse restore --namespace <NAMESPACE> --latest

==== Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

  • --background - Run restore in background without waiting for completion

==== Background restore

When using --background, the restore runs asynchronously. After starting the restore, use the following command to check status and finalize:

sts-backup clickhouse check-and-finalize --operation-id <OPERATION_ID> --wait --namespace <NAMESPACE>

The check-and-finalize command:

  • Checks the restore operation status

  • With --wait flag, waits for operation completion

  • Executes post-restore SQL commands

  • Automatically scales up StatefulSets that were scaled down during restore

  • Cleans up resources after completion

If a restore running without --background is interrupted (e.g., by Ctrl+C), you must run check-and-finalize with the operation ID to scale up StatefulSets and clean up resources.

== Telemetry data (Elasticsearch)

=== List Elasticsearch snapshots

To list the Elasticsearch snapshots, execute the following command:

sts-backup elasticsearch list --namespace <NAMESPACE>

Replace <NAMESPACE> with the namespace where SUSE Observability is installed.

The output should look like this:

Setting up port-forward to suse-observability-elasticsearch-master-headless:9200 in namespace <NAMESPACE>...
✓ Port-forward established successfully
Fetching snapshots from repository 'sts-backup'...
SNAPSHOT                                         STATE    START TIME                DURATION (ms)  FAILURES
sts-backup-20251128-1135-dpkj2dqrszo6cscpgfhrhg  SUCCESS  2025-11-28T11:35:10.967Z  329158         0

The timestamp when the snapshot was taken is part of the snapshot name.

=== Restore an Elasticsearch snapshot

The restore process automatically deletes all STS indices (matching the pattern sts*) before restoring the snapshot. This includes rolling over datastreams to ensure clean restoration.

To restore an Elasticsearch snapshot, use one of the following approaches:

==== Restore a specific snapshot

sts-backup elasticsearch restore --namespace <NAMESPACE> --snapshot <SNAPSHOT_NAME>

==== Restore the latest snapshot

sts-backup elasticsearch restore --namespace <NAMESPACE> --latest

==== Common flags

  • --yes or -y - Skip confirmation prompt (useful for automation)

  • --background - Run restore in background without waiting for completion

==== Background restore

When using --background, the restore runs asynchronously. After starting the restore, use the following command to check status and finalize:

sts-backup elasticsearch check-and-finalize --operation-id <OPERATION_ID> --wait --namespace <NAMESPACE>

The check-and-finalize command:

  • Checks the restore operation status

  • With --wait flag, waits for operation completion

  • Automatically scales up deployments that were scaled down during restore

  • Cleans up resources after completion

If a restore running without --background is interrupted (e.g., by Ctrl+C), you must run check-and-finalize with the operation ID (snapshot name) to scale up deployments and clean up resources.