मुख्य विषयवस्तु में जाएं

Back Up and Restore Object Storage

विहंगावलोकन​

इलम does not run automated backups of object storage by default. The bundled providers persist their data on PersistentVolumeClaims managed by the underlying CSI driver, and the chart preserves those PVCs across हेल्म अपग्रेड और helm rollback. Disaster recovery beyond PVC retention is operator-driven.

This page describes three layers of data protection, ordered from infrastructure-level to application-level:

  • PV snapshots via the Kubernetes VolumeSnapshot API. Point-in-time copies of the underlying volume; CSI-driver-dependent.
  • Off-cluster mc mirror copies to an external S3 backend. Logical-object-level mirrors that survive cluster loss.
  • Application-level table snapshots in Iceberg, Delta, and DuckLake. Time-travel semantics inside the table format; no infrastructure involvement.

The recipes below cover the active provider's bucket data. For recovering from misconfiguration without data loss, refer to Troubleshoot Object Storage.

Backup layers compared​

LayerRPORTOCoverageCluster loss survives?
PV snapshotSnapshot interval (typically hourly)Minutes (restore + provider restart)All buckets, including metadata indicesNo (snapshot lives on the same storage backend)
Off-cluster mc mirrorMirror interval (typically hourly)Minutes (re-mirror to new cluster)All buckets at the S3 layerहाँ
Iceberg / Delta snapshotPer-commitSeconds (VERSION AS OF)One table at a timeOnly if the table's underlying objects survive

For production deployments, combine an off-cluster mc mirror job for disaster recovery with the table-format snapshots that Iceberg, Delta, and DuckLake already provide.

Layer 1: PV snapshots via the VolumeSnapshot एपीआई​

The Kubernetes VolumeSnapshot API has been generally available since Kubernetes 1.20 (December 2020). Snapshot support is provided by the CSI driver and must be advertised by the driver itself; not every CSI driver implements snapshotting. For the upstream reference, see Volume Snapshots.

Verify CSI snapshot support​

kubectl get csidriver -ओ custom-columns=NAME:.metadata.name,SNAP:.spec.attachRequired
kubectl get volumesnapshotclass

If volumesnapshotclass returns nothing, install the external-snapshotter controller and a VolumeSnapshotClass for the cluster's CSI driver before proceeding.

एक बनाएँ VolumeSnapshotClass (one-time)​

apiVersion: snapshot.storage.k8s.io/v1
प्रकार: VolumeSnapshotClass
मेटाडेटा:
  नाम: इलम-objectstorage-स्नैपशॉट
चालक: < आपका-csi-चालक>
deletionPolicy: Retain

deletionPolicy: Retain ensures snapshots survive the deletion of the VolumeSnapshot संसाधन। मिटाना is appropriate when snapshots should be removed automatically with their parent resource.

Snapshot the active provider's PVC​

# RustFS: the chart names the PVC after the StatefulSet.
PVC=$(कुबेक्टल -n ilum get pvc -l app.kubernetes.io/name=rustfs \
  -ओ jsonpath='{.items[0].metadata.name}')

cat <<ईओएफ | कुबेक्टल -n ilum apply -स्‍त्री-विषयक -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
मेटाडेटा:
  name: ilum-objectstorage-$(खजूर +%Y%m%d-%H%M%S)
कल्पना:
  volumeSnapshotClassName: ilum-objectstorage-snapshots
  source:
    persistentVolumeClaimName: $PVC
ईओएफ

कुबेक्टल -n ilum get volumesnapshot

Restore from a snapshot​

Create a new PersistentVolumeClaim whose dataSource references the VolumeSnapshot:

apiVersion: वी 1
प्रकार: PersistentVolumeClaim
मेटाडेटा:
  नाम: rustfs-restore
  Namespace: इलम
कल्पना:
  storageClassName: < आपका-storage-कक्षा>
  dataSource:
    नाम: इलम-objectstorage->
    प्रकार: VolumeSnapshot
    एपीआई समूह: snapshot.storage.k8s.io
  accessModes: [ReadWriteOnce]
  संसाधन:
    requests:
      storage: -जैसा-source>

To swap the restored PVC into the running provider, scale the provider's StatefulSet to zero, repoint the active PVC at the restored volume, and scale back. The exact procedure depends on the CSI driver's reconciliation behavior; refer to the driver's documentation.

Limitations​

  • VolumeSnapshot है CSI-driver-dependent. Not every cloud provider implements snapshot support in their CSI driver, and on-prem drivers vary in maturity. Verify before relying on this layer.
  • Snapshots typically live on the same underlying storage backend. A failure of the backend (region outage, hardware loss) takes the snapshots with it. Layer this with off-cluster mirrors for true DR.
  • ReadWriteOnce PVCs (the bundled-provider default) can be snapshotted without quiescing the provider, but the resulting snapshot is crash-consistent rather than application-consistent. For application-consistent snapshots, quiesce writes through the alias before triggering the snapshot.

Layer 2: Off-cluster mc mirror to external S3​

mc mirror from the active provider to an external S3 backend produces a logical-object-level copy that survives full cluster loss. The typical pattern is a CronJob that mirrors every default bucket once per hour.

Provision an external destination​

Provision an external S3 backend (AWS S3, Wasabi, Backblaze B2, or any S3-compatible service) with a bucket per source bucket. The destination bucket names should match objectStorage.defaultBuckets. For provider-specific endpoint shapes, refer to Provider Reference: External S3.

Store the destination credentials in a separate Secret so they do not conflict with ilum-objectstorage-credentials:

कुबेक्टल -n ilum create secret generic ilum-backup-credentials \
  --from-literal=access-key=<external-access-key> \
  --from-literal=secret-key=<external-secret-key>

Run the mirror as a CronJob​

apiVersion: batch/v1
प्रकार: CronJob
मेटाडेटा:
  नाम: इलम-objectstorage-backup
  Namespace: इलम
कल्पना:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    कल्पना:
      backoffLimit: 1
      ttlSecondsAfterFinished: 86400
      साँचा:
        कल्पना:
          restartPolicy: OnFailure
          कंटेनर:
            - नाम: mc
              प्रतिबिंब: minio/mc:RELEASE.2025-04-16T18-13-26Z
              envFrom:
                - secretRef:
                    नाम: इलम-objectstorage-प्रत्यय-पत्र
                - secretRef:
                    नाम: इलम-backup-प्रत्यय-पत्र
              command: [sh, -के आसपास]
              आर्ग्स:
                - |
                  set -eu
                  mc alias set src http://ilum-objectstorage:9000 "$access-key" "$secret-key"
                  mc alias set dst https:// "$access-key" "$secret-key"
                  for bucket in ilum-files ilum-data ilum-tables ilum-mlflow ilum-kestra ilum-ducklake ilum-langfuse; do
                    mc mb --ignore-existing dst/$bucket
                    mc mirror --preserve --remove src/$bucket dst/$bucket
                  समाज-सम्मत

वही --remove flag mirrors deletions from source to destination. Omit it when an append-only archive is preferred. The bundled minio/mc image tag is pinned to the same release used by the in-cluster migration नौकरी, ensuring behavior parity.

Restore from the external mirror​

The restore procedure is mc mirror in reverse:

  1. Stand up a clean इलम install with the active provider enabled but the bucket-init नौकरी disabled (to avoid overwriting the restored objects).

  2. Configure the same external destination as a source alias.

  3. Mirror back to the in-cluster provider:

    के लिए bucket में ilum-files ilum-data ilum-tables ilum-mlflow ilum-kestra ilum-ducklake ilum-langfuse; करना
      mc mirror --preserve src/$bucket dst/$bucket
    समाज-सम्मत

  4. Re-enable the bundled consumers. The shared Secret and the ilum-objectstorage alias point them at the restored data automatically.

Layer 3: Application-level table snapshots​

For Iceberg and Delta tables managed by इलम, the table format itself provides point-in-time snapshots through its commit history. These cover one table at a time, not full buckets, but offer fine-grained recovery without infrastructure involvement.

हिमशैल​

-- List snapshots.
चुनना snapshot_id, committed_at, operation
से हिमशैल.<कैटलॉग>.<सारणी>.स्नैपशॉट
ORDER द्वारा committed_at DESC;

-- Time-travel read.
चुनना *
से हिमशैल.<कैटलॉग>.<सारणी>
विवरण जैसा का <snapshot_id>;

-- Roll the table back to a snapshot.
पुकार हिमशैल.प्रणाली.rollback_to_snapshot('.', <snapshot_id>);
डेल्टा झील​
वर्णन करना इतिहास <कैटलॉग>.<सारणी>;
चुनना * से <कैटलॉग>.<सारणी> विवरण जैसा का <विवरण>;
पुनर्स्थापित करना सारणी <कैटलॉग>.<सारणी> तक विवरण जैसा का <विवरण>;
DuckLake​
DuckLake snapshots are recorded in the DuckLake catalog. Refer to the
DuckLake documentation for the time-travel and rollback syntax that
matches the catalog version in use.
The retention window of these snapshots is governed by the table
format's expiration policy (Iceberg's expire_snapshots procedure,
Delta's निर्वात). Tune the retention to match the operator's recovery
objectives before relying on this layer for DR.
What is not backed up by default​
    

  • PVC snapshots are not taken automatically. The chart does not
create VolumeSnapshot resources; the operator must schedule them.
    • 

  • Off-cluster mirrors are not configured by default. वही
CronJob recipe above is operator-installed.
    • 

  • Bucket policies, lifecycle rules, and IAM users that the operator
configures directly against the provider are not part of any layer
above. Capture them separately (typically with a GitOps pipeline).
    • 

  • Hydra OIDC client registrations and Kubernetes Secrets are not
covered by object-storage backups. Use cluster-wide tooling
(Velero, Kasten K10, similar) for those.
    • 

Reference​
Last updated on