मुख्य विषयवस्तु में जाएं

DuckLake

DuckLake is the main storage solution for the DuckDB database engine used by Ilum.

Since embedded DuckDB by default supports only in-memory mode or local disk storage (single user), DuckLake provides a multi-user, concurrent storage layer that enables shared access to DuckDB datasets across the platform.

When to use DuckLake​

Use DuckLake when:

  • Multiple users or jobs need concurrent access to the same DuckDB tables
  • You need persistent table storage with metadata management
  • Your workload requires time travel or schema evolution

Skip DuckLake when:

  • You have single-user, ad-hoc analytics workflows
  • You’re prototyping and don’t need persistence
  • Direct Parquet file access is enough

सुविधाऐं​

  • Shared dataset storage using object storage (S3/MinIO/GCS)
  • ACID-like durability through snapshot isolation and cross-table transactions
  • Time travel – query previous versions of your data
  • स्कीमा विकास – add/modify columns without breaking existing queries
  • अनुकूलता with standard DuckDB file formats and Parquet

For more on DuckLake’s capabilities, see the official DuckLake documentation.

Concurrency Model​

DuckLake provides snapshot isolation for concurrent workloads:

  • Readers never block writers, and writers never block readers
  • Each query sees a consistent snapshot of data at transaction start
  • Cross-table transactions maintain atomicity across related operations

This differs from strict serializable ACID – it’s optimized for analytic workloads where high read concurrency is prioritized over write serialization.

संरूपण​

नोट

DuckLake is attached by default to all DuckDB instances in Ilum when enabled. Tables created in SQL Viewer will automatically use DuckLake.

Configure DuckLake via the Helm chart:

इलम कोर:
  एसक्यूएल:
    duckdb:
      ducklake:
        सक्षम: सच्चा                          # Set false to disable DuckLake entirely
        location: एस 3://ilum-ducklake/          # Root path for all DuckLake data
        पोस्टग्रेज़:                              # Metadata storage (required)
          मेज़बान: "ilum-postgresql-hl"           # PostgreSQL service hostname
          बंदरगाह: 5432                           # PostgreSQL port
          database: ducklake                   # Database name (created automatically if missing)
          उपभोक्ता: इलम                           # Database user
          पासवर्ड: "चेंजमीप्लीज"           # Database password
        एस 3:                                   # Data storage backend
          समापन बिंदु: इलम-मीनो:9000           # S3 endpoint (MinIO, AWS S3, GCS, etc.)
          region: us-पूर्व-1                   # S3 क्षेत्र
          keyId: मिनीएडमिन                   # Access key ID
          रहस्य: मिनीएडमिन                  # Secret access key
          urlStyle: पथ                      # Path-style access (use 'virtualHost' for AWS S3)
          ssl: गलत                          # Enable TLS for S3 connections
Disabling DuckLake

Setting enabled: false causes DuckDB to fall back to in-memory tables only. Use this only for single-user scenarios where persistence isn’t required.

उपयोग उदाहरण​

Creating Tables​

When enabled, DuckLake is automatically selected as the default storage backend for all DuckDB instances. This means any created tables in the default catalog will be stored in DuckLake.

-- Create a table (automatically stored in DuckLake)
बनाना सारणी events (
    event_id बिगिंट,
    event_time टाइमस्टैम्प,
    user_id बिगिंट,
    payload JSON
);

Limitations​

  • DuckDB-only: DuckLake catalogs cannot be accessed from Spark, Trino, or other engines
  • No branching/tagging: Unlike Nessie or Iceberg, DuckLake doesn't support Git-like version control
  • Concurrency bound by metadata DB: High-write concurrency may impact PostgreSQL performance
Last updated on