मुख्य विषयवस्तु में जाएं

Run dbt Core on Spark (Kubernetes)

This guide explains how to set up dbt Core के साथ अपाचे स्पार्क running on a Kubernetes cluster. Using Ilum as the execution engine, you can run scalable data transformation pipelines directly on your data lake.

You have two primary ways to connect dbt to Spark on Ilum:

Thrift Server vs. Spark Connect​

लक्षणMethod 1: Spark Thrift Server (Legacy)Method 2: Spark Connect (Modern)
ProtocolJDBC/ODBC (via HiveDriver)gRPC (via Spark Connect)
Connection Typeविधि: बचतविधि: सत्र
स्‍थापत्‍यशैलीRequires a dedicated Thrift Server podConnects directly to Spark Driver
PerformanceHigher latency (row-based serialization)High performance (Arrow-based)
के लिए सबसे अच्छाBI Tools (Tableau, PowerBI), Legacy appsData Engineering, Python/dbt pipelines
New to Spark Connect?

For a deep dive into the architecture, check out our Spark Connect on Kubernetes Guide.

आवश्यकताएँ​

Before starting, ensure your development environment is ready:

  • कुबेरनेट्स क्लस्टर: You need a running K8s cluster (GKE, EKS, AKS, or Minikube).
  • Tools:
  • Knowledge: Basic understanding of dbt projects and Spark concepts.

How to Configure dbt with Spark on Kubernetes​

Choose your preferred connection method:

Step 1: Deploy Spark Thrift Server​

Deploy Ilum with the SQL module (acting as a scalable Thrift server) and Hive Metastore enabled:

Helm Install
helm repo आगे कहना ilum https://charts.ilum.cloud
helm install इलम इलम / \
  --अस्त हो ilum-hive-metastore.enabled=सच्चा \
  --अस्त हो ilum-core.metastore.enabled=सच्चा \
  --अस्त हो ilum-core.metastore.type=छत्ता \
  --अस्त हो ilum-sql.enabled=सच्चा \
  --अस्त हो ilum-core.sql.enabled=सच्चा

Step 2: Connect to the Thrift Service​

1. Identify the service:

Get Service
kubectl get सेवा

इसके नाम में "sql-thrift-binary" के साथ सेवा खोजें।

2. Port-forward:

Port Forward
kubectl port-forward svc/ilum-sql-thrift-binary 10009:10009

यह थ्रिफ्ट सर्वर पर उपलब्ध बनाता है लोकलहोस्ट: 10009.

3. Test with Beeline (optional):

beeline -u "jdbc:hive2://localhost:10009/default"

चलाना:

प्रदर्शन तालिकाओं;

एक खाली सूची या मौजूदा तालिकाओं की अपेक्षा करें।

Configuring and Running dbt​

1. Clean Environment (if needed):

Uninstall Conflict
pip uninstall dbt-spark pyspark -y

2. Install dbt and dependencies:

Install dbt-spark
pip install पिस्पार्क==3.5.8
pip install डीबीटी-कोर
pip install "dbt-spark[PyHive,session]"
pip install --upgrade मितव्‍ययिता

3. Verify installation:

Verify dbt
डीबीटी --version

Create dbt Project​

1. Initialize a dbt project:

Init Project
डीबीटी इनिट ilum_dbt_project

2. Answer the setup prompts:

Setup Prompts
Which database? 1 (उत्तेजक गुण)
host: localhost
Desired authentication method: 3 (मितव्‍ययिता)
port: 10009
schema: default
threads: 1

This creates the ilum_dbt_project directory and a profiles.yml file in ~/.dbt/.

Configure dbt for Ilum​

संपादन करना ~/.dbt/profiles.yml to include both Thrift and Spark Connect targets:

~/.dbt/profiles.yml
ilum_dbt_project:
  लक्ष्य: मितव्‍ययिता  # Default target
  आउटपुट:
    मितव्‍ययिता:
      प्रकार: उत्तेजक गुण
      विधि: मितव्‍ययिता
      मेज़बान: लोकलहोस्ट
      बंदरगाह: 10009
      रूपरेखा: चूक
      धागे: 1
      connect_retries: 5
      connect_timeout: 60
      connect_args:
        यूआरएल: "jdbc:hive2://localhost:10009/default; ट्रांसपोर्ट मोड = बाइनरी; हाइव.सर्वर2.ट्रांसपोर्ट.मोड = बाइनरी"
        चालक: "org.apache.hive.jdbc.HiveDriver"
        प्रमाणन: "कोई नहीं"
    
    spark_connect:
      प्रकार: उत्तेजक गुण
      विधि: अधिवेशन
      मेज़बान: लोकलहोस्ट
      बंदरगाह: 15002
      रूपरेखा: चूक
      धागे: 1

Switch between targets:

Run dbt
# Use Thrift (default)
dbt run

# Use Spark Connect
dbt run --target spark_connect

# Or set default in dbt_project.yml
# target: spark_connect

  • परीक्षण कनेक्शन:

    Debug dbt
    cd ilum_dbt_project
    डीबीटी डीबग

    सुनिश्चित करें कि कोई त्रुटि दिखाई नहीं देती है, जो थ्रिफ्ट सर्वर से सफल कनेक्शन का संकेत देती है।

Create a Model to Write Data​

  • मॉडल बनाएँ:

  • मॉडल/sample_data.sql

    मॉडल/sample_data.sql
    {{ कॉन्फ़िग(भौतिकीकृत='टेबल') }}

    चुनना 
      परिचय,
      नाम
    से (
      मान 
        (1, 'ऐलिस'),
        (2, 'बॉब')
    ) जैसा टन(परिचय, नाम)

  • मॉडल चलाएं:

    Run sample_data
    dbt run --select sample_data

Create a Model to Read Data​

  • मॉडल बनाएँ:

  • मॉडल/read_data.sql

    मॉडल/read_data.sql
    {{ कॉन्फ़िग(भौतिकीकृत='टेबल') }}

    चुनना 
      परिचय,
      नाम,
      लंबाई(नाम) जैसा name_length
    से {{ रेफरी('sample_data') }}

  • मॉडल चलाएं:

    Run read_data
    dbt run --select read_data

Verify Results​

1. Monitor Job in Ilum UI:

  • Access the Ilum UI (URL provided in your Ilum setup, e.g. port-forward)

  • Navigate to the Jobs section

  • Look for the job named ilum-sql-spark-engine

  • Check job status, logs, and execution details to confirm successful processing

    2. Query with Beeline:

    beeline -u "jdbc:hive2://localhost:10009/default"

    3. Run query:

    चुनना * से चूक.read_data;

    Expected output:

    +----+-----+------------+
    | आईडी | नाम| name_length|
    +----+-----+------------+
    | 1 | ऐलिस| 5 |
    | 2 | बॉब | 3 |
    +----+-----+------------+

Troubleshooting dbt-spark Connections​

Common issues when connecting dbt to Spark on Kubernetes:

Error: "ThriftTransportException: Could not connect to localhost:10009"

Cause: The port forwarding tunnel is down or the Thrift Server pod is not running. विलयन:

  1. Check if the Thrift pod is running: kubectl get pods -l app.kubernetes.io/name=ilum-sql
  2. Restart port-forwarding: kubectl पोर्ट-फॉरवर्ड SVC/ilum-sql-thrift-binary 10009:10009
Error: "grpc._channel._InactiveRpcError: failed to connect to all addresses"

Cause: Your local dbt client cannot reach the Spark Connect gRPC port (15002). विलयन:

  • Ensure you have port-forwarded the Driver Pod, not the Service (unless using NodePort).
  • Verify you are using विधि: सत्र में profiles.yml.
Error: "AnalysisException: Table or view not found"

Cause: Hive Metastore connectivity issue. विलयन:

  • Ensure ilum-core.metastore.enabled=true was set during Helm install.
  • Check if the schema (database) exists in Spark: spark.sql("SHOW DATABASES").show()

वाद्यवृंदकरण​

For production orchestration using Apache Airflow, see the dedicated guide: Orchestrate dbt with Airflow

Last updated on