मुख्य विषयवस्तु में जाएं

How to Write Interactive Spark Jobs in Python (IlumJob)

This guide teaches you how to develop interactive Spark jobs in Python using the इलम जॉब interface. You'll learn how to structure your code, pass parameters at execution time, and leverage the benefits of this approach for production workloads on Kubernetes.

What is the इलम जॉब Interface?​

वही इलम जॉब interface is a Python base class used to create reusable, parameterized Spark jobs that run on interactive Ilum services. Unlike traditional स्पार्क-सबमिट scripts, इलम जॉब allows you to:

  • Receive configuration at runtime: Parameters are passed as a dictionary, allowing the same job to handle different inputs without code changes.
  • Return structured results: वही चलाना method returns a string, making it easy to extract and display results.
  • Run on-demand: Jobs can be triggered via the UI, REST API, or CI/CD pipelines.
Basic Structure
से इलम.एपीआई आयात इलम जॉब

कक्षा MySparkJob(इलम जॉब):
    डीईएफ़ चलाना(स्वयं, उत्तेजक गुण, कॉन्फिग) -> str:
        # Your Spark logic here
        लौटना "Job completed successfully"

Structure of an Interactive Spark Job​

Every interactive job consists of three essential parts:

  1. Import the interface: from ilum.api import IlumJob
  2. Define a class: Create a class that inherits from इलम जॉब.
  3. Implement चलाना: Write your Spark logic inside the run(self, spark, config) विधि।
Parameterप्रकारया क़िस्‍म
उत्तेजक गुणस्पार्कसेशनPre-initialized Spark session, ready to use.
कॉन्फिगdictA dictionary containing parameters passed at execution time.
ReturnstrA string result that will be displayed in the UI or returned via API.

How to Pass Parameters to Spark Jobs​

Parameters are passed as a JSON object when executing the job. Inside your चलाना method, you access them using standard dictionary methods.

Example: Table Inspector​

This example demonstrates reading database और सारणी parameters to inspect a Hive table.

table_inspector.py
से इलम.एपीआई आयात इलम जॉब
से पिस्पार्क.एसक्यूएल.functions आयात col, sum जैसा spark_sum

कक्षा TableInspector(इलम जॉब):
    डीईएफ़ चलाना(स्वयं, उत्तेजक गुण, कॉन्फिग) -> str:
        # Read required parameters
        table_name = कॉन्फिग.मिलना('टेबल')
        database_name = कॉन्फिग.मिलना('database')  # Optional

        अगर नहीं table_name:
            raise ValueError("Config must provide a 'table' key")

        # Set database if provided
        अगर database_name:
            उत्तेजक गुण.कैटलॉग.setCurrentDatabase(database_name)

        # Check if table exists
        अगर table_name नहीं में [टन.नाम के लिए टन में उत्तेजक गुण.कैटलॉग.listTables()]:
            raise ValueError(f"Table '{table_name}' not found in catalog")

        लोमो = उत्तेजक गुण.सारणी(table_name)

        # Build report
        report = [
            f"=== Table: {table_name} ===",
            f"Total rows: {लोमो.गिनना()}",
            f"Total columns: {len(लोमो.columns)}",
            "",
            "Schema:",
        ]
        के लिए field में लोमो.रूपरेखा.fields:
            report.append(f"  {field.नाम}: {field.dataType}")

        report.append("")
        report.append("Sample (5 rows):")
        के लिए हो-हल्‍ला में लोमो.take(5):
            report.append(str(हो-हल्‍ला.asDict()))

        # Null counts
        report.append("")
        report.append("Null counts:")
        null_df = लोमो.select([spark_sum(col(के आसपास).isNull().cast("int")).alias(के आसपास) के लिए के आसपास में लोमो.columns])
        के लिए के आसपास, v में null_df.collect()[0].asDict().items():
            report.append(f"  {के आसपास}: {v}")

        लौटना "\n".join(report)

Execution Parameters (JSON)​

When executing via UI or API, provide parameters like this:

{
  "database": "ilum_example_product_sales",
  "table": "products"
}
Before You Start

To run an interactive job, you first need to create and deploy a Job-type Service in Ilum. This service provides the Spark environment where your jobs execute.

When creating the service:

  • प्रकार: Select नौकरी
  • भाषा: Select अजगर
  • Py Files: Upload your job file (e.g., table_inspector.py)

👉 Learn how to deploy a Job Service — step-by-step guide with UI screenshots and configuration options.

Executing Jobs​

You can execute interactive jobs in three ways:

  1. यहाँ जाओ सेवाएँ → Select your Job service
  2. में अमल section:
    • कक्षा: table_inspector.TableInspector
    • Parameters: {"database": "sales", "table": "orders"}
  3. क्लिक करना अमल

The result string is displayed immediately in the UI.

Benefits of the इलम जॉब Approach​

Benefitया क़िस्‍म
ReusabilityWrite once, run many times with different parameters.
No Cold StartsInteractive services keep Spark warm, so subsequent executions are instant.
ParameterizationPass configuration at runtime—no need to hardcode values.
अवलोकन योग्यताResults are captured and visible in the UI/API for easy debugging.
API-DrivenExecute jobs programmatically from orchestrators, CI/CD, or external systems.
संस्करण नियंत्रणStore job code in Git and deploy via pipelines.

Interactive Jobs vs. Batch Jobs (Spark Submit)​

लक्षणInteractive Jobs (इलम जॉब)Batch Jobs (स्पार्क-सबमिट)
Startup TimeInstant (uses warm executors)Slow (provisions new pods)
ContextShared Spark ContextIsolated Spark Context
उपयोग का मामलाAd-hoc queries, API backends, quick reportsLong-running ETL, heavy processing
परिणामReturns string result to API/UILogs to driver stdout/file
संसाधनShared within the serviceDedicated per job

सबसे सही तरीके​

1. Validate Input Parameters​

Always validate required parameters and provide helpful error messages.

Validate Parameters
डीईएफ़ चलाना(स्वयं, उत्तेजक गुण, कॉन्फिग) -> str:
    required_keys = ['टेबल', 'output_path']
    के लिए अत्‍यंत महत्वपूर्ण में required_keys:
        अगर अत्‍यंत महत्वपूर्ण नहीं में कॉन्फिग:
            raise ValueError(f"Missing required parameter: '{अत्‍यंत महत्वपूर्ण}'")

2. Use Default Values​

For optional parameters, use config.get('key', default_value).

Use Default Values
batch_size = इंट(कॉन्फिग.मिलना('batch_size', 1000))

3. Structure Your Output​

Return a well-formatted string for readability in the UI.

Structure Output
lines = ["=== Job Summary ==="]
lines.append(f"Processed: {गिनना} records")
lines.append(f"Duration: {elapsed_time}s")
लौटना "\n".join(lines)

4. Handle Errors Gracefully​

Wrap risky operations in try/except and return meaningful messages.

Handle Errors
try:
    लोमो.लिखना.saveAsTable(output_table)
    लौटना f"Successfully wrote to {output_table}"
except Exception जैसा ई:
    लौटना f"Error writing table: {str(ई)}"

Complete Example: Transaction Report Generator​

This job generates a transaction summary report based on the transaction_anomaly_d.transactions सारणी।

transaction_report.py
से इलम.एपीआई आयात इलम जॉब
से पिस्पार्क.एसक्यूएल.functions आयात sum जैसा spark_sum, गिनना, col

कक्षा TransactionReportGenerator(इलम जॉब):
    डीईएफ़ चलाना(स्वयं, उत्तेजक गुण, कॉन्फिग) -> str:
        # Parameters
        merchant_filter = कॉन्फिग.मिलना('merchant')  # Optional filter

        # Load data from the default Ilum transactions table
        लोमो = उत्तेजक गुण.सारणी("transaction_anomaly_detection.transactions")

        अगर merchant_filter:
            लोमो = लोमो.छानना(col("Merchant") == merchant_filter)

        # Aggregate by TransactionType
        summary = लोमो.groupBy("TransactionType").agg(
            गिनना("TransactionID").alias("transaction_count"),
            spark_sum("Amount").alias("total_amount")
        ).collect()

        # Build report
        report = [
            f"=== Transaction Report ===",
            f"Merchant Filter: {merchant_filter नहीं तो 'All'}",
            "",
            "Summary by Transaction Type:",
        ]
        के लिए हो-हल्‍ला में summary:
            report.append(f"  {हो-हल्‍ला['TransactionType']}: {हो-हल्‍ला['transaction_count']} txns, ${हो-हल्‍ला['total_amount']:,.2f}")

        लौटना "\n".join(report)

Execute with:

Execute with Payload
{
  "merchant": "AcmeCorp"
}

अगले कदम​

अक्सर पूछे जाने वाले प्रश्न​

Can I use Scala for interactive jobs?

Yes. Currently, the इलम जॉब interface is primarily documented for अजगर. Check the Interactive Job Service documentation for language support details.

How do I debug an interactive job?

Since interactive jobs run on a remote cluster, you can't use a local debugger directly. Instead:

  1. प्रयोग print() statements or a logger, which will appear in the driver logs.
  2. Return error messages as part of the string result in your try/except blocks.
  3. Check the स्पार्क यूआई for the specific job execution to analyze tasks and stages.
What happens if my job fails?

If your code raises an unhandled exception, the execution will fail, and the error trace will be returned in the API response. It is best practice to wrap your logic in a try/except block to return a user-friendly error message.

Last updated on