Run Spark Jobs via REST API

विहंगावलोकन

Ilum provides a robust बाकी एपीआई that allows you to manage, submit, and execute Apache Spark jobs programmatically. This capability is essential for organizations running कुबेरनेट्स पर चिंगारी who need to automate their data workflows.

Using the API is particularly effective for:

• CI/CD Integration: Seamlessly trigger Spark jobs from GitLab CI, Jenkins, GitHub Actions, or Airflow.
• Custom Orchestration: Build your own data platforms or internal tools on top of Ilum.
• मशीनी परिचालन: Replace manual स्पार्क-सबमिट CLI commands with reliable, code-driven API calls.

REST API vs. Spark CLI

लक्षण	बाकी एपीआई	Spark CLI (स्पार्क-सबमिट)
Primary Use Case	Automation, CI/CD, Web Apps	Ad-hoc testing, Local development
Client Requirement	छल्‍ला or HTTP Client	Spark Binaries & Java installed
Feedback Loop	JSON Response (Job ID)	Console Logs (Streamed)
Firewall Friendly	Yes (Single HTTP Port)	No (Requires Random Ports)

In this guide, you will learn how to:

1. Submit a Spark job using the मल्टीपार्ट/फॉर्म-डेटा endpoint.
2. Monitor the job's status via the API.

आवश्यकताएँ

To follow this example, you will need the छल्‍ला command-line tool and a sample Spark JAR file.

• Download Example JAR (choose one):

Spark 4 (default)

• Spark 4 / Scala 2.13: spark-examples_2.13-4.1.1.jar

Spark 3

• Spark 3 / Scala 2.12: spark-examples_2.12-3.5.8.jar

Accessing the API

The Ilum Core API is exposed by default on port 9888. Depending on your environment, you can access it using one of the following methods:

API Base URL

In the examples below, replace http://localhost:9888 with your actual Ilum Core address.

1. Port Forwarding (Development)

If you are running the API on your local machine using a Kubernetes cluster (like Minikube or MicroK8s), you can use Kubectl पोर्ट-फॉरवर्ड to access it locally:

Port Forward

kubectl port-forward svc/ilum-core 9888:9888

The API will then be available at http://localhost:9888/api/v1.

2. NodePort

If your Ilum installation is configured with a नोडपोर्ट service type, you can access it via any Kubernetes node IP:

Get Nodes & Services

# Get the node IP
kubectl get nodes -ओ wide

# Get the assigned NodePort
kubectl get svc ilum-core

Access the API at http://<NODE_IP>:<NODE_PORT>/api/v1.

3. Ingress (Production)

For production environments, use an Ingress controller to expose the API. This allows you to use a custom domain and SSL/TLS encryption.

Example Ingress Path

- पथ: /api/v1/(.*)
  pathType: ImplementationSpecific
  backend:
    सेवा:
      नाम: इलम-अंतर्भाग
      बंदरगाह:
        number: 9888

Access the API at https://your-domain.com/api/v1.

Which Method Should I Use?

Method	के लिए सबसे अच्छा	Requirement
Port Forwarding	Local development, one-off tests	कुबेक्टल access to the cluster
नोडपोर्ट	Internal lab environments, simple setups	Access to Kubernetes Node IPs
Ingress	Production, Team collaboration, CI/CD	Ingress Controller (Nginx, Traefik, etc.)

Submit Apache Spark Jobs Programmatically

To submit a new Spark application, use the POST /api/v1/job/submit endpoint. This endpoint accepts मल्टीपार्ट/फॉर्म-डेटा requests, allowing you to upload your application JAR or Python script along with the job configuration. This method is the programmatic equivalent of स्पार्क-सबमिट.

Example: Submitting MiniReadWriteTest

The following छल्‍ला command submits the मिनीरीराइटटेस्ट example job (from the downloaded JAR). This job writes a file and then reads it back to verify the setup.

Spark 4 (default)

Submit Job (Spark 4)

छल्‍ला -X के बाद "http://localhost:9888/api/v1/job/submit" \
  -F "name=MiniReadWriteTest" \
  -F "clusterName=default" \
  -F "language=SCALA" \
  -F "jobClass=org.apache.spark.examples.MiniReadWriteTest" \
  -F "jobConfig=spark.executor.instances=2" \
  -F "args=/opt/spark/examples/src/main/resources/kv1.txt" \
  -F "jars=@spark-examples_2.13-4.1.2.jar"

Spark 3

Submit Job (Spark 3)

छल्‍ला -X के बाद "http://localhost:9888/api/v1/job/submit" \
  -F "name=MiniReadWriteTest" \
  -F "clusterName=default" \
  -F "language=SCALA" \
  -F "jobClass=org.apache.spark.examples.MiniReadWriteTest" \
  -F "jobConfig=spark.executor.instances=2" \
  -F "args=/opt/spark/examples/src/main/resources/kv1.txt" \
  -F "jars=@spark-examples_2.12-3.5.8.jar"

Parameter Reference

Parameter	प्रकार	या क़िस्‍म	Required	उदाहरण
नाम	तार	A unique identifier for your job.	हाँ	मिनीरीराइटटेस्ट
क्लस्टरनाम	तार	The name of the Kubernetes cluster registered in Ilum.	हाँ	चूक
भाषा	तार	The programming language of the job (SCALA नहीं तो PYTHON).	हाँ	SCALA
jobClass	तार	स्काला: The fully qualified main class name. अजगर: The script filename (without extension).	हाँ	org.apache.spark.examples.MiniReadWriteTest
jobConfig	तार	Semicolon-separated List of Spark configuration properties in key=value format.	नहीं	spark.executor.instances=2
आर्ग्स	तार	Semicolon-separated list of arguments to pass to the job's main method.	नहीं	/path/to/input.txt
जार	रेती	The application JAR file. Use the @ prefix in curl to upload the file.	Yes (for Scala)	@app.jar
pyFiles	रेती	The main Python script or ZIP package.	Yes (for Python)	@job.py

Full API Specification

For a complete list of all available parameters and their detailed descriptions, refer to the इलम एपीआई दस्तावेज़ीकरण.

Monitor Spark Job Status

Upon successful submission, the API returns a JSON response containing the जॉब आईडी. You can use this ID to poll for the job's completion status, making it easy to build wait-logic into your automation scripts.

{
  "जॉब आईडी": "20251222-0931-f56pqk5y1ap"
}

You can use this जॉब आईडी to check the current status of your job:

Get Job Status

छल्‍ला "http://localhost:9888/api/v1/job/{jobId}"

The response provides a comprehensive overview of the job's configuration, state, and execution timing.

{
  "जॉब आईडी": "20251222-0931-f56pqk5y1ap",
  "नौकरी का नाम": "MiniReadWriteTest",
  "नौकरी का प्रकार": "सिंगल",
  "भाषा": "स्केल",
  "ऐपआईडी": "spark-92b3da7ee0fa4d1e965b521ba356544c",
  "राज्य": "समाप्त",
  "सबमिट टाइम": 1766395898079,
  "स्टार्टटाइम": 1766395899941,
  "समाप्ति समय": 1766395905785,
  "jobConfig": {
    "spark.executor.instances": "2",
    "spark.kubernetes.namespace": "डिफ़ॉल्ट",
    "spark.eventLog.enabled": "सच",
    "..." : "..."
  }
}

Key fields to monitor include:

• राज्य: The current lifecycle phase (e.g., SUBMITTED, RUNNING, FINISHED, FAILED).
• appId: The Spark Application ID assigned by the cluster manager.
• startTime / endTime: Epoch timestamps (ms) for performance tracking.
• error: If the state is FAILED, this field will contain the error message or stack trace.

Troubleshooting Common Issues

If you encounter issues while submitting jobs, refer to the table below for common error codes and solutions.

HTTP Code	Error	Possible Cause & Solution
400	Bad Request	Missing Parameters: Ensure jobClass, क्लस्टरनामऔर जार (for Scala) are provided correctly in the form data.
401	Unauthorized	Auth Failure: Check if your cluster requires an API Token or Basic Auth header.
404	Not Found	Invalid Cluster: The क्लस्टरनाम specified does not exist. Verify active clusters via GET /api/v1/cluster.
500	Internal Server Error	Cluster Connection: Ilum cannot talk to the K8s API server. Check the इलम कोर logs for connectivity issues.

Frequently Asked Questions (FAQ)

Can I upload Python dependencies?

Yes. For PySpark jobs, use the pyFiles parameter to upload your .py script or a .zip archive containing your Python modules.

How do I secure the API?

We recommend placing the Ilum API behind an Ingress Controller with Basic Auth or OAuth2 enabled. You can then pass the credentials via standard HTTP headers.

What is the maximum JAR size?

The default limit is usually 100MB (configured in your Ingress or Spring Boot settings). For larger JARs, we recommend uploading them to S3/HDFS first and referencing them via spark.jars config, rather than uploading directly.