Create and Connect a Remote GKE Cluster to Data Lakehouse

परिचय

Ilum empowers you to manage a powerful multi-cluster setup from a single, central control plane. While Ilum automates the deployment and configuration of its core components, setting up the underlying infrastructure requires precise coordination.

This guide provides a comprehensive walkthrough for setting up a multi-cluster architecture on Google कुबेरनेट्स इंजन (GKE). You will learn how to:

1. Provision a central control plane (Master Cluster).
2. Set up a dedicated execution environment (Remote Cluster).
3. Establish secure communication between them using client certificates and ingress rules.

This guide walks you through the steps required to launch your first Ilum Job on a remote cluster. We use Google Kubernetes Engine (GKE) as an example, but you can follow the same flow with any Kubernetes distribution.

आवश्यकताएँ

Before starting the tutorial, make sure you have:

1. Access to a Google Cloud project with billing enabled.
2. कुबेक्टल installed and configured on your machine (version compatible with your GKE cluster).
3. पतवार installed (v3+).
4. वही Google Cloud CLI (gcloud) installed and initialized (you can run gcloud auth login और gcloud config list without errors).
5. वही gke-gcloud-auth-plugin installed and available in your PATH so that कुबेक्टल can authenticate to GKE clusters.
6. Permissions in the target Google Cloud project to:
   • create and manage GKE clusters (e.g. Kubernetes Engine Cluster Admin or equivalent),
   • create and use Cloud Storage buckets if you plan to use GCS for data.

What you'll accomplish in this guide:

Step	Task	Purpose
1	Create two GKE clusters	Set up master (control plane) and remote (job execution)
2	Install Ilum on master	Deploy Ilum's core components
3	Set up authentication	Create secure credentials for remote cluster access
4	Register remote cluster	Add cluster to Ilum's management interface
5	Configure networking	Enable communication between clusters
6	Run your first job	Verify the multi-cluster setup works

---

Step 1. Provision Master and Remote GKE Clusters

The foundation of a multi-cluster setup consists of two distinct entities:

• Master Cluster: Hosts the Ilum control plane (UI, API, Scheduler).
• Remote Cluster: dedicated environment where Ilum executes the Spark Jobs dispatched from the master.

एक प्रोजेक्ट बनाएं

• Open Google Cloud Console.
• Click the Project selector in the top-left corner.
• क्लिक करना New Project.
• Enter a project name and (if applicable) select an Organization/Folder.
• क्लिक करना बनाना.
• Select the newly created project in the project selector.

Enable Google Kubernetes Engine API

• In the Console search bar, type कुबेरनेट्स इंजन.
• Open कुबेरनेट्स इंजन.
• क्लिक करना योग्य बनाना to enable the Google Kubernetes Engine API for the selected project.

Switch to the chosen project in gcloud

• In the Console, open the project selector and copy the Project ID.
• In your terminal, set this project as active:

Set Project

gcloud config अस्त हो project PROJECT_ID

• (Optional) If you plan to create clusters in a specific region often, set a default region:

Set Default Region

gcloud config अस्त हो compute/region europe-central2

This avoids errors requiring --region / --zone.

क्लस्टर बनाएँ

Create the master cluster first:

Create Master Cluster

gcloud container clusters create master-cluster \
  --मशीन-प्रकार=n1-standard-8 \
  --num-nodes=1

Create the remote cluster with a different name:

Create Remote Cluster

gcloud container clusters create remote-cluster \
  --मशीन-प्रकार=n1-standard-4 \
  --num-nodes=1

important

Resource Requirements & Architecture:

Why two clusters? The master cluster runs Ilum's control plane (UI, API, scheduler). The remote cluster executes your Spark jobs. This separation allows independent scaling and multi-cluster management from one interface.

Sizing:

• Master cluster: This example uses n1-standard-8 (8 vCPU, 30 GB RAM) for testing only. Minimum recommended: 12 vCPUs and 48 GB RAM (e.g., n1-standard-12). Production environments with many users need significantly more.
• Remote cluster: This example uses n1-standard-4 (4 vCPU, 15 GB RAM) for testing only. Production workloads require larger machines (e.g., n1-standard-16+) and multiple nodes depending on your Spark job requirements.

Step 2. Install Ilum Control Plane on Master Cluster

Once your clusters are running, the next step is to deploy the Ilum platform on the master cluster.

kubectl में मास्टर क्लस्टर कॉन्फ़िगरेशन पर स्विच करें

Switch Context

# kubectl संदर्भ प्राप्त करें और अपने क्लस्टर के साथ एक खोजें
kubectl config get-contexts
# switch to it using this command
kubectl config use-context MASTER_CLUSTER_CONTEXT

नामस्थान बनाएं और kubectl में इसे स्विच करें

Setup Namespace

Kubectl Create Namespace ilum
kubectl config set-context --current --namespace=इलम

Install Ilum using Helm charts:

इलम स्थापित करें

helm repo आगे कहना ilum https://charts.ilum.cloud
helm install इलम -n इलम इलम /

नोट

What just happened? You've installed Ilum's control plane on the master cluster. You can now access Ilum's interface to manage jobs across multiple clusters.

Step 3. Configure Authentication for Remote Cluster Access

important

Goal: Generate secure credentials (client certificates) that authorize the Ilum control plane to deploy and manage resources on the दूरस्थ क्लस्टर.

महत्वपूर्ण: All steps in this section must be performed while connected to the दूरस्थ क्लस्टर (not the master cluster).

Why this is needed:

When you created the remote cluster with gcloud, it was automatically added to your kubeconfig with your Google Cloud account credentials. However, Ilum doesn't have access to your Google account.

Ilum needs its own authentication method - client certificates - to connect to and manage the remote cluster. This section creates those certificates.

Step	Action	Purpose
1	Create Client Key	Generate private key for authentication
2	Create CSR	Request identity verification from cluster
3	Register & Approve CSR	Cluster admin validates and signs request
4	Get Signed Certificate	Retrieve the "ID card" for Ilum
5	Grant Admin Permissions	Authorize Ilum to manage cluster resources
✓	परिणाम	Ilum can securely manage the cluster

This is similar to how कुबेक्टल authenticates to clusters, but we're doing it programmatically for Ilum.

नोक

Before you start: Make sure you're connected to the दूरस्थ क्लस्टर context:

Switch to Remote Context

kubectl config use-context gke_<your-project>_<region>_remote-cluster

You can verify with: kubectl config current-context

Step 3.1: Create a Client Key

What we're doing: Generating a private key that will be used for authentication.

First, create a dedicated directory for the certificates and work there:

Create Workspace

mkdir -p ~/remote-cluster
cd ~/remote-cluster

Now generate the private key:

Generate Private Key

openssl genpkey -algorithm RSA -out client.key

danger

Security: This private key is like a password. Keep client.key secure and never commit it to version control.

Step 3.2: Create Certificate Request

What we're doing: Creating a Certificate Signing Request (CSR) that asks the remote cluster to verify and sign our identity.

बदलें मायूसर with the username of your choice:

Create CSR

openssl req -new -key client.key -out csr.csr -subj "/CN=myuser"

नोक

What is a CSR? A Certificate Signing Request asks the Kubernetes cluster to create a signed certificate for your user. The cluster will verify and sign it, creating a trusted identity.

Step 3.3: Encode the CSR

What we're doing: Converting the CSR to base64 format, which is required by Kubernetes.

Encode CSR

cat csr.csr | base64 | tr -d '\n'

Copy the output - you'll need it in the next step.

Step 3.4: Register CSR in Kubernetes

What we're doing: Submitting the CSR to the remote cluster for approval.

बनाना सीएसआर.वाईएएमएल (replace मायूसर and paste your encoded CSR):

सीएसआर.वाईएएमएल

apiVersion: certificates.k8s.io/v1
प्रकार: CertificateSigningRequest
मेटाडेटा:
  नाम: मायूसर  # Replace with your username
कल्पना:
  निवेदन: >
  हस्ताक्षरकर्ता का नाम: kubernetes.io/kube-एपिसर्वर-ग्राहक
  उपयोग:
    - ग्राहक प्रमाणीकरण

Apply the CSR to the दूरस्थ क्लस्टर:

Register CSR

kubectl apply -स्‍त्री-विषयक सीएसआर.वाईएएमएल

Step 3.5: Approve and Retrieve the Certificate

What we're doing: As cluster admin, we approve our own CSR and retrieve the signed certificate.

Approve the CSR:

Approve CSR

kubectl प्रमाणपत्र स्वीकृत myuser

Retrieve the signed certificate:

Retrieve Certificate

kubectl get csr myuser -ओ jsonpath='{.status.certificate}' | base64 --decode > क्लाइंट.सीआरटी

नोट

What happened? You've registered your certificate request with Kubernetes, approved it (as cluster admin), and retrieved the signed certificate. This certificate + private key pair is now your authentication credential for the remote cluster.

Step 3.6: Get CA Certificate

What we're doing: Extracting the cluster's CA certificate. This will be needed for the optional test in the next section, and later in Step 4 when adding the cluster to Ilum.

Get and save the CA certificate:

Get CA Certificate

Kubectl कॉन्फिग व्यू --minify --raw -ओ jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode > सीए.सीआरटी

Verify the certificate was saved correctly:

Verify CA Certificate

openssl x509 -in सीए.सीआरटी -text -noout | head -n 5

नोक

Why do we need this?

• CA Certificate: Verifies the cluster's identity (prevents man-in-the-middle attacks)
• Flags: --minify gets only current context, --raw gets actual data instead of file paths

Step 3.7: Grant Permissions with Role Binding

What we're doing: Giving the certificate user (myuser) admin permissions on the remote cluster.

बनाना रोलबाइंडिंग.वाईएएमएल:

apiVersion: rbac.authorization.k8s.io/v1
प्रकार: क्लस्टररोलबाइंडिंग
मेटाडेटा:
  नाम: मायूसर-शासन-स्‍की वस्‍तु
विषयों:
  - प्रकार: उपभोक्ता
    नाम: मायूसर  # Match your username
    एपीआई समूह: rbac.authorization.k8s.io
roleRef:
  प्रकार: क्लस्टररोल
  नाम: गुच्‍छा-शासन
  एपीआई समूह: rbac.authorization.k8s.io

Apply the role binding to the दूरस्थ क्लस्टर:

Apply Role Binding

kubectl apply -स्‍त्री-विषयक रोलबाइंडिंग.वाईएएमएल

चेतावनी

Security: We're granting cluster-admin for simplicity. In production, create a custom role with only the permissions Ilum needs:

• Create/delete pods, services, configmaps
• Read secrets
• Manage persistent volumes

Step 3.8: Configure Service Account Permissions

What we're doing: Granting permissions to the ServiceAccount that Spark pods will use.

important

What is a Service Account? It's the Kubernetes equivalent of a user for pods. Spark driver pods use it to create executor pods and manage resources in the cluster.

By default, pods use the चूक ServiceAccount in their namespace, but it doesn't have sufficient privileges to create other pods.

बनाना sa_role_binding. यम:

sa_role_binding. यम

apiVersion: rbac.authorization.k8s.io/v1
प्रकार: क्लस्टररोलबाइंडिंग
मेटाडेटा:
  नाम: इलम-चूक-शासन-स्‍की वस्‍तु
roleRef:
  एपीआई समूह: rbac.authorization.k8s.io
  प्रकार: क्लस्टररोल
  नाम: गुच्‍छा-शासन
विषयों:
  - प्रकार: ServiceAccount
    नाम: चूक
    Namespace: इलम  # Namespace where Spark jobs will run

Apply the role binding to the दूरस्थ क्लस्टर:

Apply SA Role Binding

kubectl apply -स्‍त्री-विषयक sa_role_binding. यम

चेतावनी

Production Security: We're granting cluster-admin to the default ServiceAccount for simplicity. In production:

1. Create a dedicated ServiceAccount (e.g., spark-driver) with minimal permissions
2. Specify it in your cluster configuration:

   spark.kubernetes.authenticate.driver.serviceAccountName=spark-driver
3. Grant only the permissions needed: create/delete pods, read configmaps/secrets, etc.

(Optional) Test the Client Certificates

नोक

Why this step? This optional step verifies that the certificates work correctly before we give them to Ilum in Step 4.

We'll create a temporary kubectl context that uses the certificates (instead of your Google Cloud credentials) to confirm they have the correct permissions.

1. Verify certificates are in the directory:

All certificate files should already be in ~/remote-cluster/ from the previous steps:

List Workspace Content

ls ~/remote-cluster/
# You should see: ca.crt  client.crt  client.key  csr.csr  csr.yaml

2. Get your remote cluster name:

Get Cluster Name

Kubectl कॉन्फिग व्यू -ओ jsonpath='{.clusters[*].name}' | tr ' ' '\n' | grep बेमिलनसार

You should see something like: gke_my-project_europe-central2_remote-cluster

Copy this cluster name.

3. Add the certificate-based user to kubectl:

Set Credentials

kubectl config set-credentials myuser-cert \
  --client-certificate=$HOME/remote-cluster/client.crt \
  --client-key=$HOME/remote-cluster/client.key

This creates a new user in your kubeconfig that uses certificates instead of your Google account.

4. Create a test context:

Set Context

kubectl config set-context test-remote-certs \
  --cluster=<paste-your-remote-cluster-name-here> \
  --namespace=चूक \
  --user=myuser-cert

बदलें with the cluster name from step 2.

5. Test the certificate-based authentication:

Test Authentication

kubectl config use-context test-remote-certs
Kubectl फली प्राप्त करें

Expected output:

डिफ़ॉल्ट नामस्थान में कोई संसाधन नहीं मिला.

If you see this (or a list of pods), the certificates work! ✅

6. Switch back to your normal context:

Revert Context

# Find your original context
kubectl config get-contexts

# Switch back (replace with your actual context name)
kubectl config use-context gke_<your-project>_<region>_remote-cluster

नोट

What we just verified: The client certificates can successfully authenticate to the remote cluster with admin permissions. In Step 4, we'll provide these same certificates to Ilum so it can manage the cluster.

---

Step 4. Register the Remote Cluster in Ilum UI

With your certificates generated and permissions granted, you are now ready to connect the remote cluster to the Ilum control plane.

What you'll provide: All the credentials and connection details you created in Step 3:

Item	File/Value	Purpose
CA Certificate	सीए.सीआरटी	Verifies cluster identity
Client Certificate	क्लाइंट.सीआरटी	Your authentication credential
Client Key	client.key	Private key for authentication
सर्वर URL	From kubectl	Cluster API endpoint
Username	मायूसर	User identity

Access Ilum UI

Before you can add the remote cluster to Ilum, you need to access the Ilum web interface.

1. Switch to the master cluster context:

Switch Context

kubectl config use-context gke_<your-project>_<region>_master-cluster

You can verify with: kubectl config current-context

2. Set up port forwarding to Ilum UI:

Port Forward UI

Kubectl पोर्ट-फॉरवर्ड -n ilum svc/ilum-ui 9777:9777

Keep this terminal window open - it needs to run continuously.

3. Open Ilum in your browser:

Open http://localhost:9777 in your web browser.

4. Log in:

• Username: शासन
• Password: शासन

Now you're ready to add the remote cluster to Ilum!

---

Demo: Adding a Kubernetes Cluster

पूर्ण स्क्रीन में गाइड

क्लस्टर निर्माण पृष्ठ पर जाएं

• वर्कलोड में क्लस्टर सेक्शन में जाएं
• क्लिक करना नया क्लस्टर घुंडी

सामान्य सेटिंग्स निर्दिष्ट करें:

• इच्छित नाम और वर्णन चुनें.
• क्लस्टर प्रकार को कुबेरनेट्स
• एक स्पार्क संस्करण चुनें

स्पार्क संस्करण को स्पार्क नौकरियों के लिए एक छवि का चयन करके निर्दिष्ट किया गया है।

इलम को अपनी छवियों के उपयोग की आवश्यकता होती है, जिसमें सभी आवश्यक निर्भरताएं पूर्वस्थापित होती हैं।

यहां सभी उपलब्ध छवियों की सूची दी गई है:

स्पार्क कॉन्फ़िगरेशन निर्दिष्ट करें

स्पार्क कॉन्फ़िगरेशन जो आप क्लस्टर स्तर पर निर्दिष्ट प्रत्येक व्यक्तिगत Ilum कार्य उस क्लस्टर पर लागू करने के लिए लागू किया जाएगा।

यदि आप कॉन्फ़िगरेशन को दोहराने से बचना चाहते हैं तो यह उपयोगी हो सकता है। उदाहरण के लिए, यदि आप अपने स्पार्क कैटलॉग के रूप में आइसबर्ग का उपयोग कर रहे हैं, तो आप इसे क्लस्टर स्तर पर एक बार कॉन्फ़िगर कर सकते हैं, और फिर इस क्लस्टर पर तैनात सभी इलम कार्य स्वचालित रूप से इन कॉन्फ़िगरेशन का उपयोग करेंगे।

Important: Set the namespace for Spark jobs

में पैरामीटर section, add:

spark.kubernetes.namespace: ilum

नोट

This ensures all Spark jobs run in the इलम namespace where you created the external services (ilum-core, ilum-grpc, ilum-minio) in Step 5. Without this, jobs would run in चूक namespace and couldn't connect to master cluster services.

भंडारण जोड़ें

Default cluster storage will be used by Ilum to store all the files required to run Ilum Jobs both provided by user and by Ilum itself.

You can choose any type of storage: एस 3, जीसीएस, डब्ल्यूएएसबीएस, एचडीएफएस.

नोक

For detailed instructions on setting up storage (especially GCS), see the संग्रहण बनाएँ guide.

Quick setup in the UI:

• पर क्लिक करें "भंडारण जोड़ें" घुंडी
• Specify the name and choose the storage type (S3, GCS, etc.)
• चुनना स्पार्क बाल्टी - मुख्य बाल्टी Ilum Jobs फ़ाइलों को संग्रहीत करने के लिए उपयोग की जाती है
• चुनना डेटा बकेट - यदि आप इलम टेबल्स स्पार्क प्रारूप का उपयोग करते हैं तो आवश्यक है
• Specify storage endpoint and credentials
• पर क्लिक करें "सबमिट करें" घुंडी

आप किसी भी संख्या में स्टोरेज जोड़ सकते हैं, और Ilum Jobs को उनमें से प्रत्येक का उपयोग करने के लिए कॉन्फ़िगर किया जाएगा।

कुबेरनेट्स अनुभाग पर जाएं और कुबेरनेट्स कॉन्फ़िगरेशन निर्दिष्ट करें

यहां, हम अपने कुबेरनेट्स क्लस्टर से जुड़ने के लिए सभी आवश्यक विवरण के साथ इलम प्रदान करते हैं। प्रक्रिया kubectl टूल के माध्यम से क्लस्टर से कनेक्ट करने के समान है, लेकिन यहां इसे UI के माध्यम से किया जाता है।

You will need to provide the following:

Field	What to provide	File location
Url	Cluster API endpoint (text field)	⚠️ Switch to remote cluster first, then: kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
CaCert	Upload CA certificate file	~/remote-cluster/ca.crt (from Step 3.6)
ClientKey	Upload client private key file	~/remote-cluster/client.key (from Step 3.1)
ClientCert	Upload client certificate file	~/remote-cluster/client.crt (from Step 3.5)
Username	Username (text field)	मायूसर (or whatever you used in Step 3.2)

Once you have specified these items, Ilum will be able to access your Kubernetes cluster.

इसके अलावा, आप कर सकते हैं:

• कुबेरनेट्स को एक की आवश्यकता के लिए कॉन्फ़िगर करें पासवर्ड आपके उपयोगकर्ता के लिए। इस मामले में, आपको यूआई में यहां पासवर्ड निर्दिष्ट करना होगा।
• एक जोड़ें पासफ़्रेज़ क्लाइंट कुंजी बनाते समय। यदि आपने ऐसा किया है, तो आपको UI में यहां पासफ़्रेज़ निर्दिष्ट करना होगा।
• निर्दिष्ट करें कुंजी एल्गोरिथ्म client.key में उपयोग किया जाता है। यह अनिवार्य नहीं है, क्योंकि यह जानकारी आमतौर पर कुंजी के भीतर ही संग्रहीत होती है। हालाँकि, ऐसे मामले हो सकते हैं जहाँ आपको इसे स्पष्ट रूप से परिभाषित करने की आवश्यकता हो।

अंत में, आप पर क्लिक कर सकते हैं जमा करें क्लस्टर जोड़ने के लिए बटन।

---

अंत में, आप पर क्लिक कर सकते हैं जमा करें क्लस्टर जोड़ने के लिए बटन।

Step 5. Configure Multi-Cluster Networking

चेतावनी

The Challenge: While Ilum can now dispatch jobs to the remote cluster, those jobs need a way to report their status, logs, and metrics back to the master cluster.

Why jobs need to communicate:

Component	Action	Target Service	स्थान
स्पार्क जॉब्स	Send status updates	Ilum Core (gRPC)	Master Cluster
स्पार्क जॉब्स	Send metrics & logs	MinIO Event Log	Master Cluster
Scheduled Jobs	Trigger new jobs	इलम कोर	Master Cluster

The Solution: Expose Ilum's services from the master cluster and create DNS aliases in the remote cluster.

Step 5.1: Configure Firewall

सबसे पहले, आपको अपने जीकेई मास्टर-क्लस्टर में आने वाले ट्रैफ़िक की अनुमति देनी होगी। Google क्लाउड प्रोजेक्ट ट्रैफ़िक एक्सेस को इसके फ़ायरवॉल द्वारा प्रबंधित किया जाता है। फ़ायरवॉल को नियमों द्वारा प्रबंधित किया जा सकता है। रिमोट-क्लस्टर को मास्टर-क्लस्टर सेवाओं तक पहुंचने की अनुमति देने के लिए, हमें इस नियम को हमारे मास्टर-क्लस्टर की परियोजना:

Create Firewall Rule

gcloud compute firewall-rules create allow-ingress-traffic \
  --network चूक \
  --direction INGRESS \
  --action ALLOW \
  --rules tcp:80,tcp:443 \
  --source-ranges 0.0.0.0/0 \
  --description "Allow HTTP/HTTPS"

danger

Security Warning: --source-ranges 0.0.0.0/0 allows traffic from anywhere. In production, restrict this to your remote cluster's IP range:

--source-ranges 10.0.0.0/8  # Example: your cluster's CIDR

Step 5.2: Expose Services from Master Cluster

नोट

What is LoadBalancer? It's a Kubernetes service type that provisions a public IP address, making the service accessible from outside the cluster (including from your remote cluster).

आपको बाहरी दुनिया के लिए मास्टर क्लस्टर में सेवाओं को उजागर करने की आवश्यकता है। इसे प्राप्त करने के लिए, उनके प्रकार को LoadBalancer में बदलें। इलम हेल्म कॉन्फ़िगरेशन का उपयोग करके इस प्रक्रिया को सरल बनाता है।

To expose Ilum Core, gRPC, and MinIO to the outside world, run the following command in your terminal:

Expose Services

helm upgrade ilum -n इलम इलम / \
  --अस्त हो ilum-core.service.type="LoadBalancer" \
  --अस्त हो ilum-core.grpc.service.type="LoadBalancer" \
  --अस्त हो minio.service.type="LoadBalancer" \
  --पुन: उपयोग-मान

उसके बाद आपको कुछ मिनट इंतजार करना चाहिए और फिर चलाकर अपनी सेवाओं की जांच करनी चाहिए

Check Services

Kubectl सेवाएं प्राप्त करें

इसे देखने के लिए:

Here you can notice the ilum-core, ilum-grpc and minio services changed their type to LoadBalancer and got public IP. Now you can go to http://<public-ip>:9888/api/v1/group यह जांचने के लिए कि क्या सब कुछ ठीक है।

Step 5.3: Create External Services in Remote Cluster

नोक

What is ExternalName? It creates a local DNS alias in the remote cluster. When a job tries to connect to इलम-कोर: 9888, Kubernetes redirects it to the public IP. Jobs use familiar service names as if everything was in one cluster.

Switch to remote cluster context:

Before creating external services, make sure you're working on the दूरस्थ क्लस्टर:

Switch to Remote Cluster

kubectl config use-context gke_<your-project>_<region>_remote-cluster

बनाना external_services. यम in the remote cluster:

नोट

Use the EXTERNAL-IP addresses from the Kubectl सेवाएं प्राप्त करें output in your master cluster (the public IPs shown in the screenshot above). Replace the placeholders below with these actual IP addresses.

external_services. यम

apiVersion: वी 1
प्रकार: सेवा
मेटाडेटा:
  नाम: इलम-अंतर्भाग
  Namespace: इलम  # Must match the namespace where jobs will run
कल्पना:
  प्रकार: बाहरी नाम
  externalName: 34.118.72.123  # Replace with actual EXTERNAL-IP from master cluster
  पोर्ट:
    - बंदरगाह: 9888
      टार्गेटपोर्ट: 9888

---
apiVersion: वी 1
प्रकार: सेवा
मेटाडेटा:
  नाम: इलम-जीआरपीसी
  Namespace: इलम  # Must match the namespace where jobs will run
कल्पना:
  प्रकार: बाहरी नाम
  externalName: 34.118.72.124  # Replace with actual EXTERNAL-IP from master cluster
  पोर्ट:
    - बंदरगाह: 9999
      टार्गेटपोर्ट: 9999

---
apiVersion: वी 1
प्रकार: सेवा
मेटाडेटा:
  नाम: इलम-मीनो
  Namespace: इलम  # Must match the namespace where jobs will run
कल्पना:
  प्रकार: बाहरी नाम
  externalName: 34.118.72.125  # Replace with actual EXTERNAL-IP from master cluster
  पोर्ट:
    - बंदरगाह: 9000
      टार्गेटपोर्ट: 9000

important

Before applying:

1. Replace the example IPs (34.118.72.123, etc.) with your actual EXTERNAL-IP addresses से Kubectl सेवाएं प्राप्त करें in the master cluster
2. Make sure the namespace: ilum matches the namespace where your Spark jobs will run in the remote cluster
3. If you haven't created the इलम namespace in the remote cluster yet, create it first:
   Create Namespace

   Kubectl Create Namespace ilum

Apply the external services:

Apply External Services

kubectl apply -स्‍त्री-विषयक external_services. यम

नोट

Multi-Cluster Bridge Complete: Jobs running on the remote cluster can now reach Ilum's services on the master cluster using familiar service names (ilum-core, ilum-grpc, ilum-minio), even though they're actually connecting over the internet via public IPs.

Components in Multi-Cluster Architecture

Components compatible with multi-cluster (with additional networking):

Component	Purpose	Requires Exposure
हाइव मेटास्टोर	Metadata management for tables	✅ Yes
मार्केज़	Data lineage tracking	✅ Yes
इतिहास सर्वर	Spark application history	✅ Yes
सुरमा	Metrics collection	✅ Yes

Components restricted to single-cluster:

Component	कारण
क्यूब प्रोमेथियस स्टैक	Prometheus needs direct pod access for metrics scraping. Challenging with dynamic pods across clusters.
लोकी और प्रोमटेल	Promtail collects logs similarly to Prometheus. Same multi-cluster limitations.

सूचना-विषयक

All other Ilum services not listed above are cluster-independent and work in both single- and multi-cluster setups.

---

Step 6. Verify the Multi-Cluster Setup

The final step is to validate your configuration by running a real Spark job on the remote cluster.

Step 1: Access Ilum UI

1. Switch to master cluster context:

   kubectl config use-context gke_<your-project>_<region>_master-cluster

2. Set up port-forwarding:

   Kubectl पोर्ट-फॉरवर्ड -n ilum svc/ilum-ui 9777:9777

   Keep this terminal window open.

3. Open Ilum in browser:

   • Navigate to http://localhost:9777
   • Login with default credentials: व्यवस्थापक:व्यवस्थापक

Step 2: Create a Test Job

1. Navigate to Jobs section in Ilum UI

2. Click "New Job +" घुंडी

3. Configure the job:

   • नाम: RemoteClusterTest
   • Job Type: Spark Job
   • Cluster: Select your दूरस्थ क्लस्टर (not master)
   • कक्षा: org.apache.spark.examples.SparkPi
   • Language: स्काला

4. Add Resources:

   • यहाँ जाओ संसाधन tab
   • Jars: Upload one of these:
     • Spark 4 / Scala 2.13 (default): spark-examples_2.13-4.1.1.jar
     • Spark 3 / Scala 2.12: spark-examples_2.12-3.5.8.jar

5. Submit the job

Step 3: Verify Execution

If everything is configured correctly:

✅ Job starts successfully - Pods are created in the remote cluster

✅ Logs appear - You can see Spark initialization and execution logs

✅ Job completes - Final output shows: Pi is roughly 3.14...

Check remote cluster pods:

Check Remote Pods

kubectl config use-context gke_<your-project>_<region>_remote-cluster
Kubectl फली प्राप्त करें -n इलम

You should see Spark driver and executor pods running or completed.

नोक

For detailed job configuration options, see the Run Simple Spark Job guide.

---

Troubleshooting & FAQ

Here are solutions to common issues you might encounter when connecting a remote GKE cluster.

Why is my Spark Job stuck in "Pending" state?

This usually happens if the remote cluster lacks resources or can't pull images.

• Check Resources: Ensure your node pool has enough CPU/RAM.
• Check Events: Run kubectl get events -n ilum on the remote cluster to see scheduling errors.

Why can't the remote cluster connect to Ilum Core?

If the job runs but fails to report status:

• Verify Firewall: Ensure the master cluster's firewall allows ingress on ports 9888 (Core), 9999 (gRPC), and 9000 (MinIO).
• Check DNS: Verify the बाहरी नाम services in the remote cluster resolve to the master's public IP.

Can I use a private GKE cluster?

Yes, but you will need to configure VPC Peering or a VPN between the master and remote networks instead of using public LoadBalancers.

---

अगले कदम

Congratulations! You've successfully set up a robust multi-cluster Ilum environment. You can now:

1. Deploy Spark jobs to your remote cluster from the Ilum UI
2. Monitor job execution across all clusters from one interface
3. Scale horizontally by adding more remote clusters using the same process
4. Optimize costs by using different machine types for different workloads

important

Production Checklist:

• Restrict firewall rules to specific IP ranges
• Create custom RBAC roles instead of cluster-admin
• Set up TLS certificates for LoadBalancer services
• Configure resource quotas and limits
• योग्य बनाना monitoring and alerting
• Document your cluster configurations