Configure Cloud Object Storage (GCS, S3, Azure) for Data Lake

Ilum allows you to link जीसीएस, एस 3, डब्ल्यूएएसबीएसऔर एचडीएफएस storages to your clusters. Linking storage allows Ilum to automatically configure all your jobs to use your cloud data lakes seamlessly, eliminating the need for manual Spark parameter configuration.

Supported Storage Providers

Provider	प्रकार	या क़िस्‍म
Google क्लाउड स्टोरेज	जीसीएस	Native integration for GCP projects.
अमेज़न S3	एस 3	Standard S3 and S3-compatible storage support.
Azure ब्लॉब संग्रहण	WASBS/ABFS	Integration for Azure data lakes.
एचडीएफएस	एचडीएफएस	Connect to existing Hadoop Distributed File Systems.

---

Google क्लाउड स्टोरेज (GCS)

Google क्लाउड स्टोरेज (GCS)

Step 1: Create a GCS Bucket

डेमो:

पूर्ण स्क्रीन में गाइड

1. Create a Google Cloud Project

   • Open Google Cloud Console और पर जाएं परियोजना चयनकर्ता / Manage Resources.
   • क्लिक करना New Project / Create Project.
   • Enter a Project name, choose Organization और स्थान.

2. Create a GCS Bucket

   • In the Console, navigate to बादल भंडारण → Buckets.
   • क्लिक करना बनाना.
   • Enter a globally unique Bucket name (e.g., my-ilum-bucket) and select your Region.
   नोट

   Remember the bucket name you created - you will need it when adding this storage to Ilum.

3. Create a Service Account and JSON Key

   • यहाँ जाओ IAM & Admin → Service Accounts.
   • क्लिक करना Create Service Account, fill in details, and grant Storage Admin roles.
   • Click the created email, go to the Keys tab, and Create new key (JSON).
   • Save the downloaded JSON file securely.
   important

   Organization Policy Update: In new organizations, creating service account keys might be disabled by default. Contact your administrator if you cannot create keys.

Step 2: Add GCS to Ilum Cluster

डेमो:

पूर्ण स्क्रीन में गाइड

1. Navigate to वर्कलोड → क्लस्टर्स → संपादन करना → भंडार → Add Storage.

2. Configure General Settings:

Parameter	Value Example	या क़िस्‍म
नाम	my-gcs-storage	Unique name for this storage config.
प्रकार	जीसीएस	Select GCS provider.
स्पार्क बाल्टी	my-ilum-bucket	Bucket for Spark logs/events.
डेटा बकेट	my-ilum-bucket	Bucket for your data.

3. Configure GCS Authorization: Open your JSON key file and copy the values:

Parameter	Source Key	या क़िस्‍म
Client Email	client_email	Service account email address.
Private Key	private_key	Full key including -----BEGIN....
Private Key ID	private_key_id	Key ID string.

4. क्लिक करना जमा करें to save.

अमेज़न S3

अमेज़न S3

सूचना-विषयक

The process for adding S3 storage is nearly identical to GCS. You will need to provide your AWS credentials (Access Key and Secret Key) instead of a JSON key file.

1. Navigate to वर्कलोड → क्लस्टर्स → संपादन करना → भंडार → Add Storage.
2. Select एस 3 as the प्रकार.
3. Fill in the required fields:

Parameter	या क़िस्‍म
नाम	Unique name for this storage config.
Access Key	Your AWS Access Key ID.
Secret Key	Your AWS Secret Access Key.
Region	AWS Region of your bucket (e.g., US-East-1 (अमेरिका पूर्व -1)).
Endpoint	(Optional) Custom endpoint for S3-compatible storage (e.g., MinIO).

Azure ब्लॉब संग्रहण

Azure ब्लॉब संग्रहण

सूचना-विषयक

The process for adding Azure storage is nearly identical to GCS and S3. You will need your Azure Storage Account Name and Access Key.

1. Navigate to वर्कलोड → क्लस्टर्स → संपादन करना → भंडार → Add Storage.
2. Select Azure (or WASBS) as the प्रकार.
3. Fill in the required fields:

Parameter	या क़िस्‍म
नाम	Unique name for this storage config.
Account Name	Your Azure Storage Account name.
Account Key	Your Azure Storage Account Access Key.
Container	Name of the container to use.

---

Step 3: Verify Connection

To ensure your storage is correctly configured, run a simple Spark job.

1. Create a Code Service:

   • यहाँ जाओ वर्कलोड → सेवाएँ → New Service +.
   • Select प्रकार: कोड, भाषा: स्काला, and your गुच्‍छा.

2. Execute Test Code: Paste and run the following Scala code:

   Test Storage Connection

   // Write test data
   val डाटा = Seq(("Alice", 34), ("Bob", 45))
   val लोमो = उत्तेजक गुण.createDataFrame(डाटा).toDF("नाम", "age")
   
   // Replace with your bucket path (e.g., gs://..., s3a://..., wasbs://...)
   val पथ = "gs://my-ilum-bucket/output/" 
   
   लोमो.लिखना.मोड समूह("अधिलेखन").format("csv").save(पथ)
   
   // Read back data
   उत्तेजक गुण.पढ़ना.format("csv").load(पथ).प्रदर्शन()

3. Check Results: If the job completes and displays the data table, your storage connection is active.

---

Common Issues & FAQ

Why do I get a "Permission Denied" error?

कारण: The Service Account or User doesn't have permissions to access the bucket. विलयन:

1. Go to your cloud provider's console (e.g., Google Cloud Console).
2. Navigate to the bucket's अनुमतियाँ बिल।
3. Grant your service account the Storage Admin नहीं तो Storage Object Admin role.

Why does it say "Bucket does not exist"?

कारण: The bucket name in your code doesn't match the actual bucket name, or the region is incorrect. विलयन:

1. Verify the bucket exists in your cloud console.
2. Check that the bucket name in your code matches exactly (names are often case-sensitive).

Why do I get "Invalid credentials"?

कारण: The keys (JSON or Access Keys) were not copied correctly. विलयन:

1. Re-open your key file.
2. Carefully copy the values again. For GCS, ensure you include the -----निजी कुंजी शुरू करें----- और -----अंत निजी कुंजी----- lines.
3. Re-save the storage configuration in Ilum.