Scheduling Apache Spark Jobs

Scheduling in Ilum allows you to automate the execution of Apache Spark jobs on Kubernetes clusters at specified intervals using CRON expressions. This is essential for setting up reliable ETL pipelines, regular data analysis, or maintenance tasks that need to run without manual intervention.

Example JAR Source

You can use the jar file with spark examples from one of these links:

Spark 4 (default)

Spark 4 / Scala 2.13: spark-examples_2.13-4.1.1.jar

Spark 3

Spark 3 / Scala 2.12: spark-examples_2.12-3.5.8.jar

Step-by-Step Guide: Scheduling a Spark Job

1. Navigate to Schedules: Access the Horaires section in your Ilum dashboard.

2. Create New Schedule: Click the New Schedule + button to start setting up your automated job.

3. Fill Out Schedule Details:

   • Onglet Général :

     • Nom: Entrer ScheduledMiniReadWriteTest
     • Cluster: Select your target cluster
     • Classe: Entrer org.apache.spark.examples.MiniReadWriteTest
     • Language: Select Scala

   • Onglet de chronométrage :

     • CRON Expression: Select the Coutume onglet.
     • Custom expression: Entrer @daily
     Timing

     This configuration will trigger the job to run once every day at midnight. You can adjust this to any valid CRON expression (e.g., 0 */12 * * * for every 12 hours).

   • Configuration Tab:

     • Arguments: Entrer /opt/spark/examples/src/main/resources/kv1.txt

   • Onglet Ressources :

     • Jars: Upload the JAR file from the link above.

   • Memory Tab:

     • Leave all settings at their default values for this example.

4. Submit and Monitor:

   • Cliquer Envoyer to create the schedule.
   • You can see your new schedule in the list.
   • When the scheduled time arrives, a new job instance will be launched automatically. You can view these instances in the Emplois section.

Schedule Configuration Reference

Below is a detailed breakdown of all available settings, organized by tab as they appear in the UI.

Généralités

Parameter	Description
Nom	A unique identifier for the schedule.
Grappe	The target cluster where the scheduled jobs will be executed.
Classe	The fully qualified class name of the application (e.g., org.apache.spark.examples.SparkPi) or the filename for Python scripts.
Langue	The programming language used for the job (Scala ou Python).
Description	An optional description to explain the purpose of this schedule.
Max Retries	The maximum number of times Ilum will attempt to restart the job if it fails.

Timing

Parameter	Description
Start Time	(Optional) The specific date and time when the schedule should become active. If left blank, it starts immediately.
End Time	(Optional) The specific date and time when the schedule should stop triggering new jobs.
CRON Expression	Defines the frequency of the job execution. You can use the visual builders (Minutes, Hourly, Daily, Weekly, Monthly) or select Coutume to enter a standard Unix-style CRON expression (e.g., 0 12 * * * for noon daily).

Configuration

Parameter	Description
Paramètres	Key-value pairs for configuring Spark properties (e.g., spark.executor.memory).
Arguments	Command-line arguments passed to the main method of your application.
Tags	Custom labels to categorize and filter your scheduled jobs.

Ressources

Parameter	Description
Pots	Additional JAR files to be included in the classpath.
Files	Auxiliary files to be placed in the working directory of each executor.
PyFiles	Python dependencies (.zip, .egg, .Py) for Python jobs.
Exigences	Additional Python packages to install.
Spark Packages	Maven coordinates for Spark JAR packages to be downloaded.

Mémoire

Parameter	Description
Executors	The number of executor instances to allocate.
Driver Cores	The number of CPU cores assigned to the driver.
Executor Cores	The number of CPU cores assigned to each executor.
Driver Memory	The amount of RAM allocated to the driver.
Executor Memory	The amount of RAM allocated to each executor.
Dynamic Allocation	Enables automatic scaling of executors based on workload.

Foire aux questions

Details

Can I schedule PySpark jobs using Ilum?

Yes, Ilum fully supports scheduling for both Scala/Java (JARs) and Python (PySpark) jobs. Simply select "Python" as the language in the General tab and provide your script.

Details

How does the retry mechanism work?

If a scheduled job fails, Ilum can automatically attempt to restart it based on the "Max Retries" configuration. This ensures transient issues don't break your pipelines.

Details

What CRON formats are supported?

Ilum supports standard Unix-style CRON expressions (e.g., 0 12 * * *) as well as predefined macros like @daily, @hourly, etc.