Ilum Core
The idea of interactive Spark jobs is to give a user a possibility to run consecutive Spark jobs without a long Spark application initialization time.
Architecture
Ilum is designed to bring the power of Apache Spark to Kubernetes environments, leveraging the best of both ecosystems. At its core, it aims to simplify the process of deploying, managing, and monitoring Spark jobs, irrespective of the underlying cluster manager.
Kubernetes Cluster
Ilum integrates with Kubernetes to automate cluster management, making it easy to deploy across cloud-native environments.
Apache Kafka
Integrates seamlessly with Kubernetes clusters, where Spark jobs are executed.
Key Components
Ilum is designed to bring the power of Apache Spark to Kubernetes environments, leveraging the best of both ecosystems. At its core, it aims to simplify the process of deploying, managing, and monitoring Spark jobs, irrespective of the underlying cluster manager.
Ilum-core
The central piece of the Ilum architecture is the ilum-core, which is responsible for creating, managing, and monitoring Spark jobs. It exposes REST APIs (conforming to OpenAPI 3.0 standard) for clients to interact with, and is responsible for scheduling and executing Spark jobs on the connected Kubernetes clusters.
Ilum-ui
This is a user-friendly web interface that allows users to manage and monitor Spark jobs. It communicates with the ilum-core via the REST APIs.
Kubernetes Cluster
For users with legacy systems, Ilum also fully supports Apache Hadoop Yarn, offering flexibility across both modern and traditional infrastructures.
Object Storage
Ilum integrates with various object storage solutions, providing an alternative to Hadoop's HDFS. This allows storing and retrieving large volumes of data in a distributed and scalable manner.
MongoDB
Ilum utilizes MongoDB as its internal database for storing job metadata, cluster information, and other operational data.
Apache Kafka
Kafka can be used as a communication layer in Ilum for reliable and efficient data streaming and processing
Have some ideas how to improve Ilum?
We value your feedback and are always eager to improve. If you have suggestions that could enhance your experience or make navigating the product easier, please let us know. Join us in shaping the future of our platform! Your input matters.
Add feature request