Apache Spark Cluster
Apache Spark is a distributed data-processing engine that executes batch, streaming, SQL, and machine-learning workloads across a cluster of machines. It keeps intermediate data in-memory as resilient distributed datasets to avoid much of the disk I/O overhead that classic MapReduce jobs incur.
Baseline topologies
| Topology | Minimum nodes | Roles |
|---|---|---|
| master-plus-workers | 3 | spark-master, spark-worker |
| master-plus-workers-ml | 3 | spark-master, spark-worker |
What drives the cost
- Executor count
- Number of Spark executor processes across the cluster workers; each executor runs tasks in parallel up to executor_cores.
- Memory per executor (GB)
- JVM heap allocated to each executor process; the primary offer-selection driver for Spark worker sizing. Off-heap overhead adds approximately 10-20% on top of this value.
- Concurrent Spark jobs
- Number of Spark jobs expected to run simultaneously, informing driver and master scheduling capacity.
Where it runs
bare_metal, dedicated, cloud_vm
Size Apache Spark Cluster and get a cost estimate