When following above Getting Started with RAPIDS and Kubernetes, make sure you modify the Dockerfile to copy your application(such as jars, python files) into the docker image. Copy your application into the docker image Here we assume you already have a working Kubernetes Cluster with NVIDIA GPU support, and also built your own Spark docker image by following the above Getting Started with RAPIDS and Kubernetes. This article will mainly focus on how to use Spark Operator to do the same thing.
In the Rapids Accelerator official Doc: Getting Started with RAPIDS and Kubernetes, it shares the steps on how to use spark-submit/spark-shell to directly submit Spark jobs into a Kubernetes Cluster. Rapids Accelerator 0.4.1 with cuDF 0.18.1Īs per SPARK-33005, Spark on Kubernetes is GA in Spark 3.1.1. This article shares the steps on how to run Spark job with Rapids Accelerator using Spark Operator in a Kubernetes Cluster.
