Databricks

Spark and Redis together unlock powerful capabilities for data professionals. This section describes how to integrate these technologies for enhanced analytics, real-time processing, and machine learning applications.

Connecting to Redis

In Databricks, open your cluster settings and locate Advanced Options. Under Spark in the Spark config text area, add your Redis connection string as:

spark.redis.read.connection.uri redis://…

spark.redis.write.connection.uri redis://….

Using Databricks Secret Management

It is recommended to use Databricks secrets to store the Redis URI if it contains sensitive credentials.

To configure secrets refer to the Databricks documentation.

You can reference those secrets in your Spark cluster using the same Spark config options:

spark.redis.read.connection.uri {{secrets/redis/uri}}

spark.redis.write.connection.uri {{secrets/redis/uri}}

Use SSL/TLS to connect Databricks to Redis

To enable SSL connections to Redis, follow the instructions in the TLS section of the configuration documentation. You can provide the configurations described there as options. For example you specify the trusted certificates in the property spark.redis.read.connection.ssl.cacert.

It is recommended that you:

Store your certificates in cloud object storage. You can restrict access to the certificates only to cluster that can access Redis. See Data governance with Unity Catalog.
Store your passwords as secrets in a secret scope.

The following example uses object storage locations and Databricks secrets to enable a SSL connection:

df = spark.read \
  .option("spark.redis.read.connection.ssl.cacert", <trusted-certs>) \
  .option("spark.redis.read.connection.ssl.cert", <public-key>) \
  .option("spark.redis.read.connection.ssl.key", <private-key>) \
  .option("spark.redis.read.connection.ssl.key.password", dbutils.secrets.get(scope=<scope-name>, key=<key-name>))

Redis Spark Notebook

In this hands-on tutorial you’ll learn how to make efficient use of Redis data structures alongside Spark’s distributed computing framework. You’ll see firsthand how to extract data from Redis, process it in Spark, and write results back to Redis for application use.

Key topics include:

Setting up the Spark-Redis connector in Databricks
Reading data from Redis for application access
Writing data to Redis from Spark

You can edit and run this notebook by importing it into your Databricks account.

Select Import from any folder’s menu and paste this URL: https://raw.githubusercontent.com/redis-field-engineering/redis-spark-dist/refs/heads/main/redis-spark-notebook.ipynb