Redis Connector for Spark
Overview
The Redis Connector for Spark is a library that implements the Spark DataSource V3 API (TableProvider) for bidirectional integration between Apache Spark and Redis. It enables reading from and writing to Redis data structures including HASH, JSON, STRING, and STREAM types.
Key Features
-
Multi-type support: Works with Redis HASH, JSON, STRING, and STREAM data types
-
Streaming capabilities: Supports both batch and streaming operations (Redis streams only support streaming mode)
-
Schema inference: Automatically detects schema from Redis data structures
-
Consumer Groups: Leverages Redis Streams consumer groups for guaranteed delivery
