Why Redis Streams for feature stores

Redis Streams + RedisJSON is a practical stack for feature stores that must deliver single-digit-millisecond reads while accepting high-throughput event updates. Streams give you an append-only, ordered event log with consumer groups and replayability; RedisJSON stores the latest materialized feature state per entity in a compact document for low-latency lookups.

This guide is focused on implementation choices engineers will actually make: data layout, ingestion and processing patterns, idempotency, cluster-aware key design, TTLs/versioning, and operational monitoring. Examples below use redis-py idioms and concrete patterns you can adapt.

Core architecture and data model

• Streams: durable append-only event logs (one or more per source or partition).
• Processors: consumer-group workers that read events, enrich/validate, and materialize into RedisJSON documents keyed by entity id.
• RedisJSON keys: one JSON document per entity (entity:), containing frequently read feature fields, last_update timestamp, and a version or last_applied_event id.

Design rules:
– Co-locate frequently accessed fields in the same JSON document to avoid multi-key reads.
– Keep documents compact; store only active feature fields and metadata.
– Include last_event_id or last_op_id in the JSON document for idempotency.

Ingestion patterns

1) Event-driven ingestion (recommended for most systems)
– Producers append small change events to a Stream (xadd).
– Consumer group workers (XREADGROUP) read, enrich, and write to RedisJSON.
– Benefits: decoupling, auditability, replay/backfill, horizontal scale.

2) Synchronous upserts (direct JSON writes)
– Producers write directly to RedisJSON (json.set/json.mset).
– Benefits: lowest write latency; costs: tighter coupling, harder replay.

Practical tips:
– Batch events on producers and processors to amortize round trips.
– Put only diffs in events (changed fields + metadata).
– Add a small schema_version to events for safe evolution.

Real-time lookup strategies

• Use a single JSON key per entity: GET json document is a single network call.
• For multi-entity reads in one request, use pipelining to parallelize GETs and reduce RTT.
• Materialize derived features during processing instead of computing at read time when latency matters.
• Use TTLs for naturally ephemeral documents and have a deterministic fallback (recompute or default) on misses.

Versioning, TTLs, and schema evolution

• Embed a schema_version and last_event_id in each JSON document.
• Processors should be able to read older versions during a rolling migration and write new-version docs. Keep migration code isolated in a small worker if rewrite is expensive.
• For time-bounded features, set TTLs to free memory; for permanent features, prune unused keys via a background job.

Scaling on Redis Cluster

Key-slot strategy
– Redis Cluster shards by key hash slot. Use a single JSON key per entity so all fields for an entity land on the same slot.
– Use explicit hash tags ({…}) when you must co-locate related keys across streams and docs: e.g., stream name events:{user:123} and entity:{user:123}.

Stream partitioning
– Split high-volume ingestion into multiple streams or use sharding suffixes to distribute load across slots.
– Run consumer groups per stream/partition; maintain fixed concurrency per shard to limit contention.

Operational tactics
– Monitor slot distribution and rebalance if a few slots concentrate load.
– Avoid multi-key operations across slots; use LUA scripts or client-side transactions only when keys are co-located.

Consistency and idempotency

• Streams provide at-least-once delivery. Design idempotent processors that can safely apply the same event multiple times.
• Store last_applied_event_id inside the JSON document; on processing, compare event id and skip if already applied.
• For strict read-after-write consistency on a critical feature, you can synchronously update RedisJSON before acknowledging the producer, but accept higher write latency for those operations.

Monitoring and SLOs

Track these core metrics:
– stream length and consumer group pending entry list (PEL) size
– consumer lag and per-consumer processing rate
– Redis memory usage, eviction rates, and keyspace stats
– command latency (especially JSON and XREAD/XACK ops)

Export metrics to Prometheus, create alerts for rising lag, memory pressure, or increased command latency. Dashboards per-stream and per-consumer-group help isolate hotspots.

Python example (redis-py) — consumer pattern

from redis import Redis
from redis.commands.json.path import Path

r = Redis(host=’localhost’, port=6379, decode_responses=True)
STREAM = ‘events:entity’
GROUP = ‘workers’
CONSUMER = ‘worker-1’

Ensure group exists (safe to call at startup)

try:
r.xgroup_create(STREAM, GROUP, id=’0′, mkstream=True)
except Exception:
pass

while True:
entries = r.xreadgroup(GROUP, CONSUMER, {STREAM: ‘>’}, count=50, block=5000)
if not entries:
continue
for stream_name, messages in entries:
for msg_id, payload in messages:
entity_id = payload.get(‘entity_id’)
if not entity_id:
r.xack(STREAM, GROUP, msg_id)
continue

        # idempotency check
        json_key = f'entity:{entity_id}'
        last_id = r.json().get(json_key, Path('.last_event_id'))
        if last_id and last_id >= msg_id:
            r.xack(STREAM, GROUP, msg_id)
            continue

        # apply feature updates (example: partial set)
        pipeline = r.pipeline()
        # update fields and last_event_id and last_update timestamp
        pipeline.json().set(json_key, Path('.features'), payload.get('features', {}))
        pipeline.json().set(json_key, Path('.last_event_id'), msg_id)
        pipeline.json().set(json_key, Path('.last_update'), payload.get('ts'))
        pipeline.execute()

        r.xack(STREAM, GROUP, msg_id)

Notes:
– Keep processing idempotent and fast; offload heavy enrichment to async jobs or micro-batches.
– Use XCLAIM to recover pending messages if a consumer dies.

Backfills and migrations

• Backfills can write to streams (preferred) so normal processors handle materialization.
• Alternatively, use a separate migration consumer that tags rewritten documents with a backfill marker to avoid collisions with live updates.

Operational checklist before production

• Define key naming and hash tags for co-location in cluster mode.
• Implement idempotency and last_event_id storage.
• Add schema_version to events and docs.
• Build monitoring for stream length, consumer lag, memory, and latencies.
• Test failover scenarios (consumer crash, Redis node failure) and validate recovery paths.

Conclusion

Redis Streams + RedisJSON is a pragmatic combination for low-latency, event-driven feature stores. The common pattern—streams for durable ingestion, consumer groups for enrichment and idempotent materialization, and one JSON doc per entity for reads—scales well when you design keys for cluster co-location, control consumer parallelism, and monitor consumer lag and memory usage. Use the ingestion pattern (events vs direct upserts) that matches your latency and durability trade-offs, and implement small but essential primitives: last_event_id, schema_version, TTLs for ephemeral features, and robust monitoring.