Real-Time Feature Stores: Feast & Hopsworks for ML
Deploying Machine Learning models to production introduces a subtle but catastrophic failure mode: Training-Serving Data Skew. When feature engineering logic (such as calculating a user's 30-day purchase count or average click-through rate) is implemented in SQL during offline model training but re-written in Python/Go for real-time model inference, subtle logic discrepancies cause model accuracy to plummet in production.
ML Feature Stores provide a centralized registry and dual storage architecture for Machine Learning features. Feast (Feature Store) and Hopsworks serve features to online inference APIs with sub-10ms latency via Redis or DynamoDB, while simultaneously performing point-in-time time-travel joins over BigQuery, Snowflake, or Parquet files for offline model training. This guide details Feast feature views, dual storage synchronization, time-travel joins, and materialization pipelines.
Mental Model: Training-Serving Data Skew vs Centralized ML Feature Stores
Traditional ML engineering pipelines calculate features independently for training and serving, causing data inconsistencies:
1. Offline Training Pipeline: SQL scripts compute batch aggregations over historical warehouse tables. 2. Online Inference API: Microservice re-computes features on the fly, creating code duplication and time-lag mismatches. For vector search and edge AI model deployment, review building vector search engines faiss milvus qdrant and deploy multimodal gemini edge.
Quick reference
- Eliminates training-serving skew by sharing identical feature definitions between training and inference.
- Online storage layer (Redis/DynamoDB) serves low-latency feature vectors to model inference APIs.
- Offline storage layer (BigQuery/Snowflake) executes heavy batch queries for historical model training.
- Declarative YAML feature definitions foster feature reuse across multiple data science teams.
- Powers real-time ML systems at Uber, Spotify, DoorDash, Twitter, and CoreConcept.
Remember this
Adopt centralized ML Feature Stores to eliminate training-serving skew and standardize feature pipelines.
Feast Feature Store: Entity Definitions, Feature Views, & Declarative YAML
Feast structures ML feature definitions into declarative Python/YAML objects:
1# Feast Feature View Definition2from datetime import timedelta3from feast import Entity, Field, FeatureView, FileSource4from feast.types import Float32, Int645 6user_entity = Entity(name="user_id", value_type=Int64, description="User ID")7 8user_stats_view = FeatureView(9 name="user_purchase_stats",10 entities=[user_entity],11 ttl=timedelta(days=30),12 schema=[13 Field(name="avg_order_value", dtype=Float32),14 Field(name="purchase_count_30d", dtype=Int64),15 ],16 online=True,17 source=FileSource(path="data/user_stats.parquet", timestamp_field="datetime"),18)Quick reference
- Entities define primary join keys (user_id, item_id) for feature retrieval.
- FeatureViews define schema types, TTL retention windows, and data source mappings.
- Feast CLI applies feature store configurations directly to git-versioned repositories.
- Central registry file (registry.pb) acts as the single source of truth for features.
- Supports Python SDK, Go SDK, and REST/gRPC feature retrieval endpoints.
Remember this
Define entities and feature views declaratively in Feast to enable team-wide ML feature reuse.
Online Storage (Redis/DynamoDB) vs Offline Storage (BigQuery/Snowflake/Parquet)
Feature stores decouple online real-time inference from offline historical model training:
- Online Feature Store (Redis / DynamoDB): Key-value RAM store optimized for point queries. Stores the latest feature vector per entity (user:1092 -> [avg_order: 45.5, count: 12]), returning features in under 5 milliseconds.
- Offline Feature Store (BigQuery / Snowflake / Parquet): Columnar data lake storage optimized for heavy analytical SQL joins over years of historical data logs.
Quick reference
- Redis online store delivers sub-5ms feature vector lookups for real-time recommendation APIs.
- Offline warehouse stores complete immutable timestamped feature logs for historical training.
- Feast materialization pipeline continuously syncs new feature records from offline to online storage.
- Supports streaming sources (Kafka/Kinesis) via Spark/Flink streaming feature ingestion.
- Guarantees seamless feature vector retrieval across both training and production inference.
Remember this
Use Redis for sub-5ms online feature vector retrieval and BigQuery for offline training datasets.
Point-in-Time Correctness (Time-Travel Joins) & Materialization Pipelines
Joining historical training observation events with feature tables using standard SQL inner joins introduces Data Leakage — joining future feature values that occurred after the observation timestamp.
### Time-Travel Join Mechanics
Feast executes Point-in-Time (PIT) joins (get_historical_features): for every observation timestamp $t_{\text{obs}}$, it selects the latest feature record where $t_{\text{feature}} \le t_{\text{obs}}$ within the defined TTL window.
Quick reference
- Point-in-time time-travel joins eliminate data leakage during training dataset creation.
- Matches observation timestamps with historical feature snapshots accurately.
- Feast materialize command syncs incremental feature updates to Redis on scheduled intervals.
- Monitors feature drift and data quality metrics via Evidently AI or Great Expectations.
- Ensures model training data reflects exact historical state at prediction time.
Remember this
Execute Feast point-in-time joins to eliminate data leakage during ML training dataset generation.
Key takeaway
To test Feast locally, install via pip install feast and run feast init my_feature_store. Run feast apply followed by feast materialize-incremental 2026-08-06T00:00:00.
Related Articles
Explore this topic