Databricks Feature Store

Description

Databricks Feature Store acts as a centralized registry for features used in AI and machine learning models. By registering feature tables and models within Unity Catalog, users gain integrated governance, lineage tracking, point-in-time joins, and the ability to share and discover features across workspaces. This platform streamlines the entire model training workflow, from data ingestion and feature table creation to model training and batch inference, all within a unified environment.

Key capabilities include data pipelines for ingesting raw data, creating feature tables, training models, and performing batch inference. It also provides one-click access to model and feature serving endpoints with millisecond latency, alongside data and model monitoring. When features from Databricks Feature Store are used for training, the model automatically tracks lineage. At inference time, the model retrieves the latest feature values. The Feature Store also handles on-demand feature computation for real-time applications, eliminating training-serving skew and simplifying client-side code by managing all feature lookups and computations.

For workspaces enabled with Unity Catalog, Databricks Feature Store offers a robust set of tools for developing, discovering, sharing, and serving features. Users can create and manage feature tables, define declarative features with time-windowed aggregations, and materialize them for offline training or online serving. Feature exploration is facilitated through Catalog Explorer and the Features UI, with support for tags to categorize and manage feature tables. The platform ensures point-in-time correctness for training datasets, reflecting feature values at the time of label observation. Online Feature Stores are available for serving feature data to online applications and real-time ML models, powered by Databricks Lakebase. Model serving includes automatic feature lookup from online stores, and feature serving endpoints can be exposed for external applications. On-demand feature computation at inference time further enhances real-time capabilities. Governance and lineage are managed through Unity Catalog, controlling access and providing visibility into feature table, model, and function lineage.

Databricks Feature Store's Core Features

• Centralized feature registry

• Unity Catalog integration

• Built-in governance

• Feature lineage tracking

• Point-in-time joins

• Cross-workspace feature sharing

• Feature discovery

• Unified model training workflow

• Data pipelines for feature engineering

• Batch inference capabilities

• Online feature serving endpoints

• Real-time model serving

• On-demand feature computation

• Eliminates training-serving skew

• Simplifies client-side code

How to use Databricks Feature Store?

1. 1

   Develop features: Create and manage feature tables using declarative APIs.

2. 2

   Materialize features: Prepare declarative features for offline training or online serving.

3. 3

   Discover features: Explore and manage feature tables via Catalog Explorer and the Features UI.

4. 4

   Train models: Utilize feature tables to train machine learning models.

5. 5

   Serve features: Deploy online feature stores for real-time applications.

6. 6

   Monitor data and models: Track performance and detect drift.

Databricks Feature Store's Use Cases

• ML Model Training
• Real-time Inference
• Feature Management
• Data Lineage Tracking
• Collaborative Feature Engineering
• Batch Scoring
• Retrieval Augmented Generation

FAQ from Databricks Feature Store

• What is Databricks Feature Store?

  Databricks Feature Store is a central registry for AI and ML features, integrated with Unity Catalog, providing governance, lineage, point-in-time joins, and online serving capabilities.

• How does Databricks Feature Store help with model training?

  It allows models to automatically track lineage to training features and ensures that feature computations used at inference are the same as those used during training, eliminating training-serving skew.

• What are the benefits of using Unity Catalog with Databricks Feature Store?

  Unity Catalog integration provides built-in governance, lineage tracking, cross-workspace feature sharing, and discovery for feature tables and models.

• Can Databricks Feature Store serve features in real-time?

  Yes, it provides on-demand feature computation for real-time applications and offers online feature serving endpoints with millisecond latency.

• How does Databricks Feature Store simplify client-side code?

  It handles all feature lookups and computation, significantly simplifying the client-side code required for feature retrieval and use.

• What kind of features can be stored in Databricks Feature Store?

  It supports common machine learning feature types, including dense and sparse vectors, tensors, embeddings (as ArrayType or MapType), and text (as StringType).

• How is feature governance managed?

  Feature governance, including access control to feature tables, is managed through Unity Catalog.

• What is point-in-time correctness in Databricks Feature Store?

  It ensures that when creating a training dataset, feature values reflect the state as of the time a label observation was recorded, preventing data leakage.

• Does Databricks Feature Store support feature sharing?

  Yes, it enables cross-workspace feature sharing and discovery, allowing teams to collaborate on feature development and usage.

• What are feature serving endpoints?

  These are endpoints that serve feature data to models and applications, both within and outside of Databricks, for real-time inference.