feast.infra.materialization package
Subpackages
Submodules
feast.infra.materialization.batch_materialization_engine module
- class feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]
Bases:
abc.ABC
The interface that Feast uses to control the compute system that handles batch materialization.
- abstract materialize(registry: feast.infra.registry.base_registry.BaseRegistry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) List[feast.infra.materialization.batch_materialization_engine.MaterializationJob] [source]
Materialize data from the offline store to the online store for this feature repo.
- Parameters
registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.
- Returns
A list of materialization jobs representing each task.
- abstract teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]
Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- abstract update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]
Prepares cloud resources required for batch materialization for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
- class feast.infra.materialization.batch_materialization_engine.MaterializationJob[source]
Bases:
abc.ABC
A MaterializationJob represents an ongoing or executed process that materializes data as per the definition of a materialization task.
- abstract error() Optional[BaseException] [source]
- class feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus(value)[source]
Bases:
enum.Enum
An enumeration.
- AVAILABLE = 3
- CANCELLED = 6
- CANCELLING = 5
- ERROR = 4
- RUNNING = 2
- SUCCEEDED = 7
- WAITING = 1
- class feast.infra.materialization.batch_materialization_engine.MaterializationTask(project: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView], start_time: datetime.datetime, end_time: datetime.datetime, tqdm_builder: Callable[[int], tqdm.std.tqdm])[source]
Bases:
object
A MaterializationTask represents a unit of data that needs to be materialized from an offline store to an online store.
- end_time: datetime.datetime
- feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]
- start_time: datetime.datetime
feast.infra.materialization.local_engine module
- class feast.infra.materialization.local_engine.LocalMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]
Bases:
feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine
- materialize(registry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) List[feast.infra.materialization.batch_materialization_engine.MaterializationJob] [source]
Materialize data from the offline store to the online store for this feature repo.
- Parameters
registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.
- Returns
A list of materialization jobs representing each task.
- teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]
Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]
Prepares cloud resources required for batch materialization for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
- class feast.infra.materialization.local_engine.LocalMaterializationEngineConfig(*, type: Literal['local'] = 'local')[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Batch Materialization Engine config for local in-process engine
- type: Literal['local']
Type selector
- class feast.infra.materialization.local_engine.LocalMaterializationJob(job_id: str, status: feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus, error: Union[BaseException, NoneType] = None)[source]
Bases:
feast.infra.materialization.batch_materialization_engine.MaterializationJob
- error() Optional[BaseException] [source]
feast.infra.materialization.snowflake_engine module
- class feast.infra.materialization.snowflake_engine.SnowflakeMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]
Bases:
feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine
- generate_snowflake_materialization_query(repo_config: feast.repo_config.RepoConfig, fv_latest_mapped_values_sql: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.feature_view.FeatureView], project: str) str [source]
- materialize(registry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) List[feast.infra.materialization.batch_materialization_engine.MaterializationJob] [source]
Materialize data from the offline store to the online store for this feature repo.
- Parameters
registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.
- Returns
A list of materialization jobs representing each task.
- materialize_to_external_online_store(repo_config: feast.repo_config.RepoConfig, materialization_sql: str, feature_view: Union[feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView], tqdm_builder: Callable[[int], tqdm.std.tqdm]) None [source]
- materialize_to_snowflake_online_store(repo_config: feast.repo_config.RepoConfig, materialization_sql: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.feature_view.FeatureView], project: str) None [source]
- teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]
Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.
- update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]
Prepares cloud resources required for batch materialization for the specified set of Feast objects.
- Parameters
project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.
- class feast.infra.materialization.snowflake_engine.SnowflakeMaterializationEngineConfig(*, type: Literal['snowflake.engine'] = 'snowflake.engine', config_path: Optional[str] = '/home/docs/.snowsql/config', account: Optional[str] = None, user: Optional[str] = None, password: Optional[str] = None, role: Optional[str] = None, warehouse: Optional[str] = None, authenticator: Optional[str] = None, database: pydantic.types.StrictStr, schema: Optional[str] = 'PUBLIC')[source]
Bases:
feast.repo_config.FeastConfigBaseModel
Batch Materialization Engine config for Snowflake Snowpark Python UDFs
- database: pydantic.types.StrictStr
Snowflake database name
- type: Literal['snowflake.engine']
Type selector
- class feast.infra.materialization.snowflake_engine.SnowflakeMaterializationJob(job_id: str, status: feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus, error: Union[BaseException, NoneType] = None)[source]
Bases:
feast.infra.materialization.batch_materialization_engine.MaterializationJob
- error() Optional[BaseException] [source]