feast.infra.materialization package

Subpackages

feast.infra.materialization.lambda package

Submodules

feast.infra.materialization.batch_materialization_engine module

class feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]

Bases: abc.ABC

The interface that Feast uses to control the compute system that handles batch materialization.

abstract materialize(registry: feast.infra.registry.base_registry.BaseRegistry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) → List[feast.infra.materialization.batch_materialization_engine.MaterializationJob][source]

Materialize data from the offline store to the online store for this feature repo.

Parameters

registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.

Returns

A list of materialization jobs representing each task.

abstract teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]

Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.

abstract update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]

Prepares cloud resources required for batch materialization for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.

class feast.infra.materialization.batch_materialization_engine.MaterializationJob[source]

Bases: abc.ABC

A MaterializationJob represents an ongoing or executed process that materializes data as per the definition of a materialization task.

abstract error() → Optional[BaseException][source]

abstract job_id() → str[source]

abstract should_be_retried() → bool[source]

abstract status() → feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus[source]

task: feast.infra.materialization.batch_materialization_engine.MaterializationTask

abstract url() → Optional[str][source]

class feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus(value)[source]

Bases: enum.Enum

An enumeration.

AVAILABLE = 3

CANCELLED = 6

CANCELLING = 5

ERROR = 4

RUNNING = 2

SUCCEEDED = 7

WAITING = 1

class feast.infra.materialization.batch_materialization_engine.MaterializationTask(project: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView], start_time: datetime.datetime, end_time: datetime.datetime, tqdm_builder: Callable[[int], tqdm.std.tqdm])[source]

Bases: object

A MaterializationTask represents a unit of data that needs to be materialized from an offline store to an online store.

end_time: datetime.datetime

feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]

project: str

start_time: datetime.datetime

tqdm_builder: Callable[[int], tqdm.std.tqdm]

feast.infra.materialization.local_engine module

class feast.infra.materialization.local_engine.LocalMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]

Bases: feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine

materialize(registry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) → List[feast.infra.materialization.batch_materialization_engine.MaterializationJob][source]

Materialize data from the offline store to the online store for this feature repo.

Parameters

registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.

Returns

A list of materialization jobs representing each task.

teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]

Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.

update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]

Prepares cloud resources required for batch materialization for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.

class feast.infra.materialization.local_engine.LocalMaterializationEngineConfig(*, type: Literal['local'] = 'local')[source]

Bases: feast.repo_config.FeastConfigBaseModel

Batch Materialization Engine config for local in-process engine

type: Literal['local']: Type selector

class feast.infra.materialization.local_engine.LocalMaterializationJob(job_id: str, status: feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus, error: Union[BaseException, NoneType] = None)[source]

Bases: feast.infra.materialization.batch_materialization_engine.MaterializationJob

error() → Optional[BaseException][source]

job_id() → str[source]

should_be_retried() → bool[source]

status() → feast.infra.materialization.batch_materialization_engine.MaterializationJobStatus[source]

task: feast.infra.materialization.batch_materialization_engine.MaterializationTask

url() → Optional[str][source]

feast.infra.materialization.snowflake_engine module

class feast.infra.materialization.snowflake_engine.SnowflakeMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]

Bases: feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine

generate_snowflake_materialization_query(repo_config: feast.repo_config.RepoConfig, fv_latest_mapped_values_sql: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.feature_view.FeatureView], project: str) → str[source]

materialize(registry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) → List[feast.infra.materialization.batch_materialization_engine.MaterializationJob][source]

Materialize data from the offline store to the online store for this feature repo.

Parameters

registry – The registry for the current feature store.
tasks – A list of individual materialization tasks.

Returns

A list of materialization jobs representing each task.

materialize_to_external_online_store(repo_config: feast.repo_config.RepoConfig, materialization_sql: str, feature_view: Union[feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView], tqdm_builder: Callable[[int], tqdm.std.tqdm]) → None[source]

materialize_to_snowflake_online_store(repo_config: feast.repo_config.RepoConfig, materialization_sql: str, feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.feature_view.FeatureView], project: str) → None[source]

teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]

Tears down all cloud resources used by the materialization engine for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
fvs – Feature views whose corresponding infrastructure should be deleted.
entities – Entities whose corresponding infrastructure should be deleted.

update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]

Prepares cloud resources required for batch materialization for the specified set of Feast objects.

Parameters

project – Feast project to which the objects belong.
views_to_delete – Feature views whose corresponding infrastructure should be deleted.
views_to_keep – Feature views whose corresponding infrastructure should not be deleted, and may need to be updated.
entities_to_delete – Entities whose corresponding infrastructure should be deleted.
entities_to_keep – Entities whose corresponding infrastructure should not be deleted, and may need to be updated.

class feast.infra.materialization.snowflake_engine.SnowflakeMaterializationEngineConfig(*, type: Literal['snowflake.engine'] = 'snowflake.engine', config_path: Optional[str] = '/home/docs/.snowsql/config', account: Optional[str] = None, user: Optional[str] = None, password: Optional[str] = None, role: Optional[str] = None, warehouse: Optional[str] = None, authenticator: Optional[str] = None, database: pydantic.types.StrictStr, schema: Optional[str] = 'PUBLIC')[source]