feast.infra.materialization.contrib.bytewax package

Submodules

feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine.BytewaxMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]

Bases: feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine

materialize(registry: feast.infra.registry.base_registry.BaseRegistry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) List[feast.infra.materialization.batch_materialization_engine.MaterializationJob][source]

Materialize data from the offline store to the online store for this feature repo.

Parameters
  • registry – The registry for the current feature store.

  • tasks – A list of individual materialization tasks.

Returns

A list of materialization jobs representing each task.

teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]

This method ensures that any infrastructure or resources set up by ``update()``are torn down.

update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]

This method ensures that any necessary infrastructure or resources needed by the engine are set up ahead of materialization.

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine.BytewaxMaterializationEngineConfig(*, type: Literal['bytewax'] = 'bytewax', namespace: pydantic.types.StrictStr = 'default', image: pydantic.types.StrictStr = 'bytewax/bytewax-feast:latest', env: List[dict] = [])[source]

Bases: feast.repo_config.FeastConfigBaseModel

Batch Materialization Engine config for Bytewax

env: List[dict]

(optional) A list of environment variables to set in the created Kubernetes pods. These environment variables can be used to reference Kubernetes secrets.

image: pydantic.types.StrictStr

(optional) The container image to use when running the materialization job.

namespace: pydantic.types.StrictStr

(optional) The namespace in Kubernetes to use when creating services, configuration maps and jobs.

type: Literal['bytewax']

Materialization type selector

feast.infra.materialization.contrib.bytewax.bytewax_materialization_job

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_job.BytewaxMaterializationJob(job_id, namespace, error: Optional[BaseException] = None)[source]

Bases: feast.infra.materialization.batch_materialization_engine.MaterializationJob

error()[source]
job_id()[source]
should_be_retried()[source]
status()[source]
task: feast.infra.materialization.batch_materialization_engine.MaterializationTask
url()[source]

Module contents

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationDataflow(config: feast.repo_config.RepoConfig, feature_view: feast.feature_view.FeatureView, paths: List[str])[source]

Bases: object

input_builder(worker_index, worker_count, resume_epoch)[source]
output_builder(worker_index, worker_count)[source]
process_path(path)[source]
class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationEngine(*, repo_config: feast.repo_config.RepoConfig, offline_store: feast.infra.offline_stores.offline_store.OfflineStore, online_store: feast.infra.online_stores.online_store.OnlineStore, **kwargs)[source]

Bases: feast.infra.materialization.batch_materialization_engine.BatchMaterializationEngine

materialize(registry: feast.infra.registry.base_registry.BaseRegistry, tasks: List[feast.infra.materialization.batch_materialization_engine.MaterializationTask]) List[feast.infra.materialization.batch_materialization_engine.MaterializationJob][source]

Materialize data from the offline store to the online store for this feature repo.

Parameters
  • registry – The registry for the current feature store.

  • tasks – A list of individual materialization tasks.

Returns

A list of materialization jobs representing each task.

teardown_infra(project: str, fvs: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities: Sequence[feast.entity.Entity])[source]

This method ensures that any infrastructure or resources set up by ``update()``are torn down.

update(project: str, views_to_delete: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], views_to_keep: Sequence[Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]], entities_to_delete: Sequence[feast.entity.Entity], entities_to_keep: Sequence[feast.entity.Entity])[source]

This method ensures that any necessary infrastructure or resources needed by the engine are set up ahead of materialization.

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationEngineConfig(*, type: Literal['bytewax'] = 'bytewax', namespace: pydantic.types.StrictStr = 'default', image: pydantic.types.StrictStr = 'bytewax/bytewax-feast:latest', env: List[dict] = [])[source]

Bases: feast.repo_config.FeastConfigBaseModel

Batch Materialization Engine config for Bytewax

env: List[dict]

(optional) A list of environment variables to set in the created Kubernetes pods. These environment variables can be used to reference Kubernetes secrets.

image: pydantic.types.StrictStr

(optional) The container image to use when running the materialization job.

namespace: pydantic.types.StrictStr

(optional) The namespace in Kubernetes to use when creating services, configuration maps and jobs.

type: Literal['bytewax']

Materialization type selector

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationJob(job_id, namespace, error: Optional[BaseException] = None)[source]

Bases: feast.infra.materialization.batch_materialization_engine.MaterializationJob

error()[source]
job_id()[source]
should_be_retried()[source]
status()[source]
task: feast.infra.materialization.batch_materialization_engine.MaterializationTask
url()[source]
class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationTask(project, feature_view, start_date, end_date, tqdm)[source]

Bases: feast.infra.materialization.batch_materialization_engine.MaterializationTask

end_time: datetime.datetime
feature_view: Union[feast.batch_feature_view.BatchFeatureView, feast.stream_feature_view.StreamFeatureView, feast.feature_view.FeatureView]
project: str
start_time: datetime.datetime
tqdm_builder: Callable[[int], tqdm.std.tqdm]