feast.infra.materialization.contrib.bytewax package

Submodules

feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine.BytewaxMaterializationEngine(*, repo_config: RepoConfig, offline_store: OfflineStore, online_store: OnlineStore, **kwargs)

Bases: BatchMaterializationEngine

materialize(registry: BaseRegistry, tasks: List[MaterializationTask]) → List[MaterializationJob]

Materialize data from the offline store to the online store for this feature repo.

Parameters:

• registry – The registry for the current feature store.

• tasks – A list of individual materialization tasks.

Returns:

A list of materialization jobs representing each task.

teardown_infra(project: str, fvs: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], entities: Sequence[Entity])

This method ensures that any infrastructure or resources set up by ``update()``are torn down.

update(project: str, views_to_delete: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], views_to_keep: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity])

This method ensures that any necessary infrastructure or resources needed by the engine are set up ahead of materialization.

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_engine.BytewaxMaterializationEngineConfig(*, type: Literal['bytewax'] = 'bytewax', namespace: StrictStr = 'default', image: StrictStr = 'bytewax/bytewax-feast:latest', env: List[dict] = [], image_pull_secrets: List[dict] = [], resources: dict = {}, service_account_name: StrictStr = '', annotations: dict = {}, include_security_context_capabilities: bool = True, labels: dict = {}, max_parallelism: int = 10)

Bases: FeastConfigBaseModel

Batch Materialization Engine config for Bytewax

annotations: dict

(optional) Annotations to apply to the job container. Useful for linking the service account to IAM roles, operational metadata, etc

env: List[dict]

(optional) A list of environment variables to set in the created Kubernetes pods. These environment variables can be used to reference Kubernetes secrets.

image: StrictStr

(optional) The container image to use when running the materialization job.

image_pull_secrets: List[dict]

(optional) The secrets to use when pulling the image to run for the materialization job

include_security_context_capabilities: bool

(optional) Include security context capabilities in the init and job container spec

labels: dict

(optional) additional labels to append to kubernetes objects

max_parallelism: int

(optional) Maximum number of pods (default 10) allowed to run in parallel per job

namespace: StrictStr

(optional) The namespace in Kubernetes to use when creating services, configuration maps and jobs.

resources: dict

(optional) The resource requests and limits for the materialization containers

service_account_name: StrictStr

(optional) The service account name to use when running the job

type: Literal['bytewax']

Materialization type selector

feast.infra.materialization.contrib.bytewax.bytewax_materialization_job

class feast.infra.materialization.contrib.bytewax.bytewax_materialization_job.BytewaxMaterializationJob(job_id, namespace, error: BaseException | None = None)

Bases: MaterializationJob

error()

job_id()

should_be_retried()

status()

task: MaterializationTask

url()

Module contents

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationDataflow(config: RepoConfig, feature_view: FeatureView, paths: List[str])

Bases: object

input_builder(worker_index, worker_count, _state)

output_builder(worker_index, worker_count)

process_path(path)

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationEngine(*, repo_config: RepoConfig, offline_store: OfflineStore, online_store: OnlineStore, **kwargs)

Bases: BatchMaterializationEngine

materialize(registry: BaseRegistry, tasks: List[MaterializationTask]) → List[MaterializationJob]

Materialize data from the offline store to the online store for this feature repo.

Parameters:

• registry – The registry for the current feature store.

• tasks – A list of individual materialization tasks.

Returns:

A list of materialization jobs representing each task.

teardown_infra(project: str, fvs: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], entities: Sequence[Entity])

This method ensures that any infrastructure or resources set up by ``update()``are torn down.

update(project: str, views_to_delete: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], views_to_keep: Sequence[BatchFeatureView | StreamFeatureView | FeatureView], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity])

This method ensures that any necessary infrastructure or resources needed by the engine are set up ahead of materialization.

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationEngineConfig(*, type: Literal['bytewax'] = 'bytewax', namespace: StrictStr = 'default', image: StrictStr = 'bytewax/bytewax-feast:latest', env: List[dict] = [], image_pull_secrets: List[dict] = [], resources: dict = {}, service_account_name: StrictStr = '', annotations: dict = {}, include_security_context_capabilities: bool = True, labels: dict = {}, max_parallelism: int = 10)

Bases: FeastConfigBaseModel

Batch Materialization Engine config for Bytewax

annotations: dict

(optional) Annotations to apply to the job container. Useful for linking the service account to IAM roles, operational metadata, etc

env: List[dict]

(optional) A list of environment variables to set in the created Kubernetes pods. These environment variables can be used to reference Kubernetes secrets.

image: StrictStr

(optional) The container image to use when running the materialization job.

image_pull_secrets: List[dict]

(optional) The secrets to use when pulling the image to run for the materialization job

include_security_context_capabilities: bool

(optional) Include security context capabilities in the init and job container spec

labels: dict

(optional) additional labels to append to kubernetes objects

max_parallelism: int

(optional) Maximum number of pods (default 10) allowed to run in parallel per job

namespace: StrictStr

(optional) The namespace in Kubernetes to use when creating services, configuration maps and jobs.

resources: dict

(optional) The resource requests and limits for the materialization containers

service_account_name: StrictStr

(optional) The service account name to use when running the job

type: Literal['bytewax']

Materialization type selector

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationJob(job_id, namespace, error: BaseException | None = None)

Bases: MaterializationJob

error()

job_id()

should_be_retried()

status()

task: MaterializationTask

url()

class feast.infra.materialization.contrib.bytewax.BytewaxMaterializationTask(project, feature_view, start_date, end_date, tqdm)

Bases: MaterializationTask

end_time: datetime

feature_view: BatchFeatureView | StreamFeatureView | FeatureView

project: str

start_time: datetime

tqdm_builder: Callable[[int], tqdm]