feast.loaders package

Submodules

feast.loaders.abstract_producer module

class feast.loaders.abstract_producer.AbstractProducer(brokers: str, row_count: int, disable_progress_bar: bool)

Bases: object

Abstract class for Kafka producers

flush(timeout: int)

print_results() → None

Print ingestion statistics.

Returns

None

Return type

None

produce(topic: str, data: bytes)

class feast.loaders.abstract_producer.ConfluentProducer(brokers: str, row_count: int, disable_progress_bar: bool)

Bases: feast.loaders.abstract_producer.AbstractProducer

Concrete implementation of Confluent Kafka producer (confluent-kafka)

flush(timeout: Optional[int])

Generic flush that implements confluent-kafka’s flush method.

Parameters

timeout (Optional[int]) – Timeout in seconds to wait for completion.

Returns

Number of messages still in queue.

Return type

int

produce(topic: str, value: bytes) → None

Generic produce that implements confluent-kafka’s produce method to push a byte encoded object into a Kafka topic.

Parameters

• topic (str) – Kafka topic.

• value (bytes) – Byte encoded object.

Returns

None.

Return type

None

class feast.loaders.abstract_producer.KafkaPythonProducer(brokers: str, row_count: int, disable_progress_bar: bool)

Bases: feast.loaders.abstract_producer.AbstractProducer

Concrete implementation of Python Kafka producer (kafka-python)

flush(timeout: Optional[int])

Generic flush that implements kafka-python’s flush method.

Parameters

timeout (Optional[int]) – timeout in seconds to wait for completion.

Returns

None

Raises

KafkaTimeoutError – failure to flush buffered records within the provided timeout

produce(topic: str, value: bytes)

Generic produce that implements kafka-python’s send method to push a byte encoded object into a Kafka topic.

Parameters

• topic (str) – Kafka topic.

• value (bytes) – Byte encoded object.

Returns

resolves to RecordMetadata

Return type

FutureRecordMetadata

Raises

KafkaTimeoutError – if unable to fetch topic metadata, or unable to obtain memory buffer prior to configured max_block_ms

feast.loaders.abstract_producer.get_producer(brokers: str, row_count: int, disable_progress_bar: bool) → Union[feast.loaders.abstract_producer.ConfluentProducer, feast.loaders.abstract_producer.KafkaPythonProducer]

Simple context helper function that returns a AbstractProducer object when invoked.

This helper function will try to import confluent-kafka as a producer first.

This helper function will fallback to kafka-python if it fails to import confluent-kafka.

Parameters

• brokers (str) – Kafka broker information with hostname and port.

• row_count (int) – Number of rows in table

Returns

Concrete implementation of a Kafka producer. Ig can be:

• confluent-kafka producer

• kafka-python producer

Return type

Union[ConfluentProducer, KafkaPythonProducer]

feast.loaders.file module

feast.loaders.ingest module

feast.loaders.yaml module

feast.loaders.yaml.yaml_loader(yml, load_single=False)

Loads one or more Feast resources from a YAML path or string. Multiple resources can be divided by three hyphens ‘—’

Parameters

• yml – A path ending in .yaml or .yml, or a YAML string

• load_single – Expect only a single YAML resource, fail otherwise

Returns

Either a single YAML dictionary or a list of YAML dictionaries

Module contents