feast.dqm.profilers package

Submodules

feast.dqm.profilers.ge_profiler module

class feast.dqm.profilers.ge_profiler.GEProfile(expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite)[source]

Bases: feast.dqm.profilers.profiler.Profile

GEProfile is an implementation of abstract Profile for integration with Great Expectations. It executes validation by applying expectations from ExpectationSuite instance to a given dataset.

expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite
classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfile) → feast.dqm.profilers.ge_profiler.GEProfile[source]
to_proto()[source]
validate(df: pandas.core.frame.DataFrame) → feast.dqm.profilers.ge_profiler.GEValidationReport[source]

Validate provided dataframe against GE expectation suite. 1. Pandas dataframe is converted into PandasDataset (GE type) 2. Some fixes applied to the data to avoid crashes inside GE (see _prepare_dataset) 3. Each expectation from ExpectationSuite instance tested against resulting dataset

Return GEValidationReport, which parses great expectation’s schema into list of generic ValidationErrors.

class feast.dqm.profilers.ge_profiler.GEProfiler(user_defined_profiler: Callable[[pandas.core.frame.DataFrame], great_expectations.core.expectation_suite.ExpectationSuite], with_feature_metadata: bool = False)[source]

Bases: feast.dqm.profilers.profiler.Profiler

GEProfiler is an implementation of abstract Profiler for integration with Great Expectations. It wraps around user defined profiler that should accept dataset (in a form of pandas dataframe) and return ExpectationSuite.

analyze_dataset(df: pandas.core.frame.DataFrame) → feast.dqm.profilers.profiler.Profile[source]

Generate GEProfile with ExpectationSuite (set of expectations) from a given pandas dataframe by applying user defined profiler.

Some fixes are also applied to the dataset (see _prepare_dataset function) to make it compatible with GE.

Return GEProfile

classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfiler) → feast.dqm.profilers.ge_profiler.GEProfiler[source]
to_proto()[source]
class feast.dqm.profilers.ge_profiler.GEValidationReport(validation_result: Dict[Any, Any])[source]

Bases: feast.dqm.profilers.profiler.ValidationReport

property errors: List[feast.dqm.profilers.profiler.ValidationError]

Return list of ValidationErrors if validation failed (is_success = false)

property is_success: bool

Return whether validation was successful

feast.dqm.profilers.ge_profiler.ge_profiler(*args, with_feature_metadata=False)[source]

feast.dqm.profilers.profiler module

class feast.dqm.profilers.profiler.Profile[source]

Bases: object

abstract classmethod from_proto(proto) → feast.dqm.profilers.profiler.Profile[source]
abstract to_proto()[source]
abstract validate(dataset: pandas.core.frame.DataFrame) → feast.dqm.profilers.profiler.ValidationReport[source]

Run set of rules / expectations from current profile against given dataset.

Return ValidationReport

class feast.dqm.profilers.profiler.Profiler[source]

Bases: object

abstract analyze_dataset(dataset: pandas.core.frame.DataFrame) → feast.dqm.profilers.profiler.Profile[source]

Generate Profile object with dataset’s characteristics (with rules / expectations) from given dataset (as pandas dataframe).

abstract classmethod from_proto(proto) → feast.dqm.profilers.profiler.Profiler[source]
abstract to_proto()[source]
class feast.dqm.profilers.profiler.ValidationError(check_name: str, column_name: str, check_config: Optional[Any] = None, missing_count: Optional[int] = None, missing_percent: Optional[float] = None, observed_value: Optional[float] = None, unexpected_count: Optional[int] = None, unexpected_percent: Optional[float] = None)[source]

Bases: object

check_config: Optional[Any]
check_name: str
column_name: str
missing_count: Optional[int]
missing_percent: Optional[float]
observed_value: Optional[float]
to_dict()[source]
unexpected_count: Optional[int]
unexpected_percent: Optional[float]
class feast.dqm.profilers.profiler.ValidationReport[source]

Bases: object

abstract property errors: List[feast.dqm.profilers.profiler.ValidationError]

Return list of ValidationErrors if validation failed (is_success = false)

abstract property is_success: bool

Return whether validation was successful

Module contents