feast.dqm.profilers package¶
Submodules¶
feast.dqm.profilers.ge_profiler module¶
- class feast.dqm.profilers.ge_profiler.GEProfile(expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite)[source]¶
Bases:
feast.dqm.profilers.profiler.Profile
GEProfile is an implementation of abstract Profile for integration with Great Expectations. It executes validation by applying expectations from ExpectationSuite instance to a given dataset.
- expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite¶
- classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfile) feast.dqm.profilers.ge_profiler.GEProfile [source]¶
- validate(df: pandas.core.frame.DataFrame) feast.dqm.profilers.ge_profiler.GEValidationReport [source]¶
Validate provided dataframe against GE expectation suite. 1. Pandas dataframe is converted into PandasDataset (GE type) 2. Some fixes applied to the data to avoid crashes inside GE (see _prepare_dataset) 3. Each expectation from ExpectationSuite instance tested against resulting dataset
Return GEValidationReport, which parses great expectation’s schema into list of generic ValidationErrors.
- class feast.dqm.profilers.ge_profiler.GEProfiler(user_defined_profiler: Callable[[pandas.core.frame.DataFrame], great_expectations.core.expectation_suite.ExpectationSuite], with_feature_metadata: bool = False)[source]¶
Bases:
feast.dqm.profilers.profiler.Profiler
GEProfiler is an implementation of abstract Profiler for integration with Great Expectations. It wraps around user defined profiler that should accept dataset (in a form of pandas dataframe) and return ExpectationSuite.
- analyze_dataset(df: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.Profile [source]¶
Generate GEProfile with ExpectationSuite (set of expectations) from a given pandas dataframe by applying user defined profiler.
Some fixes are also applied to the dataset (see _prepare_dataset function) to make it compatible with GE.
Return GEProfile
- classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfiler) feast.dqm.profilers.ge_profiler.GEProfiler [source]¶
- class feast.dqm.profilers.ge_profiler.GEValidationReport(validation_result: Dict[Any, Any])[source]¶
Bases:
feast.dqm.profilers.profiler.ValidationReport
- property errors: List[feast.dqm.profilers.profiler.ValidationError]¶
Return list of ValidationErrors if validation failed (is_success = false)
feast.dqm.profilers.profiler module¶
- class feast.dqm.profilers.profiler.Profile[source]¶
Bases:
object
- abstract classmethod from_proto(proto) feast.dqm.profilers.profiler.Profile [source]¶
- abstract validate(dataset: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.ValidationReport [source]¶
Run set of rules / expectations from current profile against given dataset.
Return ValidationReport
- class feast.dqm.profilers.profiler.Profiler[source]¶
Bases:
object
- abstract analyze_dataset(dataset: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.Profile [source]¶
Generate Profile object with dataset’s characteristics (with rules / expectations) from given dataset (as pandas dataframe).
- abstract classmethod from_proto(proto) feast.dqm.profilers.profiler.Profiler [source]¶
- class feast.dqm.profilers.profiler.ValidationError(check_name: str, column_name: str, check_config: Optional[Any] = None, missing_count: Optional[int] = None, missing_percent: Optional[float] = None, observed_value: Optional[float] = None, unexpected_count: Optional[int] = None, unexpected_percent: Optional[float] = None)[source]¶
Bases:
object
- check_config: Optional[Any]¶
- class feast.dqm.profilers.profiler.ValidationReport[source]¶
Bases:
object
- abstract property errors: List[feast.dqm.profilers.profiler.ValidationError]¶
Return list of ValidationErrors if validation failed (is_success = false)