feast.dqm.profilers package
Submodules
feast.dqm.profilers.ge_profiler module
- class feast.dqm.profilers.ge_profiler.GEProfile(expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite)[source]
Bases:
feast.dqm.profilers.profiler.Profile
GEProfile is an implementation of abstract Profile for integration with Great Expectations. It executes validation by applying expectations from ExpectationSuite instance to a given dataset.
- expectation_suite: great_expectations.core.expectation_suite.ExpectationSuite
- classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfile) feast.dqm.profilers.ge_profiler.GEProfile [source]
- validate(df: pandas.core.frame.DataFrame) feast.dqm.profilers.ge_profiler.GEValidationReport [source]
Validate provided dataframe against GE expectation suite. 1. Pandas dataframe is converted into PandasDataset (GE type) 2. Some fixes applied to the data to avoid crashes inside GE (see _prepare_dataset) 3. Each expectation from ExpectationSuite instance tested against resulting dataset
Return GEValidationReport, which parses great expectation’s schema into list of generic ValidationErrors.
- class feast.dqm.profilers.ge_profiler.GEProfiler(user_defined_profiler: Callable[[pandas.core.frame.DataFrame], great_expectations.core.expectation_suite.ExpectationSuite], with_feature_metadata: bool = False)[source]
Bases:
feast.dqm.profilers.profiler.Profiler
GEProfiler is an implementation of abstract Profiler for integration with Great Expectations. It wraps around user defined profiler that should accept dataset (in a form of pandas dataframe) and return ExpectationSuite.
- analyze_dataset(df: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.Profile [source]
Generate GEProfile with ExpectationSuite (set of expectations) from a given pandas dataframe by applying user defined profiler.
Some fixes are also applied to the dataset (see _prepare_dataset function) to make it compatible with GE.
Return GEProfile
- classmethod from_proto(proto: feast.core.ValidationProfile_pb2.GEValidationProfiler) feast.dqm.profilers.ge_profiler.GEProfiler [source]
- class feast.dqm.profilers.ge_profiler.GEValidationReport(validation_result: Dict[Any, Any])[source]
Bases:
feast.dqm.profilers.profiler.ValidationReport
- property errors: List[feast.dqm.profilers.profiler.ValidationError]
Return list of ValidationErrors if validation failed (is_success = false)
feast.dqm.profilers.profiler module
- class feast.dqm.profilers.profiler.Profile[source]
Bases:
object
- abstract classmethod from_proto(proto) feast.dqm.profilers.profiler.Profile [source]
- abstract validate(dataset: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.ValidationReport [source]
Run set of rules / expectations from current profile against given dataset.
Return ValidationReport
- class feast.dqm.profilers.profiler.Profiler[source]
Bases:
object
- abstract analyze_dataset(dataset: pandas.core.frame.DataFrame) feast.dqm.profilers.profiler.Profile [source]
Generate Profile object with dataset’s characteristics (with rules / expectations) from given dataset (as pandas dataframe).
- abstract classmethod from_proto(proto) feast.dqm.profilers.profiler.Profiler [source]
- class feast.dqm.profilers.profiler.ValidationError(check_name: str, column_name: str, check_config: Optional[Any] = None, missing_count: Optional[int] = None, missing_percent: Optional[float] = None, observed_value: Optional[float] = None, unexpected_count: Optional[int] = None, unexpected_percent: Optional[float] = None)[source]
Bases:
object
- check_config: Optional[Any]
- class feast.dqm.profilers.profiler.ValidationReport[source]
Bases:
object
- abstract property errors: List[feast.dqm.profilers.profiler.ValidationError]
Return list of ValidationErrors if validation failed (is_success = false)