Data Quality Capability

The Data Quality capability provides profiling, validation, monitoring, and anomaly detection.

Contract

public interface DataQualityContract extends BaseProviderContract {
    ProfileResult profile(String dataset, Map<String, Object> options);
    ValidateResult validate(String dataset, List<QualityRule> rules);
    MonitorResult monitor(String dataset, Baseline baseline);
    TrendResult trend(String dataset, String metric, TimeRange range);
    AnomalyResult detectAnomaly(String dataset, String metric, double sensitivity);
}

Operations

Operation	Purpose	Output
profile	Statistical summary per field	null count, distinct count, min, max, mean, std dev
validate	Check data against quality rules	pass/fail per rule, overall quality score
monitor	Compare against baseline	drift detection
trend	Track metric over time	time series data
detect_anomaly	Identify statistical anomalies	anomaly score, expected range

Validation Result

record ValidateResult(
    long totalRecords,
    long passingRecords,
    long failingRecords,
    List<RuleResult> ruleResults,
    double qualityScore  // 0.0 to 1.0
) { }

record RuleResult(
    String ruleId,
    long passCount,
    long failCount,
    double passRate
) { }

Pipeline Configuration

- name: check-quality
  type: quality
  capability: data-quality
  timeout: PT5M

Failure Actions

When quality validation fails, the Policy Engine determines the action:

Action	Behaviour
quarantine	Route failing records to quarantine destination; continue processing passing records
halt	Stop pipeline execution immediately
alert	Continue processing but send notifications

Anomaly Detection

When an anomaly is detected, the Quality Provider reports:

• Metric name
• Expected range
• Observed value
• Anomaly score
• Detection timestamp

This is published through the Echo service for monitoring dashboards.

Example Providers

Provider	Strengths
Great Expectations	Rule-based validation, profiling
Soda	SQL-based checks, monitoring
Monte Carlo	Anomaly detection, observability
Custom	Organisation-specific rules