Validation Rules¶

The PipelineValidator enforces schema rules to ensure pipeline definitions are correct, complete, and technology-free.

Validation Categories¶

1. Required Field Validation¶

Field	Rule	Violation
`pipeline.name`	Non-null, non-blank	`pipeline.name is required and must not be blank`
`pipeline.tenancy`	Non-null, non-blank	`pipeline.tenancy is required and must not be blank`
`pipeline.namespace`	Non-null, non-blank	`pipeline.namespace is required and must not be blank`
`pipeline.dataset`	Non-null, non-blank	`pipeline.dataset is required and must not be blank`
`pipeline.executionMode`	Non-null	`pipeline.executionMode is required`
`stage.name`	Non-null, non-blank	`pipeline.stages[N].name is required`
`stage.type`	Non-null	`pipeline.stages[N].type is required`
`stage.capability`	Non-null, non-blank	`pipeline.stages[N].capability is required`
`stage.timeout`	Non-null, valid ISO-8601	`pipeline.stages[N].timeout is required`

2. Stage Name Uniqueness¶

Stage names must be unique within a pipeline:

pipeline.stages[1].name: duplicate stage name 'source-stage'

3. RetryPolicy Range¶

maxAttempts must be between 1 and 10:

pipeline.stages[0].retryPolicy.maxAttempts: must be between 1 and 10, got 15

4. DualModeTransformation Completeness¶

At least one mode (spark or sql) must be defined:

pipeline.stages[0].transformation: at least one transformation mode (spark or sql) must be defined

5. Provider-Specific Construct Detection¶

The validator rejects any provider-specific configuration keys:

Spark-Specific¶

Key	Violation
`sparkConf`	`contains provider-specific construct (Spark-specific configuration)`
`spark.*` (any prefix)	`contains provider-specific construct (Spark-specific configuration key)`
`numExecutors`	`contains provider-specific construct (Spark-specific configuration)`
`driverMemory`	`contains provider-specific construct (Spark-specific configuration)`
`executorMemory`	`contains provider-specific construct (Spark-specific configuration)`

Airflow-Specific¶

Key	Violation
`dag_id`	`contains provider-specific construct (Airflow-specific configuration)`
`task_id`	`contains provider-specific construct (Airflow-specific configuration)`
`operator`	`contains provider-specific construct (Airflow-specific configuration)`
`depends_on_past`	`contains provider-specific construct (Airflow-specific configuration)`
`trigger_rule`	`contains provider-specific construct (Airflow-specific configuration)`

Flink-Specific¶

Key	Violation
`flinkConf`	`contains provider-specific construct (Flink-specific configuration)`
`parallelism`	`contains provider-specific construct (Flink-specific configuration)`
`checkpointInterval`	`contains provider-specific construct (Flink-specific configuration)`

AWS-Specific¶

Key	Violation
`s3Bucket`	`contains provider-specific construct (AWS-specific configuration)`
`lambdaArn`	`contains provider-specific construct (AWS-specific configuration)`
`glueJobName`	`contains provider-specific construct (AWS-specific configuration)`

Databricks-Specific¶

Key	Violation
`clusterSpec`	`contains provider-specific construct (Databricks-specific configuration)`
`notebookPath`	`contains provider-specific construct (Databricks-specific configuration)`
`dbfsPath`	`contains provider-specific construct (Databricks-specific configuration)`

Orchestrator-Specific¶

Key	Violation	Alternative
`cron_schedule`	`contains provider-specific construct (Orchestrator-specific)`	Use `schedule` field
`retry_delay`	`contains provider-specific construct (Orchestrator-specific)`	Use `retryPolicy` field

6. Nested Detection¶

Provider-specific constructs are detected recursively in nested config maps:

config:
  settings:
    sparkConf:          # ← Detected even when nested
      key: value

Violation: pipeline.stages[0].config.settings.sparkConf: contains provider-specific construct

Violation Format¶

Every violation includes the full property path:

pipeline.stages[0].config.sparkConf: contains provider-specific construct (Spark-specific configuration)
│                │       │
│                │       └── The offending key
│                └── Stage index
└── Root path

Multiple Violations¶

The validator collects all violations before returning — it does not short-circuit on the first error:

ValidationResult<Pipeline> result = validator.validate(pipeline);
// result.violations() may contain multiple entries:
// - pipeline.stages[1].name: duplicate stage name 'source'
// - pipeline.stages[0].config.sparkConf: contains provider-specific construct
// - pipeline.stages[0].config.dag_id: contains provider-specific construct

Programmatic Usage¶

PipelineValidator validator = new PipelineValidator();
ValidationResult<Pipeline> result = validator.validate(pipeline);

if (result.valid()) {
    Pipeline validated = result.value();
    // Proceed with binding resolution and execution
} else {
    result.violations().forEach(violation -> {
        log.error("Validation error: {}", violation);
    });
}