Pipeline Definition Model

The Pipeline Definition Model (PDM) is the declarative schema that expresses pipeline structure. It is the entry point for all pipeline operations.

Responsibilities

1. Parse configuration files (YAML/JSON) into validated Pipeline records
2. Validate against the declarative schema (reject provider-specific constructs)
3. Resolve capability and provider bindings
4. Print (serialize) Pipeline objects back to configuration format
5. Support round-trip serialization: parse(print(pipeline)) ≡ pipeline

Interface

public interface PipelineDefinitionModel {

    ValidationResult<Pipeline> parse(String config);
    ValidationResult<Pipeline> validate(Pipeline pipeline);
    ResolutionResult resolveBindings(Pipeline pipeline, ProviderRegistry registry);
    String print(Pipeline pipeline);
}

Pipeline Record

public record Pipeline(
    String name,              // Pipeline identifier
    String tenancy,           // Top-level org boundary
    String namespace,         // Logical grouping within tenancy
    String dataset,           // Target dataset
    ExecutionMode executionMode,  // batch | streaming | cdc | hybrid
    List<Stage> stages,       // Ordered processing stages
    Optional<String> schedule,
    List<Map<String, Object>> policies,
    List<Map<String, Object>> extensions,
    Optional<ComputePlatformProfile> platformProfile
) { }

Parsing

The ConfigurationParser auto-detects format (YAML vs JSON) and converts to Pipeline records:

ConfigurationParser parser = new ConfigurationParser();

// Auto-detect format
ValidationResult<Pipeline> result = parser.parse(configString);

// Or be explicit
ValidationResult<Pipeline> yamlResult = parser.parseYaml(yamlString);
ValidationResult<Pipeline> jsonResult = parser.parseJson(jsonString);

Format Detection

• Starts with { or [ → JSON
• Otherwise → YAML

Enum Handling

Enums are case-insensitive and support hyphens:

Input	Parsed As
batch, BATCH, Batch	ExecutionMode.BATCH
aws-native, AWS_NATIVE	ComputePlatformProfile.AWS_NATIVE
source, SOURCE	StageType.SOURCE

Duration Format

Timeouts and backoff durations use ISO-8601 format:

Input	Duration
PT5M	5 minutes
PT30S	30 seconds
PT1H	1 hour
PT2H30M	2 hours 30 minutes

Validation

The PipelineValidator checks:

Rule	Violation Example
Required fields present	pipeline.name is required and must not be blank
Stage names unique	pipeline.stages[1].name: duplicate stage name 'source'
No provider-specific constructs	pipeline.stages[0].config.sparkConf: contains provider-specific construct
RetryPolicy range [1, 10]	pipeline.stages[0].retryPolicy.maxAttempts: must be between 1 and 10
DualMode has at least one mode	pipeline.stages[0].transformation: at least one mode must be defined

Every violation includes the full property path for precise error location.

Binding Resolution

After parsing and validation, resolve capability/provider bindings:

ResolutionResult resolution = model.resolveBindings(pipeline, registry);

if (!resolution.resolved()) {
    // Each error identifies the stage and unresolved reference
    // "Stage 'acquire-data': unresolved capability 'source-acquisition'"
    // "Stage 'transform': unresolved provider reference 'spark-provider'"
}

Resolution happens at definition time, before execution begins.

Printing (Serialization)

ConfigurationPrinter printer = new ConfigurationPrinter();

String yaml = printer.printYaml(pipeline);  // YAML output
String json = printer.printJson(pipeline);  // JSON output

Serialization Rules

• Enums → lowercase hyphenated (AWS_NATIVE → aws-native)
• Duration → ISO-8601 (Duration.ofMinutes(5) → PT5M)
• Optional fields → omitted when empty
• Collections → empty arrays when no items

Round-Trip Guarantee

PatternOps guarantees semantic equivalence on round-trip:

Pipeline original = model.parse(yaml).value();
String printed = model.print(original);
Pipeline reparsed = model.parse(printed).value();

// original and reparsed are semantically equivalent
assert original.name().equals(reparsed.name());
assert original.stages().size() == reparsed.stages().size();
// ... all fields match

This property is verified by property-based tests (jqwik) with 200+ random pipelines per test run.