Transformation Capability¶

The Transformation capability handles data transformation with dual-mode support and field-level lineage.

Contract¶

public interface TransformationContract extends BaseProviderContract {
    ValidationResult validateTransformation(TransformationDefinition definition);
    ExecutionResult execute(TransformationDefinition definition, String inputRef, String outputRef);
    Explanation explain(TransformationDefinition definition);
    FieldLevelLineage lineage(TransformationDefinition definition);
    TransformationDefinition optimise(TransformationDefinition definition);
}

Operations¶

Operation	Purpose
`validateTransformation`	Check definition correctness before execution
`execute`	Run the transformation on input data
`explain`	Show execution plan without running
`lineage`	Compute field-level lineage mappings
`optimise`	Suggest optimised version of the definition

Transformation Styles¶

Style	Description
`sql`	Pure SQL query
`dataframe`	Spark DataFrame/Dataset API
`stream`	Streaming transformation (windowed)
`rule_based`	Business rule engine
`ml_based`	ML model inference
`ai_generated`	AI-generated transformation logic

Pipeline Configuration¶

- name: transform-data
  type: transformation
  capability: transformation
  timeout: PT15M
  transformation:
    spark:
      style: dataframe
      reference: com.acme.transforms.Enrichment
      config:
        outputPartitions: 100
    sql:
      style: sql
      reference: transforms/enrichment.sql

Field-Level Lineage¶

The lineage() operation returns mappings from source to destination fields:

record FieldMapping(
    String sourceField,
    String destinationField,
    String transformation  // Optional: description of the transformation applied
) { }

This integrates with OpenLineage for interoperability with Marquez, Atlan, and DataHub.

Execution Result¶

record ExecutionResult(
    boolean success,
    String outputRef,
    long recordsRead,
    long recordsWritten,
    long durationMs,
    Map<String, Object> metadata
) { }