Writing Pipelines

Supported Formats

PatternOps supports three configuration formats for pipeline definitions:

Format	Extension	Best For
YAML	.yaml, .yml	Human-readable, concise — recommended default
JSON	.json	Programmatic generation, API responses
HOCON	.conf	Typesafe/Lightbend ecosystems, variable substitution, includes

The parser auto-detects the format based on content structure. You can also use format-specific methods (parseYaml(), parseJson(), parseHocon()) for explicit parsing.

See Pipeline Examples for the same pipeline expressed in all three formats.

Pipeline Structure

Every pipeline definition has this structure:

name: <pipeline-name>           # Required: unique identifier
tenancy: <tenancy-id>           # Required: organisation boundary
namespace: <namespace>          # Required: logical grouping
dataset: <dataset-name>         # Required: target dataset
executionMode: <mode>           # Required: batch | streaming | cdc | hybrid
platformProfile: <profile>     # Optional: databricks | snowflake | aws-native | azure-native
schedule: "<cron-expression>"   # Optional: execution schedule

stages:                         # Required: ordered list of stages
  - name: <stage-name>
    type: <stage-type>
    capability: <capability-ref>
    provider: <provider-ref>    # Optional: explicit provider
    timeout: <ISO-8601>         # Required: max execution time
    config: {}                  # Optional: stage configuration
    extensions: []              # Optional: extension hooks
    retryPolicy:                # Optional: retry configuration
      maxAttempts: 3
      backoff: PT5S
      strategy: exponential
    transformation:             # Optional: dual-mode transformation
      spark: { ... }
      sql: { ... }

policies: []                    # Pipeline-level policies
extensions: []                  # Pipeline-level extensions

Required Fields

Field	Type	Description
name	string	Pipeline identifier (non-blank)
tenancy	string	Top-level organisational boundary
namespace	string	Logical grouping within tenancy
dataset	string	Target dataset name
executionMode	enum	batch, streaming, cdc, or hybrid

Stage Required Fields

Field	Type	Description
name	string	Stage identifier (unique within pipeline)
type	enum	Stage type (see Stage Types)
capability	string	Capability reference for provider resolution
timeout	duration	Maximum execution time (ISO-8601, e.g., PT5M)

Optional Fields

Field	Default	Description
platformProfile	none	Target compute platform
schedule	none	Cron expression for scheduling
provider	auto-resolved	Explicit provider reference
config	{}	Stage-specific configuration
extensions	[]	Extension hook references
retryPolicy	none	Retry configuration
transformation	none	Dual-mode transformation definition

Minimal Pipeline

The simplest valid pipeline:

name: minimal-pipeline
tenancy: my-org
namespace: dev
dataset: test-data
executionMode: batch
stages: []
policies: []
extensions: []

Complete Example

name: order-processing-pipeline
tenancy: acme-corp
namespace: production
dataset: customer-orders
executionMode: batch
platformProfile: databricks
schedule: "0 6 * * *"

stages:
  - name: acquire-orders
    type: source
    capability: source-acquisition
    provider: jdbc-provider
    timeout: PT10M
    config:
      mode: incremental
      watermark: last_modified
      batchSize: 10000

  - name: classify-pii
    type: security
    capability: security
    timeout: PT3M

  - name: validate-quality
    type: quality
    capability: data-quality
    timeout: PT5M

  - name: transform-orders
    type: transformation
    capability: transformation
    timeout: PT15M
    retryPolicy:
      maxAttempts: 3
      backoff: PT10S
      strategy: exponential
    transformation:
      spark:
        style: dataframe
        reference: com.acme.transforms.OrderAggregation
        config:
          outputPartitions: 100
      sql:
        style: sql
        reference: transforms/order_aggregation.sql

  - name: publish-results
    type: publish
    capability: storage
    timeout: PT5M

policies: []
extensions: []

JSON Format

The same pipeline in JSON:

{
  "name": "order-processing-pipeline",
  "tenancy": "acme-corp",
  "namespace": "production",
  "dataset": "customer-orders",
  "executionMode": "batch",
  "platformProfile": "databricks",
  "stages": [
    {
      "name": "acquire-orders",
      "type": "source",
      "capability": "source-acquisition",
      "timeout": "PT10M"
    }
  ],
  "policies": [],
  "extensions": []
}

HOCON Format

The same pipeline in HOCON:

name = "order-processing-pipeline"
tenancy = "acme-corp"
namespace = "production"
dataset = "customer-orders"
executionMode = "batch"
platformProfile = "databricks"
schedule = "0 6 * * *"

stages = [
  {
    name = "acquire-orders"
    type = "source"
    capability = "source-acquisition"
    provider = "jdbc-provider"
    timeout = "PT10M"
    config {
      mode = "incremental"
      watermark = "last_modified"
      batchSize = 10000
    }
  }
]

policies = []
extensions = []

HOCON features

HOCON supports variable substitution (${ENV_VAR}), file includes (include "common.conf"), and object merging — useful for complex configurations with shared settings.

Configuration Tips

Use meaningful stage names

Stage names should describe the action: acquire-orders, validate-quality, transform-orders.

Set appropriate timeouts

Timeouts prevent runaway stages. Use realistic values based on expected data volumes.

Use retry policies for transient failures

Network issues, temporary unavailability — configure retries with exponential backoff.

Never put provider-specific config in pipelines

Spark configs, Airflow operators, cloud-specific references belong in Provider implementations, not pipeline definitions.