Platform Profiles¶

Platform profiles declare the target compute environment, enabling automatic execution strategy selection.

Available Profiles¶

Profile	Value	Capabilities
Databricks	`databricks`	Spark API, SQL, Delta Lake
Snowflake	`snowflake`	SQL only, Snowpark
AWS Native	`aws-native`	EMR (Spark), Glue, Athena (SQL)
Azure Native	`azure-native`	Synapse Spark, Synapse SQL

Usage¶

name: my-pipeline
tenancy: acme-corp
namespace: production
dataset: orders
executionMode: batch
platformProfile: databricks   # ← declares target platform

Strategy Resolution¶

The platform profile drives execution strategy selection for transformation stages:

flowchart TD
    A[Stage with Dual-Mode Transformation] --> B{platformProfile}
    B -->|databricks| C{Spark API Available}
    B -->|snowflake| D[SQL Execution]
    B -->|aws-native| E{EMR or Glue/Athena?}
    B -->|azure-native| F{Spark Pool or SQL Pool?}

    C -->|Yes| G[Spark Execution]
    C -->|No| H{SQL Fallback?}
    H -->|Yes| D
    H -->|No| I[Reject: Incompatible]

    E -->|EMR| G
    E -->|Glue/Athena| D

    F -->|Spark Pool| G
    F -->|SQL Pool| D

Optional Field¶

The platformProfile is optional. When omitted:

Execution strategy must be determinable from the stage configuration
Providers are resolved based on capability alone
Dual-mode transformations require explicit provider binding

Distribution Profiles¶

Platform profiles also drive the Build System for creating environment-specific distributions:

public record DistributionProfile(
    String name,                          // e.g., "databricks-aws"
    ComputePlatformProfile targetPlatform, // e.g., DATABRICKS
    List<String> requiredCapabilities,
    List<String> requiredProviders,
    List<String> excludedModules,
    Map<String, Object> configDefaults
) { }

A distribution for Databricks includes only Spark-capable providers. A distribution for Snowflake includes only SQL-capable providers. This minimises deployment artifact size.

Example: Multi-Platform Pipeline¶

The same pipeline definition works across platforms — only the platformProfile changes:

DatabricksSnowflakeAWS Native

platformProfile: databricks
# → Spark execution for transformations

platformProfile: snowflake
# → SQL execution for transformations

platformProfile: aws-native
# → EMR (Spark) or Athena (SQL) based on stage config

The pipeline definition itself remains unchanged.