Build System & Distributions

The Build System compiles environment-specific distributions from independently compilable modules. The DefaultBuildSystem implementation validates profiles, resolves transitive dependencies, detects conflicts, and supports incremental compilation.

Interface

public interface BuildSystem {
    DistributionArtifact compile(DistributionProfile profile);
    ProfileValidationResult validateProfile(DistributionProfile profile);
    DependencyGraph resolveDependencies(DistributionProfile profile);
    DistributionArtifact incrementalCompile(List<String> changedModules, DistributionProfile profile);
}

Distribution Profiles

A distribution profile declares what goes into a deployment artifact:

record DistributionProfile(
    String name,                          // e.g., "databricks-aws"
    ComputePlatformProfile targetPlatform, // e.g., DATABRICKS
    List<String> requiredCapabilities,     // capabilities to include
    List<String> requiredProviders,        // providers to include
    List<String> excludedModules,          // modules to exclude
    Map<String, Object> configDefaults     // default configuration
) { }

Example Profiles

Databricks on AWS

new DistributionProfile(
    "databricks-aws",
    ComputePlatformProfile.DATABRICKS,
    List.of("source-acquisition", "transformation", "storage", "state-management",
            "data-quality", "observability", "lineage"),
    List.of("spark-transform", "s3-storage", "delta-lake", "databricks-execution"),
    List.of("snowflake-provider", "azure-provider"),
    Map.of("defaultStorage", "s3", "defaultFormat", "delta")
)

Snowflake on Azure

new DistributionProfile(
    "snowflake-azure",
    ComputePlatformProfile.SNOWFLAKE,
    List.of("source-acquisition", "transformation", "storage", "state-management",
            "data-quality", "observability"),
    List.of("snowflake-transform", "adls-storage", "snowflake-execution"),
    List.of("spark-provider", "aws-provider", "databricks-provider"),
    Map.of("defaultStorage", "adls", "defaultFormat", "iceberg")
)

AWS Native (Minimal)

new DistributionProfile(
    "aws-native-minimal",
    ComputePlatformProfile.AWS_NATIVE,
    List.of("source-acquisition", "transformation", "storage"),
    List.of("glue-execution", "s3-storage", "athena-transform"),
    List.of("databricks-provider", "snowflake-provider", "azure-provider"),
    Map.of("defaultStorage", "s3", "defaultFormat", "parquet")
)

Compilation

BuildSystem buildSystem = ...;

// Validate profile first
ProfileValidationResult validation = buildSystem.validateProfile(profile);
if (!validation.valid()) {
    validation.violations().forEach(System.err::println);
    return;
}

// Resolve dependencies
DependencyGraph graph = buildSystem.resolveDependencies(profile);
if (!graph.conflicts().isEmpty()) {
    System.err.println("Dependency conflicts: " + graph.conflicts());
    return;
}

// Compile
DistributionArtifact artifact = buildSystem.compile(profile);
System.out.printf("Distribution: %s (%d bytes)%n", profile.name(), artifact.totalSize());
System.out.printf("  Core: %s%n", artifact.core().name());
System.out.printf("  Capabilities: %d modules%n", artifact.capabilities().size());
System.out.printf("  Providers: %d modules%n", artifact.providers().size());

Incremental Compilation

When only specific modules change:

DistributionArtifact artifact = buildSystem.incrementalCompile(
    List.of("spark-transform", "s3-storage"),  // changed modules
    profile
);

Module Independence

Each module compiles independently:

# Core compiles with zero internal dependencies
mvn compile -pl core

# Capabilities depends only on core
mvn compile -pl core,capabilities

# Any provider module depends on core + capabilities
mvn compile -pl core,capabilities,providers

This enables minimal distributions — include only what you need for your target platform.

Platform Compatibility

The PlatformCompatibility class encodes which providers work with which platforms:

Platform	Compatible Providers
DATABRICKS	spark-provider, delta-lake-provider, unity-catalog-provider, azure-storage-provider, aws-s3-provider, databricks-sql-provider
SNOWFLAKE	snowflake-sql-provider, snowpark-provider, aws-s3-provider, azure-storage-provider, gcs-provider, snowflake-stream-provider
AWS_NATIVE	aws-s3-provider, aws-glue-provider, aws-athena-provider, aws-lambda-provider, aws-emr-provider, spark-provider
AZURE_NATIVE	azure-storage-provider, azure-synapse-provider, azure-data-factory-provider, azure-functions-provider, spark-provider, delta-lake-provider

Dependency Resolution

The DependencyResolver knows the dependency graph:

Capability dependencies:

Capability	Depends On
data-ingestion	core
data-transformation	core
data-quality	core, data-transformation
data-governance	core, data-quality
streaming	core, data-ingestion
orchestration	core
observability	core
serverless	core, orchestration

Provider dependencies:

Provider	Requires Capabilities
spark-provider	data-transformation, data-ingestion
delta-lake-provider	data-transformation
unity-catalog-provider	data-governance
aws-glue-provider	data-transformation, orchestration
snowflake-stream-provider	streaming

Transitive dependencies are resolved automatically. If you request data-governance, you get data-quality and data-transformation included.