8-hour Databricks job to 47 minutes

The platform depended on a full reload pattern that made runtime and cost unpredictable as data volume climbed. Jobs kept growing, and so did monthly compute.

I replaced full reloads with delta loads and rebuilt the processing flow around incremental behavior and operational guardrails.

Headline outcome: runtime moved from 8 hours to 47 minutes, and monthly compute moved from $14K to $5.8K.