Solution Architect - Databricks

As a Solution Architect – Databricks, you will design and deliver
enterprise‑grade data solutions on Azure Databricks, leading a large‑scale data
modernisation programme for a telecommunications client. You will architect
migration strategies, define governance frameworks, and provide technical
leadership to ensure seamless transformation from legacy data warehousing
systems to a modern lakehouse architecture.

What You'll Do and How You'll Succeed

Architecture & Design

* Design and architect end‑to‑end data pipelines using Azure Databricks with
Medallion Architecture (Bronze Silver Gold Semantic layers).
* Lead hybrid orchestration frameworks combining DLT‑Lakeflow for CDC ingestion
and metadata‑driven frameworks for transformation layers.
* Define and implement Unity Catalog governance strategies for data access,
security, and lineage.
* Architect Databricks Workflows for job scheduling, dependency management, and
replacing legacy Control‑M orchestration.
* Design SCD Type 2 implementations and complex transformation patterns for EDW
migration.

Technical Leadership

* Provide guidance on PySpark and Spark SQL best practices for large‑scale data
processing.
* Define coding standards, modular notebook organisation, and
configuration‑driven development approaches.
* Lead technical decisions on schema evolution, data quality frameworks, and
reconciliation strategies.
* Recommend approaches for surrogate key generation, batch control mechanisms,
and selective reprocessing.

Migration & Modernisation

* Architect migration strategies from legacy systems (ODS, EDW, Informatica
PowerCenter) to Azure Databricks lakehouse.
* Design CDC ingestion patterns using Oracle GoldenGate and Informatica IDR
integration.
* Define approaches for historical data migration and parallel run strategies.
* Ensure minimal refactoring of existing SQL/PySpark code through
metadata‑driven frameworks.

Operational Excellence

* Design solutions supporting BAU operations including manual data patching,
hold/resume/skip controls, and entity‑level reprocessing.
* Architect job/cycle/batch auditing frameworks aligned with existing
operational patterns.
* Define monitoring, alerting, and logging strategies across all pipeline
layers.
* Ensure solutions support targeted fixes without requiring full refreshes.

Stakeholder Engagement

* Collaborate with Amdocs and customer architecture teams on design approvals
and technical alignment.
* Participate in discovery sessions for EDW entities (DDS, SDS, DNF).
* Present technical solutions and trade‑off analyses to leadership and steering
committees.
* Work closely with delivery teams to translate designs into implementable
solutions.

We'd Love to Hear From You If...

Experience

* You have 8+ years in data engineering and architecture roles.
* You have 4+ years of hands‑on experience with Databricks (Delta Lake, Spark,
Unity Catalog).
* You have 3+ years of experience with Azure cloud services (ADLS Gen2, Azure
DevOps, Azure Data Factory).
* You have proven experience in large‑scale data migration projects, ideally
EDW modernisation.

Technical Expertise

* You are expert in Azure Databricks (Delta Lake, DLT/Lakeflow, Databricks
Workflows, Unity Catalog).
* You are proficient in PySpark and Spark SQL for optimised transformations.
* You have hands‑on experience with Medallion Architecture
(Bronze/Silver/Gold).
* You are skilled in dimensional modelling, SCD implementations, and surrogate
key strategies.
* You are experienced in metadata‑driven ETL/ELT frameworks.
* You have worked with CDC tools such as Oracle GoldenGate or Kafka.
* You are familiar with data quality frameworks, reconciliation, and exception
handling.
* You are experienced with Git, CI/CD pipelines, and Databricks Asset Bundles
(DAB).
* Preferred: Power BI, Informatica PowerCenter/IDR, Control‑M, Terraform/IaC,
telecommunications domain.

Ways of Working

* You have strong analytical and problem‑solving abilities.
* You communicate complex technical concepts clearly to diverse audiences.
* You collaborate effectively across distributed teams and time zones.
* You mentor and guide junior team members.
* You document design decisions thoroughly (HLSD, LLD).
* You bring a collaborative mindset when working with customer stakeholders.

Certifications (Preferred)

* Databricks Certified Data Engineer Professional
* Databricks Certified Solution Architect Professional
* Microsoft Azure Data Engineer Associate (DP‑203)
* Microsoft Azure Solutions Architect Expert (AZ‑305)