Solution Architect- Databricks

Requirement: Solution Architect - Databricks
We are looking for an experienced Solution Architect - Databricks to join our Data delivery Team for a large-scale data modernization program with a leading telecommunications company. This role involves designing and architecting enterprise-grade data solutions on Azure Databricks, migrating legacy data warehousing systems to a modern lakehouse architecture.
Key Responsibilities:
Architecture & Design
Design and architect end-to-end data pipelines using Azure Databricks with Medallion Architecture (Bronze → Silver → Gold → Semantic layers)
Lead the design of hybrid orchestration frameworks combining DLT-Lakeflow for CDC ingestion and metadata-driven frameworks for transformation layers
Define and implement Unity Catalog governance strategies for data access, security, and lineage
Architect Databricks Workflows for job scheduling, dependency management, and replacing legacy Control-M orchestration
Design SCD Type 2 implementations and complex transformation patterns for EDW migration
Technical Leadership
Provide technical guidance on PySpark and Spark SQL best practices for large-scale data processing
Define coding standards, modular notebook organization, and configuration-driven development approaches
Lead technical decisions on schema evolution, data quality frameworks, and reconciliation strategies
Evaluate and recommend approaches for surrogate key generation, batch control mechanisms, and selective reprocessing
Migration & Modernization
Architect migration strategies from legacy systems (ODS, EDW, Informatica PowerCenter) to Azure Databricks lakehouse
Design patterns for CDC ingestion using Oracle GoldenGate and Informatica IDR integration
Define approaches for historical data migration and parallel run strategies
Ensure minimal refactoring of existing SQL/PySpark code through metadata-driven frameworks
Operational Excellence
Design solutions supporting BAU operations including manual data patching, hold/resume/skip controls, and entity-level reprocessing
Architect Job/Cycle/Batch auditing frameworks aligned with existing operational patterns
Define monitoring, alerting, and logging strategies across all pipeline layers
Ensure solutions support targeted fixes without full refresh requirements
Stakeholder Engagement
Collaborate with Amdocs and customer architecture teams on design approvals and technical alignment
Participate in discovery sessions for EDW entities (DDS, SDS, DNF)
Present technical solutions and trade-off analyses to leadership and steering committees
Work closely with delivery teams to translate designs into implementable solutions
Required Qualifications:
Experience
8+ years of experience in data engineering and data architecture roles
4+ years of hands-on experience with Databricks (Delta Lake, Spark, Unity Catalog)
3+ years of experience with Azure cloud services (ADLS Gen2, Azure DevOps, Azure Data Factory)
Proven experience in large-scale data migration projects (EDW modernization preferred)
Technical Skills
Must Have:
Azure Databricks - Expert level (Delta Lake, DLT/Lakeflow, Databricks Workflows, Unity Catalog)
PySpark & Spark SQL - Strong proficiency in writing optimized transformations
Medallion Architecture - Hands-on experience implementing Bronze/Silver/Gold patterns
Data Modelling - Dimensional modeling, SCD implementations, surrogate key strategies
ETL/ELT Pipelines - Experience with metadata-driven and configuration-driven frameworks
CDC Integration - Experience with Oracle GoldenGate, Kafka, or similar CDC tools
Data Quality - DQ frameworks, reconciliation, exception handling patterns
Version Control - Git, CI/CD pipelines, Databricks Asset Bundles (DAB)
Good to Have:
Power BI - Semantic layer design, report integration patterns
Informatica - PowerCenter, IDR (for migration context)
Control-M - Understanding for migration/replacement scenarios
Terraform/IaC - Infrastructure provisioning for Databricks workspaces
Telecommunications Domain
Certifications (Preferred)
Databricks Certified Data Engineer Professional
Databricks Certified Solution Architect Professional
Microsoft Azure Data Engineer Associate (DP-203)
Microsoft Azure Solutions Architect Expert (AZ-305)
Soft Skills & Competencies
Strong analytical and problem-solving abilities
Excellent communication skills - ability to articulate complex technical concepts to diverse audiences
Experience working in distributed/remote teams across time zones
Ability to mentor and guide junior team members
Strong documentation skills - HLSD, LLD, design decisions
Collaborative mindset - working with different teams, and customer stakeholders