Databricks Engineer

Summary:
As a Databricks Engineer, you will play a key role in building and operating high-quality data platforms that support analytics, machine learning, and critical business decision-making. This role requires strong end-to-end ownership of data pipelines, deep hands-on expertise with Databricks, and close collaboration with engineering, data science, and product teams to deliver reliable, scalable, and production-ready data solutions.
Responsibilities:
Data Engineering & Pipeline Ownership:
Own the end-to-end lifecycle of data pipelines, including data ingestion, data cleansing, transformation, and ML inference.
Design, build, and operate robust, scalable, and performant data pipelines using Databricks.
Ensure data solutions meet high standards for data quality, reliability, scalability, and operational excellence.
Databricks Platform Ownership:
Take ownership of the team’s Databricks workspace, including workspace configuration and optimization.
Manage clusters and jobs, security, access controls, and governance.
Define and promote best practices and standards.
Drive continuous improvement in Databricks usage, performance, and cost efficiency.
Software Development Lifecycle:
Contribute across the full development lifecycle, including requirements analysis and solution design.
Implement and automate tests, deploy, and ensure production readiness.
Maintain and support ongoing systems, applying engineering best practices such as clean code, code reviews, CI/CD integration, and documentation.
Collaboration & Delivery:
Partner closely with software engineers, data scientists, and product teams to deliver reliable, high-quality data solutions.
Translate analytical and business requirements into well-designed, production-grade data pipelines.
Support troubleshooting, incident resolution, and continuous improvement in production environments.
Requirements:
Bachelor’s degree in Information Technology, Computer Science, or equivalent education.
6 years of experience in the software engineering field.
Hands-on experience with Apache Spark or Databricks, preferably using Python or Java.
Proven experience building large-scale data or ETL pipelines that handle high-volume datasets.
Experience working in cloud-native environments, ideally on AWS.
Strong understanding of software craftsmanship, including writing clean, maintainable, and well-tested code.
Experience working in Agile/Scrum environments.
Must have knowledge of AI Agents and hands-on using AI Agents.
Strong problem-solving skills and ability to work independently on complex problems.
Strong communication skills—both verbal and written—and able to quickly learn and implement new technologies.
Strong relationship, collaborative skills, and organizational skills with a high degree of initiative and self-motivation.
Willingness and ability to learn and take on challenging opportunities.
Knowledge of payments domain and Indian payment ecosystem is desirable.
Required Skills:
Databricks & Apache Spark (Expert level)
Data Engineering & ETL Pipeline Development (Expert level)
Cloud & DevOps (AWS CI/CD) (Advanced level)
Preferred Skills:
Working knowledge of Scala, especially in the context of Spark-based workloads.
Hands-on experience managing pipeline development, deployment stages, and usage reporting within Databricks.
Experience handling regulated or sensitive data, with an understanding of data security, privacy, and compliance requirements.
Familiarity with SQL and NoSQL databases and messaging or streaming systems, such as Redis, ElastiCache, DynamoDB, Amazon S3, Kinesis, or Kafka.
Experience with monitoring, observability, and alerting tools, and supporting high-traffic, customer-facing platforms, using tools such as Grafana or Prometheus.
Experience writing automated acceptance and integration tests that are fully integrated into CI/CD pipelines.
Exposure to cloud-native tooling and infrastructure, including AWS, Docker, Kubernetes, and Infrastructure-as-Code tools such as Terraform.

#AditiConsulting
# 26-02670