Lead Databricks Data Engineer

• Design, build, and maintain scalable batch and streaming data pipelines
• Develop reusable data engineering frameworks, libraries, and templates for ingestion, transformation, validation, and publishing
• Establish standardized patterns for data modeling, transformations, and pipeline orchestration
• Implement end-to-end data workflows from raw ingestion to curated analytical datasets
• Leverage AI based techniques to automate and optimize data engineering workflows, such as:
o Intelligent schema inference and evolution
o Automated data quality checks and anomaly detection
o Pipeline failure detection and self-healing mechanisms
o Experience building AI assisted or intelligent automation for:
o Data quality monitoring
o Pipeline observability
o Cost or performance optimization
• Ensure data quality, reliability, and performance across pipelines and shared frameworks
• Support downstream consumers such as analytics, reporting, and AI/ML teams
Reliability, Operations & Intelligent Automation
• Define and monitor SLIs/SLOs for data pipelines, frameworks, and platform availability
• Participate in incident response, on-call rotations, and post-incident reviews
• Apply AI assisted monitoring and alerting to proactively detect performance issues, data drift, and operational anomalies
• Implement security, compliance, and data governance controls across shared data assets
• Drive performance tuning and cost optimization, including automated recommendations for resource utilization and workload optimization
Collaboration & Technical Ownership
• Partner with analytics, application, and platform teams to understand common data needs and platform gaps
• Adopt standardized data frameworks, automation patterns, and best practices
• Contribute to data architecture decisions, platform standards, and design guidelines

- 7+ years of IT-related work experience in Data Platform with a Bachelor's degree.
- 7+ years of strong Data engineering experience with mandatory AI assisted development
- Hands-on technical experience with AWS, Databricks data platform ecosystem of Fivetran ingestion and Acceldata observability tools
- Solid hands-on experience in Data Engineering, Data Analysis, Data Modelling and enabling Analytics solutions
- Passion to explore emerging technologies and perform Proof Of Concepts (POC)
- Ability to work independently and collaborate with business and cross functional teams
- Accountability and Ownership of the delivery in driving the projects to completion