Data/ETL Engineer

The Lead Data Engineer will be responsible for designing, building, and optimizing data pipelines and platforms to enable advanced analytics and business intelligence across the organization. This role requires strong expertise in Databricks, SQL, AWS/Azure/GCP, and Airflow, combined with a solid foundation in data analysis and engineering best practices. The ideal candidate will collaborate with cross-functional stakeholders and ensure scalable, secure, and high-performing data solutions.

Technical Design

• Develop and maintain data pipelines and ETL workflows using Databricks and Airflow.
• Good understanding of Databricks Delta Live tables, and Unity Catalog
• Build frameworks and templates for ETL/ELT, for telemetry pipelines.
• Ensure cloud-native designs aligned with AWS platform best practices.

Governance, Security & Optimization

• Ensure data quality, integrity, and governance across data platforms.
• Optimize SQL queries and data models for performance and scalability.
• Implement monitoring and observability for data workflows.

Cross-Functional Collaboration

• Work closely with architects and leads to understand data requirements and build technical solutions.
• Collaborate with analytics and business teams to deliver actionable insights through BI dashboards and reports

AI/LLM Enablement (Advantage)

• Familiarity with Agentic AI concepts for automation, data quality validation, and metadata enrichment will be an added advantage.

Required Skills & Experience

• 8+years of experience in data engineering, with at least 3 years in Databricks.
• Hands-on expertise in:
• Databricks (PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live tables)
• SQL for data modeling and query optimization
• AWS services (S3, Glue, Kinesis, Redshift, API Gateway, SNS, Lambda)
• Airflow for workflow orchestration
• Added advantage – DBT, Splunk, Atlan, Java, API, Power BI
  • Strong analytical and problem-solving skills with attention to detail.