Senior Data Engineer

Key Result Areas and Activities:

1. Data Pipeline Development & Support: Design, develop and optimize end to end batch and streaming data pipelines using Azure Databricks and Spark.
2. Data Architecture Implementation: Apply Medallion Architecture to structure data layers (raw, enriched, curated).
3. Data Quality & Governance: Ensure data accuracy, consistency, and governance using tools like Azure Purview and Unity Catalog.
4. Performance Optimization: Optimize Spark jobs, Delta Lake tables, and SQL queries for efficiency and cost-effectiveness.
5. Collaboration & Delivery: Work closely with analysts, architects, and business teams to deliver end-to-end data solutions.

Essential Skills:

• Hands-on experience with Databricks, Delta Lake, Data Factory.
• Proficiency in Python, PySpark, and SQL with strong query optimization skills.
• Deep understanding of Lakehouse architecture and Medallion design patterns.
• Experience building scalable ETL/ELT pipelines and data transformations.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Rotational Shift including 2 weeks Night Shift.

Desirable Skills:

• Knowledge of data quality frameworks and monitoring practices.
• Experience with data visualization tools.
• Experience of working in managed services

Qualifications:

• Education: Likely a degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Experience:
  • Proven hands-on experience with Azure data stack (Databricks, Data Factory, Delta Lake).
  • Experience in building scalable ETL/ELT pipelines.
  • Familiarity with data governance and DevOps practices.