Senior Data Engineer

• Pipeline development & implementation

  • Design, build, and maintain ETL/ELT pipelines across Bronze, Silver, and Gold layers using Azure Databricks, PySpark, and Delta Lake.

  • Develop data ingestion, transformation, cleansing, and enrichment processes.

  • Write unit tests and deliver robust, production-ready data pipelines.

• Data quality & observability

  • Implement automated data quality checks and validation frameworks.

  • Monitor pipeline health, freshness, and completeness.

  • Investigate and resolve data quality and production issues to maintain trusted data.
    Platform operations & CI/CD

    • Manage Databricks Asset Bundle deployments and environment promotion across development and production.

    • Optimise Spark jobs, cluster performance, and platform costs.

    • Support platform reliability, monitoring, and operational excellence.

  • Collaboration & requirements

    • Work closely with the Data Architect to implement scalable technical solutions.

    • Partner with Systems Analysts to translate business requirements into data solutions.

    • Contribute to the delivery of trusted and certified Gold layer data products.
      Technical documentation

      • Maintain clear documentation for pipelines, data contracts, configurations, and engineering standards.

      • Contribute to technical runbooks and knowledge sharing across the team.

What you bring

• Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related field (Master's preferred).

• 5–8 years of experience in Data Engineering or a related engineering role.

• Experience building and supporting enterprise-scale cloud data platforms and production data pipelines.

• Knowledge of CI/CD practices and operational support for data engineering solutions.

Technical skills

• Strong expertise in Azure Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Lake Storage Gen2.

• Proficiency in Python, SQL, data modelling, and Medallion Architecture.

• Experience with Databricks Workflows, Databricks Asset Bundles, Git, Azure DevOps, and CI/CD pipelines.

• Knowledge of Azure Monitor, Log Analytics, Azure Key Vault, pytest, data quality frameworks, and performance optimisation.

  • Working knowledge of Bicep, Unity Catalog, data contracts, and modern data governance practices.