Senior Data Engineer

wsa

Hyderabad 5 Years Exp Posted 19h ago

Pipeline development & implementation
- Design, build, and maintain ETL/ELT pipelines across Bronze, Silver, and Gold layers using Azure Databricks, PySpark, and Delta Lake.
- Develop data ingestion, transformation, cleansing, and enrichment processes.
- Write unit tests and deliver robust, production-ready data pipelines.
Data quality & observability
- Implement automated data quality checks and validation frameworks.
- Monitor pipeline health, freshness, and completeness.
- Investigate and resolve data quality and production issues to maintain trusted data.
  Platform operations & CI/CD
  - Manage Databricks Asset Bundle deployments and environment promotion across development and production.
  - Optimise Spark jobs, cluster performance, and platform costs.
  - Support platform reliability, monitoring, and operational excellence.
- Collaboration & requirements
  - Work closely with the Data Architect to implement scalable technical solutions.
  - Partner with Systems Analysts to translate business requirements into data solutions.
  - Contribute to the delivery of trusted and certified Gold layer data products.
    Technical documentation
    - Maintain clear documentation for pipelines, data contracts, configurations, and engineering standards.
    - Contribute to technical runbooks and knowledge sharing across the team.

What you bring

Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related field (Master's preferred).
5–8 years of experience in Data Engineering or a related engineering role.
Experience building and supporting enterprise-scale cloud data platforms and production data pipelines.
Knowledge of CI/CD practices and operational support for data engineering solutions.

Technical skills

Strong expertise in Azure Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Lake Storage Gen2.
Proficiency in Python, SQL, data modelling, and Medallion Architecture.
Experience with Databricks Workflows, Databricks Asset Bundles, Git, Azure DevOps, and CI/CD pipelines.
Knowledge of Azure Monitor, Log Analytics, Azure Key Vault, pytest, data quality frameworks, and performance optimisation.
- Working knowledge of Bicep, Unity Catalog, data contracts, and modern data governance practices.