Senior Data Engineer

• Design, build, and maintain scalable, reliable, and cost-efficient data pipeline architectures.
• Develop idempotent, automated ETL/ELT pipelines using Python, PySpark, Delta Lake, and MERGE INTO patterns.
• Drive automation across deployments, monitoring, incident remediation, and operational processes to improve efficiency, quality, and reliability.
• Implement robust DataOps practices, including data-contract enforcement, automated data-quality checks, and unit and integration testing.
• Establish fine-grained data governance, lineage tracking, and security access controls across environments using Unity Catalog.
• Define and manage Infrastructure as Code using Terraform and Azure DevOps for Databricks workspaces, compute clusters, serverless SQL warehouses, and storage integrations.
• Build and maintain automated CI/CD pipelines using Databricks Asset Bundles, Databricks CLI, GitHub Actions, Azure DevOps, or GitLab.
• Monitor compute consumption and costs, optimize cluster configurations, and implement cluster policies to promote efficient resource utilization.
   

WHO YOU ARE:

• At least five years of experience in data engineering, DataOps, DevOps, or a comparable technical role.
• Advanced hands-on experience with Python, PySpark, SQL, Git, and Databricks.
• Strong experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, and Databricks Serverless SQL.
• Proven experience with Infrastructure as Code, preferably Terraform, and CI/CD tooling such as Databricks Asset Bundles, GitHub Actions, Azure DevOps, or GitLab.
• Experience implementing data-quality controls, data contracts, automated testing, monitoring, and incident-management practices.
  • Strong knowledge of Azure cloud services, including security, networking, identity and access management, and storage integration. Experience with AWS or GCP is beneficial.