Azure Data Engineer
persistent
Job Description
- Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Snowflake.
- Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
- Investigate and resolve data quality issues, pipeline failures, and production incidents.
- Perform root cause analysis for framework, infrastructure, and data processing failures.
- Monitor, troubleshoot, and optimize Spark workloads for performance, reliability, and scalability.
- Collaborate with business stakeholders, architects, analysts, and development teams to gather requirements and deliver data solutions.
- Support production deployments and provide L2/L3 production support for critical data platforms.
- Implement data validation, monitoring, and governance practices to ensure data quality and integrity.
- Establish and enforce engineering best practices for coding standards, testing, performance tuning, and operational excellence.
- Develop reusable data engineering frameworks and automation capabilities.
- Participate in architecture discussions and contribute to continuous improvement initiatives.
- Work with cloud-native technologies and modern data platform architectures.