Data Engineer
zohorecruit
Job Description
- Design, develop, and maintain ETL pipelines using PySpark, Apache Airflow, and Azure Data Factory (ADF).
- Build and optimize distributed data processing jobs using PySpark.
- Orchestrate and schedule workflows using Apache Airflow.
- Develop and manage data ingestion and transformation pipelines in Azure Data Factory.
- Write clean, efficient, and reusable code using Python .
- Develop and optimize complex SQL queries for MySQL and PostgreSQL databases.
- Work with MongoDB for handling semi-structured and unstructured data.
- Perform data analysis using Pandas and NumPy to support business insights.
- Create basic to intermediate data visualizations using Matplotlib, Power BI, and Streamlit.
- Monitor data pipelines, troubleshoot issues, and ensure data quality and performance.
- Collaborate with cross-functional teams including analysts, data scientists, and product teams.