Data Engineer
happiestminds
Job Description
- Key tasks & accountabilities??
- Responsible for Creation of ETL Pipelines.?
- Responsible for designing and building new Data Models and?optimizing?existing data models.?
- Exploit elasticity and agility of the cloud.?
- Ownership of physical Data models of serving layer?
- Responsible and accountable for pipelines reliability and availability.?
- Responsible of Data analysis for troubleshooting and exploration?
- Understand the meaning,?domain?and the context of the data.?
- Professional & Technical skills?
- Essential:??
- Azure Technology Stack: SQL DW/Synapse- Azure Data Factory - Azure Databricks - Azure Data Lake Storage.?
- Ability to write complex SQL scripts and stored procedures.?
- Demonstrated experience in design and delivering data platforms for Business Intelligence or Data Warehouse, including data ingestion, ETL and data integration.?
- Strong skills in handling and analysing complex, high volume data with excellent attention in details.?
- Familiar with Agile?methodology?and Agile working environment.?
- Ability to work along with team leads,?BAs, data architects, other.?
- Develop, design, and implement data pipelines using Azure Data Factory for efficient data ingestion, transformation, and processing.?
- Utilize Databricks to build scalable and optimized data processing workflows and perform advanced analytics tasks.?
- Design and manage Delta tables for handling large-scale?structured and semi-structured data efficiently.?
- Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.?
- Optimize?and fine-tune?PySpark?scripts for performance improvement and scalability.?
- Create and?maintain?SQL queries for data extraction, manipulation, and analysis.?
- Leverage Feature Store for efficient management and sharing of features across machine learning pipelines (added advantage).?
- Monitor and troubleshoot data pipeline issues to ensure data integrity and reliability.?
- Stay updated with the latest Azure data engineering technologies and best practices.?
- Leverage Git, manage Release Cycles.?