Data Engineer

Design, develop, and maintain data ingestion pipelines using Azure Data Factory (ADF).

Develop pipelines to extract data from REST APIs, including handling:

·       Authentication mechanisms (OAuth, API Keys, Bearer Tokens)

·       Pagination

·       Incremental and historical data loads (multiple months of data)

·       Error handling and retries

Load and transform data into Azure target systems such as:

·       Azure Blob Storage

·       Azure SQL Database

Develop and optimize SQL queries, stored procedures, views, and data transformations.

Handle ad hoc data extraction, transformation, and reporting requests from business stakeholders.

Monitor scheduled pipelines, investigate failures, and perform production support activities.

Maintain source code, pipelines, and deployment artifacts using Azure DevOps with proper version control and branching strategies.

Collaborate with business users to understand reporting requirements and build interactive Power BI dashboards and reports.

Ensure data quality, consistency, and completeness across ingestion and reporting processes.

Participate in code reviews and follow development best practices.

Required Skills

Azure

·       3–4 years of hands-on experience with Microsoft Azure.

·       Strong experience with Azure Data Factory (ADF).

·       Experience creating and scheduling pipelines, triggers, parameterized pipelines, and reusable datasets.

·       Good understanding of linked services and integration runtimes.

·       Experience integrating data from REST APIs.

·       Ability to work with JSON payloads.

·       Experience handling:

o   Pagination

o   Rate limiting

o   Authentication

o   Historical backfills

o   Incremental loads

·       Azure DevOps

o   Experience maintaining code repositories in Azure DevOps.

o   Knowledge of Git branching strategies.

o   Experience with pull requests and deployment pipelines.

o   Data Storage

o   Experience working with:

§  Azure Blob Storage

§  Azure SQL Database