Data Engineer

Design, develop, and maintain robust data pipelines using Azure Data Factory (ADF).- Build ETL/ELT processes that ingest, transform, and load data from various sources into data lakes, warehouses, or databases. - Implement both batch and near real-time data flows.

- Create, schedule, and monitor ADF pipelines. - Implement complex data workflows, including dependency management, parameterization, and dynamic pipeline design. - Integrate ADF with other Azure services (Blob Storage, Synapse, Databricks, Key Vault, etc.). - Implement logging, alerting, and retry logic for production stability.

- Analyze large datasets to identify trends, issues, and transformation needs. - Write performant SQL queries and transformations. - Collaborate with data analysts to understand data requirements and structure transformations accordingly. - Ensure data is accurate, consistent, and aligned with business logic.

- Quickly diagnose and resolve data quality issues, pipeline failures, or performance bottlenecks. - Perform root cause analysis for pipeline errors and implement long-term fixes. - Debug and optimize Spark or SQL transformations in environments like Databricks.

- Implement data validation and quality checks within pipelines. - Perform reconciliation and sanity checks across data sources. - Maintain data dictionaries and metadata repositories.

- Work closely with data architects, analysts, BI developers, and business users. - Translate business data needs into technical implementations. - Document pipeline logic, data flows, and troubleshooting procedures.

- Work closely with data architects, analysts, BI developers, and business users. - Translate business data needs into technical implementations. - Document pipeline logic, data flows, and troubleshooting procedures.

- Optimize ADF pipeline execution time and cost. - Improve query performance using indexing, partitioning, and efficient transformation logic. - Recommend improvements in data infrastructure and tooling.