Data Engineering

Data Ingestion & Integration
Design, build, and maintain data pipelines using Azure Data Factory for batch and incremental data ingestion.
Connect to various data sources (SQL Server, REST APIs, CSV, JSON, SAP, etc.) and integrate into the Azure ecosystem.
Develop metadata-driven and parameterized pipelines to improve reusability.
Implement data validation, error handling, and logging frameworks in ADF.
Data Transformation & Processing
Use Azure Databricks (PySpark) for data cleansing, transformation, and enrichment.
Optimize Spark jobs for performance and cost efficiency.
Implement ETL/ELT workflows with Delta Lake and Medallion (Bronze, Silver, Gold) architecture.
Data Storage & Modeling
Work with Azure Data Lake Storage Gen2 for raw and curated data zones.
Develop data models in Azure Synapse Analytics / SQL Server for reporting and analytics.
Implement partitioning, indexing, and performance tuning strategies.
Deployment & DevOps
Implement CI/CD pipelines using Azure DevOps or GitHub Actions for data workflows.
Collaborate with architects to automate deployments and version control using Git.
Security & Governance
Manage data access using Azure RBAC, Managed Identities, and Key Vault.
Ensure data security, compliance, and privacy as per organizational standards.
Collaboration
Work with data analysts, BI developers, and business users to define data requirements.
Participate in code reviews and adhere to best practices in data engineering.