Data Engineering

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory.
• Build scalable data processing solutions using Azure Databricks and Apache Spark.
• Ingest data from multiple sources including databases, APIs, flat files, and cloud storage.
• Develop and maintain data transformation workflows using PySpark, Spark SQL, and SQL.
• Implement data quality checks, monitoring, and error-handling mechanisms.
• Integrate data solutions with Azure services such as:
  • Azure Data Lake Storage
  • Azure Synapse Analytics
  • Azure SQL Database
• Optimize Spark jobs for performance and cost efficiency.
• Collaborate with data architects and business teams to understand requirements and deliver solutions.
• Implement CI/CD pipelines and deployment automation.
• Ensure security, governance, and compliance standards are followed.

Required Skills

Technical Skills

• Strong experience with:
  • Azure Databricks
  • Azure Data Factory
  • Apache Spark
  • PySpark
  • SQL
• Experience with Azure cloud services.
• Knowledge of ETL/ELT concepts and data warehousing.
• Experience working with large-scale structured and unstructured datasets.
• Understanding of Delta Lake architecture.
• Experience with Git-based version control.
  • Familiarity with REST APIs and data integration patterns.