Data Engineering

We are seeking a highly skilled Azure Databricks Data Engineer with strong experience in designing, developing, and optimizing scalable data integration and analytics solutions on the Microsoft Azure platform. The ideal candidate will have expertise in Databricks, PySpark, Spark SQL, Python, SQL, Azure Data Factory (ADF), and Azure data services, along with a solid understanding of modern data architectures, data warehousing, and cloud-based ETL/ELT frameworks.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, PySpark, Spark SQL, Python, SQL, and Azure Data Factory (ADF).
• Build and optimize data integration solutions for batch and real-time data processing across enterprise data platforms.
• Develop and manage data ingestion, transformation, cleansing, enrichment, and orchestration processes from multiple structured and unstructured data sources.
• Implement and support modern Lakehouse architectures leveraging Delta Lake and Azure data services.
• Design and maintain enterprise-grade data models, data warehouses, and analytical data structures.
• Utilize Azure services including Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure Functions, Event Hub, and related Azure components.
• Collaborate with Data Architects, Business Analysts, QA teams, and business stakeholders to translate business requirements into scalable technical solutions.
• Ensure data quality, governance, security, compliance, and metadata management best practices are followed across data platforms.
• Monitor, troubleshoot, and optimize Spark jobs, SQL queries, data pipelines, and cloud-based workloads for performance and cost efficiency.
• Develop reusable frameworks, automation scripts, and best practices to improve platform reliability and operational efficiency.
• Implement CI/CD pipelines and deployment automation using Git, Azure DevOps, and DevOps best practices.
• Participate in Agile ceremonies including sprint planning, backlog grooming, code reviews, and release activities.
  • Support enterprise reporting, business intelligence, advanced analytics, and data transformation initiatives.