Technical Lead-Data Engg

 Solution Architecture & Technical Leadership
•    Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake.
•    Architect end-to-end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization.
•    Define technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams.
•    Provide technical leadership across the project lifecycle—requirements analysis, solution blueprinting, estimation, development, and deployment.
2. Data Pipeline Engineering
•    Build, optimize, and maintain scalable, high performance ELT/ETL pipelines to process large volumes of structured and unstructured data.
•    Set up complex data ingestion frameworks, enabling seamless integration with on-premise systems, cloud services, APIs, and third-party sources.
•    Ensure high availability, data reliability, and error-resilient orchestration workflows in Azure Data Factory.
3. Azure Databricks & PySpark Expertise
•    Design and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity.
•    Utilize Delta Lake capabilities—ACID transactions, schema evolution, versioning, time travel—to manage enterprise-grade datasets.
•    Perform cluster-level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization.
•    Manage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines
________________________________________
Mandatory Skills & Experience
•    10–13 years of overall experience in data engineering and enterprise data platforms.
•    Minimum 3 years of hands-on project experience in Azure Databricks (beyond POCs).
•    Minimum 5 years of experience building and orchestrating pipelines in Azure Data Factory (ADF).
•    Minimum 2+ years of strong PySpark experience with complex data transformation logic.
•    6+ years of ETL & Data Warehouse experience, including dimensional modelling, data partitioning, and performance optimization.
•    Strong SQL expertise—complex queries, optimization, stored procedures, analytical functions.
•    Proven experience working with Azure Data Lake Storage (ADLS), Delta Lake, and modern data processing patterns.