Senior Data Engineer

abb

Bangalore 8 Years Exp Posted 3d ago

Design, develop, and maintain scalable ETL/ELT solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS Gen2).
Develop PySpark and Scala-based data processing jobs to handle large-scale batch and real-time data workloads.
Build and optimize cloud-based data pipelines for high-volume, complex data transformation and integration requirements.
Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers) ensuring data quality, governance, and scalability.
Orchestrate data workflows using Azure Data Factory and manage CI/CD pipelines through Azure DevOps.
Perform data modeling, partitioning, optimization, and performance tuning across the data platform.
Monitor and optimize data pipelines for reliability, performance, and cost efficiency.
Collaborate with product owners, data scientists, architects, and business stakeholders to translate business requirements into data solutions. Lead data engineering initiatives, establish coding standards, and mentor junior engineers and Promote a culture of continuous improvement, innovation, and technical excellence within the team.

Qualifications for the role :

Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or related field with 8+ years of experience in Data Engineering and Big Data technologies.
Minimum 5 years of hands-on experience with Azure cloud platforms including Azure Databricks, Azure Data Factory, ADLS Gen2, and Azure DevOps.
Strong expertise in PySpark, Scala, Python, and SQL for large-scale data processing and analytics solutions.
Proven experience designing and implementing scalable ETL/ELT pipelines and modern data architectures.
Strong knowledge of Data Warehousing, Lakehouse architecture, Delta Lake, Parquet, and Data Governance practices.
Experience in building and optimizing distributed data processing solutions in production environments. Knowledge of Git, CI/CD practices, and Agile software development methodologies.
Exposure to real-time data processing technologies such as Azure Event Hub, Stream Analytics, and Spark Streaming is an added advantage.
Knowledge of Machine Learning, Deep Learning, and related frameworks is an added advantage. Experience in Industrial IoT, Manufacturing, Automotive, Energy, or Sensor Data Analytics domains is an added advantage.