Senior Data Engineer

Job Summary**
We are seeking a highly skilled and experienced Sr Data Engineer to join our team at Becton Dickinson Technology Campus India LLP. You will be responsible for designing, developing, and maintaining robust and scalable data pipelines and solutions using the Databricks platform on Microsoft Azure. This role requires expertise in large-scale data processing and a deep understanding of Databricks Delta Live Tables, Databricks SQL, and Databricks Unity Catalog.

Job Responsibilities**

* Design, develop, and implement efficient and reliable data pipelines using Databricks Delta Live Tables for real-time and batch data processing.* Leverage the full capabilities of the Databricks Platform on Microsoft Azure to build and optimize data solutions.
* Write and optimize complex SQL queries using Databricks SQL for data extraction, transformation, and loading.
* Implement and manage data governance and metadata using Databricks Unity Catalog to ensure data quality and discoverability.
* Develop and maintain large-scale data processing solutions, ensuring performance, scalability, and security.
* Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver appropriate data solutions.
* Monitor, troubleshoot, and optimize existing data pipelines and data infrastructure.
* Stay up-to-date with the latest trends and technologies in data engineering and the Databricks ecosystem.
* Provide technical guidance and mentorship to junior data engineers.

Job Qualifications**

* Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
* 7+ years of experience in data engineering, with a strong focus on large-scale data processing.* Proven expertise with Databricks Delta Live Tables for building resilient and efficient data pipelines.
* Extensive experience with the Databricks Platform, including Databricks notebooks, clusters, and jobs.
* Proficiency in writing and optimizing complex queries using Databricks SQL.
* Hands-on experience with Databricks Unity Catalog for data governance, metadata management, and access control.
* Solid understanding and experience with Microsoft Azure Databricks.
* Strong programming skills in Python or Scala.
* Experience with various data warehousing concepts and ETL/ELT methodologies.
* Excellent problem-solving, analytical, and communication skills.
* Ability to work independently and as part of a collaborative team.