Azure Databricks

• Design robust data models and queries using Databricks SQL that enable reliable analytics and reporting for critical business functions in research and development and regulatory compliance domains.
• Develop configure and optimize Databricks Workflows that orchestrate end to end data pipelines supporting ingestion transformation quality checks and scheduled delivery of curated datasets.
• Implement efficient data transformation logic that improves performance reduces compute cost and supports scalable processing of large structured and semi structured datasets.
• Collaborate closely with product owners data analysts and subject matter experts to translate complex analytical and compliance requirements into well engineered Databricks based solutions.
• Create reusable SQL patterns views and parameterized queries that standardize data access and support consistent metrics across regulatory and research initiatives.
• Monitor production workloads running on Databricks Workflows analyze failures or performance bottlenecks and implement systematic fixes that enhance reliability and stability.
• Apply data quality checks validation rules and reconciliation procedures that improve trust in analytical outputs and support audit ready regulatory reporting.
• Document technical designs data flow diagrams and operational procedures in clear language that helps future maintenance and onboarding of new team members.
• Partner with cloud platform and security teams to align Databricks implementations with enterprise standards for access control encryption logging and governance.
• Contribute to continuous improvement by evaluating new Databricks features proposing enhancements and sharing best practices that improve team productivity and solution quality.
• Support hybrid collaboration by participating in both onsite and virtual ceremonies providing timely status updates and coordinating effectively across time zones when needed.
• Assist business stakeholders in understanding delivered datasets and SQL logic by creating reference queries usage guidelines and example dashboards that showcase value.
• Align daily development activities with the company purpose by delivering data solutions that strengthen scientific research improve regulatory transparency and indirectly benefit society.

Qualifications

• Possess eight to nine years of hands on experience in data engineering or analytics development with strong exposure to cloud based data platforms in large scale environments.
• Demonstrate advanced proficiency in Databricks SQL including writing complex joins window functions incremental logic and performance tuned queries for large datasets.
• Show substantial experience in designing and maintaining Databricks Workflows or equivalent orchestration solutions to manage production grade data pipelines and schedules.
• Exhibit strong understanding of relational data modeling query optimization techniques and best practices for handling large volume data in distributed compute environments.
• Display practical knowledge of research and development or regulatory compliance processes where data accuracy traceability and audit readiness are critical to business outcomes.
• Bring solid experience in working within hybrid work models using modern collaboration tools and structured delivery practices such as agile methods tailored for data projects.
• Communicate clearly with technical and non technical partners explaining complex SQL logic and workflow behavior in accessible language that supports informed decision making.
  • Preferably hold prior experience with other cloud data services scripting languages and visualization tools that complement Databricks solutions and broaden contribution to the team.