Databricks Lead

Technical Design Design

· Build design specifications, data models, pipeline patterns, and solution architectures for the workstream.

· Lead the design, development, and maintenance of data pipelines and ETL workflows using Databricks and Airflow.

· Build reusable frameworks, templates, and engineering standards for ETL/ELT, streaming, and batch pipelines.

· Ensure cloud-native designs aligned with AWS platform best practices.

Governance, Security & Optimization

· Ensure data quality, integrity, and governance across data platforms.

· Optimize SQL queries and data models for performance and scalability.

· Build standards for Unity Catalog, lineage, metadata, glossary, and security policies.

· Build data quality frameworks, observability, and operational monitoring.

Cross-Functional Collaboration

· Work closely with stakeholders to understand data requirements and translate them into technical solutions.

· Collaborate with analytics and business teams to deliver actionable insights through BI dashboards and reports.

· Support estimations, and sizing for new programs and migrations migrations.

AI/LLM Enablement (Advantage)

· Design frameworks for agentic workflows, GenAI-based automation, code generation, data quality validation, and metadata enrichment.

· Build Databricks Model Serving, Vector Search, and AI/BI integration.

Required Skills & Experience

· 6-8 8-18 years of experience in data engineering, with at least 3 years in Databricks.

· Deep hands-on expertise in:

o PySpark, Spark SQL, Delta Lake, Unity Catalog

o Databricks Workflows, Delta Live Tables (DLT), SQL Warehouses, Photon

o Streaming, batch, and real-time architectures

· Expertise in Databricks for big data processing and analytics.

· Strong proficiency in SQL for data modeling and query optimization.

· Solid understanding of AWS services (S3, Redshift, Glue, Lambda, etc) for data engineering.

· Strong analytical and problem-solving skills with attention to detail.

· Excellent communication and leadership skills to manage teams and stakeholders.

Preferred Qualifications

· Databricks Certified Data Engineer Professional or Databricks Architect certification.

· Experience with Databricks AI/BI, Lakehouse Federation, and Airflow.

· Familiarity with Airflow DAGs.

· AWS certification in data analytics or related specialization.

· Knowledge of LLMs, agentic frameworks, or GenAI automation (strong advantage).