Senior Databricks Engineer

bridgenext

pune 3 Years Exp Posted 46d ago

Develop, optimize, and maintain ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
Design and deploy distributed data processing workflows for batch and streaming use cases.
Implement best practices for performance tuning, cost optimization, and cluster configuration.
Work with Delta Lake for data versioning, incremental pipelines, and reliability.

Cloud Data Platform Integration

Build solutions on Azure, AWS, or GCP using cloud-native services integrated with Databricks.
Ingest, transform, and process large datasets using cloud storage and compute services.
Work with APIs, connectors, and cloud-native data orchestration tools.

Azure Data Services (Good to Have)

Exposure to Azure Data Lake (ADLS), Azure Data Factory, Azure Synapse Analytics, Azure SQL, Event Hub, Azure Functions, etc.
Support end-to-end pipelines covering ingestion, transformation, storage, governance, and monitoring.

Data Engineering & Development

Collaboration & Delivery

Work closely with Data Architects, Analysts, and Scientists to implement scalable data solutions.
Contribute to solution design documents, data models, and architecture diagrams.
Ensure solutions adhere to security, governance, and compliance standards (e.g., Unity Catalog).

Required Skills & Qualifications

3–6 years of experience in Data Engineering or Big Data platforms.
Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
Experience with at least one cloud provider (Azure, AWS, or GCP).
Strong SQL programming and data modeling concepts.
Understanding of distributed computing, performance tuning, and cost-efficient design.
Experience with Git, CI/CD, and basic DevOps practices.
Familiarity with workflow orchestration (Databricks Workflows, Airflow, ADF, etc.).