Senior Data Engineer
siemens
Job Description
- BE / B.Tech / MCA / ME / M.Tech
- 7+ years of experience in Data Engineering / Analytics Engineering
Key Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks and Azure data services
- Integrate internal/external data sources and build reusable, modular data components
- Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications
- Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing
- Drive data quality, governance, reliability, root-cause analysis, and exploratory data analysis (EDA)
- Collaborate with stakeholders to define KPIs, business metrics, and analytics-ready datasets
- Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns
- Design semantic and metadata-driven datasets and enable downstream AI and BI consumption
- Support Qlik / BI performance optimization and ensure consistency of business metrics
- Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps
- Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration
Required Skills
Data Engineering & Platform
- Spark 3.x (DataFrames, SQL, Batch & Structured Streaming)
- Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines)
- Azure Data Services and Lakehouse / Medallion Architecture
- Parquet / Delta, partitioning, compaction, and performance optimization