Senior Data Engineer – AI & Analytics

siemens-health

Bangalore 7 Years Exp Posted 6d ago

Key Responsibilities

Design, develop, and optimize scalable data pipelines using Databricks and Azure data services
Integrate internal/external data sources and build reusable, modular data components
Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications
Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing
Drive data quality, governance, reliability, root-cause analysis, and exploratory data analysis (EDA)
Collaborate with stakeholders to define KPIs, business metrics, and analytics-ready datasets
Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns
Design semantic and metadata-driven datasets and enable downstream AI and BI consumption
Support Qlik / BI performance optimization and ensure consistency of business metrics
Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps
Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration

Required Skills

Data Engineering & Platform

Spark 3.x (DataFrames, SQL, Batch & Structured Streaming)
Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines)
Azure Data Services and Lakehouse / Medallion Architecture
Parquet / Delta, partitioning, compaction, and performance optimization

Programming & Analytics

Strong Python and SQL (Spark SQL, TSQL, HiveQL)
Data quality, EDA, KPI-driven analytical modeling
Understanding of statistical concepts and data readiness for analytics/recommendation use cases
Experience building reusable, analytics-ready, and AI-ready datasets

AI, BI & Delivery

Good to Know

ADLS, Managed Identity, Azure AI Foundry
Feature engineering concepts
Airflow / ADF / Synapse Pipelines
Scala or Java
Data Catalogs (Purview, Unity Catalog, Apache Atlas)
- Healthcare domain experience (preferred)