Data Engineer
caterpillar
Job Description
Data Pipelines & Integration
- Design, build, and maintain scalable batch/streaming pipelines (e.g., telematics/IoT, SAP, PLM, MES, service/warranty, dealer systems).
- Implement CDC, schema evolution, and data quality checks (Great Expectations/Deequ/custom).
- Develop feature pipelines for ML and analytics (feature stores, time-series transformations).
Data Modelling & Storage
- Model data for lakes/lakehouses/warehouses (Delta/Parquet/Iceberg) following medallion or similar architectures.
- Optimize partitioning, compaction, and performance for large datasets.
AI/ML Enablement
- Partner with Data Scientists to productionize models (batch/real-time), build inference pipelines/APIs, and manage feature stores.
- Contribute to MLOps (CI/CD for models and data, model versioning, monitoring, drift detection).
- Support GenAI use cases (RAG pipelines, vector indexing, prompt orchestration, governance).
Platform & DevOps
- Implement infrastructure‑as‑code, CI/CD and observability (lineage, cataloging, logging/metrics).
- Enforce security, privacy, and compliance (PII handling, role-based access, data contracts).
What you will have
- Bachelor’s or master’s degree in computer science, Data Engineering, Industrial Engineering, or related field.
- Overall, at least 7+ years of experience in Data engineering
- Strong proficiency in Snowflake, SQL, Python, and ETL tools (e.g. Snaplogic).
- Familiarity with cloud platforms (AWS, Azure, GCP) and data warehousing solutions (e.g., Snowflake, Redshift).
AI/MLOps Exposure:
- Building feature pipelines, packaging and deploying models (batch/real‑time).
- Model serving (Databricks Model Serving, MLflow, Azure ML etc), model registry, experiment tracking.
- Familiarity with vector databases and RAG patterns.
Behavioural Competencies
- Ownership & bias for action; thrives in a product operating model.
- Clear communicator who can bridge business and engineering.
- Pragmatic, security‑first, automation‑oriented mindset.
- Collaborative partner across global, cross‑functional teams.
- This position requires candidate to work a 5-day -a -week schedule in the office
- Shift Timing-01:00PM-10:00PM IST