Data Engineer

caterpillar

Bengalor 7 Years Exp Posted 1h ago

Job Description

Data Pipelines & Integration

  • Design, build, and maintain scalable batch/streaming pipelines (e.g., telematics/IoT, SAP, PLM, MES, service/warranty, dealer systems).
  • Implement CDC, schema evolution, and data quality checks (Great Expectations/Deequ/custom).
  • Develop feature pipelines for ML and analytics (feature stores, time-series transformations).

Data Modelling & Storage

  • Model data for lakes/lakehouses/warehouses (Delta/Parquet/Iceberg) following medallion or similar architectures.
  • Optimize partitioning, compaction, and performance for large datasets.

AI/ML Enablement

  • Partner with Data Scientists to productionize models (batch/real-time), build inference pipelines/APIs, and manage feature stores.
  • Contribute to MLOps (CI/CD for models and data, model versioning, monitoring, drift detection).
  • Support GenAI use cases (RAG pipelines, vector indexing, prompt orchestration, governance).

Platform & DevOps

  • Implement infrastructure‑as‑code, CI/CD and observability (lineage, cataloging, logging/metrics).
  • Enforce security, privacy, and compliance (PII handling, role-based access, data contracts).

What you will have

  • Bachelor’s or master’s degree in computer science, Data Engineering, Industrial Engineering, or related field.
  • Overall, at least 7+ years of experience in Data engineering
  • Strong proficiency in Snowflake, SQL, Python, and ETL tools (e.g. Snaplogic).
  • Familiarity with cloud platforms (AWS, Azure, GCP) and data warehousing solutions (e.g., Snowflake, Redshift).

AI/MLOps Exposure:

  • Building feature pipelines, packaging and deploying models (batch/real‑time).
  • Model serving (Databricks Model Serving, MLflow, Azure ML etc), model registry, experiment tracking.
  • Familiarity with vector databases and RAG patterns.

Behavioural Competencies

  • Ownership & bias for action; thrives in a product operating model.
  • Clear communicator who can bridge business and engineering.
  • Pragmatic, security‑first, automation‑oriented mindset.
  • Collaborative partner across global, cross‑functional teams.
  • This position requires candidate to work a 5-day -a -week schedule in the office
    • Shift Timing-01:00PM-10:00PM IST

Similar Openings for You