Lead Data Engineer

bnpparibas

Mumbai, India 8 Years Exp Posted 3h ago

Job Description

Direct Responsibilities

  • Design and develop real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow.
  • Implement and maintain ETL/ELT processes, ensuring data quality and integrity across various data sources.
  • Integrate and manage data flows across diverse systems using NiFi and Kafka topics.
  • Monitor and optimize the performance of data pipelines and troubleshoot issues proactively.
  • Work with the ELK stack (Elasticsearch, Logstash, Kibana) for logging, monitoring, and real-time analytics.
  • Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
  • Collaborate with data analysts and business stakeholders to develop meaningful Power BI dashboards and reports.
  • Maintain documentation for data workflows, processes, and architecture.

Required Skills

  • Strong experience with Apache NiFi for data flow management.
  • Expertise in Kafka (streaming data, Kafka Connect, schema management).
  • Proficiency in Apache Airflow for orchestrating workflows.
  • Solid understanding of SQL (writing complex queries, tuning, indexing).
  • Experience working with the ELK stack – Elasticsearch for search and analytics, Logstash for 
    ingestion, Kibana for visualization.
  • Experience with Power BI – data modeling, DAX, dashboard development.
  • Knowledge of data warehousing concepts, data modeling, and data lake architecture. 

Contributing Responsibilities

The Digital Platform domain within the CIB ITO PRODUCTION entity is in charge of the following scopes:

  • IAAS (Infrastructure As A Service): Server / OS / Storage / Backup / Virtualization
  • CAAS (Container As A Service): Containers / Middleware / Sharepoint
  • PAAS (Platform As A Service): Databases / Data Analtics
  • Infrastructure Services: Micro segmentation / Load Balancing / API gateway / Automation
  • Risk & Improvement: Obsolescence / Security / Continuous improvement / ITIL Process
  • Work as a member of a team, adhere to corporate standards and change management processes, and work to improve IT standards and policies. 
  • Proficient in English.
  • Good interpersonal and communication skills.