Lead Data Engineer

Direct Responsibilities

• Design and develop real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow.
• Implement and maintain ETL/ELT processes, ensuring data quality and integrity across various data sources.
• Integrate and manage data flows across diverse systems using NiFi and Kafka topics.
• Monitor and optimize the performance of data pipelines and troubleshoot issues proactively.
• Work with the ELK stack (Elasticsearch, Logstash, Kibana) for logging, monitoring, and real-time analytics.
• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
• Collaborate with data analysts and business stakeholders to develop meaningful Power BI dashboards and reports.
• Maintain documentation for data workflows, processes, and architecture.

Required Skills

• Strong experience with Apache NiFi for data flow management.
• Expertise in Kafka (streaming data, Kafka Connect, schema management).
• Proficiency in Apache Airflow for orchestrating workflows.
• Solid understanding of SQL (writing complex queries, tuning, indexing).
• Experience working with the ELK stack – Elasticsearch for search and analytics, Logstash for 
  ingestion, Kibana for visualization.
• Experience with Power BI – data modeling, DAX, dashboard development.
• Knowledge of data warehousing concepts, data modeling, and data lake architecture. 

Contributing Responsibilities

The Digital Platform domain within the CIB ITO PRODUCTION entity is in charge of the following scopes:

• IAAS (Infrastructure As A Service): Server / OS / Storage / Backup / Virtualization
• CAAS (Container As A Service): Containers / Middleware / Sharepoint
• PAAS (Platform As A Service): Databases / Data Analtics
• Infrastructure Services: Micro segmentation / Load Balancing / API gateway / Automation
• Risk & Improvement: Obsolescence / Security / Continuous improvement / ITIL Process
• Work as a member of a team, adhere to corporate standards and change management processes, and work to improve IT standards and policies. 
• Proficient in English.
• Good interpersonal and communication skills.