Ingestion Engineer/Senior Software Engineer

In this role, you will:

• Ensure data quality, data governance, and compliance with regulatory requirements.
• Monitor and optimize data pipeline performance.
• Troubleshoot and resolve data-related issues promptly.
• Implement monitoring and alerting systems for data processes.
• Troubleshoot and resolve technical issues optimizing system performance ensuring reliability.
• Create and maintain technical documentation for new and existing system ensuring that information is accessible to the team.
• Implementing and monitoring solutions that identify both system bottlenecks and production issues.

Requirements

To be successful in this role, you should meet the following requirements:

• Good communication skills as the candidate need to work with globally dispersed and diversified teams.
• Flexible attitude - open to learn new technologies based on project requirements.
• Proficiency in Python/Scala/Bash for data pipeline development and automation
• Familiarity with CI/CD pipelines for deploying and managing data pipeline.
• Proven experience building and maintaining scalable data movement pipelines,
• Good understanding of Hadoop and GCP environments for data storage and data processing
• Familiarity with ETL tools and distributed data processing frameworks such as Spark
• Good understanding of scheduling and orchestration tools such as AirFlow or Control-M
• Good understanding of data principles, data integrity, data best practices etc