Data Engineer

Responsibilities:
? Build data pipelines to integrate diverse data from multiple internal and external sources, including social media and public datasets.
? Collaborate with cross-functional teams to source data and ensure its availability for downstream consumption.
? Design effective solutions in collaboration with the team to meet business needs.
? Maintain regular communication with key stakeholders to address concerns regarding initiative delivery, risks, and issues.
? Escalate and manage dependencies and challenges (risks), ensuring critical issues are communicated to the Sponsor and/or Head of Data Engineering.
? Manage timelines, milestones, and delivery to ensure the initiative's value is achieved without compromising quality and within budget.
? Develop and implement a coordinated communications plan for initiative execution and delivery, both internally and externally.
? Oversee the final handover of initiatives to business-as-usual processes, conduct post-implementation reviews as necessary, and ensure that lessons learned are integrated into future initiatives.

Mandatory Skills:
? Proficiency in Python is essential.
? Experience with at least one Cloud Infrastructure provider (Azure or AWS).
? Strong background in building data pipelines using batch processing with Apache Spark SQL or Hive.
? Experience in building streaming data pipelines using Apache Spark or Kafka Stream.
? Knowledge of NoSQL databases, with experience in Azure Cosmos DB (NoSQL), RESTful services, and GraphQL being advantageous.
? Familiarity with ETL/ELT processing tools, data modeling, and data mapping.
? Experience with Hive and Hadoop file formats (Avro, Parquet, ORC).
? Basic scripting knowledge (Scripting).
? Experience working with multiple data sources, including relational databases (SQL Server, Oracle, DB2, Netezza), NoSQL/document databases, and flat files.
? Basic understanding of Jenkins/Bamboo tools.
? Basic understanding of Devsecops practices using Git version control.
? Ability to debug, fine-tune, and optimize large-scale data processing jobs.

Preferred Skills:
? Certifications related to Data and Analytics are a plus.

Qualifications:
? Master?s or Bachelor?s degree in a STEM field (Science, Technology, Engineering, Mathematics).

Language:
? Fluency in written and spoken English is required.