Data Engineer

Direct Responsibilities:

• Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service.
• Implement data transformation and quality to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations.
• Set up CI/CD pipelines to automate deployments, unittestingand development management.
• Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.
• Automate data pipelines and streamline data ingestion through the implementation of different orchestrators and scheduling processes (Airflow as aService mainly).
• Writing technical documentation (specifications, operational documents) to ensureknowledgecapitalization.