Cloud Data Engineer

Responsibilities

Qualifications

•  Design, implement, and maintain cloud-based data pipelines to enable efficient storage, processing, and analysis of large volumes of data.
•  Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures.
•  Evaluate and select appropriate cloud infrastructure and services to support data storage, processing, and analytics needs, considering factors such as scalability, performance, cost, and security.
•  Implement data ingestion pipelines, integrating various data sources and ensuring data quality, integrity, and timeliness.
•  Design and optimize data storage solutions, including data lakes, data warehouses, and data marts, to support different types of data processing and analysis.
•  Implement data processing workflows using technologies such as Apache Spark, Apache Hadoop, or cloud-native data processing services.
•  Develop and maintain data governance and security policies, ensuring compliance with data protection regulations and industry best practices.
•  Monitor and optimize the performance of data platforms, identifying bottlenecks and implementing optimizations to improve data processing speed and efficiency.
•  Collaborate with DevOps teams to automate deployment, monitoring, and management of data platforms using infrastructure-as-code and CI/CD practices.
•  Stay up-to-date with emerging technologies and industry trends in cloud computing, big data processing, and data engineering.
•  Provide guidance and mentorship to junior team members, fostering a culture of learning and innovation.
  •  Bachelor's or Master's degree in computer science, engineering, or a related field.
  •  Proven experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data platforms.
  •  Strong experience in cloud computing platforms such as Microsoft Azure
  •  Proficiency in the latest Big Data tools/technologies like Hive, Hadoop, Yarn, Kafka, Spark Stream.
  •  Proficiency in data processing frameworks such as Apache Spark, Apache Hadoop, or cloud-native data processing services (Azure Data Lake, Azure Data factory, Azure Databricks, Azure Synapse, Snowflake,CosmosDB)
  •  Experience with data integration and ETL (Extract, Transform, Load) processes, including tools like Apache Airflow or cloud-native orchestration services.
  •  Experience with workload and job optimization for cost reduction
  •  Knowledge of database systems (relational and NoSQL) and data modeling principles.
  •  Familiarity with data governance, data security, and compliance frameworks.
  •  Understanding of distributed computing principles and scalable architectures.
  •  Experience with infrastructure-as-code tools like Terraform
  •  Excellent problem-solving and troubleshooting skills, with the ability to address complex data platform challenges.
    •  Strong communication and collaboration skills to work effectively with cross-functional teams.