Senior Data Engineer
avathon
Job Description
- Building and optimizing data architectures to handle large-scale data from various sources IoT, PLC, SCADA including cloud and on-premise systems.
- Architecting scalable ETL and data workflows to support advanced analytics and machine learning models in Renewable domains
- Collaborating with cross-functional teams to deliver end-to-end data solutions that drive business insights.
- Owning and leading projects that focus on data platform modernization, ensuring high performance, security, and scalability.
- Working on complex data engineering challenges that push the boundaries of cloud data infrastructure.
You'll have:
- A Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field.
- 5+ years of experience designing and implementing large-scale data platforms, with at least 2 years of experience leading or mentoring data engineers.
- Hands-on experience with Python (Pandas, Polars) and modern big data tools such as Databricks (Spark), Flink, or Kafka.
- Strong understanding of data pipeline development, ETL frameworks, and data lakes/warehouses.
- Extensive experience with cloud data platforms such as AWS (S3, Redshift, Lambda, Glue), GCP (BigQuery, Dataflow, Dataproc), or Snowflake.
- Familiarity with workflow orchestration tools like Airflow or Prefect.
- Proven ability to lead data engineering teams and mentor junior engineers.
It would be great if you have:
- Experience working with IoT or real-time streaming data.
- Knowledge of data governance frameworks and best practices.
- Exposure to MLOps workflows and integrating data with machine learning models.
- Experience in the energy or manufacturing sector or familiarity with time-series data.
- A strong understanding of data warehousing concepts, including Kimball model design.
- A deep understanding of data security and compliance requirements, particularly in cloud environments.