Senior Data Engineer

Pipeline Development and Architecture

• Design, build, and maintain high-performance Apache Flink ETL,ELT pipelines for real-time data synchronisation from PostgreSQL to StarRocks.
• Architect robust CDC (Change Data Capture) solutions using Flink CDC connectors and Debezium for PostgreSQL source ingestion.
• Implement Flink SQL and DataStream API pipelines for complex transformation logic, aggregations, and data enrichment.
• Develop and maintain custom Flink connectors and sinks for StarRocks integration using the StarRocks Flink Connector.
• Design fault-tolerant, exactly-once or at-least-once pipelines with appropriate checkpointing and state management strategies.
  • Evaluate and implement schema evolution strategies to handle upstream PostgreSQL schema changes gracefully.