Lead I - Data Engineering

The Opportunity:

·       As a Senior Data Engineer, you will Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).

·       Build and optimize scalable data architectures, including data lakes, data warehouses (BigQuery), and streaming platforms.

·       Performance Tuning: Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness

·       Data Quality: Implement data quality checks, monitoring, and ing systems to ensure data accuracy and consistency.

What you need:

• Programming: Strong proficiency in Python, SQL, and potentially Scala/Java.
• Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
• Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
• Cloud: Familiarity with GCP, Azure data services.
• Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
  • Tools: Experience with Airflow, Databricks, Docker, Kubernetes is a plus.