Data Engineer
soothsayeranalytics
Job Description
Data Pipeline Development:
·Build and maintain scalable ETL/ELT pipelines for structured and unstructured data
·Ingest data from diverse sources (APIs, streaming, batch systems).
Data Modeling & Warehousing
·Design efficient data models to support analytics and AI workloads.
·Develop and optimize data warehouses/lakes using Redshift, BigQuery, Snowflake, or Delta Lake.
Big Data & Streaming
·Work with distributed systems like Apache Spark, Kafka, or Flink for real-time/large-scale data processing.
·Manage feature stores for ML pipelines
Collaboration & Best Practices
·Work closely with Data Scientists and ML Engineers to ensure high-quality training data.
·Implement data quality checks, observability, and governance frameworks.