Senior Data Engineer

firmable

Remote 4 Years Exp Posted 8d ago

Partner with data quality, product, and analytics on extraction schema, coverage, and accuracy standards
Translate sourcing requirements into extractor designs and ETL architecture
Be the technical voice for sourcing in cross-functional discussions on coverage and data quality

What We're Looking For

4+ years building production extraction, collection, or ETL pipelines in business-critical environments
Strong Python expertise — pandas, numpy, production-grade code, performance-aware. You write systems, not notebooks.
Advanced SQL — complex queries, performance optimisation, comfort across large datasets
Extensive Airflow (or equivalent) experience — end-to-end orchestration, dependency management, recovery patterns in production
Shipped real work with agentic IDEs — Claude Code, Cursor, or equivalent. Not "tried it" — built and merged real extraction systems with it.
Deep, demonstrable expertise integrating LLMs into extraction pipelines — explicit prompts with rubrics, structured outputs, eval sets, prompt versioning. You can show us the repos.
You operate LLMs as production systems — you've designed eval harnesses, run labelled eval sets, versioned prompts, logged traces, and debugged LLM extractions on precision/recall
Sharp judgement on rules vs. LLMs — you reach for a parser when the structure allows, and don't default to an LLM because it feels modern
Strong knowledge of web extraction at scale — anti-bot defences, proxy strategy, JS rendering, schema drift handling
A product mindset — you understand that extraction quality directly impacts customer value

Experience with cloud data platforms — Snowflake, Databricks, Redshift, or RDS
Hands-on AWS experience for pipeline deployment (Lambda, S3, EC2, Glue)
Understanding of data warehousing concepts and how extraction feeds downstream systems
Experience building reusable agents, skills, or tool-calling pipelines that other engineers or agents invoke
Exposure to vector databases and embeddings for matching and deduplication
- Knowledge of data privacy and compliance considerations