Senior Data Engineer

• Design end-to-end data pipelines processing large volumes of unstructured enterprise data (documents, PDFs, transaction records, email)
• Build data ingestion frameworks supporting multiple sources and formats with automated validation and quality checks
• Implement large-scale processing using distributed computing frameworks (Spark, Flink, AWS Glue) handling terabytes efficiently
• Develop advanced feature engineering pipelines — document classification, entity extraction, semantic tagging from unstructured data
• Design data warehousing architecture supporting both near-real-time operational and analytical queries for agentic AI reasoning
• Build data quality frameworks ensuring accuracy critical for agent decision-making and regulatory compliance
• Implement data governance — lineage tracking, metadata management, audit trails for regulated environments
• Lead data security for sensitive information (PII, financial data, healthcare records) with encryption and access controls

Must-Have Technical Skills

• Unstructured data expertise — production ingestion and processing of documents, PDFs, images, text, logs at scale
• Distributed data processing — Apache Spark, Flink, or AWS Glue at production scale
• Feature engineering — advanced techniques for ML systems, automated feature extraction and transformation
• Expert Python — data processing, ETL pipeline development, data science workflows; not notebook-level
• NLP/text processing — document understanding, entity extraction, semantic processing (spaCy, transformers)
• Data architecture — data warehouses, data lakes, or lakehouse architectures supporting batch and real-time processing
• ETL/ELT pipeline design — production-grade with error handling, retry logic, and monitoring
• AWS data services — S3, Athena, Glue, RDS, DynamoDB, MSK
• Data quality & governance — metadata management, lineage tracking, compliance frameworks (GDPR, HIPAA, SOC2)
•