Senior Data Engineer
yash
Job Description
- Design and maintain an integrated data architecture that connects Palantir Foundry, AWS services, and Snowflake, ensuring secure, scalable, and performant data access.
- Define and govern enterprise-wide standards for data modeling, metadata, lineage, and security across hybrid environments.
- Architect high-throughput data pipelines that support batch and real-time ingestion from APIs, structured/unstructured sources, and external platforms.
- Collaborate with engineering, analytics, and product teams to implement analytical-ready data layers across Foundry, Snowflake, and AWS-based lake houses.
- Define data optimization strategies including partitioning, clustering, caching, and materialized views to improve query performance and reduce cost.
- Ensure seamless data interoperability across Palantir objects (Quiver, Workshop, Ontology), Snowflake schemas, and AWS S3-based data lakes.
- Lead DataOps adoption including CI/CD for pipelines, automated testing, quality checks, and monitoring.
- Govern identity and access management using platform-specific tools (e.g., Foundry permissioning, Snowflake RBAC, AWS IAM).
- Drive compliance with data governance frameworks, including auditability, PII protection, and regulatory requirements (e.g., GxP, HIPAA).
- Evaluate emerging technologies (e.g., vector databases, LLM integration, Data Mesh) and provide recommendations.
- Act as an architectural SME during Agile Program Increment (PI) and Sprint Planning sessions.