Technical Lead-Data Engg

1. Snowflake Architecture & Enterprise Data Platform Design

• Lead architecture and implementation of scalable Snowflake data platforms:
  • Multi-layered architecture (Landing → Raw → Staging → Curated → Data Marts)
• Develop secure cross-account data sharing strategies.
• Implement:
  • Snowpipe for automated ingestion
  • Streams & Tasks for CDC-based incremental processing
  • Time Travel & Zero-copy cloning for environment management
• Implement data masking, row-level security, and RBAC frameworks.
• Optimize storage, partitioning (micro-partition pruning), and query performance.

2. Data Migration & Modernization

• Participate in end-to-end data migration initiatives including:
  • Legacy data warehouse (Teradata, Oracle, SQL Server, Netezza) to Snowflake
  • On-prem to cloud modernization programs
  • Source system analysis and profiling
  • Data quality assessment and remediation planning
  • Schema conversion and transformation mapping
  • Incremental migration strategies
  • Parallel-run validation strategies
• Perform reconciliation and data validation between legacy and target systems.
• Develop automated validation scripts using SQL and DBT tests.
• Support cutover planning and production readiness.

3. Data Ingestion & Multi-Source Integration

Design and implement ingestion frameworks for structured, semi-structured, and unstructured data from multiple enterprise systems:

Structured Sources

• Oracle, SQL Server, SAP, PostgreSQL
• Clinical systems (EDC, CDMS, CTMS)
• Regulatory systems (RIM)
• Commercial systems (CRM, ERP)

Semi-Structured Sources

• JSON, XML, Avro files
• API responses
• External vendor feeds

4. DBT – Enterprise Transformation Framework

• Design/ Develop DBT transformation layers:
  • Staging models
  • Intermediate models
  • Data marts
• Implement:
  • Incremental models
  • Snapshot strategies for historical tracking
  • Surrogate key management
• Develop custom macros and reusable transformation components.
  • Optimize DBT models specifically for Snowflake compute efficiency.