Data Engineer

Design, develop, and maintain robust ETL/ELT pipelines using Databricks, PySpark, and Azure Data Factory (ADF). • Build scalable and efficient data ingestion frameworks for structured and unstructured data. • Optimize pipeline performance through performance tuning and orchestration best practices. Data Modeling & Management • Develop and maintain data models using modern tools (DBT preferred). • Implement Master Data Management (MDM) solutions to ensure data consistency and integrity. • Design scalable and efficient Snowflake schemas (star/snowflake schema, dimensional modeling). Database & Query Optimization • Write and optimize advanced SQL queries across Snowflake, Azure SQL, and Synapse. • Develop and manage stored procedures and database objects. • Ensure efficient data retrieval through indexing, partitioning, and query optimization. Cloud & Platform Integration • Work with Azure data services including: o Azure Data Factory (ADF) o Azure Data Lake Storage (ADLS) o Azure Synapse Analytics o Azure SQL Database • Integrate and maintain Snowflake with Azure ecosystem. Python Development • Develop data transformation and automation scripts using Python libraries: o pandas o pyodbc o SQLAlchemy • Build reusable components for data processing and validation. Data Quality, Validation & Monitoring • Implement data validation rules, quality checks, and anomaly detection frameworks. • Perform root cause analysis for data inconsistencies. • Develop dashboards or tools for data quality monitoring. Collaboration & DevOps • Use GitHub for version control, branching strategies, and code reviews. • Manage workload scheduling and dependency management for pipelines. • Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders. Required Skills & Qualifications • Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field. • Strong experience in data engineering and data platform development.