Data Engineering Consultant

• Developing Data Pipelines: Creating and maintaining optimal data pipeline architectures to ensure efficient data flow and processing
• Data Integration: Assembling large, complex data sets from various sources to meet business requirements
• Process Improvement: Identifying and implementing internal process improvements, such as automating manual processes and optimizing data delivery
• Infrastructure Building: Designing and building the infrastructure required for data extraction, transformation, and loading (ETL) using technologies like
• SQL and AWS Collaboration: Working with stakeholders, including data scientists, analysts, and executives, to support their data infrastructure needs and resolve data-related technical issues

• Data Quality: Running tests to isolate errors and updating systems to accommodate changes in company needs.

• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications:

• Experience with DBT Core
• AI experience to generate and prepare test cases using GitHub Copilot
• Data Security: Knowledge of data security best practices, encryption, and access controls
• Medallion Architecture: Knowledge of the medallion architecture, which organizes data layers (Bronze, Silver, Gold) in a lakehouse
• Health care domain knowledge
• Programming Languages: Familiarity with languages like Python or Java for data analysis and application development
• Delta Lake: Familiarity with Delta Lake, an open-table format that combines data lake flexibility with ACID transactions
• Data Warehousing: Understanding of data warehousing concepts, including data storage, retrieval, and optimization
• Cloud Computing: Understanding of cloud platforms (e.g., Azure, AWS, GCP) and their data services
• Data Vault: Understanding of the Data Vault methodology for scalable and flexible data warehousing
• Data Modeling: Proficiency in creating logical and physical data models to represent business requirements and system architecture
• Database Management Systems (DBMS): Proficiency in relational databases (e.g., Microsoft SQL Server) and NoSQL databases
• Azure Data Factory (ADF): Proficiency in using ADF for orchestrating data workflows, data movement, and transformation
  • ETL (Extract-Transform-Load): Proven expertise in ETL processes and tools like DBT (Data Build Tool) and DBT framework/packages