Data Engineering Consultant

Primary Responsibilities:

• Design, develop, and maintain Python-based APIs for high-performance, secure, and reliable data and application services
• Build, optimize, and maintain ETL/ELT pipelines using Databricks, leveraging PySpark and Spark SQL for large-scale data processing
• Develop and optimize PostgreSQL schemas, queries, and stored procedures to support analytical and transactional workloads
• Implement and maintain CI/CD pipelines using GitHub Actions, ensuring high code quality, automated testing, and reliable deployments
• Build and deploy Azure Functions to support event-driven and serverless workloads
• Monitor, troubleshoot, and improve platform reliability and performance using Azure Log Analytics and observability tools
• Collaborate closely with architects, platform teams, and business stakeholders to deliver robust, production-ready data solutions
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Solid hands-on experience with Python for data engineering and API development
• Solid experience with Databricks, including PySpark and Spark SQL
• Experience building CI/CD pipelines, preferably using GitHub Actions or similar tools
• Working knowledge of Azure services, especially Azure Functions and monitoring/logging capabilities
• Proficiency in SQL, with practical experience in PostgreSQL or similar relational databases

Preferred Qualifications:

• Experience working in enterprise-scale data platforms
• Experience with Infrastructure-as-Code (IaC) using Terraform to automate cloud infrastructure provisioning and configuration
• Exposure to cloud security, access controls, and compliance-driven data platforms
• Solid debugging, performance tuning, and problem-solving skills
  • Proven ability to work effectively in a distributed, cross-functional team environment