Senior Data Engineering Lead

• Participate in scrum process and deliver stories/features according to the schedule
• Collaborate with team, architects and product stakeholders to understand the scope and design of a deliverable
• Participate in product support activities as needed by the team.
• Understand product architecture, features being built and come up with product improvement ideas and POCs
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements.

Required Qualifications:

• Undergraduate degree or equivalent experience
• Deep experience in Data Analysis, including source data analysis, data profiling and mapping
• Good experience in building data pipelines using ADF/Azure Databricks
• Proven hands-on experience with a large-scale data warehouse
• Experience with DevOps, implementation of Bigdata, Apache Spark and Azure Cloud
• Experience with Large scale data processing using PySpark on azure ecosystem
• Experience with implementation of self-service analytics platform ETL framework using PySpark on Azure
• Hands-on data migration experience from legacy systems to new solutions, such as from on-premises clusters to Cloud
• Tools/Technologies:
  • Programming Languages: Python, PySpark
  • Cloud Technologies: Azure (ADF, Databricks, WebApp, Key vault, SQL Server, function app, logic app, Synapse
    Azure Machine Learning, DevOps)
• Solid proficiency in SQL and complex queries
• Proven expert skills in Azure data processing tools (Azure Data Factory, Azure Databricks)
• Proven ability to learn and adapt to new data technologies
• Proven good problem solving skills
  • Proven good communication skills