Data Engineer (PySpark+SQL)

Must-Have Skills:

• Good experience in Pyspark - Including Dataframe core functions and Spark SQL

• Good experience in SQL DBs - Be able to write queries including fair complexity.

• Should have excellent experience in Big Data programming for data transformation and aggregations

• Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.

• Good customer communication.

• Good Analytical skill

Technology Skills (Good to Have):

• Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
• Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. 
• Designing and implementing data engineering, ingestion, and transformation functions

• Azure Synapse or Azure SQL data warehouse

• Spark on Azure is available in HD insights and data bricks

Good to Have: 

• Experience with Azure Analysis Services
• Experience in Power BI
• Experience with third-party solutions like Attunity/Stream sets, Informatica
• Experience with PreSales activities (Responding to RFPs, Executing Quick POCs)
  • Capacity Planning and Performance Tuning on Azure Stack and Spark.