Data Engineer
synoptek
Job Description
- Design and implement end-to-end data pipelines utilizing Azure services, including Data Factory, Databricks, Synapse Analytics, and Event Hubs, to ingest, process, and transform data from various sources (e.g., Oracle Mercury Gate, D365, NetSuite, telematics systems, sensors).
- Leverage Kafka and StreamSets to ensure real-time data ingestion and low latency processing for critical operational insights.
- Develop and maintain data models that optimize data storage, retrieval, and performance within Azure Data Lake Storage and Snowflake.
- Write, optimize, and troubleshoot SQL queries within the Snowflake environment.
- Collaborate with business stakeholders to understand data requirements and translate them into actionable data solutions.
- Implement data governance best practices to ensure data quality, consistency, and compliance with industry regulations.
- Monitor and optimize data pipelines for performance, scalability, and cost-effectiveness.
- Develop and maintain data quality checks and data cleansing routines to ensure data integrity throughout the data lifecycle.
- Develop reports and visualization using Tableau to communicate data insights effectively to technical and non-technical audiences.
- Stay current with the latest advancements in Azure data services, data management practices, and transportation & logistics data solutions.