Sr. Data Engineer
articconsulting
Job Description
- Hands-on experience building and maintaining OLTP databases, including relational data modeling, schema design, normalization, SQL query development, indexing, transaction handling, performance troubleshooting, and data integrity controls
- Design, develop, and maintain Power BI reports, dashboards, DAX expressions, KPIs, and scorecards using both Import and DirectQuery modes.
- Build and orchestrate scalable ETL/ELT workflows using Fabric Data Pipelines, Dataflows Gen2, and Azure Data Factory.
- Write and tune complex T-SQL and KQL queries, stored procedures, and views for performance in Synapse SQL and SQL Server environments.
- Implement data models based on star/snowflake schemas and support modern data warehousing and Lakehouse architectures using Microsoft Fabric.
- Manage and optimize Azure SQL application databases, including schema design, indexing strategies, stored procedures, query optimization, data integrity, security, monitoring, backup/recovery awareness, and operational reliability for client-facing business applications.
- Partner with application, analytics, and client teams to ensure Azure SQL transactional databases are performant, secure, well-structured, and ready for integration with reporting, data warehousing, Microsoft Fabric, and AI/ML workloads.
- integrate structured and unstructured data sources (e.g., SQL, Excel, APIs, Blob Storage), and transform them efficiently using Fabric Notebooks (Spark/PySpark) or Dataflows.
- Diagnose and resolve pipeline failures, logic errors, and performance bottlenecks across the data engineering lifecycle.
- Automate repetitive data processes using Azure Functions, Logic Apps, PowerShell, or Python scripting within the Fabric ecosystem.
- Collaborate with stakeholders to gather business requirements and translate them into scalable data solutions.
- Ensure data governance, privacy, and compliance standards (e.g., GDPR, HIPAA, ISO) are adhered to, including sensitive data handling policies.
- Apply best practices for item-level security, workspace-based access models, and data lineage using Microsoft OneLake and Fabric tools.
- Utilised AI-assisted tools such as GitHub Copilot and Microsoft Copilot to accelerate SQL development, data pipeline implementation, code reviews, troubleshooting, and performance optimization.
- Support AI initiatives by preparing, transforming, and managing high-quality datasets for model training and inference.
- Integrate AI-powered services such as Azure AI Services, Azure OpenAI, and Microsoft Fabric AI capabilities into data solutions.
- Develop and maintain data pipelines that support Generative AI, predictive analytics, and machine learning workloads.