Data Engineer
jacobs
Job Description
- Design, develop and maintain scalable data pipelines and data integration solutions using cloud-based technologies.
- Build batch and, where required, near-real-time ingestion and transformation workflows across structured and unstructured data sources.
- Use Python and SQL to extract, transform, validate and manage data.
- Work with cloud data platforms and services, with Microsoft Azure experience particularly relevant. Technologies may include Azure Data Lake, Azure Data Factory, Azure Functions, Microsoft Fabric, Databricks and equivalent services.
- Apply data processing technologies such as Apache Spark, PySpark and Delta Lake when appropriate.
- Develop and maintain data models, data structures and curated datasets for analytics, enterprise applications and AI/ML use cases.
- Profile, cleanse, standardise and validate data, working with subject matter experts where domain interpretation is required.
- Implement data quality checks, testing, monitoring, logging, security and performance controls.
- Support processing of documents and other unstructured information, including metadata extraction and transformation where appropriate.
- Contribute to data architecture, integration patterns and technical design decisions.
- Work with AI engineers, software developers, solution architects, analysts and business stakeholders to translate requirements into deliverable solutions.
- Support deployment, troubleshooting, maintenance and continuous improvement of data workflows.
- Produce clear technical documentation and contribute to knowledge transfer across distributed teams.