Staff Data Engineer
tjx
Job Description
- Design, develop, and deploy scalable, cloud-native data engineering solutions on Azure using Databricks, ADLS Gen2, and Snowflake.
- Build and operationalize end-to-end data pipelines for batch and streaming workloads, enabling reliable ingestion, transformation, and consumption of structured and unstructured data.
- Leverage Generative AI and LLM capabilities to enhance data engineering workflows, including data ingestion automation, schema inference, data quality validation, metadata enrichment, and analytics enablement.
- Implement and manage modern data architectures such as Lakehouse patterns using Delta Lake, optimizing data reliability, performance, and governance.
- Develop secure, scalable APIs and data services to expose curated datasets and GenAI-enabled insights to downstream analytics and applications.
- Implement CI/CD pipelines for data engineering workloads using Azure DevOps or GitHub, supporting automated testing, deployment, and environment promotion.
- Ensure data platform security and compliance by applying best practices such as encryption, access controls, role-based permissions, and secure data sharing.
- Monitor and troubleshoot data pipelines and platforms using tools such as Databricks monitoring, Azure Monitor, Log Analytics, and related observability solutions.
- Collaborate in an Agile/Scrum environment with data scientists, analytics engineers, architects, and business stakeholders to deliver end-to-end data solutions.
- Create and maintain comprehensive documentation, including data models, pipeline designs, operational runbooks, and GenAI integration guidelines.
- Apply responsible and ethical AI principles when enabling GenAI use cases on data platforms, including data quality, lineage, transparency, and bias awareness.
- Stay current with advancements in data engineering, Databricks, Snowflake, cloud-native data services, and emerging GenAI patterns relevant to analytics and data platforms.