Data Engineer
vanguardjobs
Job Description
- Develop and maintain efficient ETL (Extract, Transform, Load) pipelines that ingest, process and transform large datasets from diverse sources. Build and manage data storage solutions using relational and non-relational databases, ensuring high availability, performance and scalability.
- Work with APIs, third-party data sources and internal databases to create unified datasets that provide actionable insights for business stakeholders.
- Partner with business intelligence analysts, data scientists and software engineers to design and implement data-driven solutions that meet business requirements.
- Design optimized data schemas and structures that support analytical workloads and improve query performance.
- Leverage orchestration tools like Apache Airflow or Prefect to automate ETL workflows and improve operational efficiency.
- Provide technical guidance, review code and promote best practices within the data engineering team.
- Lead and mentor Data and AI Engineers, providing technical direction, coaching, and engineering guidance.
- Lead solution architecture and technical design, ensuring alignment with business objectives, product priorities, and engineering standards.
- Design and optimize scalable data pipelines, cloud data platforms, data models, and storage solutions supporting analytical and operational workloads.
- Architect reliable batch and real-time data processing solutions, incorporating data quality, governance, security, performance, and scalability.
- Partner with Product Owners, business stakeholders, Architects, Analysts, and Engineering teams to translate requirements into sustainable data solutions.
- Take end-to-end technical ownership across design, development, deployment, production reliability, and continuous improvement.
- Evaluate technical options and make informed architecture decisions considering performance, scalability, security, cost, reliability, and maintainability.
- Establish engineering and operational practices covering CI/CD, automation, testing, observability, monitoring, release readiness, and production support.
- Lead complex production incident resolution, root-cause analysis, preventive actions, and reliability improvements, while guiding engineers in troubleshooting and reducing operational risk.
- Drive engineering excellence through architecture/design/code reviews, reusable patterns, DevOps practices, and AI-assisted development.
- Strong product mindset, connecting technical decisions and engineering investments to business and customer outcomes.
- Proven ability to lead, mentor, and influence engineers while remaining technically hands-on.
- Strong analytical, problem-solving, troubleshooting, and technical decision-making skills.
- Effective communication and stakeholder management across technical, business, and product teams.
- Ability to navigate ambiguity, balance competing priorities, and proactively manage risks, dependencies, and trade-offs.
- Strong commitment to engineering excellence, operational reliability, and continuous improvement.
- Willingness to provide technical leadership during critical production incidents, major releases, and business-critical situations outside regular working hours when required.