Data Engineer II

• Optimize existing data pipelines built using PySpark for performance and scalability.
• Review and tune SQL queries to improve execution time and resource utilization.
• Design and develop new batch data pipelines that meet evolving business needs (real-time decisioning is not required; systems can tolerate up to 6 hours of delay).
• Automate pipeline generation and metadata management to streamline development.
• Implement data quality and validation frameworks to ensure reliability and trust in data assets.
• Collaborate with teams to integrate and share data via DataHub, leveraging connectors such as SFTP and Snowflake Data Share.
• Enhance observability by integrating and monitoring pipelines through Grafana and other telemetry tools.
• Work closely with other engineers to improve coding practices and leverage AI-powered developer assistants like Cursor, Copilot, or Windsurf for productivity.
• Participate in design reviews, documentation, and operational support for production systems.

Work Experience

What You’ve Done

• 5–8 years of experience as a Data Engineer working with large-scale distributed data systems.
• Strong proficiency in Python (especially PySpark) and hands-on experience optimizing Spark jobs on EMR or similar environments.
• Deep knowledge of SQL tuning and query optimization across analytical databases (preferably Snowflake).
• Experience working with Airflow for workflow orchestration and Livy for Spark job management.
• Exposure to Flink and Apache Iceberg for batch or incremental data processing.
• Strong understanding of data quality frameworks, testing, and validation techniques.
• Familiarity with DataHub or similar data catalog and sharing platforms.
• Experience setting up dashboards and alerts in Grafana or equivalent monitoring tools.
• Exposure to AI coding assistants (Cursor, GitHub Copilot, Windsurf) for accelerating development is a plus.
  • Passionate about automation, continuous improvement, and building scalable, maintainable systems.