Process Manager

Data Pipeline Development

• Design, develop, and maintain scalable and reliable data pipelines on GCP.
• Build batch and, where required, near-real-time data ingestion and transformation pipelines.
• Develop robust ETL/ELT frameworks for large volumes of structured and semi-structured data.
• Ingest data from multiple sources into Google Cloud Storage and BigQuery.
• Develop reusable and modular data engineering components.
• Implement appropriate error handling, retry mechanisms, logging, and data validation.

BigQuery & Data Engineering

• Develop complex and optimized SQL queries in BigQuery for data transformation and analysis.
• Optimize BigQuery performance through:
  • Partitioning
  • Clustering
  • Query optimization
  • Efficient table design
  • Appropriate data types and storage strategies
• Monitor and optimize BigQuery processing costs.
• Design scalable data models suitable for large-scale campaign, customer, and clickstream datasets.
• Troubleshoot data quality, performance, and pipeline-related issues.

Airflow / Cloud Composer

• Develop and maintain DAGs using Apache Airflow / Cloud Composer.
• Implement scheduling, dependency management, retries, failure handling, and alerting.
• Monitor production workflows and proactively resolve failed or delayed jobs.
• Build reusable operators and workflow components where required.
• Ensure critical data pipelines meet agreed SLAs.

Python / PySpark

• Develop data processing and transformation logic using Python and/or PySpark.
• Write clean, reusable, scalable, and production-ready code.
• Optimize PySpark jobs for performance and efficient resource utilization.
• Implement appropriate testing and validation for data transformation processes.

GCP Services

Work extensively with GCP services, particularly:

• Google BigQuery
• Google Cloud Storage (GCS)
• Cloud Composer / Airflow

Exposure to additional GCP services such as Cloud Functions, Pub/Sub, Dataflow, Cloud Run, Secret Manager, IAM, or Cloud Monitoring would be an advantage.

Automation, CI/CD & DevOps

• Automate deployment and execution of data engineering workflows.
• Work with Git-based development and version-control workflows.
• Implement and maintain CI/CD pipelines using tools such as Jenkins or equivalent.
• Follow code review, branching, deployment, and release-management processes.
• Implement monitoring, logging, and alerting for production data pipelines.

Production Support & Troubleshooting

• Monitor daily data pipeline execution and resolve production issues within defined SLAs.
• Investigate pipeline failures, data discrepancies, performance issues, and processing delays.
• Perform root-cause analysis and implement permanent fixes.
• Coordinate with application, analytics, infrastructure, and business teams to resolve data-related issues.
• Participate in production deployments and provide post-deployment support.

Stakeholder Collaboration

• Work closely with Data Analysts, Data Scientists, Product Teams, Business Stakeholders, and Technology Teams to understand data requirements.
• Translate business requirements into scalable technical solutions.
• Communicate technical issues, risks, dependencies, and delivery status effectively.
  • Participate in technical discussions, design reviews, and solution development.