Data/ETL Engineer
ascendion
Job Description
- Data Engineering and Processing:
- Develop, deploy, and manage scalable ETL/ELT pipelines using AWS Glue and PySpark.
- Work with AWS Data Catalog for metadata management and data discovery.
- Optimize query performance and data analysis workflows using AWS Athena.
- Implement serverless data workflows using AWS Lambda for real-time and batch data processing.
- Database and Storage Management:
- Design and manage relational databases on AWS RDS (PostgreSQL) and non-relational databases on DynamoDB.
- Store, manage, and optimize data in AWS S3 for performance, security, and cost efficiency.
- Infrastructure as Code (IaC) (good to have):
- Build and deploy AWS infrastructure using Terraform and Terragrunt.
- Automate the provisioning of scalable, secure, and reliable resources.
- Cloud Services and Integrations (good to have):
- Manage containerized data applications on AWS ECS for efficient deployment and scaling.
- Design and implement event-driven architectures using AWS SQS and SNS.
- Configure AWS CloudFront for secure, low-latency content delivery.
- Security and Access Management (should understand how its implemented):
- Design and implement IAM roles and policies to ensure secure access and compliance.
- Ensure data security, including encryption at rest and in transit.
- Collaboration and Agile Practices:
- Collaborate with cross-functional teams, including data scientists, analysts, and developers, to understand data requirements.
- Actively participate in Agile ceremonies, such as sprint planning, standups, and retrospectives.
- Version Control and Continuous Integration:
- Utilize GitHub for version control, collaboration, and CI/CD workflows.
- Ensure best practices in code management and deployment processes.
- Monitoring and Optimization:
- Monitor and troubleshoot data workflows and AWS resources using AWS CloudWatch and other tools.
- Optimize data pipelines and resource configurations for performance and cost.