Specialist Software Engineer - Python, Devops

societegenerale

Bangalore 5 Years Exp Posted 3h ago

Design and implement robust data pipelines using Apache Kafka, Apache NiFi, Apache Spark, and Sqoop.
Manage and optimize distributed data storage systems including Hadoop, HDFS, Druid, and ElasticSearch.
Integrate and maintain data visualization and monitoring tools like Kibana, Grafana, and Logstash.
Ensure efficient data ingestion, transformation, and delivery across various platforms.

Programming & Scripting:

Develop automation scripts and data processing utilities using Python 3 and Shell scripting.
Build reusable components and libraries for data manipulation and system integration.

Observability & Monitoring:

Implement and configure observability agents such as Fluentd, Telegraf, and Logstash.
Collaborate with platform teams to integrate OpenTelemetry for distributed tracing and metrics collection (good to have).
Maintain dashboards and alerts for system health and performance monitoring.

DevOps & CI/CD:

Contribute to CI/CD pipeline development using GitHub Actions.
Collaborate with DevOps teams to ensure seamless deployment and integration of data services.

Collaboration & Documentation:

Work closely with cross-functional teams including data scientists, platform engineers, and product managers.
Document system architecture, data flows, and operational procedures.
Participate in code reviews, knowledge sharing sessions, and technical mentoring.

Required Skills & Qualifications:

Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
5+ years of experience in Big Data engineering and scripting.
Strong hands-on experience with:
- Kafka, NiFi, Hadoop, HDFS, Spark, Sqoop
- ElasticSearch, Druid, Kibana, Grafana
- Python3, Shell scripting
- Logstash, Fluentd, Telegraf
Familiarity with GitHub Actions and basic DevOps practices.
Exposure to OpenTelemetry is a plus.
Excellent problem-solving, analytical, and communication skills.

Preferred Qualifications:

Experience in building real-time data streaming applications.
Knowledge of data governance, security, and compliance in Big Data environments.
- Certifications in Big Data technologies or cloud platforms (AWS/GCP/Azure) are a plus.