Data Engineer
idfcfirst
Job Description
Primary Responsibilities
- Minimum 2 to 3 years of hands-on experience in data engineering.
- Proven expertise in SQL, Spark, and the Hadoop ecosystem.
- Experience handling multiple terabytes of data from ingestion to consumption.
- Collaborate with business stakeholders to identify and document high-impact business problems and potential solutions.
- Strong understanding of Data Lake/Lakehouse architecture and exposure to Hadoop (Cloudera, Hortonworks) and/or AWS.
- Manage the full data lifecycle: ingestion, transformation, and consumption.
- Proficient in Spark, Spark Streaming, Hive, and SQL.
- Experience with big data infrastructure including MapReduce, Hive, HDFS, YARN, HBase, Oozie, etc.
- Deep knowledge of relational databases and data modeling.
- Technical debugging skills and experience with Git for version control.
- Create and maintain technical design documentation for data pipelines and projects.
Secondary Responsibilities
- Ability to work independently and manage personal development efforts.
- Strong oral and written communication skills.
- Learn and apply internal analytics technologies.
- Identify key performance indicators and develop strategies to achieve them.
- Use educational background in data engineering to perform data mining and analysis.
- Collaborate with BI analysts/engineers to build prototypes.
- Participate in the delivery and presentation of data solutions.
What We Are Looking For
Education
- Bachelor’s degree in Computer Science, Information Technology, Data Engineering, or a related field.
Experience
- 2 to 5 years of relevant experience in data engineering.
Skills and Attributes
- Strong coding and debugging skills.
- Proficiency in big data tools and cloud platforms.
- Ability to work with large datasets and complex data architectures.
- Effective communication and stakeholder engagement.
- Self-driven with a continuous learning mindset.
- Strong documentation and organizational skills.
Key Success Metrics
- Timely and high-quality delivery of data engineering projects.
- Excellent code quality and maintainability.
- Identification and resolution of data issues.
- Effective technical solutioning and error-free execution.