Hadoop Developer

Job Description:

• Design, develop, and operate end-to-end big-data pipelines on Hadoop, ingesting data from various sources.
• Build robust ETL/ELT workflows using Apache Spark, Hive, Pig, and Sqoop.
• Develop high-throughput streaming data pipelines using Kafka, Spark Streaming, or Flink.
• Optimize Spark and MapReduce jobs to meet SLAs at minimal cost.
• Design and maintain data models and storage layouts on HDFS and related formats.
• Implement data governance and quality controls.
• Build robust monitoring and logging strategies for big-data pipelines.
• Partner with data scientists to deliver reliable datasets.
• Automate pipeline orchestration using Airflow or Oozie.
• Continuously evaluate and adopt new technologies in the big-data ecosystem.
• Mentor junior engineers and contribute to the team’s engineering standards.

Requirements:

• 5+ years of professional experience designing and operating big-data pipelines on Hadoop.
• Strong hands-on expertise with Apache Spark (Scala, Python, or Java) in production environments.
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem.
• Hands-on experience with streaming data platforms such as Kafka, Spark Streaming, or Flink.
• Strong SQL skills and experience working with both relational and NoSQL data stores.
• Experience with workflow orchestration tools such as Airflow or Oozie.
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance.
• Strong scripting skills in Python or Shell.
• Excellent troubleshooting, debugging, and documentation skills.

Benefits:

• Comprehensive benefits
• Competitive compensation packages
• Supportive work-life balance