Lead Big Data Engineer

Position: Lead Big Data Engineer
Company: EXL
Experience: 7–9 Years
Location: All EXL Locations (PAN India)
Work Mode: Hybrid
 

Job Summary

We are looking for a highly skilled Lead Big Data Engineer with 7–9 years of experience in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate should have strong expertise in Python, Apache Spark, Hadoop, Hive, and SQL , along with experience building scalable ETL pipelines and working with distributed data platforms.

Key Responsibilities

• Design, develop, and maintain scalable Big Data pipelines using Python and Apache Spark.
• Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
• Work extensively with components of the Hadoop ecosystem, such as Hive and HDFS.
• Develop efficient SQL queries and optimize data models for analytics and reporting.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and cross-functional teams.
• Ensure data quality, reliability, scalability, and performance of data platforms.
• Troubleshoot production issues and implement performance tuning for Spark and Hadoop jobs.
• Mentor junior engineers and contribute to technical design discussions and code reviews.
• Follow best practices in coding, testing, deployment, and documentation.
• Support cloud-based data engineering initiatives where applicable.