Databricks

Job Description – Lead Data Engineer (Azure, Databricks)

Job Title: Lead Data Engineer
Experience: 8–12+ Years
Location:  Delhi NCR, Pune, Bengaluru, Hyderabad
Employment Type: Full-Time

Job Summary

We are seeking an experienced Lead Data Engineer to design, build, and lead the development of scalable cloud-based data platforms and enterprise data pipelines. The ideal candidate will have deep expertise in Azure , Azure Databricks , Azure Data Lake , Azure Data Factory (ADF) , SQL , Python , and ETL/ELT pipeline development . This role requires technical leadership, architecture design, mentoring team members, and collaborating with cross-functional stakeholders to deliver high-quality data solutions.

Key Responsibilities

• Lead the design, development, and implementation of scalable data engineering solutions on Microsoft Azure.
• Architect and build robust ETL/ELT pipelines using Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage (ADLS).
• Develop high-performance data transformation workflows using Python, SQL, and PySpark.
• Design and optimize enterprise data models, data lakes, and modern data platforms.
• Ensure data quality, governance, security, and performance across the data ecosystem.
• Optimize SQL queries and data processing jobs for large-scale datasets.
• Collaborate with business stakeholders, data architects, analysts, and engineering teams to define data requirements.
• Lead code reviews, establish engineering best practices, and mentor junior and mid-level data engineers.
• Implement CI/CD pipelines and DevOps best practices for data engineering projects.
• Monitor production data pipelines, troubleshoot issues, and ensure high availability and reliability.
• Drive continuous improvements in data architecture, automation, and operational efficiency.

Required Skills

• 8+ years of experience in Data Engineering with at least 2+ years in a technical lead role.
• Strong hands-on expertise in Microsoft Azure services.
• Extensive experience with Azure Databricks and PySpark .
• Strong experience with Azure Data Factory (ADF) .
• Expertise in Azure Data Lake Storage (ADLS) .
• Advanced proficiency in SQL with query optimization and performance tuning.
• Strong programming skills in Python .
• Proven experience designing and maintaining scalable ETL/ELT pipelines .
• Experience with data warehousing, data modeling, and big data processing.
• Knowledge of Delta Lake, Lakehouse architecture, and modern cloud data platforms.
• Experience with Git, Azure DevOps, CI/CD, and Agile methodologies.

Preferred Skills

• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Exposure to Spark optimization and distributed data processing.
• Knowledge of streaming technologies such as Kafka or Event Hubs.
• Experience with Power BI or other BI/reporting tools.
• Azure Data Engineer Associate certification is a plus.