Databricks Lead

Job Summary:

We are seeking a highly skilled Databricks Lead with over 8 years of experience to spearhead our core engineering initiatives. This role plays a pivotal part in driving our data engineering processes, enhancing our cloud data solutions, and optimizing data management practices across the organization.

Key Responsibilities:

• Lead the design and implementation of data engineering solutions using Databricks and Snowflake, ensuring scalability and performance.
• Collaborate with cross-functional teams to gather requirements, analyze data needs, and develop production-grade APIs for seamless data integrations.
• Execute comprehensive data analysis on large datasets to extract insights and support data-driven decision-making across departments.
• Manage cloud infrastructure on Azure, AWS, or GCP, ensuring optimal resource utilization and security compliance.
• Utilize SQL for querying databases and developing efficient data pipelines, focusing on data quality and integrity.
• Implement and oversee CI/CD pipelines to facilitate smooth deployment and monitoring of data workflows.
• Mentor junior engineers and promote best practices in data engineering and cloud technologies.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 8+ years of experience in data engineering, with a strong focus on Databricks and data analytics.
• Proficiency in Python and PySpark for data manipulation and API development.
• Extensive experience in working with large volumes of data and performing complex data analyses.
• Familiarity with cloud platforms (Azure, AWS, GCP) and associated data services.
• Solid understanding of SQL for data querying and processing.
• Experience with container technologies, CI/CD practices, and data monitoring tools.

Preferred Qualifications:

• Hands-on experience with data governance and management best practices.
• Knowledge of machine learning frameworks and data pipeline orchestration tools.
• Familiarity with Apache Spark and real-time data streaming technologies.
• Certifications in cloud services or data engineering will be considered a plus.