Databricks Developer / Data Engineer

Job Summary:

We are seeking a skilled Databricks Developer / Data Engineer with 5+ years of experience in building scalable data engineering solutions and modern lakehouse architectures. The ideal candidate will leverage their strong hands-on expertise in Databricks, PySpark, Spark SQL, Delta Lake, and Azure Cloud to design, develop, and optimize high-performance data pipelines that make a significant impact on our data-driven decision-making.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake.
• Build and optimize data ingestion frameworks, transformation logic, and end-to-end data workflows for both batch and streaming use cases.
• Implement Delta Lake-based architectures, including schema evolution, versioning, and ACID-compliant pipelines to ensure data integrity.
• Develop scalable Lakehouse solutions while adhering to Medallion Architecture best practices for efficient data organization.
• Collaborate with business stakeholders, analysts, and fellow data engineers to gather requirements and deliver effective data solutions that meet organizational needs.
• Manage and optimize Databricks clusters, jobs, notebooks, and workflows for maximum performance and cost efficiency.
• Ensure data quality, governance, observability, and reliability through automated validation and monitoring frameworks while contributing to ongoing data modeling and platform modernization initiatives.
• Implement CI/CD pipelines and adhere to DevOps best practices for streamlined data engineering workflows.
• Troubleshoot and resolve performance bottlenecks in Spark applications and data pipelines, employing best practices for continuous improvement.

Requirements:

• 5+ years of experience in Data Engineering with a focus on building scalable solutions.
• 2+ years of hands-on experience with Databricks in production environments.
• Strong expertise in PySpark, Spark SQL, and a comprehensive understanding of distributed data processing.
• Solid experience with SQL and large-scale data transformation projects.
• Strong understanding of Delta Lake, Lakehouse Architecture, Medallion Architecture (Bronze, Silver, Gold), data modeling, and data quality frameworks.
• Experience with Azure Databricks and Azure-based data solutions is essential.
• Good knowledge of CI/CD, Git, and DevOps practices to enhance development workflows.
• Proficiency in working with both structured and unstructured datasets.
• Strong troubleshooting, optimization, and performance tuning skills to enhance data processing efficiency.

Preferred Qualifications:

• Experience in implementing modern data architecture patterns and solutions.
• Familiarity with advanced data governance frameworks and tools.
• Knowledge of additional programming languages like Python.
• Ability to communicate complex technical concepts to non-technical stakeholders.
• Passion for learning about new technologies and continuous professional development.

Benefits:

• Competitive salary and performance-based incentives.
• Comprehensive health, dental, and vision insurance plans.
• Flexible working hours with a supportive work-from-office environment.
• Opportunities for professional development and continuous learning.
• Collaborative and inclusive company culture that values diversity.
• Access to the latest tools and technologies in data engineering.
• Generous paid time off and holiday policies, encouraging work-life balance.