Databricks Platform Engineer

Job Summary:

The Databricks Platform Engineer will play a pivotal role in the architectural design, implementation, and optimization of Databricks workspaces and data pipelines across multiple environments. This role will significantly enhance our data processing capabilities, drive efficiency in data governance, and support machine learning workloads, ultimately contributing to the success of data-driven initiatives within the organization.

Key Responsibilities:

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.
• Configure and manage Databricks Unity Catalog to enforce data governance, access control, and fine-grained permissions.
• Optimize cluster configurations and auto-scaling policies to ensure cost-effectiveness and optimal performance.
• Implement best practices for workspace management, including folder structures, access controls, and secret management strategies.
• Manage Databricks jobs, workflows, and multi-task job orchestration with proficient dependency management.
• Develop and maintain scalable batch and streaming data pipelines utilizing PySpark and Spark SQL.
• Set up and manage MLflow for model lifecycle management and support AI/ML workloads effectively.

Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience with Databricks platform, Delta Lake, and data lakehouse architecture.
• Strong proficiency in building data pipelines using PySpark and SQL.
• Experience with CI/CD practices and tools such as Azure DevOps and GitHub Actions.
• Familiarity with cloud-native data storage solutions like Azure Data Lake Storage (ADLS) or AWS S3.
• Excellent understanding of data governance principles and experience with Unity Catalog.
• Strong problem-solving skills and the ability to work collaboratively in a team environment.

Preferred Qualifications:

• Master’s degree in a relevant field.
• Experience in AI/ML model deployment and management on Databricks.
• Knowledge of Delta Live Tables and related best practices for ETL/ELT processes.
• Certification in Databricks or relevant cloud platforms (AWS/Azure).
• Familiarity with containerization and orchestration tools like Docker and Kubernetes.

Benefits:

• Competitive salary with performance-based bonuses.
• Comprehensive health, dental, and vision insurance plans.
• Flexible working hours and remote work options.
• Generous paid time off policy, including vacation and sick leave.
• Professional development opportunities with training and certifications.
• Collaborative and inclusive work environment that values diversity.
• Access to cutting-edge technology and tools for your role.