Data Engineer

Job Summary:

As a Data Engineer specializing in Databricks and Delta Lake, you will play a critical role in designing and implementing robust data pipelines that enable the organization to maximize the value of its data. Your expertise will drive data-driven decision-making across the company by ensuring reliable and high-quality data is readily available to stakeholders.

Key Responsibilities:

• Design and develop scalable data pipelines using Databricks and Delta Lake to support data ingestion, transformation, and analysis.
• Collaborate with data scientists and analysts to understand data requirements and ensure data availability for analytics and reporting.
• Implement data quality checks and monitoring to maintain the integrity and reliability of data sets.
• Optimize data storage and retrieval processes to enhance performance and reduce costs in cloud environments.
• Document data architecture, data models, and ETL processes for team knowledge sharing and compliance.
• Troubleshoot and resolve data-related issues and performance bottlenecks to ensure seamless operations.
• Stay current on emerging technologies and best practices in data engineering, Databricks, and Delta Lake, and recommend improvements.

Requirements:

• Proven experience as a Data Engineer or similar role with a strong focus on Databricks and Delta Lake.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficiency in programming languages such as Python or Scala for data processing.
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) for data storage and processing.
• Knowledge of SQL and data query languages to access and manipulate data effectively.
• Excellent problem-solving skills and attention to detail.
• Strong communication skills and the ability to work collaboratively in a team environment.

Preferred Qualifications:

• Experience with Apache Spark and big data technologies.
• Familiarity with machine learning frameworks and libraries.
• Knowledge of data governance and security practices.
• Prior experience with real-time data processing and streaming technologies.
• Relevant certifications in data engineering or cloud platforms.

Benefits:

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance plans.
• Flexible work hours and remote work options to support work-life balance.
• Access to ongoing professional development and training programs.
• Generous paid time off and holiday policies.
• Retirement savings plan with company matching contributions.
• Inclusive workplace culture that values diversity and encourages innovation.