Senior Data Engineer

Job Summary:

We are seeking a skilled Senior Data Engineer with a strong background in Python, PySpark, and Azure Cloud services to join our dynamic team. In this role, you will play a critical part in designing and implementing data solutions that drive business insights and support strategic decision-making.

Key Responsibilities:

• Design and develop scalable data pipelines and ETL processes using Python and PySpark.
• Implement and manage cloud-based data solutions on Microsoft Azure, ensuring data integrity and accessibility.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions.
• Optimize existing data workflows and systems for improved performance and reduced costs.
• Utilize Databricks for big data processing and analytics, ensuring seamless integration with Azure services.
• Monitor and troubleshoot data pipeline performance and availability issues.
• Document processes, data flows, and technical specifications to facilitate knowledge sharing and project transparency.

Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 8-10 years of experience in data engineering or related roles, with extensive knowledge of Python and PySpark.
• Proficiency in Microsoft Azure, including services such as Azure Data Lake, Azure SQL Database, and Azure Data Factory.
• Strong experience with Databricks for data analytics and processing.
• Solid understanding of data architecture principles and best practices, including data modeling and schema design.
• Excellent problem-solving skills and the ability to troubleshoot complex data issues efficiently.
• Effective communication skills, with the ability to collaborate across teams and present technical concepts to non-technical stakeholders.

Preferred Qualifications:

• Experience with machine learning frameworks and libraries, such as TensorFlow or Scikit-learn.
• Knowledge of data warehousing concepts and experience with tools like Snowflake or Redshift.
• Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.
• Prior experience in an Agile or Scrum development environment is a plus.

Benefits:

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance plans.
• Opportunities for career advancement and professional development.
• Flexible work hours with options for remote work.
• Generous paid time off and holiday packages.
• A collaborative and innovative work environment with a focus on employee well-being.