Data Engineer

Job Summary:

The Data Engineer will play a pivotal role in designing and implementing scalable data solutions to support our organization's analytics and data-driven decision-making processes. This position focuses on leveraging Databricks, Python, PySpark, SQL, and Azure to transform raw data into structured formats, driving efficiency and innovation across the business.

Key Responsibilities:

• Design and develop data pipelines using Databricks to ingest, process, and store large datasets.
• Implement data transformation processes in Python and PySpark to ensure data quality and integrity.
• Create and optimize SQL queries for efficient data retrieval and analysis.
• Collaborate with data scientists and analysts to understand data requirements and provide relevant data sets.
• Utilize Azure services to manage and deploy data solutions effectively within cloud environments.
• Monitor and troubleshoot data processing workflows, ensuring timely resolution of issues.
• Document data architectures, processes, and best practices to support knowledge sharing and compliance.

Requirements:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Proven experience as a Data Engineer or in a similar role focusing on data pipeline construction.
• Strong proficiency in Python and PySpark for data processing and manipulation.
• Experience with Azure cloud services and data lake architecture.
• Solid understanding of SQL for database management and query optimization.
• Knowledge of data modeling concepts and ETL processes.
• Excellent problem-solving skills and attention to detail.

Preferred Qualifications:

• Certifications in Azure or Databricks.
• Experience with big data technologies such as Apache Spark or Hadoop.
• Familiarity with tools like Power BI or Tableau for data visualization.
• Background in machine learning or statistical analysis.
• Strong communication and collaboration skills to work effectively within cross-functional teams.

Benefits:

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Generous paid time off and flexible working hours.
• Opportunities for professional development and continuous learning.
• Dynamic and inclusive work environment with a focus on innovation.
• Access to cutting-edge technology and tools to enhance your workflow.
• Employee wellness programs and team-building activities.