Data Engineer

Job Summary:

The Data Engineer plays a crucial role in optimizing our data infrastructure and ensuring seamless data flow across systems. With a focus on ETL/ELT processes and big data pipelines, this position is vital in empowering analytics initiatives that drive business decision-making and strategy.

Key Responsibilities:

• Design and implement robust ETL/ELT pipelines leveraging GCP cloud data platforms to support analytics and operational workflows.
• Monitor and troubleshoot production data pipelines, ensuring consistency, reliability, and data quality throughout various processes.
• Utilize BigQuery for creating and managing datasets, tables, and views with a focus on SQL optimization and cost efficiency.
• Collaborate with data analysts and architects to establish data quality checks, reconciliation processes, and alert systems.
• Develop and maintain streaming and batch frameworks using Dataflow or Dataproc, facilitating real-time data processing needs.
• Build and orchestrate workflows with Cloud Composer/Airflow for efficient data pipeline management and scheduling.
• Implement solutions for event streaming using Pub/Sub or other equivalent technologies.

Requirements:

• Minimum 8 years of experience in data engineering, analytics engineering, or related fields.
• At least 4 years of expertise in GCP cloud data platforms, with a strong focus on BigQuery.
• Proficiency in Python and advanced SQL, with a solid understanding of data quality frameworks.
• Experience in production pipeline monitoring, troubleshooting, and ensuring data quality.
• Demonstrated ability to handle complex datasets and optimize queries effectively in BigQuery.
• Familiarity with cloud orchestration tools, specifically Cloud Composer or similar DAG development environments.
• Knowledge of CI/CD principles for data solutions and monitoring tools for production workflows.

Preferred Qualifications:

• Experience with Dataform, dbt, or similar data transformation tools.
• Hands-on knowledge of BigQuery ML and capabilities in building machine learning models.
• Familiarity with Dataflow templates for efficient pipeline execution.
• Strong understanding of Looker/LookML for data visualization and reporting.
• Proficiency in using Terraform for the provisioning and management of data infrastructure.

Benefits:

• Competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance plans.
• Generous paid time off (PTO) and flexible work-from-home options.
• Continuous learning and development opportunities, including training and certifications.
• Collaborative and dynamic work environment with a focus on innovation.
• Retirement savings plan with company matching.
• Access to the latest tools and technologies to enhance productivity and creativity.