Data Engineering Lead - Spark, Scala, AWS

Data Engineering Lead - Spark, Scala, AWS

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start  Caring. Connecting. Growing together.

 

Primary Responsibilities:

We are seeking a hands-on Technical Lead / Data Engineering Leader with 7+ years of experience to design, build, and scale enterprise-grade data platforms using Apache Spark, Scala, and AWS cloud services. The ideal candidate will combine solid technical depth with leadership capability to drive architecture, delivery, and team excellence.

• Technical Leadership and Architecture
  • Design and architect scalable, distributed data processing systems using Spark and Scala.
  • Define and implement cloud-native data architectures on AWS (EMR, S3 etc.).
  • Lead technical design reviews and enforce best practices across teams.
  • Drive system reliability, scalability, and security standards across data platforms.
• Data Engineering and Platform Development
  • Build and optimize step functions for large-scale batch processing
  • Develop high-performance solutions leveraging Spark (batch + streaming) and Scala
  • Enable ingestion and processing of structured, semi-structured, and streaming data sources
  • Ensure data quality, integrity, and governance across the data lifecycle
• Cloud and DevOps Excellence
  • Implement scalable solutions using AWS cloud ecosystem (EMR, S3, Lambda etc.)
  • Drive CI/CD, automation, and monitoring frameworks for data pipelines
  • Optimize pipelines for performance, cost, and fault tolerance
• Team Leadership and Delivery
  • Lead and mentor a team of data engineers and developers, ensuring high-quality delivery
  • Drive Agile execution, sprint planning, and delivery commitments
  • Collaborate with product managers, architects, and business stakeholders to align solutions with business goals
• Foster a culture of code quality, ownership, and continuous improvement
• Innovation and Strategy
  • Evaluate and implement new technologies and tools to optimize data platforms
  • Work closely with data science teams to enable analytics, ML, and AI use cases
  • Contribute to data platform roadmap and modernization initiatives
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Bachelor’s degree or equivalent experience
• 6+ years of experience in Data engineering role
• Working experience on AWS Cloud
• Solid experience in using Spark & Scala
• Solid experience in Python, Kafka, MySQL, and Redis
• Experience with CI/CD tools (Jenkins), GitHub, and Snowflake
• Solid knowledge of data modeling, evaluating large data sets from multiple sources and data transformation
• Proficient with AI Tools and Technologies like LLM’s, Agents, Vector Database, Chatbots, etc.

 

Preferred Qualifications:

• Knowledge of data warehousing, distributed systems, and parallel computing
• Solid understanding of database concepts SQL and NoSQL and data processing tools (Spark)
• Proven ability to work independently, mentor others, and lead initiatives with minimal supervision

Proven excellent problem-solving skills and the ability to perform under pressure