Data engineer-Databricks

About the Role:

 

Databricks and Python Developer with strong background in AWS. This position is responsible for Development and Technical Support with a high degree of customer service, technical expertise and timeliness. The primary purpose of the position is to develop and support Bank data solutions.

Experience in Machine Learning/AI projects is a plus.

Experience in the area of Banking or Finance is desirable.

The ideal candidate will have Databricks on AWS Development and/or Support background with the ability to quickly adapt those skills to new environments. This position will work with Business units and the Technology Group. Occasional travel will be required.

 

 

Duties include  

• Analysis, design, development, support and maintenance of DW, Data Lake and Lakehouse infrastructures 
• Technical expertise on data pipelines and ETL processes. 
• Solid understanding of SQL and Data Modelling
• Hands on experience with Databricks on AWS cloud solutions
• In-depth knowledge of Databricks ETL and Orchestration, notebook development and job scheduling
• Collaborate with cross-functional teams of business analysts, business users and IT stakeholders gathering, documenting, analysing and implementing requirements
• Collaborate with multiple teams to develop and maintain data pipelines, harvesting data from systems of records to create data products
• Create and optimize relational databases schema, tables, indexes, views, and stored procedures
• Perform data migration, transformation, and integration tasks between different database systems
• Develop and maintain ETL processes to load and transform data from various sources
• Work with other development groups to advise, guide and assist in their integrations
• Familiarity with version control systems (e.g. GIT) and Agile development methodologies
• Interact with vendor for support and consultancy
• Work directly with reporting manager to ensure clear and accurate communication of current status, dependencies and estimated delivery timelines
• Clearly communicate issues, risks and proposed solutions to relevant stakeholders​
• Coordinate the release process with business groups, operations in compliance with ITIL Change Management discipline
• Communicate detailed descriptions of functionality changes and provide follow up support as required
• Support QA testing 
• Provide direction & support to the Business Divisions for User Acceptance Testing 
• Utilizing Reporting tools to rapidly create, modify, refresh and update dashboards and reports

 

 

EDUCATION, EXPERIENCE & SKILLS:

 

DW / Business Intelligence / Cloud

• 5-7 years’ experience in DW/BI hands-on development and data-modelling experience
• 3-5 years’ experience in Databricks on Amazon Web Services cloud computing platforms
• 3-5 years’ experience in Reporting and Visualization tools with deep understanding of relevant architecture and framework
  
   

Development

• Experience with business intelligence, data analysis, data modelling and visualization solutions on Cloud, on-premises and hybrid
• Experience with Databricks platform 
• Experience with Python, SQL, relational database objects (Oracle and/or SQL Server) and procedural languages (PL/SQL and/or T-SQL)
• Experience with Version Control systems
• Web Services integration using REST/SOAP/JSON
• Machine Learning/Artificial Intelligence is a plus
• Exposure to SQL Server Integration Services

 

General

• Must have excellent communication skills and be able present diagnostic, troubleshooting steps and conclusions to a varied audience including both business and technical stakeholders
  • excellent listening and analytic skills
  • excellent oral/written communication skills   
  • strong interpersonal and communication skills and ability to deal effectively in a team environment
• Bachelor's degree in Computer Science, or similar discipline, or equivalent education and experience
• Ability to work independently with or without direction and or supervision, portray professional demeanour

 

 

Additional Beneficial Attributes/Skills:

• Agile and SCRUM methodologies
• Exposure generating and manipulating XML and XBRL formats
• Hedge Funds or Financial systems industry domain experience