Lead Data Engineer ( Data Integration)
Bombardier
- Dorval, QC
- Permanent
- Full-time
- Lead/Administer the enterprise data platform (DWH, Data Lake).
- Create and maintain optimal/reliable data pipeline architecture to meet business needs.
- Define and operate the infrastructure required for optimal extraction, transformation and loading (ETL) of data from a wide variety of data sources using SQL and 'big data' technologies as needed (Hadoop, MapReduce, Hive, Spark, Kafka, Pig, data streaming, NoSQL, SQL, programming)
- Design and implement life cycle management processes (DevOps) to enable continuous integration and continuous deployment (CICD) of data systems.
- Integrate data from various resources (including external data sources and IoT) and manage the big data as a key enterprise asset.
- Create and maintain backend data solutions for data analysts and data scientists. Assist them in unlocking insight from enterprise data.
- Identify, design, and implement internal process and framework (e.g. elimination of manual processes, optimizing data delivery, evolving data infrastructure capabilities, etc.)
- Work with stakeholders including product, data and architecture SME to assist with data-related technical issues and support their data infrastructure needs.
- Ensure compliance to data architecture and security requirements.
- Identify data quality issue and make recommendation for addressing root causes.
- Setup observability/monitoring to measure reliability of the data pipelines and act quickly (e.g. operational support) in case of incident.
- You hold a bachelor's degree in computer science, Statistics, Informatics, Information Systems or another quantitative field.
- You have 10 years of experience in a Data Engineer / Data Specialist role .
- You have experience coaching/leading a small team (technical leadership).
- You have knowledge of Agile / SCRUM project delivery, DevOps and CICD practices.
- You have advanced knowledge of SQL, query authoring and relational databases.
- You have experience optimizing 'big data' pipelines (storage, file format, partitioning, spark, python, streaming)
- You are efficient at performing root cause analysis to address issues and applying long-term fix.
- You have experience designing and building data transformation, data structures, metadata framework and automated workload management.
- You have experience in data protection measures, data privacy and collaborating with Cyber team
- You have good knowledge of Azure data services (Azure Data Factory, Synapse, Azure Data Lake Storage, Event Hub, Polybase, Databricks, Delta lake, Cognitive Services, …)
- You have good knowledge of Object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
- You have good people skills and are a team player, motivated by developing other people.
- You are a good communicator, who can simplify complex technical issue for non-technical people. You can work in an environment with a mix of French and English languages.