Lead Data Engineer (.net Python/PySpark & DevOps)

Atlantis

  • Toronto, ON
  • Permanent
  • Full-time
  • 2 days ago
  • Apply easily
Lead Data Engineer (.net Python/PySpark & DevOps)TorontoAIRFLOW, PYSPARK, DATABRICKS, PYTHON, KAFKAEssential Skills: Job DescriptionDevelop and maintain backend systems using Python, PySpark ensuring high performance, scalability, and reliabilityParticipate in the design and implementation of data engineering and ETL pipelines using PySparkCollaborate with cross-functional teams to identify and prioritize project requirementsMentor and guide engineers, providing technical guidance and code reviewsStay up-to-date with the latest technologies and frameworks, and apply this knowledge to improve existing systems and processesLead the technical direction of the team, including architecture, design, and implementation of software systemsCollaborate with product owners to define and prioritize requirementsDevelop and maintain technical documentation, including architecture diagrams, and technical specificationsParticipate in code reviews, ensuring high-quality code and adherence to coding standardsCollaborate with DevOps and Operations teams to ensure smooth deployment and operation of applicationsExpert in DevOps practices and tools for CICD pipelines.Willingness to learn new technologies and adapt to new challengesProven experience in leading technical teams, including technical guidance, mentoring, and coachingStrong technical leadership skills, including the ability to make technical decisions, prioritize tasks, and manage technical resourcesExcellent communication and collaboration skills, with the ability to work effectively with cross-functional teamsAbility to drive technical innovation, including researching new technologies, evaluating technical options, and recommending technical solutionsStrong problem-solving skills, with the ability to debug complex issues, optimize system performance, and ensure high-quality software deliveryExperience with Agile development methodologies, including Scrum or Kanban, and the ability to apply these principles to lead the teamExperience with cloud platforms, including Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)Experience with PySpark for Data EngineeringETL pipelinesFamiliarity with containerization using Docker and orchestration using Kubernetesy
Desirable Skills: 8+ years of strong design, development experience using c#, .net core, Web API, angular/react, asp.net, mvc 3+ years of strong development experience using sqlserver/oracle/sybase, complex procedures and SSIS/SSRS Working experience in financial domain Experience working with devops, source code management tools like git, svn, jira, ucd, jenkins. Experience working with Agile/Scrum/RAD development methodology Nice to have container based technologies like docker, kubernetes, openshift ,cloud platform and familiar with big-data and cloud based technologies Nice to have experience in setting application in Linux/Windows, IIS/nginx Nice to have working experience with APAC Regulatory Reporting Strong technical analytical skills to solve complex problems Strategic thinker with excellent interpersonal skills to work across functions and businesses

Atlantis