Lead Data Engineer (.net Python/PySpark & DevOps)

Atlantis

  • Toronto, ON
  • Permanent
  • Full-time
  • 2 days ago
  • Apply easily
Role - Lead Data Engineer (.net Python/PySpark & DevOps)Location - Toronto ( Hybrid )Essential Skills: Job DescriptionDevelop and maintain backend systems using Python, PySpark ensuring high performance, scalability, and reliabilityParticipate in the design and implementation of data engineering and ETL pipelines using PySpark Collaborate with cross-functional teams to identify and prioritize project requirements Mentor and guide engineers, providing technical guidance and code reviewsStay up-to-date with the latest technologies and frameworks, and apply this knowledge to improve existing systems and processesLead the technical direction of the team, including architecture, design, and implementation of software systemsCollaborate with product owners to define and prioritize requirementsDevelop and maintain technical documentation, including architecture diagrams, and technical specificationsParticipate in code reviews, ensuring high-quality code and adherence to coding standardsCollaborate with DevOps and Operations teams to ensure smooth deployment and operation of applicationsExpert in DevOps practices and tools for CICD pipelines.Willingness to learn new technologies and adapt to new challengesProven experience in leading technical teams, including technical guidance, mentoring, and coachingStrong technical leadership skills, including the ability to make technical decisions, prioritize tasks, and manage technical resources Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teamsAbility to drive technical innovation, including researching new technologies, evaluating technical options, and recommending technical solutionsStrong problem-solving skills, with the ability to debug complex issues, optimize system performance, and ensure high-quality software deliveryExperience with Agile development methodologies, including Scrum or Kanban, and the ability to apply these principles to lead the teamExperience with cloud platforms, including Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP)Experience with PySpark for Data Engineering ETL pipelinesFamiliarity with containerization using Docker and orchestration using Kubernetesy
Desirable Skills:8+ years of strong design, development experience using c#, .net core, Web API, angular/react, asp.net, mvc3+ years of strong development experience using sqlserver/oracle/sybase, complex procedures and SSIS/SSRSWorking experience in financial domainExperience working with devops, source code management tools like git, svn, jira, ucd, jenkins.Experience working with Agile/Scrum/RAD development methodologyNice to have container based technologies like docker, kubernetes, openshift ,cloud platform and familiar with big-data and cloud based technologiesNice to have experience in setting application in Linux/Windows, IIS/nginxNice to have working experience with APAC Regulatory ReportingStrong technical analytical skills to solve complex problemsStrategic thinker with excellent interpersonal skills to work across functions and businesses

Atlantis