Senior Azure Data Engineer - Databricks
Recutify Inc.
- Toronto, ON
- $65.00 per hour
- Permanent
- Full-time
This job is for an existing, active vacancy and We are looking for a Senior Azure Data Engineer with strong expertise in Azure Databricks, Delta Lake, and Medallion Architecture to design and deliver scalable, high performance data platforms.This role will lead complex data engineering initiatives, influence architectural decisions, and partner closely with analytics, data science, and platform teams.Key ResponsibilitiesData Architecture & LeadershipDesign, implement, and evolve Medallion Architecture (Bronze, Silver, Gold) using Delta Lake on Azure Databricks.Lead data architecture decisions, establish engineering standards, and define best practices.Review designs and code from team members, ensuring scalability, performance, and data quality.Data Engineering & DevelopmentBuild and maintain high volume, fault tolerant ETL/ELT pipelines using Azure Databricks and Azure Data Factory.Develop optimized PySpark transformations for large scale batch and streaming data.Implement reusable frameworks for ingestion, validation, transformation, and enrichment.Data Ingestion & StreamingArchitect and develop ingestion pipelines for structured, semi structured, and unstructured data.Implement batch and real time streaming pipelines using Azure Event Hubs / Kafka / Structured Streaming.Support incremental loads and CDC based data ingestion strategies.Performance, Optimization & ReliabilityOptimize Spark workloads using partitioning, caching, Z Ordering, and file sizing strategies.Tune Delta Lake tables for performance and cost efficiency.Monitor pipeline health, troubleshoot failures, and improve data platform reliability.Cloud, Security & GovernanceLeverage Azure services including ADLS Gen2, Synapse Analytics, Key Vault, Azure Monitor, and networking/security patterns.Implement role based access control, secrets management, and data protection best practices.Align data solutions with governance, compliance, and auditing requirements.Collaboration & MentorshipWork closely with data scientists, analysts, product owners, and business stakeholders.Enable advanced analytics and machine learning workloads.Mentor junior and mid level data engineers and contribute to team skill growth.Required Skills & Qualifications7+ years of data engineering experience with strong Azure cloud expertise.Hands on expertise with Azure Databricks, Delta Lake, and Medallion Architecture.Advanced proficiency in PySpark and Python.Strong experience with Azure Data Factory for orchestration.Deep understanding of data modeling, data lakes, and big data processing.Proven ability to design scalable, production grade data platforms.Strong communication skills and experience working in agile environments.Preferred QualificationsExperience implementing CI/CD pipelines for data platforms (Azure DevOps, GitHub Actions).Exposure to streaming technologies (Kafka, Event Hubs, Structured Streaming).Familiarity with data governance, cataloging, and lineage tools (e.g., Azure Purview).Experience supporting ML and advanced analytics workloads.Prior experience in financial services, healthcare, retail, or large scale enterprise environments.
We are sorry but this recruiter does not accept applications from abroad.