CAN_Engineer
Varite View all jobs
- Mississauga, ON
- Permanent
- Full-time
The QA & Automation Engineer will be responsible for validating ETL pipelines, data migration processes, and cloud data platform integrations across Azure services.
This role involves hands-on testing of large-scale data platforms, automation development, advanced SQL validation, and ensuring end-to-end data quality across enterprise data engineering initiatives.Key Responsibilities:
1. ETL & Data Pipeline Testing
Perform end-to-end testing of ETL pipelines built using Azure Data Factory, Azure Databricks, Azure Synapse, and SSIS.
Validate data transformations, mappings, data quality rules, and data lineage.
Conduct source-to-target (S2T) reconciliation, data profiling, completeness, and accuracy checks.
Verify schema changes, incremental loads, delta loads, and historical load processes.2. Data Migration Testing
Design and execute test strategies for large-scale on-prem to cloud or cross-cloud data migration projects.
Validate ETL/ELT processes and post-migration data accuracy.
Perform count checks, checksum validation, CDC validation, duplicate checks, and referential integrity validation.
Review and validate data mapping documents, business rules, and acceptance criteria.3. Data Lake & Cloud Platform Testing
Test ingestion pipelines into Azure Data Lake Storage (ADLS Gen2) from multiple source systems.
Validate partitioning, folder structures, file formats (Parquet, CSV, JSON), and governance standards.
Conduct performance and scalability testing for large data workloads.
Validate integration flows across Data Lake, Synapse, ADF, and Power BI.4. Automation Testing
Develop automated data validation and regression frameworks using:
Python
PySpark
SQL automation scripts
ADF automated validation frameworks
Integrate automated tests within CI/CD pipelines using Azure DevOps.
Build reusable automation templates, accelerators, and validation utilities.5. Defect & Test Management
Create and maintain test plans, scenarios, and test cases in Azure DevOps or Jira.
Log, track, triage, and validate defects in collaboration with development and data engineering teams.
Publish daily/weekly QA health metrics, defect reports, coverage summaries, and quality dashboards.Essential Skills:
4–10 years of experience in Data Warehouse / Data Engineering QA.
Strong hands-on expertise with:
Azure Data Factory (ADF)
Azure Data Lake (ADLS Gen2)
Azure Synapse Analytics
ETL tools such as ADF, SSIS, Informatica, Databricks
Expert-level SQL for complex data validation and reconciliation.
Strong understanding of data warehousing concepts (SCDs, facts/dimensions, star schema).
Experience with:
Python-based test automation
Azure DevOps (Repos, Pipelines, Boards)
Strong analytical, debugging, and problem-solving skills.Preferred Skills:
Test automation framework development.
Quality management and defect lifecycle management.
Integration of automated tests into CI/CD pipelines using Azure DevOps.Keywords / Technical Skills:
Azure Data Factory
ETL & Data Pipeline Testing
Data Migration Testing
Quality Management
ERP / CRM TestingExperience Required: 4–10 years
Skills: Category Name Required Importance Experience
SkillCategoryTest1_MN ERP/CRM Testing - Performance Testing Oracle Yes 1