Lead real-time data integration using Azure Databricks, PySpark, Kafka, ADF, and ADLS. Build and optimize medallion-style ETL pipelines processing over 10 million daily records. Improve transformation efficiency, data quality, validation, and downstream reliability by identifying upstream data issues, implementing root-cause fixes, and collaborating with stakeholders.
Real-time data pipelines using Azure Databricks, PySpark, Kafka, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS), enabling reliable ingestion and transformation of 10M+ records daily. • Developed and optimized SQL and PySpark transformations within medallion-style ETL workflows, improving processing efficiency, maintainability, and data quality for downstream analytics. • Identified requirement gaps and data quality issues stemming from upstream data sources and solutioned root-cause fixes by implementing validation rules, refining transformation logic, and collaborating with stakeholders to improve downstream data reliability.
Similar Jobs
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, develops CI/CD workflows, and mentors engineering teams delivering secure, actionable analytics for banking environments.
Top Skills:
AzureAzure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, establishes CI/CD workflows, supports banking-grade security, and mentors engineering teams delivering actionable analytics.
Top Skills:
Azure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
Artificial Intelligence • Information Technology • Professional Services • Consulting
Designs, develops, and maintains end-to-end integrations between Oracle EPM Cloud applications and source or target systems. Responsibilities include data mapping, transformation, validation, automation with scripts and REST APIs, integration monitoring, troubleshooting, stakeholder collaboration, and technical documentation. The role requires extensive Oracle EPM integration experience, SQL proficiency, scripting or development skills, and familiarity with testing and Oracle Integration Cloud.
Top Skills:
Ansi SqlArcsC#Cloud Data ManagementEpbcsEpm AutomateEpm Integration AgentFccsFdmeeGroovyJavaJythonOdiOracle Epm CloudOracle Epm Rest ApisOracle Integration CloudOracle SqlPcmcsPipelinesPowershellPythonRelational DatabasesT-SqlVbscript
What you need to know about the Ottawa Tech Scene
The capital city of Canada and the nation's fourth-largest urban area, Ottawa has proven a rapidly growing global tech hub. With over 1,800 tech companies, many of which are leaders in their sectors, the city's tech talent now makes up more than 13 percent of its total workforce. This growth is driven not only by the big players like UL Solutions and Dropbox, but also by a thriving startup ecosystem, as new businesses emerge to follow in the footsteps of those that came before them.
