Lead real-time data integration using Azure Databricks, PySpark, Kafka, ADF, and ADLS. Build and optimize medallion-style ETL pipelines processing over 10 million daily records. Improve transformation efficiency, data quality, validation, and downstream reliability by identifying upstream data issues, implementing root-cause fixes, and collaborating with stakeholders.
Real-time data pipelines using Azure Databricks, PySpark, Kafka, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS), enabling reliable ingestion and transformation of 10M+ records daily. • Developed and optimized SQL and PySpark transformations within medallion-style ETL workflows, improving processing efficiency, maintainability, and data quality for downstream analytics. • Identified requirement gaps and data quality issues stemming from upstream data sources and solutioned root-cause fixes by implementing validation rules, refining transformation logic, and collaborating with stakeholders to improve downstream data reliability.
Similar Jobs
Information Technology • Consulting
Leads data integration development using Azure, Hadoop, Spark, Scala, Java, SQL, and relational databases. Responsibilities include data modeling, analytics, solution design, requirements analysis, development, maintenance, and quality delivery in an agile, client-focused environment. The role requires strong problem-solving, interpersonal, communication, and technical skills, along with adaptability, motivation, and the ability to learn quickly.
Top Skills:
AgileSparkAzure CloudAzure Data Factory (Adf)Azure Data Lake Storage (Adls)Azure Databricks (Adb)Data AnalyticsData ModelingDelta LakeHadoopHbaseHdfsHiveIbm Db2J2EeJavaJdbcJSONMapreduceMicrosoft Sql ServerOracleScalaSpark SqlSpringSqoopWeb ServicesXMLYarn
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, develops CI/CD workflows, and mentors engineering teams delivering secure, actionable analytics for banking environments.
Top Skills:
AzureAzure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
Information Technology • Consulting
Leads hands-on data integration and engineering initiatives using Azure Databricks, Azure Data Factory, PySpark, Python, and SQL. Modernizes legacy on-premises systems into cloud-native architectures, implements Medallion Architecture and data warehousing, improves pipeline reliability and compute efficiency, establishes CI/CD workflows, supports banking-grade security, and mentors engineering teams delivering actionable analytics.
Top Skills:
Azure Data FactoryAzure DatabricksCi/CdData WarehousingMedallion ArchitecturePysparkPythonSQL
What you need to know about the Ottawa Tech Scene
The capital city of Canada and the nation's fourth-largest urban area, Ottawa has proven a rapidly growing global tech hub. With over 1,800 tech companies, many of which are leaders in their sectors, the city's tech talent now makes up more than 13 percent of its total workforce. This growth is driven not only by the big players like UL Solutions and Dropbox, but also by a thriving startup ecosystem, as new businesses emerge to follow in the footsteps of those that came before them.
