Design, orchestrate, and optimize large-scale LLM pre-training across 1,000+ GPUs. Implement 3D parallelism, manage GPU clusters (SLURM/Kubernetes), optimize InfiniBand/RDMA networking and memory, and automate checkpointing and failure recovery for long training runs.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
Artificial Intelligence • Fintech • Information Technology • Marketing Tech • Payments • Social Impact • Software
Leads and builds Donorbox’s product marketing function, owning positioning, messaging, product-led growth, launches, enablement, competitive intelligence, and performance measurement. Partners with Product, Sales, Customer Success, and Marketing to drive acquisition, conversion, retention, and expansion. Manages one direct report, establishes scalable processes and KPIs, influences roadmap priorities, and presents data-driven insights to executive leadership.
Top Skills:
Product-Led Growth (Plg)SaaS
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Lead and develop a Canada Scaled Customer Success team to drive onboarding, adoption, expansion, and renewals via digital scale programs. Hire and coach CSMs, manage KPIs/OKRs, improve processes and tooling, collaborate cross-functionally, and escalate high-stakes customer moments to ensure measurable customer outcomes.
Top Skills:
CRMCs PlatformDigital Engagement ToolsGainsightProduct AnalyticsTotango
.png){kind=logo}
AdTech • eCommerce • Information Technology • Software • Travel • Generative AI
Leads Expedia B2B Operations’ global business enablement portfolio, focusing on continuous improvement, process design, automation, AI-enabled servicing, operational readiness, governance, crisis response, and partner communications. The role redesigns workflows, builds business cases, improves service and financial metrics, governs operational changes, supports launches, and drives adoption across teams, vendors, partners, Product, and Technology.
Top Skills:
Artificial IntelligenceAutomationConversations ApiLean Six SigmaProcess MappingProperty Message CentreRoot-Cause AnalysisSix Sigma
What you need to know about the Ottawa Tech Scene
The capital city of Canada and the nation's fourth-largest urban area, Ottawa has proven a rapidly growing global tech hub. With over 1,800 tech companies, many of which are leaders in their sectors, the city's tech talent now makes up more than 13 percent of its total workforce. This growth is driven not only by the big players like UL Solutions and Dropbox, but also by a thriving startup ecosystem, as new businesses emerge to follow in the footsteps of those that came before them.
