# GPU Cluster Engineer, Systems & Platform at Sciforium

> Onsite · United States · Unspecified · Software engineering · open. Source: https://alderhq.app/jobs/ashby%3ASciforium%3A62d7ad5e-d00b-4e6d-b5be-b02ea9ef65ea (read 2026-11-04).

## Facts (observed)

- Company: [Sciforium](https://alderhq.app/companies/sciforium)
- Location text: San Francisco, CA
- Where it can be done from (inferred): Onsite · United States (evidence: "Location: San Francisco, CA")
- Posted: 2026-08-06 (90 days ago)
- First seen by alder: 2026-11-02 · last verified: 2026-11-03
- Reposted: 0 times · edited: 0 times
- Apply on the employer's site: https://jobs.ashbyhq.com/Sciforium/62d7ad5e-d00b-4e6d-b5be-b02ea9ef65ea
- Skills listed (inferred): Node.js, Python, Kubernetes, Docker, PyTorch

## Pay and sponsorship

- Posted pay: not posted
- Market band for similar posted roles (SF Bay Area, n 1378): p25 $163k, median $200k, p75 $248k
- H-1B filings at this company for similar roles, FY26: 4, median offered wage $188k ([US DOL OFLC LCA disclosure data](https://www.dol.gov/agencies/eta/foreign-labor/performance))

## Description (the employer's text)

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.

### About the Role

We are looking for a GPU Cluster Engineer to own the entire software stack of our GPU clusters — from kernel tuning and GPU drivers up through schedulers, containers, and ML frameworks. While our Hardware Operations team keeps the physical machines healthy and connected, you define what a production-ready node looks like in software: you author the images, playbooks, and pipelines that take a freshly provisioned server to a fully validated GPU node, and you keep the fleet consistent, upgradable, and fast. You will serve two demanding customer groups — our foundation model training teams and our model serving/product teams — ensuring both run on correctly configured, well-managed, high-performance infrastructure.

### Key Responsibilities

### OS Bring-Up & Node Lifecycle Engineering

Golden Images & Automated Bring-Up: Own the node software definition — versioned OS images, kernel tuning (NUMA, hugepages, IRQ affinity, cgroups), GPU/NIC driver stacks — and the automated pipeline that takes a node from base OS to production-ready.

Validation & Burn-In: Build automated acceptance suites (DCGM diagnostics, nccl-tests/RCCL tests, bandwidth and topology checks, HPL) that gate every node before it enters a scheduler pool.

Fleet Maintenance: Execute rolling kernel/driver/toolkit upgrades with minimal disruption to running workloads; enforce configuration consistency, detect drift, and maintain the driver ↔ CUDA/ROCm ↔ framework compatibility matrix across the fleet.

Self-Healing Operations: Automate detection of unhealthy nodes (Xid/ECC errors, link flaps, thermal throttling), with cordon/drain/reboot/re-image workflows and clean handoff to Hardware Operations for physical repair or RMA.

### Configuration Management & Automation

Infrastructure as Code: Manage all node and cluster configuration through Ansible/SaltStack playbooks in Git, with peer-reviewed changes, CI validation, and canary rollouts before fleet-wide deployment.

Provisioning Pipelines: Build and maintain image/provisioning tooling (PXE, MaaS, Packer, or similar) so new or re-imaged nodes are reproducible, not hand-crafted.

Operational Tooling: Develop Python/Bash tooling for cluster operations, health reporting, and workflow automation.

### Orchestration & Scheduling (Kubernetes & Slurm)

Kubernetes for Serving: Deploy and operate GPU-enabled Kubernetes for inference workloads — NVIDIA GPU Operator, device plugins, node feature discovery, topology-aware scheduling, and MIG/MPS partitioning where appropriate.

Training Schedulers: Operate Slurm (or Run:AI) for multi-node training — partitions, QoS, preemption, accounting, and container integration (enroot/pyxis).

Container Platform: Maintain base images, registries, and the NVIDIA Container Toolkit / ROCm container stack; keep training and serving images lean, current, and reproducible.

### GPU Driver & ML Stack Engineering

Driver & Runtime Lifecycle: Build, deploy, and debug the full accelerator stack — NVIDIA (CUDA toolkit, cuDNN, NCCL, Fabric Manager) and AMD (ROCm, RCCL) — including kernel modules (DKMS), GPUDirect RDMA/Storage, and the RDMA software stack (MOFED/DOCA).

Framework Environments: Maintain curated, optimized PyTorch and JAX environments with sane dependency and version management for researchers and production services.

Distributed Performance: Tune NCCL/RCCL across NVLink/NVSwitch and InfiniBand/RoCE fabrics, ensure topology-aware job placement, and run continuous communication/throughput benchmarks to catch regressions.

### Advanced Debugging & Observability

Escalation Point: Own the hard problems — NCCL hangs and timeouts, CUDA memory leaks, ROCm kernel crashes, straggler nodes, and unexplained throughput drops.

---
Data: alder (https://alderhq.app). Cite https://alderhq.app/jobs/ashby%3ASciforium%3A62d7ad5e-d00b-4e6d-b5be-b02ea9ef65ea and the date you read it. Observed facts come from the employer's own listing; location reach, level, family and skills are inferences.