Skip to content
TensorWave

Senior Hardware Diagnostics Engineer - Infrastructure

Las Vegas, NevadaOnsite · United States
Senior · FullTime · Engineering
POSTED 23 OCTVERIFIED 3H AGO
Apply Ask
Open for
10d
closes ~28d here
H-1B · FY26
1
certified LCAs
Hiring · 30d
42
−20 net · open roles
Node.jsPython

About TensorWave

Our mission is simple: deliver seamless, secure, reliable, and resilient AI compute at scale. We've built a versatile cloud platform that eliminates infrastructure barriers, empowering builders to focus on innovation instead of fighting their stack. Because breakthrough AI should move at the speed of ideas, not infrastructure.

About the Role

We are building and operating large-scale infrastructure platforms to support high-performance AI workloads across multiple data centers. Our environment includes GPU-intensive systems, high-throughput networking, and rapidly scaling compute clusters.

We are looking for a Senior Hardware Diagnostics Engineer to own that surface: which versions we run, how we prove a version is safe, and how we push it across a fleet without breaking production. You'll set the bar for what validated hardware means here, build the tooling that enforces it, and be the person we escalate to when a failure doesn't make sense.

Firmware is where a lot of our hardest problems live. A BIOS revision changes how a GPU enumerates. A BMC update fixes one thing and breaks telemetry.

What You’ll Do

Own the firmware lifecycle across BIOS, BMC, NIC, GPU, and storage controllers: version qualification, approved baselines, staged rollout, and rollback

Design and build the validation harness that qualifies a firmware release before it reaches production, including regression and performance comparison between versions

Build automation against Redfish, IPMI, and vendor management APIs so firmware and validation work runs at fleet scale rather than node by node

Define what passing burn-in means: the test matrix, the thresholds, and the evidence we keep

Make fleet-wide changes safe and predictable through pre-flight validation, blast radius control, staged waves, and a rollback path that has actually been tested

Lead root cause analysis on hardware and firmware failures nobody else can reproduce, including PCIe error analysis, sensor and event log forensics, and thermal and power telemetry

Own technical escalation with OEM, ODM, and silicon vendors on firmware defects, and track fixes to closure

Prepare validation and bring-up capability for next-generation AMD platforms before the hardware lands

Mentor the diagnostics engineer on the team, and turn recurring findings into runbooks and tooling

Participate in incident response and root cause analysis

Contribute to runbooks, documentation, and operational best practices

Who You Are

Required Qualifications

6+ years in infrastructure engineering, hardware validation, systems engineering, or a closely related field

Deep hands-on experience with BMCs and out-of-band management — Redfish, IPMI, iDRAC, iLO, or equivalent — including scripted interaction, not clicking through a web UI

Direct firmware experience flashing, qualifying, and troubleshooting BIOS, BMC, and device firmware on enterprise servers, and a clear sense of how badly this can go wrong at scale

Strong Linux systems skills: boot process, kernel and driver behavior, and hardware diagnostics such as {{dmesg}}, {{lspci}}, {{ipmitool}}, SMART, and vendor CLIs

Ability to read low-level hardware signals: PCIe topology and error output, SEL and sensor data, thermal and power telemetry

Python for automation against REST APIs — Redfish, vendor tooling, and internal systems

A rigorous diagnostic method: you form a hypothesis, design a test that can disprove it, isolate one variable at a time, and document the evidence behind your conclusion

Experience driving hardware issues with vendors through to resolution

Clear technical writing for runbooks, RCAs, and vendor escalations

Preferred Qualifications

AMD GPU and ROCm experience: {{amd-smi}}, {{rocm-smi}}, ROCm Validation Suite

GPU and HPC validation at scale: stress testing, collective communication testing with RCCL or NCCL, node health frameworks

Low-level bus and firmware work: I2C, SMBus, BMC firmware behavior, Redfish schema extensions

Kernel-level debugging, including crash dump analysis

Ansible for fleet operations, and Git-based workflows with CI/CD

Source · TensorWaveFirst seen 2 NovOriginal ↗