Skip to content
ALQUA PROOF

Busy is not the same as working.

Proof checks every training run and inference endpoint while it runs. Within minutes it tells you whether a slowdown is the job or the infrastructure, and shows the evidence.

{{ laneTitle }}{{ pct }}%
{{ l.id }}
{{ lgComp }}{{ lgWait }}{{ timeTxt }}
CASE FILE 01 — THE CUSTOMER'S JOB The rotating straggler

A 16-GPU fine-tune runs 30% slower than paid for. Every GPU reports busy, the fabric is clean, and no error appears in the log. Each step, the rank that draws the longest documents holds the others back, and the slow rank changes every step.

▸ engine active ≈ 100% on all ranks▸ tensor cores idle on waiting ranks▸ cross-rank activity correlation collapses
VERDICT → SEQUENCE-LENGTH IMBALANCE. FIX THE SAMPLER, KEEP THE GPUS.
 CASE FILE 02 — THE INFRASTRUCTURE Checkpoint stalls

At every save the whole job goes quiet. Nothing fails, the log prints "checkpoint saved", and the run finishes a day late. The writes land on an overloaded shared volume and block every rank.

▸ multi-second gaps between kernel launches▸ gaps line up with save lines in the job log▸ shared storage throughput collapses, zero errors
VERDICT → THE STORAGE, NOT THE TRAINING. CHECKPOINTS MOVED TO LOCAL NVME.
 CASE FILE 03 — INFERENCE Half-empty batches

An Arabic chat endpoint runs on 8 GPUs. Latency is inside the SLA, the bill is not: cost per million tokens is 2.4 times the target. Long prompts are processed one by one while decode slots sit empty.

▸ batch occupancy: 3 of 32 slots on average▸ KV cache 71% free at peak▸ prefill takes most GPU time on long prompts
VERDICT → CONTINUOUS BATCHING AND PREFIX CACHE. SAME SLA ON 3 GPUS.
 CASE FILE 04 — THE DISPUTE Replace the GPU

Loss explodes around step 800 with no infrastructure event anywhere near it. The customer's team suspects a GPU computing wrong values and asks for a node swap and credits. The counters say otherwise: hardware deltas are zero on every GPU, the loss drifts before it explodes, and a rerun on other nodes diverges at the same step. The recipe is fp8 without per-tensor scaling.

▸ ECC, XID, row-remap and throttle deltas: zero on all GPUs▸ one old row-remap, flat all run: pre-existing, not the cause▸ loss drifts, then explodes; rerun diverges at the same step
VERDICT → THE RECIPE, HARDWARE CLEARED. NODE SWAP AVOIDED.
01THE REAL PRICE

You pay per GPU-hour. You get per productive hour.

Divide the list price by the share of time the GPU actually computes. That is the number that decides your budget, and the one Proof moves.

{{ effLabel }}${{ effTxt }}
{{ proofLabel }}${{ proofTxt }}
{{ wasteLine }}
02NOT ANOTHER DASHBOARD

Dashboards show symptoms. We return verdicts.

You can assemble DCGM, Prometheus and Grafana yourself and still be staring at thirty charts at 3 a.m. The hard problem is correlation: joining GPU counters, kernel traces, collectives, scheduler state and the training log into one answer. The most expensive failures produce no hardware symptom at all.

DIY OPEN-SOURCE STACKMONITORING SAASALQUA PROOF
GPU dashboards and alertsYes, DCGM exporter and GrafanaYes, DCGM-based GPU monitoringYes. That part is table stakes.
Every signal keyed to job, rank, PID and GPUWeeks of label plumbing, usually never landsHost and container tags, no job or rank identityDone at collection time, inside the agent
Training progress: loss, steps, checkpointsNot visibleLogs if you ship them; loss and steps mean nothing to itRead from job stdout via kernel-level capture
Per-rank CUDA and NCCL tracingRequires SDKs and code changesNot offered; profiling stops at the processeBPF uprobes, zero instrumentation
Who correlates the incidentAn engineer with thirty dashboardsAlerts fire; the investigation is still yoursAn agent, with a cited verdict in minutes
Runs that complete but learned nothingInvisible: exit code 0, all greenInvisible: the infrastructure looks healthyA first-class failure family
03HOW IT WORKS

One agent per node. One verdict per incident.

01 · COLLECTGPU counters, kernels, collectives, logsA single binary on each node reads DCGM, NVML, eBPF probes and the job's own output. No SDK, no code changes.
02 · JOINEvery signal keyed to one identityJob, step, rank, process and GPU are attached at collection time, so correlation is a lookup, not a forensic project.
03 · INVESTIGATEAn agent reads the evidenceIt runs on a local model, so telemetry never leaves the site. It checks the job, the node, the fabric and the storage in turn.
04 · VERDICTThe cause, named and citedThe job or the infrastructure, with the GPU, interval and signal attached. Sent to you before you open a ticket.
ONE IDENTITY ACROSS EVERY SIGNAL
DCGM counterCUDA traceNCCL collectiveIB portSlurm recordjob stdout job · step · rank · PID · GPU

Because the join happens at collection time, correlation is a lookup, not a forensic project.

850+GPU metric series per node: 157 DCGM fields, tensor pipes, throttle causes, ECC, XID, row-remap, NVLink errors
3,700+eBPF series per cluster: CUDA API latency, kernel launch gaps, NCCL collectives by type, dtype and message size
46kernel uprobes on libcuda and libnccl, aggregated inside the kernel at 40K launches per second per rank
1static binary per node: DCGM, NVML, eBPF and log capture, about 0.15 CPU cores and 240 MiB on an 8×H100 node
04FOR OPERATORS

Why operators run Proof on their own fleet.

Not another tool to resell. Leverage for the operator itself: support, SLAs, fleet health, and what your GPU-hour is worth against everyone else's.

01It's your hardware, answered in minutesA customer escalates: the cluster is at fault. You see the real cause, pinpointed to the GPU or link, or cleared with counters attached.
02Proactive outreach, not ticketsInvestigations run live while the job burns. You message the customer first: your run is 40% slower than it should be, here is why and the fix.
03Utilization becomes the offerSell capacity with an efficiency layer attached: more productive training per rented hour, measured per job.
04SLA talks start from evidenceEvery dispute opens with per-rank evidence: job, node, GPU, interval and signal. It ends with a verdict instead of a goodwill credit.
05Fleet health ahead of failuresRising ECC rates, row-remaps, thermal throttling and degrading links flagged days before they kill a job. Drain and replace on your schedule.
06Support cost down, resolution upTier-1 hands the customer a cited verdict, not a wall of dashboards. Week-long escalations close in one pass.
05WHERE IT RUNS
ON ALQUA CLOUD Included on every cluster. Proof runs on every ALQUA GPU from day one. Verdicts appear in the console and in your team's chat. SEE ALQUA CLOUD →
ON YOUR FLEET SaaS, on-premise or air-gapped. For neoclouds and data centres. Answer "it's your hardware" in minutes and sell utilization, not just hours. Priced per GPU per month. Talk to Us