Your Temporal GNN Is Wasting
98% of Your GPU
and the Research Community
Doesn't Talk About It

There's a number buried in a 2022 profiling paper on dynamic graph neural network inference that deserves more attention than it gets. TGAT — one of the canonical temporal GNN architectures, the model cited in hundreds of follow-up papers — achieves GPU utilization of 5 to 6 percent during inference. JODIE achieves 1.5 to 2.5 percent. DyRep achieves under 2 percent.

To be precise about what this means: for every 100 units of compute your GPU is capable of delivering, these models are using between 1.5 and 6. The rest sits idle. You could run this workload on hardware that costs 5% as much and get identical throughput.

The temporal GNN literature mostly doesn't discuss this. The papers report Average Precision, AUC, and occasionally training time. GPU utilization appears in one systems paper. The implication — that temporal GNN computation is structurally incompatible with how GPUs work — is rarely stated directly.

Here's why it happens, what it means for anyone trying to deploy these models, and what a March 2026 paper does about it.

// Why GPUs fail at this

GPUs are optimised for one thing: executing the same operation on many data elements simultaneously. The computational primitive they're designed around is the matrix multiplication — large, regular, dense. Thousands of cores fire in lockstep on contiguous memory, the data flows in predictable patterns, and hardware prefetchers do their job.

Temporal GNN computation is the opposite of this in almost every respect.

When a new edge arrives in a temporal graph, the model has to update the memory states of the affected nodes, then recompute their embeddings by aggregating over their temporal neighborhoods. The neighborhood of node v before time t is the set of all historical interactions involving v up to t. This is irregular — some nodes have thousands of historical neighbors, others have two. The access pattern into memory is scattered and non-contiguous. And critically, the computation is sequential: the memory state after edge i depends on the state after edge i-1, which means batch i+1 cannot start until batch i finishes its memory update. You can't pipeline this without breaking temporal causality.

The result is that GPU cores sit idle waiting for memory operations to complete, waiting for the previous batch's sequential dependency to resolve, waiting for scattered reads across a graph structure that doesn't fit in L2 cache. The memory bandwidth utilization during TGAT inference, according to the profiling data: 10–30% of peak. The GPU occupancy during the memory update stage: 15–25%. Everything that GPUs are good at is the wrong shape for what temporal GNNs need.

// The inference problem nobody was solving

Here's a fact that surprised me when I read it: every temporal GNN training system published before March 2026 — TGL, ETC, SIMPLE, SWIFT — optimised exclusively for training. None of them modified the inference pipeline. At serving time, they all executed the same full-recomputation procedure as the baseline: every new edge triggers O(|V|) embedding updates, where |V| is the total number of nodes in the graph.

On a million-node graph, one new edge causes the system to recompute embeddings for all one million nodes, even though only the nodes within a few hops of the new edge are mathematically affected. The rest don't change. The system recomputes them anyway.

The key observation is simple once you see it: a new edge in an L-layer TGN affects only nodes within L hops of the two endpoints. On sparse real-world graphs, this is a tiny fraction of the total node population — under 0.2% on Stack-Overflow (2.6 million nodes). StreamTGN maintains persistent GPU-resident node memory, tracks which nodes are "dirty" after each batch using lightweight flag propagation, and only recomputes embeddings for those nodes.

// The numbers

On eight real-world temporal graphs spanning four orders of magnitude in scale, StreamTGN achieves 4.5×–739× speedup for TGN and up to 4,207× for TGAT. On Stack-Overflow (2.6 million nodes, 48 million edges), inference time for a batch of 600 edges drops from 33,984ms to 46ms. The key metric is the global index refresh — the cost of maintaining up-to-date embeddings for the full graph after each batch. TGL: 31,625ms. StreamTGN: 42.79ms. Same accuracy, every time.

The 739× figure is the one that gets quoted, and it is real — but it's worth understanding what drives it. Speedup is inversely proportional to the affected ratio: Speedup ≈ |V| / |A|. On Stack-Overflow, only 0.13% of nodes are affected per batch. On WIKI, it's 17.4% — and the speedup drops to 5.7×. The relationship is exact and predictable. Large, sparse graphs with localized edge arrivals benefit most. Small, dense graphs benefit less. This matters for choosing when to use this architecture.

Two additional findings from the paper's sensitivity analysis are worth noting. First, speedup degrades as batch size increases — larger batches dirty more nodes, shrinking the advantage. Second, rebuild interval barely matters: even with no periodic full rebuild, speedup and accuracy remain nearly identical. The dirty set doesn't accumulate the way you'd expect on real temporal graphs, which means the incremental approach is more stable than it should theoretically be.

// The 10,000 GPU hours paper

A separate paper from December 2024 (arXiv:2412.20256) spent 10,000 GPU hours systematically searching the temporal GNN design space — module choices, sampling strategies, aggregators — and derived empirical design principles that the standard benchmark-centric literature hadn't surfaced.

The most practically significant finding: most-recent neighbor sampling consistently outperforms uniform sampling. This matters because most-recent sampling is more compatible with the incremental approach — it focuses computation on recently active nodes, which are precisely the ones most likely to be in the dirty set. The design space search and the systems optimisation are pointing in the same direction.

The paper also found that static node memory is an effective alternative to dynamic memory in many settings. This is relevant to the utilization problem: static memory eliminates the GRU update step — the sequential batch dependency that contributes to 15–25% GPU occupancy in TGN — at the cost of some representational capacity. On datasets where the dynamic memory isn't providing much benefit (which is more common than the TGN paper implies), this is a straightforward trade.

// What this changes

The mainstream temporal GNN narrative is: design a better architecture, benchmark on standard datasets, report improved Average Precision. The systems reality is: if you deploy any of these architectures at production scale, you're running at 2–6% GPU utilisation and recomputing the entire graph on every edge arrival.

StreamTGN fixes the second problem without changing the first. It's architecture-agnostic — it wraps TGN, TGAT, and DySAT and achieves identical accuracy on all of them. It's orthogonal to training optimisations — you can stack SWIFT for training with StreamTGN for inference. And its speedup guarantee has a closed-form expression: if you know your graph size and expected edge locality, you can predict the speedup before you deploy.

What it doesn't fix: the fundamental mismatch between irregular graph computation and GPU architecture. The GRU memory update is still the dominant cost in StreamTGN's own pipeline (53–67% of runtime). Coalesced memory access helps feature retrieval but doesn't solve the underlying problem of sparse, irregular neighborhoods. For the specific class of large, sparse temporal graphs with localized edge arrivals, StreamTGN closes most of the gap. For dense graphs where many nodes are affected by each batch, the problem remains open.

The honest framing is that temporal GNNs are computationally expensive in a way that doesn't match the hardware most ML practitioners have access to, and the research community has spent most of its effort on the part of the problem that's already relatively well-served — training throughput — while leaving the part that matters for deployment — inference latency — almost entirely unaddressed until this year.

That's the paper I'm glad exists. That's also the paper that should have existed in 2022.