Processing-in-Memory for AI: What Must Improve Beyond the Chip?

I see processing-in-memory as an opportunity for Korean memory suppliers to take a larger role in AI execution—but only if the gain survives integration. I would judge it by the complete request, including unsupported operations, transfers and coordination, rather than the fastest operation in isolation.

A PIM result needs two boundaries: which operations run in memory, and where the performance timer starts and stops. A fast operation, an accelerator card and a responsive service are different achievements. Public Korean demonstrations provide a way to examine that distinction without assuming that PIM replaces an entire GPU.

Start with what the memory actually computes

PIM places selected computational capability within memory, reducing the need to move some operands to a separate processor. The supported operations and their scheduling determine what work can move. Simply adding logic somewhere in a memory package does not establish that the package executes an LLM's workload.

Samsung's 2021 HBM-PIM announcement described an AI engine inside memory banks. It is a historical architecture example, not evidence that every later HBM product includes those engines.

SK hynix's 2023 AiMX prototype report describes an accelerator card built around GDDR6-AiM. This is a different implementation and packaging level. Its performance statement also carries a condition about developing the AiM control hub as an ASIC. Dropping that condition would make the reported result sound more broadly established than the source allows.

There is no reason to compare those announcements by placing their largest speedup numbers side by side. Their workloads, implementation assumptions and system boundaries need to match first.

Put the announcements on a scope ladder

Use this ladder when reading a PIM claim. Each row asks for evidence the preceding row cannot supply on its own.

Evidence level	What to ask	What remains unresolved
Operation or memory chip	Which math, data type and data layout are supported?	Coverage of the complete model
Accelerator card	What controller and host transfers are included?	Server integration and contention
Server demonstration	What other processors execute the remaining work?	Behavior across production traffic
User-facing service	Does it meet quality and latency targets under load?	Wider adoption and economics

The 2025 SK hynix demonstration used a server combining GPUs and AiMX cards. That makes cooperation between devices a concrete design to study. It is a more useful starting point than imagining all computation disappearing into a memory chip.

In September 2026, the company again displayed an AiM chip, AiMX card and equipped server with an LLM demonstration. This establishes a public demonstration at multiple hardware levels. It leaves open the particular deployment economics, customer adoption and repeatability that a production assessment would need.

A four-times-faster operation can yield a 1.54-times-faster request

Here is a hypothetical serial execution path, not a vendor benchmark. The original request takes 100 ms: 60 ms in work eligible for PIM and 40 ms elsewhere. Suppose the eligible segment becomes four times faster, taking 15 ms.

Now add 10 ms of new integration overhead, such as exposed coordination or transfer time. The resulting request takes:

15 ms + 40 ms + 10 ms = 65 ms

Its overall speedup is 100 ÷ 65 ≈ 1.54×. The operation's fourfold improvement is real within the example, but it applies to only part of the original time.

AI NodeLab hypothetical serial-time calculation, not a Samsung or SK hynix benchmark. The eligible operation is four times faster; the complete request is about 1.54 times faster.

If the new overhead were 50 ms instead, total time would become 105 ms. The accelerated operation would remain faster while the complete request became slower. For the stated inputs, overhead must stay below 45 ms, the 60 − 15 ms saved, to improve total latency.

The general worksheet is:

New total = eligible time ÷ eligible speedup + remaining time + added overhead

Use non-overlapping time on the request's critical path. If transfers overlap useful computation, count only their exposed delay; adding both full durations would overstate total time. Hold output quality, model behavior and the measurement boundary fixed.

Even eliminating the eligible segment entirely, with no added overhead, leaves 40 ms. That gives a 2.5× ceiling under these particular assumptions. Our memory-bandwidth article explains the related distinction between faster kernels and a faster application; PIM adds the practical question of moving eligible work into another execution location.

The missing evidence is often workload coverage

A small eligible fraction limits the benefit before any implementation problem appears. A workload that spends more time in supported operations offers more room, but only if the transfer and scheduling costs remain controlled.

Ask for a breakdown that identifies the operations sent to PIM, the work retained on the host or GPU, and the time exposed at the boundary. Also ask how the distribution changes with prompt length, output length and concurrent requests. A single favorable operation is an insufficient description of that mix.

Energy deserves its own complete-system boundary. Reduced movement inside one component does not directly state how much energy the entire service consumes. Compare the same useful work and include the devices needed to complete it.

This is why first-token delay and later token delivery should remain separate measurements. A change concentrated in one phase can leave another part of the user experience almost unchanged.

A stronger Korean PIM story would show the integration

The public evidence makes Samsung's HBM-PIM and SK hynix's AiM/AiMX relevant technical efforts to follow. It does not establish one universal PIM architecture or a market winner.

What would strengthen my confidence is a reproducible, full-system comparison: the same task quality and request mix, explicit eligible operations, complete latency and energy boundaries, and a clear description of the controller and software. If the gain survives busy traffic and integration overhead, the memory maker's execution capability has delivered something the user can notice.

If it disappears outside one favorable operation, the development may still be useful—but for a narrower workload. Before treating the next PIM headline as a service forecast, fill in the eligible-time worksheet and place the evidence on the scope ladder. Those two steps reveal which improvement was actually demonstrated and which one still needs proof.