I see local AI as an opportunity for Korean low-power memory suppliers, especially if devices retain longer conversations. But I would judge that opportunity by the useful workload a device can sustain, not the size of a model download. A file labeled 4 GB does not establish that the application needs just 4 GB of working memory—or that all of the device's RAM is available to it.
Start with resident weights, add the retained KV cache and runtime buffers, then account for the operating system, other apps and a safety margin. For Korean LPDDR suppliers, the relevant opportunity is supporting a usable local workload within that shared budget, rather than maximizing one isolated capacity number.
Disk size, resident size and available memory
The downloaded file is a storage artifact. Loading it may require additional metadata, buffers or a representation different from the file's compressed encoding. Inference then creates state whose size depends on the model and the live request.
Apple's Core ML Llama deployment example combines compressed weights with a stateful KV cache. The important lesson is the combination: reducing weights does not remove the state needed during generation. This is a documented implementation example, not a compatibility claim for every device or runtime.
Shared memory also serves more than the model. Apple Metal exposes a recommended maximum working-set size as approximate guidance for allocations without affecting GPU runtime performance. It is not a promise that an application may consume all physical RAM, or a universal hard limit for every allocation.
The title's “4 GB” describes a generic download-size claim. The worked example below uses GiB and an assumed resident weight size. Those units and boundaries are deliberately kept separate.
Build the budget from the device down
Imagine a device with 16 GiB of shared physical memory. For one chosen workload, assume 6 GiB is occupied by the operating system and other apps. Hold 2 GiB aside as reserve. The model's resident weights take 4 GiB and its non-KV runtime buffers take 1 GiB.
| Hypothetical allocation | Memory |
|---|---|
| Operating system and other apps | 6 GiB |
| Resident model weights | 4 GiB |
| Non-KV runtime buffers | 1 GiB |
| Reserve | 2 GiB |
| Remaining KV cache budget | 3 GiB |
| Total | 16 GiB |
These are AI NodeLab assumptions, not specifications or recommended reservations for an Apple or Android device. The reserve is planned headroom, not memory already allocated to an application.
AI NodeLab hypothetical budget. The resident weight size and every allocation are assumptions, not specifications for an Apple, Samsung or SK hynix product.
The useful calculation is 16 − 6 − 4 − 1 − 2 = 3 GiB. A weights-only estimate would have left the reader with a very different impression of the space available for a conversation. If the operating system or runtime permits a smaller application working set, that tighter constraint must also be respected.
One long conversation fits; two may not
For an illustrative full-attention model, assume 32 layers, four KV heads per layer, a head dimension of 128 and two bytes per stored value. Key and value together require:
2 × 32 × 4 × 128 × 2 = 65,536 bytes per retained token
That is 64 KiB per token. The KV cache calculation guide explains the formula and where uniform full-attention assumptions stop applying.
One sequence retaining 32,768 tokens needs 2 GiB of logical KV payload. Two independent sequences of that length need 4 GiB, exceeding the illustrative 3 GiB cache budget. The model weights have not grown; the live workload has.
Dividing 3 GiB by 64 KiB gives 49,152 retained tokens across the assumed independent sequences. Treat this as an arithmetic upper bound before cache allocation padding and other unmodeled costs, not an advertised context setting.
There is no prefix sharing or offloading in this example. A real model may use a different attention layout, precision or sharing policy. Apply the actual model's rules before converting a memory budget into tokens.
Compression changes more than one line
Weight compression reduces the weight allocation when the runtime can use that compressed representation effectively. Cache compression acts on retained state. They need separate entries in the ledger, and both require quality and implementation checks.
Apple's 2025 foundation-model description provides a dated example of designing these components together: different representations for weights, embeddings and cache, plus cache sharing across parts of its on-device model. It should not be used as evidence that an unrelated model can apply the same reduction.
Apple's June 2026 AFM 3 Core Advanced description adds another placement example: the full model remains in flash while selected expert weights enter DRAM. It makes routing decisions per prompt, with periodic reselection during generation, rather than swapping experts for every token. That architecture reinforces the need to count the resident working set separately from stored model size; it does not change the all-resident assumptions of the worksheet here.
In the hypothetical budget, saving another GiB of resident weights would leave more room for KV, provided runtime allocations and the available working set stayed unchanged. But a conversion that creates a larger temporary workspace could consume some of that gain. Inspect peak allocations as well as the steady state.
A smaller download also does not prove faster inference. The hardware and software must execute the representation efficiently. Our memory-bandwidth-versus-compute article offers a framework for examining the operation that actually limits performance.
Where Korea fits without guessing the device supplier
Samsung's LPDDR5X product discussion explicitly includes on-device AI applications. SK hynix's FMS 2026 report presents LPDDR and mobile storage in its edge portfolio. These are direct reasons to connect local AI to Korean low-power memory capabilities.
They do not identify the memory supplier inside a particular Apple device, establish a design win for a named model, or show that each increase in local AI use raises memory-company revenue. Avoid filling those gaps with a plausible-sounding supply-chain story.
My confidence in that opportunity would grow when an application demonstrates a useful task locally with acceptable quality, latency and sustained resource use. It would weaken when acceptable quality requires a much larger model, the working set repeatedly exceeds the device budget, or the service must move most of the task elsewhere.
Test the workload you will actually keep open
Load the intended model and record resident and peak memory with the target runtime. Repeat with the intended conversation lengths and simultaneous sessions, while the other applications expected in normal use remain open. Check a cold start and an extended run, not just whether a short prompt completes once.
Record latency and output quality beside the allocations. Fitting is the first gate; running well is the second. A useful device claim states both.
When the next local model arrives with a reassuringly small download, start with the five-line budget. Ask how much space remains for live state after everything else is counted. That is the number that makes a long conversation feasible—or explains why the file size was never the answer.
Comments
Post a Comment