Harvard MadSys Lab · Storage systems
Model hubs now hold tens of petabytes of weights, and generative platforms add billions of images a year. We study how this data is stored and used, and build systems that keep it in a fraction of the space.
We also look ahead to the media this data will live on: dense QLC flash and ceramic archives. The goal is storage that stays affordable and sustainable as AI keeps producing more.
Model hubs are now among the largest storage systems in the world. Public models on Hugging Face took 0.28 PB in 2023 and more than 76 PB by the end of 2025.
More than 99% of those bytes are fine-tuned variants of a much smaller set of base models. Generic deduplication and compression treat model files as opaque bytes and miss most of this redundancy. We studied every public LLM repository on Hugging Face and found structure that a storage system can use:
Efficient LLM storage via tensor deduplication and delta compression
ZipLLM groups models into families by bit-level similarity, deduplicates identical tensors, and compresses the rest with BitX, a fast lossless codec for the XOR between a fine-tuned tensor and its base.
A compact, tensor-centric storage system for modern AI models
Model cards are unreliable: only about a quarter of Hugging Face repositories name their model family. TensorDex pairs every tensor with its best base by content, using small sketches to estimate similarity without reading whole tensors, and keeps clusters up to date as new models arrive.
AI image generation reached 15 billion images in about a year and a half; photography took 149 years to get there. Adobe Firefly alone passed 24 billion images by mid-2025, and Midjourney makes more than 34 million a day.
Stored as 1024×1024 PNGs, 100 billion images take about 150 PB, and the pile grows every day. Today’s photo stores keep each one as an opaque blob.
An AI image is different from a photo because it can be recreated exactly. Diffusion spends seconds producing a small latent tensor, and that work happens once. Turning the latent back into pixels is a deterministic decode that takes tens of milliseconds on a GPU. A storage system can therefore keep the latent and rebuild the image when someone asks for it.
The same trade-off applies to music, video and text: when is it cheaper to keep what a model made, and when is it cheaper to make it again?
A latent-first storage system for AI-generated images
We designed LatentStore from the first large-scale study of how people access AI images: a 35-month, 2-billion-request trace from a production generative-content platform. Popular images stay cached as pixels for fast hits. Colder images are kept as compact latents, and the split between the two caches adapts as popularity shifts.
A cost model for AI-generated content at scale
Our model covers corpus growth, HDD and tape prices, drive replacement, electricity, request skew, caching, and falling GPU cost. Regenerating images from the prompt stays more expensive than storing them until around 2040. Keeping a compact intermediate representation and decoding it on a miss is cheaper than both today.
The media under storage systems are changing. Dense QLC flash is starting to take over read-heavy capacity from hard disks, and ceramic archives can keep data without power for far longer than tape or disk. Both break assumptions that today’s storage software was built on.
QLC stores four bits in every cell. That density puts more than 100 TB in one SSD at a lower cost per terabyte than TLC flash, but write endurance and write speed pay for it. Most AI data is written once and read many times: model weights, checkpoints, generated media. That makes it a good match for QLC, if the storage system above it shapes writes carefully.
Builds on our flash caching work: Kangaroo (SOSP ’21, best paper) caches billions of tiny objects on flash within a strict write budget.
Ceramic storage writes bits with a laser into a ceramic coating a few nanometers thick on a glass carrier. The medium is write-once, needs no power or climate control to keep data, and is designed to last far longer than tape or hard disks. That makes it a candidate for the data we never want to lose: base models, training corpora, and scientific records.
An archive that never needs migrating changes how storage systems should be built. Today’s archives copy data to new media every few years. Ceramic archives would skip those copies, but they rule out updates and deletion in place.
Caching is our longest-running line of work, and it has its own home. Eviction algorithms such as SIEVE and S3-FIFO, the libCacheSim simulator, and our open collection of production cache traces all live on cachemon.org.
Our storage course is COMPSCI 2640: Modern Storage Systems, first taught in Spring 2026.
We are looking for students who want to work on storage systems. Read the notes for prospective students on the lab homepage before you get in touch.