Recent Paper Reading: ACOS, LESS, Bidaw, FlashAttention
09 November 2026
1. The Past, the Present and the Future of Local Storage in Cloud [2026, 0 refs, FAST26 best paper, Alibaba]
https://www.usenix.org/conference/fast26/presentation/yang
https://yiyibooks.cn/boywithoutname2/alibaba-local-storage-fast26-yang/index.html
https://chatgpt.com/c/69d2fd63-3720-83a1-b9ff-66bfec6a6689
1. Good. To learn how Alibaba "Local Storage" leverages DPU (ARM + ASIC).
At the moment, LATTE is still a proof of-concept (PoC)
2. My readings out
1. What is the "Local Storage" vs other Alibaba products like OSS, EBS and PolarDB?
1. Host attached "Local Storage", Latte is integrating to EBS
2. The present: Ristretto: From ASIC to ASIC/SoC Co-design
1. How it solves the problem in Doppio ASIC?
1. Insert ARM SoC to implement complex management logic instead of ASIC
Leave fixed logic still to ASIC
2. Why it needs three layers? ASCI virtualization backend, ARM SoC, ASIC-based SSD Driver
1. First, note VM uses its OS NVMe Driver to access the whole Ristretto stack
In Doppio ASIC, it does NVMe emulation -> namespace management -> Host NVMe driver
In Ristretto, the namespace management part is replaced to ARM SoC
LVM is implemented in the ARM SoC. Note there is no Linux kernel at all now
3. Why not replace all three layers into ARM
1. ARM is still CPU, keeping layers with fixed logic at ASIC is still CapEx optimal
3. The future: Latte: Local-Cloud Combined Storage
1. What is it?
1. To find the applicable user scenario with EBS
2. Caching implementation
1. Maintaining Write-ordering
1. Both write-to-cache and flush-to-backend paths operate in append only mode
2. ML-based I/O dispatching model
1. We collect per-I/O traces of cache and backend latency, I/O size, and queue depth (QD)
2. Using a slide window (by default 5 IOs) as input,
3. the model outputs binary classification (i.e., forwarding I/O to cache or backend)
4. Model selection: We choose linear-SVM
5. Model retrain
1. we collect the throughput and average latency of LATTE every 60 seconds for statistic analysis.
2. If the variance is above a pre-defined threshold (10% by default), we retrain the model
3. by short I/O traces and update the weights accordingly.
4. Given the small scale of traces dataset and the use of a lightweight model, the process only takes 5s on average
3. Admission and eviction
1. Use S3 FIFO [76] structure for better caching instead of classic LRU
3. Why attach NVMe to EBS, given EBS may already be running on NVEe flash?
1. NVMe on local storage, while using cheaper EBS
4. Why local storage has to be a cache? anything else than cache?
1. Caching is the least intrusive way to work with EBS.
Reasonable to be future first step
2. Other alternatives are (not in paper)
1. Local SSD = primary, EBS = replica (sync / async)
2. Local SSD = primary, EBS = backup
3. Local SSD = hot tier, EBS = Hot tier = cold tier
3. Local SSD = write buffer / log, EBS = coalescing destination
4. Anything done for AI workload?
1. Latte, reduce cost of AI workload and still the scale via EBS
3. Highlights
1. Timeline
1. ESPRESSO - 2017
1. What
1. SPDK userspace IO
2. 3.2GBps per SSD, 480K IOPS per SSD
2. Problems
1. No baremetal support, need to dedicate CPU cores for IO
2. Busy polling wastes 40% CPU utilization
3. Overhead from context switches between VMs and hypervisor
2. DOPPIO - 2019
1. What
1. Use ASIC DPU, which alleviates CPU inefficiency
2. uses fast hardware interrupts instead of software ones
3. 3.2GBps per SSD, 500K IOPS per SSD
2. Problems
1. ASIC-based DPUs have limited processing power, and thus are not able to exploit the latest NVMe SSD with 1.5M IOPS
2. the fixed hardware logic of ASIC cannot well support the emerging cloud features (e.g., Logical Volume Management).
3. While using high-performance FPGA may alleviate some of the issues, it would incur high energy cost and Capital Expenditure (CapEx).
3. Ristretto - 2023
1. What
1. 900K IOPS, 6GBps throughput for single VD / SSD
2. Build LVM etc namespace management logic into ARM instead of ASIC
3. Now Linux kernel is replaced into ASIC->ARM->ASIC
2. Problems
1. Local storage limitation
1. Unable to scale
2. certain AZs, limited to region
3. subject to data loss
4. Latte - Future
1. What
1. Integrate Local Storage with cheaper EBS
2. machine learning based I/O dispatcher and cache admission control
n. Related materials
1. CacheSack: Admission Optimization for Google Datacenter Flash Caches
https://www.usenix.org/system/files/atc22-yang-tzu-wei.pdf
2. Google L4 Cache at Colossus, see algorithm at CacheSack
https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance
3. What’s the Story in EBS Glory: Evolutions and Lessons in Building Cloud Block Store
https://www.usenix.org/system/files/fast24-zhang-weidong.pdf
4. [76] S3 FIFO: FIFO queues are all you need for cache eviction [2023, 110 refs, SOSP23]
https://github.com/Thesys-lab/sosp23-s3fifo
https://blog.jasony.me/system/cache/2023/08/01/s3fifo
https://dl.acm.org/doi/pdf/10.1145/3600006.3613147
1. Good queuing algorithm. Simple, and out performs LRU on cloud storage
2. Highlights
1. Key idea
1. Most data is useless (one-hit wonders), evict quickly unless proven useful
2. Key design
1. Small queue (S, filter, ~10% cache) -> Main queue (M, real queue, ~90% cache) -> Ghost queue (G, metadata only)
2. Write to S, admit to M on second hit. Otherwise evict to G.
3. Compare with Google CacheSack
1. S3 FIFO is similar with AdmitOnSecondMiss
2. ACOS: Apple’s Geo-Distributed Object Store at Exabyte Scale [2026, 0 refs, FAST26]
https://www.usenix.org/conference/fast26/presentation/baron
https://chatgpt.com/c/69d362f0-0cc8-839c-9bff-34a5affac54d
1. Very good. ACOS 2.0
Intra-stamp
1. 3 out of 5 quorum write,
2. cross-data EC
3. LRC 20+2+2
inter-stamp
1. inline cross-region EC, one object is chunked into 4 regions.
2. Cross-region strong consistency metadata store backed by RocksDB and Raft
2. My readings out
1. How is the object store architectured and different from other systems?
1. See Figure 3, the cross-data EC without n-replica=>EC rewrite is worth noting.
2. ACOS 1.0 is using Dual Cassandra for metadata
3. ACOS 2.0 is very different, very good. See my notes in highlights
2. How does ACOS support being Geo-distributed?
1. ACOS 1.0 - Geo-paired replication, replication and failover to the second stamp
2. my questions
1. replication is async, but no mentioning how async lag is handled
3. What is the production deployment status of ACOS?
1. Prod at Apple > 10 years.
3. What is special in Apple's scenarios and Apple's technology stack
1. Apple mobile system, Apple iCloud, Apple TV, Apple Map, Apple Music
4. Any relation with Apply FoundationDB
1. Not directly mentioned. FoundationDB probably handles transaction, metadata; while ACOS handles object storage, both service Apple services
5. Erasure coding in ACOS
1. ACOS v1: LRC (12,2,2), see Figure 3
1. local region: LRC (20,2,2)
2. cross region: XOR coding
3. Highlights
1. ACOS 1.0
1. A store is two geo-paired stamps (deployment cluster).
load balancer evenly distribute traffic to each
2. Storage runs JBOD HDD
3. An object is sharded into 64MB parts, stored in containers
4. A container is a large file from 4 to 32 GB. The "cluster"
1. Very good here, see Figure 3. The EC is very different from Azure Storage
1. Data is first written as replicas then ECed after seal, but
2. In n-replica stage, each data has 5 replicas.
Replication is via ZAB protocol. Majority ack means complete.
3. In n-replica stage, 12 data containers (files) are pre-grouped for EC
The group is called "cluster".
After the cluster is filled up (not mentioning it is sequentially filling or round-robin),
it is sealed, then the group can be directly converted into an EC group, where each original data container now is an EC symbol in LRC(12,2,2)
So that, no data rewrite is needed like in Azure Storage
2. A container is a large file ranging from 4 to 32 GiB
3. My questions
1. Why use 5-replica inside a stamp in the replicated cluster?
1. ZAB uses 5-replicas, quorum size is probably 3. The container cluster is sealed when filled up then immediately converted to EC format. So 5-replica won't incur significant storage cost, because LRC is the final data format.
5. sealing process: when container is full, convert replicas to EC
6. Replicated containers use a custom distributed transaction log based on ZAB [34] for data flow but uses Raft [30] algorithm for leader election,
1. with optimizations specific to object storage services:
(i) minimize flushes for multi-chunk objects and
(ii) track aborted partially written objects.
2. ACOS 2.0
1. Strong consistency metadata service across regions - ClassVI
1. Cross-region replication, backed by RocksDB, Raft consensus. Similar to BigTable. Not multi-row transaction.
2. Write is strong consistency. Single replica read is eventual consistency
2. Write process
1. The object length must be known upfront so as to do chunking
2. An object is chunked into 4 segments. Write to 4 regions. Generate 1 XOR parity on the fly. Write completes when 4 out of 5 done.
3. Intra-stamp level, it still uses LRC(20,2,2), so overall storage overhead is 1.2*1.25=1.5
4. Intra-stamp level, it still uses container files. An object segment needs to store in global metadata about its cluster, container index, and offset within container
5. My questions
1. So its inline cross-region EC, in the schema to shard a single object to regions
3. regional load balance and migration
1. a unified endpoint DNS that load balances to different regions
2. intra-region data migration rebalance
3. deployment-level rate limiter and stamp-level rate limiter
4. Problems
1. ACOS 2.0 is worse at first byte latency (TTFB) compared to 1.0, because the cross-region chunking
2. Latency optimization
1. Select the closest datacenter
2. Fast metadata propagation and non-quorum fast read
3. inter-stamp traffic bypasses load balancer
n. Related materials
1. Apple FoundationDB: A Distributed Unbundled Transactional Key Value Store
https://www.foundationdb.org/files/fdb-paper.pdf
https://github.com/apple/foundationdb
3. LESS is More for I/O-Efficient Repairs in Erasure-Coded Storage [2026, FAST26, 1 refs]
https://www.usenix.org/conference/fast26/presentation/cheng
https://github.com/adslabcuhk/less
ChatGPT: https://chatgpt.com/c/6a33a575-4ac8-83ec-a52c-a2b9e4b80bf2
1. Very good paper. LESS maintains the same storage overhead, recoverability, etc properties with RS code, but able to greatly reduce the disk IO (also network IO) size required in node failure repair.
The research track is good. We need new codecs optimized for disk IO rather than network IO (regenerating, MSR codes). This is because network speed has been growing way too fast, but HDD is larger but disk throughput isn't changing much.
1. So, we can see soon we will have brand new future research spaces for erasure coding
1. Optimize disk IO while assuming network bandwidth is large enough
2. Low sub-packetization that is less aggressive than MSR Clay Code
3. Since MSR / Clay Code is already a solved problem. It should be possible to find a middle ground where IO saving is still valid but it has less aggressive sub-packetization.
2. My readings out
1. How does it work?
1. RS code -> sub-striping -> extend to several longer RS stripes
2. What are the saving
1. (n,k,alpha)=(6,4,2) is extended to 3 * (8,6) sub-stripes
Single block repair costs 25% less disk IO (and network IO) than RS code
2. What are the penalty
1. the failure recovery, compared to the original RS code, is impaired.
the after-extend (8,6) codes actually has 3 stripes. how to place it in nodes? Does it still tolerate 2 node failures? (but the node count will be bigger anyway.)
2. Although disk IO size is reduced to repair failure in LESS code, the disk seek count is slightly increased compared to RS code
3. Coefficient construction, must be careful, more than raw Vandermonde matrix, to achieve MDS like RS.
4. Others
1. sub-packetization, complexity
2. Encoding throughput - RS reaches 2.8 GiB/s, while LESS-4 reaches 1.6 GiB/s
3. What are NOT the penalty
1. In LESS construction illustration - https://youtu.be/i6ppVyIOG9U?si=7L6xvHkMXFWWF8cU&t=462
The physical storage layout is B1~B6, the same with RS code (except sub-packetization)
X1~X3 are encoding/decoding view, not mapping to storage. X1~X2 defines how parity sub-packets are encoded. X3 automatically satisfies.
Here shows how X3 automatically satisfies: https://youtu.be/i6ppVyIOG9U?si=nJiHOhQp_ajH8JrC&t=568
1. This is very good trick
2. LESS code doesn't impair failure recovery ability compared to RS code
LESS code is MDS. See https://youtu.be/i6ppVyIOG9U?si=qKsXq87RMK52crzc&t=734
4. Codec parameter limitations
1. k < n
2 ≤ α ≤ n-k # at least 2 parities
2. MDS requires GF field size larger than 2^w >= nα + (n−k −1)C(n−1, k)
3. Highlights
1. Good representation using parity check matrix to decode lost blocks
https://youtu.be/i6ppVyIOG9U?si=3DkDC5izZHeLnVbI&t=437
2. LESS construction illustration
https://youtu.be/i6ppVyIOG9U?si=7L6xvHkMXFWWF8cU&t=462
3. The magic of disk IO saving
https://youtu.be/i6ppVyIOG9U?si=OFWniqUmtkvezzqB&t=638
4. Clay Code's problem
1. Sub-packetization level is α = (n-k)^ceil(n/(n-k)) , i.e. exponential 256 to (14,10)
2. Clay Code is Disk optimal, amount read from helper disk = amount sent by that helper over the network
But, due to sub-packets are small, and get amplified by disk sector size, the real physical disk bytes read is probably no saving compared to RS
5. LESS code compares to RS, Clay Code, Hitchhiker, HashTag, ET
https://youtu.be/i6ppVyIOG9U?si=Y2suHLEmKcdLpX7N&t=779
6. Rule of partitioning groups and sub-stripes in LESS code
B_i = block i, for 1 ≤ i ≤ n
b_i,j = sub-block j inside block B_i, for 1 ≤ j ≤ α
X_z = extended sub-stripe z, for 1 ≤ z ≤ α+1
G_z = block group z, for 1 ≤ z ≤ α+1
The paper says LESS first partitions the n blocks into α+1 block groups whose sizes differ by at most one. Then it organizes the nα sub-blocks into α+1 extended sub-stripes. For the first α extended sub-stripes, X_z contains all sub-blocks in block group G_z plus all sub-blocks in row/sub-stripe z. The last one, X_{α+1}, contains all sub-blocks in G_{α+1} plus the diagonal sub-blocks b_i,j where the block’s group index g_i equals the sub-block row j.
7. The shape of parity check matrix: Figure 6: Parity-check matrix of an (n, k,α) LESS stripe.
1. Caveat: consecutive rows, the row block, are the same sub-block index from different parities.
They are not rows from the same parity.
8. X_{α+1} is can be automatically derived from X1~Xα, then why introduce it?
1. X_{α+1} is introduced so every block group has one repair sub-stripe that contains all sub-packets of blocks in that group.
2. In another word, X_{α+1} is introduced to tell how to repair a lost parity node with saved disk IO.
4. DRBoost: Boosting Degraded Read Performance in MSR-Coded Storage Clusters [2026, FAST26, 1 refs]
https://www.usenix.org/conference/fast26/presentation/niu
ChatGPT: https://chatgpt.com/c/6a369ea7-0e44-83ec-80a7-09225295a9af
1. Problem: rec-read for a small piece of data in Clay Code suffers from the non-contiguous disk IO layout. The Clay Code usually needs a large EC chunk size and you need to return the whole chunk for the small rec-read.
Optimize for the disk layout problem for Clay Code. Core technique - introducing "basic layout unit"
2. My readings out
1. What is the exact meaning of chunk, sub-chunk, stripe, sub-stripe in this paper?
1. A node hosts a chunk, the sub-packets are sub-chunks. The full EC group is a stripe. In Clay Code, a small set of sub-chunks, across the nodes, are self-recoverable.
2. What is the optimization techniques exactly?
1. Partial chunk reconstruction
1. Instead rec-read the sub-chunk, you only need to reconstruct the sub-stripe
1.5. Exploit "sub-stripe reuse"
1. Choose a sub-stripe that includes more of the client request data
1.6. Exploit "request reuse"
1. Cache the sub-chunks read and reuse them of the next request
2. Use a reconstruction-friendly coding layout
1. Introducing "basic layout unit": all basic layout units can cover the entire stripe data without duplication or omission.
"Furthermore, objects aligned with basic layout units naturally conform to sub-stripe reuse, enabling multiple lost sub-chunks to be reconstructed together."
2.5. Fragmentation-Free Storage Layout
1. "First, sub-chunks within each data node are rearranged so that those belonging to the same basic layout unit are placed contiguously, allowing the basic layout unit to be viewed as a whole.
Second, the basic layout units within each data node are adjusted to preserve their sequential order of allocation"
5. NCBlob: Revisiting Network Coding for Warm Blob Storage [2025, FAST25, 8 refs]
https://www.usenix.org/conference/fast25/presentation/gan
https://chatgpt.com/c/6a54c102-1fcc-83ec-83bc-1f36eec7da02
1. A trial of non-systematic MSR codes instead of Clay Code
2. My readings out
1. How does it work exactly?
1. F-MSR: non-systematic, to read any data, you need to read two parities
2. To mitigate this extra read issue, introduce locality-based encoding
https://youtu.be/TDnlprDSyoE?si=azNEajTXO6Bxi432&t=432
1. Let the two reads read out two blobs.
3. To mitigate the problem that F-MSR doesn't work well with large blob
1. Fallback to Clay Code for large blob
4. Limitation
1. State-of-the-art non-systematic MSR codes only support n-k=2
With rotation-based sub-block selection, support n-k=3
6. TapeOBS: Cost-efficient Archive Cloud Storage with Tape: Design and Deployment [2026, FAST26, 1 refs, Huawei]
https://www.usenix.org/conference/fast26/presentation/wang
ChatGPT: https://chatgpt.com/c/6a54ccf9-abdc-83ec-8f2e-bfd11512c591
1. Looks like using tape as archival storage is becoming common. Used in Huawei Cloud.
2. My readings out
1. What exactly makes this system different?
1. Append-only units called PLog to store data
2. Bulk scheduling of object restore/write
1. Use HDD pool as buffering and write staging
3. GC
1. Move objects with a similar lifetime to the same tape
4. Dedicated Dries
1. Write drives, read drives, internal drives
5. Batch erasure coding
1. Batch multiple objects into one single PLog append to do EC, instead of doing per object EC.
7. ServiceLab: Preventing Tiny Performance Regressions at Hyperscale through Pre-Production Testing [2024, OSDI24, Facebook]
https://www.usenix.org/conference/osdi24/presentation/chow
1. Highlights
1. ServiceLab solutions
1. Problem: Insufficient identical machines for hyperscale testing
1. Use heterogeneous machines and pick "nearly identical machines"
2. Analyze test records to find out which machine key factors of variance
2. Problem: Detect tiny regressions
1. Apply adaptive statistics methods
3. Problem: Cover a very diverse set of services
1. Record & replay RPC traffic to generate testing workloads
2. Allow direct call to dependent production services
8. A Survey of the Past, Present, and Future of Erasure Coding for Storage Systems [2025, 55 refs]
https://keyuncheng.github.io/files/publications/tos24ecsurvey.pdf
1. Highlights
1. Table 1. Erasure coding deployment in production
1. Don't trust prod vendors would really reveal their SKU.
Included systems: Google Colossus, Faebook F4, Tencent ultra-cold.
2. New constructions of erasure coding
1. LRC, regenerating code
2. Sector-Disk Codes
3. Piggyback codes
4. Acceleration of Encoding/Decoding Operations
1. Interesting and useful reference here
5. Erasure coding for emerging architectures
1. Erasure Coding for Flash Storage
2. Erasure Coding for In-Memory Storage
n. Related materials
1. Tencent Ultra-Cold Storage System Optimization with Intel ISA-L - A Case Study
https://www.intel.cn/content/dam/develop/external/us/en/documents/case-study-of-tencent-ultra-cold-storage-system-optimization-720747.pdf
9. MRC: Resilient AI Supercomputer Networking using MRC and SRv6 [2026, OpenAI, 2 refs]
https://cdn.openai.com/pdf/resilient-ai-supercomputer-networking-using-mrc-and-srv6.pdf
ChatGPT: https://chatgpt.com/c/6a72df23-df94-83ec-8205-3d205a01ad29
1. Good work after PFC->DCQCN and for AI age. MRC sprays at transport layer, low-latency failover without involving routing layer.
1. Core idea: multi-plane physical topology
+ MRC multi-path RDMA transport
+ static SRv6 source routing
2. Highlights
1. Multi-plane physical topology
1. multi-plane design needs around two-thirds as many optics and three-fifths as many switches as the compared three-tier 800 Gb/s design
1. conventional 3-tier clogs -> 64K NICs, cannot connect 100K GPUs
2. proposed 2-tier bisect 8-plane networking, see Figure 1(b), has 131,072 NICs.
2. a link failure impacts less networking capacity
2. What MRC changes in RoCE
1. MRC means Multipath RC.
1. Conventional QP: QP → one ECMP-selected path
2. MRC QP: QP → EV set → hundreds of explicit paths
2. Packet spraying means packets can arrive out of order.
1. Ordinary RC packet processing relies significantly on sequence order
2. MRC instead includes remote virtual address in each packet, so receiving NIC can placement each packet directly into its final memory location
3. Selective retransmission
1. the network is intentionally lossy, MRC cannot retransmit everything after one gap
2. receiver sends selective acknowledgments describing exactly which packets arrived. Only missing packets are retransmitted
4. Packet trimming
1. For congestion loss, supported switches can strip the payload from a packet sends only header to receiver. The receiver then responses NACK to let send retransmit immediately
2. This provides faster loss notification than waiting for a timeout or inferring the gap from later traffic.
3. MRC disables PFC and runs Ethernet as a best-effort, lossy fabric
1. The motivation is that a bounded, selective retransmission is often less damaging to synchronous AI traffic than PFC spreading congestion to unrelated flows
3. Entropy values are path objects, not merely hashes
1. Each EV carries path info therefore maps algorithmically to one explicit physical path
2. every EV is an individually measurable and disableable path
3. ECN is repurposed as a path-selection signal
1. Packet on EV-17 receives ECN mark
↓
Receiver echoes EV-17 congestion state
↓
Sender temporarily reduces or stops use of EV-17
↓
Traffic moves to less-congested EVs
4. Why static SRv6 is used
1. Static switches are not resilient by themselves. MRC endpoints make the complete system resilient.
2. static routing eliminates reconvergence concerns when a T1 fails
1. i.e. "static" gives every path a stable identity
4. Important limitations
1. Limited RDMA operation set: RDMA write-only, missing remote read
2. Hardware requirements: NIC support for MRC semantics, switch to perform SRv6 uSID forwarding and to support packet trimming
3. The paper primarily targets the back-end training network. Not Remote KV-cache transfer
5. PFC vs DCQCN MPC selective retransmission
1. PFC aims to prevent packet loss, it sends hop-by-hop pause frame, the control granularity is one priority on one physical link
1. Why PFC can spread congestion? PFC identifies the congested priority and link, not the flow that caused congestion.
2. DCQCN aims to prevent persistent congestion, it sends ECN mark->CNP, it's control granularity is one RDMA flow/QP
1. Why DCQCN still struggles with abrupt incast? The feedback has a delay.
For a 7-to-1 incast, the excess arrival rate is: 2,800−400=2,400 Gb/s
So the queue can grow extremely quickly before the rate reductions take effect
The original DCQCN design therefore retained PFC as a necessary safety net.
2. The MRC paper’s 15-to-1 experiment illustrates this trade-off:
3. MRC selective retransmission aims to recover efficiently after loss, it sends SACK/NACK, its control granularity is one missing packet
1. How MRC approaches the problem differently? - very good
1. Do not build a network that must prevent every loss. Build a transport that can cheaply repair individual losses.
2. PFC is disabled. Ethernet runs in best-effort mode.
10. AITURBO: Fast Cloud Storage for AI Jobs via Grouped I/O API with Transparent Read/Write Optimizations [2026, 1 refs, Huawei]
https://www.usenix.org/conference/fast26/presentation/hao
chatgpt: https://chatgpt.com/c/6addd77c-8fc8-83ec-874d-e728e20367d0
1. introduce group IO concept and a planning later to improve AI job IO perf to backing cloud storage.
2. My reading outs
1. What is the target scope of this paper?
1. Cloud storage in Huawei cloud AI training jobs, mostly on checkpoint writing (3.9x+ improvements)
IO perf optimization without costly hardware and large application modification.
2. What is interesting in background and introduction?
1. Automatic checkpoint deduplication
1. Bytecheckpoint: A unified checkpointing system for large foundation model development [2025, 90 refs]
https://www.usenix.org/conference/nsdi25/presentation/wan-borui
3. What is the key technique of AITURBO?
1. exposes grouped I/O APIs, e.g. group_getfile.
A grouped IO layer is introduced in middle of AI job and backend storage.
AI job issues grouped IO + metadata, the middle layer do global IO planing, deduplicate, load balancing.
IO and results are tracked in staging CPU DRAM, while application can go forward.
2. See "Figure 10", checkpoint write has duplication, it can be deduped.
After dedup, you need to rebalance the writes (load balancing)
3. 但训练作业有一个优势：checkpoint 的结构在多轮迭代里很稳定。AITURBO 会缓存第一次生成的写计划，后续直接复用。
https://zhuanlan.zhihu.com/p/2057929324775454027
n. Related materials
1. AITURBO：把 AI 训练和推理的存储 I/O 优化下沉到云存储层 - 废柴
https://zhuanlan.zhihu.com/p/2057929324775454027
2. FAST’26 论文导读 | 分组 I/O API 与透明读写优化 AI 作业快速云存储 - 团子云技术
https://zhuanlan.zhihu.com/p/2018120513138999413
3. Bytecheckpoint: A unified checkpointing system for large foundation model development [2025, 90 refs, ByteDance]
https://www.usenix.org/conference/nsdi25/presentation/wan-borui
1. Mentioned in AITURBO about checkpoint deduplication and write plan.
Features in Load-time resharding - agnostic to parallelism configuration.
2. Highlights
1. Load-time resharding - load checkpoint into a new parallelism configuration
2. generic checkpoint saving/loading workflow - accommodate multiple training frameworks, support different storage backends
3. full-stack optimizations to ensure high I/O efficiency and scalability
1. Tree-based hierarchical communication
4. a suite of monitoring tools to streamline large-scale performance analysis and bottleneck detection
5. Checkpointing at Scale
1. HDFS, fully rewritten with C++, and use NNProxy for federated NameNode deployment
2. Move cool checkpoint from SSD to HDD.
6. Checkpoint deduplication and write plan gneeration
1. Why checkpoint has duplication?
1. In training scenarios employing data parallelism (DP), model states are replicated across all DP groups, resulting in duplicated model states
2. To address this challenge, we implement a workload-balancing deduplication mechanism during the planning procedure
3. When loading checkpoints into parallelism configurations that include data parallelism, ByteCheckpoint optimizes the process by eliminating repetitive tensor reading across DP groups
n. Good. This is what I'm looking for in AITURBO.
1. Could AITURBO produce an even better checkpoint write plan?
1. ByteCheckpoint's Worst-Fit scheme operates on tensor-shard tasks. It can have difficulty perfectly balancing workloads when individual tensors are large and their sizes are uneven.
2. AITURBO supports smaller chunks, down to 4 MB, and can independently assign chunks to different sources and destinations.
11. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness
https://link.zhihu.com/?target=https%3A//www.usenix.org/system/files/fast26-hu-shipeng.pdf
1. Good. Simple but very effective caching policy for LLM KV cache. In another word, the classic LRU doesn't fit KV cache anymore.
2. My reading outs
1. What is the targeting scope of this paper?
1. LLM inference in chat conversation rounds, two-tier KV cache storage - DRAM + SSD.
How to leverage the coordination between the compute engine and the two-tier storage to reduce response latency.
2. Bidaw - a new KV caching approach
2. What are the key results?
1. Bidaw reduces response latency by up to 3.58× and improves throughput by up to 1.83× over state-of-the-art approaches
3. What are the key techniques?
1. request scheduling based on the KV-loading latency from two-tier storage
1. Only requests in “ready queue” can be scheduled for GPU inference. Rather than let GPU blocked by slower loading KV cache from SSD
2. promote the requests with shorter estimated I/O times to “ready queue” earlier for minimizing overall request queuing delays
3. example: Figure 11: I/O-aware request scheduling strategy reduces request blocking
2. KV eviction strategy guided by the length of the model answer in compute engine
1. in interactive LLM serving, the weighted reuse distance of a KV access is positively correlated with the length of LLM’s previous answer generated by compute engine (§3.3.1). This is because, longer model answer leads to more time for the user to read/listen and comprehend the answer
2. Bidaw captures the latest round’s model answer generated by compute engine for each user, and predicts the weighted reuse distance of the next KV access. We then estimate the hit probabilities for different weighted reuse distances by maintaining a ghost cache that uses an optimal eviction strategy (knowing the future information), leveraging the past I/O traces. Finally, Bidaw identifies KVs whose next accesses have the lowest estimated hit probabilities and evicts them to the capacity layer when the performance layer is full.
1. Steps: 1) estimating the hit potentials for accesses across different ranges of weighted reuse distances; 2) predicting the range of the weighted reuse distance of each user’s next KV access.
2. Algorithm explained: https://chatgpt.com/c/6adde5bd-f410-83ec-beb8-c48537392e2f
4. Why LRU Often Fails for LLM KV Caching
1. See interesting analysis in https://chatgpt.com/c/6adde5bd-f410-83ec-beb8-c48537392e2f
2. These problems of LRU
1. Poor temporal locality: In interactive conversations, users may not return for a long time, even when their KV will eventually be reused.
2. Scan pollution: Large amounts of newly generated, potentially one-time-use KV can displace older but repeatedly useful KV.
3. Unequal miss costs: A 10 MB KV miss and a 1 GB KV miss can have dramatically different consequences.
4. Structured sharing: KV blocks form prefix dependencies and may be shared by many requests, unlike independent cache entries.
5. Compute/storage coordination: The scheduler may already know which KVs will be accessed next, but LRU ignores this information.
n. Related materials
1. Fast26 | KV Cache 的两层存储(内存+SSD)到底卡在哪？ - marsggbo
https://zhuanlan.zhihu.com/p/2038938063070607023
2. 多轮对话的 KV 放上 SSD，延迟为什么反涨 3.8 倍
https://github.com/ForceInjection/AI-fundamentals/blob/main/09_inference_system/kv_cache/03_fast26/bidaw.md
12. Sharpen the Spec, Cut the Code: A Case for Generative File System with SYSSPEC [2026, 10 refs, FAST26 best paper]
https://www.usenix.org/conference/fast26/presentation/liu-qingyuan
https://www.usenix.org/system/files/fast26_slides_liu-qingyuan.pdf
chatgpt: https://chatgpt.com/c/6ade8572-7ba4-83ec-97e4-a91bd3869ad0
1.
2. Highlights
1. challenges
1. Challenge I: Specification semantic gap
2. Challenge II: Complex component composition
3. Challenge III: Unreliable LLM capability
2. specs
1. functional spec
1. pre-condition
2. post-condition
3. cases
2. modular spec
1. Predefined Structures/Functions
2. Exported Interface
3. concurrency spec
1. Locking Specifications of xxx
3. DAG-Structured Specification Patch
1. The leaf node: a self-contained change
2. Intermediate nodes: building on guarantees
3. The root node: the integration point
4. The SYSSPEC Toolchain
1. The SpecCompiler agent
1. translating the high-level, specification-based file system into a low-level C-language implementation
2. two-phase prompting
1. first phase generates a correct sequential implementation of the module, focusing only on its core functionality.
2. In the second phase, it uses the dedicated concurrency specification to instrument this sequential code with the necessary locking and concurrent behaviors
2. SpecValidator
1. SpecEval - it does not simply report failure; instead, it generates specific, actionable feedback
3. SpecAssistant
1. streamlines specification development through a human-in-the-loop process
n. my questions
1. There many papers talking about spec and rewrite linux filesystem. What makes this paper different?
1. SYSSPEC builds directly on AtomFS — from the same research group
2. Traditional formal-methods papers typically ask: Given a specification and an implementation, how can we prove that the implementation is correct?
SYSSPEC asks: Given a specification, how can we generate an implementation that is sufficiently likely to be correct, and evolve it without manually rewriting code?
2. The missing piece in SYSSPEC: Who verifies the specification? What if the spec is conflicting by itself？
1. The authors acknowledge specification errors and provide an LLM-assisted refinement mechanism, but they do not demonstrate a systematic way to detect logically contradictory or semantically incorrect requirements.
13. Principles and Methodologies for Serial Performance Optimization [2025, 5 refs, OSDI25]
https://www.usenix.org/conference/osdi25/presentation/park-sujin
1. A useful collection of common performance optimization techniques for a serial program
2. Highlights
1. 3 fundamental principles
1. Task Removal: Eliminate unnecessary operations entirely, reducing the amount of work on the critical path.
2. Task Replacement: Substitute expensive operations with cheaper alternatives that achieve the required result.
3. Task Reordering: Change execution order to improve locality, reduce dependencies, or move work away from performance-sensitive paths.
2. 8 performance optimization methodologies
1. Batching: Combine multiple operations to amortize fixed overhead, eliminate redundant work, and improve spatial or temporal locality; e.g., group commit, batched I/O, and write coalescing.
2. Caching: Store and reuse previously computed results to avoid repeated expensive operations; e.g., memoization, metadata caching, and caching scheduling decisions.
3. Precomputing: Execute anticipated work early, preferably outside the critical path, to eliminate future stalls; e.g., prefetching, proactive garbage collection, and speculative execution.
4. Deferring: Postpone work until it becomes necessary or more efficient, potentially eliminating it entirely; e.g., lazy evaluation, optimistic concurrency control, and deferred writes.
5. Relaxation: Trade strict guarantees such as accuracy, consistency, durability, or availability for faster execution when the application permits; e.g., sampling, approximate computation, and relaxed consistency.
6. Contextualization: Exploit runtime information and workload characteristics to make more efficient decisions than a general-purpose design; e.g., workload-aware scheduling, adaptive locking, and hot-data identification.
7. Hardware Specialization: Adapt execution to particular hardware capabilities or move operations to more suitable devices; e.g., NUMA-aware placement, FPGA acceleration, SmartNIC offloading, and persistent-memory-aware algorithms.
8. Layering: Restructure abstraction boundaries to reduce overhead or expose optimization opportunities, including bypassing (skip layers), delayering (merge layers), and decoupling (split layers into independently optimizable components).
n. Related materials
1. OSDI '25 论文《Principles and Methodologies for Serial Performance Optimization》技术解读 - 胡津铭
https://zhuanlan.zhihu.com/p/1961744461177553026
14. How to write to SSDs [2026, 5 refs, VLDB26]
https://www.vldb.org/pvldb/vol19/p1469-lee.pdf
https://arxiv.org/abs/2603.09927
chatgpt: https://chatgpt.com/c/6ae422f9-2238-83ec-a3fe-b8b695802a18
1. Optimize DB's B-tree writes with Out-of-place writes, compression, lifetime grouping, NoWA. It proves: 1) compression is the most effective, 2) ZNS SSD is the right direction, 3) NoWA is a invented technique (but not useful after ZNS SSD). Built based on LeanStore
2. Highlights
1. Evaluation
1. LeanStore, Physical flash writes per 4 KiB logical B-tree page write 18.85 KiB -> 2.42 KiB
2. Key techniques
1. Out-of-place writes
1. Established knowledge: Used by LSM trees, log-structured filesystems, SSD FTLs
2. Paper contribution: A practical redesign of an in-place B-tree engine that enables coordinated write placement
2. Database-level write amplification
1. Established knowledge: Logging, compaction, compression, GC have been widely studied
2. Paper contribution: Combines compression, alignment-aware page packing, and predicted page deathtime, deathtime grouping
3. SSD-level write amplification
1. Established knowledge: Hot/cold separation, GC-unit alignment, ZNS, FDP are known
2. Paper contribution: Develops and evaluates host write patterns intended to achieve SSD WAF ≈ 1, including a NoWA pattern for conventional SSDs, DB aligns to SSD GC unit
4. NoWA's solution: coordinate concurrent write groups
1. Rules
1. Do not open a new group of DB zones until the currently active group has been completely written.
2. Track differences in how quickly concurrently written zones become invalid, and compensate for imbalances before SSD GC is forced.
1. Even if the FTL interleaves data from different logical zones, it does not cause GC amplification if those zones become invalid together.
2. But what if A and C are very different in lifetime, or GC strongly prefers reclaiming B rather than C? That is where NoWA's compensation mechanism comes in.
5. How B-tree is made write out-of-place
1. When flushing the dirty page to disk, the modified implementation
1. Computes the page's estimated deathtime (EDT).
2. Compresses the page.
3. Finds a zone whose existing pages have similar EDTs.
4. Allocates space at that zone's current append position.
2. ZLeanStore's PID2OffsetTable maintains this logical-to-physical mapping.
n. My questions
1. Table 1, fully optimized wo. compression WAF is 3.58, while fully optimized w. compression WAF is 0.60. So, somehow the compression is the best optimization?
1. This invalidates the effort of this paper somehow.
2. Since SSD FTL already does out-of-place writes, can it replace what this DB side out-of-place optimization?
1. In Table 1, we can see it is actually the NoWA that is effective.
3. How can the authors measure SSD-internal write amplification when the FTL is hidden?
1. they use device-exposed physical NAND write counters.
4. How exactly do they infer the SSD's hidden GC-unit size?
1. The authors estimates, by altering write size vs guessed GC unit size, and observe amplification. When write size matches GC unit size, zero GC amplification should happen
n. Related materials
1. 数据库应该如何写SSD - 胡津铭
https://zhuanlan.zhihu.com/p/2084769109485605709
2. 从一到无穷大 #104：面向 SSD 的数据库写入设计——从 FTL 到应用协同 - 李兆龙
https://zhuanlan.zhihu.com/p/2098140491866961090
3. How to write SSD - 公园野鸭
https://zhuanlan.zhihu.com/p/2077144036058588719
4. How to Write to SSDs 解析 - 意珥
https://zhuanlan.zhihu.com/p/2079942546315273614
15. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [2026, 58 refs]
https://arxiv.org/abs/2604.25850
chatgpt: https://chatgpt.com/c/6ae284aa-5324-83ec-85cc-34b0d849e756
1. Let agent evolves its own harness by constructing the observability
2. Highlights
1. The central idea of this paper is to make coding agents improve their own engineering infrastructure, rather than improving the underlying LLM.
1. Today, when a coding agent fails, engineers generally inspect its execution logs and manually improve its system prompt, tools, memory, or execution framework. The paper proposes automating that engineering process.
2. The authors build a system called Agentic Harness Engineering (AHE), where another LLM agent observes coding-agent failures, diagnoses the root causes, modifies the coding-agent framework, and experimentally evaluates whether those modifications improve performance.
2. Solution: Three observability layers implement this principle
1. Component observability
2. Experience observability
3. Decision observability
1. Instead of allowing the Evolve Agent simply to say, "I improved the harness," the system requires it to document each modification.
1. The evidence motivating the change.
The inferred root cause.
What was modified.
Which previously failing tasks should improve.
Which previously successful tasks might regress.
n. related materials
1. Microsoft: Hypervelocity Engineering: Accelerating Enterprise AI with Azure AI Landing Zones
https://techcommunity.microsoft.com/blog/azurearchitectureblog/hypervelocity-engineering-accelerating-enterprise-ai-with-azure-ai-landing-zones/4536192
https://github.com/microsoft/hve-core
16. Deep dive FlashAttention
0. Very good.
1. First, understand GPU architecture
1. GPU Architecture & Working intuitively explained
https://pub.towardsai.net/gpu-architecture-working-intuitively-explained-2acce00f2fec
2. Hugging Face: Flash Attention basic intro
https://huggingface.co/docs/text-generation-inference/conceptual/flash_attention
2. Next, standard math representation for matrix operations
Chatgpt: https://chatgpt.com/c/6aefc1fb-d310-83ec-9206-73c4c7d602f4
1. Calculate attention.
1. Attention formula: O = softmax(Q*K^T+M)*V
M is causal mask, which uses negative infinite to mask out the attention from Token_i with to Token_i+1.
This is fundamentally why KV cache is possible, because k_i+1 and v_i+1 is only needed later at i+1.
2. Step by step calculation:
// A row is a token vector
// Python slicing, where a:b, a is inclusive, b is exclusive, index starting from zero
Scores = S := Q[i:i+1] @ K[:i+1,:].T // Dimension (1,i)
// softmax axis=-1 means to work on row level, while axis=0 means to work on a column. Softmax works on a single vector rather than a whole maxtrix
Weights = W := softmax(S, axis=-1) // Dimension (1,i)
// @ is the operator of matrix multiplication
O[i:i+1, :] = W @ V[:i+1,:]
// Expanding softmax
W[j] = exp(S[j]) / sum(exp(S[k]) for k in 0..i+1)
3. Good, with ascii matrix math, which is compatible with python, in a few lines it directly shows why KV cache is necessary
3. Basic FlashAttention in python ascii representation
chatgpt: https://chatgpt.com/c/6aefc1fb-d310-83ec-9206-73c4c7d602f4
1. Tile vs KV token/block
1. They are totally different. Tile is a sub-matrix, it can take multiple rows (multiple tokens) but won't take the full columns (all token dimensions). Tile is pure computation concept used in GPU attention calculation. Unlike KV cache, tile is not cached in tiering storage.
2. The problem that FlashAttention is facing is, the full Q@K.T matrix is way too large, context length * content length, to put into SRAM in GPU (HBM->SRAM->Registers->Tensor Cores) so as to participation. We need to partition the matrix. The partition is called "tile".
1. Sub-matrix operation is actually easy. They are standard math. The true problem that FlashAttention solved is how to "incrementally" calculate softmax tile by tile, which is called "online softmax"
2. Attention scores tile by tile
1. S := Q@K.T // Q dim = context length * d_k, K dim = context length * d_k
Then,
S[a:b,c:d], which is a tile = Q[a:b,:]@K[c:d,:].T
3. Online softmax tile by tile
1. Softmax only cares row by row, so
W[row,c:d] = exp(S[row,j]) / sum(exp(S[row,k]) for k in 0..ctx_len) for j in c..d
Then, the final output of attention
O[row,:] = W[row,:]@V
= sum(exp(S[row,j])/sum(exp(S[row,k]) for k in 0..ctx_len) *V[j,:] for j in 0..ctx_len)
= sum(exp(S[row,j])*V[j,:] for j in 0..ctx_len) / sum(exp(S[row,k]) for k in 0..ctx_len)
So, we can decompose O[row,:] into two partss
U[row,:] = sum(exp(S[row,j])*V[j,:] for j in 0..ctx_len) // Dim is (1, d_token)
L[row] = sum(exp(S[row,k]) for k in 0..ctx_len) // A scalar value
O[row,:] = U[row,:]/L[row]
Now, to tell in natural language, in online softmax, we only need to track the U and L values per row. Rather than the softmax value itself. U and L have clear linear accumulating across tiles:
Define U[row](c,d) := sum(exp(S[row,j])*V[j,:] for j in c..d)
Then U[row] = sum(U[row](c,d) for (c,d) in each tile)
Similar define L[row][c,d] := sum(exp(S[row,k]) for k in c..d)
Then L[row] = sum(L[row][c,d] for (c,d) in each tile)
Now, we don't need to put the big S or W matrix into SRAM. We only need to put U, L in SRAM, and transfer Q, K, V tile by tile throughout the GPU calculation. The comparing baseline is materializing the big S thus you need to write/read S from HBM.
Additionally, the loading of the next tile and the computation of the current tile can overlap, this further hides the IO time of loading tile into SRAM.
4. Problem: Since we separate U and L, we are exposing the big exp numbers in calculation.
Solution: FlashAttention introduced techniques for number stability
1. softmax has a nice property: softmax(s) = softmax(s-c), so we choose c=max(s[j] for j in 0..len(s))
2. So, if we put the c=max(tile) into U, L calculation. When new tile comes and the max is switched, we need to multiple a scaling factor to the old U and L
scaling factor = exp(-c_new - (-c_old))
5. Putting all together, the flash attention algorithm
chatgpt: https://chatgpt.com/c/6aefc1fb-d310-83ec-9206-73c4c7d602f4
---
# Q: [Nq, d_k]
# K: [Nk, d_k]
# V: [Nk, d_v]
# Output: [Nq, d_v]
#
# Bq: number of query rows per tile
# Bk: number of key/value rows per tile
O = zeros((Nq, d_v))
for q_start in range(0, Nq, Bq):
q = Q[q_start:q_start+Bq, :]
# q: [bq, d_k]
bq = q.shape[0]
m = full((bq, 1), -inf) # Row maxima
l = zeros((bq, 1)) # Row denominators
u = zeros((bq, d_v)) # Weighted sums
for k_start in range(0, Nk, Bk):
k = K[k_start:k_start+Bk, :]
v = V[k_start:k_start+Bk, :]
# 1. Calculate score tile
s = (q @ k.T) / sqrt(d_k)
# s: [bq, bk]
# 2. Update running maxima
tile_max = max(s, axis=1, keepdims=True)
m_new = maximum(m, tile_max)
# 3. Rescale old accumulators
alpha = exp(m - m_new)
# 4. Exponentiate current tile
p = exp(s - m_new)
# 5. Update denominator
l = alpha * l + sum(
p, axis=1, keepdims=True
)
# 6. Update weighted value sum
u = alpha * u + p @ v
m = m_new
# 7. Normalize once, after all K/V tiles
O[q_start:q_start+bq, :] = u / l
---
4. FlashAttention V1, V2 and V3 evolving
chatgpt: https://chatgpt.com/c/6aefc1fb-d310-83ec-9206-73c4c7d602f4
1. FA1: Avoid transferring unnecessary intermediate data.
1. The breakthrough was tiled, fused attention with online softmax. It avoided materializing the full attention matrix in HBM and used recomputation for training.
2. FA2: Make better use of GPU parallelism and reduce expensive non-matrix operations.
1. Reduced non-matmul work, improved occupancy by parallelizing over query sequence blocks, and changed how work is divided among GPU warps.
2. FA1 - Split KV across warps and share Q => FA2 - Split Q across warps and share KV
1. FA2 changes the partitioning so that each warp can independently calculate its own output rows.
This is a classic parallel-computing optimization: prefer partitioning work by independent outputs rather than partitioning a reduction across workers.
1. I.e. FA1 forces the warps to merge to the same output row, high contention, bad.
OK .. this means FA1 will actually cause you problems, it's just the authors didn't realize it.
Interesting problem.
3. FA3: Overlap data transfer, matrix multiplication and softmax more effectively using asynchronous hardware execution.
1. Exploited Hopper's TMA and WGMMA, overlapped GEMM with softmax, and introduced FP8 optimizations including block quantization and incoherent processing.
5. The original papers
1. FA1: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness [2022, 7086 refs]
https://arxiv.org/abs/2205.14135
2. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning [2023, 3955 refs]
https://arxiv.org/abs/2307.08691
3. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision [2024, 847 refs]
https://arxiv.org/abs/2407.08608
6. "Tile" isn't a new idea
1. Tiling (also called blocking) is a common technique in numerical computing, even before Neural Networks and FlashAttention.
2. In fact, FlashAttention's novelty is not tiling matrix multiplication. That was already standard practice. Its major innovation was making the entire attention operation, including softmax, work efficiently with tiles without materializing the large intermediate attention matrix.
3. A multi-GPU tensor shard and an on-chip computation tile are not the same thing. Sharding determines ownership across devices; tiling determines how a local computation is broken down. They can be combined.
Create an Issue or comment below