Recent Articles Reading: KV Cache, Jev, AI Storage
02 November 2026
1. KV cache in AI storage
1. ChatGPT: https://chatgpt.com/c/6a62da76-9c18-83ec-9b37-07588b58cef6
1. ChatGPT for vLLM vs SGLang: https://chatgpt.com/c/6a62ddc3-8c94-83ec-a00a-829dba397835
1. PageAttention and RadixAttention
2. KV cache workload pattern characteristics
1. Cache block is typically 16 tokens, aligned with GPU allocation page
2. A token is larger than it sounds: S_token=2×L×H_KV×D_H×S_element
For a Llama-3.1-70B-like GQA configuration with 80 layers, 8 KV heads and 128-dimensional heads
- 2×80×8×128×2=320 KiB/token
3. A 2026 trace of Claude Code and Codex found a median step with approximately:
119,000 cached prefix tokens.
875 newly appended tokens.
214 output tokens.
1. TraceLab: Characterizing Coding Agent Workloads for LLM Serving
https://arxiv.org/html/2606.30560v2
4. A storage-oriented system should preserve or reconstruct sequentiality by packing consecutive prefix blocks into large extents or chunks
5. Spatial locality: early blocks are much hotter: system prompts, tools, template
1. Hot application prefix
Examples:
System prompt.
Tool descriptions and schemas.
Safety instructions.
Few-shot examples.
Stable agent instructions.
A common RAG document prefix.
These may be shared by many requests and deserve:
Pinning in DRAM or HBM.
Replication across workers.
Cache-aware routing.
Longer retention.
Possibly eager preloading.
2. Warm active-session trunk
This includes the user-specific conversation history.
It normally has:
Very high reuse probability for the next turn.
Low or zero reuse by unrelated users.
A relatively predictable next-use time.
Increasing size over the session.
It is a strong candidate for DRAM or fast shared flash when the user or tool is temporarily idle.
3. Hot but short-lived tail
The latest assistant output and tool result are likely to be used almost immediately by the next agent step. They should usually remain close to the active GPU or be written asynchronously to the next tier.
6. Temporal locality is strong but workload-dependent
The Aliyun production study found:
In human-facing workloads, 80% of KV reuse occurred within 10 minutes.
In programmatic API workloads, 80% occurred within 10 seconds.
Ninety percent of blocks in the human-facing trace received no further reuse after approximately 612 seconds.
API-system-prompt blocks had extremely short common reuse intervals but also some long-lived outliers.
1. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
https://arxiv.org/html/2506.02634v3
7. Is cross-conversation sharing rare?
1. Programmatic single-turn API calls
Same 1,000-token translation instruction
+ different document each time
2. Shared application templates
3. RAG and agent workloads
8. A hierarchy is a basic requirement
L0 Active request pages in GPU HBM
L1 Other GPU or inactive local HBM
L2 Host DRAM / CXL memory
L3 Local NVMe
L4 Pod-level shared DRAM or flash
L5 Large shared storage
↓
Recompute when cheaper
9. KV workloads that fit S3/object store
1. SSD-backed
1. Active or recently paused conversations; long-running agents; popular system prompts; prompt templates; frequently reused RAG documents; cross-worker shared prefixes
2. HDD-backed S3/object store
1. Long-idle but resumable sessions; low-frequency tenant contexts; very large agent histories; the cold tail of a huge KV corpus
3. KV cache architecture shift - Good
1. As of July 2026, the major architectural shift is this: KV cache is moving from an internal GPU-memory optimization to a distributed, tiered and independently managed data service.
2023 2024 2025–2026
GPU page allocator → prefix-aware cache → cluster-wide KV service
↓
HBM / DRAM / CXL
↓
local and shared NVMe
↓
pod-level “context memory”
1. Trend 1: Pages and token blocks have become the physical storage unit
1. The first major transition came from vLLM’s PagedAttention
2. Trend 2: Prefix trees and content addressing are replacing session-affine storage
1. SGLang’s RadixAttention organizes reusable KV state around a radix tree of token prefixes
2. global namespace. The llm-d KV indexer
3. Trend 3: KV cache is becoming an independent distributed service
1. Mooncake, used by Moonshot AI’s Kimi service, was one of the clearest early demonstrations
4. Trend 4: Prefill/decode disaggregation turns KV into a network protocol
1. Prefill is relatively compute-intensive.
2. Decode repeatedly reads model weights and accumulated KV state and is commonly memory-bandwidth-sensitive.
6. Trend 5: The hierarchy is deepening beyond HBM and host DRAM
1. Mooncake, LMCache, NVIDIA Dynamo, vLLM connectors and llm-d now expose different portions of this hierarchy
L0: active HBM KV pages
L1: inactive HBM or peer-GPU memory
L2: local host DRAM
L3: rack or pod disaggregated DRAM / CXL memory
L4: local NVMe SSD
L5: shared low-latency NVMe or flash appliance
L6: general shared storage or object storage
L7: discard and recompute
2. The placement decision therefore needs a cost model such as:
1. cache value ≈ P(future reuse) × prefill cost saved − fetch cost − capacity and network cost
This is more appropriate than ordinary LRU.
7. Trend 6: Compute scheduling and storage scheduling are merging
1. A global KV store can make load balancing harder. Routing solely to the worker with the longest prefix match creates hotspots; routing solely to the least-loaded worker destroys locality.
2. A 2025 production-workload study found that reuse is highly skewed, that single-turn reuse is as important as multi-turn reuse in its traces, and that reuse patterns become more predictable after requests are divided into categories.
worker score = prefix match benefit
- worker queueing delay
- KV transfer time
- expected prefill time
- memory pressure
- SLO risk
8. Trend 7: The CPU is being removed from the KV I/O path
1. Putting KV on SSD does not automatically make SSD-backed caching efficient. Paged GPU layouts can generate many small, scattered transfers, while CPU submission and buffer management can prevent modern NVMe devices from reaching useful throughput.
2. Several approaches are appearing:
1. GPUDirect Storage and RDMA bypass unnecessary copies through host memory.
2. NIXL creates a common abstraction for GPU, host and storage movement.
3. Storage vendors are adding NIXL plugins and GPU-direct paths.
4. Tutti, currently a 2026 preprint, proposes a GPU-centric KV object store and GPU-side I/O submission to eliminate the CPU from both the data and I/O-control path.
5. NVIDIA’s announced BlueField-4 STX architecture moves storage processing, protection and KV I/O handling onto a DPU rather than the application CPU.
3. The likely long-term split is:
CPU/control processor:
namespace, policy, admission, scheduling
GPU/DPU data plane:
tensor movement, checksums, encryption,
decompression and storage I/O
9. Trend 8: Compression is becoming an adaptive storage and network policy
1. CacheGen treats cached context as a compressed stream. It exploits structure in KV tensors and adapts compression levels to available bandwidth, trading transfer size against generation quality.
10. Trend 9: Reuse is expanding beyond exact same-model prefixes
1. Classic prefix caching is exact and position-dependent. A block is reusable only when the same model sees the same preceding tokens at the same logical positions.
Research is relaxing those constraints ... CacheSlide
11. Trend 10: Database storage techniques are entering KV-cache implementations
1. The earliest external cache systems often stored one file or object per KV chunk. At scale, that can create metadata pressure, poor prefix locality and too many small I/Os.
2. SGLANG-LSM, currently a research preprint, proposes applying LSM-tree ideas to KV-cache storage
2. The agentic workload is accelerating the change
1. Unlike single-turn inference - chatbots, Agent workloads look very different
Long existing context
+ small tool response
+ short reasoning step
+ another tool call
+ repeated for tens or hundreds of turns
3. Others
1. Hugging Face’s documentation accordingly describes KV caching as an inference mechanism and warns against enabling it during ordinary training
1. KV caching is useful when the model repeatedly extends the same already-processed sequence. Standard training processes the complete training sequence once, in parallel, and then backpropagates through it.
2. How it maps to Prefill and Decode stages
1. Prefill
1. For a prompt of P tokens, the prefill stage processes the prompt largely in parallel.
2. Prefill resembles a training forward pass. The model calculates Q,K,V for all prompt positions.
2. Decode
1. After prefill, decode generates one token—or occasionally a small speculative group—at a time
n. Related materials
1. How KV cache works
https://medium.com/@joaolages/kv-caching-explained-276520203249
2. TraceLab: Characterizing Coding Agent Workloads for LLM Serving [2026, 0 refs]
https://arxiv.org/html/2606.30560v2
1. ChatGPT: https://chatgpt.com/c/6a635b79-36f0-83ec-b548-2588acce2c9b
2. Highlights
1. Coding agent characteristics
1. short burst of tool calls. at <5min time, high KV cache hit. prefix-to-append ratio approximately 136x
2. 5min gap for human thinking, KV cache is likely to expire/evict, then causes miss.
3. The cache-retention knee: evict timeout, hit rate, suspended/active KV ratio, prefill amplification
4. The paper’s approximately 119K-token median prefix would occupy: 119K*320K = 36.3GB. To restore KV state in 3 seconds, it means 12.1GB/s throughput.
3. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider [2025, ATC25, Alibaba, 48 refs]
https://www.usenix.org/conference/atc25/presentation/wang-jiahao
1. ChatGPT: https://chatgpt.com/c/6a635b79-36f0-83ec-b548-2588acce2c9b
2. Highlights
1. Compared to TraceLab, this paper records anonymized token-prefix identities, therefore, it can reconstrut prefix matching results
2. Finding two: single-turn requests can dominate reuse
1. Counter-intuitive to that multi-turn request is reusing the most.
2. This is because single-turn requests reuse the system prompt.
3. Finding three: reuse is strongly skewed
1. 10% of KV blocks→77% of reuse events
2. Trace B, A p90 KV-block lifespan of only 0.3 seconds.
4. Finding five: temporal locality depends heavily on workload class
1. the cache-retention decision should use application type:
5. Spatial locality: cache the head, not the middle or tail
1. Blocks near the beginning of the request are substantially more reusable.
6. Lifespan is different from reuse time
1. This is why LFU performs poorly
1. A system prompt: app ends, block is dead
LFU: Very frequent; retain it forever
Workload aware policy: Expected application-specific lifespan has expired; evict it
7. The authors capacity conclusion
1. For their Qwen2-7B configuration, 16 tokens consume approximately 0.875 MB of KV.
These contexts are dramatically smaller than the approximately 119K-token median prefix observed by TraceLab for coding agents.
2. The authors estimate the capacity needed to achieve the infinite-cache hit rate using an ideal oracle:
1. Add a request’s blocks if they will be reused.
2. Remove blocks immediately after their final reuse.
3. Measure the maximum live reusable KV state.
4. Normalize it against HBM available for KV.
3. For GQA models, the result is relatively modest:
1. In Trace A, Llama3-70B requires approximately four times the available KV HBM to attain the ideal hit ratio.
2. In Trace B, the required capacity is smaller than the reserved GPU HBM.
3. A typical server with eight A100 GPUs and 1 TB of host DRAM supplies about 128 GB of host memory per GPU, which the authors argue is approximately four times the GPU memory reserved for KV in their example.
4. Their conclusion is:
1. For these conventional chatbot and API workloads using GQA models, HBM plus local host DRAM may be sufficient; remote DRAM and SSD are not necessarily required.
2. This is not a universal conclusion about AI storage.
5. It depends on:
1. Relatively short request lengths.
2. GQA reducing KV bytes per token.
3. A single model per trace.
4. Their observed request rates.
5. Their specific chatbot and API workloads.
6. An idealized capacity calculation.
7. No large coding-agent contexts.
6. For MHA models, the paper finds much larger cache requirements.
1. For MHA vs MQA vs GQA vs MLA
https://medium.com/@zaiinn440/mha-vs-mqa-vs-gqa-vs-mla-c6cf8285bbec
8. The workload-aware eviction policy
1. A block is more valuable when:
1. Its workload-specific reuse score is higher.
2. It is closer to the beginning of the request.
2. Why it outperforms LRU
Block A:
text-1
First request of a conversation
Low probability of continuation
Block B:
text-10
Conversation already has ten turns
High probability of continuation
1. LRU sees equal age.
The workload-aware policy prefers Block B.
3. Why it outperforms LFU
Block C:
Past frequency = 500
Application batch has finished
Future reuse probability ≈ 0
Block D:
Past frequency = 1
Active conversation is likely to continue
Future reuse probability is high
1. LFU retains C. The proposed policy retains D.
4. Making eviction scalable
1. The authors observe that, within a workload category, the fitted score is naturally ordered by recency. They therefore maintain an LRU queue for each workload:
During eviction:
1. Select the oldest block from every workload queue.
2. Calculate priorities only for those candidates.
3. Evict the lowest-priority candidate.
2. They report approximately 79 microseconds per eviction and aggregated overhead equal to roughly 1.2% of vLLM scheduling overhead.
3. Limitations
1. Single-provider scope. The observations come from two workloads at one cloud provider, and the paper explicitly notes that newer reasoning workloads were outside its study.
2. Short-context bias. Mean contexts of approximately 973 and 5,953 tokens do not represent coding-agent sessions containing hundreds of thousands of tokens.
3. The data covers two one-week periods, one in December 2024 and one in February 2025
4. NVIDIA Dynamo
https://github.com/ai-dynamo/dynamo/blob/main/docs/design-docs/kvbm-design.md
1. ChatGPT: https://chatgpt.com/c/6a65a3c6-5c18-83ec-9cb8-29b19aaf3550
2. Highlights
1. vLLM and SGLang optimize model execution inside serving workers. NVIDIA Dynamo optimizes how many such workers operate together as a distributed inference system.
2. KVBM / HiCache / LMCache / FlexKV
1. They are mostly alternatives
KVBM
Engine connector → KVBM block pools → NIXL → DRAM/SSD/remote
HiCache
SGLang scheduler + HiRadixTree → GPU/DRAM/shared L3
LMCache
Inference engine → LMCache daemon/service → DRAM/SSD/remote
FlexKV
Inference engine → in-process FlexKV library
→ RadixTree + CPU/SSD/distributed storage
2. Strongest characteristic
1. KVBM: Portable block/data plane abstraction from NVIDIA Dynamo
2. HiCache: Tight engine integration, originated from SGLang, index by HiRadixTree rather than block hash
3. LMCache: Standalone opensource KV cache. Rich features and persistence service.
4. FlexKV: Originated from Tencent Cloud TACO. Fast SSD/distributed storage path.
3. Radix tree indexing vs hash indexing
1. Radix tree
1. preserve prefix topology, nice to analysis and advanced caching algorithms. The cost is complex metadata and slower lookup
2. Can answer
1. What are all descendants of this prefix?
How many requests share this prefix?
Which leaf branch is coldest?
If I evict this node, which later blocks become unusable?
Which waiting requests belong to the same prefix subtree?
2. Hash indexing
1. faster lookup, avoid to index middle nodes
6. LMCache
https://docs.lmcache.ai/developer_guide/architecture.html
1. ChatGPT: https://chatgpt.com/c/6a65a3c6-5c18-83ec-9cb8-29b19aaf3550
2. Highlights
1. In current LMCache multiprocess mode (P2P mode), there is no single global directory that maps KV key -> node/tier/device
Instead, the requesting LMCache server resolves the object hierarchically:
vLLM GPU metadata
↓
local LMCache L1 object table
↓
enabled external sources:
peer LMCache L1 caches
configured L2 adapters
↓
recompute
1. MP Coordinator: maintains the live fleet membership. it only manages node membership, rather than data placement.
2. Every live, P2P-enabled LMCache server registered with the same coordinator becomes a peer of every other live, P2P-enabled server.
2. Architecture diagram
Inference cluster
┌──────────────────────────────────────────────────────────┐
│ │
│ Node A Node B │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ vLLM GPU/HBM │ │ vLLM GPU/HBM │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ ZMQ / GPU transfer │ │
│ ┌──────▼────────────┐ ┌──────▼────────────┐ │
│ │ LMCache MP server │ │ LMCache MP server │ │
│ │ │ │ │ │
│ │ L1: pinned DRAM │◄──RDMA──►│ L1: pinned DRAM │ │
│ │ or │ P2P │ or │ │
│ │ GDS NVMe slab │ │ GDS NVMe slab │ │
│ └──────┬────────────┘ └──────┬────────────┘ │
│ │ async L2 adapters │ │
└────────┼───────────────────────────────┼──────────────────┘
│ │
└──────────┬────────────────────┘
▼
Shared or distributed L2 storage
S3 / shared FS / Redis / Valkey /
Mooncake / Bigtable / Aerospike / etc.
MP Coordinator
membership, heartbeats, quota,
control operations, L2 accounting
7. Bidaw: Enhancing Key-Value Caching for Interactive LLM Serving via Bidirectional Computation–Storage Awareness [2026, FAST26, 2 refs]
https://www.usenix.org/conference/fast26/presentation/hu-shipeng
1. Key problems to solve
1. the compute engine and the two-tier storage are mutually unaware
2. the twotier storage’s KV eviction strategy does not consider the user conversation patterns in compute engine, and only uses its own past KV accesses (queuing) information
2. Highlights:
1. ChatGPT: https://chatgpt.com/c/6a65c642-2f4c-83ec-88ae-f6fa64c38c72
2. What Bidaw is really about
1. It caches the historical tensors of an ongoing user conversation after each interaction round. When the user returns with the next question, those tensors are restored rather than recomputing the entire history.
8. DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference [2026, DeepSeek, 7 refs]
https://arxiv.org/html/2602.21548v2
https://www.youtube.com/watch?v=6iPItyp82FU
1. Roughly improves throughput to 1.5x. The design is simple but effective, good.
2. Highlights
1. ChatGPT: https://chatgpt.com/c/6a65ce8f-8338-83ec-a078-dd736b3d6f91
2. Key problem to solve:
1. storage network bandwidth is unevenly utilized across engines. The bandwidth of prefill engines are persistently saturated, while decoding engines remain underutilized.
3. Key solutions
1. Figure 1.Existing bottleneck (left) and DualPath (right).
1. in addition to the conventional storage-to-prefill path, KV-Cache can be loaded into decoding engines and then transferred to prefill engines via high-performance RDMA
2. To avoid congestion and keep balanced usage
1. an optimized dual-path loading data path design
2. Traffic manager and priority isolation
3. a dynamic scheduling policy that jointly balances computation and network utilization
3. Baseline techniques
1. PD Disaggregation
2. Layerwise prefill, which avoids HBM bottlenecks
4. CNIC-Centric Traffic Manager
1. Existing data transfer technologies in LLM inference systems
1. on-chip CUDA copy engine
2. GPUDirect Storage
2. For the InfiniBand-based network, we leverage virtual lanes (VLs)
1. For RoCE, leverage Traffic Class (TC) and Differentiated Services Code Point (DSCP) markings
3. CNIC-Assisted KV-Cache Copy
1. Instead of GPUDirect Storage, transfer data using DRAM as the middle bridge. This leverages CNIC to provide QoS and allows VL.
2. In DualPath, CNIC is the network-interface card attached to the high-speed GPU compute network. The paper also calls it the east–west NIC.
4. Others
1. Bottleneck-Free Analysis
2. Cloudflare Meerkat：全球控制面为什么想摆脱 Raft Leader - 废柴的文章
https://zhuanlan.zhihu.com/p/2061020697636369839
1. Key problems:
1. Raft 的问题：Leader 和 Timeout 在广域网里很脆
1. 但 Cloudflare 的场景跨整个 Internet。广域网延迟会变，链路会抖，队列会积压，数据中心可能失联。Leader-based consensus 在这种环境下会遇到两个现实问题。
第一，leader 故障或变慢会让写入停住，直到新的 leader election 完成。若 timeout 设得短，网络延迟稍微抖动就可能触发频繁选主；若 timeout 设得长，真正故障时恢复又很慢。多个 replica 同时发起选主还可能互相干扰。
第二，leader 即使没故障，也可能成为性能瓶颈。所有写都要转发给 leader，leader 所在数据中心的链路质量、CPU、队列状态都会影响全局写入。对于一个 330+ 数据中心网络来说，这个集中点很容易和“任意位置都能写”的控制面目标冲突。
2. Key solutions
1. QuePaxa：Leader 是快路径，系统进展不依赖它
QuePaxa 试图摆脱 timeout 驱动的 liveness。Cloudflare 文章给出的简化解释是：客户端可以联系任意 replica，那个 replica 可以为最新 slot 推进 consensus。系统里仍然存在 leader，但 leader 的意义是更快，通常可以用一轮往返推进；非 leader 也能推进，只是可能需要 3 轮或更多往返。
这带来一个重要性质：leader 下线或变慢，不会让系统必须等待 timeout 和 election。客户端可以联系其他健康 replica。多个 replica 同时 proposal 时，它们不会像 Raft leader election 那样破坏性干扰，而是共同推进某个值被决定。
2. 性能边界：全球强一致绕不开多数派 RTT
Meerkat 没有把成本藏起来。所有 consensus 算法都要付出往返通信成本。QuePaxa 决定一个 proposal 通常需要一到三轮在 proposer 与多数派之间的往返，leader 快路径可以更短，非 leader 路径更长。若 replica 分布得很远，写入和强一致读取的延迟会按多数派之间的网络延迟上升。
Cloudflare 提到几种缓解方式。开发者可以选择更近的 replica 放置，适用于不需要真正全球分布的服务。写入可以 batch，例如 10ms 内收到的多次写可以合进一个 proposal。部分读可以使用 stale-but-consistent 本地读，避免每次都进 consensus。KV store 还支持 compare-and-swap 类写入，把多步操作绑定到一个 consensus round。
这些优化不能消除物理延迟。Meerkat 短期定位是小块、低频、必须一致的控制面状态，并非通用数据库。
3. Cloudflare 已经跑过最多 50 个 replica 的全球 proof-of-concept，文章称 leader 不断失败时 cluster 仍能运行且 error rate 没有升高，但它仍未进入生产。
n. Related materials
1. QuePaxa: Escaping the tyranny of timeouts in consensus [2023, 28 refs]
https://github.com/dedis/quepaxa
https://dl.acm.org/doi/10.1145/3600006.3613150
1. QuePaxa is cited as "beyond Raft", the new generation consensus system. Adopted in Cloudflare Meerkat, targeting the cross region Internet scenario
1. Good paradigm shift paper any. Good to see it is adopted in production at Cloudflare after these years.
2. Highlights
1. Leader is just the fast path not the critical path
2. Compared to Paxos where each participant trying to shoot each other, QuePaxa is cooperative. The ordering is pre-determined by random numbers. The participants still retry after a short delay, the retry are idempotent rather than trying to overthrow the original leader.
3. ChatGPT: https://chatgpt.com/c/6a66c901-6b54-83ec-8b16-e8399ec08267
QuePaxa: concurrent conflicting proposals
---
QuePaxa: 3 recorders R1,R2,R3; quorum=2
pA=<priorityA,A,X>, pB=<priorityB,B,Y>, and pA>pB
Phase 0 / step 4: choose candidate
First proposals stored:
R1.F=pA, R2.F=pB, R3.F=pB
A reads {R1,R2}: best(pA,pB)=pA -> A carries pA
B reads {R2,R3}: best(pB,pB)=pB -> B carries pB
Phase 1 / step 5: spread candidates
A sends pA to {R1,R2}
B sends pB to {R2,R3}
Phase-1 aggregates:
R1=max(pA)=pA
R2=max(pA,pB)=pA
R3=max(pB)=pB
Phase 2 / step 6: gather and possibly decide
A reads previous-step aggregates from {R1,R2}:
{pA,pA}; best=pA == A's pA
-> A decides X
B reads previous-step aggregates from {R2,R3}:
{pA,pB}; best=pA != B's pB
-> B cannot decide Y
During phase 2:
R1 records pA
R2 records both pA and pB; aggregate=max(...)=pA
R3 records pB
Phase 3 / step 7: converge for next round
B reads previous-step aggregates from {R2,R3}:
{pA,pB}; best=pA
-> B adopts pA/X as its candidate for the next round
---
2. Traditional Basic Paxos: competing proposers
---
Basic Paxos: 3 acceptors R1,R2,R3; quorum=2
A wants X using ballot 10
B wants Y using ballot 11
1. A -> {R1,R2}: Prepare(10)
R1,R2 promise not to accept ballots <10
2. B -> {R2,R3}: Prepare(11)
R2,R3 promise not to accept ballots <11
R2's promise(11) preempts A's ballot 10
3. A -> {R1,R2}: Accept(10,X)
R1 accepts (10,X)
R2 rejects it because R2 promised ballot 11
A has only one acceptance -> no quorum
4. B examines Prepare(11) replies
Neither R2 nor R3 reports an accepted value
-> B may choose Y
5. B -> {R2,R3}: Accept(11,Y)
R2,R3 accept (11,Y)
-> Y is chosen by a quorum
6. A retries with ballot 12:
A -> {R1,R2}: Prepare(12)
R1 reports accepted (10,X)
R2 reports accepted (11,Y)
Highest accepted ballot is (11,Y)
-> A MUST propose Y, not X, in ballot 12
---
4. Caveats
1. The priority is generated randomly in each round. It doesn't use a fixed ordering, such as machine id, to avoid being bottleneck-ed by a slow and constant leader.
2. The random portions of two QuePaxa priorities can be equal. When the random priority is equal, the proposer id must break the tie. Or, you detect the tie in the vote round, and hope they don't collide in the next round.
3. The published QuePaxa protocol assumes a well-known, static replica set. If you have a large number of machines dynamically join/exit, you'd be careful.
3. AWS S3 架构深度解析 - 李明
https://zhuanlan.zhihu.com/p/2055746074715894477
1. Namespace / Metadata Service, 把 metadata 拆成很多 partition
1. range partition 是 namespace 的主分区方式，因为它服务 LIST；hash、前缀随机化、adaptive split 更多是为了处理热点，而不是替代有序 namespace
2. 早期 S3 曾经建议用户随机化 key prefix，本质上就是在 range 分区模型下避免顺序 key 把压力集中到单个热 range。后来 S3 自动扩展 prefix 性能后，用户不再需要手工随机化，但底层仍然要同时维护“可顺序扫描”和“可热点拆分”这两个目标。
2. S3 的一个关键思想是 ... 有大量客户的 bucket 会分布到数百万块 drive 上
3. S3 的 storage node 是 key-value store，ShardStore，不是“文件路径 -> 文件”
1. key 是 shard identifier
2. ShardStore 的内部再把 shard bytes 写成 chunk，并把 chunk 放进 extent
4. ShardStore 论文还说明：ShardStore 用 LSM tree 存索引，shard data 存在 LSM 外部，以降低写放大
1. ShardStore 选择自己管理 extent、append-only write、LSM index 和 GC。这样 S3 可以把单盘当作“shard KV device”，而不是普通目录树。
2. 论文说明它使用类似 soft updates 的 crash consistency 协议
5. Background Operations / Data Services 与 Namespace / Metadata Service 分离
6. Frontend / Request Router is a separate service
7. Heat management：把请求压力摊到整个 fleet
1. S3 面临的基础物理矛盾是：HDD 容量增长远快于随机 IOPS 增长
2. 对象存储不能只按容量均衡放置，还必须按未来可能的请求热度均衡放置。
3. 但写入时系统并不知道未来哪些对象会热。S3 的解法不是准确预测每个对象，而是利用巨大多租户规模：
- 单个 workload 往往突发且稀疏。
- 大量 workload 聚合后，整体曲线更平滑。
- 如果每个 bucket 的对象被广泛摊到 fleet 上，客户突发就能借用大 fleet。
- replication/EC 让读路径有多个候选 shard，可以避开热点盘。
小集群必须猜测热点；S3 的规模让许多热点在聚合后被摊平。
8. Placement service - Placement is also a separate service?
1. No public source confirmed. Only "placement function" so far
https://chatgpt.com/c/6a73237f-bf34-83ec-ae4a-7a58071d8972
4. 让高性能存储变得无聊：Google Cloud 如何打造全球最快的 Lustre 文件系统 - 李明
https://zhuanlan.zhihu.com/p/2059739227819184886
ChatGPT: https://chatgpt.com/c/6a73c107-b1fc-83ec-ac4c-10a3f1196e8d
1. Good. The deep dive reveals why the 29-age old Lustre regain popular in the AI storage age
2. Highlights
1. Lustre has become one of the leading storage choices for large AI training clusters
1. not the universal storage system for all AI data.
2. Goal change in AI storage
1. Old, before AI age, store data cheaply and conveniently
operational simplicity;
multi-tenancy;
inexpensive capacity;
cloud-native APIs;
Kubernetes integration;
unified block, file, and object access;
geographic durability.
2. New, Large-scale AI training, never let thousands of extremely expensive GPUs wait for data
1. Lustre was originally designed to solve: feeding a tightly coupled supercomputer
2. Dissecting AI storage
1. “AI storage” actually contains several very different workloads
Raw datasets
Training samples
Temporary preprocessing output
Training checkpoints
Final model weights
Experiment metadata
Inference KV cache
Vector indexes
Logs and user data
2. Other storage systems in AI scenarios
1. Long-term source of truth - cloud object storage
2. Intensive training & checkpointing - Parallel FS, Lustre
3. KV cache storage - Node local NVM + SSD cache pool + cloud storage
1. LMCache, NVIDIA Dynamo KVBM, Mooncake Store
4. Cold tier - cloud object storage
5. Inference model hosting - Node local NVM + object storage registry
3. How Lustre works
1. File is statically stripped and stored in OSTs
1. q = file offset / chunk size
OST number = q mod OSD count
2. My questions
1. static allocation, no consistent hashing?
2. Metadata is managed in MDTs, hosting paths and folders
1. each inode has a home MDT, folder/path mapping is specified at inode creation. the selection is basically least-capacity or by a placement policy, or explicitly specified by an admin.
2. There is no dynamic tree partitioning like Ceph.
3. Unlike Ceph who persists MDS data into OSD, Lustre's MDT manages data itself, it is a target anyway
4. How client knows the initial path->MDT mapping?
1. MGS hosts MDT index -> MDT server addresses
3. File is locked at the internal extent level, data access lock is managed in OST who hosts that data, metadata access lock is manged in the MDT that hosts the metadata
1. client has lock cache to speedup. But OST lock is still the sole authority. Event the client side lock passes, the OST server can still reject the session.
2. The namespace has a fixed starting anchor: MDT0000 -> "/"
4. Lustre does not manage replication, replication is managed in the under layer of OST, RAID/ZFS plus OST/MDT failover
1. A key limitation
5. Index organization inside MDT
1. directory -> children: ext4's HTree
1. key is hashed. B-tree hosts directory block. inside block it is linear
2. inode table: inode number -> inode record
1. it is a linear array, grouped, and uses inode bitmap
3. FID -> inode: object index
1. an extension of the ldiskfs HTree directory code
4. File layout in MDT inode - Layout Extended Attribute,
1. Mapping file offset to OST. It is an ordered array.
6. Index organization inside OST
1. FID → local backend object inode: object index
1. an extension of the ldiskfs HTree directory code
2. object offset → physical block: extent tree
1. from ext4, it is B-tree like.
7. ldiskfs
1. Lustre often uses a modified ext4 filesystem as the local persistence engine underneath every MDT and OST, called ldiskfs
2. This is why in the index part, you see Lustre keeps borrowing ext4 hash-tree and ext4 extent tree
8. ZFS
1. Every Lustre MDT and OST needs a local persistent storage engine. The two major supported choices are ldiskfs and ZFS
4. Google managed Lustre
1. Lustre is infamous for its
1) complex config and deployment
2) not managing replication and durability
Google Cloud right solves both
2. Lustre provides POSIX FS API, which is the key need to customers
1. Lustre regains popularity in AI storage age because:
1. mature POSIX compatibility
known failure behavior
direct parallel I/O
RDMA
large existing HPC ecosystem
vendor and cloud support
acceptable performance on modern NVMe
3. The competing alternative, DAOS, isn't necessary, because Google Persistent Disks solves what Lustre misses, while DAOS's becomes unnecessary.
And DAOS is new but not as stable enough
4. Other competing alternative AI filesystems
1. FalconFS
2. WEKA NeuralMesh
3. VAST Data
4. JuiceFS
5. 3FS
5. Other competing alternative AI stores
1. NVIDIA AIStore
2. Alluxio
n. My questions
1. Lustre seems to be using the old generation technology. The persistence is provided by Google Persistent Disk. From this sense, Lustre is just a "cache" that provides filesystem interface and makes checkpoint writing parallel and fast and Lustre is not even optimized for native flash/NVMe.
So, why not let's build a new storage atop cloud storage, just NVM cache POSIX FS?
n. Related materials
1. The Lustre Book: The Lustre Storage Architecture
https://arxiv.org/pdf/1903.01955
2. The Lustre whitepaper: Lustre: A Scalable, High-Performance File System
https://cse.buffalo.edu/faculty/tkosar/cse710/papers/lustre-whitepaper.pdf
3. Google Cloud: Storage best practices for AI/ML workloads on TPU VMs
https://docs.cloud.google.com/tpu/docs/storage-best-practices
About storage services for AI and ML workloads
https://docs.cloud.google.com/ai-hypercomputer/docs/storage?hl=en
1. Training is recommending Google "Cloud Storage Rapid Bucket" over Lustre
Lustre is however best recommended for saving model checkpoints
5. FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline —— 论文阅读 - 妙BOOK言
https://zhuanlan.zhihu.com/p/1932023003895542265
ChatGPT: https://chatgpt.com/
1. Paper: FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline [2026, 1 refs, NSDI 26, IPADS, Huawei]
https://www.usenix.org/conference/nsdi26/presentation/xu
2. Highlights
1. Problems to solve
1. training workload access many files in random access. Unlike traditional workload, there is no locality.
2. Key techniques
1. Instead of client side path resolution, do it at the server side. Server uses custom metadata objects which is smaller sized. And there are less number of servers so cache is more concentrated in one server thus higher cache hit rate.
1. Interesting. Smart, simple, but effective optimization
2. index and partitioning of the directory tree
1. hash + selective redirection
2. full replication top layers
3. merge requests
6. 解读CIDR'25论文《Bullion》：为机器学习量身定做的列式存储 [2024, 16 refs]
https://zhuanlan.zhihu.com/p/1965080536386086325
1. Paper: Bullion: A Column Store for Machine Learning
https://vldb.org/cidrdb/papers/2025/p26-liao.pdf
2. Bullion is a great new Parquet replacement. But, the paper has many immature parts.
Bullion identifies real problems. Parquet/ORC really do have structural mismatches with modern ML training data
chatgpt: https://chatgpt.com/c/6a746865-7a78-83ec-b1c7-2bf03161cd52
3. Highlights
1. 现有列式存储在机器学习场景下的不足之处
1. 严苛的数据合规性要求 (Deletion Compliance)
1. 在列式存储中，一行数据（比如一个用户的所有信息）被拆散存储在成百上千个独立的列文件中。要删除这一行，理论上需要重写所有包含该行数据的文件。对于动辄PB级别的数据表，这种操作的I/O开销是无法接受的。
2. 另一种常见的做法是使用“删除向量”（Deletion Vector），即在一个附加文件中标记某一行已被删除。读取数据时，系统会过滤掉这些行。这种方法避免了重写文件，但数据本身并未被物理删除，只是逻辑上不可见了，这在严格的合规要求下是行不通的。
2. 长序列稀疏特征与超宽表 (Sparse Features and Wide Tables)
1. 稀疏特征：推荐模型严重依赖稀疏特征，例如用户的历史点击广告ID序列。在Parquet中，这通常被存储为 list<int64> 类型。正如论文中的 Table 1 所示，一个生产环境的广告数据表中，仅 list<int64> 类型的列就多达16,256个。
2. 当数据按照用户ID和时间戳排序后，同一个用户在相邻时刻的行为序列通常非常相似，呈现出一种“滑动窗口”模式。比如，新的行为序列可能只是在旧序列的头部增加了一个ID，同时在尾部移除了一个ID。现有的通用编码方案无法有效利用这种模式来极致地压缩数据。
3. 超宽表：为了进行A/B测试和模型迭代，特征表会变得非常宽，动辄包含数万个列。然而，一次模型训练任务通常只需要读取其中的一小部分特征（比如10%）。对于Parquet这类格式，它的元数据（Metadata）记录了文件中所有列的信息。当表变得极宽时，仅仅是读取和反序列化这个庞大的元数据文件，其耗时就可能与读取实际数据所需的时间相当，造成严重的性能瓶颈。
3. 大模型（LLM）带来的新挑战
1. 多模态数据存储：LLM的训练需要处理文本、图片、音频、视频等多种类型的数据。将视频这类非结构化的大对象直接塞进列式存储效率很低。通常的做法是，将元数据（如文本、图片URL）存储在列存中，而将视频文件存储在对象存储中。这样做的问题在于，训练时需要根据元数据去对象存储中拉取对应的视频文件，导致大量的随机I/O，严重影响训练吞吐量。
2. 存储量化 (Storage Quantization)：现代AI硬件（如GPU）为了追求极致的计算速度，越来越多地使用低精度浮点数（如FP16, BF16）或整数进行计算。然而，大多数存储系统原生支持的是高精度的FP32或FP64。这意味着数据从存储中读出后，需要经过一次从FP32到FP16的转换才能送入GPU，这不仅浪费了存储空间（FP32是FP16的两倍大
），还增加了数据预处理的CPU开销。
2. Bullion 的核心设计与技术创新
1. File data structure
1. See chatgpt: https://chatgpt.com/c/6a746865-7a78-83ec-b1c7-2bf03161cd52
2. It is still organized as row groups like Parquet. Except in binary footer Bullion uses FlatBuffers like representation, while Parquet uses Thrift metadata tree
1. Parquet's problem: duplicate column metadata per row group. So move them to footer to dedup
2. deletion-compliant column storage
1. in-place page modification + use deletion vector to preserve row position
2. RLE encoding is tricky
3. also need to maintain checksum due to in-place page modification
n. my questions
1. Object storage incompatibility: S3-style object storage does not normally support arbitrary byte-range overwrites.
2. Atomicity and crash consistency: A deletion may touch thousands of column pages, the deletion vector, page hashes, row-group hashes, and the root checksum.
3. sliding-window encoding for sparse feature
1. Default parquet RLE is for scalar values. Bullion proposes RLE for vector lists
4. wide-table metadata
1. At 10,000 columns, the reported metadata time for extracting one feature was
Parquet: 52 ms
Bullion: 1.2 ms
2. The gain comes from replacing a flexible, self-describing, recursively deserialized metadata representation with a compact fixed binary structure.
n. My questions
1. The 52ms from Parquet isn't slow considering a file provides bulk operations
5. storage quantization
1. Bullion argues that quantization should happen before data reaches the training job
2. model quantization usually optimizes model weights or activations, whereas Bullion applies the same thinking to persisted training features and embeddings.
6. multimodal organization
1. Tradition Training first selects rows from the meta table and then performs separate media lookups, producing fragmented I/O
2. Bullion proposes placing selected, reduced-resolution key frames directly into the columnar record
7. cascading encodings
n. Related works
1. Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads [2018, 12 refs, VLDB18]
https://www.vldb.org/pvldb/vol18/p4964-song.pdf
1. Interesting compare with Bullion. ByteDance overhauled the entire system
2. Highlights
1. Krypton format: Magnus's answer to extremely wide Parquet
2. Instead of copy data: training data + search copy + vector copy, use three indexes on the same data
3. Git-like branching or features and models
4. Merge-on-read, with transaction and serialization (partially)
2. Lance：针对机器学习的新兴文件格式中最具影响力的新贵 - 胡津铭
https://zhuanlan.zhihu.com/p/1966901915553100807
1. Good. This has industry implementation, real users, and looks more mature then Bullion
1. LanceDB
https://www.reddit.com/r/dataengineering/comments/1rbcjqu/lance_vs_parquet/
2. the F3 paper: F3: The Open-Source Data File Format for the Future [2025, 18 refs]
https://db.cs.cmu.edu/papers/2025/zeng-sigmod2025.pdf
2. Highlights
1. Lance最早的设计动机, flagging parquet's problems
1. 支持高效的随机访问（random access）
2. 对于图像、视频、音频和向量嵌入（embedding）的一等公民的待遇
3. 对于二级索引的支持
3. 对于schema的演进
1. Decoupled Column Pages
1. Lance 则完全抛弃了“行组”的概念。它将每一列视为一个独立的数据流。它的切分单位是物理大小固定的“页” (Page)
写入数据时，系统会沿着某一列持续写入
对于不同数据类型的列，一个页内包含的行数是不同的
2. 代价：读取逻辑变得相对复杂。当查询 SELECT col_A, col_B WHERE row_id = 500 时，引擎需要分别计算第500行在 A 列的哪个页、哪个位置，以及在 B 列的哪个页、哪个位置。这需要一个额外的元数据映射层来处理。
1. 这里要提一下Parquet为什么要用行组这样的设计 - This is the good part
1. 我的理解是一方面行组可以去维护每个行组的zonemap（每一列的最小值、最大值等信息），做谓词下推更方便，另一方面，行组天然地把一个大文件切成了一系列的块，做读取与计算的并行更方便。
2. 那么Lance的这种设计是怎么解决这两个问题的呢？实际上，Lance的zonemap并不是做到页（page）这一层，因为不同的列的page无法对齐，Lance的zonemap仍然是针对固定行数的。因此，可以认为，Lance对于固定行数（例如1M行）去维护一个zonemap，而这1M行，Lance会拆成一个一个小的8MB的页来存储。至于并行，页本身就可以作为并行的单位。
2. 页面内部布局：根据数据本身的特点做优化 - 在一个8MB的页内部
1. Mini-Block 布局：主要用于整数、小字符串等尺寸较小或可变的数据
2. Full Zip 布局：专为向量等尺寸较大的定长数据设
3. Blob布局：这种布局用来存储更加大的数据，例如图片、视频等
3. 数据更新与版本管理：安全的“追加-提交”模型
1. 写入新文件：将新列的数据写入一个全新的 .lance 文件。旧的数据文件完全不被触碰。
创建新清单 (Manifest)：创建一个新的元数据清单文件，这个文件会同时引用旧的数据文件（包含旧列）和新的数据文件（包含新列）。
原子性更新指针：最后，原子性地修改一个 _latest.manifest 文件，将其指向刚刚创建的新清单。
7. Introducing System One Models - Jev
https://typesafe.ai/blog/introducing-system-one-models-and-jev
1. Highlights
1. Outputs - Type-safe structured values.
2. Reinforcement Learning for Calibrated Decisions (RLCD)
3. Cost and speed
1. 10x to 100x cheaper
4. Demo
1. Jev can play Doom game
5. RLHF vs RLVR vs RLCD
1. RLHF（基于人类反馈的强化学习）
- 全称：Reinforcement Learning from Human Feedback
- 优化目标：优化“人类评审偏好的文字”，让模型学会“好好说话、讨好人类、符合人类价值观”。
- 奖励机制：稠密且相对的（Reward Model 对任意两个回答进行打分比较，给出相对高低）。
- 典型场景：自由对话、创意写作、通用AI助手（如 ChatGPT、Claude 对话端）。
- 潜在问题：容易产生“讨好型AI”或 Reward Hacking（看起来对，但事实上有逻辑漏洞或幻觉）。
2. RLVR（基于可验证奖励的强化学习）
- 全称：Reinforcement Learning with Verifiable Rewards
- 优化目标：优化“可被程序验证的输出”，确保逻辑和计算结果客观正确。
- 奖励机制：稀疏且绝对的（结果非对即错，要么为 1，要么为 0）。
- 典型场景：数学推理、代码编写、逻辑解题（代表模型为 DeepSeek 推理系列等）。
- 潜在问题：强依赖于客观规则或编译器验证，无法直接评估主观语气或复杂社会常识。
3. RLCD（面向校准决策的强化学习）
- 全称：Reinforcement Learning for Calibrated Decisions
- 优化目标：优化“校准的决策与认识论上诚实的概率”。模型在给出选择或评分的同时，必须附带一个数学上严格校准的置信度（例如模型宣称有 80% 的把握，长期统计下该判断确实有约 80% 是正确的）。
- 奖励机制：基于严格适当评分规则（Strictly Proper Scoring Rules），迫使概率输出真实反映不确定性。
- 典型场景：机器原生（Machine-native）决策、自动化工作流、后台条件分支判断、不需要生成长篇大论文本的 System 1 结构化任务。
- 背景与代表：由前 OpenAI 研究员 Diogo Almeida 创立的 TypeSafe AI 及其发布的 Jev 模型 首次将该范式带入大众视野。
4. RLCD 的核心目标就是：让模型输出的置信概率（Stated Confidence）在统计上完全吻合其真实的准确率（Actual Accuracy）
1. RLCD 改变了这一逻辑，它的训练分为两部分：
1. 强制的架构约束（Schema-Conditioned Output）：RLCD 训练的模型（如 TypeSafe 的 Jev）通常不生成长篇大论的文本，而是被限制在严格的格式（如 JSON、选择、评分）中，直接输出一个概率分布向量 \(q\)。
2. 引入“严格对等评分规则”作为强化学习奖励：这是 RLCD 能训练置信概率的数学秘密。
5. Reddit: How is RLCD (jev) RL?
https://www.reddit.com/r/MachineLearning/comments/1wk6iei/how_is_rlcd_jev_rl_d/
0. Very good
1. We know that despite treating this as a "new kind of foundation model" it is in reality an open source LLM with a few extra nn.Modules to nail their output shape and some post-training to reinforce the task.
2. We know that they trained it 100% with synthetic data generated by bigger LLMs.
3. We know based on their benchmarks that the LLM that they fine-tuned is not very large.
4. We also know that their benchmarks are all generated by Astra/Fable and "success" means "agreement with Astra/Fable" which happen to be in the distillation lineage of our model...oops!
5. We know based on their team and open positions that their main focus is hosting and productization, not ML research.
6. We know that they took $40m of monopoly money and are under intense pressure to bring something to market.
7. They talk about "two years of research" like it establishes substance, and I can't contradict that someone had some kind of idea of something two years ago, but realistically, a problem of this magnitude doesn't need two years of research to execute.
8. As for RLCD, it's likely just conventional SFT with a dash of RL and they invented 'RLCD' as a marketing term--just like RLHF.
9. Lots of puffery here and I don't believe they have much of a moat. Hopefully it works out for them and their investors.
10. It's very obviously an organized media push. That's why you're hearing it from all angles at once. 100% chance there's a PR form orchestrating this.
11.
6. RLHF+RLVR=？普林斯顿 RLMT 框架把数学推理的 「思考术」 搬进日常聊天 - tomsheep
https://zhuanlan.zhihu.com/p/1958279086897074259
1. OK .. it is RLMT not RLCD
2. my questions
1. Jev API allows to output a choice between options. But does it tell me what options I have? Like what I commonly use ChatGPT for.
1. No. This is a fundamental limitation
2. What is the model structure of Jev? What is the training materials of Jev? How cost effective is Jev? How reliable are the results of Jev?
1. TypeSafe has not disclosed Jev's actual architecture, parameter count, pretraining corpus, or exact RLCD algorithm.
2. Cost: Compared with frontier LLM models, TypeSafe claims to be 444.6x cheaper, 193.6x faster, 70-500ms end-to-end latency.
3. TypeSafe didn't publish RLCD algorithm. But OpenJev-RLCD paper gives the internal
1. OpenJev-RLCD: A Working RLCD Implementation [2026, 0 refs]
https://arxiv.org/abs/2609.38850
https://github.com/ZimmyGao/openjev-rlcd
1. The core idea - Reward the entire probability distribution. RLVR's problem is is rewarding a single output.
2. Techniques
1. Brier score
Chatgpt: https://chatgpt.com/c/6ac4663b-7054-83ec-bcb9-122d30062451
3. How does RLVR and RLCD work? How exactly RLCD calibration for the correct probability output?
1. TypeSafe didn't publish RLCD algorithm. But OpenJev-RLCD paper gives the internal
8. 苹果的 AirTag 技术原理是什么？ - 柴健翌
https://www.zhihu.com/question/456603466/answer/1986453517
1. Good explanation
9. 为什么说大模型训练很难？ - bigpunch
https://www.zhihu.com/question/498271491/answer/1981120245616906843
1. Good insider talk
10. Claude Code产品负责人：Anthropic的产品团队为何行动速度远超其他团队丨Lenny's Podcast
https://mp.weixin.qq.com/s/462YIl5peql-9XMC-LU5DQ
1. Good and interesting thoughts about how to reorganize how we work today.
2. Highlights
1. Dev 跳过 PM 做到每周甚至每天发布 feature 给客户并拿到反馈
2. 代码越便宜，人的“品味”越重要
11. OpenAI 等 5 大巨头发布 MRC 协议，将如何重塑 AI 训练网络？
https://www.zhihu.com/question/2035703353049657557/answer/2035839566360356290
1. It shares the idea of many modern AI networking design
2. Highlights
1. Multi-pathing + error path detection and handling
2. Path load balancing
这种多平面拓扑、喷射、负载均衡和修剪等技术的结合
3. 有了 MRC，动态路由的必要性就降低了 ... 禁用动态路由
12. 现在招聘Oracle DBA的越来越少了，以后Oracle会不会完全被取代？ - 艾米粒
https://www.zhihu.com/question/40156914/answer/2044706910360822466
1. Good experiences
2. Highlights
1. 国有大行：保守策略，稳扎稳打
第一家是某国有大行的省级分行，他们的策略用一个词形容就是”保守”。
他们从决定迁移到真正上线，前后花了整整18个月。其中前6个月什么都没干，就是做POC验证，跑了不下100个核心业务场景。光是兼容性测试报告就写了300多页，每个SQL语句都要对比执行计划和性能差异。
迁移方式也是最稳妥的”双轨并行”：GoldenDB和Oracle同时跑了3个月，每天对账，发现不一致就停掉排查。那段时间我们实施团队每天加班到凌晨，就是为了确保万无一失。
结果： 虽然周期长、成本高，但上线后确实没出什么大问题。现在运行两年多，核心系统稳定性达到99.99%。但代价是迁移成本是预算的2.5倍，时间是预期的3倍。
13. 【SOSP’25 论文解读】Sandman：让睁一只眼睡觉的存储——把 NVMe 用户态栈的功耗砍掉 39%
1. Sleeping with One Eye Open: Fast, Sustainable Storage with Sandman [2025, 1 refs, SOSP25]
https://zhuanlan.zhihu.com/p/2043397264656314796
2. Highlights
1. This is an interesting topic: Sandman 的核心 insight: 用户态 SSD 栈需要一个自适应的睡眠/唤醒调度器
2. Approach
Sandman 给现代用户态 SSD 栈 (SPDK 风格) 增加了一个轻量调度框架, 让 CPU 在 idle 时进入一个浅睡眠 (shallow sleep) 状态：
浅睡眠: 不进入深层 CPU C-state, 不释放 cache line ownership, 保持 register pinning;
快速唤醒: 通过轻量中断 (MSI-X) + spinning hint 在 1-2μs 内重新进入轮询模式;
预测式入睡: 基于历史 I/O 到达率和 queue depth 决定入睡时机;
混合模式: 在高负载时完全不睡, 退化为标准 busy-poll。
14. 如何看待Spacex新发布的AI卫星？ - OwlLite
https://www.zhihu.com/question/2047773618428437482/answer/2048034738611528030
1. Good example of project analysis using the first principle.
We can do a good financial analysis even on a project that isn't fully in the IT area
"从目前的物理参数和行业数据，已经足够为SpaceX的AI1 卫星构建财务模型了"
15. 比SkipList更高效的Columnar MemTable的设计与实现 - 胡津铭
https://zhuanlan.zhihu.com/p/1952405750363162542
1. 胡津铭 与 @无敌大饺子 设计并开源了一个全新的高性能内存 KV 存储引擎：Columnar MemTable, 是传统 Skip-List 实现的 3.5倍
2. Highlights
1. Columnar MemTable 的高性能基石可以总结为一句话：将写入热点（Hot Path）与后台处理（Cold Path）彻底分离。
2. Columnar MemTable本质上就是一个内存中的LSM-Tree的设计。
3. 局限性
1. 当前的设计并不适合直接作为 RocksDB 的 MemTable 插件。RocksDB 的核心操作之一是需要提供一个能够遍历整个 MemTable 所有数据的有序迭代器（Iterator）
2. 目前的 ColumnarMemTable 专门为以下场景设计
1. 极高的写入吞吐量和并发点查：例如，实时指标监控、用户行为日志记录等写多读少的场景。
2. OLAP 引擎的内存缓冲层: 它天然的列式存储格式 (ColumnarBlock) 使其非常适合作为 OLAP 数据库（如 ClickHouse）的内存数据暂存区。
16. 我为什么放着存储 leader 不做，去当了售前架构师 - 架构师修仙之路
https://mp.weixin.qq.com/s/-cnP0aTtAODPaI6Nv_3_SQ
1. Good industry and savvy experiences
17. KV 存储技术综述：从 KV Cache 到分布式 Attention Store - 夜.月
https://zhuanlan.zhihu.com/p/2082615179595073265
1. Very good article. It walked through all major works with in-depth drill and good future prediction insights.
2. Highlights
1. KV cache designs
1. PagedAttention（vLLM）：把 OS 分页做进 kernel
2. RadixAttention（SGLang）：基数树前缀复用
3. vAttention：把分页做进 MMU - 用 CUDA Virtual Memory Management API
1. PagedAttention 把分页做进 kernel，vAttention 把分页做进 MMU
2. 架构演进对 KV 存储的冲击：MLA 与 KDA
1. MLA（DeepSeek）- TPLA
2. KDA（Kimi K3）打破 per-token 假设
1. 实测 60K 上下文下 KDA 混合 cache 比纯 MLA 省 70%
2. 架构冲击：per-token 线性增长 -> 固定递归状态 + 稀疏 checkpoint
1. Good, paradigm change again
3. Attention Store 系统
1. 单卡 HBM 装不下 KV Cache 时，就需要把它扩展到 CPU DRAM、SSD 和分布式存储——这类系统统称 Attention Store。核心思想是“以存换算”：用更多的存储资源换取更少的计算。
2. 存储层次谱系
1. 层级 介质 典型容量（单节点） 典型带宽
L1 GPU HBM 80–96 GB 2–3.3 TB/s
L2 CPU DRAM 0.5–2 TB 50 GB/s
L3 NVMe SSD 4–100 TB 7 GB/s（单盘）~25 GB/s（RAID）
L4 分布式池 / 远端对象存储 PB 级 10–100 Gbps 网络带宽
3. 带宽落差是数量级的（NVMe→HBM 差约 480×），因此所有系统都在做同一件事：用预取、重叠、粒度设计把搬运藏进计算。
1. 重叠隐藏：逐层 load/store 与计算重叠（Mooncake/LMCache/CachedAttention/LayerKV）、多 CUDA stream 并行、预取窗口；
2. 未来信息 > 历史信息：用调度器队列的前瞻而非 LRU 的历史做放置与驱逐，命中率差距是数量级的（86% vs 48–58%）；
3. 粒度选择：传输粒度要大（聚合到 chunk 才能饱和带宽），管理粒度要细（页/层/会话各取所需）。
4. CachedAttention / AttentionStore：会话级 KV 外存的奠基
5. Mooncake：KVCache-Centric 分离架构的生产标杆
6. LMCache：引擎中立的 KV 存储传输层
7. HiCache：分层基数树（SGLang/阿里云 Tair）
8. 端到盘与直通存储：KVSwap 与 Phoenix
4. 命中率经济学：Attention Store 的商业闭环
1. 浴盆模型
1. 命中部分定价仅为未命中的 10%–25%（无问芯穹，服务 GLM/Kimi）。
2. 由此推出浴盆模型：TPS_hit 随命中率单调上升，TPS_miss 近似线性下降；命中率 40%→70% 时未命中吞吐再降 20%，而命中部分提升不足 3 倍——中段（”偶然失效区间”）可能命中率越高越亏钱；
3. 且存在 20%/40%/70% 奇异点（末尾残块不满 chunk 独立计算）。
4. 优化方向：把浴盆变浅、加大 cache 容量逃离盆底、cache-aware 负载均衡。
2. 命中率天花板由 trace 决定，容量只是入场券（kvcache.ai 模拟器逆向）
1. Mooncake 真实 trace 的复用天花板（无限容量）仅 59.1%——Belady 最优在 16GiB 就到顶（有效复用工作集极小，仅 ~3.7% 唯一块有未来复用价值），LRU 却要 256GiB 才追平（分不清”将复用”与”死块”）。
2. trace 的复用结构决定上限，模型结构只决定多少 GiB 换多少块。 Agent 负载改变命中率来源：传统对话输入 8K、命中率 50%；Agent 任务输入 64–128K、输出 ≤1K、命中率 90%。但前缀易变字段破坏命中：Agent ID、工作目录、时间戳、工具 schema 每次微变即 miss。
3. TokenPilot（浙大）证明仅把易变字段替换成静态占位符 + 摄入边界确定性整理，就把宏观命中率 38.7%→79.2%、费用降 61–87%——上下文管理与 KV 存储是同一枚硬币的两面。
5. 总结
1. KV 存储技术沿两条主线演进，且正加速融合：
1. 引擎内管理（PagedAttention → RadixAttention → vAttention → KDA 状态缓存）：从分页分配到前缀复用到虚拟内存下沉。分页+前缀树已是标准范式；架构演进（MLA/KDA）正在把缓存对象从”per-token 张量”变为”per-prefix 状态”，前缀缓存被重新定义。
2. 跨介质存储系统（CachedAttention → Mooncake → LMCache → HiCache → Phoenix）：从单机两级到分布式全局池。“以存换算”已成行业共识；决定性工程问题是粒度（16MB 才饱和 400Gbps）、重叠（加载快于计算调度才生效）与寻址（页哈希 vs token 序列）。
2. 一个正在成形的判断：KV Cache 是 Agent 时代的第一等数据。Agent 负载（输入 64–128K、输出 ≤1K、命中率 >90%、轮间隔秒级）天然是 KV 存储的最佳场景，其经济学（命中定价 10–25%、浴盆模型、trace 决定天花板）已经反过来塑造系统设计——命中率的竞争前移到了上下文管理（TokenPilot）与 KV-aware 路由（kv_events/Dynamo）。未来的 KV 存储不再是孤立模块，而是横跨推理引擎、存储系统、网络传输与硬件介质的全栈协同设计。
18. PostgreSQL 的 2026 与下一个十年
https://zhuanlan.zhihu.com/p/2080352181098754373
1. Good article, many insights with industry experiences
2. PostgreSQL 未来十年
1. PostgreSQL 能在内核或近内核生态里更好地支持多种存储形态，比如 zheap、列存、时序存储、Append-only 存储、Temporal 存储，甚至 Index-organized Table
2. 逻辑复制：从"能复制"走向"更像系统能力"
3. 可分支数据库：数据也要像代码一样被管理
1. AI Agent 和新一代开发工具带来的变化
4. Postgres everywhere：PG 不一定只跑在服务端
1. PGlite 这类东西真正打开的，是另一类场景：本地开发、CI 测试、浏览器 IDE、Agent 沙箱、离线应用、边缘计算、前端近端查询。
5. 自动 DBA 会是一个很大的方向
1. 哪个事务正在拖住 vacuum；
哪个 JSONB 查询应该建 GIN；
哪个索引长期没用可以考虑清理；
哪类 SQL 应该单独设置 statement_timeout；
哪张表的 autovacuum 参数不合理；
哪个连接池配置正在把数据库顶爆；
哪次版本升级风险最大，应该先验什么。
6. PostgreSQL + Lakehouse：谁能把事务数据自然带到分析侧，谁就有牌
1. Databricks 买 Neon，Snowflake 买 Crunchy Data，AWS 收 DuckLabs，这几件事放在一起看，背后有一条暗线：事务数据、分析数据、AI 数据不想再各玩各的。
业务系统里的数据，刚写进 PostgreSQL，分析侧就想看；AI 应用刚产生状态，评估、审计、特征、画像就想接着用。
7. PG 兼容：数据库世界里的流量入口
2. Neon 便是一个很典型的例子 - Databricks 2025 年宣布收购 Neon
1. Neon 不是简单托管一个 PostgreSQL，它把 PostgreSQL 的计算和存储拆开，引入可分支、可回溯、可弹性伸缩的存储架构。对开发者来说，这意味着一个数据库可以像代码一样创建分支、回滚、隔离测试环境；对 AI Agent 来说，这意味着它可以快速创建自己的数据库环境，不需要每次都等一个传统数据库实例慢慢拉起来。
2. 但是 AI Agent 和新一代开发工具带来的变化是：数据库可能变成一种短生命周期资源。写一个 demo 要一个库，跑一个测试要一个库，验证一次 schema migration 要一个库，一个 Agent fork 一份状态也要一个库。用完就扔，错了就回滚，互相隔离，最好还不用人工干预。
1. Neon 最值钱的地方，是把数据库从一个笨重实例，变成了可以被程序快速创建、复制、回滚和丢弃的状态资源。
19. 直播/点播技术全景：从数据流到架构演进 - 架构师修仙之路
https://mp.weixin.qq.com/s/6_fzrUCLipWQ1K1EtJqQKg
1. Good summary of the architecture change history of 直播
2. Highlights
1. Generation 2 - 多CDN + HTTP-FLV 时代
2. Generation 3 - 快直播不是简单的协议替换，而是整个传输模型的变革——从TCP+缓冲变为UDP+实时反馈，延迟从秒级降到毫秒级。
引入WebRTC和类WebRTC协议实现毫秒级延迟。各云厂商推出"快直播"产品——阿里云RTS（Real-Time Streaming）、腾讯云快直播、声网Agora。连麦场景使用WebRTC实现主播间实时互动。转码开始使用GPU硬件加速。
3. Generation 4 - 边缘智能 + AI增强时代
大模型和AI技术爆发，边缘计算能力持续提升，视频编码标准迭代（H.265/AV1普及），用户对画质和体验的要求不断提高。
架构新特征：
1. AI超分辨率：在边缘节点使用AI模型将720p实时增强到1080p甚至4K，降低源端带宽消耗。
2. 智能转码：AI驱动的感知编码（Per-Title Encoding），根据画面复杂度动态调整码率，相同画质下节省30%-50%带宽。
3. 边缘转码：将转码任务从中心节点下沉到边缘节点，减少回源延迟，实现就近转码就近分发。
4. Serverless流处理：云端无服务器架构按需弹性处理转码和分发，无需预留资源。
5. 多CDN智能编排：基于AI的实时流量调度，综合考虑延迟、成本、画质体验做最优决策。
代表厂商：Akamai（边缘计算+AI）、Cloudflare Stream、阿里云视频云、Mux、Bitmovin。
总结：新一代架构的核心理念是'算力下沉、智能调度'——把转码和AI增强推到离用户最近的边缘，用算力换带宽，用智能换成本。
20. AI Storage compare and study
Chatgpt: https://chatgpt.com/c/6ae18e24-a8b4-83ec-af0e-448153f557bd
1. Nebius Storage - A first-party distributed filesystem and performance-tiered object store
https://nebius.com/storage
1. Combines its own high-performance shared filesystem, S3-compatible object storage with multiple performance classes, network block storage, and third-party systems such as WEKA and VAST. Its primary strength is offering multiple storage interfaces optimized for different stages of AI training and inference.
2. Highlights
1. Nebius published filesystem interface in S3 compatible object buckets
2. CoreWeave Storage - Distributed VAST filesystem plus a GPU-node cache over object storage
https://www.coreweave.com/products/storage
1. Combines partner-powered distributed filesystems with a newer AI-native object storage service. Its Local Object Transport Accelerator (LOTA) is particularly interesting: it caches objects directly on GPU nodes, aiming to provide near-local read throughput while keeping a unified object dataset.
2. Highlights
1. LOTA acceleration technology (GPU-adjacent caching)
1. S3 objects are cached in GPU/CPU nearby NVMe cache. Can be cached in a different node. 3TiB NVMe cache per node by default. Distributed cache placement. LRU eviction.
2. Cross-region data mobility
1. The high-level technique is to acknowledge writes after making the data durable locally and synchronizing metadata with the target region, then relocate object bytes to the destination in the background
1. No recovery guarantee in region failure declared
3. Google Rapid Storage - Exposing Colossus's stateful protocol through object storage
https://cloud.google.com/blog/products/storage-data-transfer/high-performance-storage-innovations-for-ai-hpc
1. Google has extended its object storage architecture with Rapid Bucket, based on Colossus, and Rapid Cache, an SSD-backed zonal read cache. Unlike a traditional object store, Rapid Bucket supports persistent streaming connections, native appends, and very high operation rates.
2. Highlights
1. The major innovation: appendable objects
1. Traditional object flow: Produce data -> Complete upload -> Publish object -> Read object
2. Rapid Bucket
Producer opens object
|
Append chunk 1 ------> Readers see committed data
Append chunk 2 ------> Readers see additional data
Append chunk 3 ------> Readers continue reading
|
Finalize object
3. Rapid Cache is a different solution
1. Rapid Cache is not Rapid Bucket's write cache. It is an SSD-backed zonal read cache in front of existing Cloud Storage buckets.
2. Google caches in 2 MB increments, automatically manages cache capacity, and preserves the source bucket's role as the durable storage layer.
4. Google describes an optimized RDMA-like internal protocol, writes to multiple disks in parallel, and quorum-based durable append operations within Colossus. Rapid Bucket builds on this with gRPC streaming
5. Persistent streaming connection
1. The key innovation is turning many independent, stateless object operations into repeated operations on an already-open, stateful object stream.
2. This is conceptually closer to opening a file and issuing multiple pread() calls than repeatedly requesting byte ranges through a conventional S3 API.
6. Why based on Colossus?
1. Regular Google Cloud Storage already uses Colossus
Google describes Colossus as its foundational distributed storage system, used by Cloud Storage, BigQuery, Spanner, and many other products
1. Colossus under the hood: a peek into Google’s scalable storage system
https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system
2. Colossus under the hood: How we deliver SSD performance at HDD prices - L4
https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance
2. Google also documents Colossus's L4 distributed SSD cache, which transparently identifies and caches hot data. So even regular Cloud Storage can benefit from SSD acceleration without being entirely stored on SSDs.
21. Weight-Space Programming: LLMs can natively read & write neural network weights
https://rikiyat.github.io/blog/weight-space-programming/
1. Very good. This is paradigm shift breakthrough
2. Highlights
1. GPT-6 Astra can tell and natively alter model weights to improve/specialize its performance
2. The current stage is very early results
22. 商业存储列传 · 第 08 篇 — XtremIO，30亿美元神话死于内战 - 架构师修仙之路
https://mp.weixin.qq.com/s/pZ6CSWShST2zJOO5f6ETGA
1. Good
2. Highlights
1. 内容寻址（content-based addressing）最后变成的致命的架构缺陷，难以支持更多更全面的功能
23. 商业存储列传 · 第 07 篇 — Pure Storage，从破发到唯一赢家 - 架构师修仙之路
1. Good
2. Highlights
1. DirectFlash - 绕开 SSD FTL。
“时间一长，DirectFlash 的意义从省钱变成了容量密度和产品节奏的自主权。单模块容量从早期的几 TB 一路做到 2023 年预告的 75TB，2024 到 2025 年间做到 150TB 级别（具体时间为约数，官方未精确披露）。闪存供应上，Pure 与 Kioxia、Micron、SK hynix 先后签了供应链合作（2024 年 12 月至 2025 年 5 月间），直接对接 NAND 原厂。这项技术后来成了它向云厂商授权的核心资产，第五节细说。”
2. “用高密度模块把闪存塞进容量型市场。这一步也是后来 AI 行情的伏笔。大模型训练的主存储，要的首先是容量和带宽，其次才是 IOPS。Meta RSC （AI Research SuperCluster）那单 175PB，正是容量型需求。”
24. 商业存储列传 · 第 09 篇 — Violin，从8.6亿市值到2300万的三年自由落体 - 架构师修仙之路
1. Highlights
1. 4 亿美元风投，必须承诺估值和风投退出时间表
2. HP 单一渠道。最大的分销商同时是最直接的潜在竞争者
Create an Issue or comment below