llm-d / llm-d-router

项目摘要

llm-d Router is a Kubernetes-native inference gateway that provides intelligent, KV-cache and load-aware routing for LLM inference requests. It uses an Endpoint Picker (EPP) that integrates with L7 proxies via the ext-proc protocol to make optimal request placement decisions based on KV-cache locality, current load, and request priority.

为什么值得关注

It introduces KV-cache-aware routing to the data plane via ext-proc protocol integration, addressing a key production challenge in distributed LLM serving. With 276 contributors and 10 releases in 6 months, it represents a significant community effort to build production-grade AI inference infrastructure on top of the Kubernetes Gateway API standard.

优势

• KV-cache and load-aware routing optimizes LLM inference throughput and latency by placing requests near cached KV data
• Flexible deployment model supporting standalone Envoy proxy or cloud-managed L7 load balancers via Kubernetes Gateway API
• Active development with 276 contributors, 10 releases in 6 months, CI, and tests

局限性

• Relatively new project (May 2025) with 201 open issues, still maturing toward production stability
• No Docker support or examples provided, increasing barrier to adoption

使用场景

• Distributed LLM inference routing with KV-cache affinity to minimize recomputation overhead
• Production Kubernetes-based AI inference infrastructure with Gateway API integration
• A/B testing and traffic splitting for model serving via InferenceModelRewrite API