2026-09 Model Serving Top 10

LLM inference in C/C++

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Speech-to-text, text-to-speech, speaker diarization, speech enhancement, source separation, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, HarmonyOS, Raspberry Pi, RISC-V, RK NPU, Axera NPU, Ascend NPU, x86_64 servers, websocket server/client, support 12 programming languages

An open, programmable decision layer for models and compute.

Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.

A high-throughput and memory-efficient inference and serving engine for LLMs

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Community maintained hardware plugin for vLLM on Huawei Ascend

A retargetable MLIR-based machine learning compiler and runtime toolkit.

A Datacenter Scale Distributed Inference Serving Framework