ashhart / TensorFold

项目摘要

TensorFold is a high-performance LLM inference engine that serves language models on both Apple Silicon (via MLX) and NVIDIA GPUs through an OpenAI-compatible API. It provides per-model custom kernels and draft verification techniques like MTP heads and context copies to optimize decoding speed, supporting multiple model families including Nemotron, Qwen, GLM, Gemma, and DeepSeek.

为什么值得关注

TensorFold stands out for its dual-backend architecture that delivers optimized inference across both Apple Silicon and NVIDIA GPUs from a single codebase, with per-model kernel specialization and integrated draft verification for faster decoding. Its OpenAI-compatible endpoint makes it a drop-in replacement for cloud APIs while leveraging local hardware acceleration.

优势

• Dual backend support for Apple Silicon (MLX) and NVIDIA CUDA from unified codebase
• Per-model custom kernels and draft verification (MTP heads, context copies, DFlash2) for optimized decoding
• OpenAI-compatible API with additional Anthropic Messages API support
• Active development with 12 releases in 6 months and 14 contributors
• Vision/multimodal input support for compatible models

局限性

• No CI pipeline configured despite having tests
• No Docker support for containerized deployment
• No examples directory for quick-start guidance
• Hardware restrictions: requires Apple Silicon or NVIDIA GPUs with compute capability 8.9+

使用场景

• Local LLM serving on Apple Silicon Macs with OpenAI-compatible API
• High-performance NVIDIA GPU inference with custom kernel optimization
• Multi-modal text and image inference on supported models
• Drop-in local replacement for OpenAI/Anthropic APIs in development workflows