PrismML-Eng / Bonsai-demo
Runs a 27B ternary-weighted reasoning model at 5.9 GB locally with vision, tool calling, and 262K context — near full-precision quality on consumer hardware
星标趋势
AI 分析
项目摘要
Bonsai-demo is a local inference demo for Bonsai 2 27B, a ternary-weighted (1.75 bits/weight) reasoning LLM that retains 98.2% of FP16 intelligence at ~5.9 GB. It provides shell scripts to run the model via a PrismML-forked llama.cpp or MLX backend, offering chat, vision, and agentic tool calling through a local OpenAI-compatible server.
为什么值得关注
Demonstrates extreme quantization (ternary weights at 1.75 bits/weight) preserving near-full-precision reasoning quality on a 27B model, making high-capability inference feasible on consumer hardware. The hybrid-attention backbone enables a 262K-token context window on-device, and native MCP server integration with OpenAI-style tool calling is built in.
优势
- Extreme ternary quantization achieving 98.2% FP16 parity at ~1/9th model size
- Multi-backend support (llama.cpp fork + MLX) across Metal, CUDA, Vulkan, ROCm, and CPU
- Integrated vision, agentic tool calling with MCP, and configurable reasoning effort
- Clear two-command quick start with well-documented generation parameters
局限性
- Requires a custom PrismML fork of llama.cpp — incompatible with stock llama.cpp
- No Docker support or formal releases in the last 6 months
- Demo-focused rather than a general-purpose serving framework; tightly coupled to Bonsai models
使用场景
- Running a high-capability 27B reasoning model locally on consumer hardware
- Local AI assistant with vision, tool calling, and long-context support
- Experimenting with extreme quantization techniques for LLM deployment