cactus-compute / needle

项目摘要

Needle is a highly optimized foundation model designed for edge devices, ranging from 8-29 MB in size while delivering competitive performance on tool calling and structured extraction. It uses a novel Laddered Simple Attention Network architecture with engram n-gram memory and supports 2-bit quantization, enabling deployment on phones, wearables, robots, and microcontrollers.

为什么值得关注

Needle achieves remarkable efficiency by beating models 10x its size on mobile tool calls while maintaining a tiny footprint, making on-device AI automation feasible for resource-constrained environments. Its laddered architecture allows every depth from 2 to 20 layers to be a deployable model, providing unique flexibility for different compute budgets.

优势

• Exceptional model efficiency with 8-29 MB size and 2-bit quantization support
• Novel architecture (Laddered Simple Attention Network with engram memory) optimized for tool calling
• Strong benchmark performance beating much larger models on mobile tool calls
• Calibrated confidence scores and grammar-constrained decoding for reliable outputs
• Active development with 27 contributors and 12k+ stars

局限性

• No official releases in the last 6 months despite active development
• Trades general chat capacity for specialized tool calling and extraction tasks
• Limited examples and no Docker support for easier onboarding
• Requires understanding of specific architecture and schema compilation

使用场景

• Mobile and wearable app automation with local tool calling
• IoT and smart home device control without cloud dependency
• Structured data extraction from text on edge devices
• Local text embedding and semantic search for privacy-sensitive applications
• Robotics and automotive systems requiring offline AI capabilities