Header image: Tomás Saraceno – new connectome (working title) by Fulvio Spada, CC BY-SA 2.0, via flickr via Openverse — cropped to 16:9 and colour-adjusted.
Key takeaways
- Ternary Bonsai 2 27B fits a 27B model in 5.9GB using ternary quantization
- Retains 98.2% of Qwen3.8 27B performance across benchmarks
- Enables local 27B model inference on consumer GPUs and edge devices
PrismML just released Ternary Bonsai 2 27B. That’s a 27-billion-parameter model. It fits in 5.9GB. Not 59GB. Not 15GB with some aggressive quantization tricks. Five. Point. Nine. Gigabytes. That’s smaller than some vacation photos on your phone. Yet it still hits 98.2% of Qwen3.8 27B’s full-precision performance on benchmarks. If that doesn’t make you pause, you haven’t been watching the compression wars closely enough.
The Compression Paradox: Smaller Usually Means Dumber
Compression has always been a trade-off. Always. You shrink the model, you lose something. FP16 to 8-bit? Maybe a 2-3% drop on benchmarks. 8-bit to 4-bit? Closer to 5-7%. And if you push further—2-bit, ternary, binary—you’re looking at double-digit declines. The rule was simple: more compression, more capability lost.
Most compressed models hover around 90-95% of the original performance. That’s acceptable for some use cases. But it’s not near-lossless. It’s a compromise. And compromises add up. A 5% drop on MMLU might not sound dramatic, but it translates to weaker reasoning, flakier outputs, and more edge cases where the model fails outright. That’s why most high-performance models still run in full precision—or at least 8-bit—when accuracy matters.
PrismML’s claim? Ternary Bonsai 2 27B retains 98.2% of aggregate benchmark performance. That’s a leap.
Inside Ternary Weights: {-1, 0, +1} and the 1.76-Bit Miracle
So how did they pull this off? The secret isn’t just quantization. It’s ternary quantization. Most low-bit models use uniform quantization—4-bit, 2-bit—where weights are rounded to the nearest representable value in a fixed range. Ternary weights restrict values to just three options: -1, 0, and +1.
At first glance, that sounds restrictive. Three values? Really? But here’s the trick: ternary weights are sparse. That 0 isn’t just another value. It’s a signal to skip computation entirely. Fewer multiply-accumulate operations. Faster inference. Lower memory bandwidth.
But three values alone aren’t enough. The real magic is in group-wise scaling. PrismML applies a single FP16 scaling factor to groups of ternary weights. This preserves precision where it matters, avoiding the blunt-force rounding errors that plague uniform quantization.
The result? An effective 1.76 bits per weight. It’s less than 2 bits, thanks to the sparsity from those zeros. For a 27B-parameter model, the math works out:
“ 27 billion weights × 1.76 bits/weight ÷ 8 bits/byte = 5.9GB “
That’s a 9x reduction from the ~50GB FP16 footprint. And crucially, it’s achieved without the performance collapse that usually comes with extreme compression.
The Performance Retention Breakthrough: 98.2% Isn’t Just Marketing
It’s an aggregate across multiple benchmarks:
- MMLU: General knowledge and reasoning.
- GSM8K: Grade-school math problems.
- HumanEval: Coding tasks.
- Vision tasks: Multimodal capabilities, since Qwen3.8 is a vision-language model.
For comparison, the previous Bonsai 27B generation retained 95% of performance.
In qualitative tests, PrismML reports stronger reasoning, coding, and vision capabilities—all while running in a fraction of the memory.
At 98.2%, it’s an unlock.
The Deployment Unlock: Why 5.9GB Changes Everything
Here’s what it enables:
Consumer GPUs
- RTX 4090 (24GB VRAM): Fits comfortably, even with overhead for KV cache and batching.
- RTX 4080 (16GB VRAM): Still works, though you’ll need to watch batch sizes.
- High-end laptops: A 16GB GPU can now run a 27B model locally, without cloud offloading.
Edge Devices
- Jetson Orin: NVIDIA’s edge AI platform tops out at 64GB memory, but most configurations are 16-32GB. 5.9GB is trivial.
- Apple M-series chips: The M3 Max has 128GB unified memory, but even the M1 Pro (16GB) can handle this.
- Raspberry Pi 5 (8GB RAM): Not quite there yet, but give it a year. We’re approaching single-board computer territory.
Energy Efficiency
Memory usage isn’t just about capacity. It’s about bandwidth. A 5.9GB model requires far less data movement than a 50GB one. That translates to:
- Lower power draw (critical for mobile and embedded use).
- Less heat (no more server-room cooling costs).
- Longer battery life (for laptops, drones, robots).
Throughput
Smaller models don’t just fit in more places. They run faster in those places. Reduced memory bandwidth means higher local inference speeds, even on consumer hardware. PrismML hasn’t released latency benchmarks, but the implication is clear: this isn’t just about fitting into memory. It’s about using that memory efficiently.
Cost
- Cloud savings: Smaller instances, less memory, lower bills.
- On-prem savings: Fewer GPUs needed for the same throughput.
- Accessibility: High-performance AI is no longer gated by hardware budgets.
Ternary vs. The Alternatives: How Bonsai 2 27B Stacks Up
Here’s how they compare to other compression methods:
4-bit Quantization (e.g., bitsandbytes)
- Pros: Simple, widely supported, good retention (~95-97%).
- Cons: Still 4x larger than ternary (23.6GB for a 27B model). No sparsity benefits.
2-bit Quantization
- Pros: Smaller footprint (~11.8GB for 27B).
- Cons: Significant performance drops (~90-93% retention). No group-wise scaling, so precision loss is uniform.
Pruning
- Pros: Can remove up to 50% of weights with minimal performance loss.
- Cons: Irregular sparsity is hard to accelerate. Often requires fine-tuning, which ternary avoids.
Binary Weights
- Pros: Extreme compression (~3.4GB for 27B).
- Cons: Severe performance drops (~80-85% retention). Limited expressivity.
The group-wise scaling is key.
The Broader Implications: Compression as the New Scaling Law
It’s about a shift in how we think about AI deployment.
Democratization of AI
27B models are no longer the exclusive domain of cloud providers and data centers. With Ternary Bonsai 2 27B, they’re now accessible on consumer hardware. That changes who can build with AI:
- Startups: No need for massive cloud bills.
- Researchers: No need for institutional GPU clusters.
- Developers: Local fine-tuning and inference, without latency.
Energy Efficiency
AI’s carbon footprint is a growing concern. Training a model like Qwen3.8 27B emits tons of CO₂. But inference? That’s a daily cost. A 9x reduction in memory usage translates to a 9x reduction in data movement, a major driver of energy consumption. For edge devices, this could mean the difference between a model that’s always-on and one that drains the battery in an hour.
Edge AI Revolution
Real-time, high-capability AI is no longer tethered to the cloud. Drones, robots, IoT devices—anything with a GPU can now run a 27B model locally. That unlocks:
- Privacy: No need to send data to a server.
- Latency: No round-trip delays.
- Reliability: No dependency on network connectivity.
Research Direction
If ternary weights can scale this well, what’s next? Could we see:
- Sub-ternary: Even lower bit counts, with more aggressive sparsity?
- Hybrid approaches: Ternary for weights, low-bit for activations?
- Pre-training with ternary: Currently, ternary is applied post-training. Could future models be trained directly with ternary weights?
The Catch: Where Ternary Bonsai 2 27B Still Falls Short
Here’s where Ternary Bonsai 2 27B still has limitations:
Training Constraints
Ternary weights are applied post-training. That means:
- You still need the full-precision model to start with.
- Fine-tuning after quantization isn’t trivial. PrismML hasn’t clarified if this is supported.
- The training process itself doesn’t benefit from the efficiency gains.
Hardware Support
Ternary inference isn’t natively accelerated on most GPUs. That means:
- Early adopters may need custom kernels.
- Throughput gains depend on software optimizations (e.g., sparse tensor cores).
- Some hardware (e.g., older GPUs) may not see the full benefits.
Benchmark Gaps
98.2% is an aggregate. That means some tasks perform worse:
- Long-context reasoning: Compression can hurt attention mechanisms.
- Multimodal fine-tuning: Vision-language models may see larger drops on image-heavy tasks.
- Specialized domains: Niche benchmarks might fall below 98%.
Open Questions
- Scaling: Can ternary work for 100B+ models without performance degradation?
- Generalization: Does 98.2% retention hold across all tasks, or are there edge cases?
- Future-proofing: Will ternary remain competitive as models grow larger and more complex?
The Road Ahead: What’s Next for PrismML and Model Compression
Possible directions:
- Lower bits: Could we see sub-ternary weights (e.g., {-1, 0} or {0, +1})?
- Larger models: Will ternary scale to 50B, 100B, or beyond?
- Hybrid methods: Combining ternary with other compression techniques (e.g., pruning, distillation).
Competitor Response
Other labs will take notice. Expect:
- More ternary experiments: Will Meta, Mistral, or others adopt this approach?
- Alternative methods: Could we see new compression techniques (e.g., learned sparsity, neural architecture search for quantization)?
- Hardware optimizations: GPU vendors may add native support for ternary inference.
Industry Adoption
Early use cases will likely focus on:
- Coding assistants: Local, high-performance code generation.
- Vision models: On-device image and video analysis.
- On-device agents: Personal AI assistants that don’t require cloud sync.
Long-Term Vision
If compression keeps improving at this rate, the next bottleneck won’t be memory. It’ll be imagination. We’re approaching a world where:
- A 100B model fits on a phone.
- AI is truly ubiquitous, not just in data centers.
- The "cloud vs. edge" debate becomes moot.
Why Ternary Bonsai 2 27B Is a Watershed Moment
98.2% retention at 9x smaller isn’t just an improvement.
A 27B model that fits in 5.9GB isn’t just a technical achievement.