Document change
https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf
24
Lines added
7
Lines removed
Was read
2026-09-25
98f4a6c74b5e88a4…
Changed by
2026-10-31
57fa037cfa812c9e…
24 lines added, 7 lines removed
Left column is the version read 2026-09-25, right is 2026-10-31. Unchanged stretches are elided. Both bodies are kept in full and are addressed by the hashes above.
⋯ 40 unchanged lines
4141 - **Retains thinking, reasoning, and agentic behavior** deep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92
4242 - **End-to-end ternary language weights** across embeddings, attention projections, MLP projections, and LM head, at a *true* 1.72 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships as a separate Q8_0 mmproj pack
4343 - **262K-token context** on-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (\~75% linear attention)
4444 - **Two GGUF packings** with custom ternary hybrid-attention kernels for llama.cpp (CUDA, Metal) — **PTQ1_0** packs trits densely (1.75 bits/weight, 5.95 GB), **PQ2_0** stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB); packed weights are consumed directly, never expanded back to FP16
45+- **Runs on stock llama.cpp too**: a `Q2_0` packing (2.25 bits/weight, 7.63 GB) runs in mainline llama.cpp from release b11556 onward, no fork needed
4546 - **MLX companion**: also available as [Ternary-Bonsai-2-27B-mlx-2bit](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) for native Apple Silicon inference
4647
4748 ## Resources
4849
⋯ 32 unchanged lines
8182 | FP16 (baseline) | 16.0 | \~54 GB | 1.0x |
8283 | Ternary g128 (ideal) | 1.72 | 5.8 GB | \~9.3x |
8384 | **GGUF PTQ1_0** (dense trits) | **1.75** | **5.95 GB** | **\~9.0x** |
8485 | **GGUF PQ2_0** (2-bit slots) | **2.13** | **7.21 GB** | **\~7.5x** |
86+| **GGUF Q2_0** (stock llama.cpp) | **2.25** | **7.63 GB** | **\~7.1x** |
8587
86−Practical deployment needs packing formats that efficient kernels can consume, and this repo ships two: **PTQ1_0** packs trits densely and lands essentially on the information-theoretic target, while **PQ2_0** stores each trit in a 2-bit slot, trading footprint for cheaper unpacking. Neither is uniformly faster — see the throughput table below for where each wins. These sizes describe the language…
88+Practical deployment needs packing formats that efficient kernels can consume, and this repo ships two: **PTQ1_0** packs trits densely and lands essentially on the information-theoretic target, while **PQ2_0** stores each trit in a 2-bit slot, trading footprint for cheaper unpacking. Neither is uniformly faster — see the throughput table below for where each wins. A third file, **Q2_0**, uses llam…
8789
8890 ### Shipped Components
8991
9092 The vision tower ships alongside the language model as an optional component (on-disk sizes):
⋯ 1 unchanged line
9294 | Component | Pack | Size | Residency |
9395 | :------------- | :--------------------------------- | -------: | :--------------------------------- |
9496 | Language model | ternary g128 (PTQ1_0) | 5.95 GB | resident |
9597 | Language model | ternary g128 (PQ2_0) | 7.21 GB | resident |
98+| Language model | ternary (Q2_0, stock llama.cpp) | 7.63 GB | resident |
9699 | Vision tower | mmproj (Q8_0) | 0.63 GB | optional — multimodal input only |
97100 | Vision tower | mmproj BF16 (reference) | 0.93 GB | optional |
98101
99102 The Q8_0 file carries the vision tower in an 8-bit container. It is usually offloaded, loaded only when an image actually arrives, so text-only serving never pays for it.
⋯ 32 unchanged lines
132135 > **[PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for
133136 > running these models.** It carries the tested setup for every backend, pins a known-good binary
134137 > release, and is kept current as the kernels move. Where anything here disagrees with it, it is right.
135138
136−### These files need our llama.cpp build
139+### Which llama.cpp runs which file
137140
138−The ternary hybrid-attention kernels live in the
139−[PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) fork. **Stock llama.cpp will not
140−run these files.** It rejects `PQ2_0` and `PTQ1_0` as unknown types, and it loads `Q2_0` without any
141−warning and produces garbage, because it has no Hadamard activation runtime. Use a binary from the
142−fork.
141+`PQ2_0` and `PTQ1_0` need the ternary hybrid-attention kernels in the
142+[PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) fork. Stock llama.cpp rejects them
143+as unknown types. Use a binary from the fork.
143144
145+`Q2_0` runs on stock llama.cpp from release
146+[b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) onward, which added the Hadamard
147+activation runtime these weights need. **Older stock builds load `Q2_0` without any warning and
148+produce garbage**, so check your build first (`llama-cli --version`). Apps that bundle their own
149+llama.cpp need an engine built from b11556 or newer.
150+
144151 ```bash
145152 # prebuilt, pick the archive for your platform
146153 # https://github.com/PrismML-Eng/llama.cpp/releases/latest
147154 tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1
⋯ 8 unchanged lines
156163 ```
157164
158165 ```bash
159166 ./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
167+ -ngl 99 -fa on -c 32768 \
168+ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
169+ -p "Explain quantum computing in simple terms." -n 16384
170+```
171+
172+With stock llama.cpp (b11556 or newer), use the `Q2_0` file and the same flags:
173+
174+```bash
175+hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-Q2_0.gguf --local-dir .
176+llama-cli -m Ternary-Bonsai-2-27B-Q2_0.gguf \
160177 -ngl 99 -fa on -c 32768 \
161178 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
162179 -p "Explain quantum computing in simple terms." -n 16384
163180 ```