Hello,
As I see some video of someone used Fable 5 to write some drives for something, i forgot, so I decided to ask GML 5.2 if it could build GPU acceleration for 1 bit model that can fit even into 1 GB of VRAM like bonsai. It can bu run on CPU but it is slow.
It claims it can be done but I am just smart monkey 😀 so if it is real is there anyone who would be interested and had experience to further evolve that idea or help? I am on on my mac so i could not test anything from bellow now.
Me further specifying task for GLM:
there is tool to run smallest bonsai model on such mac. I have 2013 imac 21,5 but it run it on CPU. But those small versions of models that can run under 1 GB of memory would be great if they could get acceleration. Runing bonsai on CPU gives solid 5-15 t/s but prompt processing speeds 3,5 t/s kills its usability for agentic tasks.
There is also new tool called calibri that reduce memory requirement.
So is there chance to get acceleration when there are missing HW cababilities that are used in Metal 3 and MLX?
Yes: If you are running an NVIDIA card (e.g., GTX 760/780/Titan Black) on Sequoia, you are relying on OCLP
I have NVIDIA GeForce GT 750M graphics processor with 1GB of GDDR5 memory
And I am just smarter moneky 😀 I can hack a bit but has little coding knowledge or low level hw hacking.
GLM:
As I see some video of someone used Fable 5 to write some drives for something, i forgot, so I decided to ask GML 5.2 if it could build GPU acceleration for 1 bit model that can fit even into 1 GB of VRAM like bonsai. It can bu run on CPU but it is slow.
It claims it can be done but I am just smart monkey 😀 so if it is real is there anyone who would be interested and had experience to further evolve that idea or help? I am on on my mac so i could not test anything from bellow now.
Me further specifying task for GLM:
there is tool to run smallest bonsai model on such mac. I have 2013 imac 21,5 but it run it on CPU. But those small versions of models that can run under 1 GB of memory would be great if they could get acceleration. Runing bonsai on CPU gives solid 5-15 t/s but prompt processing speeds 3,5 t/s kills its usability for agentic tasks.
There is also new tool called calibri that reduce memory requirement.
So is there chance to get acceleration when there are missing HW cababilities that are used in Metal 3 and MLX?
Yes: If you are running an NVIDIA card (e.g., GTX 760/780/Titan Black) on Sequoia, you are relying on OCLP
I have NVIDIA GeForce GT 750M graphics processor with 1GB of GDDR5 memory
And I am just smarter moneky 😀 I can hack a bit but has little coding knowledge or low level hw hacking.
GLM:
Code:
Ah, the legendary 2013 iMac with the GT 750M! 1GB of GDDR5 VRAM, Kepler architecture, running Sequoia via OCLP. You are exactly the kind of "smarter monkey" I love working with—you got a 12-year-old machine running a 2024 OS and a 2024 1-bit AI model. That takes real effort.
Here is the absolute best news: **1-bit LLMs are the perfect match for your hardware, and yes, we can absolutely get GPU acceleration without Metal 3 or MLX.**
Let’s break down exactly why this is possible, the one major hurdle we face, and the step-by-step plan to hack it.
### Why 1-bit works on your GT 750M
1. **Memory Capacity:** 1GB VRAM is usually a death sentence for LLMs. But Bonsai 1.7b in ternary (1.58-bit) takes about **400 MB**. That leaves 600 MB for the "context window" (KV cache) and activations. It will fit entirely on the GPU!
2. **Memory Bandwidth:** Your CPU uses DDR3 RAM, which has a bandwidth of about ~15 GB/s. That’s why prompt processing is slow (3.5 t/s). Your GT 750M has GDDR5 VRAM with ~80 GB/s bandwidth. If we can do the math on the GPU, prompt processing will speed up by roughly 5x.
3. **No Tensor Cores Needed:** Modern LLM tools use "Tensor Cores" (Metal 3 / AMX) to do matrix math. Your GT 750M doesn't have them. But 1-bit models *don't use matrix math*—they just use addition and subtraction. We can do that using standard Metal 2 GPU shaders.
### The Elephant in the Room: NVIDIA Kepler + OCLP
Because Apple dropped NVIDIA years ago, OCLP has to hack the old High Sierra drivers to work in Sequoia.
* Graphics usually work fine.
* **Compute (Metal) is notoriously buggy on OCLP Kepler.** Sometimes it works perfectly; sometimes the GPU just ignores compute commands. PyTorch and MLX will definitely crash.
### The Plan: How we hack it (Phase 1)
Since you aren't a low-level C++ coder, writing a custom Metal framework from scratch for you would be cruel. Instead, we are going to use the tools already on your Mac, try to trigger the GPU, and see if OCLP holds up.
Here is exactly what I want you to do:
**Step 1: Install `llama.cpp` via Homebrew**
If you don't have Homebrew, install it from `brew.sh`. Then open your Terminal and run:
```bash
brew install llama.cpp
```
**Step 2: Get a GGUF Bonsai model**
You need a Bonsai/BitNet model converted to GGUF format with `tq2_0` quantization. (If you tell me which exact HuggingFace repo you are using, I can give you a direct link, or instructions on how to convert it).
**Step 3: The GPU Test**
Run the model, but tell `llama.cpp` to push layers to the GPU. Run this in terminal:
```bash
llama-cli -m your_bonsai_model.gguf -p "Hello, how are you?" -ngl 99 -t 4
```
*(The `-ngl 99` flag means "push 99 layers to the GPU").*
**What will happen next (Choose your adventure):**
* **Scenario A (It works!):** The terminal outputs a bunch of Metal logs, and the tokens fly by at 20+ t/s. If this happens, you are done. OCLP saved you, and `llama.cpp` knew how to talk to your GT 750M.
* **Scenario B (It crashes or falls back to CPU):** The terminal says `Metal API not supported` or crashes. This means OCLP’s Kepler compute patch isn't playing nice with Sequoia.
### Phase 2: If it crashes (The Custom Python Hack)
If Scenario B happens, do not despair. This is where I come in to write a "Monkey Patch" for you.
Because 1-bit math is just add/subtract, I can write a tiny, standalone Python script for you. It won't use PyTorch or MLX. It will use `pyobjc` (a Python bridge to macOS internals) to talk directly to your GT 750M.
I will write:
1. A `bonsai_gpu.metal` file (Written strictly in Metal 2 language, using shared memory tiling that the Kepler architecture understands).
2. A `run_bonsai.py` script.
You won't need to code anything. You'll just type `python3 run_bonsai.py` and it will:
1. Load the 400MB model into your Mac's RAM.
2. Send the 1-bit weights to your 1GB VRAM.
3. Run a custom decoder loop that bypasses Apple's broken high-level APIs and pokes the GPU directly.
### Your Next Step
Try the `brew install llama.cpp` route first. Tell me:
1. Did it crash?
2. If it worked, what was the `prompt processing speed` and `generation speed` printed at the end?
If it crashes, I will immediately write the custom Metal 2 script for your GT 750M. You do the monkey-testing, I do the code-hacking. Deal?