A 26.4 billion parameter model that fits on a laptop sounds like a trick, and in a sense it is one. Inkling 2 Mini stores 26.4 billion parameters, but its sparse design activates only a slice of them for each token, and its middle layers run three times to add reasoning depth without adding stored weights. Thinking Machines says the full checkpoint takes about 20 GB. The GGUF route trims that further, and it runs through llama.cpp, the local inference engine most open weight tools are built on.
This tutorial uses the GGUF files published by the ggml org, the team behind llama.cpp itself. That choice matters. Support for looped models in local runtimes is new, the model card says so plainly, and the safest place to get a build of a brand new architecture is from the people who maintain the runtime.
What You Will Build or Learn
You will end with Inkling 2 Mini running on your own machine in two forms. First, a local server with an OpenAI compatible API and a browser chat page, ready for your own tools to call. Second, the knowledge of which GGUF file you actually need, because picking the wrong one is the fastest way to run out of memory on a machine that should have been fine.
Inkling 2 Mini accepts text and image input and produces text. It has 24 layers, 128 experts with 8 active per token, and a context window of up to 1 million tokens on paper. That last number is a model limit, not a local promise. On a laptop you will set a far smaller context, and Step 3 explains why that is the right call.

Before You Begin
Two warnings belong up front, before any download starts.
The model card also carries a candid limit worth knowing before you build plans on this model. Thinking Machines says Inkling 2 Mini can fall into long reasoning loops, above all when it debugs code, and recommends limits on output tokens and agent turns. It also says the model may take destructive actions in agent setups without confirmation. Run it as a chat and coding assistant first. Give it tools and file access later, in a sandbox, with confirmation switched on. Our tutorial on giving a local model real tools with Ollama and MCP covers that leash work in detail.
Step 1: Install llama.cpp
Action: install llama.cpp with the official installer for your system. On macOS and Linux, run this in a terminal.
curl -LsSf https://llama.app/install.sh | sh
On Windows, run this in PowerShell, or install the package with winget install llama.cpp.
irm https://llama.app/install.ps1 | iex
Homebrew users on macOS or Linux can use brew install llama.cpp instead. All of these routes install the same project. What matters is that the build is current, because the Inkling 2 architecture support is recent.
Expected result: the installer finishes and the llama command answers. Check it before you download 17 GB of model.
llama --help
The help text lists the subcommands, among them llama cli for terminal chat and llama serve for the server. If the shell says the command is missing, open a fresh terminal window so the updated path loads, and run the check again.
Step 2: Pick Your GGUF File
Action: choose one row from the official GGUF table before you run anything. The ggml org publishes three sizes for this model.
| File | Size on disk | What the publisher says |
|---|---|---|
| IQ4_NL | 17.6 GB | Calibrated with an importance matrix; 4 bit experts in layers 0 to 19, everything else at Q8_0 |
| Q8_0 | 27.8 GB | Near lossless; all weight matrices at Q8_0 |
| BF16 | 52.2 GB | 16 bit precision; the original experts upcast to BF16 |
The publisher measured both smaller files against BF16 on the same test text. Q8_0 picked the same top token 97.6 percent of the time. IQ4_NL picked it 95.5 percent of the time. That gap is real but modest, and the disk gap is 10 GB.
Expected result: a clear choice. Pick IQ4_NL if your machine sits near the 32 GB memory target, which is most laptops that qualify at all. Pick Q8_0 if you have memory to spare and want the closer match. Skip BF16 for local use; it exists as a reference, not as a laptop file. Remember that memory use is the file size plus the context cache plus working buffers, so a 27.8 GB file on a 32 GB machine leaves very little room to breathe.
Step 3: Serve the Model
Action: start the server with the file you picked. This command uses IQ4_NL. Swap the suffix to Q8_0 if you chose the larger file.
llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL --port 8080
The -hf flag tells llama.cpp to download the model from Hugging Face and cache it on your machine, so the first start includes the full download. The --port 8080 flag sets the address on purpose. Recent llama.cpp builds have changed their default port, and setting it yourself means your tools and this tutorial agree on where the server lives.
Keep the context modest at first. The model supports up to 1 million tokens, but every token of context costs memory on your machine, and the looped layers already spend extra compute per token. If you want a set size from the start, add -c 8192 to the command and raise it later only if your work needs it and your memory allows it.
Expected result: after the download and load, the terminal reports the server listening on 127.0.0.1 at port 8080. Leave that terminal open. Closing it stops the server.
Step 4: Chat in the Terminal Instead
Action: if you only want to talk to the model, skip the server and use the chat command in a fresh terminal. It downloads the same cached file.
llama cli -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL
Expected result: a chat prompt opens once the model loads. Ask it something you can check, such as asking it to explain a short piece of your own code or to summarise a paragraph you paste in. A correct summary of text you supplied is a better first test than a trivia question, because this model is weaker on factual recall than its larger sibling, by its own card. Type your exit command as shown in the chat help when you are done.
Step 5: Add Image Input Only If You Need It
Action: skip this step for text work. For image input, the GGUF repo also publishes a vision encoder file, called mmproj, at 642 MB in BF16 or 346 MB in Q8_0. Download the mmproj file from the same repo and pass it when you start the server.
llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL --port 8080 --mmproj /path/to/mmproj-file.gguf
Replace the example path with the real path to the file you downloaded. The model card says each image dimension should sit between 40 and 4096 pixels for best results.
Expected result: the server starts with the vision file loaded alongside the model, and the browser chat page accepts image input. If the server fails at this step and names the mmproj file, the path is wrong or the download is incomplete. Text chat keeps working without this file, so remove the flag and confirm the base setup first.
Step 6: Set a Reasoning Limit
Action: keep the model from thinking in circles. The GGUF notes say reasoning depth is set with reasoning_effort, with values of none, minimal, low, medium, high, xhigh and max, plus numbers from 0.0 to 0.99. High is the default. For quick local questions, set a lower value in your client request, and cap the output length when your client allows it.
Expected result: shorter, faster answers on simple tasks, with less chance of the long reasoning loops the model card warns about. Raise the setting for hard coding work, where the extra passes earn their time. This model scored 48.2 percent on DeepSWE v1.1 in the lab tests published on its card, ahead of every open weight model under 100 billion parameters in that table, so the reasoning is worth spending on work that needs it and worth trimming on work that does not.
Verification
Do not trust the terminal banner alone. Check the server from a second terminal while it runs.
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models
The health call should report the server as ready, and the models call should list the Inkling 2 Mini GGUF you started. Then open http://127.0.0.1:8080/ in your browser and send one message in the built in chat page. A reply there proves the full loop: download, load, server, and generation, all on your machine.
For an API check, send one chat request with curl. Use the model name exactly as the models endpoint returned it.
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "MODEL_NAME_FROM_MODELS_ENDPOINT", "messages": [{"role": "user", "content": "Name three primary colours."}]}'
A JSON reply with your answer inside it means your own tools can now call this endpoint the same way they would call a hosted API, with no account and no per token bill. That is the whole point of the build, the same local first pattern we used in the Hermes Agent setup.
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| Unknown architecture or the model refuses to load | The llama.cpp build predates Inkling 2 support | Update llama.cpp with the Step 1 installer, then run the same serve command again |
| Out of memory during load | Q8_0 plus context cache leaves too little room on a 32 GB machine | Switch the suffix to IQ4_NL and add -c 8192 to keep the context small |
| The download restarts or stalls | The connection dropped during a 17 GB or larger download | Run the same command again; llama.cpp resumes from its cache where it can, and a free Hugging Face account can make large downloads more stable |
| The browser page does not open | The server is on a different port, or the serve terminal was closed | Check the serve terminal for the listening address, keep it open, and confirm you used --port 8080 |
| Answers ramble or loop on simple questions | Reasoning effort is at the high default for a task that does not need it | Lower reasoning_effort as in Step 6 and cap the output length in your client |
| Image input is missing in the chat page | The server started without the vision file | Restart with the --mmproj flag and a correct path, as in Step 5 |
| The server answers slowly | The context is set far above what the work needs, on a looped model that spends extra compute per token | Lower the context with -c, close other heavy apps, and keep expectations matched to laptop hardware |
Next Steps
Once the server answers, point one tool you already use at the local endpoint. Most OpenAI compatible clients accept a custom base address, and http://127.0.0.1:8080/v1 is the address to give them. Start with low stakes work, such as summaries and first drafts of code, and keep a hosted model nearby for factual questions, where this model is at its weakest.
From there, the honest comparison is worth doing. Run the same three prompts through IQ4_NL and, if your memory allows, through Q8_0, and see whether the larger file changes answers you care about. The publisher numbers say the gap is small. Your own work is the test that counts, and now it costs you nothing but time and electricity to run it. For background on how this runtime reached its current shape, see our report on llama.cpp reaching 1.0.
Commands, file sizes and model facts checked against the ggml org GGUF repo, the official Inkling 2 Mini model card and the Thinking Machines release post, November 21, 2026. This procedure was verified against those documents and was not run end to end on local hardware for this article.
Sources
- ggml org, Inkling 2 Mini GGUF repo on Hugging Face. The serve command, the IQ4_NL, Q8_0 and BF16 file sizes, the mmproj vision files, the token match figures and the reasoning effort setting.
- Thinking Machines Lab, Inkling 2 Mini model card on Hugging Face. Hardware targets, architecture, context limit, release date, Apache 2.0 licence, limitations and safety notes.
- Thinking Machines Lab, Introducing Inkling 2 Mini, November 20, 2026. Looping design, benchmark claims and the laptop flight story, all presented as lab claims.
- llama.cpp project, llama.app. Installer commands for macOS, Linux and Windows, and the unified llama cli and llama serve commands.
- Image credit: the body image is the header image from the official Inkling 2 Mini model card on Hugging Face, by Thinking Machines Lab. The hero image is a Dirty Static house style illustration.
END OF TRANSMISSION
Stay curious. Question everything.


