DIRTY STATIC / DISPATCH Tutorials21 NOV 2026 • 11 MIN READ
A laptop on a concrete desk in a brutalist concrete room, a ring of model weight blocks beside it, caution yellow hazard stripes on the floor and one yellow glowing line on the terminal screen, black and white with caution yellow as the only colour.
TutorialsINTELLIGENCE FOR THE MACHINE AGE

Install Inkling 2 Mini With llama.cpp GGUF

Thinking Machines released Inkling 2 Mini on November 20, 2026, as open weights under Apache 2.0, and the ggml org published GGUF builds for llama.cpp on day one. This tutorial installs a current llama.cpp build, picks between the IQ4_NL and Q8_0 files, serves the model on your own machine, and verifies the local server before you trust it with real work.

Start reading ↓

A 26.4 billion parameter model that fits on a laptop sounds like a trick, and in a sense it is one. Inkling 2 Mini stores 26.4 billion parameters, but its sparse design activates only a slice of them for each token, and its middle layers run three times to add reasoning depth without adding stored weights. Thinking Machines says the full checkpoint takes about 20 GB. The GGUF route trims that further, and it runs through llama.cpp, the local inference engine most open weight tools are built on.

This tutorial uses the GGUF files published by the ggml org, the team behind llama.cpp itself. That choice matters. Support for looped models in local runtimes is new, the model card says so plainly, and the safest place to get a build of a brand new architecture is from the people who maintain the runtime.

What You Will Build or Learn

You will end with Inkling 2 Mini running on your own machine in two forms. First, a local server with an OpenAI compatible API and a browser chat page, ready for your own tools to call. Second, the knowledge of which GGUF file you actually need, because picking the wrong one is the fastest way to run out of memory on a machine that should have been fine.

Inkling 2 Mini accepts text and image input and produces text. It has 24 layers, 128 experts with 8 active per token, and a context window of up to 1 million tokens on paper. That last number is a model limit, not a local promise. On a laptop you will set a far smaller context, and Step 3 explains why that is the right call.

Header image from the official Inkling 2 Mini model card on Hugging Face

Before You Begin

Two warnings belong up front, before any download starts.

The model card also carries a candid limit worth knowing before you build plans on this model. Thinking Machines says Inkling 2 Mini can fall into long reasoning loops, above all when it debugs code, and recommends limits on output tokens and agent turns. It also says the model may take destructive actions in agent setups without confirmation. Run it as a chat and coding assistant first. Give it tools and file access later, in a sandbox, with confirmation switched on. Our tutorial on giving a local model real tools with Ollama and MCP covers that leash work in detail.

Step 1: Install llama.cpp

Action: install llama.cpp with the official installer for your system. On macOS and Linux, run this in a terminal.

Terminal

curl -LsSf https://llama.app/install.sh | sh

On Windows, run this in PowerShell, or install the package with winget install llama.cpp.

Terminal

irm https://llama.app/install.ps1 | iex

Homebrew users on macOS or Linux can use brew install llama.cpp instead. All of these routes install the same project. What matters is that the build is current, because the Inkling 2 architecture support is recent.

Expected result: the installer finishes and the llama command answers. Check it before you download 17 GB of model.

llama --help

The help text lists the subcommands, among them llama cli for terminal chat and llama serve for the server. If the shell says the command is missing, open a fresh terminal window so the updated path loads, and run the check again.

Step 2: Pick Your GGUF File

Action: choose one row from the official GGUF table before you run anything. The ggml org publishes three sizes for this model.

FileSize on diskWhat the publisher says
IQ4_NL17.6 GBCalibrated with an importance matrix; 4 bit experts in layers 0 to 19, everything else at Q8_0
Q8_027.8 GBNear lossless; all weight matrices at Q8_0
BF1652.2 GB16 bit precision; the original experts upcast to BF16

The publisher measured both smaller files against BF16 on the same test text. Q8_0 picked the same top token 97.6 percent of the time. IQ4_NL picked it 95.5 percent of the time. That gap is real but modest, and the disk gap is 10 GB.

Expected result: a clear choice. Pick IQ4_NL if your machine sits near the 32 GB memory target, which is most laptops that qualify at all. Pick Q8_0 if you have memory to spare and want the closer match. Skip BF16 for local use; it exists as a reference, not as a laptop file. Remember that memory use is the file size plus the context cache plus working buffers, so a 27.8 GB file on a 32 GB machine leaves very little room to breathe.

Step 3: Serve the Model

Action: start the server with the file you picked. This command uses IQ4_NL. Swap the suffix to Q8_0 if you chose the larger file.

Terminal

llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL --port 8080

The -hf flag tells llama.cpp to download the model from Hugging Face and cache it on your machine, so the first start includes the full download. The --port 8080 flag sets the address on purpose. Recent llama.cpp builds have changed their default port, and setting it yourself means your tools and this tutorial agree on where the server lives.

Keep the context modest at first. The model supports up to 1 million tokens, but every token of context costs memory on your machine, and the looped layers already spend extra compute per token. If you want a set size from the start, add -c 8192 to the command and raise it later only if your work needs it and your memory allows it.

Expected result: after the download and load, the terminal reports the server listening on 127.0.0.1 at port 8080. Leave that terminal open. Closing it stops the server.

Step 4: Chat in the Terminal Instead

Action: if you only want to talk to the model, skip the server and use the chat command in a fresh terminal. It downloads the same cached file.

Terminal

llama cli -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL

Expected result: a chat prompt opens once the model loads. Ask it something you can check, such as asking it to explain a short piece of your own code or to summarise a paragraph you paste in. A correct summary of text you supplied is a better first test than a trivia question, because this model is weaker on factual recall than its larger sibling, by its own card. Type your exit command as shown in the chat help when you are done.

Step 5: Add Image Input Only If You Need It

Action: skip this step for text work. For image input, the GGUF repo also publishes a vision encoder file, called mmproj, at 642 MB in BF16 or 346 MB in Q8_0. Download the mmproj file from the same repo and pass it when you start the server.

Terminal

llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL --port 8080 --mmproj /path/to/mmproj-file.gguf

Replace the example path with the real path to the file you downloaded. The model card says each image dimension should sit between 40 and 4096 pixels for best results.

Expected result: the server starts with the vision file loaded alongside the model, and the browser chat page accepts image input. If the server fails at this step and names the mmproj file, the path is wrong or the download is incomplete. Text chat keeps working without this file, so remove the flag and confirm the base setup first.

Step 6: Set a Reasoning Limit

Action: keep the model from thinking in circles. The GGUF notes say reasoning depth is set with reasoning_effort, with values of none, minimal, low, medium, high, xhigh and max, plus numbers from 0.0 to 0.99. High is the default. For quick local questions, set a lower value in your client request, and cap the output length when your client allows it.

Expected result: shorter, faster answers on simple tasks, with less chance of the long reasoning loops the model card warns about. Raise the setting for hard coding work, where the extra passes earn their time. This model scored 48.2 percent on DeepSWE v1.1 in the lab tests published on its card, ahead of every open weight model under 100 billion parameters in that table, so the reasoning is worth spending on work that needs it and worth trimming on work that does not.

Verification

Do not trust the terminal banner alone. Check the server from a second terminal while it runs.

curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models

The health call should report the server as ready, and the models call should list the Inkling 2 Mini GGUF you started. Then open http://127.0.0.1:8080/ in your browser and send one message in the built in chat page. A reply there proves the full loop: download, load, server, and generation, all on your machine.

For an API check, send one chat request with curl. Use the model name exactly as the models endpoint returned it.

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "MODEL_NAME_FROM_MODELS_ENDPOINT", "messages": [{"role": "user", "content": "Name three primary colours."}]}'

A JSON reply with your answer inside it means your own tools can now call this endpoint the same way they would call a hosted API, with no account and no per token bill. That is the whole point of the build, the same local first pattern we used in the Hermes Agent setup.

Troubleshooting

ProblemLikely causeFix
Unknown architecture or the model refuses to loadThe llama.cpp build predates Inkling 2 supportUpdate llama.cpp with the Step 1 installer, then run the same serve command again
Out of memory during loadQ8_0 plus context cache leaves too little room on a 32 GB machineSwitch the suffix to IQ4_NL and add -c 8192 to keep the context small
The download restarts or stallsThe connection dropped during a 17 GB or larger downloadRun the same command again; llama.cpp resumes from its cache where it can, and a free Hugging Face account can make large downloads more stable
The browser page does not openThe server is on a different port, or the serve terminal was closedCheck the serve terminal for the listening address, keep it open, and confirm you used --port 8080
Answers ramble or loop on simple questionsReasoning effort is at the high default for a task that does not need itLower reasoning_effort as in Step 6 and cap the output length in your client
Image input is missing in the chat pageThe server started without the vision fileRestart with the --mmproj flag and a correct path, as in Step 5
The server answers slowlyThe context is set far above what the work needs, on a looped model that spends extra compute per tokenLower the context with -c, close other heavy apps, and keep expectations matched to laptop hardware

Next Steps

Once the server answers, point one tool you already use at the local endpoint. Most OpenAI compatible clients accept a custom base address, and http://127.0.0.1:8080/v1 is the address to give them. Start with low stakes work, such as summaries and first drafts of code, and keep a hosted model nearby for factual questions, where this model is at its weakest.

From there, the honest comparison is worth doing. Run the same three prompts through IQ4_NL and, if your memory allows, through Q8_0, and see whether the larger file changes answers you care about. The publisher numbers say the gap is small. Your own work is the test that counts, and now it costs you nothing but time and electricity to run it. For background on how this runtime reached its current shape, see our report on llama.cpp reaching 1.0.

Commands, file sizes and model facts checked against the ggml org GGUF repo, the official Inkling 2 Mini model card and the Thinking Machines release post, November 21, 2026. This procedure was verified against those documents and was not run end to end on local hardware for this article.

Sources

  • ggml org, Inkling 2 Mini GGUF repo on Hugging Face. The serve command, the IQ4_NL, Q8_0 and BF16 file sizes, the mmproj vision files, the token match figures and the reasoning effort setting.
  • Thinking Machines Lab, Inkling 2 Mini model card on Hugging Face. Hardware targets, architecture, context limit, release date, Apache 2.0 licence, limitations and safety notes.
  • Thinking Machines Lab, Introducing Inkling 2 Mini, November 20, 2026. Looping design, benchmark claims and the laptop flight story, all presented as lab claims.
  • llama.cpp project, llama.app. Installer commands for macOS, Linux and Windows, and the unified llama cli and llama serve commands.
  • Image credit: the body image is the header image from the official Inkling 2 Mini model card on Hugging Face, by Thinking Machines Lab. The hero image is a Dirty Static house style illustration.
■

END OF TRANSMISSION

Stay curious. Question everything.

CONTINUE THE DESCENT

MORE FROM DIRTY STATIC ↘