83 lines
7.7 KiB
Markdown
83 lines
7.7 KiB
Markdown
# Running Local LLMs
|
|
I wanted to revisit locally hosting LLMs. Since the last time I tried, there have been drastic improvements in both the models and the quality of software surrounding them. I also wanted to document the process of setting it up - Dad is interested in doing the same thing, so it would be great to have information he can look at.
|
|
|
|
My specs as of writing this guide:
|
|
* AMD Ryzen 7 9800 X3D CPU
|
|
* Nvidia RTX 5070 Ti GPU (16GB VRAM)
|
|
* 64 GB DDR5 RAM
|
|
|
|
## Tech Stack
|
|
There are a few necessary components to a Local LLM stack. There's more technical information later, but at a very high level:
|
|
1. **Large Language Model** (LLM) - Qwen3.5-27B & Qwen3-Coder-30B-A3B-Instruct
|
|
A neural network trained to recognize and make predictions based on patterns in text.
|
|
|
|
2. **Inference engine** - Ollama
|
|
The software that's loading our model and performing computations to generate text.
|
|
|
|
3. **Chat UI** - OpenWebUI
|
|
A user interface for talking to an LLM. Many inference engines have a built-in CLI-based UI, but separate software is usually preferable for multiple sessions, conversation history, and so forth.
|
|
|
|
4. **Agent harness** (only for agentic operations/coding) - OpenCode
|
|
Software designed to give an LLM the ability to act autonomously, typically for tasks like coding. Often provides "tools" that allow reading and writing files or running shell commands.
|
|
|
|
## Setup
|
|
1. Install Ollama: https://ollama.com/download
|
|
```
|
|
curl -fsSL https://ollama.com/install.sh | sh
|
|
```
|
|
By default, Ollama runs as a `systemctl` service. If it doesn't start automatically, start it with `ollama serve` or `sudo systemctl start ollama`.
|
|
|
|
2. Use Ollama to download an LLM: `ollama pull qwen3.8:27B` and/or `ollama pull qwen3-coder:latest`. Once finished, test by running your model name with ollama: `ollama run qwen3.8:27B`.
|
|
|
|
3. By default, Ollama has a very small model context window (the number of tokens the model can "remember" at one time), which means longer inputs or responses can lead to the model getting cut off. Ollama allows us to make modifications to the models we just downloaded using Modelfiles - see `./Ollama/`. The modelfile dictates which model to use as a base, and then what things to modify about it; those two modelfiles only modify the `num_ctx` (context window length) parameter. To create models with them, run `ollama create <model_name_an_arbitrary_string> --file <path_to_modelfile>`. Do this for each of the models you downloaded.
|
|
|
|
4. Install OpenWebUI as a Docker container: https://docs.openwebui.com/getting-started/quick-start?setup-method=docker-compose
|
|
I added an environment variable `WEBUI_AUTH=False` to the `docker-compose.yml` file to prevent the need for registering with an email address.
|
|
In the same directory as `docker-compose.yml`, run `docker compose up -d`. Then, visit [localhost:3000](http://localhost:3000/). It might have local Ollama models already set up already (TODO: verify).
|
|
|
|
5. I also changed some of OpenWebUI's default settings since they annoyed me. I think this is all of them, but I might have forgotten some.
|
|
* turn off default prompt suggestions on the front page: Settings -> AI -> Models -> Model Defaults (Configure) -> Prompt Suggestions (Configure) -> remove all that exist
|
|
* turn off follow-up question generation: Settings -> Experience -> Interface -> turn off Follow Up Generation (near the bottom)
|
|
* turn off AI autocomplete: Settings -> Experience -> Interface -> turn off Autocomplete Generation (near the bottom)
|
|
|
|
6. Set up OpenCode (optional, for agentic coding): https://github.com/anomalyco/opencode#installation
|
|
```
|
|
curl -fsSL https://opencode.ai/install | bash
|
|
```
|
|
To add models, fill out `~/.config/opencode/opencode.jsonc` according to the template on [the OpenCode website](https://opencode.ai/docs/providers/#ollama). Change the `models` object to reflect your currently installed models. For me:
|
|
```
|
|
"models": {
|
|
"qwen3-coder-modified:latest": {
|
|
"name": "qwen3-coder-modified:latest"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Stack Selection & Technical Details
|
|
### Model Selection
|
|
Obviously, model selection is crucial here - some models will be better than others depending on the architecture, number of parameters, desired use-cases, and whether they can even run on your hardware.
|
|
|
|
I want models to be good at a few main things. These don't have to all be fulfilled by the same model.
|
|
1. Coding - both agentic and chat-based. Agentic is less important for my personal use, since I still like writing code myself.
|
|
2. General knowledge - I mostly use LLMs for cooking (science, not recipes), but other domains would be helpful too.
|
|
* One other thing to look into is Retrieval-Augmented Generation (RAG). Modern big tech models sometimes use web searches to retrieve up-to-date information, but I might want to, say, point it to my library of cooking books.
|
|
3. Multi-language - I'd like to learn some more Spanish, and having a personal tutor will be great for an intro level.
|
|
|
|
I chose Qwen3-coder primarily for coding, given its name. It scores well on benchmarks and doesn't support chain-of-thought reasoning. I chose Qwen3.8:27b for everything else because it's just about the right size for my hardware and performs well on a wide variety of tasks. The model weights are slightly too large to fit entirely in my 16GB VRAM, but I have enough system RAM (64GB) and a fast enough CPU that I can still get good performance by offloading some of the model to CPU/system RAM.
|
|
|
|
### Inference Engine
|
|
There are three main inference engines that I'm aware of at the time of writing: Ollama, llama.cpp, and vLLM.
|
|
|
|
Ollama is widely regarded as the easiest inference engine to set up; my experience with it has been "curl the install script and go" every time I've set it up. However, this smoothness comes with the cost of having less control over how LLMs are run. It's a good place to start, but for more control I'd choose something else.
|
|
|
|
llama.cpp provides more control over how models are run, quantized, and distributed among resources - it's able to set up custom GPU/CPU offloading ratios and run models with more parameter customization. For limited hardware, I'd like to explore this more in the future.
|
|
|
|
vLLM is optimized for running multiple models simultaneously, usually with multiple GPUs and multuple users. This is probably a little overkill for what we need.
|
|
|
|
### Chat UI
|
|
There are lots of options for a chat UI, and I tried quite a few. One thing that I really dislike is when software treats me like an idiot by abstracting away too much of what's going on under the hood. Many of the desktop chat UIs do that; they're too polished and don't give me any control. For example: AnythingLLM, Jan, and (to some extent) Cherry Studio. Several of them even try to manage their own inference engine stack. Web browser UIs are better about this. I tried both LibreChat and OpenWebUI, finding problems with both of them. LibreChat requires registering with an email address, which I find a little ridiculous. It also requires a lot of editing config files to add and update models - not a big deal for me, but that is an obstacle for someone less technical, and it's more brittle. OpenWebUI had a slightly smoother setup, but still felt over-engineered and had some minor annoyances; they're trying to shoehorn AI prompt suggestions into the UI. I also tried LM Studio a while back, and had a mostly positive experience, from what I remember - I'll probably revisit that again some other time. However, it's not open-source.
|
|
|
|
### Agent Harness
|
|
There are lots of options here: OpenCode, OpenClaw, Qwen Code, Claude Code, etc. I chose OpenCode because I'd heard of it before. This is something to play around with more in the future, if I ever care enough - I don't see myself doing much agentic coding.
|
|
|