add setup guide

setup.md

@@ -0,0 +1,63 @@
+# Running Local LLMs
+I wanted to revisit locally hosting LLMs. Since the last time I tried, there have been drastic improvements in both the models and the quality of software surrounding them. I also wanted to document the process of setting it up - Dad is interested in doing the same thing, so it would be great to have information he can look at. 
+
+My specs as of writing this guide:
+* AMD Ryzen 7 9800 X3D CPU
+* Nvidia RTX 5070 Ti GPU (16GB VRAM)
+* 64 GB DDR5 RAM
+
+## Tech Stack
+There are a few necessary components to a Local LLM stack. There's more technical information later, but at a very high level:
+1. **Large Language Model** (LLM) - Qwen3.5-27B & Qwen3-Coder-30B-A3B-Instruct
+   A neural network trained to recognize and make predictions based on patterns in text. 
+
+2. **Inference engine** - Ollama
+   The software that's loading our model and performing computations to generate text. 
+
+3. **Chat UI** - OpenWebUI
+   A user interface for talking to an LLM. Many inference engines have a built-in CLI-based UI, but separate software is usually preferable for multiple sessions, conversation history, and so forth. 
+
+4. **Agent harness** (only for agentic operations/coding) - OpenCode
+   Software designed to give an LLM the ability to act autonomously, typically for tasks like coding. Often provides "tools" that allow reading and writing files or running shell commands. 
+
+## Setup
+1. Install Ollama: https://ollama.com/download
+   ```
+   curl -fsSL https://ollama.com/install.sh | sh
+   ```
+   By default, Ollama runs as a `systemctl` service. If it doesn't start automatically, start it with `ollama serve` or `sudo systemctl start ollama`. 
+
+2. Use Ollama to download an LLM: `ollama pull qwen3.8:27B` and/or `ollama pull qwen3-coder:latest`. Once finished, test by running your model name with ollama: `ollama run qwen3.8:27B`. 
+
+3. By default, Ollama has a very small model context window (the number of tokens the model can "remember" at one time), which means longer inputs or responses can lead to the model getting cut off. Ollama allows us to make modifications to the models we just downloaded using Modelfiles - see `./Ollama/`. The modelfile dictates which model to use as a base, and then what things to modify about it; those two modelfiles only modify the `num_ctx` (context window length) parameter. To create models with them, run `ollama create <model_name_an_arbitrary_string> --file <path_to_modelfile>`. Do this for each of the models you downloaded. 
+
+4. Install OpenWebUI as a Docker container: https://docs.openwebui.com/getting-started/quick-start?setup-method=docker-compose
+   I added an environment variable `WEBUI_AUTH=False` to the `docker-compose.yml` file to prevent the need for registering with an email address. 
+   In the same directory as `docker-compose.yml`, run `docker compose up -d`. Then, visit [localhost:3000](http://localhost:3000/). It might have local Ollama models already set up already (TODO: verify). 
+
+5. I also changed some of OpenWebUI's default settings since they annoyed me. I think this is all of them, but I might have forgotten some. 
+    * turn off default prompt suggestions on the front page: Settings -> AI -> Models -> Model Defaults (Configure) -> Prompt Suggestions (Configure) -> remove all that exist 
+    * turn off follow-up question generation: Settings -> Experience -> Interface -> turn off Follow Up Generation (near the bottom)
+    * turn off AI autocomplete: Settings -> Experience -> Interface -> turn off Autocomplete Generation (near the bottom)
+
+6. Set up OpenCode (optional, for agentic coding): https://github.com/anomalyco/opencode#installation
+   ```
+   curl -fsSL https://opencode.ai/install | bash
+   ```
+   To add models, fill out `~/.config/opencode/opencode.jsonc` according to the template on [the OpenCode website](https://opencode.ai/docs/providers/#ollama). Change the `models` object to reflect your currently installed models. For me:
+   ```
+   "models": {
+     "qwen3-coder-modified:latest": {
+       "name": "qwen3-coder-modified:latest"
+     }
+   }
+   ```
+
+## Model Selection
+Obviously, model selection is crucial here - some models will be better than others depending on the architecture, number of parameters, desired use-cases, and whether they can even run on your hardware. 
+
+I want models to be good at a few main things. These don't have to all be fulfilled by the same model. 
+1. Coding - both agentic and chat-based. Agentic is less important for my personal use, since I still like writing code myself. 
+2. General knowledge - I mostly use LLMs for cooking (science, not recipes), but other domains would be helpful too. 
+    * One other thing to look into is Retrieval-Augmented Generation (RAG). Modern big tech models sometimes use web searches to retrieve up-to-date information, but I might want to, say, point it to my library of cooking books. 
+3. Multi-language - I'd like to learn some more Spanish, and having a personal tutor will be great for an intro level.