Blog

Installing Ollama on Arch (CachyOS) and Running the GPT‑OSS‑120B Model

Details
Written by: MrHorda
Category: Blog
Hits: 11412

1. Overview

This guide shows how to:

  1. Install Docker on Arch Linux (CachyOS).
  2. Deploy the containerized open‑webui front‑end and connect it to a local Ollama server.
  3. Enable ROCm for the AMD GPU and run Ollama in mixed GPU + CPU mode.
  4. Configure Ollama as a systemd service with the custom environment you provided.
  5. Apply optimal model parameters for the 120 B‑parameter GPT‑OSS model.
  6. Set a system prompt that enforces the desired behaviour.

All commands assume you are using a regular user with sudo privileges.


First what we need is to Install Linux OS

2. Prerequisites

Component
Specification
OS
Arch Linux (CachyOS) – up‑to‑date
GPU
AMD Radeon RX 7900 XTX (ROCm‑compatible)
CPU
Intel i9‑13900K
RAM
96 GB
Model
GPT‑OSS‑120B (local)
Web UI
open‑webui (Docker)
LLM engine
Ollama (installed from the Arch repo)

3. System Update & ROCm Packages

 

# Full system upgrade
sudo pacman -Syu
 
# Install ROCm core packages
sudo pacman -S rocm-hip-sdk rocm-opencl-sdk
 

The ROCm stack provides the hip runtime and OpenCL drivers needed for the RX 7900 XTX.

4. Install Ollama

# Install Ollama from the official repo
sudo pacman -S ollama
 

After installation, verify the binary:
ollama --version

5. Install Docker

sudo pacman -S docker
sudo systemctl enable --now docker
 

Add your user to the docker group to avoid sudo for each command:
sudo usermod -aG docker $USER
newgrp docker # reload groups

6. Deploy open‑webui (Docker)

Create a directory for the UI configuration:
mkdir -p ~/openwebui && cd ~/openwebui

Create a simple docker-compose.yml:

version: "3.8"
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_HOST=http://host.docker.internal:11434   # Connect to local Ollama
    restart: unless-stopped
 

Start the stack:
docker compose up -d

 

Open http://localhost:8080 in a browser – the UI will forward requests to the Ollama server running on the host.

7. Configure Ollama Systemd Service (Mixed GPU + CPU)

Create a drop‑in directory for custom settings:
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo nano /etc/systemd/system/ollama.service.d/custom.conf

Paste the following content (exactly as you provided):

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="ROCM_VISIBLE_DEVICES=0"
Environment="OLLAMA_KEEP_ALIVE=10m"
Environment="OLLAMA_MAX_VRAM=24GiB"
Environment="OMP_NUM_THREADS=24"
Environment="KMP_AFFINITY=granularity=fine,compact,1,0"
 

Reload systemd and restart Ollama:
sudo systemctl daemon-reload
sudo systemctl enable --now ollama

 

You can verify that Ollama is listening on all interfaces:
ss -tlnp | grep 11434

 

The environment variables enable:

  • GPU detection (ROCM_VISIBLE_DEVICES) – selects the first AMD device.
  • Mixed‑mode execution – the HSA_OVERRIDE_GFX_VERSION and KMP_AFFINITY flags let the runtime use both GPU and CPU cores efficiently.
  • Memory limits (OLLAMA_MAX_VRAM=24GiB) – caps VRAM usage to leave headroom for the system.

8. Pull & Run the GPT‑OSS‑120B Model

Ollama stores models in its own registry. Assuming the model is available as gpt-oss-120b:
ollama pull gpt-oss-120b

 

Create a configuration file for the model (optional but convenient):
mkdir -p ~/.ollama/models/gpt-oss-120b
cat > ~/.ollama/models/gpt-oss-120b/config.yaml <<EOF
num_ctx: 8192
temperature: 0.7
top_p: 0.9
top_k: 40
repeat_penalty: 1.1
system_prompt: |

You are GPT-OSS-120B, a highly capable open-source language model.
Follow these rules:
1. Be concise, factual, and logically structured.
2. Give direct answers before optional detail.
3. Use bullet points when listing information.
4. Avoid filler, disclaimers, and unnecessary explanations.
5. When reasoning, think step‑by‑step internally and present only the final result.
6. Avoid hallucinating by admitting when information is unknown.
7. Prioritize accuracy, clarity, and practicality.
You behave as a helpful expert assistant with strong reasoning ability.
EOF
 

Restart Ollama to load the new config:
sudo systemctl restart ollama

Now the UI (open‑webui) will use the 120 B model with the parameters you specified.

9. Verify GPU Utilisation

Install rocm-smi to monitor the AMD GPU:
sudo pacman -S rocm-smi
watch -n 1 rocm-smi

When you send a request from the UI, you should see GPU memory usage rise (up to the 24 GiB limit) while CPU threads are also active, confirming mixed‑mode operation.

10. Troubleshooting Tips

Symptom
Likely Cause
Fix
Ollama does not start, logs show “GPU not found”
ROCm not loaded or wrong ROCM_VISIBLE_DEVICES
Verify ls /dev/kfd exists; ensure the kernel module amdgpu is loaded.
open‑webui cannot reach Ollama (connection refused)
Service not listening on 0.0.0.0
Confirm OLLAMA_HOST=0.0.0.0 in the custom.conf and reload systemd.
Out‑of‑memory errors
OLLAMA_MAX_VRAM too low for the model
Increase to 30GiB if your card has enough free VRAM, or lower num_ctx.
Docker containers cannot see the host network
Missing host.docker.internal mapping on Linux
Add --add-host=host.docker.internal:host-gateway to the compose file under openwebui service.

11. Full Command Summary

# 1. System update & ROCm
sudo pacman -Syu
sudo pacman -S rocm-hip-sdk rocm-opencl-sdk
 
# 2. Install Ollama
sudo pacman -S ollama
 
# 3. Install Docker
sudo pacman -S docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
 
# 4. open-webui (docker compose)
mkdir -p ~/openwebui && cd ~/openwebui
cat > docker-compose.yml <<'EOF'
version: "3.8"
services:
  openwebui:
    image: ghcr.io/open-webui/open-webui:latest
    container_name: openwebui
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_HOST=http://host.docker.internal:11434   # Connect to local Ollama
    restart: unless-stopped
docker compose up -d
 
# 5. Ollama systemd custom config
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/custom.conf > /dev/null <<'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="ROCM_VISIBLE_DEVICES=0"
Environment="OLLAMA_KEEP_ALIVE=10m"
Environment="OLLAMA_MAX_VRAM=24GiB"
Environment="OMP_NUM_THREADS=24"
Environment="KMP_AFFINITY=granularity=fine,compact,1,0"
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now ollama
 
# 6. Pull model & set parameters
ollama pull gpt-oss-120b
mkdir -p ~/.ollama/models/gpt-oss-120b
cat > ~/.ollama/models/gpt-oss-120b/config.yaml <<'EOF'
num_ctx: 8192
temperature: 0.7
top_p: 0.9
top_k: 40
repeat_penalty: 1.1
system_prompt: |
You are GPT-OSS-120B, a highly capable open-source language model.
Follow these rules:
1. Be concise, factual, and logically structured.
2. Give direct answers before optional detail.
3. Use bullet points when listing information.
4. Avoid filler, disclaimers, and unnecessary explanations.
5. When reasoning, think step‑by‑step internally and present only the final result.
6. Avoid hallucinating by admitting when information is unknown.
7. Prioritize accuracy, clarity, and practicality.
You behave as a helpful expert assistant with strong reasoning ability.
EOF
sudo systemctl restart ollama

12. Conclusion

You now have a fully functional local LLM stack on Arch Linux (CachyOS) that:

  • Leverages the AMD Radeon RX 7900 XTX via ROCm.
  • Runs the GPT‑OSS‑120B model in mixed GPU + CPU mode for optimal performance.
  • Provides a web‑based chat UI through open‑webui.
 

Feel free to tweak the environment variables (e.g., OMP_NUM_THREADS or OLLAMA_MAX_VRAM) to match future hardware upgrades. Happy prompting!

Login Form

Older Posts