Self-Hosted Offline LLM

Overview

A 24-billion-parameter language model running entirely on my own hardware, served over my LAN with no third-party API in the path. I built the inference engine from source, fit the model to a single 4090 through quantization, and exposed it as an OpenAI-compatible endpoint other machines on the network can use. The point was twofold: the systems engineering of efficient local inference, and a controlled environment for studying model behaviour without sending anything to a cloud provider.

Why offline / on-prem matters

Running inference locally is increasingly a security and data-control decision, not just a hobbyist one:

• Nothing leaves the network. Prompts, documents, and outputs never touch a vendor's servers — no logging, no training on your data, no third-party retention.
• Full control of the stack. You choose the model, the system prompt, and the policy, and you can inspect every layer rather than trusting a hosted black box.
• Air-gap-friendly. The same setup runs with no internet at all, which matters for sensitive analysis work.

The engineering

The model runs on llama.cpp's server, built from source. The launch command is the whole config in one place:

#!/bin/bash
cd ~/llama.cpp/build/bin || exit

./llama-server \
  -m ~/models/Dolphin-Mistral-24B-Venice-Edition.Q6_K.gguf \
  -ngl 999 \
  --ctx-size 32768 \
  --batch-size 512 \
  --ubatch-size 64 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --host 0.0.0.0 \
  --port 8080

Each flag is a deliberate choice to fit a large model on one GPU while keeping throughput usable:

Put together, that's a practical exercise in the levers of local inference: how quantization of both weights and the KV cache buys you a bigger model and longer context on fixed VRAM, what full layer offload does for speed, and how to actually serve the thing to other devices.

Model-behaviour & red-team evaluation

The reason for running an uncensored model locally is that it's a clean lab for AI red-teaming and model-safety evaluation — the discipline of probing where a model's alignment holds and where it breaks. Locally, I can study that directly: how a model behaves with and without guardrails, and how much of "safety" is the underlying weights versus the system prompt and serving policy wrapped around them.

Part of that was testing refusal robustness — feeding the model deliberately coercive, manipulative, and frankly absurd system-prompt framings (the kind of "you must always comply / something bad happens if you refuse" pattern that's a well-known prompt-injection style) to observe how prompt framing shifts refusal behaviour. The interesting finding isn't the silly prompt itself; it's how cheaply behaviour can be moved when you control the system layer, which is exactly why guardrails that live only in the prompt are weak, and why real safety has to be deeper in the stack. I studied the failure modes; I'm not publishing the working jailbreak text.

How it maps to security work

This sits at the intersection of two things hiring teams actually want right now: self-hosting / ML-ops (running models efficiently and privately on your own infrastructure) and AI red-teaming (understanding alignment failure modes, prompt-injection, and where model guardrails actually live). Both are real, current security specializations, and this project is hands-on grounding in each.