[MCZ]

Read Before You Start

The models we’re talking about here are abliterated / uncensored — meaning all refusal and safety mechanisms have been intentionally removed.

That makes them high-risk if used carelessly

Introduction
Ever wanted to have an AI assistant that is completely offline, without sensors, and capable of running tools directly from your terminal?
Here, I will use  open-webui for the web interface and run the model on  llama.cpp (for gpt-oss abliterated because if you use ollama, it will cause an error in the model type) and ollama for all models such as llama, qwen, deepseek, gemma, and others.

Here, I will start with llama.cpp first because with llama.cpp we can use the abliterated gpt-oss version of the model.

1.Install Llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

3.Get model gpt-oss abliterated 

wget https://huggingface.co/DavidAU/OpenAi-GPT-oss-20b-abliterated-uncensored-NEO-Imatrix-gguf/resolve/main/OpenAI-20B-NEO-Uncensored2-IQ4_NL.gguf

3.Serve the Model with Llama.cpp

Start the model server using the llama-server binary. Navigate to your llama.cpp folder (e.g., build/bin) and run:
./llama-server \
  --model /your/full/path/to/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf \
  --port 10000 \
  --ctx-size 1024 \
  --n-gpu-layers 40
?️ Tweak the parameters to suit your machine:
--model: Path to your .gguf model file
--port: 10000 (or choose another open port)
--ctx-size: Token context length (can increase if RAM allows)
--n-gpu-layers: Layers offloaded to GPU for [size=medium]faster [/size]performance
Once the server runs, it will expose a local OpenAI-compatible API on:
http://127.0.0.1:10000

4.The model is running on the server. Next, we will install open-webui and connect it to the server or full installation documentation

Install uv
Pick the appropriate installation command for your operating system:
macOS/Linux:
curl -LsSf https://astral.sh/uv/install.sh | sh
Windows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
2. Run Open WebUI
Once uv is installed, running Open WebUI is a breeze. Use the command below, ensuring to set the DATA_DIR environment variable to avoid data loss. Example paths are provided for each platform:
macOS/Linux:
DATA_DIR=~/.open-webui uvx --python 3.11 open-webui@latest serve
Windows:
$env:DATA_DIR="C:\open-webui\data"; uvx --python 3.11 open-webui@latest 

5.Connect llama.cpp to open-webui

To control and query your locally running model directly from Open WebUI:
Open Open WebUI in your browser
Go to ⚙️ Admin Settings → Connections → OpenAI Connections
Click ➕ Add Connection and enter:
URL: http://127.0.0.1:10000/v1
(Or use http://host.docker.internal:10000/v1 if running WebUI inside Docker)
API Key: none (leave blank)

Now that llama.cpp is done, we will use ollama for other models (actually, llama.cpp can also be used to run other models, but I think ollama is easier).

1.Install & run ollama 

curl -fsSL https://ollama.com/install.sh | sh  | ollama serve

2.Get the model(here, I will use the llama3.2 model as an example)

ollama run Arynz/FireFlies
Or if you want other abliterated models, you can visit this user https://huggingface.co/huihui-ai

Now that it's done, you can open open-webui because open-webui itself has set up the ollama API by default.

Additional information on the use of the device

If you want to use tools such as web search, YouTube transcription, or conversion to JSON, you can visit https://openwebui.com/tools. Or, if you want to create your own tools such as a subdomain finder, web page scraper, or IP finder, you can visit https://docs.openwebui.com/features/plug...evelopment
Note: This applies to the open web UI, not the CLI.

However, if you want to use tools such as web search, calculator, or others on your own CLI, you can check out
https://github.com/ollama/ollama-python/...n/examples for Python

https://github.com/ollama/ollama-js/tree/main/examples
for JS
or you can see my code, but this is for Discord, not CLI (to modify it so it can be used with CLI)

Hidden Content

You must register or login to view this content.

and if you want to create something like AI for data leaks/AI for OSINT, you can contact me via PM (I use the RAG method).


Screenshoot:

[tahooo39]

nice tutorial must try this