Local LLM ChatUI

Project Architecture & Overview

Private, offline generative AI chat studio. Interfaces with Ollama and llama.cpp runtimes to execute 4-bit and 8-bit quantized models (Llama 3, Mistral, Gemma) locally on consumer NVIDIA GPUs with optimal VRAM memory allocation and streaming Server-Sent Events (SSE).

Key Engineering Milestones

• ✦Zero cloud dependency: 100% private and confidential local inference
• ✦Real-time markdown, syntax highlighted code blocks, and math LaTeX rendering
• ✦Fine-grained parameter controls: Temperature, Top-P, Context Window, System Prompts
• ✦Optimized layer offloading maximizing GPU VRAM efficiency on modest hardware