Architecture Breakdown: Engineering an In-House, Context-Aware AI Assistant on Bare Metal

When deploying an interactive AI assistant for an independent engineering studio, the default industry playbook usually defaults to managed SaaS platforms: subscribing to third-party chat widgets, hooking up brittle multi-step no-code relays, and incurring unpredictable monthly per-token API charges.

For Veronami, Amaya architected a self-hosted conversational pipeline designed to run directly across her private server infrastructure. The goal was clear:

• Deliver an ultra-responsive, non-intrusive floating chat widget on the Ghost CMS frontend.

The Infrastructure Topology: Physical Separation of Concerns

Self-hosting Large Language Models (LLMs) on standard CPU instances requires strict boundaries. CPU-bound inference demands significant RAM bandwidth and thread utilization. Running an inference runtime on an instance already managing high-memory mail daemons (such as Mailcow’s ClamAV and Solr) or MariaDB transactional databases risks Out-of-Memory (OOM) process terminations and service downtime.

Amaya structured the hardware topology to ensure production stability:

[ Visitor Browser (veronami.com) ]
                │
                ▼ (Lightweight HTTPS / POST)
[ Production Host: RS 1000 G11 (12M) ]
├── Ghost CMS (Static frontend delivery)
├── Nginx Reverse Proxy (SSL Termination)
├── Node.js API Gateway (port 41006)
│   ├── Input sanitization & trust proxy rate limiting
│   └── Cheerio-based page summary caching
└── ERPNext Instance (erp.veronami.com)
                │
                ▼ (Encrypted WireGuard VPN / 10.14.0.0/24)
[ Dedicated AI Engine: VPS 2000 G12 (10.14.0.1) ]
└── Ollama Daemon (Port 11434)[cite: 2]
    └── Quantized Small Language Model (qwen2.5:1.5b)

Step 1: The AI Inference Node (Ollama Configuration)

On the dedicated inference node (10.14.0.1), Amaya deployed Ollama running a quantized small language model (qwen2.5:1.5b). A 1.5B 4-bit model keeps the memory footprint under 2 GB while sustaining 20–35 tokens per second purely on CPU threads—well above typical human reading speeds.

Because Ollama binds by default to host loopback (127.0.0.1:11434), it actively rejects traffic arriving over an internal VPN interface. Amaya applied a systemd drop-in configuration to expose the daemon to the internal tunnel:

# /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

After executing systemctl daemon-reload and restarting the daemon, host firewall rules were locked down using ufw so that only the web host's internal VPN address could reach port 11434:

sudo ufw allow from 10.14.0.2 to any port 11434 proto tcp

Step 2: The Gateway Middleware (Rate Limiting & Page Scrapes)

Exposing an inference engine directly to the web invites token drainage and system lockups. Amaya engineered an Express-based gateway proxy on the production host.

The middleware manages three critical operational tasks:

1. Reverse Proxy & Header Trust: With Nginx terminating SSL, Express is configured with app.set('trust proxy', 1) so that express-rate-limit accurately tracks visitor IP addresses via X-Forwarded-For rather than throwing validation errors.
2. Context Ingestion & In-Memory Caching: When a visitor opens the chat on a blog post, the frontend passes the URL pathname and any client-scraped summaries contained in a <p id="blog-summary"> element. If absent, the backend fetches the page using Cheerio and stores the excerpt in an in-memory Map to ground the model without latency.
3. Payload Truncation: Chat history is strictly sliced to the latest 6 turns with an 800-character ceiling per turn, preventing prompt-injection overflow and context-window blowouts.

// Excerpt from /opt/veronami/chat_handler/chat_api.js
import express from 'express';
import helmet from 'helmet';
import rateLimit from 'express-rate-limit';
import sanitizeHtml from 'sanitize-html';
import axios from 'axios';

const app = express();
app.set('trust proxy', 1); // Crucial for reverse-proxied rate limiters
app.use(helmet());
app.use(express.json({ limit: '20kb' }));

const pageContextCache = new Map();
const OLLAMA_BASE = process.env.OLLAMA_HOST || 'http://10.14.0.1:11434';

const chatLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 12,
  standardHeaders: true,
  legacyHeaders: false,
  message: { error: 'Rate limit exceeded. Please wait a moment.' }
});

app.post('/api/chat', chatLimiter, async (req, res) => {
  const { messages, sitePath, clientSummary } = req.body;
  if (!Array.isArray(messages) || messages.length === 0) {
    return res.status(400).json({ error: 'Payload requires a messages array.' });
  }

  // Resolve dynamic page context
  const normalizedPath = sitePath?.trim() || '/';
  let activeContext = pageContextCache.get(normalizedPath);

  if (!activeContext && clientSummary) {
    activeContext = sanitizeHtml(clientSummary, { allowedTags: [] }).trim();
    pageContextCache.set(normalizedPath, activeContext);
  }

  let systemPrompt = `You are Vero, the digital assistant for Veronami Digital and Prototyping Services.
We specialize in web API development, systems architecture, and custom FDM 3D printing.
Direct booking inquiries to our 30-minute consultation session (₹500).`;

  if (activeContext) {
    systemPrompt += `\n\nPage context for (${normalizedPath}): "${activeContext}"`;
  }

  // Validate and sanitize the last 6 message turns
  const validatedMessages = messages.slice(-6).map(item => ({
    role: item.role,
    content: sanitizeHtml(item.content, { allowedTags: [] }).trim().slice(0, 800)
  }));

  try {
    const response = await axios.post(`${OLLAMA_BASE}/api/chat`, {
      model: 'qwen2.5:1.5b',
      messages: [{ role: 'system', content: systemPrompt }, ...validatedMessages],
      stream: false,
      options: { temperature: 0.4, num_predict: 256 }
    }, { timeout: 30000 });

    return res.status(200).json({ reply: response.data?.message?.content });
  } catch (err) {
    console.error('[Inference Gateway Error]:', err.message);
    return res.status(502).json({ error: 'Assistant temporarily unreachable.' });
  }
});

Step 3: The Zero-Bloat Vanilla Frontend

To maintain fast load times and clean styling on Ghost, Amaya bypassed bloated JavaScript widget libraries. Using Ghost's Code Injection settings, she added a discrete, accessible floating launcher and a minimal chat dialogue styled directly with CSS variables:

<!-- Ghost Footer Injection -->
<div id="v-chat-trigger" onclick="toggleVeroChat()" role="button" aria-label="Open support chat">
  <span>💬</span>
  <span>Chat</span>
</div>

<div id="v-chat-panel" style="display: none;" role="dialog" aria-modal="true">
  <div id="v-chat-header">
    <span>Veronami Assistant</span>
    <button onclick="toggleVeroChat()">&times;</button>
  </div>
  <div id="v-chat-thread"></div>
  <form id="v-chat-form" onsubmit="submitVeroChat(event)">
    <input type="text" id="v-chat-input" placeholder="Ask about services or consultation..." maxlength="800" required />
    <button type="submit" id="v-chat-send">Send</button>
  </form>
</div>

On submit, the script checks the DOM for a #blog-summary paragraph, pulls the current window.location.pathname, and sends the payload to /api/chat.

Key Architectural Takeaways

By establishing clean hardware boundaries and relying on native protocols, Amaya built a self-contained production system:

• Predictable Hardware Footprint: The inference daemon runs strictly on dedicated compute, keeping the mail stack and ERP databases completely isolated.
• Zero SaaS Tolls: Lead intake, knowledge grounding, and natural language answering operate without monthly per-seat fees or external API meters.
• Deterministic Accounting: Visitor queries turn directly into actionable, structured data within ERPNext, ensuring every interaction is accounted for in the core business ledger.

Sign up with Veronami so that you will know when our developers build an in-house hosted ai support system that has secure access to the database and can do basic support ticket resolutions.