IntelliDemo AI — Streaming LLM Features in a Web App

Delivery

Token streaming

Cost control

Model routing + caching

Safety

Schema-validated output

Status

Live demo available

Overview

IntelliDemo is a platform for AI-assisted content generation: a user describes what they want, the system streams a response, and structured results feed back into the interface. React front end, Node.js API layer, language-model providers behind an abstraction.

The problem

Calling a model API is a few lines of code. Everything that makes the feature usable is the surrounding engineering — responses that take several seconds feel broken without streaming, costs scale linearly with success, and untrusted text reaching a model is an injection surface.

Architecture

Providers sit behind a single interface so the application never imports a vendor SDK directly. That abstraction earned its keep more than once: swapping providers, or routing different request types to different models, became a configuration change rather than a refactor.

• Streaming responses over the full request path, verified end to end rather than only on localhost.
• Difficulty-based routing — cheap fast models for routine requests, stronger models reserved for hard ones.
• A hash-keyed response cache for repeated prompts.
• Schema validation on every structured output, with a defined fallback when parsing fails.

Technical challenges

Streaming that survives the deployment path

Streaming worked locally and arrived as a single payload in the deployed environment. The cause was buffering in the layer in front of the application — the server was streaming correctly and something downstream was accumulating chunks before forwarding them.

Fixing it meant setting the response headers that disable proxy buffering and confirming behaviour with curl against production. The general lesson: a stream is only as good as the least cooperative hop in its path, and localhost tells you nothing about the others.

Keeping cost proportional to value

A long system prompt sent on every request is a fixed cost multiplied by traffic. Unbounded conversation history is worse — it grows quadratically, because every turn resends everything before it.

Three controls addressed most of it: a rolling history window with a running summary of older turns, an explicit output-token ceiling, and routing by difficulty so the expensive model only handles requests that need it. Token counts are logged per request, which made it possible to answer which features were actually costly rather than guessing.

Treating model output as untrusted

Model output is generated text, not a trusted API response. Interpolating it into queries, markup or downstream commands without validation is the same mistake as trusting a form field.

Structured responses are parsed against a schema and rejected on mismatch rather than assumed well-formed. Authorisation decisions are made in the handler, never delegated to the model — a model can be talked into things, and an access check cannot.

Outcome

The platform runs as a live demo with streaming generation and the cost and safety controls described above. The most valuable structural decision was the provider abstraction — it turned model selection into an operational lever rather than a commitment baked into application code.

Published May 28, 2026 by Cenedy Udoy Palma.