LLM Usage in a Multi-Surface Application

personal // research / Apr 24, 2026

Large language models are at its peak importance when treated as infrastructure with clear limits rather than the quintessential answer to every problem (yet, that's not how it goes most of the time). Most teams work linearly and add one prompt for routing, another for summarizing, another for pulling out data. In this practice, the system is full of model calls with no real thinking behind them, and no clear answer for when the system actually needs the model, how the team measures quality, or where plain deterministic code would do the job better. Duhh. Latency climbs, costs get hard to trace, prompts drift, and bugs turn into a headache to chase down because retrieval, prompt, and model quality are all tangled together. A more durable approach is to treat LLM use like any other engineered part of the system: clear task boundaries, real ways to measure quality, token budgets, and fallback plans for when things break. Teams need to start implementing a shared inference client, telemetry tagged by purpose, retrieval-backed flows, and a handful of different ways to track how they lean on the model. The plan shouldn't chase more model-powered features. It should figure out which calls pull their weight, which can get dropped, and how to govern the rest. For each call, ask whether it earns its place against simpler code, a good retrieval step, and the budget the team has to spend can be considered later. A good place to start is separating deterministic work from generative work; the predictable stuff from the creative stuff. Plenty of tasks never need the model to generate anything. Routing usually comes down to simple rules for the common cases. Confirmations, acknowledgments, and other low-content replies can be templated. Many "transformations" are structured mapping or field extraction, not open-ended writing. In those cases the model call only buys us latency, cost, and randomness with nothing to show for it. That's an architectural decision, and it comes first: find the low-value paths and pull the model out of them. Once those are cleared out, the next big category is retrieval-backed work. For anything that leans on facts, retrieval is still the most powerful lever someone can use. Lewis et al.'s framing of Retrieval-Augmented Generation is the right foundation here, stating that models stay more accurate and current when they're conditioned on the right evidence rather than relying on whatever they absorbed during training [1]. A lot of the times, we blame the model for getting things wrong when really, context could have been delivered in a much more structured way. Retrieve the wrong evidence, or way too much of it, and even a strong model will answer badly. So when teams dig into quality, they shouldn't open with prompt tuning or swapping models. The process should start with what the model is being fed: how it's retrieved, what evidence is selected, and most importantly, I cannot emphasize this enough again, the context being sent. Measurement is the next piece, and it has to happen at the level of the task. In a real product, you can't judge model quality from generic benchmarks or a vendor's pitch (all the while, a good objective place to start). Classification, field extraction, retrieval-backed answering, and synthesis all break in different ways, so they each need their own metrics. OpenAI's evaluation guidance serves us this to think about: prompt and model changes should be judged against real test sets and clear acceptance criteria, not just a human feeling or a quick read-through [2][3]. Without checks at the task level, teams want outputs that look nicer instead of ones that are measurably more correct. When a single system has many uses of the model, skipping those checks leaves a real gap where ownership can blend in, and a prompt tweak in one spot can break something that matters somewhere else. Latency deserves the same seriousness as correctness. Systems can fail by being too slow as often as by being wrong. Groq's guidance lays out the trade-off cleanly, stating that input tokens mostly drive the cost of reading the prompt, output tokens drive the cost of writing the answer, and end-to-end latency is more than the model's run time [4]. The main issue is that teams reach for a model choice before they've measured how big their prompts are, how long the answers run, or how much extra context they're stuffing in. One of the highest-leverage things an engineer can do is trim prompt content, keep answers shorter, and sort tasks by how latency-sensitive they are. Some tasks earn heavier reasoning or a longer answer. Most don't. Without a latency budget per task, it's super easy to overspend effort on trivial requests and underinvest in the paths that get hit with more requests. Prompts. Prompts deserve the same treatment as code. A prompt carries a lot: what the task assumes, what shape the answer should take, how edge cases are handled, and where the safety lines are. That earns it the same versioning, review, and testing we typically give real code (at least, pre-AI). OpenAI's prompt guidance leans on clarity, directness, and structure, but the bigger operational lesson is that prompts shouldn't evolve informally [5]. Once you've got prompts spread across routing, extraction, synthesis, and retrieval-backed answers, there's a bit of a reliability issue. Instead, the plan should include a prompt registry, a versioning approach, and one firm rule: a prompt change has to pass its evals before it ships, no matter how much nicer it reads. I also want to emphasize the movement to design for graceful degradation. No part of the system should assume the model, the network, or the retrieval service will always be up and quick. Some tasks can fall back cleanly to a template, a partial retrieval, or cached context. Others should defer, abstain, or return a constrained response instead of improvising. In turn, every task should map to a fallback mode, with clear rules for timeouts, retries, and when to skip the answer. This goes beyond fault tolerance and it keeps the system "honest" when things degrade instead. I think that's better than hiding a drop in quality. The sequence of all this matters. First, systems have to inventory every way the model is currently being used and group them into families: routing, extraction, retrieval-backed answering, enrichment, synthesis. Then, classify each family; is it truly generative, mostly a retrieval problem, or something deterministic code could handle? Only after that can evals be designed for each family. Once the evals exist, a team's telemetry can point them at the priciest or slowest paths, and then, and only then does it make sense to compare models, providers, or prompt revisions. What I want people to walk away with this is a compact, practical framework. The pieces are straightforward. A registry of every model call, a taxonomy of the task types, a prompt registry, sets of eval examples, dashboards for latency and token use per purpose, and a matrix of fallback plans. With those in hand, changes stop being ad hoc. It becomes necessary AND needed for a new model call to justify why simpler code or a retrieval step can't do the job. Prompt changes have to pass their evals, latency-sensitive paths get token budgets, retrieval-backed paths get evidence policies, low-value calls become easy to spot and cut. The most important work in this principal is building real engineering discipline around how people will actually use the model, not polishing prompts without considering impact. The question is which tasks deserve the model, how these should be measured, and what operational envelope they have to work within. For a system that already has centralized inference and solid telemetry, the highest-value next move is to split the work into clear tracks. Do this, and we end up with systems that are easier to reason about, cheaper to run, and a lot steadier as AI keeps changing. [1] Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks [2][3] OpenAI, evaluation guidance [4] Groq, production/latency guidance [5] OpenAI, prompt engineering guidance