Why LLM placement matters more than LLM adoption
A lot of teams start the conversation in the wrong place. They ask whether they should use an LLM at all, as if the choice is a clean yes or no. In production, that’s usually the less interesting question. The real decision is where the model sits in the stack, because placement decides how much it can reshape behavior and how much pain you’ll have when it gets something odd, slippery, or just plain wrong.
The model’s position matters more than its mere presence. Put it near the edge, and it can clean up messiness. Put it deeper in the system, and it starts steering outcomes.
That tradeoff shows up all over LLM production systems. At a high level, most systems have a few layers: input handling, retrieval, ranking, and downstream execution. Input handling deals with raw user text, which is often noisy, incomplete, or oddly phrased. Retrieval fetches candidate documents, records, or results. Ranking decides what rises to the top. Execution takes an action, whether that means sending a reply, creating a ticket, updating a record, or kicking off a workflow. An LLM can sit in any of those layers, but it does not behave the same way in each one.
Close to the user, the model usually has a narrower job. It might rewrite a search query, extract intent, normalize a date, or map freeform text into a schema. That kind of use is fairly easy to reason about because the rest of the system still does the heavy lifting with deterministic logic. Move the model into retrieval, and it starts deciding what data even gets considered. Put it into ranking, and it can change which candidates win. Place it in execution, and now the model is no longer just shaping text or sorting results. It is influencing actions that may reach customers, internal systems, or external APIs.
That extra reach is the tradeoff. More flexibility means more leverage. It also means a bigger blast radius when the model drifts, misreads intent, or fails on an edge case that only shows up every third Thursday after lunch. Debugging gets harder as soon as the model’s output affects more of the pipeline. If an input rewrite looks wrong, you can often inspect the prompt and the transformed query. If a retrieval step misses the right document, you may need to examine candidate generation, embedding behavior, filters, and reranking logic. If execution goes sideways, you are no longer asking, “Did the model understand the request?” You are asking why a bad guess turned into a live side effect.
Search is a good example. If an LLM only cleans up the query, the system can still fall back to standard keyword and semantic search. If it ranks results, a small prompt change can reorder the entire page. Support triage has a similar shape. An LLM at intake can classify tickets into billing, login, or shipping. That is one thing. Let it choose the agent, draft the response, or trigger an account change, and the stakes climb fast. Workflow automation goes even further. When a model merely drafts a task description, a human can correct it. When it decides which workflow runs next, the system starts depending on its judgment in a much more direct way.
That is why LLM architecture and LLM orchestration are really about control surfaces, not novelty. The same model can be a tidy text cleaner in one design and a decision-maker in another. The difference is not the model. It is the layer where you let it act.
In the next section, we’ll stay near the edge and look at the safest place to start: messy user input that needs a little cleanup before the rest of the system touches it.
Keep the model close to the user when the input is messy
Once you accept that model placement matters, the safest place to start is usually the edge of the system. That’s where user input first arrives, and that’s also where the mess tends to live. People type half-formed questions, paste logs with missing context, mix product names with slang, and ask for two things at once. A model can clean that up before anything irreversible happens.
In practice, this layer does a few fairly plain jobs: query rewriting, intent extraction, normalization, and schema mapping. A customer might write, “refund for last order pls, or maybe swap if possible.” A model can turn that into a cleaner search query, a refund intent, or a support record with structured fields the backend can actually use. A developer might say, “need EU pricing for enterprise plan and maybe SSO too.” That can become two separate intents, or one ticket routed to the right queue. No magic, just translation from human shorthand into something deterministic systems can process.
The safest LLM use is the one that can be wrong without breaking anything irreversible.
That last part matters. When the model sits close to the user, the rest of the stack can still behave like a normal system. The query gets rewritten, classified, or mapped to a schema, then conventional code takes over. If the model makes a bad call, the blast radius is usually contained. A search query might return slightly off results. A support ticket might land in the wrong triage bucket. Annoying, yes. Catastrophic, usually not.
This is why edge-layer work feels so comfortable in production. The model is assisting, not deciding the final state of the world. If you’re using it to clean up search input, for example, a typo-heavy query like “blue tooth headfones under 50, noise canceling pls” can be normalized into something the search backend understands. You can still run ordinary filtering, faceting, and ranking after that. The model’s job is to strip away noise, not to pick the final answer from scratch.
The same pattern works well for support ticket classification. A user writes one long paragraph with billing complaints, setup problems, and a note about an error code they saw three days ago. The model can extract the main issue, tag the ticket, and map the payload into a schema that your support workflow expects. If the classification is a little off, a human agent or a later rules step can correct it. You haven’t handed the model the power to trigger refunds, close accounts, or change entitlements on its own.
Structured outputs help a lot here. If you need the model to emit JSON, enums, or a fixed field set, use a schema instead of hoping for polite behavior from the prompt. Google’s structured output guidance is one example of that pattern: constrain the response shape so downstream code doesn’t have to guess. Anthropic’s tool use overview covers a similar idea from another angle, where the model produces a tool call rather than free-form text. That’s a good fit when you want intent extraction or routing decisions, but still want your application code to own the actual action.
For conversational systems, a small amount of state can help the model stay honest about what the user has said so far. The Gemini interactions overview is useful reading if you’re building multi-turn flows where one message depends on the last one. That matters for things like “use the same shipping address as before” or “make it the cheaper option,” because the model needs enough context to normalize the request without inventing missing pieces.
In routing workflows, the edge model can do something even simpler: decide where a request should go next. “I can’t log in,” “please change my invoice,” and “ship this report to Slack every Monday” all map to different systems. One model call can convert those free-form requests into clean routing labels, then a conventional workflow engine takes over. If the model tags a billing issue as account access, the wrong team gets paged, but the underlying account data still hasn’t been touched. That separation buys you time to catch the mistake.
The nice thing about this setup is that it keeps the expensive parts boring. Search still searches. Rules still rule. Workflow engines still route based on explicit fields. The model just reduces ambiguity before the rest of the pipeline sees it. You can log the raw input, the rewritten query, the extracted intent, and the final route, then compare them later when something looks odd. That makes debugging far less painful than trying to explain why a model buried deep in the stack decided to change the meaning of a request halfway through execution.
Of course, edge placement doesn’t remove error. It just limits where error can go. A bad normalization step can still send a user down the wrong path, and a sloppy schema mapper can drop useful detail. Still, those are usually easier problems to catch than a model making hidden decisions inside retrieval or action selection. At the edge, you can inspect the transformation directly, patch prompts or schemas quickly, and keep the rest of the system stable while you improve it.
For most teams, that’s the sweet spot. Put the model where the input is noisy, the output can be checked, and the consequences of a mistake stay modest. Then let the deterministic parts of the stack do what they’ve always done well.
When it reaches deeper: retrieval, ranking, and execution
Once the model moves past input cleanup, the stakes change fast. At the retrieval layer, it can decide what gets pulled from a search index or knowledge base. In reranking, it can sort candidates by relevance instead of leaving that job to term matches or static heuristics. In candidate selection, it can prune a long list down to the few items worth spending compute on. In execution, it can choose the next action, whether that means calling a tool, opening a ticket, sending a message, or writing into a system of record.
That extra reach is useful when the domain is messy in a way that simple rules struggle with. A support system, for example, might need to understand that “my invoice got weird after the plan change” should pull billing docs, recent account events, and past tickets, even if none of those words appear verbatim. A product search flow might need to map “black running shoe, wide fit, under $120” to a shelf of items that use different wording in the catalog. In a RAG architecture, that sort of semantic retrieval can be a better fit than brittle keyword search, because the model can reason over intent, synonyms, and domain jargon before the downstream system makes a final decision.
The deeper the model sits, the more it can correct bad structure, and the more painful its mistakes become.
That tradeoff shows up in reranking very quickly. A plain search engine can return ten decent candidates, but an LLM reranker can look at the query, the snippet, and the surrounding context, then decide which result is actually about the user’s problem. For a knowledge base with inconsistent titles, stale labels, or awkward taxonomy, that often improves relevance more than another round of prompt tweaking in production. The same applies to candidate selection. If a workflow has twenty possible handlers, the model can narrow the field before a deterministic rule engine takes over. Done well, that keeps the system from drowning in noise.
Still, the deeper you go, the less obvious the failure modes become. A bad rewrite at the retrieval step can fetch the wrong documents. The reranker then promotes those documents because they seem to fit the query. The execution layer reads the wrong context and takes an action that looks reasonable on paper, but makes no sense in the real world. One weak decision feeds the next. By the time someone opens the dashboard, the original mistake has turned into a chain of plausible-looking steps.
This is where debugging gets annoying in the ordinary, human sense. The failure doesn’t look like a crash. It looks like a confident answer with a wrong source, or an action taken for a reason nobody can quite reconstruct. Without good AI observability, teams end up guessing. Was the retrieval corpus missing the right document? Did the reranker overvalue a misleading snippet? Did the prompt leave too much room for interpretation? Did a tool call return malformed data, then get passed through anyway? If you can’t trace those steps, you’re left with folklore.
The execution layer is the place where caution matters most, because mistakes can reach outside the system. A model that picks the wrong article is annoying. A model that sends the wrong email, changes the wrong record, or triggers the wrong workflow can create real mess. That’s why tool choice and action choice usually need tighter controls than query rewriting or summary generation. Even when an LLM is allowed to participate, the final action often needs a hard constraint layer, explicit confirmation, or a scoped function call with strict inputs and outputs. The function calling documentation for Gemini is a decent example of how model output can be boxed into a narrower shape before anything happens downstream.
The same logic applies when a system has to talk to external tools or agents. Standards and boundaries matter more than clever prompts. If the model is choosing between actions, the interface should be boring in the best possible way: named parameters, typed fields, and a short list of allowed operations. The MCP docs from Anthropic are useful here because they frame tools as something a model can call without pretending the model should improvise its own protocol. That distinction sounds small until a tool call goes sideways and you’re trying to figure out which layer invented the problem.
Prompting in production gets harder here too. At the edge, prompt quality mostly affects how well the model parses a query. Deeper in the stack, the prompt can shape what data is fetched, what gets ranked, and what action is taken. A tiny wording change might alter retrieval enough to move the whole response. That’s why teams often separate prompts by layer instead of reusing one giant template everywhere. Retrieval prompts should stay narrow. Reranking prompts should judge relevance with a clear rubric. Execution prompts should be almost stubbornly specific.
Good tracing helps more than clever wording. If your system logs the original input, the rewritten query, the retrieved candidates, the reranked order, the selected tool, and the final action, you can usually find where things drifted. If it doesn’t, you’re flying blind with a dashboard that looks prettier than it is useful. Microsoft’s Semantic Kernel guidance on observability is a sensible reminder that AI observability isn’t a bonus feature. It’s the only way to tell whether a deeper model placement is actually improving the system or just moving the mistakes into harder-to-see places.
Used carefully, deeper placement can make a system feel much smarter than a simple edge-layer rewrite ever could. It can also make a system much harder to trust. That’s the bargain. When the model starts choosing what to fetch, what to rank, and what to do next, you get better intent matching in domains that don’t fit clean labels. You also inherit a larger mess when something goes wrong.
A practical way to choose the right layer in production
Once you’ve seen what happens when a model sits near retrieval, ranking, or execution, the next question gets less glamorous and a lot more useful: where should it actually live in your system?
A good starting filter is domain predictability. Some domains have tidy rules, repeated phrasing, and a fairly small set of valid outcomes. Tax classification, ticket routing, product search cleanup, and internal knowledge lookup often fit that mold. In those cases, a model can sit a bit deeper because the system around it can still catch odd cases. Other domains are messier. Customer support emails written at 2 a.m. Incident reports, open-ended sales requests, or agentic workflows that touch external tools tend to produce odd edge cases all the time. When the input space keeps shifting under your feet, deeper model placement gets harder to trust.
That doesn’t mean “stable equals safe” and “messy equals forbidden.” It’s more about how much damage a bad guess can do. If the model misreads a search query, the user might get irrelevant results and try again. Annoying, yes. If it chooses the wrong action in an execution step, maybe it sends the wrong email, opens the wrong ticket, or updates the wrong record. That’s a different class of problem. The more expensive the mistake, the more constrained the model path should be. Put another way, if you’d hesitate to let a junior engineer make that call without a review, don’t let the model make it with no guardrails.
Put the model where a bad guess costs the least, then widen its reach only after you can explain what it did.
That last part matters more than teams sometimes admit. If you can’t tell why the model produced a result, you probably can’t safely expand its role. Before you let it shape retrieval or select actions, build boring but reliable observability around the path. Log the raw input, the normalized input, the prompt or tool call, the retrieved candidates, the final output, and the fallback that fired when the model hesitated. Keep traces tied to user-facing requests so one weird result can be followed all the way back through the stack. Run evals on real examples, not just polished demo cases. A model that looks fine in a notebook can behave very differently once it starts seeing typos, partial context, duplicate records, and the occasional user who pastes a wall of text with three unrelated asks.
Fallback logic deserves its own seat at the table. If the model can’t extract a confident intent, route to a simpler rule. If reranking looks unstable, keep the original ranking and log the miss. If an action would have external effects, require a threshold, a confirmation step, or a narrower tool set. None of this is glamorous. It’s the software equivalent of checking whether the door is locked before you leave. Not thrilling, but it saves headaches.
A practical rollout usually starts at the edge. Let the model rewrite queries, normalize fields, classify tickets, or map free text into a schema. Measure accuracy, latency, and failure modes. Then widen its influence only if the system is understandable and the gains are real. Maybe deeper placement improves retrieval quality by a lot. Maybe it just makes debugging miserable for a 2% lift. Either answer can be fine. What matters is that you found out with evidence instead of hope.
Teams sometimes want to jump straight to fancy agentic workflows because the demo looked neat. Fair. Demos are good at that. In production, though, the disciplined move is usually the unflashy one: start shallow, record everything, set a cutoff for uncertainty, and earn the right to let the model do more. That gives you control when you need it and flexibility when the data says you can afford it.
