How to Make an AI Agent: Four Decisions, and the Model Is the Least of Them

The four decisions that matter

Most teams begin by choosing a model. That choice has almost no influence on whether the agent solves a real problem. The four decisions that decide success are the source of knowledge, the scope of answers, the memory between turns, and the escalation path. Each can be configured independently of the model, and each can be got wrong in a way that makes the agent useless within minutes of live traffic. They are the same four decisions whichever tool you build on, including the no-code AI agent builders.

The model is the engine, but the four decisions are the steering wheel, the brakes, the fuel and the map. A powerful engine with no steering will crash.

Each of those four decisions can be tested separately, and each has a failure mode that is visible within minutes of launch. The source decision determines whether the agent can answer at all. The scope decision determines what it is allowed to say. The memory decision determines whether it feels like a conversation or a series of isolated questions. The escalation decision determines whether a human ever sees the user.

Where the answers come from

An agent that has no source will invent. It will confidently state a return policy that your website does not contain, or quote a price that changed last month. The first decision is therefore which documents, databases and live endpoints the agent may read.

Start with the material that already answers most questions. For a consumer brand that is usually the help center, the product pages and the order‑status system. For a SaaS product it is the documentation, the status page and the billing portal. Feed the agent only what it needs, and refresh that feed on a schedule that matches how often the information changes.

An append‑only audit log, which Asyntai provides on Enterprise, lets you review every answer the agent gave and trace it back to the source document. If the source changes, the next answer should reflect the change; if it does not, the log shows where the pipeline broke.

The cost of getting this wrong is that the agent becomes a liability. Every confident wrong answer costs a support ticket, and the user loses trust in the next answer as well.

Why the source must be auditable

The audit log is not a nice‑to‑have. It is the difference between an agent you can trust and one you have to hope is right. When a user reports that the agent gave incorrect information, you need to know which document it read and when it read it. Without that trace you are guessing whether the source was out of date, the feed failed to refresh, or the agent misinterpreted a correct document.

An append‑only log also protects you from accidental deletion. If a junior member of staff edits a help article and removes a paragraph that contained a key answer, the old version remains in the log. You can compare the answer the agent gave before the edit with the answer it gives now, and you can restore the missing content if it was removed by mistake.

The log should be reviewed weekly. Pick ten recent conversations, follow each answer back to its source, and check that the source still says what the agent claimed. If it does not, the feed is stale and the schedule needs tightening.

Defining the scope

Scope is the list of questions the agent may answer on its own, and the list of questions it must hand over. Write both lists before you write a single prompt.

A useful test is to take the last hundred real inquiries and mark each one as either answerable from your source material or requiring a human. The agent should cover the first group and refuse the second. Refusing is not a failure; it is the correct behaviour. An agent that tries to guess is the one that produces the memorable error.

Set the refusal message explicitly. It should name what the agent can do and offer the next step, such as opening a ticket or joining the queue. A vague apology does not move the conversation forward.

Asyntai’s output classifier is separate from the prompt, so you can define a policy that blocks answers outside the scope without editing the prompt each time. The classifier runs after the model generates a response, which means the model can be creative while the policy stays strict.

Scope and the refusal message

The refusal message is the first place where a user realises the agent is not a person. It must be written with the same care as the answers. A good refusal does three things: it confirms that the question is outside the agent’s reach, it restates what the agent can do, and it offers a concrete next step.

The next step should be a single click. “Open a ticket” is better than “contact support”, because it removes one more decision from the user’s day. If the agent can also estimate a response time, say so. Knowing that a human will reply within two hours changes the feeling of being refused.

Test the refusal message with real users. Ask a colleague who has never seen the agent to ask a question that is clearly outside scope, and see whether they understand what to do next. If they hesitate, the message needs another rewrite.

Memory: what it keeps between turns

Memory is the difference between a chatbot and an agent. Without memory the user must repeat their order number, their previous question and the reason they are contacting you in every turn.

Decide what the agent may retain across a session. Useful items are the user’s identity, the order they are discussing, the last answer they received and any preference they stated. Anything that is not needed for the next turn should be dropped. Keeping irrelevant data increases the chance that the agent will reference something the user never mentioned.

The mistake here is to store everything. A long transcript gives the model more context, but it also gives it more opportunities to confuse two similar questions. Trim the memory to the facts that the next turn actually needs.

Configurable chat retention, which Asyntai offers, lets you set how long a conversation’s memory persists and when it is auto‑deleted. That keeps the store small and reduces the surface for a leak.

Memory hygiene

Memory hygiene is the practice of deciding what to forget. The default instinct is to keep everything, but that instinct is wrong for an agent. Each extra piece of context is a chance for the agent to confuse one user’s order with another’s, or to repeat a detail that the user already corrected.

A practical rule is to keep only the facts that are directly needed for the next turn. If the user is asking about a refund, the agent needs the order number and the reason for the request. It does not need the colour of the product, the date of the original inquiry, or the name of the agent who handled the previous call.

Review the memory schema after the first week of live traffic. Look for fields that were never used in a follow‑up question and remove them. The goal is a memory that is just large enough to feel like a conversation and no larger.

Escalation: when a human takes over

Escalation is the part of the agent that users judge most harshly, because it is the moment the conversation stops being automated. A good escalation feels like a handover, not a rejection.

The agent should state why it is handing over, summarise what it has already learned, and pass that summary to the human. The human then picks up the conversation without asking the user to repeat themselves.

Escalation triggers need to be configured explicitly. Common ones are a request for a refund, a complaint about a delivery, a mention of legal terms, or the user stating that they want to speak to a person. Do not rely on the model to decide; set the rules in your own code.

Per‑tab permissions and per‑website scoping, which Asyntai supports, let you route different types of conversation to different human teams. A billing question goes to finance, a technical issue goes to engineering, and the agent never has to guess which queue is correct.

Escalation as a handover

The moment of handover is where trust is either earned or lost. A user who has been chatting for five minutes expects the human to know everything the agent knew. If the human starts with “can you tell me again what the problem is”, the agent’s earlier work is wasted and the user feels penalised for using the automated channel.

The summary that the agent passes should be short, factual and ordered. It should include the user’s original request, any details the user provided, the agent’s attempted answers, and the reason for the handover. The human can then read it in ten seconds and start with “I see, you are asking about X”.

Test the handover by having a colleague act as the human. They should be able to pick up the conversation without asking the user to repeat anything. If they need to ask for clarification, the summary is not complete enough.

Checks before your agent meets a customer

Before you switch the agent on for real traffic, run these checks on a test version.

1. Ask it the ten most common questions and verify each answer against your published help center.
2. Ask it a question that is outside its scope and confirm that it refuses with a clear handover message.
3. Start a multi‑turn conversation, then ask a follow‑up that depends on something the agent said earlier. Check that it remembers the earlier answer.
4. Feed it a deliberately misleading statement and see whether it corrects the user or repeats the error.
5. Check that any personal data it receives is stored only for the duration of the session and is deleted afterwards.

If any of these checks fail, the agent is not ready. Fix the underlying decision rather than adding more prompts.

Testing in production

The checks above are a starting point, but they are not enough on their own. Before you open the agent to real traffic, run it in a shadow mode where it answers questions but its responses are not shown to the user. Compare its answers to the ones a human would have given, and count the disagreements.

A low disagreement rate is not the goal. The goal is to understand where the agent is confident and wrong, where it is hesitant, and where it refuses correctly. Each of those categories needs a different fix. A confident wrong answer needs a source update. A hesitant answer needs a prompt rewrite. A correct refusal needs no change.

Shadow mode should run for at least a few hundred real inquiries. That is enough to see patterns without exhausting your message allowance. Once the disagreement rate on in‑scope questions is consistently below a threshold you set, you can switch the agent on for real traffic.

What it costs to run an agent

The running cost is measured in messages, not chats. Each chat is a series of back‑and‑forth exchanges, and each exchange is a message. To convert a chat forecast into a message forecast, multiply by the average number of messages per chat.

Asyntai publishes its allowance in messages per month on its pricing page. The self‑serve tiers are:

• Free: 50 messages
• Starter: 800 messages for $39
• Standard: 15,000 messages for $139
• Pro: 50,000 messages for $449

Enterprise is a signed order form with a negotiated allowance and is not sold self‑serve.

The same arithmetic works for any other forecast. Change the chat count or the messages‑per‑chat figure and the maths block updates the total. The plan prices stay the same because they are published figures.

Monitoring after launch

After the agent is live, the work is not finished. You need to watch three numbers: the percentage of conversations that end without a human, the average number of messages per conversation, and the rate of user‑reported errors.

The first number tells you whether the agent is handling the volume you expected. If it is too low, the scope may be too narrow or the source may be missing answers. The second number tells you whether conversations are becoming longer, which may indicate that the agent is struggling and the user is repeating themselves.

The third number is the most important. Every error report is a chance to improve the source, the scope, the memory or the escalation rules. Fix the underlying decision, not the wording of a single prompt. A prompt can be rewritten in minutes; a missing source document can take days to replace.

Review these numbers weekly for the first month, then monthly. An agent that is not improving is slowly becoming a liability, because the business changes around it while the agent stays the same.