The AI Engineering Handbook
A practical guide to architecture, reliability, cost, and testing for teams shipping AI products.
Ganesh Ghatti
Founder1. Start with a clear boundary around the model
A model response is one part of an AI product. Your application still needs to decide who may ask, what information the model may see, which actions it may take, and what happens when it cannot answer. Write those decisions down before you choose a model.
Input
Check identity, validate the request, and remove data the model does not need.
Decision
Give the model a narrow job, approved context, and explicit tool permissions.
Output
Check the result before showing it or allowing it to change a system.
This is the practical difference between a convincing demo and a product people can rely on. The demo proves the model can answer. The product also has to handle the second request, the failed request, and the request it should refuse.
The three layers behind a useful answer
Think in terms of a gateway, a processing layer, and a response layer. The gateway checks identity and limits requests. Processing gathers context and decides whether a tool is needed. The response layer checks the result, records what happened, and shows the user either an answer or a clear next step. Keeping these jobs separate makes failures easier to locate.
2. Build a request path that can fail safely
Keep the route from user to answer easy to inspect. A complicated chain of agents is harder to debug than a straightforward request with clear checks at each step.
Receive
Authenticate the user and set a request limit.
Prepare
Retrieve only the context this task needs.
Generate
Call the model with a time limit and a defined output shape.
Verify
Validate the answer or hand the task back to a person.
Set timeouts and retry rules deliberately. If a tool or model is unavailable, a clear fallback is better than a request that spins indefinitely. Log enough detail to investigate a failure without recording more personal data than you need.
Keep tool calls narrow
A model may ask to use a tool, but your application decides whether to run it. Check the requested action, its inputs, and the user's permission first. Whether tools are connected directly or exposed through a shared interface such as MCP, the same rule applies: the model should not gain more access than the task requires.
Plan for a provider outage
A circuit breaker stops sending requests to a failing service for a short period. Pair it with a timeout and a fallback response. Retrying every failed request without a limit can make an outage slower and more expensive for everyone.
3. Choose the right way to give the model context
Prompting, retrieval, and fine-tuning solve different problems. They are not three levels of the same upgrade path.
| Approach | Use it when | Watch for |
|---|---|---|
| Prompt and examples | You need clearer instructions or a consistent response shape. | Instructions that grow until nobody can maintain them. |
| Retrieval (RAG) | Answers must use current documents or company knowledge. | Missing sources, stale documents, and permission leaks. |
| Fine-tuning | A repeat task needs a learned response pattern that prompting cannot reliably provide. | Training data quality and a harder update cycle. |
For a knowledge-heavy product, test retrieval separately from the final answer. If the right document never reaches the model, a better prompt will not repair the missing evidence. Microsoft's RAG design guide walks through this distinction.
Embeddings and memory are different jobs
Embeddings help a search system find material that is similar in meaning to a question. They do not tell you whether a document is current, trustworthy, or visible to this user; those checks still belong in your retrieval design. Memory is about what the product keeps between interactions. Save only what helps the next task, set a retention period, and give users a way to correct or remove it.
4. Measure speed and cost per completed task
A cheap model call can still make an expensive product if it retries often, fetches too much context, or requires a person to correct the result. Measure the whole task, not the price of one API request.
Time to useful answer
Record retrieval time, model time, tool time, and the delay the user actually experiences. Set a budget for each step.
Cost to finish the job
Include retries, failed calls, long context, storage, and human review. Compare that total with a successful outcome.
Cache stable information where appropriate, limit unnecessary context, and reserve slower or more expensive models for tasks that need them. Keep rate-limit and provider failures in your test plan; they will happen eventually.
Model settings are part of the test
Temperature changes how varied a model's responses can be. A lower setting may help a repeat extraction task stay consistent, but it does not guarantee correctness. Record the model, prompt, settings, and tool versions alongside each evaluation result so you can explain why behavior changed.
5. Test the work your users actually do
Build a small set of real tasks before launch. Include easy requests, ambiguous ones, missing information, and requests the product must decline. For each task, write down what a good outcome looks like.
A useful evaluation record
Question asked → source or tool used → answer given → expected answer → pass or fail → reason for failure.
Run the same set after every meaningful prompt, retrieval, model, or tool change. For search-backed answers, check whether the source was retrieved as well as whether the final answer was correct. Microsoft's evaluation guidance describes groundedness, completeness, and relevance as separate checks.
A working prototype can fail after launch because real users phrase requests differently, documents change, and tools return unexpected data. Use those failures to expand your test set. Do not treat a single successful demo as a reliability measure.
6. Launch with a way to see and correct mistakes
Start with a narrow task and a small group of users. Watch the requests that fail, the places people take over, and the cases where the system sounds confident without enough evidence. Those examples should become new tests.
Before opening access, decide what the system will do when it reaches a provider rate limit. Queue work when waiting is acceptable, give the user a useful message when it is not, and alert the team if failures keep rising. A graceful limit is part of the product experience.
Before release
Agree on permissions, a human handoff, and the cases the AI must not handle.
During the pilot
Review failed tasks, response times, and the cost of completed work.
After release
Keep a rollback path and rerun evaluations when anything important changes.
A few questions worth asking before launch
What happens if the model is wrong? Decide whether to correct, retry, or hand the task to a person before a user encounters the problem.
Do we need a vector database? Only if the retrieval task calls for one. Start with the smallest search setup that passes your own tests.
When should we fine-tune? After simpler prompts and retrieval have been tested against the same examples, not as a default first step.
Building an AI product for real users?
See how we scope, test, and launch AI systems around a specific job your team needs done.
Founder at Squirrel Technologies, architecting production AI systems, workflow automations, and rapid 2-week MVPs for fast-moving businesses.
Keep exploring