Interview: Evaluation and Observability

What you will be able to do

• ✓Explain what to trace and why
• ✓Design an evaluation set for an agent
• ✓Use and judge LLM-as-judge evaluation
• ✓Evaluate trajectories, not only answers
• ✓Design guardrails and cost monitoring

The idea, in plain English

Anyone can build an agent demo. Interviewers want to know whether you can prove it works, find out why it fails, and stop it from getting worse.

These questions cover observability (seeing what happened), evaluation (measuring how good it is) and guardrails (stopping bad inputs and outputs).

Module 5 did all three with LangSmith; the ideas are the same with any tool.

Worked example: Fourteen questions on measuring agents, tied to the LangSmith tracing, evaluation and guardrail lessons of Module 5.

Basic: What is tracing, and why do agents need it?

Tracing records every step of a run - each model call with its prompt and output, each tool call with arguments and result, plus tokens and time - as a tree you can inspect. Agents take different paths each time, so without traces you cannot tell why one failed.

• •Lesson 5.2 sets up LangSmith tracing; Lesson 5.3 reads a trace.

Basic: What is the difference between offline and online evaluation?

Offline evaluation runs the agent on a fixed test set before release. Online evaluation measures real traffic after release - user feedback, success signals, sampled reviews.

• •Offline catches regressions; online catches what your test set missed.

Basic: What metrics would you track for an agent?

Task success rate, answer correctness, tool-call accuracy, number of steps, latency, tokens and cost per task, error rate, and user feedback.

• •Pick a few that match the product goal.
• •Lesson 5.6 measures cost and latency.

Intermediate: How do you build an evaluation dataset?

Collect real or realistic inputs, write the expected answer or the criteria for a good answer, include hard and edge cases, and add every production failure as a new case.

• •Start small (20-50 cases) but representative.
• •Keep it versioned. Lesson 5.4.

Intermediate: What is LLM-as-judge, and what are its problems?

Using a model to grade outputs against a rubric - useful when answers are free text. Problems: the judge can be biased (toward longer answers or its own style), inconsistent, and wrong - so check it against human labels.

• •Use clear rubrics and yes/no criteria.
• •Prefer exact checks wherever possible.

Intermediate: What is trajectory evaluation?

Evaluating the path, not just the final answer: were the right tools called, with the right arguments, in a sensible order, without wasted steps?

• •A right answer by a wrong path may fail next time.
• •Compare the trajectory with a reference or with rules.

Intermediate: What are guardrails?

Checks around the agent that block or fix bad inputs and outputs - for example rejecting prompt-injection attempts, validating output format, removing personal data, or refusing off-topic requests.

• •Enforce in code; prompts alone are not reliable.
• •Lesson 5.5.

Advanced: How do you stop an agent from getting worse when you change a prompt or model?

Run the evaluation set before every change goes live, compare scores with the current version, block the release on regressions, and keep monitoring the same metrics in production.

• •Treat prompts and models like code: versioned and tested.

Advanced: How do you find out why an agent fails in production?

Find failing runs (errors, low feedback, long runs), open their traces, locate the first wrong step - retrieval, tool choice, arguments, or reasoning - fix it, and add the case to the evaluation set.

• •Group failures by cause to fix the biggest first.

Advanced: How would you monitor cost and latency in production?

Record tokens, model, time and cost per call in traces, aggregate per task, user and feature, set budgets and alerts, and look at the slowest and most expensive runs regularly.

• •Usually a few runaway runs dominate the bill.

Advanced: How do you evaluate an agent that takes actions, not just answers?

Run it in a sandbox with fake or test systems, check the final state of those systems (was the ticket created correctly?), and check that no forbidden actions were attempted.

• •State-based checks are more reliable than reading the agent's text.

Advanced: How many test cases are enough?

Enough to cover the main task types and known failures, and to make score differences meaningful - often tens to a few hundred. Grow it from real failures over time.

• •Small differences on 20 cases may be noise; repeat runs for non-deterministic setups.

Advanced: What is the difference between a guardrail and an evaluation?

A guardrail runs on every request in real time and blocks or fixes problems. An evaluation measures quality on a test set or sample, usually offline, to guide changes.

• •Guardrails protect users now; evaluations improve the system over time.

Advanced: How do you collect useful user feedback for an agent?

Make it easy (thumbs up/down, a short reason), link feedback to the trace of that run, and review negative feedback regularly to create new test cases.

• •Feedback without the trace is hard to act on.

Interview questions