Lesson 5.7 · LangSmith & Tooling

Packaging & Deployment

Turn the Module 3 agent into something other people can use: an installable command (ask-agent "question") and a FastAPI service with /ask and /health. Settings from environment variables, clear errors, a timeout, a limit on parallel runs - and the bug we found only because we tested the packaged version.

deployment

What you will be able to do

Split an agent into a reusable core, a command-line entry point and an HTTP entry point
Read settings (model, URL, limits) from environment variables
Make a command with pyproject.toml, and use exit codes and stderr correctly
Serve the agent with FastAPI: build once at startup, validate input, health check
Protect the service: a timeout, a limit on parallel runs, clear error codes
Test the packaged version - not only the notebook version

The idea, in plain English

So far every agent in this course ran inside one script that only you could start. To be useful, other people - and other programs - must be able to use it. That means packaging (making it installable, with a clear entry point) and deployment (running it where others can reach it).

We take Lesson 3.4’s agent (llama3 with a weather tool and a calculator) and give it two "doors": a command-line tool, and a small web service with FastAPI (see the FastAPI course for the basics). The agent itself is written once and shared by both.

Everything below was run: LangGraph 1.2.14, langchain-ollama 1.1.0, FastAPI 0.143.0, uvicorn 0.54.0, llama3 on Ollama.

Worked example: The weather-and-calculator agent from Lesson 3.4, packaged as helpdesk-agent: installed with pip, run from the terminal, and served over HTTP - with three questions sent at the same time.

workflowOne agent, two doorsstep 1 / 3

a person

in a terminal

another program

a website, a script

cli.py

ask-agent "..."

✓

server.py

POST /ask, GET /health

agent.py

build_graph() - written once

Ollama

llama3

React Flow

1 - The command-line door

pip install makes a real command, ask-agent. The answer goes to stdout, the info line to stderr, and the exit code tells scripts what happened.

umbrella question

3 steps, 6.0 s

exit code

0 = answered

Ollama down

exit code 2

> answer.txt

only the answer

The same graph, used from a terminal and over HTTP. Real results from our runs.

Words you will see in this lesson

A few words about making software usable by others.

Small dictionary

PackagingMaking your code installable with pip, with a name, a version and its dependencies.

DeploymentRunning your code where other people or programs can reach it.

Entry pointWhere a program starts: a command, or an HTTP endpoint.

pyproject.tomlThe file that describes a Python package.

Environment variableA setting given to a program from outside: OLLAMA_URL=http://...

Exit codeA number a command returns when it ends. 0 means OK; other numbers mean a problem.

stdout / stderrTwo output streams: results, and messages for people.

Health checkAn endpoint that says "I am working" - used by servers and monitors.

An everyday example: a home cook opens a restaurant

A home cook makes great food - but only family can eat it, and only when the cook is home. (A script only you can run.)
Opening a restaurant means a fixed menu, a front door, and opening hours. (A package with clear entry points.)
The same kitchen serves the dining room and a take-away window. (One agent, a command and an HTTP service.)
A restaurant also needs rules: how many orders at once, what to say when the oven breaks, how long a customer waits. (Limits, error codes, timeouts.)

Step 1 - one core, two doors

First, separate the agent from the ways people use it:

agent.py: the graph and the tools. No input(), no print(), no web code. It has a function build_graph() that returns the compiled graph.
cli.py: reads the question from the command line, runs the graph, prints the answer.
server.py: reads the question from HTTP, runs the graph, returns JSON.
Settings come from environment variables with safe defaults: OLLAMA_URL, AGENT_MODEL, AGENT_MAX_STEPS. The same code then runs on your laptop and on a server - only the variables change.

The project

helpdesk-agent/
  pyproject.toml
  helpdesk_agent/
    __init__.py
    agent.py      # the graph - written once
    cli.py        # door 1: the terminal
    server.py     # door 2: HTTP

helpdesk_agent/agent.py

"""The agent: Lesson 3.4's ReAct loop, with settings read from the environment."""
import ast
import operator
import os
from typing import Literal, TypedDict

from langchain_ollama import ChatOllama
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel, Field

OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")   # where the model runs
MODEL = os.environ.get("AGENT_MODEL", "llama3")
MAX_STEPS = int(os.environ.get("AGENT_MAX_STEPS", "6"))

WEATHER = {"mumbai": "Rainy, 27°C", "delhi": "Sunny, 34°C", "hyderabad": "Cloudy, 29°C"}
OPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv}


def get_weather(city: str) -> str:
    return WEATHER.get(city.strip().lower(), "No weather data for that city.")


def calculate(expression: str) -> str:
    def ev(node):
        if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
            return node.value
        if isinstance(node, ast.BinOp) and type(node.op) in OPS:
            return OPS[type(node.op)](ev(node.left), ev(node.right))
        raise ValueError("only + - * / and numbers are allowed")
    try:
        return str(ev(ast.parse(expression, mode="eval").body))
    except Exception as error:
        return f"Error: {error}"


TOOLS = {"get_weather": get_weather, "calculate": calculate}


class Step(BaseModel):
    action: Literal["calculate", "get_weather", "final_answer"]
    action_input: str = Field(description="The expression, the city name, or the final answer text")


SYSTEM = (
    "You answer questions using two tools: get_weather(city) and calculate(expression). "
    "Choose ONE next action. Use a tool for each piece of information you do not have yet. "
    "Never repeat an action that already has an Observation. "
    "When the Observations contain everything needed, reply with final_answer."
)


class AgentState(TypedDict):
    question: str
    history: list
    step: int
    action: str
    action_input: str
    answer: str


def build_graph():
    decide = ChatOllama(model=MODEL, base_url=OLLAMA_URL, temperature=0).with_structured_output(Step)

    def agent(state: AgentState):
        transcript = "\n".join([f"Question: {state['question']}"] + state["history"])
        step = decide.invoke([("system", SYSTEM), ("human", transcript)])
        update = {"step": state["step"] + 1, "action": step.action, "action_input": step.action_input}
        if step.action == "final_answer":
            update["answer"] = step.action_input
        return update

    def tools(state: AgentState):
        call = f"Action: {state['action']}({state['action_input']!r})"
        if call in state["history"]:                                   # the same call again (Lesson 3.4)
            done = state["history"][state["history"].index(call) + 1]
            note = f"Observation: you already have this result ({done[13:]}). Do not repeat it; answer now."
            return {"history": state["history"] + [call, note]}
        result = TOOLS[state["action"]](state["action_input"])
        return {"history": state["history"] + [call, f"Observation: {result}"]}

    def give_up(state: AgentState):
        return {"answer": f"Stopped after {state['step']} steps without an answer."}

    def should_continue(state: AgentState) -> Literal["tools", "give_up", "__end__"]:
        if state["answer"]:
            return END
        if state["step"] >= MAX_STEPS:
            return "give_up"
        return "tools"

    builder = StateGraph(AgentState)
    builder.add_node("agent", agent)
    builder.add_node("tools", tools)
    builder.add_node("give_up", give_up)
    builder.add_edge(START, "agent")
    builder.add_conditional_edges("agent", should_continue, ["tools", "give_up", END])
    builder.add_edge("tools", "agent")
    builder.add_edge("give_up", END)
    return builder.compile()


def start_state(question: str) -> dict:
    return {"question": question, "history": [], "step": 0, "action": "", "action_input": "", "answer": ""}

Step 2 - the command-line door

cli.py follows the rules every good command follows:

argparse reads the question and an optional --json flag, and makes --help for free.
It checks Ollama FIRST, with a 3-second timeout, and stops with a clear message if it is down. No long Python traceback for the user.
The answer goes to stdout. The "(3 steps, 6.0 s, llama3)" line goes to stderr - so a script that saves the output gets only the answer.
Exit codes: 0 = answered, 1 = the agent gave up, 2 = the model server is unreachable. Other scripts can check these.

helpdesk_agent/cli.py

"""Command-line entry point: ask-agent "question" """
import argparse
import json
import sys
import time

import httpx

from helpdesk_agent.agent import MODEL, OLLAMA_URL, build_graph, start_state


def main() -> int:
    parser = argparse.ArgumentParser(prog="ask-agent", description="Ask the helpdesk agent a question.")
    parser.add_argument("question", help="the question, in quotes")
    parser.add_argument("--json", action="store_true", help="print the result as JSON")
    args = parser.parse_args()

    try:                                                     # fail fast and clearly if Ollama is down
        httpx.get(f"{OLLAMA_URL}/api/tags", timeout=3).raise_for_status()
    except httpx.HTTPError:
        print(f"error: cannot reach Ollama at {OLLAMA_URL} - is it running?", file=sys.stderr)
        return 2

    start = time.perf_counter()
    result = build_graph().invoke(start_state(args.question), {"recursion_limit": 20})
    seconds = round(time.perf_counter() - start, 1)
    gave_up = result["answer"].startswith("Stopped after")

    if args.json:
        print(json.dumps({"answer": result["answer"], "steps": result["step"], "seconds": seconds, "model": MODEL}))
    else:
        print(result["answer"])
        print(f"({result['step']} steps, {seconds} s, {MODEL})", file=sys.stderr)
    return 1 if gave_up else 0


if __name__ == "__main__":
    sys.exit(main())

Step 3 - make it a real command

pyproject.toml describes the package. The important lines:

dependencies: what pip must install, with version ranges (Lesson 4.3 explained why: a future major version must not break you).
[project.scripts] ask-agent = "helpdesk_agent.cli:main" - pip creates a command called ask-agent that calls main().
pip install -e . installs it in "editable" mode: you can keep changing the code without installing again.

pyproject.toml

[project]
name = "helpdesk-agent"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "langgraph>=1.2,<2",
    "langchain-ollama>=1.1,<2",
    "fastapi[standard]>=0.143",
    "httpx",
]

[project.scripts]
ask-agent = "helpdesk_agent.cli:main"     # the command name = module:function

[build-system]
requires = ["setuptools>=69"]
build-backend = "setuptools.build_meta"

Install and use it - real runs

$ pip install -e .
$ ask-agent --help
usage: ask-agent [-h] [--json] question

Ask the helpdesk agent a question.

positional arguments:
  question    the question, in quotes

options:
  -h, --help  show this help message and exit
  --json      print the result as JSON

$ ask-agent "Should I bring an umbrella in Mumbai, and what is 12 * 8?"
(3 steps, 6.0 s, llama3)
Based on the weather in Mumbai, it is likely to rain, so you should bring an umbrella. Additionally, the calculation 12 * 8 equals 96. Therefore, the final answer is: Bring an umbrella and the result of the calculation is 96.

$ ask-agent --json "What is 125 * 48?"
{"answer": "6000", "steps": 3, "seconds": 3.8, "model": "llama3"}

$ ask-agent "What is 250 / 4?" > answer.txt; echo "exit=$?"
(4 steps, 5.3 s, llama3)
exit=0
$ cat answer.txt
62.5

$ OLLAMA_URL=http://localhost:11999 ask-agent "hi"; echo "exit=$?"
error: cannot reach Ollama at http://localhost:11999 - is it running?
exit=2

Reading the command-line output

The info line "(3 steps, 6.0 s, llama3)" printed BEFORE the answer here. stderr is shown at once; stdout waits a moment when it is not a terminal. That is normal.
> answer.txt saved only "62.5" - the info line went to stderr and stayed on the screen.
With Ollama down, the user got one clear line and exit code 2 - not a 40-line traceback.
--json is for other programs: one line, easy to parse.

Step 4 - the HTTP door

server.py turns the agent into a web service. Read it in five parts:

lifespan: the graph is built ONCE when the server starts, not on every request.
Question: a Pydantic model - the question must be 3 to 500 characters. Anything else gets 422 automatically (FastAPI Module 2).
GET /health: checks that Ollama answers. Load balancers and monitors call this.
busy = asyncio.Semaphore(2): at most 2 questions run at the same time; others wait their turn. A local model can only do so much at once.
asyncio.wait_for(..., timeout): a run that takes too long gets 504, and Ollama being down gets 503 - not a crash.

helpdesk_agent/server.py

"""HTTP entry point: fastapi run helpdesk_agent/server.py"""
import asyncio
import os
import time
from contextlib import asynccontextmanager

import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

from helpdesk_agent.agent import MODEL, OLLAMA_URL, build_graph, start_state

TIMEOUT_SECONDS = float(os.environ.get("AGENT_TIMEOUT", "60"))
busy = asyncio.Semaphore(2)          # at most 2 questions run at the same time


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.graph = build_graph()  # build once, at startup - not on every request
    yield


app = FastAPI(title="Helpdesk agent", lifespan=lifespan)


class Question(BaseModel):
    question: str = Field(min_length=3, max_length=500)


class Answer(BaseModel):
    answer: str
    steps: int
    seconds: float


@app.get("/health")
async def health():
    try:
        async with httpx.AsyncClient(timeout=3) as client:
            (await client.get(f"{OLLAMA_URL}/api/tags")).raise_for_status()
    except httpx.HTTPError:
        raise HTTPException(status_code=503, detail="model server unreachable")
    return {"status": "ok", "model": MODEL}


@app.post("/ask")
async def ask(body: Question) -> Answer:
    start = time.perf_counter()
    async with busy:                                     # wait here if 2 runs are already going
        try:
            result = await asyncio.wait_for(
                app.state.graph.ainvoke(start_state(body.question), {"recursion_limit": 20}),
                timeout=TIMEOUT_SECONDS,
            )
        except asyncio.TimeoutError:
            raise HTTPException(status_code=504, detail="the agent took too long")
        except httpx.ConnectError:
            raise HTTPException(status_code=503, detail="model server unreachable")
    return Answer(answer=result["answer"], steps=result["step"], seconds=round(time.perf_counter() - start, 1))

Run it and call it - real runs

$ fastapi run helpdesk_agent/server.py --port 8601

$ curl -i localhost:8601/health
HTTP/1.1 200 OK
{"status":"ok","model":"llama3"}

$ curl -X POST localhost:8601/ask -H 'content-type: application/json' \
       -d '{"question": "Should I bring an umbrella in Mumbai, and what is 12 * 8?"}'
{"answer":"Based on the weather in Mumbai, it is likely to rain, so you should bring an umbrella. Additionally, the calculation 12 * 8 equals 96. Therefore, the final answer is: Bring an umbrella and the result of the calculation is 96.","steps":3,"seconds":5.9}

$ curl -X POST localhost:8601/ask ... -d '{"question": "hi"}'                    -> 422
$ AGENT_TIMEOUT=2 fastapi run ...   then the umbrella question                    -> 504 {"detail":"the agent took too long"}
$ OLLAMA_URL=http://localhost:11999 fastapi run ...   then GET /health           -> 503 {"detail":"model server unreachable"}
                                                        and POST /ask             -> 503 {"detail":"model server unreachable"}

Step 5 - three questions at the same time

We sent three questions at once. With Semaphore(2), two ran together and the third waited:

17 * 23 + 9 finished first, after 6.3 s.
Delhi weather finished at 13.1 s.
250 / 4 waited for a free place and finished last, at 15.8 s.
Alone, each takes about 4 to 6 seconds. Together they share one local model, so each one gets slower. Measure this before you promise users a response time (Lesson 5.6).

Output - three requests sent together

{"answer":"400","steps":3,"seconds":6.3}  <- What is 17 * 23 + 9?, total 6.286767s
{"answer":"Sunny, 34°C","steps":5,"seconds":13.1}  <- Weather in Delhi?, total 13.142510s
{"answer":"62.5","steps":4,"seconds":15.8}  <- What is 250 / 4?, total 15.798997s

The bug we found by testing the package

Our FIRST packaged agent did not have Lesson 3.4’s repeat guard - we left it out to keep the code short. The test results:

"Weather in Delhi?" and "What is 250 / 4?" both answered "Stopped after 6 steps without an answer." - alone AND in parallel.
The trace showed why: llama3 called get_weather('Delhi') six times in a row. The prompt rule "Never repeat an action" was not enough.
We put the repeat guard back. Then every question was answered (the outputs above).
The lesson: ship the version you tested. A shortcut made while packaging changed the behaviour.

Output - the first version, traced with stream_mode="updates"

agent {'step': 1, 'action': 'get_weather', 'action_input': 'Delhi'}
tools ["Action: get_weather('Delhi')", 'Observation: Sunny, 34°C']
agent {'step': 2, 'action': 'get_weather', 'action_input': 'Delhi'}
tools ["Action: get_weather('Delhi')", 'Observation: Sunny, 34°C']
agent {'step': 3, 'action': 'get_weather', 'action_input': 'Delhi'}
tools ["Action: get_weather('Delhi')", 'Observation: Sunny, 34°C']
agent {'step': 4, 'action': 'get_weather', 'action_input': 'Delhi'}
tools ["Action: get_weather('Delhi')", 'Observation: Sunny, 34°C']
agent {'step': 5, 'action': 'get_weather', 'action_input': 'Delhi'}
tools ["Action: get_weather('Delhi')", 'Observation: Sunny, 34°C']
agent {'step': 6, 'action': 'get_weather', 'action_input': 'Delhi'}
give_up {'answer': 'Stopped after 6 steps without an answer.'}

Watch out: Keep a small set of test questions with known answers (Lesson 5.4) and run them against the packaged command or the running service - not only against your notebook.

Before real users

This service is a starting point. Before real users, add:

Authentication: our /ask is open to anyone who can reach it (FastAPI course, security module).
Streaming: for answers that take seconds, stream progress as in Lesson 3.9.
Tracing and logs: record every run (Lesson 5.1) so you can see what went wrong.
A container: package the service and its dependencies with Docker (see the Docker course) so it runs the same everywhere.
A real model server: Ollama on a laptop is fine to learn; production needs a machine sized for the traffic you measured.

The recipe

1. Put the agent in one module with a build function. No input(), no print().
2. Read settings from environment variables, with safe defaults.
3. Add a command: argparse, stdout for results, stderr for messages, meaningful exit codes, [project.scripts].
4. Add a service: build once at startup, validate input, /health, a timeout, a limit on parallel runs, clear error codes.
5. Test the packaged version with known questions - alone and in parallel.

Packaging & deployment at a glance

Setting from outside

With a default.

os.environ.get("OLLAMA_URL", "http://localhost:11434")

A command

In pyproject.toml.

[project.scripts]
ask-agent = "helpdesk_agent.cli:main"

Install for development

Editable.

pip install -e .

Messages for people

Not mixed with results.

print(..., file=sys.stderr)

Exit code

From main().

sys.exit(main())   # return 0, 1 or 2

Build once

At server start.

@asynccontextmanager
async def lifespan(app): app.state.graph = build_graph(); yield

Limit parallel runs

Others wait.

busy = asyncio.Semaphore(2)
async with busy: ...

Timeout

504 when too slow.

await asyncio.wait_for(graph.ainvoke(...), timeout=60)

Try it yourself

The code does not change. Swap the content string and the program does something else entirely.

Another model

“Run AGENT_MODEL=gemma3 ask-agent "..." - does structured output still work?”

Exit code 1

“Set AGENT_MAX_STEPS=1 and ask the umbrella question. Which exit code do you get?”

One at a time

“Change the Semaphore to 1 and send three questions together. How do the total times change?”

Read /docs

“Open localhost:8601/docs and call /ask from the browser.”

What usually goes wrong

Building the graph on every request

Build it once at startup with lifespan and reuse it.

✗ @app.post("/ask")
async def ask(body):
    graph = build_graph()

✓ app.state.graph = build_graph()   # in lifespan

Settings written into the code

A hard-coded localhost URL breaks on the server. Read it from an environment variable.

Printing messages to stdout

A script that saves the output then gets your info lines too. Messages go to stderr.

No timeout, no limit

One slow question - or many at once - can block the whole service. Add a timeout and a Semaphore.

Testing only the notebook

Our packaged agent lost the repeat guard and gave up on easy questions. Test the thing you ship.

Key points

Keep the agent in one module with a build function; add a command and a service around it.
Read settings from environment variables, with safe defaults.
[project.scripts] in pyproject.toml turns a function into a command.
Commands: results to stdout, messages to stderr, meaningful exit codes.
Services: build once at startup, validate input, /health, timeout (504), model down (503), a limit on parallel runs.
Test the packaged version - ours lost the repeat guard and gave up until we did.

Quick check before you move on

Why does agent.py have no print() or input()?

So both the command and the web service can use it; each door handles input and output its own way.

What does [project.scripts] do?

It makes pip create a command (ask-agent) that calls a Python function (helpdesk_agent.cli:main).

What did /ask return when the agent took longer than the timeout?

504 with "the agent took too long".

Why did the third parallel question take 15.8 s?

The Semaphore allowed only 2 runs at once, so it waited, and all runs shared one local model.

Quiz

1.
A script runs ask-agent "..." > answer.txt. Why does the file contain only the answer?
2.
Which exit code did ask-agent return when Ollama was down, and why is that useful?
3.
Why build the graph in lifespan?
4.
What bug did testing the packaged agent find?

Interview questions

How would you expose an LLM agent as an API?

Keep the agent in a module with a factory, build it once at startup, validate requests, add a health check, enforce timeouts and concurrency limits, map failures to clear status codes, stream long answers, and add auth, tracing and evaluation before production.

Why limit concurrency for an agent service?

Each run holds model capacity for seconds; unlimited parallel runs overload a shared model server and make every request slower or fail. A semaphore or queue keeps latency predictable.

What belongs in configuration rather than code?

Model names, endpoints, limits, timeouts and secrets - read from environment variables or a secrets manager - so the same build runs in every environment.

Comments

Loading comments...