Agent systems and standards · Chapter 7 of 8
Observability: traces and runs
Recording every model call, tool call and step as a tree of runs, and reading it to find what is slow or broken.
Goals
- Find the slowest model or tool run in the trace.To do
- Find the failed tool run and its parent.To do
- Export the runs in LangSmith's format.To do
- Pass the check.To do
Step 1 of 4: The read
Runs and traces
An agent's answer hides many steps. Tracing records each one as a run: a graph, a node, a model call or a tool call, with its inputs, outputs, start and end times, and any error. The runs behind one request form a tree, called a trace.
LangChain.js and LangGraph.js trace through callbacks. Pass a tracer such as LangChainTracer in the config's callbacks, and every runnable reports its start and end to it.
LangSmith's run format
Each run has a run_type, such as chain, llm or tool, an id, a trace_id shared by the whole tree, and a parent_run_id. Its dotted_order is a key built from the start time and id of each run from the root down to it, so sorting by it lays out the tree in the order things happened. Model runs carry usage_metadata with token counts. LangGraph adds thread_id, langgraph_node and langgraph_step to the metadata.
Supersteps and the run tree
LangGraph runs a graph in supersteps. In each superstep the nodes that are due run, and their updates are applied together before the next superstep starts. langgraph_step is that superstep's number, so one trace reads two ways: as a run tree, each run under its parent, and as a superstep timeline, one group of runs per step. The waterfall in the lab groups its rows by superstep.
What to look for
Latency: which run is long, and is it the model or a tool. Errors: a failed tool run and the node that called it. Tokens: which calls read the most. Loops: the same step again and again.
Traces hold prompts, tool results and customer data, so treat them as sensitive. The LangSmith SDK can run an anonymizer on every run before it is sent. OpenTelemetry is the general standard for traces across services; LangSmith accepts OpenTelemetry traces too.
Step 2 of 4: Try it: The run waterfall
The run waterfall
Five steps: run the agent and watch its trace grow, read its supersteps, find the slowest run and the failed tool, then export the runs.
Mode:RealSimulatedReal: LangGraph.js runs the agent, and LangChain's own tracer and the LangSmith SDK produce these runs in this page, in LangSmith's run format. Nothing is sent anywhere. Simulated: the model follows a script, the timings are scripted delays, the data is fictional, and token counts are estimated as characters / 4.
Step 3 of 4: Check
Check
Next: Question 2 of 5
Step 4 of 4: What you can do now
What you can do now
- Read a superstep timeline and a run tree.
- Find the slow or failed run in a trace, and the run that called it.
- Name the main fields of LangSmith's run format.
- Redact personal data before a trace leaves your system.