Skip to content
Back to Agent systems and standards

Agent systems and standards · Chapter 7 of 8

Observability: traces and runs

Recording every model call, tool call and step as a tree of runs, and reading it to find what is slow or broken.

20 minutes0 of 4 goals done

Goals

  • Find the slowest model or tool run in the trace.To do
  • Find the failed tool run and its parent.To do
  • Export the runs in LangSmith's format.To do
  • Pass the check.To do

Step 1 of 4: The read

Runs and traces

An agent's answer hides many steps. Tracing records each one as a run: a graph, a node, a model call or a tool call, with its inputs, outputs, start and end times, and any error. The runs behind one request form a tree, called a trace.

LangChain.js and LangGraph.js trace through callbacks. Pass a tracer such as LangChainTracer in the config's callbacks, and every runnable reports its start and end to it.

LangSmith's run format

Each run has a run_type, such as chain, llm or tool, an id, a trace_id shared by the whole tree, and a parent_run_id. Its dotted_order is a key built from the start time and id of each run from the root down to it, so sorting by it lays out the tree in the order things happened. Model runs carry usage_metadata with token counts. LangGraph adds thread_id, langgraph_node and langgraph_step to the metadata.

Supersteps and the run tree

LangGraph runs a graph in supersteps. In each superstep the nodes that are due run, and their updates are applied together before the next superstep starts. langgraph_step is that superstep's number, so one trace reads two ways: as a run tree, each run under its parent, and as a superstep timeline, one group of runs per step. The waterfall in the lab groups its rows by superstep.

What to look for

Latency: which run is long, and is it the model or a tool. Errors: a failed tool run and the node that called it. Tokens: which calls read the most. Loops: the same step again and again.

Traces hold prompts, tool results and customer data, so treat them as sensitive. The LangSmith SDK can run an anonymizer on every run before it is sent. OpenTelemetry is the general standard for traces across services; LangSmith accepts OpenTelemetry traces too.

Step 2 of 4: Try it: The run waterfall

The run waterfall

Five steps: run the agent and watch its trace grow, read its supersteps, find the slowest run and the failed tool, then export the runs.

Mode:RealSimulatedReal: LangGraph.js runs the agent, and LangChain's own tracer and the LangSmith SDK produce these runs in this page, in LangSmith's run format. Nothing is sent anywhere. Simulated: the model follows a script, the timings are scripted delays, the data is fictional, and token counts are estimated as characters / 4.

Step 3 of 4: Check

Check

Question 1 of 5What does dotted_order give you?

Next: Question 2 of 5

Question 2 of 5Two runs in a trace have the same langgraph_step. What does that tell you?
Question 3 of 5What is a trace?
Question 4 of 5A tool run failed, but the agent still answered. Why is the parent run not marked failed?
Question 5 of 5Which should you avoid sending to a shared tracing service?

Step 4 of 4: What you can do now

What you can do now

  • Read a superstep timeline and a run tree.
  • Find the slow or failed run in a trace, and the run that called it.
  • Name the main fields of LangSmith's run format.
  • Redact personal data before a trace leaves your system.