Trace a Quarkus LangChain4j App in LangSmith

Build a small Quarkus assistant, send its OpenTelemetry traces to LangSmith, and learn how plain chat, tool calls, and failures show up in one trace tree.

LangSmith still looks like a Python-and-LangChain product in a lot of Java teams. When I started wiring Quarkus LangChain4j into it, I assumed I would need a Java SDK bridge or some Arconia-style semantic-convention shim before the traces would look sane in the UI.

Turns out I did not. Quarkus LangChain4j already emits OpenTelemetry spans with GenAI attributes when quarkus-opentelemetry is on the classpath. LangSmith accepts generic OTLP from non-LangChain apps and maps those attributes into its tracing UI. So the work here is exporter wiring, deciding how much prompt and tool payload to ship, and generating traces worth opening.

We build SignalDesk, a fictional on-call support assistant with one REST endpoint, one AI service, one runbook tool, and three request shapes: plain chat, tool call, and controlled failure. You run it locally against Ollama, export traces to LangSmith, and keep CI green with a deterministic test stub.

For a much simpler Quarkus observability story in a hand-crafted way, see LLM observability with Quarkus and LangChain4j. This post stays focussed on standards: LangSmith over OTLP from a Quarkus AI service.

What we build

SignalDesk exposes POST /signaldesk/assist and returns:

• answer — model text

• usedTool — whether a tool ran

• toolName — e.g. lookupRunbook

• outcome — OK, TOOL_FAILED, or DEGRADED

Three prompts drive three trace stories:

• Plain chat — “What is our SLA for SEV-2?” (no tool)

• Tool path — “SEV-1 database failover — which runbook?” (lookupRunbook)

• Failure path — “Trigger runbook lookup for UNKNOWN-PLAN” (tool returns an error; HTTP stays 200 with outcome: TOOL_FAILED)

What you need

You have run Quarkus in dev mode and called a JSON endpoint before. LangChain4j AI service interfaces should look familiar.

• JDK 21

• Ollama on http://localhost:11434

• with a tool-capable model (defaults to llama3.2; set OLLAMA_MODEL if you standardize on something else)

• LangSmith account, API key, and the OTLP endpoint from your project settings

• About 4-5 ☕️

Project setup

Follow along or clone the project from my Github. From the repo root (adjust the path if you nest the module elsewhere):

quarkus create app dev.signaldesk:signaldesk-langsmith \
  --package-name=dev.signaldesk \
  --extensions='rest-jackson,quarkus-langchain4j-ollama,quarkus-opentelemetry' \
  --java=21 \
  --no-code
cd signaldesk-langsmith

The generator already adds quarkus-langchain4j-bom to dependencyManagement when you pick the LangChain4j Ollama extension. That is the same outcome you get from code.quarkus.io with that extension selected.

Add rest-assured dependency to your pom.xml:

• rest-assured — HTTP contract tests

Enable parameter names on maven-compiler-plugin (<parameters>true</parameters>) for {{question}} templates.

Package root: dev.signaldesk.

Assistant and runbook tool

I keep the assistant deliberately small here. One AI service, one short system message, and one tool box are enough for LangSmith to show a trace tree you would actually want to inspect: parent chat span, child tool span, and token usage without extra noise.

SignalDeskAssistant:

@RegisterAiService
@ApplicationScoped
public interface SignalDeskAssistant {

    @SystemMessage(
            """
            You are SignalDesk, an internal support assistant for on-call engineers.
            Answer SLA and policy questions directly when no runbook lookup is needed.
            For SEV-1 failover or explicit runbook requests, call lookupRunbook with service and severity.
            Keep answers short.""")
    @UserMessage("{{question}}")
    @ToolBox(RunbookTools.class)
    String assist(String question);
}

RunbookTools.lookupRunbook records invocations on a request-scoped AssistTrace and returns a fake runbook string. For UNKNOWN-PLAN it records a tool failure and returns an ERROR: line. LangChain4j then feeds that back to the model instead of throwing through the whole stack, which is a lot easier to demo and inspect:

@Tool("Looks up the on-call runbook for a service and severity. Use for failover or incident response.")
public String lookupRunbook(String service, String severity) {
    if (service != null && service.toUpperCase().contains("UNKNOWN-PLAN")) {
        assistTrace.recordToolFailure(TOOL_NAME);
        return "ERROR: runbook not found for service " + service;
    }
    assistTrace.recordTool(TOOL_NAME);
    return "runbook-" + service + "-" + severity + ": page platform-oncall, follow failover checklist RB-12";
}

REST endpoint and AssistTrace

SignalDeskResource delegates to SignalDeskService, which resets AssistTrace, calls the assistant, and maps trace state into AssistResponse. When the tool reports failure, outcome becomes TOOL_FAILED even though HTTP stays 200. I like that split here because API success and tool success are different stories, and the trace should make that visible.

OpenTelemetry → LangSmith

Add quarkus-opentelemetry (included if you used the create command above). Configure OTLP export in src/main/resources/application.properties:

quarkus.application.name=signaldesk-langsmith

quarkus.langchain4j.ollama.base-url=http://localhost:11434
quarkus.langchain4j.ollama.chat-model.model-id=${OLLAMA_MODEL:llama3.2}
quarkus.langchain4j.ollama.chat-model.temperature=0.2
quarkus.langchain4j.timeout=120s

quarkus.otel.exporter.otlp.protocol=http/protobuf
quarkus.otel.exporter.otlp.traces.protocol=http/protobuf
quarkus.otel.exporter.otlp.traces.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:${LANGSMITH_OTLP_ENDPOINT:https://api.smith.langchain.com/otel}}
quarkus.otel.exporter.otlp.traces.headers=x-api-key=${LANGSMITH_API_KEY:},Langsmith-Project=${LANGSMITH_PROJECT:signaldesk-langsmith}
quarkus.otel.traces.sampler=parentbased_always_on

Three details burned me during smoke testing:

1. Use the OTLP base URL ending in /otel, not /otel/v1/traces. LangSmith’s OpenTelemetry guide often shows …/otel as the endpoint. With http/protobuf, Quarkus appends v1/traces automatically (Quarkus OpenTelemetry guide). If you also put /v1/traces in the property, export silently hits a double path (…/otel/v1/traces/v1/traces) and the dashboard stays empty even though spans exist locally in the logs.

2. Quarkus does not read the Python LangChain SDK env vars. These are ignored for this app:

• LANGSMITH_TRACING

• LANGSMITH_OTEL_ENABLED

• LANGSMITH_ENDPOINT (API host, not the OTLP exporter URL)

Use OTLP variables instead:

• LANGSMITH_API_KEY → x-api-key header

• OTEL_EXPORTER_OTLP_ENDPOINT or LANGSMITH_OTLP_ENDPOINT → traces endpoint (base /otel URL)

• LANGSMITH_PROJECT → Langsmith-Project header (must match the project name in the UI)

3. Regional hosts matter. EU accounts need the EU OTLP host, not the US default:

• EU: https://eu.api.smith.langchain.com/otel

• US: https://api.smith.langchain.com/otel

Export credentials in the same shell you use to start dev mode, then restart:

export LANGSMITH_API_KEY=lsv2_pt_...
export OTEL_EXPORTER_OTLP_ENDPOINT=https://eu.api.smith.langchain.com/otel
export LANGSMITH_PROJECT=signaldesk-langsmith
./mvnw quarkus:dev

Use a slug for LANGSMITH_PROJECT (no spaces). Values like “Quarkus Test App” break Quarkus comma-separated headers and LangSmith may only receive Langsmith-Project=Quarkus. If you need spaces, set the full header string via LANGSMITH_OTLP_HEADERS and wire that property in application.properties instead.

Startup check: OtelExportConfigProbe

The demo includes a small startup bean that logs the resolved OTLP endpoint, the project name it sees, whether an API key reached the exporter config, and a few common-footgun warnings. On boot you want something like:

OTLP traces: protocol=http/protobuf endpoint=https://eu.api.smith.langchain.com/otel project=signaldesk-langsmith apiKeySet=true

If apiKeySet=false or the endpoint still shows …/otel/v1/traces, fix the env vars and restart before you curl.

The app still starts and answers requests even when export is misconfigured. LangSmith just stays empty until the endpoint, key, and project line up. Once they do, traces usually show up in the named project within about 10 to 30 seconds of each request.

Quarkus LangChain4j records spans for chat model calls with attributes such as gen_ai.operation.name, gen_ai.system, model id, and token usage and there is no custom instrumentation class required for the baseline path.

Rich trace content (and why it is dangerous)

LangSmith is much easier to debug when prompts and tool payloads are on the span. Quarkus exposes that as configuration, not code:

quarkus.langchain4j.tracing.include-prompt=true
quarkus.langchain4j.tracing.include-completion=true
quarkus.langchain4j.tracing.include-tool-arguments=true
quarkus.langchain4j.tracing.include-tool-result=true

I turn these on for local debugging and leave them off by default in production unless there is redaction, a retention policy, and actual legal sign-off. Customer text, tokens, and runbook arguments end up in a third-party trace store very quickly.

For local export failures, the demo turns on %dev debug logging for io.opentelemetry.exporter and io.quarkus.opentelemetry — search the console for 401, 404, or Failed to export after a curl.

Optional: app metadata on spans

If you want an app-specific filter that survives in LangSmith, implement ChatModelSpanContributor:

@ApplicationScoped
public class SignalDeskSpanContributor implements ChatModelSpanContributor {

    @Override
    public void onRequest(ChatModelRequestContext requestContext, Span currentSpan) {
        currentSpan.setAttribute("signaldesk.workflow", "signaldesk-assist");
    }
    // onResponse / onError — same attribute
}

That sits beside standard gen_ai.* data, not instead of it.

Prove it

CI (no Ollama, no LangSmith):

./mvnw test

src/test/resources/application.properties:

%test.quarkus.langchain4j.ollama.devservices.enabled=false
%test.quarkus.langchain4j.devservices.enabled=false
%test.quarkus.otel.sdk.disabled=true
%test.quarkus.langchain4j.tracing.include-prompt=false
%test.quarkus.langchain4j.tracing.include-completion=false
%test.quarkus.langchain4j.tracing.include-tool-arguments=false
%test.quarkus.langchain4j.tracing.include-tool-result=false

Tests use SignalDeskStubChatModel via SignalDeskStubProfile (getEnabledAlternatives()), with keyword-driven tool versus no-tool behavior that matches the three curl recipes.

Manual traces (Ollama + LangSmith):

Plain chat:

curl -s -X POST http://localhost:8080/signaldesk/assist \
  -H 'Content-Type: application/json' \
  -d '{"question":"What is our SLA for SEV-2?"}' | jq

Tool path:

curl -s -X POST http://localhost:8080/signaldesk/assist \
  -H 'Content-Type: application/json' \
  -d '{"question":"SEV-1 database failover — which runbook?"}' | jq

Failure path:

curl -s -X POST http://localhost:8080/signaldesk/assist \
  -H 'Content-Type: application/json' \
  -d '{"question":"Trigger runbook lookup for UNKNOWN-PLAN"}' | jq

LangSmith checklist after each call (project signaldesk-langsmith in the UI):

Open the Runs tab for your project. After the tool-path curl, the table should look roughly like this:

• POST /signaldesk/assist — HTTP entry (~0.8s), often the slowest row

• langchain4j.services.SignalDeskAssistant.assist (or truncated) — AI service span

• completion llama3.2 — model span with Input/Output columns showing prompt and answer text; Tokens populated (exact count varies by model and prompt)

• lookupRunbook — tool span with Input database (or similar) and Output runbook-database-SEV-…

• OTEL_SPAN_ID in Metadata on each row — confirms generic OTLP export, not a LangChain-only SDK path

Drill into one run for the trace tree (HTTP → service → completion → tool). The flat Runs table is enough to prove export. The tree view is the part worth keeping for the article screenshot.

Per recipe:

• Plain chat — completion rows without a lookupRunbook row

• Tool path — lookupRunbook plus one or more completion llama3.2 rows (tool-capable models may take two model hops)

• Failure path — lookupRunbook with error content in Output; JSON shows "outcome":"TOOL_FAILED" even when HTTP stays 200

Make it survive production

Keep the claims modest. This setup gives you useful LangSmith traces from Quarkus over OTLP. It does not promise full parity with LangSmith’s language-specific SDKs, every LangSmith feature via generic OTLP, or a safe default for shipping full prompts on production traffic.

Nothing in the dashboard? Work through this order — it matches what broke in practice:

1. Endpoint suffix — property must end with /otel, not /otel/v1/traces

2. Region — EU UI needs eu.api.smith.langchain.com, not api.smith.langchain.com

3. Env vars in the dev shell — restart quarkus:dev after export; IDE runs often miss them

4. Project name — LANGSMITH_PROJECT must match the LangSmith project; avoid spaces in the value unless you use LANGSMITH_OTLP_HEADERS

5. Wrong tab — traces land under the named project’s Runs, not “default”

6. Batch delay — wait 10–30 seconds and refresh

Missing API key — the app should still answer; OTLP export may warn or drop spans. OtelExportConfigProbe logs apiKeySet=false when the key did not reach the JVM.

PII and secrets — prompts, completions, and tool arguments can contain customer data. Once traffic is real, I would rather put sampling, redaction, or a collector in front of LangSmith than pretend the debug-friendly defaults are somehow production policy. The follow-up that belongs in its own post is one Quarkus app with collector fan-out to LangSmith for AI traces and Tempo for everything else.

Metrics — traces answer “what happened on this request?” Micrometer dashboards answer “what is the burn rate?” The sibling piece in this series is production-style AI metrics, not this article.

Close

SignalDesk is small on purpose. Quarkus LangChain4j already speaks the OpenTelemetry GenAI dialect LangSmith understands, so the real decision is how you export and how much content you let leave the JVM. Once the plain chat, tool, and failure traces look right in LangSmith, you have the same debugging loop Python teams enjoy, without rewriting the app in LangChain.