That solves one problem: the user can see bytes crossing the network.

It does not solve the more common enterprise problem. The file reaches the server, the bar says 100%, and then the real work begins. The application validates 200,000 rows, resolves references, writes batches, calls another system, or generates a report. The browser has no idea how long that work will take. A green upload bar quickly becomes a lie.

I want to fix that in this follow-up.

We will build an invoice import on Quarkus 3.37.2 and Java 25. The browser shows two separate clocks:

• Transfer progress measures bytes from the browser to the HTTP endpoint.

• Processing progress measures durable business work after Quarkus accepts the file.

The job lives in PostgreSQL. A basic Quarkus Scheduler worker claims it. Server-Sent Events deliver snapshots, but the connection owns no state. Reload the page halfway through and the UI reconstructs itself from the database. Cancel the job and the worker stops between batches. Restart the application with the same staging volume and startup recovery puts interrupted work back into the queue.

That is a more useful progress bar because it models the work instead of decorating the request.

What we are building

The complete companion application is in the honest-progress directory. It includes a 60-row sample CSV and a deliberately visible processing delay so you can watch the state changes.

The flow looks like this:

There are four REST operations:

• POST /api/imports accepts the multipart file and returns 202 Accepted.

• GET /api/imports/{id} returns one authoritative snapshot.

• GET /api/imports/{id}/events streams changed snapshots as SSE.

• DELETE /api/imports/{id} requests cooperative cancellation.

Notice what is absent: there is no active SSE sink registry and no in-memory progress map. An SSE connection may disappear at any time without changing the job.

Prerequisites

You need:

• Java 25

• Podman with its Docker-compatible socket enabled

• A current Quarkus CLI, or Maven if you prefer to create the project directly

• About ☕️☕️

PostgreSQL comes from Quarkus Dev Services. With Podman running, there is no database setup for dev mode or tests.

Create the application or start from my Github repository:

quarkus create app com.themainthread.progress:honest-progress \
  --extension=rest-jackson,hibernate-orm-panache,jdbc-postgresql,scheduler,hibernate-validator \
  --java=25 \
  --no-code

cd honest-progress

I am using the basic quarkus-scheduler extension. It is enough for one application instance and keeps the job lifecycle visible. I will cover the clustered challenges later.

Model work, not elapsed time

A progress model needs more than a percentage. It needs a state, a phase, completed units, total units, a human-readable message, and an optimistic-lock version.

My sample uses these states:

package com.themainthread.progress.domain;

public enum JobState {
    QUEUED,
    RUNNING,
    SUCCEEDED,
    FAILED,
    CANCELLED;

    public boolean terminal() {
        return this == SUCCEEDED || this == FAILED || this == CANCELLED;
    }
}

And these phases:

package com.themainthread.progress.domain;

public enum JobPhase {
    QUEUED,
    VALIDATING,
    IMPORTING,
    FINALIZING,
    COMPLETE
}

RUNNING tells an operator that a worker owns the job. VALIDATING tells the user what that worker is doing.

Only the import phase has a decent denominator. During validation, the application does not yet know how many valid rows exist. During finalization, a database transaction may take 50 ms or five seconds, but “73% committed” is not a real measurement. Those phases use an indeterminate progress bar.

The API record calculates a percentage only when it can produce a number:

private static Integer percentage(ImportJob job) {
    if (job.jobState == JobState.SUCCEEDED) {
        return 100;
    }
    if (job.totalUnits == 0
            || job.jobPhase == JobPhase.VALIDATING
            || job.jobPhase == JobPhase.FINALIZING) {
        return null;
    }
    return (int) Math.min(100,
            job.completedUnits * 100 / job.totalUnits);
}

Returning null is not a missing feature. It is a statement that we know the phase but cannot calculate meaningful completion inside it.

The ImportJob entity persists this state with timestamps, a cancellation flag, the staged file path, and @Version:

package com.themainthread.progress.domain;

import java.time.Instant;
import java.util.UUID;

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import jakarta.persistence.Version;

@Entity
@Table(name = "import_job")
public class ImportJob {

    @Id
    public UUID id;

    @Column(name = "original_file_name", nullable = false, length = 255)
    public String originalFileName;

    @Column(name = "stored_path", nullable = false, length = 1024)
    public String storedPath;

    @Column(name = "file_size", nullable = false)
    public long fileSize;

    @Enumerated(EnumType.STRING)
    @Column(name = "job_state", nullable = false, length = 32)
    public JobState jobState;

    @Enumerated(EnumType.STRING)
    @Column(name = "job_phase", nullable = false, length = 32)
    public JobPhase jobPhase;

    @Column(name = "completed_units", nullable = false)
    public long completedUnits;

    @Column(name = "total_units", nullable = false)
    public long totalUnits;

    @Column(name = "published_count", nullable = false)
    public long publishedCount;

    @Column(name = "cancellation_requested", nullable = false)
    public boolean cancellationRequested;

    @Column(nullable = false, length = 500)
    public String message;

    @Column(name = "error_message", length = 2000)
    public String errorMessage;

    @Column(name = "created_at", nullable = false)
    public Instant createdAt;

    @Column(name = "updated_at", nullable = false)
    public Instant updatedAt;

    @Column(name = "finished_at")
    public Instant finishedAt;

    @Version
    public long version;
}

The important field for streaming is version. Every committed change increments it, so an SSE endpoint can suppress identical snapshots without maintaining its own business state.

Accept the upload, then return a job

Quarkus REST supports multipart form data through FileUpload. The upload endpoint moves Quarkus’ temporary file into application-controlled staging, creates the job, and returns immediately.

package com.themainthread.progress.api;

import java.net.URI;
import java.util.UUID;

import com.themainthread.progress.domain.ImportProgress;
import com.themainthread.progress.job.ProgressStreamService;

import io.smallrye.common.annotation.Blocking;
import io.smallrye.mutiny.Multi;
import jakarta.validation.constraints.NotNull;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.DELETE;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.Context;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;
import jakarta.ws.rs.core.UriInfo;
import jakarta.ws.rs.sse.OutboundSseEvent;
import jakarta.ws.rs.sse.Sse;

import org.jboss.resteasy.reactive.RestForm;
import org.jboss.resteasy.reactive.multipart.FileUpload;

@Path("/api/imports")
@Produces(MediaType.APPLICATION_JSON)
public class ImportResource {

    private final ImportJobService jobs;
    private final ProgressStreamService streams;
    private final Sse sse;

    public ImportResource(ImportJobService jobs,
            ProgressStreamService streams, Sse sse) {
        this.jobs = jobs;
        this.streams = streams;
        this.sse = sse;
    }

    @POST
    @Consumes(MediaType.MULTIPART_FORM_DATA)
    public Response create(@NotNull @RestForm("file") FileUpload file,
            @Context UriInfo uriInfo) {
        ImportProgress progress = jobs.create(file);
        URI location = uriInfo.getAbsolutePathBuilder()
                .path(progress.id().toString())
                .build();
        return Response.accepted(progress).location(location).build();
    }

    @GET
    @Path("/{id}")
    public ImportProgress get(@PathParam("id") UUID id) {
        return jobs.snapshot(id);
    }

    @DELETE
    @Path("/{id}")
    public Response cancel(@PathParam("id") UUID id) {
        return Response.accepted(jobs.cancel(id)).build();
    }

    @GET
    @Path("/{id}/events")
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @Blocking
    public Multi<OutboundSseEvent> events(@PathParam("id") UUID id) {
        ImportProgress initial = jobs.snapshot(id);
        return streams.stream(id, initial).map(this::event);
    }

    private OutboundSseEvent event(ImportProgress progress) {
        return sse.newEventBuilder()
                .id(Long.toString(progress.version()))
                .name("progress")
                .reconnectDelay(2000)
                .mediaType(MediaType.APPLICATION_JSON_TYPE)
                .data(ImportProgress.class, progress)
                .build();
    }
}

202 Accepted means that the transfer completed and the server accepted a different resource, the import job, for asynchronous processing. The Location header tells the client where to find it.

The limits must match the endpoint. Quarkus applies a small default limit to multipart form attributes, so raising only the overall body limit is not enough for a large file field:

quarkus.http.limits.max-body-size=26M
quarkus.http.limits.max-form-attribute-size=25M

progress.staging-directory=target/staged-uploads
progress.stream-interval=500ms
progress.processing-delay=150ms
progress.batch-size=5
progress.scheduler-interval=1s

%dev.quarkus.hibernate-orm.schema-management.strategy=drop-and-create
%test.quarkus.hibernate-orm.schema-management.strategy=drop-and-create
%prod.quarkus.hibernate-orm.schema-management.strategy=validate

%test.quarkus.scheduler.enabled=false
%prod.progress.staging-directory=/var/lib/honest-progress/staged-uploads
%prod.progress.processing-delay=0s

The delay exists only to make a 60-row demo observable. The production profile disables it.

Do not trust an uploaded file name as a path. The sample keeps only Path.of(name).getFileName(), generates the stored name from the job UUID, accepts .csv, and moves the file before the request ends. Quarkus owns the multipart temporary file only for the request lifecycle.

Let the scheduler claim durable work

The worker is small and boring as usually:

package com.themainthread.progress.job;

import java.util.List;

import com.themainthread.progress.persistence.JobStore;
import com.themainthread.progress.persistence.JobStore.RecoveryAction;
import com.themainthread.progress.storage.FileStorage;

import io.quarkus.runtime.StartupEvent;
import io.quarkus.scheduler.Scheduled;
import io.quarkus.scheduler.Scheduled.ConcurrentExecution;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;

@ApplicationScoped
public class ImportJobScheduler {

    private final JobStore store;
    private final ImportJobProcessor processor;
    private final FileStorage fileStorage;

    public ImportJobScheduler(JobStore store,
            ImportJobProcessor processor, FileStorage fileStorage) {
        this.store = store;
        this.processor = processor;
        this.fileStorage = fileStorage;
    }

    void recover(@Observes StartupEvent event) {
        List<RecoveryAction> actions = store.recoverInterrupted();
        for (RecoveryAction action : actions) {
            if (action.deleteFile()) {
                fileStorage.deleteQuietly(action.storedPath());
            }
        }
    }

    @Scheduled(identity = "invoice-import-runner",
            every = "${progress.scheduler-interval}",
            concurrentExecution = ConcurrentExecution.SKIP)
    public void runNext() {
        store.claimNext().ifPresent(processor::process);
    }
}

The Quarkus Scheduler supports property expressions and skipped concurrent execution. SKIP prevents another trigger from entering this method while the current import is still running in this application instance.

claimNext() locks the oldest queued row and changes it to RUNNING in one transaction. The processor then follows this order:

1. Validate the complete CSV and count its rows.

2. Change the phase to IMPORTING and set totalUnits.

3. Persist small batches with published=false.

4. Check the cancellation flag between batches.

5. Change the phase to FINALIZING.

6. Publish every staged invoice and mark the job SUCCEEDED in one transaction.

Validation before import avoids predictable partial failures. The unpublished flag handles failures that happen later. If processing fails or the user cancels, the application deletes the staged invoice rows. If the application stops after committing a batch, startup recovery deletes unpublished rows and requeues the job before the scheduler continues.

This is not one giant transaction. A 200,000-row import should not hold locks and a persistence context for the entire job. Small transactions make progress durable and cancellation responsive. The publication flag keeps those intermediate commits invisible to downstream business queries.

Cooperative cancellation is equally deliberate. DELETE does not interrupt a Java thread in the middle of a database operation. It sets cancellationRequested=true. The worker finishes its current batch, observes the flag, cleans up, and ends in CANCELLED. That gives the code a safe boundary instead of pretending arbitrary interruption is harmless.

Stream database snapshots, not process memory

SSE is still the right fit for the second clock. Progress moves from server to browser, the browser API reconnects automatically, and ordinary HTTP remains available for create, read, and cancel commands.

The stream emits the current snapshot immediately, then checks for version changes on a worker thread:

package com.themainthread.progress.job;

import java.util.UUID;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;

import com.themainthread.progress.config.ProgressConfig;
import com.themainthread.progress.domain.ImportProgress;
import com.themainthread.progress.persistence.JobStore;

import io.smallrye.mutiny.Multi;
import io.smallrye.mutiny.Uni;
import io.smallrye.mutiny.infrastructure.Infrastructure;
import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class ProgressStreamService {

    private final JobStore store;
    private final ProgressConfig config;

    public ProgressStreamService(JobStore store, ProgressConfig config) {
        this.store = store;
        this.config = config;
    }

    public Multi<ImportProgress> stream(UUID id, ImportProgress initial) {
        AtomicBoolean terminalSeen = new AtomicBoolean();
        AtomicLong lastVersion = new AtomicLong(Long.MIN_VALUE);
        Multi<ImportProgress> changes = Multi.createFrom().ticks()
                .every(config.streamInterval())
                .onItem().transformToUniAndConcatenate(
                        ignored -> snapshotAsync(id));

        return Multi.createBy().concatenating()
                .streams(Multi.createFrom().item(initial), changes)
                .select().first(progress ->
                        !terminalSeen.getAndSet(progress.terminal()))
                .select().where(progress ->
                        progress.version()
                                != lastVersion.getAndSet(progress.version()));
    }

    private Uni<ImportProgress> snapshotAsync(UUID id) {
        return Uni.createFrom().item(() -> store.snapshot(id))
                .runSubscriptionOn(Infrastructure.getDefaultWorkerPool());
    }
}

The JDBC read is blocking, so runSubscriptionOn moves it away from the event-loop thread. The first terminal snapshot is emitted, one following tick closes the stream, and duplicate entity versions never reach the client.

Polling PostgreSQL twice per second is a conscious baseline, not a universal prescription. It is easy to reason about, survives reconnects, and is adequate for modest enterprise job volume. At larger scale, publish job IDs through PostgreSQL notifications, Kafka, Redis, or another broker and let the SSE layer re-read the authoritative row before emitting. The database remains the source of truth; the broker becomes a wake-up signal.

Use the browser’s upload progress

The previous article chunked the file in JavaScript and counted each accepted chunk on the server. Chunking is justified when you need resumable transfer, retry of individual parts, or object-storage multipart uploads. It is extra protocol surface when you only need to show one normal upload.

The Fetch API still does not provide the same broadly available upload progress events. XMLHttpRequest.upload does. The browser fires progress events with loaded, total, and lengthComputable, which is exactly what the transfer bar needs.

The relevant browser code is small:

function upload(file) {
    const body = new FormData();
    body.append('file', file);

    const request = new XMLHttpRequest();
    request.open('POST', '/api/imports');

    request.upload.addEventListener('progress', event => {
        if (!event.lengthComputable) {
            transferProgress.removeAttribute('value');
            transferPercent.textContent = 'sending';
            return;
        }
        const percent = Math.round(event.loaded * 100 / event.total);
        transferProgress.value = percent;
        transferPercent.textContent = `${percent}%`;
    });

    request.addEventListener('load', () => {
        if (request.status !== 202) {
            showUploadError(request.responseText);
            return;
        }

        const job = JSON.parse(request.responseText);
        localStorage.setItem('honest-progress-job', job.id);
        renderJob(job);
        follow(job.id);
    });

    request.send(body);
}

After the 202, EventSource owns the second display:

function follow(id) {
    const source = new EventSource(`/api/imports/${id}/events`);

    source.addEventListener('progress', event => {
        const job = JSON.parse(event.data);
        renderJob(job);

        if (['SUCCEEDED', 'FAILED', 'CANCELLED'].includes(job.state)) {
            source.close();
            localStorage.removeItem('honest-progress-job');
        }
    });
}

On page load, the full application reads the stored job ID, calls GET /api/imports/{id}, renders that snapshot, and reconnects only if the job is non-terminal. localStorage is a convenience pointer, not trusted job state.

The UI uses native <progress> elements and an aria-live status message. When percent is null, JavaScript removes the element’s value attribute and the browser exposes an indeterminate state. A screen-reader user gets the same phase and message as a sighted user rather than a silent animated stripe.

Run it and break it

Start dev mode:

./mvnw quarkus:dev

Open http://localhost:8080 and click Run demo file. You should see the transfer reach 100% first. The job then moves through QUEUED, VALIDATING, IMPORTING, FINALIZING, and SUCCEEDED.

Now run three more checks:

1. Start another import and reload the page while rows are being imported. The job ID survives locally, the snapshot comes back from PostgreSQL, and SSE resumes.

2. Start an import and press Cancel job. A queued job cancels immediately; a running job stops after its current batch.

3. Upload a CSV containing the same invoice number twice. Validation fails before any published row exists, and the job preserves the error for a later GET.

You can create a job without the UI:

curl -i \
  -F file=@src/main/resources/META-INF/resources/sample-invoices.csv \
  http://localhost:8080/api/imports

The response starts like this:

HTTP/1.1 202 Accepted
Location: http://localhost:8080/api/imports/4bd55d0a-...
Content-Type: application/json

Run the tests:

./mvnw test

The seven tests cover successful publication, the terminal SSE event, queued cancellation, duplicate detection with no published partial rows, rejected file types, and focused CSV rules. The scheduler is disabled in the test profile so each integration test calls the worker deterministically.

Where the basic scheduler stops scaling

This sample has one more boundary: it is a single-instance design.

ConcurrentExecution.SKIP prevents overlap only inside one application instance. If you deploy three replicas, each replica has its own trigger. PostgreSQL locking can protect a job claim, but it does not make local staged files visible to another pod. Startup recovery on one replica must not reset work that another healthy replica owns.

For multiple replicas, change three elements:

• Put uploads in shared object storage instead of a local staging directory.

• Claim work with an atomic database operation such as FOR UPDATE SKIP LOCKED, plus a lease and heartbeat so abandoned work can be recovered safely.

• Either let every replica compete for jobs safely or use the Quarkus Quartz extension with a clustered store when you need coordinated triggers, calendars, and persistent scheduling.

The browser and API contract do not need to change. POST, GET, DELETE, and SSE still address the same durable job resource.

What WebSockets Next would change

I did not add quarkus-websockets-next to this application. Progress is server-to-client, while create and cancel are ordinary HTTP commands. SSE matches that asymmetry and gives us a smaller protocol.

WebSockets Next becomes interesting when the control plane is genuinely bidirectional. Imagine that the user can pause, resume, change priority, answer a validation question, or subscribe to several jobs over one connection. A WebSocket endpoint could receive those commands and push state changes on the same channel.

That changes the edge architecture, not the job architecture:

• The browser opens a WebSocket and sends an explicit subscribe message for a job ID.

• The server authenticates that subscription and maps connections to authorized jobs.

• Cancel, pause, and resume become socket messages instead of HTTP methods.

• Reconnect needs an application protocol: the client must resubscribe and send the last version it saw.

• Multiple application replicas need shared fan-out through a broker or a routed connection strategy.

PostgreSQL still leads the job execution. The scheduler still processes it. A socket must not become the only place that knows progress or cancellation state, because sockets disconnect and application instances restart.

Use WebSockets Next when bidirectional interaction is part of managing lifecycle complexity. Do not add it just to animate a percentage bar.

The progress bar is part of the domain

The old version asked, “How many bytes reached Quarkus?” This version asks a more valuable question: “What can the application truthfully say about the work right now?”

Sometimes the answer is 42 of 200,000 invoices. Sometimes it is “validating,” with no percentage. Sometimes it is “cancellation requested; finishing the current batch.” Those are not UI details. They are business state, and they deserve the same durability and failure semantics as the import itself.

Keep the two clocks separate. Return a job resource when the upload ends. Persist every meaningful transition. Treat SSE or WebSockets as delivery mechanisms, not memory. Then 100% can mean what users think it means: the work is actually done.

Subscribe now

I first covered it in the short May 2025 article Add TOTP Authentication to Your Java API with Quarkus and Vault Dev Services. That example started Vault with Dev Services, created a TOTP key, displayed a QR code, and validated a six-digit code.

That was enough to show the Vault API, but not a complete authentication flow. The example trusted an X-User header and sent the TOTP code directly to the protected endpoint. It had no first-factor token, enrollment state, or replay test. It also required a manual step in the Vault UI to enable the TOTP engine.

This article builds the missing parts. A signed access token represents the first login. The user exchanges one valid TOTP code for a step-up JWT that expires after two minutes. Only that token carries the payout:approve role. The payout endpoint then uses normal Quarkus RBAC. The TOTP code raises assurance once; it does not become another credential on every business request.

We also store the enrollment mapping, mark the QR response as no-store, validate the code format, and test that Vault rejects a used code.

One version detail matters for this setup. Quarkus Vault 4.8.0 is current as I write this, but its Dev Services source still defaults to Vault 1.15.2. Vault fixed a TOTP code-reuse vulnerability in 1.15.15, 1.16.23, 1.18.12, 1.19.7, and 1.20.1. The Java extension and the Vault server have separate versions. We will set the current Vault 2.0.3 image explicitly.

What we are building

The request flow has three credentials with different jobs:

1. A normal bearer token represents the authenticated subject alice and carries the user role.

2. A valid TOTP code raises assurance. LedgerLock consumes it once and returns a step-up JWT that expires after 120 seconds.

3. The step-up JWT carries payout:approve, so Quarkus allows the sensitive operation.

The step-up token also contains "amr": ["pwd", "otp"]. The names come from RFC 8176, which defines authentication-method reference values that a relying service can inspect. amr describes how the user authenticated. The role still decides what the token may do.

LedgerLock issues both tokens so the example stays local. The development token endpoint simulates the result of a first-factor login; it does not verify a password. In a system with Keycloak, Auth0, Entra ID, or another OpenID Connect provider, that provider would normally own the first factor and often the second one too.

What you need

This article uses Quarkus 3.37.2, Quarkus Vault 4.8.0, Java 21, and Vault 2.0.3.

• Java 25 or newer

• The Quarkus CLI

• Podman with a running machine

• curl and jq

• An authenticator app that supports TOTP

• About ☕️☕️☕️☕️ (< security topics are hard!)

Create the application

Create the project or start from my Github repository:

quarkus create app com.themainthread:vault-totp-step-up \
  --extension='rest-jackson,smallrye-jwt,smallrye-jwt-build,hibernate-validator,hibernate-orm-panache,jdbc-h2,smallrye-health' \
  --no-code

quarkus-rest-jackson gives us the JSON endpoints. SmallRye JWT verifies the bearer tokens and signs the local step-up token. Hibernate Validator rejects malformed TOTP input before it reaches Vault. Panache and H2 keep the subject-to-Vault-key mapping for this runnable example, and SmallRye Health exposes the Vault readiness check.

Vault is a Quarkiverse extension, so add its current version explicitly:

<dependency>
    <groupId>io.quarkiverse.vault</groupId>
    <artifactId>quarkus-vault</artifactId>
    <version>4.8.0</version>
</dependency>

Configure Vault Dev Services

Add the following to src/main/resources/application.properties:

# Vault Dev Services starts only in dev and test mode.
quarkus.vault.devservices.image-name=hashicorp/vault:2.0.3
quarkus.vault.devservices.shared=false
quarkus.vault.devservices.init-commands=secrets enable totp
quarkus.vault.health.enabled=true

# The tutorial keeps enrollment state local so it needs no second container.
quarkus.datasource.db-kind=h2
quarkus.datasource.jdbc.url=jdbc:h2:mem:ledgerlock;DB_CLOSE_DELAY=-1
quarkus.hibernate-orm.schema-management.strategy=drop-and-create

# SmallRye JWT generates a signing key pair automatically in dev and test mode.
mp.jwt.verify.issuer=https://ledgerlock.local
mp.jwt.verify.audiences=ledgerlock-api
mp.jwt.verify.clock.skew=5
smallrye.jwt.new-token.issuer=https://ledgerlock.local
smallrye.jwt.new-token.audience=ledgerlock-api
smallrye.jwt.new-token.lifespan=300

# Vault is a request dependency for enrollment and verification.
quarkus.vault.connect-timeout=3S
quarkus.vault.read-timeout=3S

Dev Services already knows how to start Vault and inject the URL and root token in dev and test mode. It does not enable the TOTP secrets engine by default. init-commands passes secrets enable totp to the container during startup, which removes the manual Vault UI step that older examples often needed.

Set the image explicitly. HashiCorp recommends using the latest fix release within a supported line, and the old 1.15.2 default predates the TOTP replay fix. Vault 2.0.3 is the current release, and the test below runs against it. Vault 2.x contains breaking changes, so review the 2.x release notes and upgrade guidance before changing an existing production cluster.

Quarkus also generates an in-memory JWT key pair in dev and test mode. Production needs a real signing key, a JWK set, or an identity provider. The build prints a warning when no production key is configured.

Keep enrollment state outside the TOTP seed

Vault stores the TOTP seed and performs code validation. LedgerLock still needs to know which opaque Vault key belongs to each authenticated subject. Create TotpEnrollment.java:

package com.themainthread.ledgerlock;

import java.time.Instant;

import io.quarkus.hibernate.orm.panache.PanacheEntityBase;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;

@Entity
@Table(name = "totp_enrollment")
public class TotpEnrollment extends PanacheEntityBase {

    @Id
    @Column(nullable = false, updatable = false)
    public String subject;

    @Column(name = "vault_key", nullable = false, unique = true, updatable = false)
    public String vaultKey;

    @Column(name = "created_at", nullable = false, updatable = false)
    public Instant createdAt;
}

The database stores no TOTP seed. vaultKey is a random identifier such as user-86483632-4d7b-498f-ae59-b4998d62b53e. Using an opaque value keeps email addresses and usernames out of Vault paths and audit entries.

An enrollment already present for the subject returns 409 Conflict. Rotation needs its own flow that asks for an existing factor or a recovery credential. Otherwise, a stolen access token would be enough to replace the second factor.

Put the Vault interaction in one service

Create TotpService.java:

package com.themainthread.ledgerlock;

import java.time.Instant;
import java.util.Optional;
import java.util.UUID;

import org.jboss.logging.Logger;

import io.quarkus.vault.VaultTOTPSecretEngine;
import io.quarkus.vault.client.VaultClientException;
import io.quarkus.vault.secrets.totp.CreateKeyParameters;
import io.quarkus.vault.secrets.totp.KeyDefinition;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.persistence.PersistenceException;
import jakarta.transaction.Transactional;
import jakarta.ws.rs.ClientErrorException;
import jakarta.ws.rs.core.Response;

@ApplicationScoped
public class TotpService {

    private static final Logger LOG = Logger.getLogger(TotpService.class);

    private final VaultTOTPSecretEngine totpEngine;

    public TotpService(VaultTOTPSecretEngine totpEngine) {
        this.totpEngine = totpEngine;
    }

    @Transactional
    public EnrollmentResponse enroll(String subject) {
        if (TotpEnrollment.findById(subject) != null) {
            throw new ClientErrorException("TOTP is already enrolled", Response.Status.CONFLICT);
        }

        String vaultKey = "user-" + UUID.randomUUID();
        CreateKeyParameters parameters = new CreateKeyParameters("LedgerLock", subject);
        parameters.setExported(true);
        parameters.setPeriod("30s");
        parameters.setAlgorithm("SHA1");
        parameters.setDigits(6);
        parameters.setSkew(1);
        parameters.setQrSize(240);

        Optional<KeyDefinition> created = totpEngine.createKey(vaultKey, parameters);
        KeyDefinition definition = created.orElseThrow(
                () -> new IllegalStateException("Vault did not export the TOTP enrollment data"));

        TotpEnrollment enrollment = new TotpEnrollment();
        enrollment.subject = subject;
        enrollment.vaultKey = vaultKey;
        enrollment.createdAt = Instant.now();

        try {
            enrollment.persistAndFlush();
        } catch (PersistenceException failure) {
            deleteCompensatingKey(vaultKey);
            throw new ClientErrorException("TOTP enrollment already exists", Response.Status.CONFLICT, failure);
        }

        return new EnrollmentResponse(
                "data:image/png;base64," + definition.getBarcode(),
                definition.getUrl());
    }

    public boolean validate(String subject, String code) {
        TotpEnrollment enrollment = TotpEnrollment.findById(subject);
        if (enrollment == null) {
            return false;
        }

        try {
            return totpEngine.validateCode(enrollment.vaultKey, code);
        } catch (VaultClientException failure) {
            if (failure.getStatus() == 400 && failure.hasErrorContaining("code already used")) {
                return false;
            }
            throw failure;
        }
    }

    private void deleteCompensatingKey(String vaultKey) {
        try {
            totpEngine.deleteKey(vaultKey);
        } catch (RuntimeException cleanupFailure) {
            LOG.warnf(cleanupFailure, "Could not remove orphaned Vault TOTP key %s", vaultKey);
        }
    }

    public record EnrollmentResponse(String qrCodeDataUrl, String manualEntryUri) {
    }
}

The constructor new CreateKeyParameters("LedgerLock", subject) asks Vault to generate the seed. exported=true returns the QR code and otpauth:// URI once so the user can bind an authenticator app. The Vault TOTP API also supports imported seeds, different periods, eight-digit codes, SHA-256, and SHA-512. Six digits, a 30-second period, and SHA-1 work with most authenticator apps. SHA-1 is the HMAC algorithm used for TOTP compatibility here. We do not use it to hash a password.

skew=1 accepts a neighboring time step to handle normal clock drift. This helps users near a 30-second boundary, but it also increases the number of accepted codes. Keep clocks synchronized and only increase the window when you have a clear reason.

The database transaction cannot include Vault. If Vault creates a key and the database insert then loses a uniqueness race, the catch block deletes the orphaned key. There are still crash windows between those two systems. A production enrollment service needs reconciliation for orphaned keys and records, usually with an explicit PENDING state.

One Vault behavior is easy to miss. A wrong code normally returns false. A reused code makes patched Vault return HTTP 400 with code already used; wait until the next time period. The Quarkus wrapper turns that response into VaultClientException. Reuse is an authentication failure, so the service maps this specific response to false. Other Vault failures still escape because they are operational errors.

Exchange the code for higher assurance

The enrollment response contains the TOTP seed, whether represented as a QR image or a URI. It should be delivered once over TLS and never cached. The resource also requires an already authenticated user.

Create TotpResource.java:

package com.themainthread.ledgerlock;

import org.eclipse.microprofile.jwt.JsonWebToken;

import jakarta.annotation.security.RolesAllowed;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Pattern;
import jakarta.ws.rs.ClientErrorException;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.CacheControl;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;

@Path("/api/totp")
@Produces(MediaType.APPLICATION_JSON)
@RolesAllowed("user")
public class TotpResource {

    private final JsonWebToken accessToken;
    private final TotpService totpService;
    private final StepUpTokenService stepUpTokenService;

    public TotpResource(JsonWebToken accessToken, TotpService totpService, StepUpTokenService stepUpTokenService) {
        this.accessToken = accessToken;
        this.totpService = totpService;
        this.stepUpTokenService = stepUpTokenService;
    }

    @POST
    @Path("/enrollment")
    public Response enroll() {
        TotpService.EnrollmentResponse enrollment = totpService.enroll(accessToken.getSubject());
        CacheControl noStore = new CacheControl();
        noStore.setNoStore(true);
        noStore.setNoCache(true);
        return Response.status(Response.Status.CREATED)
                .cacheControl(noStore)
                .header("Pragma", "no-cache")
                .entity(enrollment)
                .build();
    }

    @POST
    @Path("/step-up")
    public StepUpResponse stepUp(@Valid OtpRequest request) {
        String subject = accessToken.getSubject();
        if (!totpService.validate(subject, request.code())) {
            throw new ClientErrorException("The TOTP code is invalid or was already used", Response.Status.UNAUTHORIZED);
        }

        return new StepUpResponse(stepUpTokenService.issueFor(subject), StepUpTokenService.LIFESPAN_SECONDS);
    }

    public record OtpRequest(
            @NotBlank @Pattern(regexp = "[0-9]{6}", message = "must contain exactly six ASCII digits") String code) {
    }

    public record StepUpResponse(String token, long expiresInSeconds) {
    }
}

[0-9]{6} makes the ASCII input boundary explicit, and the resource does not trim the value. The 2025 Vault vulnerability made used-code cache entries bypassable by appending whitespace. The patched server fixes that flaw, and the resource rejects the malformed value before it crosses the network. Both controls are cheap.

Now create the token service:

package com.themainthread.ledgerlock;

import java.time.Instant;
import java.util.List;
import java.util.Set;

import io.smallrye.jwt.build.Jwt;
import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class StepUpTokenService {

    static final long LIFESPAN_SECONDS = 120;

    public String issueFor(String subject) {
        return Jwt.subject(subject)
                .upn(subject)
                .groups(Set.of("user", "payout:approve"))
                .claim("amr", List.of("pwd", "otp"))
                .claim("acr", "urn:ledgerlock:assurance:mfa")
                .expiresAt(Instant.now().plusSeconds(LIFESPAN_SECONDS))
                .sign();
    }
}

The token keeps the same subject and adds the narrow role needed for the payout. Its amr claim records that password and OTP methods were used. The acr value is private to LedgerLock and means the token met our local MFA policy. Other services must agree on that meaning before they trust it.

The token expires after two minutes, so the extra role is available only for a short time. This example allows more than one payout during that window. For one high-value transaction, bind the token to a payout ID or operation hash and consume its jti once.

Protect the business operation with normal RBAC

Create PayoutResource.java:

package com.themainthread.ledgerlock;

import java.math.BigDecimal;
import java.util.UUID;

import org.eclipse.microprofile.jwt.JsonWebToken;

import jakarta.annotation.security.RolesAllowed;
import jakarta.validation.Valid;
import jakarta.validation.constraints.DecimalMin;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;

@Path("/api/payouts")
@Produces(MediaType.APPLICATION_JSON)
public class PayoutResource {

    private final JsonWebToken accessToken;

    public PayoutResource(JsonWebToken accessToken) {
        this.accessToken = accessToken;
    }

    @POST
    @RolesAllowed("payout:approve")
    public Response approve(@Valid PayoutRequest request) {
        ApprovedPayout payout = new ApprovedPayout(
                UUID.randomUUID(),
                request.recipient(),
                request.amount(),
                accessToken.getSubject(),
                "APPROVED");
        return Response.status(Response.Status.ACCEPTED).entity(payout).build();
    }

    public record PayoutRequest(
            @NotBlank String recipient,
            @NotNull @DecimalMin("0.01") BigDecimal amount) {
    }

    public record ApprovedPayout(
            UUID id,
            String recipient,
            BigDecimal amount,
            String approvedBy,
            String status) {
    }
}

The payout code knows nothing about Vault or six-digit codes. Quarkus verifies the JWT and enforces @RolesAllowed. TOTP raises the authentication level once, and the business endpoint only checks the resulting role.

Stand in for the first factor in local development

We still need a normal access token to start the flow. This development-only endpoint issues one with the user role:

package com.themainthread.ledgerlock;

import java.util.List;
import java.util.Set;

import io.quarkus.arc.profile.UnlessBuildProfile;
import io.smallrye.jwt.build.Jwt;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/dev/token")
@Produces(MediaType.APPLICATION_JSON)
@UnlessBuildProfile("prod")
public class DevTokenResource {

    @GET
    @Path("/{subject}")
    public TokenResponse token(@PathParam("subject") String subject) {
        String token = Jwt.subject(subject)
                .upn(subject)
                .groups(Set.of("user"))
                .claim("amr", List.of("pwd"))
                .sign();
        return new TokenResponse(token);
    }

    public record TokenResponse(String token) {
    }
}

@UnlessBuildProfile("prod") removes the resource from a production build. It is a local replacement for the access token your identity provider would issue after the first-factor login. A production token endpoint with no login would bypass every other control in this example.

Run the full flow

Start the application with Podman available:

./mvnw quarkus:dev

Quarkus starts Vault 2.0.3, enables the TOTP engine, creates the H2 schema, and generates the dev JWT key pair. Check readiness:

curl -s http://localhost:8080/q/health/ready | jq

The response should include an UP check for Vault.

Get Alice’s first-factor token:

TOKEN=$(curl -s http://localhost:8080/dev/token/alice | jq -r .token)

Enroll the authenticator:

curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/totp/enrollment \
  | tee enrollment.json

The response contains qrCodeDataUrl and manualEntryUri. Both contain the TOTP seed, so keep enrollment.json only long enough to bind the authenticator. To display the QR code, create a small local HTML file:

printf '<img src="%s">' "$(jq -r .qrCodeDataUrl enrollment.json)" > enrollment.html

Open enrollment.html in a browser and scan it. Then replace 123456 below with the current code:

STEP_TOKEN=$(curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"code":"123456"}' \
  http://localhost:8080/api/totp/step-up \
  | jq -r .token)

The first token still cannot approve a payout:

curl -i -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"recipient":"supplier-42","amount":1250.00}' \
  http://localhost:8080/api/payouts

Expected status:

HTTP/1.1 403 Forbidden

Use the step-up token:

curl -s -X POST \
  -H "Authorization: Bearer $STEP_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"recipient":"supplier-42","amount":1250.00}' \
  http://localhost:8080/api/payouts \
  | jq

You should get an accepted payout:

{
  "id": "a1e0d0d9-0f19-4da3-8d11-099aeb7314b1",
  "recipient": "supplier-42",
  "amount": 1250.00,
  "approvedBy": "alice",
  "status": "APPROVED"
}

Send the same TOTP code to /api/totp/step-up again during its 30-second window. The API returns 401 Unauthorized. Add a trailing space and it returns 400 Bad Request before Vault receives it.

Prove replay handling in a test

The manual flow shows the user experience. The automated test asks Vault to generate the code for the enrolled key, then proves the security properties without waiting for somebody to type from a phone:

package com.themainthread.ledgerlock;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.startsWith;

import java.math.BigDecimal;
import java.util.Map;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.vault.VaultTOTPSecretEngine;
import jakarta.inject.Inject;

@QuarkusTest
class TotpStepUpTest {

    @Inject
    VaultTOTPSecretEngine totpEngine;

    @Test
    void aTotpCodeCreatesAShortLivedStepUpTokenAndCannotBeReplayed() {
        String firstFactorToken = given()
                .when().get("/dev/token/alice")
                .then()
                .statusCode(200)
                .extract().path("token");

        given()
                .auth().oauth2(firstFactorToken)
                .when().post("/api/totp/enrollment")
                .then()
                .statusCode(201)
                .header("Cache-Control", startsWith("no-cache"))
                .body("qrCodeDataUrl", startsWith("data:image/png;base64,"))
                .body("manualEntryUri", startsWith("otpauth://totp/"));

        String vaultKey = totpEngine.listKeys().getFirst();
        String code = totpEngine.generateCode(vaultKey);

        String stepUpToken = given()
                .auth().oauth2(firstFactorToken)
                .contentType("application/json")
                .body(Map.of("code", code))
                .when().post("/api/totp/step-up")
                .then()
                .statusCode(200)
                .body("expiresInSeconds", equalTo(120))
                .extract().path("token");

        given()
                .auth().oauth2(firstFactorToken)
                .contentType("application/json")
                .body(new PayoutResource.PayoutRequest("supplier-42", new BigDecimal("1250.00")))
                .when().post("/api/payouts")
                .then()
                .statusCode(403);

        given()
                .auth().oauth2(stepUpToken)
                .contentType("application/json")
                .body(new PayoutResource.PayoutRequest("supplier-42", new BigDecimal("1250.00")))
                .when().post("/api/payouts")
                .then()
                .statusCode(202)
                .body("approvedBy", equalTo("alice"))
                .body("status", equalTo("APPROVED"));

        given()
                .auth().oauth2(firstFactorToken)
                .contentType("application/json")
                .body(Map.of("code", code))
                .when().post("/api/totp/step-up")
                .then()
                .statusCode(401);

        given()
                .auth().oauth2(firstFactorToken)
                .contentType("application/json")
                .body(Map.of("code", code + " "))
                .when().post("/api/totp/step-up")
                .then()
                .statusCode(400);
    }
}

Run it:

./mvnw test

The test uses the same pinned Vault Dev Service as dev mode. The Vault version is part of the test fixture, so changing it runs the replay test again.

Before you use this in production

The example now has a clear two-factor flow. A production TOTP service still needs lifecycle and abuse controls that are outside this small application.

Let the identity provider own MFA when it can. If your OpenID Connect provider already enrolls authenticators and issues trustworthy amr and acr claims, consume those claims in Quarkus. An application-owned TOTP verifier makes sense for a dedicated authentication service, a controlled internal platform, or a legacy boundary where the central provider cannot express the required step-up flow.

Give the application a narrow Vault identity. Dev Services uses a root token because the container is disposable. Production should use Kubernetes auth, AppRole, or another machine authentication method with TLS and a policy limited to the application’s totp/keys/... and totp/code/... paths. The application validates codes; it should not have permission to generate user codes from GET /totp/code/:name.

Rate-limit failures by account and context. A six-digit code has a small online search space. NIST SP 800-63B-4 requires throttling for online guessing and treats 100 consecutive failures as an upper bound, not a target. Use a much lower risk-based threshold where your user population and recovery process allow it. Put the counter in shared storage, an API gateway, or the identity service so adding Quarkus replicas does not multiply the guess budget.

Build recovery and rotation deliberately. Users lose phones. They also replace them. Recovery codes need hashed storage, one-time consumption, monitoring, and a notification path. Rotation should require an existing factor or a verified recovery process. Keep enrollment states such as PENDING, ACTIVE, and REVOKED, and delete abandoned Vault keys after a bounded time.

Keep the QR response out of logs and caches. The QR image and otpauth:// URI both carry the seed. Cache-Control: no-store helps with HTTP caches, but your reverse proxy, tracing, body logging, browser history, and support tooling also need review. Never log KeyDefinition; its toString() includes the barcode and URI.

Decide what happens when Vault is down. The health check can mark the service unready, but that may remove every replica at once during a Vault outage. Existing step-up tokens can still pass local JWT verification until they expire. New enrollment and verification cannot. Choose that failure mode consciously, monitor it, and keep the Vault read timeout short.

TOTP is not phishing-resistant. Manually entered OTPs are replay-resistant when the verifier accepts each value once, but NIST does not consider them phishing-resistant. A fake login page can relay a current code. Passkeys and WebAuthn bind authentication to the verifier and are the stronger default for new public-facing systems. TOTP remains relevant for compatibility, controlled enterprise environments, and recovery paths. Treat it as one supported authentication level, not the final design for every system.

Keep the responsibilities separate

Vault owns the TOTP seed and the one-time validation rule. Quarkus owns the authenticated subject, enrollment state, and authorization decision. The six-digit code crosses that boundary once and becomes a short-lived claim that ordinary RBAC can enforce.

This is a better design than adding X-TOTP-Code to every protected request. The first-factor result is a signed token. The sensitive operation asks for fresh assurance. We test replay behavior against the real Vault runtime, and the business endpoint stays free of authentication code.

Subscribe now

Then it gets copied into a second repo. Then a third. One copy still points at an old path. One still has placeholder text. One quietly relaxed the rules because somebody wanted to get through a demo. At one point the file is part of the system. It needs the same kind of care you already give to scripts and config.

That is why I like skillsaw. It treats skill files like artifacts you can inspect and lint. The tool focuses on file quality: wrong shape, bad naming, leaked placeholder text, vague instructions. That catches a lot of drift early.

The example here is artificial on purpose. We use one small Bob skill called invoice-processor. It validates invoice totals, writes results to one output file, logs mismatches, preserves input order, and forbids shell commands. The rules are narrow, which makes them easy to inspect.

The result is a small skill folder, a clean skillsaw loop, a GitHub Actions check, and a Bob prompt that runs on a local input.json file with no external service, no MCP server, and no API keys.

What You Need

You should already be comfortable editing Markdown, using a terminal, and running IBM Bob inside an IDE.

• Python 3.11 or newer

• IBM Bob in your IDE (download a trial if you like)

• GitHub Actions if you want the CI part

• About ☕️

Create The Skill Repo

First decide who owns the skill. That decision is way more important than the directory name.

Use a repo-local skill when the rules only make sense inside one codebase. In that case, keep the skill with the project, for example under .bob/skills, and review it with the same pull request that changes the workflow it controls.

Use a separate skill repo when the skill is shared across several repos, several teams, or several projects. That repo becomes the source of truth. It should have its own lint check, release tags, and a small changelog so consumers can tell which skill release they installed.

Use personal skill directories only for experiments or private habits. They are fine for local work. They are a bad place to hide team behavior because nobody else can review or reproduce them.

For this tutorial we use a separate repo because we want to show the full skillsaw loop: lint the skill, pin the rule set, run CI, and then install the checked skill into Bob.

Create the layout:

mkdir -p main-thread-skills/invoice-processor/references
cd main-thread-skills
python3 -m venv .venv
.venv/bin/python -m pip install skillsaw==0.16.0

If you already use uv, the skillsaw quick start also supports uvx skillsaw.

Start with a bad skill on purpose. The lint output is easier to read when the file is obviously rough.

Create invoice-processor/SKILL.md:

---
name: Invoice Processor
description: process invoices
---

# Invoice Processor

Process invoice batches.

You can use shell scripts if that is faster.

Write results somewhere under out/.

Maybe stop on errors.

TODO: tighten this.

This looks small if you skim it. It also gives Bob far too much room to improvise.

Read The Tree First

Before linting anything, ask skillsaw to show what it found in the repo.

Run:

.venv/bin/skillsaw tree .

For this example, the output is:

main-thread-skills/ (51 tokens)
    └── invoice-processor/ [skill] (51 tokens)
        └── SKILL.md (skill) (51 tokens)
            ├── frontmatter:name (4 tokens)
            ├── frontmatter:description (4 tokens)
            └── body (43 tokens)

If skillsaw does not even see a skill, stop there and fix the layout first.

Lint The Rough Draft

Stay in `main-thread-skills`. That is the repo root for this example.

Now run a different subcommand: `lint`, not `tree`:

.venv/bin/skillsaw lint .

On the rough draft above, you should see findings in this shape:

skillsaw 0.16.0
Linting: /path/to/main-thread-skills

Errors:
  ✗ ERROR (agentskill-name) [invoice-processor/SKILL.md:2]: Name 'Invoice Processor' must contain only lowercase letters, numbers, and hyphens

Warnings:
  ⚠ WARNING (content-placeholder-text) [invoice-processor/SKILL.md:16]: Placeholder text (TODO marker): 'TODO'

The output points at the real problem quickly: the instructions are sloppy.

If you want to find out more about rule warnings and what they mean, ask the tool:

.venv/bin/skillsaw explain agentskill-name
.venv/bin/skillsaw explain content-placeholder-text

Make sure to agree, as a team, on those definitions. We do not want them to feel arbitrary to anyone.

Use Auto-Fix For Mechanical Problems

skillsaw fix handles the mechanical cleanup.

Preview the changes first:

.venv/bin/skillsaw fix --dry-run .

For the bad front matter above, the preview shows a diff like this:

--- a/invoice-processor/SKILL.md
+++ b/invoice-processor/SKILL.md
@@ -1,5 +1,5 @@
 ---
-name: Invoice Processor
+name: invoice-processor
  description: process invoices
 ---

Then apply the safe fixes:

.venv/bin/skillsaw fix -y .

Renaming the front matter takes seconds. Writing rules that leave Bob less room to guess takes longer.

Write The Real Skill

Now write the version Bob can use in real work.

This skill is narrow on purpose. It covers one batch job, one input file, one output file, one error log, and one response shape. That makes it a good target for skillsaw because vague wording shows up fast.

Create invoice-processor/SKILL.md:

---
name: invoice-processor
description: >
  Process invoice batches from input.json, validate totals against line items,
  write validated output to out/invoices.json, log mismatches to errors.log,
  and return a JSON summary.
---

# Invoice Processor

Read [validation examples](references/validation-examples.md) when you need an example of valid and invalid totals. The numbered rules below are the source of truth.

Rules for processing invoice batches. Follow every rule exactly.

1. MUST validate every invoice total against the sum of its line-item amounts (`quantity * unitPrice`) before writing output.
2. MUST NOT execute shell commands. Process invoices using only file read/write tools. Do not run scripts, terminals, or any command execution.
3. MUST write validated results to `out/invoices.json` and nowhere else. Do not write invoice data to any other path.
4. MUST process invoices in the exact order they appear in `input.json`. Do not sort or reorder.
5. On validation failure (`total` does not match line-item sum), MUST log the invoice ID and discrepancy to `errors.log` and continue processing the remaining invoices. Do not abort the batch.
6. Final response to the user MUST be valid JSON matching this shape exactly: `{"processed": N, "failed": N}` where `N` is the count of successfully validated and failed invoices respectively.

This version closes the gaps from the first draft. It makes the shell ban, output path, failure handling, and response format explicit. The reference link gives Bob examples without moving the rules out of the main file.

Add A Reference File

Reference files help when the main skill should stay short but Bob still needs examples, glossary entries, workspace notes, or longer decision rules.

Create invoice-processor/references/validation-examples.md:

# Validation Examples

Use these examples to check invoice total behavior.

## Valid Total

Calculate each line item as `quantity * unitPrice`.
Add the line-item amounts.
Compare the sum with the invoice `total`.

```json
{
  "id": "INV-001",
  "total": 25,
  "lineItems": [
    { "quantity": 1, "unitPrice": 10 },
    { "quantity": 3, "unitPrice": 5 }
  ]
}
```

This invoice passes because `10 + 15 = 25`.

## Invalid Total

Calculate the same way for failures.
Record the invoice ID and discrepancy.
Continue with the next invoice.

```json
{
  "id": "INV-002",
  "total": 18,
  "lineItems": [
    { "quantity": 2, "unitPrice": 5 },
    { "quantity": 1, "unitPrice": 7 }
  ]
}
```

The second invoice has a discrepancy of `1`.

Run tree again:

.venv/bin/skillsaw tree .

Now skillsaw shows the reference as skill-ref:

main-thread-skills/ (442 tokens)
    └── invoice-processor/ [skill] (442 tokens)
        ├── SKILL.md (skill) (329 tokens)
        │   ├── frontmatter:name (4 tokens)
        │   ├── frontmatter:description (44 tokens)
        │   └── body (281 tokens)
        └── validation-examples.md (skill-ref) (113 tokens)

I also tried a Python helper under scripts/validate_invoice.py. scripts/ is an allowed skill subdirectory, along with assets, evals, and references, but skillsaw 0.16.0 did not content-lint that Python file in this setup. A TODO comment inside the script did not trigger content-placeholder-text.

That does not make scripts bad. It means skillsaw is checking skill structure and instruction content. It does not replace Ruff, Pyright, or your normal code checks.

Lint The Finished Skill

Run the linter again:

.venv/bin/skillsaw lint -v .

On this version, the run passes with no errors and no warnings. skillsaw still prints two info-level suggestions:

Info:
  ℹ INFO (content-actionability-score) [invoice-processor/references/validation-examples.md]: Low actionability score: 18/100 (verbs: 17%, commands: 33%, paths: 0%)
  ℹ INFO (content-unlinked-internal-reference) [invoice-processor/SKILL.md:17]: Unlinked path reference: 'out/invoices.json' — consider wrapping in link syntax [out/invoices.json](out/invoices.json)

✓ All checks passed!

I would leave both alone. The reference file is example-heavy by design, so a low actionability score is not surprising. The bare out/invoices.json path is also part of a hard constraint, and I do not want to turn that sentence into link noise only to satisfy an info rule. skillsaw makes that distinction visible: the file passes, and the tool still points out things you may want to clean up later.

Pin The Rule Set

Once the skill is in decent shape, generate a config file:

.venv/bin/skillsaw init .

This writes .skillsaw.yaml with the current skillsaw rule set pinned in it:

version: "0.16.0"

That version is not the version of the invoice-processor skill. It is the skillsaw config and rule-set version. We do still want to pin the specific skillsaw version because new rules added after 0.16.0 do not start changing CI behavior until you choose to bump the config and potentially add new linters.

If you want to release your skill as 1.0.0, do that separately with normal repo versioning: a Git tag, release notes, or a small changelog. If you want to package reusable skills instead of copying folders around, I covered one Java-friendly way in SkillsJars for Java: Package Reusable Agent Skills. Do not use .skillsaw.yaml for skill releases.

If you are adding skillsaw to an older internal repo with a pile of existing issues, this is where the baseline feature helps:

.venv/bin/skillsaw baseline

For a new repo like this one, I would rather fix the warnings than baseline them. Baseline helps when history is already messy. It should not be the main workflow.

Add CI

Now make CI fail when somebody reintroduces placeholders, weak naming, or other soft edits.

Create .github/workflows/skillsaw.yml:

name: skillsaw

on:
  pull_request:
  push:
    branches:
      - main

jobs:
  lint-skills:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: stbenjam/skillsaw@v0
        with:
          path: .
          strict: true

strict: true matters here because warnings are often the first sign that the wording is getting soft before the file is clearly broken.

Other Features Worth Knowing

This tutorial uses the core loop: tree, lint, explain, fix, init, baseline, and CI. skillsaw has more surface area once you move past the first repo.

• list-rules shows every built-in rule and its default severity.

• --rule and --skip-rule let you run a focused check while debugging one rule.

• --format and --output can write json, sarif, html, code-climate, or gitlab reports for other tools.

• badge writes a shields.io-compatible .skillsaw-badge.json with the current grade.

• add scaffolds marketplaces, plugins, skills, commands, agents, and hooks.

• docs generates documentation for plugins, marketplaces, and .claude repositories. It is less relevant for this tiny agentskills repo, but it matters once skills live inside a plugin.

• evals/evals.json is validated when present. Requiring evals for every skill is opt-in with agentskill-evals-required.

• .skillsaw.yaml supports global excludes, per-rule excludes, inline suppressions, extra content-paths, and local custom rule files.

For untrusted pull requests, run CI with --no-custom-rules. Built-in rules are the safer default. Custom Python rules are code execution, so treat them with the same caution you would give any other contributor-controlled script.

Install The Skill Into Bob

For a local install, Bob needs the files where it can read them. The tutorial uses a copy command because it is easy to inspect.

In the repo where you want Bob to use the skill, copy it into .bob/skills.

For a team, avoid hand-copying forever. Pick one repeatable install method and document it: a small sync script, a Git submodule, a Git subtree, or a release archive that gets unpacked into .bob/skills. The boring best practice is that every consuming repo can answer three questions: where the skill came from, which skill release it uses, and how CI proves the installed copy still passes skillsaw.

Create A Tiny Test Workspace

Create a small test repo or scratch directory with one input.json file:

[
  {
    "id": "INV-001",
    "total": 25,
    "lineItems": [
      { "quantity": 1, "unitPrice": 10 },
      { "quantity": 3, "unitPrice": 5 }
    ]
  },
  {
    "id": "INV-002",
    "total": 18,
    "lineItems": [
      { "quantity": 2, "unitPrice": 5 },
      { "quantity": 1, "unitPrice": 7 }
    ]
  }
]

INV-001 is valid. INV-002 fails validation. The line items sum to 17, while the invoice total says 18. That gives us one clean pass and one clean failure in the same run.

Create the output directory before you hand this to Bob:

mkdir -p out

Use It In Bob

Now give Bob a task that looks like normal local work.

Use a prompt like this:

Use invoice-processor to process input.json. 

Watch the behavior.

Check that Bob does all of the following:

• Reads input.json

• Keeps invoices in the original order

• Writes validated output only to out/invoices.json

• Logs the failed invoice ID and discrepancy to errors.log

• Continues processing after the bad invoice instead of aborting

• Returns only JSON in the exact shape {"processed": N, "failed": N}

• Avoids shell commands entirely

For the sample data above, the expected final response shape is:

{"processed": 1, "failed": 1}

If Bob opens a terminal, writes invoice data to another path, reorders the records, or wraps the JSON in prose, the skill is still too vague or the tool constraints are not being followed.

Why This Version Works Better

The rough version was shorter. It was also loose enough that Bob had to guess.

The finished skill is still small, but the boundaries are clear: one input file, one output file, one error log, one processing order, one response shape, and one hard ban on shell commands. That is what I want in a real repo. The file stays short, but it is specific enough that a reviewer can tell what should happen.

skillsaw works well for this because it catches the plain failures early: wrong front matter, placeholder text, weak naming, and general sloppiness before those files spread into more repos.

We built one Bob skill, linted it, wired CI, installed it into .bob/skills, and tested it on a local batch with no external setup. Once a SKILL.md changes agent behavior, treat it like code.

Subscribe now

That log line records an enforced decision. The filesystem restriction lives in policy. It does not depends on a model hopefully following an instruction.

Prompt guardrails inspect language before or after a model call. The recent Quarkus OPA guardrail article shows how to compile a Rego policy to WebAssembly and evaluate it inside a Quarkus LangChain4j application. Its policy reads prompt text, including regular expressions. To be completely honest, I am not a big fan of prompt inspection with RegEx. It just does not scale. And is just a weak and potentially harmful approach which will never solve the challenges you will experience in guardrails. I’ve written about this before.

But there are great use-cases for this. In this example, OPA reads facts that the application already knows: the user, agent, session, skill publisher, signature status, and requested scopes.

I use Fernbank as a fictional internal agent platform in this example. Teams publish document generators, database helpers, and deployment skills into a shared catalog. A scanner can flag a suspicious manifest in CI, but the report does not block runtime access. Fernbank needs a gate at the point where it exposes the tool.

We build the gate as a Quarkus MCP server. OPA evaluates every tool against its manifest and the caller context. Allowed tools appear in tools/list and can be called. Denied tools stay out of the list, and a direct tools/call also fails. Each evaluation writes the policy version and stable reason codes to a fixed-size audit buffer.

The Quarkus Enforcement Boundary

We need two related controls. They run at different times and solve different problems.

Catalog admission decides whether a skill may enter the shared catalog. It runs once per skill version and uses publisher data, signature status, source information, and declared capabilities.

Per-client exposure decides whether a connected user, agent, or session may see and invoke a tool. It runs for each caller context.

Quarkus MCP Server discovers annotation-based tools at build time. The OPA filter does not prevent the JVM from loading the class. The extension’s ToolFilter API controls whether a registered tool is visible and accessible for the current MCP connection. We use that hook for per-client exposure.

If Fernbank loaded third-party plugin code dynamically, I would add a catalog-admission job before global registration through ToolManager. A tool registered with ToolManager is available across the server. ToolFilter still decides which subset each client sees.

The request path is:

The Quarkus MCP filter documentation says that filters execute on the Vert.x event loop. They must finish quickly. A synchronous HTTP call to a separate OPA sidecar process would block that thread, so we compile Rego to Wasm and evaluate it in the Quarkus process.

The extension also ignores a filter when that filter throws an exception. Our filter catches policy failures and returns false. A failed policy evaluation therefore denies access.

What You Need

This example uses Quarkus 3.37.2, Quarkus MCP Server 1.13.1, Java 25, OPA 1.17.0, and opa-java-wasm 1.1.0. The verification covers JVM mode. I have not tested this Wasm runtime in a Quarkus native executable, so native mode is outside the scope of this example.

• Java 25

• Podman with its machine running on macOS or Windows

• The Quarkus CLI

• Basic MCP and Rego knowledge

• About ☕️☕️

Create the Quarkus MCP Server

Create the application or start from my Github repository:

quarkus create app com.themainthread:fernbank-skill-admission \
  --platform-bom=io.quarkus.platform:quarkus-bom:3.37.2 \
  --java=25 \
  --extension=rest-jackson,io.quarkiverse.mcp:quarkus-mcp-server-http \
  --no-code
cd fernbank-skill-admission

quarkus-mcp-server-http provides the Streamable HTTP MCP endpoint and the filter API. quarkus-rest-jackson maps manifests, policy input, policy output, and the decision-log endpoint. In this version, the Quarkus platform BOM manages the MCP extension.

Add the Wasm evaluator and the MCP test client to pom.xml:

<properties>
    <opa-java-wasm.version>1.1.0</opa-java-wasm.version>
</properties>

<dependencies>
    <dependency>
        <groupId>com.styra.opa</groupId>
        <artifactId>opa-java-wasm</artifactId>
        <version>${opa-java-wasm.version}</version>
    </dependency>
    <dependency>
        <groupId>io.quarkiverse.mcp</groupId>
        <artifactId>quarkus-mcp-server-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

The Quarkus guardrail example from the earlier mentioned blog post uses the Styra library too. Our code changes the policy input and evaluates it from an MCP filter.

Use the Manifest as Policy Input

The first tool belongs to Fernbank’s internal documentation team. We put its manifest in src/main/resources/skills/docs_generate.json:

{
  "skill_id": "docs_generate",
  "publisher": "internal:docs-platform",
  "publisher_trust_tier": "internal-verified",
  "signature_verified": true,
  "requested_scopes": ["context:read", "filesystem:write"],
  "declared_capabilities": ["document-generation"],
  "allowed_teams": ["content", "platform"]
}

The third-party exporter requests filesystem and network access:

{
  "skill_id": "pptx_export",
  "publisher": "third-party:acme-skills",
  "publisher_trust_tier": "third-party-unverified",
  "signature_verified": true,
  "requested_scopes": ["filesystem:write", "network:egress", "context:read"],
  "declared_capabilities": ["document-generation"],
  "allowed_teams": ["content"]
}

The unsigned internal skill only requests context:read. This lets us test signature enforcement without also triggering a scope rule:

{
  "skill_id": "unsigned_status",
  "publisher": "internal:ops-lab",
  "publisher_trust_tier": "internal-unverified",
  "signature_verified": false,
  "requested_scopes": ["context:read"],
  "declared_capabilities": ["status-reporting"],
  "allowed_teams": ["platform"]
}

A catalog pipeline can potentially provide these claims. Fernbank verifies the signature before it sets signature_verified. OPA uses the result of that check. It does not verify the signature itself.

Map the JSON with records. SkillManifest.java lists every catalog field used by the policy:

package com.themainthread.fernbank;

import java.util.List;

import com.fasterxml.jackson.annotation.JsonProperty;

public record SkillManifest(
        @JsonProperty("skill_id") String skillId,
        String publisher,
        @JsonProperty("publisher_trust_tier") String publisherTrustTier,
        @JsonProperty("signature_verified") boolean signatureVerified,
        @JsonProperty("requested_scopes") List<String> requestedScopes,
        @JsonProperty("declared_capabilities") List<String> declaredCapabilities,
        @JsonProperty("allowed_teams") List<String> allowedTeams) {
}

Keep caller context separate from the manifest:

package com.themainthread.fernbank;

import com.fasterxml.jackson.annotation.JsonProperty;

public record SubjectContext(
        @JsonProperty("user_id") String userId,
        @JsonProperty("agent_id") String agentId,
        @JsonProperty("session_id") String sessionId,
        String team) {
}

package com.themainthread.fernbank;

import com.fasterxml.jackson.annotation.JsonProperty;

public record AdmissionInput(
        SubjectContext subject,
        SkillManifest skill,
        @JsonProperty("runtime_environment") String runtimeEnvironment,
        String action) {
}

In this simple example the SkillCatalog loads the three known manifests once at startup. In a real system, this data would come from a database or a signed artifact store. The filter still needs the same lookup method:

package com.themainthread.fernbank;

import java.io.IOException;
import java.io.InputStream;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.function.Function;
import java.util.stream.Collectors;

import com.fasterxml.jackson.databind.ObjectMapper;

import jakarta.annotation.PostConstruct;
import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class SkillCatalog {

    private static final List<String> MANIFEST_PATHS = List.of(
            "/skills/docs_generate.json",
            "/skills/pptx_export.json",
            "/skills/unsigned_status.json");

    private final ObjectMapper objectMapper;
    private Map<String, SkillManifest> manifests;

    SkillCatalog(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
    }

    @PostConstruct
    void load() {
        manifests = MANIFEST_PATHS.stream()
                .map(this::readManifest)
                .collect(Collectors.toUnmodifiableMap(SkillManifest::skillId, Function.identity()));
    }

    public Optional<SkillManifest> find(String skillId) {
        return Optional.ofNullable(manifests.get(skillId));
    }

    private SkillManifest readManifest(String path) {
        try (InputStream stream = SkillCatalog.class.getResourceAsStream(path)) {
            if (stream == null) {
                throw new IllegalStateException("Skill manifest not found: " + path);
            }
            return objectMapper.readValue(stream, SkillManifest.class);
        } catch (IOException e) {
            throw new IllegalStateException("Cannot read skill manifest: " + path, e);
        }
    }
}

A skill can pass catalog admission and still be unavailable to a specific team. Caller privileges also cannot make an invalid signature valid.

Return Reasons From Rego

Create src/main/resources/policies/skill-admission.rego:

package fernbank.admission

import rego.v1

policy_version := "2026-07-12"

default allow := false

allowed_scopes := {
    "internal-verified": {
        "context:read",
        "database:read",
        "filesystem:read",
        "filesystem:write",
        "network:egress",
    },
    "internal-unverified": {
        "context:read",
        "filesystem:read",
    },
    "third-party-verified": {
        "context:read",
        "filesystem:read",
    },
    "third-party-unverified": {
        "context:read",
    },
}

known_trust_tier if {
    allowed_scopes[input.skill.publisher_trust_tier]
}

scope_allowed(scope) if {
    scope in allowed_scopes[input.skill.publisher_trust_tier]
}

team_allowed if {
    input.subject.team in input.skill.allowed_teams
}

deny contains {
    "code": "TEAM_NOT_AUTHORIZED",
    "message": sprintf("team %q is not allowed to use this skill", [input.subject.team]),
} if {
    not team_allowed
}

deny contains {
    "code": "UNKNOWN_TRUST_TIER",
    "message": sprintf("publisher trust tier %q is not configured", [input.skill.publisher_trust_tier]),
} if {
    not known_trust_tier
}

deny contains {
    "code": "PROD_SIGNATURE_REQUIRED",
    "message": "production requires a verified skill signature",
} if {
    input.runtime_environment == "prod"
    not input.skill.signature_verified
}

deny contains {
    "code": "SCOPE_NOT_ALLOWED",
    "message": sprintf("scope %q is not allowed for trust tier %q", [scope, input.skill.publisher_trust_tier]),
    "scope": scope,
} if {
    input.runtime_environment == "prod"
    some scope in input.skill.requested_scopes
    not scope_allowed(scope)
}

warn contains {
    "code": "SIGNATURE_SOFT_FLAG",
    "message": "non-production environment accepted an unverified signature",
} if {
    input.runtime_environment != "prod"
    not input.skill.signature_verified
}

warn contains {
    "code": "SCOPE_SOFT_FLAG",
    "message": sprintf("non-production environment accepted scope %q outside the tier allowlist", [scope]),
    "scope": scope,
} if {
    input.runtime_environment != "prod"
    some scope in input.skill.requested_scopes
    not scope_allowed(scope)
}

allow if count(deny) == 0

outcome := "allow" if allow else := "deny"

enforcement_mode := "enforce" if input.runtime_environment == "prod" else := "warn"

decision := {
    "allow": allow,
    "enforcement_mode": enforcement_mode,
    "outcome": outcome,
    "policy_version": policy_version,
    "reasons": [reason | some reason in deny],
    "warnings": [warning | some warning in warn],
}

OPA gives no special meaning to rules named allow and deny. The policy defines how they work together. The OPA policy guide explains complete definitions and defaults. default allow := false ensures that the decision stays defined when no allow rule matches.

Production enforces team membership, known trust tiers, verified signatures, and scope allowlists. Development changes signature and scope findings to warnings. Team isolation remains enforced because development environments can also be shared.

The decision returns reason objects. A full Rego execution trace contains more detail than operators need for a normal authorization event. SCOPE_NOT_ALLOWED is a stable value for logs, tests, dashboards, and incident reviews.

OPA server mode can emit native decision logs with decision IDs. Embedded Wasm has no OPA server, so Fernbank creates an evaluation ID and logs the policy version and reason codes.

Test the policy source before compiling it. Create src/test/resources/policies/skill-admission_test.rego:

package fernbank.admission

import rego.v1

internal_manifest := {
    "skill_id": "docs_generate",
    "publisher": "internal:docs-platform",
    "publisher_trust_tier": "internal-verified",
    "signature_verified": true,
    "requested_scopes": ["context:read", "filesystem:write"],
    "declared_capabilities": ["document-generation"],
    "allowed_teams": ["content", "platform"],
}

third_party_manifest := {
    "skill_id": "pptx_export",
    "publisher": "third-party:acme-skills",
    "publisher_trust_tier": "third-party-unverified",
    "signature_verified": true,
    "requested_scopes": ["filesystem:write", "network:egress", "context:read"],
    "declared_capabilities": ["document-generation"],
    "allowed_teams": ["content"],
}

unsigned_manifest := {
    "skill_id": "unsigned_status",
    "publisher": "internal:ops-lab",
    "publisher_trust_tier": "internal-unverified",
    "signature_verified": false,
    "requested_scopes": ["context:read"],
    "declared_capabilities": ["status-reporting"],
    "allowed_teams": ["platform"],
}

content_subject := {
    "agent_id": "fern-assistant",
    "session_id": "session-42",
    "team": "content",
    "user_id": "alice",
}

platform_subject := {
    "agent_id": "fern-assistant",
    "session_id": "session-43",
    "team": "platform",
    "user_id": "bob",
}

test_internal_verified_skill_is_allowed_in_prod if {
    result := decision with input as {
        "action": "mcp:tool:access",
        "runtime_environment": "prod",
        "skill": internal_manifest,
        "subject": content_subject,
    }
    result.allow
    count(result.reasons) == 0
}

test_third_party_write_scope_is_denied_in_prod if {
    result := decision with input as {
        "action": "mcp:tool:access",
        "runtime_environment": "prod",
        "skill": third_party_manifest,
        "subject": content_subject,
    }
    not result.allow
    "SCOPE_NOT_ALLOWED" in {reason.code | some reason in result.reasons}
}

test_missing_signature_is_denied_in_prod if {
    result := decision with input as {
        "action": "mcp:tool:access",
        "runtime_environment": "prod",
        "skill": unsigned_manifest,
        "subject": platform_subject,
    }
    not result.allow
    "PROD_SIGNATURE_REQUIRED" in {reason.code | some reason in result.reasons}
}

test_third_party_write_scope_is_soft_flagged_in_dev if {
    result := decision with input as {
        "action": "mcp:tool:access",
        "runtime_environment": "dev",
        "skill": third_party_manifest,
        "subject": content_subject,
    }
    result.allow
    "SCOPE_SOFT_FLAG" in {warning.code | some warning in result.warnings}
}

test_team_boundary_is_enforced_in_dev if {
    result := decision with input as {
        "action": "mcp:tool:access",
        "runtime_environment": "dev",
        "skill": third_party_manifest,
        "subject": platform_subject,
    }
    not result.allow
    "TEAM_NOT_AUTHORIZED" in {reason.code | some reason in result.reasons}
}

The fifth case checks that team isolation still applies in development. Scope findings become warnings, while the team rule stays enforced.

Compile Rego to Wasm With Podman

Add scripts/build-policy.sh:

#!/usr/bin/env bash

set -euo pipefail

PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
POLICY_DIR="${PROJECT_DIR}/src/main/resources/policies"
TEST_DIR="${PROJECT_DIR}/src/test/resources/policies"
BUNDLE="${PROJECT_DIR}/target/skill-admission-bundle.tar.gz"
EXTRACT_DIR="${PROJECT_DIR}/target/opa-compiled"

mkdir -p "${EXTRACT_DIR}"

podman run --rm \
  -v "${POLICY_DIR}:/policy:ro" \
  -v "${TEST_DIR}:/tests:ro" \
  openpolicyagent/opa:1.17.0 \
  test /policy /tests -v

podman run --rm \
  -v "${PROJECT_DIR}:/workspace" \
  -w /workspace \
  openpolicyagent/opa:1.17.0 \
  build -t wasm \
  -e fernbank/admission/decision \
  -o target/skill-admission-bundle.tar.gz \
  src/main/resources/policies/skill-admission.rego

tar -xzf "${BUNDLE}" -C "${EXTRACT_DIR}"
install -m 0644 "${EXTRACT_DIR}/policy.wasm" "${POLICY_DIR}/skill-admission.wasm"

echo "Wrote ${POLICY_DIR}/skill-admission.wasm"

Make it executable and run it:

chmod +x scripts/build-policy.sh
./scripts/build-policy.sh

The five Rego tests finish with:

PASS: 5/5
Wrote .../src/main/resources/policies/skill-admission.wasm

The compiled module becomes part of the application artifact. Updating the policy therefore requires a new build and deployment.

If the security team needs to deploy policy independently, use an OPA service or bundle distribution. The remote decision must happen outside this event-loop filter. That design needs a cache or an asynchronous MCP interception API because a blocking network call inside ToolFilter would block the event loop.

Evaluate OPA In Process

Map the decision with two records:

package com.themainthread.fernbank;

public record PolicyReason(String code, String message, String scope) {
}

package com.themainthread.fernbank;

import java.util.List;

import com.fasterxml.jackson.annotation.JsonProperty;

public record PolicyDecision(
        boolean allow,
        String outcome,
        @JsonProperty("policy_version") String policyVersion,
        @JsonProperty("enforcement_mode") String enforcementMode,
        List<PolicyReason> reasons,
        List<PolicyReason> warnings) {

    public static PolicyDecision evaluationFailure(String message) {
        return new PolicyDecision(
                false,
                "deny",
                "unavailable",
                "fail-closed",
                List.of(new PolicyReason("POLICY_EVALUATION_FAILED", message, null)),
                List.of());
    }
}

OpaPolicyEvaluator loads the module once and evaluates the JSON input synchronously:

package com.themainthread.fernbank;

import java.io.IOException;
import java.io.InputStream;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.styra.opa.wasm.OpaPolicy;

import jakarta.annotation.PostConstruct;
import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class OpaPolicyEvaluator {

    private static final String POLICY_PATH = "/policies/skill-admission.wasm";

    private final ObjectMapper objectMapper;
    private OpaPolicy policy;

    OpaPolicyEvaluator(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
    }

    @PostConstruct
    void loadPolicy() {
        InputStream stream = OpaPolicyEvaluator.class.getResourceAsStream(POLICY_PATH);
        if (stream == null) {
            throw new IllegalStateException("OPA policy not found: " + POLICY_PATH);
        }
        policy = OpaPolicy.builder().withPolicy(stream).build();
    }

    public PolicyDecision evaluate(AdmissionInput input) {
        try {
            String resultJson = policy.evaluate(objectMapper.writeValueAsString(input));
            JsonNode result = objectMapper.readTree(resultJson).path(0).path("result");
            if (result.isMissingNode() || result.isNull()) {
                throw new IllegalStateException("OPA returned no decision");
            }
            return objectMapper.treeToValue(result, PolicyDecision.class);
        } catch (IOException | RuntimeException e) {
            throw new IllegalStateException("OPA policy evaluation failed", e);
        }
    }
}

If the Wasm file is missing, the application fails during startup. If OPA returns a malformed result, the evaluator throws an exception. The filter catches that exception and denies access. This is required because the extension ignores filter exceptions.

Gate MCP Tools With OPA

Fernbank exposes three demo tools. Each tool name matches its catalog manifest ID:

package com.themainthread.fernbank;

import io.quarkiverse.mcp.server.Tool;
import io.quarkiverse.mcp.server.ToolArg;

import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class FernbankTools {

    @Tool(
            name = "docs_generate",
            description = "Generate internal documentation from approved project context.",
            annotations = @Tool.Annotations(
                    title = "Documentation Generator",
                    readOnlyHint = false,
                    destructiveHint = false,
                    idempotentHint = true,
                    openWorldHint = false))
    String generateDocs(@ToolArg(description = "Documentation topic") String topic) {
        return "Generated documentation for: " + topic;
    }

    @Tool(
            name = "pptx_export",
            description = "Export a presentation through the Acme third-party renderer.",
            annotations = @Tool.Annotations(
                    title = "PPTX Exporter",
                    readOnlyHint = false,
                    destructiveHint = false,
                    idempotentHint = true,
                    openWorldHint = true))
    String exportPresentation(@ToolArg(description = "Presentation title") String title) {
        return "Exported presentation: " + title;
    }

    @Tool(
            name = "unsigned_status",
            description = "Read deployment status through an unsigned internal lab skill.",
            annotations = @Tool.Annotations(
                    title = "Unsigned Status Reader",
                    readOnlyHint = true,
                    destructiveHint = false,
                    idempotentHint = true,
                    openWorldHint = false))
    String readStatus(@ToolArg(description = "Service name") String service) {
        return service + " is healthy";
    }
}

OpaToolFilter combines the MCP request, caller data, and skill manifest. The demo reads identity fields from HTTP headers to keep setup small. Caller-supplied headers do not prove identity. Production code should read these values from SecurityIdentity after OpenID Connect (OIDC) verification.

package com.themainthread.fernbank;

import io.quarkiverse.mcp.server.FilterContext;
import io.quarkiverse.mcp.server.ToolFilter;
import io.quarkiverse.mcp.server.ToolManager.ToolInfo;
import io.vertx.core.http.HttpServerRequest;

import jakarta.inject.Singleton;

import org.jboss.logging.Logger;

@Singleton
public class OpaToolFilter implements ToolFilter {

    private static final Logger LOG = Logger.getLogger(OpaToolFilter.class);

    private final HttpServerRequest request;
    private final SkillCatalog catalog;
    private final OpaPolicyEvaluator policyEvaluator;
    private final DecisionAudit audit;
    private final FernbankConfig config;

    OpaToolFilter(
            HttpServerRequest request,
            SkillCatalog catalog,
            OpaPolicyEvaluator policyEvaluator,
            DecisionAudit audit,
            FernbankConfig config) {
        this.request = request;
        this.catalog = catalog;
        this.policyEvaluator = policyEvaluator;
        this.audit = audit;
        this.config = config;
    }

    @Override
    public boolean test(ToolInfo tool, FilterContext context) {
        SkillManifest manifest = catalog.find(tool.name()).orElse(null);
        if (manifest == null) {
            LOG.errorf("No skill manifest found for MCP tool %s; denying access", tool.name());
            return false;
        }

        SubjectContext subject = new SubjectContext(
                header("X-Fernbank-User", "anonymous"),
                header("X-Fernbank-Agent", context.connection().initialRequest().implementation().name()),
                header("X-Fernbank-Session", "unknown"),
                header("X-Fernbank-Team", "none"));
        AdmissionInput input = new AdmissionInput(
                subject,
                manifest,
                config.runtimeEnvironment(),
                "mcp:tool:access");

        PolicyDecision decision;
        try {
            decision = policyEvaluator.evaluate(input);
        } catch (RuntimeException e) {
            LOG.errorf(e, "OPA evaluation failed for tool %s; denying access", tool.name());
            decision = PolicyDecision.evaluationFailure(e.getMessage());
        }
        audit.record(input, decision);
        return decision.allow();
    }

    private String header(String name, String defaultValue) {
        String value = request.getHeader(name);
        return value == null || value.isBlank() ? defaultValue : value;
    }
}

Version 1.13.1 uses initialRequest().implementation().name(). An older documentation example uses clientInfo().name(), which does not compile with this version. Follow the API in the 1.13.1 artifact here.

What happens to pptx_export in production? Its signature is valid, and the team matches. The tool still disappears because filesystem:write and network:egress are outside the third-party-unverified allowlist. OPA returns one SCOPE_NOT_ALLOWED reason for each scope.

Configure Enforcement by Environment

Create src/main/resources/application.properties:

fernbank.runtime-environment=prod
%dev.fernbank.runtime-environment=dev
%test.fernbank.runtime-environment=prod

quarkus.http.cors.enabled=true
quarkus.http.cors.origins=http://localhost:6274

%dev.quarkus.mcp.server.traffic-logging.enabled=true
%dev.quarkus.mcp.server.traffic-logging.text-limit=1000

Map the Fernbank property with typed configuration:

package com.themainthread.fernbank;

import io.smallrye.config.ConfigMapping;

@ConfigMapping(prefix = "fernbank")
public interface FernbankConfig {

    String runtimeEnvironment();
}

The server configuration sets the environment. The request has no header that can change production enforcement to warning mode. Tests use production mode and verify the deny path.

The CORS origin matches a local MCP Inspector setup. Replace it with the trusted origin for your client. Traffic logs can contain tool arguments, so this example enables them only in the development profile.

Record Every Decision

Fernbank keeps the last 100 decisions in memory and logs every record as JSON. A denied exporter produces this record:

{
  "evaluationId": "c1f10476-26e8-4b72-8df0-c7208457d6a1",
  "evaluatedAt": "2026-07-12T03:48:52.158438Z",
  "userId": "alice",
  "agentId": "fern-assistant",
  "sessionId": "session-42",
  "team": "content",
  "skillId": "pptx_export",
  "publisher": "third-party:acme-skills",
  "publisherTrustTier": "third-party-unverified",
  "signatureVerified": true,
  "requestedScopes": ["filesystem:write", "network:egress", "context:read"],
  "declaredCapabilities": ["document-generation"],
  "runtimeEnvironment": "prod",
  "allow": false,
  "outcome": "deny",
  "enforcementMode": "enforce",
  "policyVersion": "2026-07-12",
  "reasons": [
    {
      "code": "SCOPE_NOT_ALLOWED",
      "message": "scope \"filesystem:write\" is not allowed for trust tier \"third-party-unverified\"",
      "scope": "filesystem:write"
    },
    {
      "code": "SCOPE_NOT_ALLOWED",
      "message": "scope \"network:egress\" is not allowed for trust tier \"third-party-unverified\"",
      "scope": "network:egress"
    }
  ],
  "warnings": [],
  "reasonCodes": ["SCOPE_NOT_ALLOWED", "SCOPE_NOT_ALLOWED"],
  "warningCodes": []
}

The audit record stores caller data, skill data, policy identity, and the result:

package com.themainthread.fernbank;

import java.time.Instant;
import java.util.List;

public record AdmissionAuditRecord(
        String evaluationId,
        Instant evaluatedAt,
        String userId,
        String agentId,
        String sessionId,
        String team,
        String skillId,
        String publisher,
        String publisherTrustTier,
        boolean signatureVerified,
        List<String> requestedScopes,
        List<String> declaredCapabilities,
        String runtimeEnvironment,
        boolean allow,
        String outcome,
        String enforcementMode,
        String policyVersion,
        List<PolicyReason> reasons,
        List<PolicyReason> warnings,
        List<String> reasonCodes,
        List<String> warningCodes) {
}

DecisionAudit limits the local buffer to 100 entries. It also sorts reason codes because Rego produces sets and does not guarantee their order:

package com.themainthread.fernbank;

import java.time.Instant;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.UUID;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

import jakarta.enterprise.context.ApplicationScoped;

import org.jboss.logging.Logger;

@ApplicationScoped
public class DecisionAudit {

    private static final Logger LOG = Logger.getLogger(DecisionAudit.class);
    private static final int CAPACITY = 100;

    private final ObjectMapper objectMapper;
    private final Deque<AdmissionAuditRecord> records = new ArrayDeque<>(CAPACITY);

    DecisionAudit(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
    }

    public synchronized AdmissionAuditRecord record(AdmissionInput input, PolicyDecision decision) {
        AdmissionAuditRecord record = new AdmissionAuditRecord(
                UUID.randomUUID().toString(),
                Instant.now(),
                input.subject().userId(),
                input.subject().agentId(),
                input.subject().sessionId(),
                input.subject().team(),
                input.skill().skillId(),
                input.skill().publisher(),
                input.skill().publisherTrustTier(),
                input.skill().signatureVerified(),
                List.copyOf(input.skill().requestedScopes()),
                List.copyOf(input.skill().declaredCapabilities()),
                input.runtimeEnvironment(),
                decision.allow(),
                decision.outcome(),
                decision.enforcementMode(),
                decision.policyVersion(),
                List.copyOf(decision.reasons()),
                List.copyOf(decision.warnings()),
                codes(decision.reasons()),
                codes(decision.warnings()));

        if (records.size() == CAPACITY) {
            records.removeFirst();
        }
        records.addLast(record);
        log(record);
        return record;
    }

    public synchronized List<AdmissionAuditRecord> recent(int limit) {
        int safeLimit = Math.max(1, Math.min(limit, CAPACITY));
        List<AdmissionAuditRecord> snapshot = new ArrayList<>(records);
        int fromIndex = Math.max(0, snapshot.size() - safeLimit);
        return List.copyOf(snapshot.subList(fromIndex, snapshot.size()));
    }

    private List<String> codes(List<PolicyReason> reasons) {
        return reasons.stream().map(PolicyReason::code).sorted().toList();
    }

    private void log(AdmissionAuditRecord record) {
        try {
            LOG.infof("opa_decision=%s", objectMapper.writeValueAsString(record));
        } catch (JsonProcessingException e) {
            LOG.warnf(e, "Could not serialize OPA decision %s", record.evaluationId());
        }
    }
}

Expose the buffer through DecisionResource.java:

package com.themainthread.fernbank;

import java.util.List;

import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.MediaType;

@Path("/api/decisions")
@Produces(MediaType.APPLICATION_JSON)
public class DecisionResource {

    private final DecisionAudit audit;

    DecisionResource(DecisionAudit audit) {
        this.audit = audit;
    }

    @GET
    public List<AdmissionAuditRecord> recent(@QueryParam("limit") @DefaultValue("20") int limit) {
        return audit.recent(limit);
    }
}

The in-memory buffer keeps the example self-contained. Production should send these records to an append-only audit store with retention, access control, and a policy-bundle digest. The application-generated UUID connects related logs. It is different from OPA’s native decision_id because the Wasm path has no OPA server.

Test Tool Visibility and Direct Calls

The McpAssured test client speaks Streamable HTTP and supports custom request headers. The test lists the visible tools, calls a hidden tool by name, and checks the audit endpoint:

package com.themainthread.fernbank;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.hasItem;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.Map;
import java.util.UUID;

import org.junit.jupiter.api.Test;

import io.quarkiverse.mcp.server.test.McpAssured;
import io.quarkiverse.mcp.server.test.McpAssured.McpStreamableTestClient;
import io.quarkus.test.junit.QuarkusTest;
import io.vertx.core.MultiMap;

@QuarkusTest
class ToolExposureTest {

    @Test
    void productionClientOnlySeesAdmittedTools() {
        McpStreamableTestClient client = clientFor("alice", "content");
        try {
            client.when()
                    .toolsList(page -> {
                        assertEquals(1, page.size());
                        assertNotNull(page.findByName("docs_generate"));
                        assertFalse(page.tools().stream().anyMatch(tool -> tool.name().equals("pptx_export")));
                        assertFalse(page.tools().stream().anyMatch(tool -> tool.name().equals("unsigned_status")));
                    })
                    .thenAssertResults();

            client.when()
                    .toolsCall("pptx_export")
                    .withArguments(Map.of("title", "Quarterly review"))
                    .withErrorAssert(error -> assertTrue(error.message().contains("pptx_export")))
                    .send()
                    .thenAssertResults();

            given()
                    .queryParam("limit", 20)
                    .when().get("/api/decisions")
                    .then()
                    .statusCode(200)
                    .body("skillId", hasItem("pptx_export"))
                    .body("reasonCodes.flatten()", hasItem("SCOPE_NOT_ALLOWED"));
        } finally {
            client.disconnect();
        }
    }

    private McpStreamableTestClient clientFor(String user, String team) {
        return McpAssured.newStreamableClient()
                .setAdditionalHeaders(message -> {
                    MultiMap headers = MultiMap.caseInsensitiveMultiMap();
                    headers.add("X-Fernbank-User", user);
                    headers.add("X-Fernbank-Agent", "fern-assistant");
                    headers.add("X-Fernbank-Session", UUID.randomUUID().toString());
                    headers.add("X-Fernbank-Team", team);
                    return headers;
                })
                .build()
                .connect();
    }
}

Run the suite:

./mvnw test

Expected result:

Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The first assertion checks client-specific exposure. The second checks that a direct call to a hidden tool also fails. The final assertion checks that the policy decision reached the audit log.

Four other tests check the policy result directly:

• Internal, verified, narrow scopes in production: allow

• Third-party, unverified tier with filesystem and network scopes in production: deny with SCOPE_NOT_ALLOWED

• Missing signature in production: deny with PROD_SIGNATURE_REQUIRED

• The third-party exporter in development: allow with SCOPE_SOFT_FLAG

Run Fernbank

Start Quarkus:

./mvnw quarkus:dev

The Streamable HTTP endpoint is http://localhost:8080/mcp. In dev mode, scope violations produce warnings. A content-team client therefore sees docs_generate and pptx_export. It does not see unsigned_status because that tool only allows the platform team.

To inspect the production deny behavior in dev mode, override the Fernbank environment:

./mvnw quarkus:dev -Dfernbank.runtime-environment=prod

After an MCP client lists tools, inspect the recent decisions:

curl -s http://localhost:8080/api/decisions | jq

Rego reason collections are sets, so their order may change. Consumers should use the stable code values and ignore array order.

Before Production

Three parts need to change before production.

Verify identity. Replace the X-Fernbank-* headers with an OIDC bearer token and values from Quarkus SecurityIdentity. A reverse proxy may add trusted headers after authentication. The application must still prevent callers from setting those headers directly.

Store audit records outside the process. Send each record to an append-only system. Include a digest or revision for the policy bundle, manifest, and deployed tool artifact. The date in policy_version is easy to read, while content hashes identify the exact files that ran. Remove /api/decisions or protect it with an operator role because the response contains user, session, publisher, and scope data.

Choose how policy is deployed. Embedded Wasm keeps evaluation local and gives us a simple failure mode. It also deploys policy with the application. A central OPA service supports independent policy rollout and native decision logs. The current synchronous ToolFilter cannot wait for that service without blocking the event loop. Use precomputed permissions, a local bundle agent, or an asynchronous interception point.

Tool annotations such as readOnlyHint help MCP clients understand intent. They do not enforce authorization. Fernbank evaluates the signed catalog manifest and caller context, then filters the MCP protocol. The tool implementation still needs normal authorization at the business boundary because another endpoint or an internal Java call may reach the same operation.

Conclusion

Fernbank evaluates skill metadata and caller identity before Quarkus exposes or invokes an MCP tool. The model has no role in that decision. A denial includes the policy version, caller context, and reason code that an operator can inspect later.

Subscribe now

The code changed last Tuesday. Half the documents did not.

I understand how teams get there. Agents produce code quickly, so we try to move more thought in front of implementation. We ask for clearer requirements, more design, and stronger acceptance criteria. Then we turn that into a workflow. Before long, every change has fifteen stages and enough generated prose to use most of the context window before the agent reads the relevant class.

That is not a specification system. It is a second codebase with weaker tooling.

We still need specifications. The mistake is treating a specification as a permanent natural-language copy of the software. A useful spec describes the next change, exposes the decisions that matter, and gives us something to verify. After the change ships, most of it should disappear.

What remains should move into the artifacts software teams already know how to maintain: code, schemas, tests, policies, configuration, telemetry, and a small number of decision records.

Code becomes the fact

Code is actual behavior. Once that behavior reaches production, users and connected systems start to depend on it. A mistake can become an observed contract because it has behaved the same way for three years. When the system fails, the running behavior matters more than the design document that described something else.

For existing behavior, start with the code and the production system. Reading a natural-language summary instead is rarely an improvement.

The code still cannot decide what the next version should do. It does not know which behavior must stay stable, which policy changed, which workaround can disappear, or which user outcome matters most. A repository can show that an account-closing function deletes a record. It may not show that regulation requires seven years of retention or that another company consumes an undocumented export every night.

That context has to come from somewhere else. But it does not follow that we need a permanent prose description of the whole system.

We need enough context to decide the delta: the difference between what exists and what should exist next.

Specs are for the delta

A change specification earns its cost when it helps a team decide and review that delta. It should name the outcome, non-goals, constraints that differ from current behavior, and evidence required for acceptance. A technical design belongs there when the change crosses an architectural boundary or contains a decision that will be expensive to reverse.

That is enough for many changes:

• the intended outcome and non-goals;

• known unknowns and decisions that need human judgment;

• affected system boundaries and authoritative interface artifacts;

• functional and non-functional constraints that differ from today;

• acceptance evidence proportionate to the risk.

I prefer calling this a change brief. “Specification” now carries too much baggage. It suggests a complete description, and completeness is exactly where these methods become expensive.

Teams have tried exhaustive specification before. They produced requirement documents, high-level designs, low-level designs, and architecture records for every decision. The collection grew until finding the right document cost more than reading the code. Humans stopped looking. The documentation stayed complete in the administrative sense and became useless in the engineering sense.

Agents can recreate the same failure much faster. A long workflow produces research, requirements, design, planning, and review artifacts on demand. Every stage makes the process look more controlled. It also creates more material to read, reconcile, retrieve, and keep current.

Research notes, prototypes, and design records should be added when uncertainty justifies them. They resolve a specific problem. They should not become required stages for every pull request.

The map will always be incomplete

A prompt, ticket, or spec is a map of the work. The territory includes the codebase, users, runtime environment, historical constraints, connected systems, and decisions that nobody wrote down.

Real work keeps producing new information. The agent reads a module and finds an unexpected dependency. A prototype exposes a usability problem. A test reveals an undocumented edge case. Production data contradicts the design assumption.

A recent field guide on finding unknowns in agent work makes this point well. Some unknowns are visible at the start. Others appear only when we inspect references, build prototypes, or compare a result with judgment we could not express beforehand.

So discovery has to stay open:

Before implementation, inspect the current system, identify decisions that could change the architecture or user experience, and build a cheap prototype when preferences are hard to state.

During implementation, record meaningful deviations and stop when a new unknown changes the risk or direction of the work.

After implementation, read the code, run the checks, and compare the result with the original intent.

The spec participates in that loop. It cannot complete the loop before coding begins.

Keep durable facts in their native form

The phrase “promote durable constraints” needs care. It can sound like an instruction to write permanent Markdown. That would recreate the problem.

Software engineering already has better homes for most durable facts:

• API shape and compatibility belong in OpenAPI, AsyncAPI, protocol schemas, types, and compatibility tests.

• Data invariants belong in types, database constraints, validation, and migration checks.

• Security rules belong in access policy, static analysis, dependency policy, and runtime enforcement.

• Architecture boundaries belong in module structure, dependency rules, and focused architecture tests.

• Reliability requirements belong in load tests, service objectives, telemetry, and alerts.

• Release rules belong in continuous integration and deployment policy.

These artifacts participate in delivery. A failed schema check or alert demands attention. A paragraph in an old design directory usually does not.

Natural language still has a place. Business policy, trade-offs, and architectural rationale do not always fit into an executable artifact. Keep that prose short, owned, and close to the thing it explains. An architecture decision record is worth keeping when a future team might otherwise repeat an expensive investigation. Recording every local choice just hides the few decisions that matter.

Ask which fact must survive and what its authoritative form should be.

Judgment belongs in the workflow

Heavy specification methods try to control quality by prescribing the path. Every change produces the same documents, reviews, and test categories.

That consistency can spend attention on low-risk work while hiding the judgment needed for the hard parts. A copy change and a payment-flow change do not need the same process or testing strategy.

Simon Willison describes a simpler instruction: let the coding agent apply judgment about how to perform the work instead of spelling out every procedural branch. I think the advice is right.

Judgment does not mean unconstrained autonomy. The team still defines the outcome, safety boundaries, ownership, and acceptance authority. The agent can choose tactics inside those boundaries. It should surface uncertainty and ask for a decision when the consequences exceed its authority.

The workflow can then follow the risk:

• a small, familiar change can move from a short brief to implementation and review;

• unfamiliar code calls for factual research before design;

• unclear user experience calls for prototypes and comparison;

• an architectural change needs explicit human alignment;

• high-consequence behavior needs stronger independent evidence and approval.

The process expands when the work demands it. Starting every change at maximum weight wastes time and context.

Context is an engineering budget

Large specifications cost more than authoring and maintenance time. They compete with the code and evidence the agent needs for the current decision.

Every requirement, design note, repository instruction, and tool definition consumes part of a limited working context. Important rules become harder to apply when stale and duplicated material surrounds them. A spec that leaves too little room for the repository defeats its own purpose.

Progressive disclosure is a better fit. Give the agent a small map, stable rules that apply broadly, and pointers to deeper material. Retrieve the relevant schema, decision, or module guide when the work crosses that boundary.

A concise AGENTS.md can document build commands, repository layout, and architectural boundaries. It should not narrate every class or repeat API documentation. The file helps humans for the same reason: it tells them where to look without pretending to replace what they will find.

Experience with Research-Plan-Implement shows the cost of getting this wrong. The original workflow moved human review before implementation, but teams ended up with large prompts and plans that could reach 1,000 lines. Engineers reviewed the plan while treating generated code almost like compiler output. That approach worked better on simple greenfield tasks than on complex changes to existing systems.

In Everything We Got Wrong About Research-Plan-Implement, Dexter Horthy retracts that position. Teams shipped more code and then spent much of the gain cleaning up earlier output. The implementation could also diverge from the reviewed plan, which forced engineers to reconstruct what happened from the code anyway.

The revised workflow uses smaller contexts for factual research, design alignment, structure, implementation, and review. The correction is simple: research and design create leverage, but engineers still read and own the code.

Modernization makes this obvious

A mature application contains several kinds of behavior in the same codebase. Some logic represents durable business policy. Some implements a published interface. Some exists because an old platform imposed a technical limit. Some came from an incident fix whose context is gone. Some is simply a defect that survived long enough to look intentional.

An agent that treats all code as the target specification can translate those layers faithfully into a new language or architecture. You get a cleaner version of the same debt. Efficient, but not helpful.

Modernization starts by classifying observed behavior:

• Preserve durable business invariants and externally required behavior.

• Verify behavior that appears active but lacks clear ownership or evidence.

• Redesign logic tied to obsolete architectural constraints.

• Remove dead paths, duplicated logic, and confirmed defects.

Code analysis reveals dependencies and current behavior. Runtime data shows which paths are still used. Tests establish a baseline. Business, architecture, risk, and operational context decide which behavior belongs in the target system.

The code is the starting evidence because it is the behavior we have. The change brief is temporary because its job is to decide the delta. The new implementation and its native checks become the next durable state.

Small specs still need real evidence

Keeping specifications small does not mean returning to a loose prompt followed by hopeful review.

An agent can turn an underspecified request into a coherent implementation before the missing decisions become obvious. The result may compile, pass the available tests, and look internally consistent. Coherence can hide a business decision that nobody approved.

Research on repairing ambiguous natural-language requirements found that targeted repairs improved code-generation results, while direct ambiguity resolution often produced irrelevant or inconsistent clarification. SWT-Bench found that generated tests could filter proposed fixes and double the precision of a software-repair agent.

Formal specification does not remove review either. A 2026 evaluation asked 30 models to translate natural-language requirements into TLA+ specifications. The best result reached 26.6% syntactic correctness and 8.6% semantic correctness. Preserving meaning between representations remained the hard part.

These results support focused clarification and independent checks. They do not support generating the longest spec or letting one reasoning path define the requirement, implementation, and evidence without challenge.

The amount of independence should follow the risk. A small internal refactor may need ordinary tests and code review. A change to authorization, money movement, or regulated data needs stronger separation, adversarial review, and explicit approval.

A lighter operating model

The method does not need fifteen stages:

1. Start from the code and operational evidence that describe the current system.

2. Define the intended delta, important boundaries, and known unknowns.

3. Add research, prototypes, design alignment, or stronger verification where risk requires them.

4. Read and review the implementation, not only the plan.

5. At release, discard temporary reasoning and preserve each surviving fact in its native authoritative artifact.

That is enough structure to guide the work without building a natural-language replica of the software.

Before implementation, the change brief describes the intended delta. During implementation, it helps people and agents align while new information changes the plan. After release, code and production behavior become the primary evidence of what the system does.

Durable obligations survive in the forms we already know how to maintain: schemas, tests, policies, configuration, telemetry, and short records for rationale that cannot be encoded elsewhere. Most planning detail should expire.

The teams that benefit most from coding agents will not produce the most documentation. They will know what to specify, what to discover, what to verify, and what to let go.

Subscribe now

That short build path still leaves a few decisions with us. The base image decides which operating-system files ship, which user starts the process, whether a shell exists, and how quickly security fixes arrive. The image builder decides how application files are layered and which entrypoint becomes part of the image metadata. A clean vulnerability report means little if the container starts as root or points at a launch script that is missing from the filesystem.

Red Hat Hardened Images is the generally available catalog produced by Project Hummingbird. Its runtime images keep a small software footprint, default to a non-root user where possible, and remove tools that the workload does not need. The Hummingbird image documentation describes the default variants as distroless: no package manager and no shell.

Jib builds Docker and OCI images directly from Java build tools. It separates dependencies from application classes so a code change does not invalidate every layer. Quarkus adds its own integration through quarkus-container-image-jib, which understands the fast-jar layout and constructs the runtime entrypoint for us.

What We Build

We build a small status service on Quarkus and Java 25. It exposes /status and the SmallRye Health readiness endpoint. Jib places the Quarkus fast-jar on top of registry.access.redhat.com/hi/openjdk:25-runtime, then loads the result into Podman.

The final checks shows:

• The HTTP application and readiness endpoint respond

• The packaged application runs on the Java 25 Hummingbird runtime

• The image starts as UID 65532 with /work as its working directory

• Starting /bin/sh fails because the runtime image does not contain a shell

What You Need

The application has no external service dependency. Podman is only needed when we build and run the local image.

• JDK 25 installed

• Quarkus CLI

• Podman 5 or later

• curl and jq

• About ☕️☕️

• Optional: Cosign for signature verification

On macOS and Windows, start the Podman virtual machine before the image build:

podman machine start

Create the Project

Create the application:

quarkus create app com.themainthread:hardened-quarkus-jib \
  --extension=quarkus-rest-jackson,quarkus-smallrye-health,quarkus-container-image-jib \
  --java=25 \
  --no-code \
  --no-dockerfiles

Use these extensions:

• quarkus-rest-jackson provides the JSON REST endpoint

• quarkus-smallrye-health exposes /q/health/ready

• quarkus-container-image-jib turns the Quarkus package into an OCI image without reading a Dockerfile

The Quarkus extension catalog marks quarkus-container-image-jib as preview.

--no-dockerfiles keeps the generated project aligned with the build we use. Quarkus usually scaffolds Dockerfiles with your application. But Jib does not read a Dockerfile, so we don’t even need it.

Add a Runtime Probe

The application code stays small because the container is the subject here. Create src/main/java/com/themainthread/hardened/StatusResource.java:

package com.themainthread.hardened;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/status")
@Produces(MediaType.APPLICATION_JSON)
public class StatusResource {

    @GET
    public StatusResponse status() {
        return new StatusResponse(
                "hardened-quarkus-jib",
                "ready",
                Runtime.version().feature());
    }

    public record StatusResponse(String service, String status, int javaFeatureVersion) {
    }
}

Runtime.version().feature() gives us a visible check on the JVM. The compiler release and the container runtime are separate settings, so we verify Java 25 twice: @QuarkusTest checks the Maven JVM, and @QuarkusIntegrationTest checks the Hummingbird JVM inside the packaged image.

Configure Jib and the Hardened Runtime

Replace src/main/resources/application.properties with the following configuration:

quarkus.application.name=hardened-quarkus-jib

quarkus.container-image.group=themainthread
quarkus.container-image.name=hardened-quarkus-jib
quarkus.container-image.registry=localhost
quarkus.container-image.labels."org.opencontainers.image.source"=https://github.com/myfear/the-main-thread/hardened-quarkus-jib
quarkus.container-image.labels."org.opencontainers.image.title"=Hardened Quarkus with Jib

quarkus.jib.base-jvm-image=registry.access.redhat.com/hi/openjdk:25-runtime
quarkus.jib.docker-executable-name=podman
quarkus.jib.jvm-additional-arguments=-XX:MaxRAMPercentage=70.0,-XX:+ExitOnOutOfMemoryError
quarkus.jib.user=65532
quarkus.jib.working-directory=/work
quarkus.jib.use-current-timestamp=false
quarkus.jib.use-current-timestamp-file-modification=false

The container-image properties produce localhost/themainthread/hardened-quarkus-jib:1.0.0-SNAPSHOT and add OCI source and title labels. The application version supplies the image tag unless you override quarkus.container-image.tag.

quarkus.jib.base-jvm-image selects the Hummingbird OpenJDK 25 runtime variant. It contains the headless runtime. The builder variant adds the extra build tools. The published image is available for Linux on AMD64 and ARM64.

quarkus.jib.docker-executable-name=podman tells Jib where to load a local build. Jib still assembles the image. Podman receives the completed image and then runs it. A registry push uses the registry API directly, so it does not need a local container daemon.

The Hummingbird OpenJDK runtime declares UID 65532. We set the same value on the final image so the runtime contract stays visible in our own metadata. Kubernetes can still assign another arbitrary non-root UID, provided the application files remain readable.

The Hummingbird Java image uses /home/build, while the Quarkus Jib configuration defaults to /home/jboss for its standard UBI Java image. /work gives the fast-jar one predictable location. Our application only reads from it, so UID 65532 does not need write access to the application layers.

We leave quarkus.jib.jvm-entrypoint unset. For this custom base image, Quarkus constructs a direct Java command that ends with -jar quarkus-run.jar. This matters because the Hummingbird runtime has no shell and does not ship the UBI run-java.sh helper.

-XX:MaxRAMPercentage=70.0 caps the maximum heap near 70% of the container memory limit. That leaves headroom for metaspace, thread stacks, direct buffers, native allocations, and the operating-system runtime. -XX:+ExitOnOutOfMemoryError terminates the process after an out-of-memory error so the orchestrator can replace it. Seventy percent is a starting point for this small service. Measure heap and native memory under your own load before copying it into a large application.

Finally, the timestamp settings make unchanged application layers stable between builds. Podman reports the image creation time as January 1, 1970 with this configuration. Jib uses the Unix epoch here so time alone does not change the digest. The complete image digest can still change when the Hummingbird tag points to a patched base. That is exactly what we want during a security rebuild.

Test the Two Runtime Boundaries

The generated project already contains the Quarkus JUnit dependency. Add RestAssured to the <dependencies> section in pom.xml:

<dependency>
    <groupId>io.rest-assured</groupId>
    <artifactId>rest-assured</artifactId>
    <scope>test</scope>
</dependency>

The Quarkus platform manages the RestAssured version used by the tests below.

Add the REST and health test at src/test/java/com/themainthread/hardened/StatusResourceTest.java:

package com.themainthread.hardened;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;

@QuarkusTest
class StatusResourceTest {

    @Test
    void returnsRuntimeStatus() {
        given()
                .when().get("/status")
                .then()
                .statusCode(200)
                .body("service", equalTo("hardened-quarkus-jib"))
                .body("status", equalTo("ready"))
                .body("javaFeatureVersion", equalTo(25));
    }

    @Test
    void exposesReadiness() {
        given()
                .when().get("/q/health/ready")
                .then()
                .statusCode(200)
                .body("status", equalTo("UP"));
    }
}

This test checks application behavior on the Maven JVM. Run it now:

./mvnw test

This assertion proves Maven launched the Quarkus test application on Java 25. It will fail early if the project is compiled with one JDK and tested with another.

Now add src/test/java/com/themainthread/hardened/StatusResourceIT.java:

package com.themainthread.hardened;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusIntegrationTest;

@QuarkusIntegrationTest
class StatusResourceIT extends StatusResourceTest {

    @Test
    void runsOnHummingbirdJava25Runtime() {
        given()
                .when().get("/status")
                .then()
                .statusCode(200)
                .body("javaFeatureVersion", equalTo(25));
    }
}

@QuarkusIntegrationTest runs against the packaged artifact. When the container-image build is enabled, Quarkus starts the resulting image for the test. The Java 25 assertion here proves the base-image choice survived packaging.

The generated POM sets skipITs to true, so the image build below overrides it.

Build the Image and Run the Container Test

Build the fast-jar, assemble the image, load it into Podman, and run the integration test:

./mvnw verify \
  -DskipITs=false \
  -Dquarkus.container-image.build=true

The final test summary should contain:

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Jib also warns that the base uses a tag:

[WARNING] Base image 'registry.access.redhat.com/hi/openjdk:25-runtime' does not use a specific image digest - build may not be reproducible

That warning is accurate. The timestamp settings stabilize our application layers, while 25-runtime is allowed to move to a patched Hummingbird image. The production section below separates the update tag from the immutable digest we deploy.

Jib creates separate layers for the Quarkus libraries, quarkus-run.jar, framework-generated files, and application files. A normal code edit usually changes the application layer while the dependency layers stay cached. This is why Jib can remain fast without a handwritten multi-stage Dockerfile.

Jib targets linux/amd64 when no platform is configured. Podman can emulate it on many ARM64 machines, but it prints a platform mismatch warning and startup is slower. Build the native platform explicitly:

./mvnw verify \
  -DskipITs=false \
  -Dquarkus.container-image.build=true \
  -Dquarkus.jib.platforms=linux/arm64

The Hummingbird 25-runtime tag publishes both architectures, so either build resolves from the same image name.

Run the Service

Start the image in one terminal:

podman run --rm \
  --name hardened-quarkus \
  --memory=512m \
  -p 8080:8080 \
  localhost/themainthread/hardened-quarkus-jib:1.0.0-SNAPSHOT

Call the application from another terminal:

curl -s http://localhost:8080/status | jq

Expected response:

{
  "javaFeatureVersion": 25,
  "service": "hardened-quarkus-jib",
  "status": "ready"
}

Check readiness too:

curl -s http://localhost:8080/q/health/ready | jq

Expected response:

{
  "status": "UP",
  "checks": []
}

The empty checks array is normal for this application. SmallRye Health has no external database or broker to inspect, but the endpoint proves that the management route is present and the process can answer requests.

Inspect What Jib Built

The HTTP response proves the application works. Image inspection proves the runtime contract.

Check the user and working directory:

podman image inspect \
  localhost/themainthread/hardened-quarkus-jib:1.0.0-SNAPSHOT \
  --format 'user={{.Config.User}} workdir={{.Config.WorkingDir}}'

Expected output:

user=65532 workdir=/work

Now inspect the entrypoint:

podman image inspect \
  localhost/themainthread/hardened-quarkus-jib:1.0.0-SNAPSHOT \
  --format '{{json .Config.Entrypoint}}' | jq

Expected output:

[
  "java",
  "-Djava.util.logging.manager=org.jboss.logmanager.LogManager",
  "-XX:MaxRAMPercentage=70.0",
  "-XX:+ExitOnOutOfMemoryError",
  "-jar",
  "quarkus-run.jar"
]

Before the next command, predict the result. The entrypoint uses java directly, and we chose the distroless runtime variant. There should be no /bin/sh available to run id:

podman run --rm \
  --entrypoint=/bin/sh \
  localhost/themainthread/hardened-quarkus-jib:1.0.0-SNAPSHOT \
  -c 'id'

The command must exit with a non-zero status and report that /bin/sh cannot be found. If it prints a user ID, you built from a different image variant or replaced the entrypoint with something that pulled a shell into the final filesystem.

Push Directly to a Registry

The local build needs Podman because Jib must load the image somewhere and the integration test must run it. A separate CI packaging job can push straight to a registry:

./mvnw package \
  -Dquarkus.container-image.push=true \
  -Dquarkus.container-image.image=quay.io/acme/hardened-quarkus-jib:1.0.0

Provide registry credentials through the CI secret store with QUARKUS_CONTAINER_IMAGE_USERNAME and QUARKUS_CONTAINER_IMAGE_PASSWORD. Jib talks to the registry API, so this packaging path needs neither a Dockerfile nor a privileged Docker-in-Docker service. It runs @QuarkusTest, but it does not launch the pushed image. Keep the packaged-container test in a job that has a container runtime, or run it after deployment against the promoted image.

For a multi-architecture release, push both supported platforms:

./mvnw package \
  -Dquarkus.container-image.push=true \
  -Dquarkus.container-image.image=quay.io/acme/hardened-quarkus-jib:1.0.0 \
  -Dquarkus.jib.platforms=linux/amd64,linux/arm64

A local image store can only run one platform at a time. The registry is the right destination for the multi-architecture index.

Verify the published index:

podman manifest inspect \
  quay.io/acme/hardened-quarkus-jib:1.0.0 \
  | jq '.manifests[].platform'

Expected platforms:

{
  "architecture": "amd64",
  "os": "linux"
}
{
  "architecture": "arm64",
  "os": "linux"
}

Make the Image Survive Production

The hardened base removes a large amount of operating-system content. It does not remove our responsibility for updates, provenance, application dependencies, or runtime diagnosis.

Rebuild tags and deploy digests

25-runtime tracks the Java 25 line and receives patched image revisions. Rebuilding against that tag is convenient because Jib picks up the new base layers. A released workload should deploy an immutable digest so a rollback always returns to the same bytes.

Use both behaviors in the pipeline: resolve and verify the current approved Hummingbird tag, build the application, push the result, and deploy the final application image by digest. Schedule rebuilds even when the application source has not changed. Hardened images are immutable, so a base-image fix reaches production only after another application image is built and deployed.

Verify the base signature

The Hummingbird documentation provides Red Hat’s public signing key. Verify the base before the build:

cosign verify \
  --key https://security.access.redhat.com/data/63405576.txt \
  --insecure-ignore-tlog \
  registry.access.redhat.com/hi/openjdk:25-runtime

This checks the signature against the supplied key. --insecure-ignore-tlog skips transparency-log verification, so record the command as signature-only verification without a Rekor inclusion check.

Debug through the process boundary

A missing shell changes the debugging routine. Use podman logs hardened-quarkus, HTTP health endpoints, metrics, traces, heap dumps written to an approved volume, and an external debug container. Installing a shell into the production image restores the tools the hardened base deliberately removed.

The same rule applies to startup scripts. An entrypoint such as /bin/sh run-java.sh works with many general-purpose base images and fails immediately here. The direct Java entrypoint we inspected earlier is part of the compatibility proof.

Keep trust stores and SBOMs separate

Enterprise services often call internal TLS endpoints. The Hummingbird Java image reads its truststore from /etc/pki/ca-trust/extracted/java/cacerts. Mount a prepared truststore read-only, or create a derived image through the documented builder-to-runtime flow. On SELinux systems, remember the :Z or :z relabel option for Podman volumes.

The base-image SBOM describes the Hummingbird content. It does not include the Quarkus libraries we add in later layers. Scan the final application image and generate an application SBOM as a separate release artifact. A quiet base scan can still sit underneath a vulnerable Maven dependency. Containers have a sense of humor about organizational boundaries.

The FIPS variants have another hard boundary: the Hummingbird documentation states that validation applies when they run on RHEL systems installed in FIPS mode. Changing an image tag cannot make an arbitrary host compliant.

Conclusion

We built a Quarkus JVM image with a direct Java entrypoint, a non-root Hummingbird runtime, stable application layers, and tests that cross the real container boundary. Jib removes the Dockerfile maintenance, while the inspection and failure checks keep the runtime assumptions explicit.

The complete code is available in the hardened Quarkus Jib example.

Subscribe now

It is not a round anniversary. It does not look good on a sticker. It has no obvious meaning unless you already know the counter.

For me, it means 510 daily posts on The Main Thread.

That is a lot of words. It is also a lot of small Quarkus applications, build failures, agent experiments, screenshots, wrong assumptions, late evenings, and mornings where I opened the editor because the streak was still alive.

When I started this, the motivation was simple:

• Do I still know Quarkus well enough to explain it from the inside?

• Can I keep up with what is happening in Java, cloud-native development, standards, and AI-assisted engineering?

• Can I use agents better by using them every day, on real writing and real code, with real constraints?

The Main Thread was never meant to be a place for polished summaries from a distance. I wanted to build something, test it, break it, fix it, and then explain what survived. A working notebook, basically. For people who still care about what the system does after the diagram looks nice.

Publishing every day helped with that.

It kept me close to the material. If Quarkus changed, I had to notice. If an extension behaved differently than expected, I had to find out why. If an agent produced code that looked fine but failed under a real test, I had to deal with the failure before publishing.

That discipline helped and it also had a cost.

Daily publishing changes how you look at your day. Every topic becomes a possible post. Every experiment becomes a thing you owe yourself to finish. Every unfinished idea waits in the corner (an almost endless Note on my phone in fact) and quietly adds pressure.

From the outside, this looks productive. For a while, it is. From the inside, it can also become heavy. You start measuring time in publishable units, which is a strange way to live and a worse way to think.

The last 510 posts also taught me a lot about agents.

Agents help. They really do.

They help with structure. They help with first drafts. They help with small code changes, test scaffolding, option comparison, and finding cases I missed. They are useful sparring partners when I already know what I am trying to build.

But agents do not magically create high-value technical content.

They can produce a lot of text. That does not mean the text contains insight.

They can generate code. That does not mean they understand the failure mode.

They can summarize documentation. That does not mean they know why a senior engineer should care.

The valuable part still needs judgment. It needs technical memory. It needs taste. It needs the ability to say: this example is too clean, this abstraction hides the real problem, this tutorial needs a failure case, this claim needs verification, this topic is not ready yet. And sometimes this judgment needs confirmation from engineering or the upstream maintainers even. That takes time.

Sometimes the agent makes that work faster. Sometimes it creates more work because the draft sounds confident before it is correct. For daily publishing, that difference matters. Speed helps only when verification keeps up.

After 510 daily posts, I know two things better than before.

First, I still enjoy the craft. I still enjoy Quarkus. I still enjoy Java. I still enjoy the plain parts that make systems reliable: standards, tests, contracts, observability, failure handling, and small design choices that keep code understandable. And I enjoy working with the new agentic overlords. It’s fun to “write” software again. And an even bigger joy to figure out how to eliminate all that vibecoded mess and turn it into something that is architecturally sound.

Second, daily publishing has done its job.

So I am changing the cadence.

This is not the end of The Main Thread. I will keep writing about Quarkus, Java, standards, agentic engineering, and the engineering practices we need around generated code. I will keep testing ideas in code. I will keep looking at where agents help, where they fail, and where they need tighter boundaries.

But I want to give the next posts more room.

Some examples need a second test. Some topics need a few days before they are worth publishing. Some claims need more proof. Some articles should sit overnight so the confident parts can become the correct parts. And some topics do even need multiple parts.

The daily streak was good at making me publish. It was less good at making me wait and think.

510 posts is not a clean milestone. It is enough evidence for me though.

Consistency works. Agents help when there is engineering judgment around them. And a streak is only worth keeping while it serves the work.

I am keeping the work.

I am changing the rhythm.

Subscribe now

Quarkus source tree splits into core/, extensions/, and independent-projects/. The same class names show up in deployment and runtime modules. Generated config classes, synthetic beans, and build-item scheduling sit between the code you can read and the behavior you care about. If you just use rg it might work on the first jump. By the third or fourth jump you are reading whole files again and hoping you opened the right module. And the same is true for coding agents.

codebase-memory-mcp builds a local knowledge graph from a checkout and exposes it through MCP. I wanted to see whether that graph helps IBM Bob on a real Quarkus platform tree and not a toy service with twelve files and one obvious call chain.

I indexed Quarkus, wired the MCP server into Bob, and ran the same @ConfigMapping trace twice: once with shell tools only, once with graph tools. The interesting part is not whether Bob can answer the question. Both paths got there. The interesting part is what each path costs, what it misses, and what that says about agent tooling on framework-scale codebases.

The setup

codebase-memory-mcp ships as a single static binary. No Docker, no API keys, no built-in LLM. Bob still does the thinking. The MCP server answers structural questions about the indexed graph.

curl -fsSL https://raw.githubusercontent.com/DeusData/codebase-memory-mcp/main/install.sh | bash

The installer puts the binary at ~/.local/bin/codebase-memory-mcp. If your shell cannot see it:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

On macOS, Gatekeeper sometimes blocks a fresh download:

xattr -cr ~/.local/bin/codebase-memory-mcp
codesign --force --sign - ~/.local/bin/codebase-memory-mcp

The install script does cover all of this. Just in case you want to do this manually.

Bob reads MCP config from .bob/mcp.json in the workspace root. Open your Quarkus checkout as the Bob workspace and create the file there:

{
  "mcpServers": {
    "codebase-memory-mcp": {
      "command": "codebase-memory-mcp",
      "args": [],
      "groups": ["read", "mcp"],
      "disabled": false
    }
  }
}

If Bob says the command is missing, set "command" to the full path under ~/.local/bin. The groups entry is what mattered in my runs: Bob could call graph tools from the Quarkus workspace without extra wiring.

Reload MCP. You should see tools such as index_repository, search_graph, query_graph, trace_path, get_code_snippet, and get_architecture.

Indexing

I indexed from the shell first:

codebase-memory-mcp index ~/Projects/quarkus

On my laptop, with v0.8.1 and cache under /tmp, the full Quarkus index reported 29,206 files, 375,498 nodes, 1,587,540 edges, and about 24 seconds of wall time. Peak RSS hit roughly 12 GB during the heavy passes. That is not instant, but it is fast enough to treat as a repeatable step before a debugging session.

Graph data lives under ~/.cache/codebase-memory-mcp/ by default. Bob names the project from the checkout path. Run list_projects once and use whatever name Bob shows you in the prompts below.

You can also ask Bob to index through MCP. Paste this into a fresh chat:

The Quarkus source checkout is open in this workspace.

Index it with codebase-memory-mcp if it is not indexed already.

Rules for this run:

- Call list_projects first.
- If the Quarkus project is missing, call index_repository with the workspace path.
- If the project already exists, say so and do not re-index.
- Do not start the @ConfigMapping trace yet.
- Do not spawn subagents.

Report back with:

1. Whether the repo was already indexed or needed a fresh index
2. File, node, and edge counts if the tool returns them
3. Any error or slow step you hit

Bob called list_projects, found the existing index at 356k nodes and 1.5M edges, and stopped. Cost about nine cents. list_projects itself took over a minute on cold start. That is worth remembering if you are waiting on Bob and wondering whether the index failed. Indexing cost happen.

The question

The benchmark task:

Trace how a @ConfigMapping class moves from deployment scanning and generated mapping classes to synthetic bean registration and runtime creation.

“Weaker” versions of this question send you in the wrong direction. “Who calls discoverConfigMappings?” sounds precise. Quarkus does not schedule build steps through normal Java call chains. It schedules them through build-item producers and consumers, which are objects passed between @BuildStep methods. A build item is not a method call. A synthetic bean is a CDI bean registered at build time. @ConfigMapping injection often ends there before runtime code sees a constructor you can grep.

Better questions:

• Which build steps produce and consume ConfigMappingBuildItem?

• Which deployment code registers the synthetic bean for a mapping?

• Which runtime creator calls SmallRyeConfig#getConfigMapping(...)?

The path crosses core/deployment, extensions/arc/deployment, extensions/arc/runtime, and core/runtime. That is the point. A tool that only works on single-module apps would not tell us much.

Build-time work sits on the left. Runtime materialization sits on the right. The line you have to draw in your head is the build-item boundary: ConfigMappingBuildItem is the handoff from core deployment to arc deployment. trace_path will not draw that line for you.

A map before the trace

Before the @ConfigMapping question, I asked Bob for a repo map. Fresh chat, graph indexed, MCP loaded:

The Quarkus source tree is indexed in codebase-memory-mcp.

Project name: Users-meisele-Projects-quarkus

Give me a first-pass onboarding map of the repo.

Rules for this run:

- Call get_architecture first with the project name above.
- Do not spawn subagents.
- Do not open source files.
- Do not create HTML artifacts.
- Summarize dominant languages, the largest package areas, and rough boundaries between major subsystems.

Report back with:

1. The tool output you used (name only — do not paste the full JSON)
2. The five largest package areas you would tell a new contributor about
3. One subsystem you would inspect first if the task were extension work in arc or resteasy-reactive

get_architecture returned in 1.4 seconds and named resteasy-reactive, arc, bootstrap, qute, vertx-http, and tools as dominant package areas. Bob did not open a single source file. Cost about 0.2 Bobcoins.

On the first pass I had left subagent instructions in the prompt. Bob spawned explore subagents, they could not see MCP in that run, and Bob recovered from get_architecture alone. Subagents can use MCP if you configure them. For graph-first work I still keep them out of the prompt. The default explore subagent reaches for grep and adds noise.

Conclusion: get_architecture is worth the index cost on its own. For a platform repo, five minutes of orientation without reading files is a real win. It does not replace code reading. It tells you which neighborhood to start in.

Baseline: grep and file reads

Fresh Bob chat. MCP disabled for the task. Paste:

Trace how @ConfigMapping classes move from deployment scanning and generated mapping classes to synthetic bean registration and runtime creation.

Rules for this run:

- Use only shell commands and file reads (execute_command, grep, read_file).
- Do not use codebase-memory-mcp or any MCP graph tools.
- Do not load skills, spawn subagents, or create HTML artifacts.
- Start with rg ConfigMappingBuildItem and open files from the hits.
- Stop when you can name the main deployment steps, the arc deployment step, and the runtime creator.

Report back with:

1. Files you opened, in order (count them)
2. Which steps are build-time and which are runtime
3. What you are still unsure about

The first useful rg hits look promising:

core/deployment/.../ConfigGenerationBuildStep.java:196:    void discoverConfigMappings(
core/deployment/.../ConfigGenerationBuildStep.java:225:    void generateConfigMappings(
extensions/arc/deployment/.../ConfigMappingProcessor.java:86:    void registerConfigMappingBeans(
extensions/arc/runtime/.../ConfigMappingCreator.java:13:public class ConfigMappingCreator implements BeanCreator<Object> {

Bob still opened eight files. It loaded the Quarkus writing-build-steps skill despite the prompt. The final answer traced ExtensionLoader through ConfigGenerationBuildStep, ConfigMappingProcessor, and ConfigMappingCreator with correct build-item language. It went deeper on bootstrap than the graph runs did. Cost about eighty-seven cents.

Conclusion: Plain tools can answer hard Quarkus questions if the agent already knows the platform model. The cost is context. Every file read pulls deployment code, static imports, and generated-class references into the chat. Bob got the answer. It paid for that answer in tokens and in files it had to open before it knew which methods mattered.

The writing-build-steps skill

One reason the baseline answer sounded like Quarkus and not generic Java: Bob loaded the writing-build-steps skill before its first rg pass, even though the prompt said not to load skills.

That skill does not come from codebase-memory-mcp. It comes from quarkus-agent-mcp, the Quarkus project’s MCP server, which loads skills from the quarkusio/skills repository. When you open the Quarkus checkout in Bob with quarkus-agent-mcp in .mcp.json, which many Quarkus contributors already have, Bob can pull in bundled skills through quarkus_skills. writing-build-steps is one of them. It documents how @BuildStep methods produce and consume build items, how ordering follows dependency edges rather than source order, and when to use recorders in deployment modules.

Graph path: the same question, different tools

Fresh chat. MCP loaded. Index warm. Paste (replace the project name with yours from list_projects):

The Quarkus source tree is indexed in codebase-memory-mcp.

Project name: Users-meisele-Projects-quarkus

Trace how @ConfigMapping classes move from deployment scanning and generated mapping classes to synthetic bean registration and runtime creation.

Rules for this run:

- Use codebase-memory-mcp MCP tools only for the first 15 tool calls: search_graph, query_graph, get_code_snippet, and trace_path if a plain call chain is the right question.
- Do not spawn subagents.
- Do not create HTML artifacts.
- Do not load skills.
- Use grep or read_file only after the graph narrows the target (maximum one read_file for confirmation).
- Quarkus build steps connect through build items, not normal Java call chains. Say that explicitly in your answer.

Required MCP sequence:

1. list_projects — confirm project name
2. query_graph with this Cypher:

MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method)
WHERE c.name IN ['ConfigGenerationBuildStep', 'ConfigMappingProcessor']
RETURN c.name AS class_name, m.name AS method_name, m.file AS file_path
ORDER BY class_name, method_name

3. search_graph for ConfigMappingCreator and ConfigMappingBuildItem
4. get_code_snippet for the methods you need — not whole files
5. Optional: trace_path on ConfigMappingCreator only if you explain why a call chain applies

Report back in plain markdown (no HTML):

1. MCP tools you called, in order
2. Files and methods the graph surfaced
3. Which steps are build-time and which are runtime
4. Any graph edge you distrust and why (especially in_degree = 0 on build steps)

If get_code_snippet fails with "symbol not found", run search_graph again and use the qualified_name from the result, not a Java package name you guess.

I ran this twice.

First pass: loose prompt, no guardrails. Bob made about thirty graph tool calls, skipped query_graph, fell back to grep and one read_file, and generated an HTML artifact nobody asked for. The answer was strong. It explicitly distrusted in_degree = 0 on @BuildStep methods. Cost 2.45 Bobcoin and nearly a million input tokens. Most of that was Bob context, workspace rules, Quarkus AGENTS.md, and an eight-thousand-token HTML detour, not the MCP queries themselves.

Second pass: same prompt with the guardrails above. Bob called query_graph first. Cypher returned methods but empty file_path values. Bob recovered through search_graph qualified names, then get_code_snippet. No HTML. No file reads. It stated plainly that trace_path is the wrong tool for build-item wiring. Cost 0.49 Bobcoin.

That five-fold cost drop is the first lesson. On framework repos, prompt shape dominates tool choice. The graph did not get five times faster. Bob stopped doing expensive things the prompt now forbids.

The second lesson is about metadata gaps. Cypher listed 29 methods on ConfigGenerationBuildStep and 4 on ConfigMappingProcessor in under a second. get_code_snippet failed until Bob passed qualified_name values from search_graph, not guessed package paths. The graph knows structure. It does not always surface file paths in the shape your agent expects. Plan for a two-step lookup: Cypher to enumerate, search_graph to locate.

The third lesson is about what the graph cannot see. I ran a short follow-up with graph tools only:

The Quarkus graph project is Users-meisele-Projects-quarkus.

This is a short graph-only check. Do not open source files. Do not spawn subagents. Do not create HTML artifacts.

Run these MCP tools in order and summarize the results:

1. query_graph:

MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method)
WHERE c.name IN ['ConfigGenerationBuildStep', 'ConfigMappingProcessor']
RETURN c.name AS class_name, m.name AS method_name, m.file AS file_path
ORDER BY class_name, method_name

2. search_graph with name pattern ConfigMappingBuildItem, label Class, limit 10
3. search_graph with name pattern ConfigMappingCreator, label Class, limit 10
4. search_graph with name pattern discoverConfigMappings, label Method, limit 5
5. search_graph with name pattern generateConfigMappings, label Method, limit 5

Report back with:

1. Methods listed for each deployment class
2. Whether discoverConfigMappings or generateConfigMappings show in_degree = 0
3. One sentence on why that matters for Quarkus build steps vs Java call graphs

Both discoverConfigMappings and generateConfigMappings showed in_degree = 0. Bob explained why that is normal: @BuildStep entry points are invoked by the build framework, not by other Java methods in the graph. Cost 0.16 Bobcoins. No file reads.

If you treat in_degree = 0 as dead code, you will misread Quarkus deployment. If you treat trace_path as proof of build-step ordering, you will misread it too. The graph is a search accelerator, not a build engine simulator.

What held up

Indexing is practical. A full Quarkus platform tree in under half a minute, on a laptop with enough RAM, changes the economics. You can afford to index before a hard debugging session instead of treating graph search as a batch job.

Java support is good enough for navigation. codebase-memory-mcp advertises Hybrid LSP support for Java as of v0.8.0. The old “Java is syntax-only” claim is stale. On this task the graph found the right classes and methods across modules quickly. The gaps are semantic, not syntactic.

Both paths reached the right answer. Baseline and graph runs named ConfigGenerationBuildStep, ConfigMappingProcessor, and ConfigMappingCreator. They separated build-time discovery and generation from synthetic bean registration and runtime materialization. Neither path invented a fake CALLS chain through ConfigMappingBuildItem.

Graph path used fewer files. Baseline: eight full file reads. Graph v2: snippets only. On a platform repo where deployment files run to hundreds of lines, that difference compounds across a session.

What still breaks

Build items are invisible to call-graph tools. Producer/consumer relationships between @BuildStep methods do not show up as CALLS edges. You still need Quarkus build-phase knowledge to interpret what the graph returns.

JVM edges have holes. Static imports, generated classes, recorders, and synthetic bean paths are hard cases. Some deployment helpers show in_degree = 0 even when the source clearly calls them. Do not trust a graph edge on Quarkus deployment code without opening the file once.

Agents ignore prompt guardrails. Bob loaded writing-build-steps on the baseline run and generated HTML on graph v1 despite explicit instructions not to. Tighter prompts helped. They did not guarantee obedience. Factor that into cost estimates.

Agent context is the real bill. The graph MCP calls were cheap. Bob’s workspace rules and Quarkus documentation loaded into every chat were not. The expensive run was not slow Cypher. It was a million tokens of context plus an unsolicited HTML report.

How I would use this in practice

For a platform or large monorepo checkout:

1. Index once. Keep the cache warm.

2. Run get_architecture before you ask a trace question. Know which subsystem you are in.

3. Use Cypher to enumerate methods on known anchor classes. Use search_graph when you need qualified names or file paths.

4. Fetch snippets, not files, until you need surrounding context.

5. Ask follow-ups that force the agent to classify edges: normal Java call, build-item handoff, synthetic bean path, or “I would not trust this without reading the source.”

Score your own runs on four questions:

1. Did it find the right files?

2. Did it separate build-time work from runtime work?

3. Did it claim a plain call chain where Quarkus uses build items?

4. How much full-file reading was still needed after the first structured result?

On my runs, baseline won on bootstrap depth. Graph v2 won on file count, cost, and speed to the anchor methods. Neither replaced knowing how Quarkus build phases work.

Quarkus isn’t really hard for users

But Quarkus is hard to search because the real logic spreads across build-time phases, runtime modules, generated config classes, and synthetic beans. codebase-memory-mcp shrinks that search space on a full platform tree. get_architecture gives an agent a usable map in seconds. Cypher and search_graph get you to the right methods across modules without opening every deployment file along the way.

The graph does not understand build-item scheduling. It will not tell you which step consumes ConfigMappingBuildItem. It will tell you where to look, faster than grep alone, if you ask the right questions and read the answers with platform knowledge in the loop.

The companion repo with scripts, and the scoring rubric can be found on my main-thread Github repository.

Subscribe now

For coding agents, start there. Then evaluate the workflow.

The work your team cares about starts after the model begins acting. It opens files. It reads repo rules. It loads skills, or ignores them. It calls tools. It edits code. It runs tests. It explains what it changed. A human then has to decide whether the result is safe to merge.

The eval has to cover that whole loop.

Model scores still matter. Token charts still matter. A polished demo can still show what is possible. But none of that tells you whether the workflow keeps wrong turns small, recovers from bad tool output, lowers review cost, or gives reviewers results they can trust.

I want evals to answer that.

Benchmarks Measure A Smaller Unit

Model benchmarks are good for comparing base capability. They tell you whether one model is generally stronger than another under a fixed test setup. For coding work, they can show signal around bug fixing, repository reasoning, code generation, and test repair.

I would still look at them. Choosing a weak model and hoping process will save you just creates more work later.

But a coding-agent workflow has more moving parts than the model. The agent gets instructions. It reads local files. It may have skills for a framework or codebase. It may call search, shell, test, browser, GitHub, database, or deployment tools. It may have permission gates. It may have memory from earlier turns. It may be forced to recover after a bad assumption.

Most of that never reaches the benchmark.

One-off demos have a different problem. They feel close to real work because you can watch the agent do things. It reads files, makes a patch, runs a command, and gives a tidy summary.

One good demo proves that one path worked once. It leaves out how often the agent reads the wrong file, trusts stale docs, deletes a test, changes more than asked, or burns 10 turns recovering from a mistake it made in turn two.

The expensive part of agent work is often recovery and review. Evaluate that directly.

Workflow Evaluation

Workflow evaluation means the run is the unit under test.

A run includes the task prompt, model, skills, tool calls, files read, files changed, commands run, failures, recoveries, final answer, and human review. The model response is only one part of that run.

For coding agents, ask questions like these:

• Did the agent read the right files before editing?

• Did it keep the change inside the request?

• Did it notice missing tests?

• Did it run checks that prove the changed behavior?

• Did it handle stale docs or repeat them?

• Did it recover from a failing command by reading the error?

• Did it stop before unsafe changes?

• Did the final summary give the reviewer real evidence?

The final answer can sound good while the diff is wrong. We have all seen that version of confidence. It happens often enough to measure.

So the eval has to inspect the work around the answer: diff, logs, tool trace, tests, touched files, approval points, and review notes.

Use Realistic Task Suites

Start with small sample projects. A sample project can be a tiny repo, a copied slice of a real service, or a made-up project that keeps one failure mode without exposing private code.

The sample should be small enough for a human to inspect. It should still contain the mess that makes agent work costly: stale files, partial tests, unclear requests, dangerous shortcuts, and tools that sometimes fail or mislead.

I would include these paths.

Easy path
The task is clear, local, and covered by tests. The agent should make a small diff and run the obvious checks. This is your baseline.

Unclear request path
The request has two possible meanings. The agent should ask, state its assumption, or choose the lower-risk path and say so. Silent guessing should cost points.

Failure path
The first test run fails. Watch what happens next. A good agent reads the failure, maps it to the right file, and fixes the cause. A weak workflow starts guessing.

Stale docs path
The README points to an old API or framework pattern. The code says something else. The agent should prefer current local evidence or verified docs.

Missing tests path
The code change is easy, but no test covers the behavior. Existing tests passing should not be enough. The agent should add focused coverage or explain why the sample cannot support it.

Unsafe changes path
The tempting fix deletes a test, changes a public contract, rewrites build files, gives the code more access, or runs a risky command. The right behavior may be to stop and ask.

Bad tool output path
A search misses a file. A test runner returns incomplete output. A tool reports success while the output file is missing. The agent should check again before it trusts the tool.

These tasks can be small. They need to keep the same failure.

Measure the Review Cost

Pass/fail is not enough.

Two runs can both pass tests. One changes two files, adds one focused test, and gives the reviewer the exact command output. The other changes 18 files, removes a failing check, and writes a clean summary. Treating both as a pass destroys the signal.

Track the parts a reviewer pays for.

Recovery turns
How many turns or tool cycles did the agent need after the first wrong move? This shows whether the workflow can repair itself cheaply.

Review time
How long did a human need to approve or reject the result? This is rough, but it is honest. If the patch passes tests and still takes an hour to trust, the workflow is expensive.

Diff size
How much changed? A small diff can be wrong. A large diff can be necessary. But unnecessary churn is review tax.

Files touched
Which files changed? Edits outside the expected area should show up immediately.

Test evidence
Which checks ran? What were the exit codes? Do the checks cover the changed behavior? “Tests passed” without this detail is almost useless.

Rule and policy breaks
Did the agent delete tests, change public APIs without calling it out, ignore an approval gate, edit generated files, or use a tool that was not allowed?

Result quality
Is the final patch, doc, migration note, or design write-up something a human can use? Automated checks can catch many problems. They cannot fully judge whether a design belongs in your system.

Human approval points
Did the agent stop when the task crossed into business risk, architecture risk, or hard-to-undo work? Good escalation is a feature.

A single score hides too much. These measurements show whether a model is faster but messier, a skill reduces recovery turns, or a prompt change improves test evidence while increasing diff size.

Now the team has something to fix.

A Specific Skill Eval

Let’s make this concrete with one skill. I have added the folder to my Github repository if you want to copy it and play along.

The example skill is review-standards. Its job is to guide an agent that reviews authentication code. The sample project contains one small file, src/auth.ts, with two problems:

• Passwords are hashed with SHA-1

• Reset tokens are compared with ===

The skill has three states:

• No skill loaded

• A stale skill that finds auth problems but ignores what the user asked for

• An improved skill that finds both issues and stays inside the request

The runnable sample lives in review-standards-skill:

review-standards-skill/
  promptfooconfig.yaml
  scripts/mock-review-agent.js
  samples/
    no-skill/
      src/auth.ts
    v1/
      .agents/skills/review-standards/SKILL.md
      src/auth.ts
    v2/
      .agents/skills/review-standards/SKILL.md
      src/auth.ts

The source file is the same in all three sample projects. Only the skill changes.

The first version of the skill knows about password hashing and token comparison. It does not know that the user may ask for a smaller review. The improved version adds this rule:

Respect what the user asked for. If the user says to review only password
handling, report password issues and leave token findings out of the result.

That rule sounds small. The eval makes it visible.

The Promptfoo Config

promptfoo/promptfoo gives us a repeatable way to run the same tasks against each skill state. The Promptfoo skill guide uses the same basic pattern: keep the task and runtime steady, swap the skill version, and compare behavior.

For this article, I use a local exec: provider. It reads the sample project, returns JSON, and keeps the dashboard runnable without an API key. In a real coding-agent eval, replace the exec: providers with openai:codex-sdk, anthropic:claude-agent-sdk, or another agent provider. Promptfoo’s provider docs list custom script providers and agent providers, and the skill guide shows the Codex .agents/skills/... layout.

The config compares three providers:

providers:
  - id: "exec: node scripts/mock-review-agent.js"
    label: no-skill-baseline
    config:
      sampleDir: samples/no-skill

  - id: "exec: node scripts/mock-review-agent.js"
    label: stale-review-standards
    config:
      sampleDir: samples/v1

  - id: "exec: node scripts/mock-review-agent.js"
    label: improved-review-standards
    config:
      sampleDir: samples/v2

Then the tests ask two questions:

tests:
  - description: Finds both auth review issues
    vars:
      request: Review src/auth.ts for password handling and token comparison issues.
      expectedIssues:
        - weak-password-hash
        - timing-unsafe-compare
      outOfRequestIssues: []

  - description: Stays inside password-only review request
    vars:
      request: Review src/auth.ts only for password handling issues.
      expectedIssues:
        - weak-password-hash
      outOfRequestIssues:
        - timing-unsafe-compare

The first test asks whether the run found both known issues. The second test asks whether the run stayed inside the request. When the user asks only for password handling, the token issue should stay out.

The Promptfoo check scores four things:

• Expected issue IDs were found

• Issue IDs outside the request were not returned

• Evidence was included

• The skill was used

For real agents, Promptfoo also supports skill-specific checks. The Promptfoo docs show skill-used checks for agent providers that expose skill/tool calls directly. You can also use trace-based checks when you need to prove that a Codex run read the matching SKILL.md.

Running It

Run the sample:

cd review-standards-skill
npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json

This eval is supposed to fail some results. That is the point. The weak cases should stay red. Promptfoo returns a non-zero exit code when results fail, which is what you want in CI.

My run produced this result:

Results:
  ✓ 3 passed (50.00%)
  ✗ 3 failed (50.00%)
  0 errors (0%)
Duration: 0s (concurrency: 4)

The dashboard is the easiest way to read it:

npx promptfoo@latest view --port 15500

Open http://localhost:15500

and select the latest eval. The grid tells the story without much explanation:

• no-skill-baseline fails both tasks because it finds no issues and has no skill evidence

• stale-review-standards passes the broad auth review but fails the password-only review because it reports the token issue anyway

• improved-review-standards passes both tasks

This is the result I want from a skill eval. It shows the behavior change: the improved skill still found the real issues, and it stopped reporting issues the user did not ask for.

The same shape works with a real agent. Keep the sample and checks. Swap the provider. Let the dashboard show whether the skill reduced wrong turns, added evidence, stayed inside the request, or just made the answer sound more serious.

The Eval Loop

The loop is small:

skill version
  -> same sample project
    -> same agent/provider
      -> output JSON + trace + files
        -> promptfoo checks
          -> dashboard review
            -> skill edit
              -> rerun

The dashboard shows why the run failed: it missed expected issues, reported issues outside the request, gave no evidence, or did not use the skill. Now you have something to fix.

What Promptfoo Should Not Decide Alone

Automated evals are good at catching repeatable failures. They are weaker at judging whether a solution belongs in your architecture.

Promptfoo should not be the final authority on:

• Architecture judgment

• Business risk

• Security acceptance for real production data

• Whether a migration is safe for your customers

• Whether the design is maintainable in your codebase

• Final human acceptance

You can catch many bad patterns with checks: large diffs, missing tests, files that should not change, unsafe commands, weak evidence, and broken output shape. You can also use model-graded scoring rules for some quality checks when scale matters.

But architecture and business risk still need humans. A good eval suite reduces review cost. It does not make the final merge decision.

Avoid Chasing the Score

Score chasing starts when the number becomes more valuable than the behavior.

The signs are easy to spot. The suite has only happy paths. The checks look at wording because wording is easy to grade. The samples are tuned until the preferred workflow wins. “Flaky” failures disappear before anyone asks what they show. The final report has a clean percentage, and no reviewer can explain what went wrong.

That gives you confidence without trust.

Design evals so failures stay visible.

Keep the suite small enough to understand
A dozen good cases beat 300 shallow ones when you are still learning what breaks. Add scale after the signal is clear.

Include failure paths
Broken tests, stale docs, missing files, unsafe changes, and bad tool output should be normal eval inputs.

Freeze samples and expected behavior
Version the sample project, task prompt, skills, and scoring scripts. If the task changes every run, the score tells you very little.

Store run output
Keep the diff, logs, final answer, tool trace when available, and human review notes. Scores without those files are hard to debug.

Separate hard gates from diagnostics
Deleted tests, files that should not change, unsafe commands, and missing required evidence can fail CI. Larger diffs, extra recovery turns, or more files read may be review signals instead.

Reward correct escalation
Sometimes the right output is “I need approval before this change.” If your eval marks that as failure because no patch was produced, you are training the workflow to be reckless.

Watch what the numbers encourage
Small-diff scoring can discourage necessary tests. Test-pass scoring can encourage weak tests. Low-token scoring can make the agent stop reading too early. Each number needs another number or review check beside it.

Check with humans
Have reviewers sample passing and failing runs. If humans disagree with the automated score, fix the scoring.

An eval suite should make failures cheaper to find and easier to explain. If it only gives you a better slide, it has become part of the problem.

A Practical Starting Set

If I were starting from zero, I would create one small sample repo and five tasks:

1. A clean bug fix with an existing failing test

2. The same bug fix with no test coverage

3. An unclear request where the agent should ask or state assumptions

4. A stale-docs task where the README points to an old API

5. An unsafe-change task where the tempting fix edits a public contract

For each task, I would collect:

• Final diff

• Files touched

• Commands run and exit codes

• Test evidence

• Number of recovery turns

• Rule and policy breaks

• Human review verdict and review time

Then I would run the suite across:

• Current workflow

• Workflow with one prompt change

• Workflow with one skill change

• One stronger model

• One cheaper model

• A plain model baseline where possible

This is enough to start learning. You are not trying to prove a universal law on day one. You are trying to see whether the workflow fails in small, understandable ways.

After that, add more samples, more models, and more automation.

The Real Score

The real score is whether the workflow keeps wrong turns small, recovers with evidence, respects approval boundaries, and leaves a human with work they can review without replaying the whole session in their head.

Model benchmarks help you choose candidates. Token charts help you watch cost. Demo runs show what is possible. Workflow evaluation tells you whether the whole system is becoming safer and cheaper to use.

Coding agents do not fail as benchmark rows. They fail as diffs, missed tests, stale assumptions, unsafe tool calls, and tired reviewers.

A good eval loop makes those failures visible while they are still small enough to fix.

Subscribe now

That is the problem Wanaku is trying to solve.

Wanaku is an open-source MCP router built on Quarkus and Apache Camel. Your agent connects to one Wanaku endpoint over the Model Context Protocol, and Wanaku manages the catalog behind it: tools and forwards to other MCP servers. That gives you one place to manage what the agent can see instead of hand-wiring every capability into every client.

I wanted to try Wanaku since a while and had this on my to-do list. Finally got a chance to run an example and share what I learned. From idea to this tutorial, a few parts did not survive contact with the real CLI. The working path below uses a local Wanaku router, one imported HTTP toolset, and one forwarded upstream MCP server.

What We Are Building

• A local Wanaku router on http://localhost:8080

• One imported HTTP toolset from the official wanaku-toolsets repository

• One forwarded upstream MCP server: the official Everything reference server from the MCP servers repository

• A verified catalog that shows local HTTP tools and remote MCP tools behind the same router

This is not the exact bridge I wanted to use. I started with the official filesystem server because that looks closer to real work. In Wanaku 0.1.3, that path failed for reasons I explain later. The bridge below is the one that worked end to end.

What You Need

• Java 21 or later

• A recent Node.js with npx

• Internet access on first run, because Wanaku and the MCP server packages need to be downloaded (obvl)

• ☕️☕️☕️

I recommend Java 21 for this article. In my validation run, Wanaku 0.1.3 installed on Java 25, but the CLI needed extra JVM flags because of module access rules.

Check the local runtime first:

java -version
node -v
npx --version

If you are already on Java 25 and want to stay there, I had to use this during validation:

export JAVA_TOOL_OPTIONS='--add-opens=java.base/java.lang=ALL-UNNAMED'

That is a workaround. I could not find official docs about Java version support with Wanaku yet.

Install the Wanaku CLI

For a Mac tutorial, the JBang path is the simplest one because it avoids hardcoding a release ZIP names.

Install JBang:

brew install jbangdev/tap/jbang
jbang version
jbang app setup

Open a new terminal, or reload your shell so the JBang app path is available. After that, install Wanaku:

jbang app install wanaku@wanaku-ai/wanaku
wanaku --version

If you want to run this on Java 25, use wanaku --help to check if it works. In my run, wanaku --version did not.

Start Wanaku Locally

Before starting the stack, ask Wanaku which local services it can launch:

wanaku start local --list-services

In my run, the local service catalog was:

• service-http

• service-tavily

• service-exec

That list matters later because it explains why my file resource example did not work.

Now start the local router:

wanaku start local

Keep that terminal open. Local mode disables authentication automatically, which is fine for a laptop demo and wrong for anything shared.

After startup, check the router health:

curl -s http://127.0.0.1:8080/q/health
curl -s http://127.0.0.1:9000/q/health

In my validation run, both the router on 8080 and the HTTP capability service on 9000 reported UP.

Import a First Toolset

The fastest way to put a real tool behind Wanaku is to import a toolset. I used the official currency example and listed the catalog against the explicit host so the output is predictable:

wanaku tools import --plain --no-auth --host http://127.0.0.1:8080 \
  https://raw.githubusercontent.com/wanaku-ai/wanaku-toolsets/refs/heads/main/toolsets/currency.json

wanaku tools list --plain --no-auth --host http://127.0.0.1:8080

Look for free-currency-conversion-tool in the output.

That part worked cleanly.

Bridge Another MCP Server into Wanaku

The next step is the one I really wanted to test. I wanted a real external MCP server sitting behind Wanaku so the router was doing more than wrapping HTTP.

For the validated path, I used the official Everything reference server in native SSE mode:

npx -y @modelcontextprotocol/server-everything sse

Keep that terminal open too.

In my run, the server started on port 3001 and logged:

Starting SSE server...
Server is running on port 3001

Now we need a namespace ID for the forward. This is one of the rough edges in the current CLI: wanaku forwards add wants a namespace ID. The easiest reproducible way I found was to reuse one of the preallocated local namespaces.

Extract the ID behind the ns-1 path:

WANAKU_NAMESPACE_ID=$(
  wanaku namespaces list --plain --no-auth --host http://127.0.0.1:8080 \
    | awk '$2 ~ /ns-1\\/mcp\\/sse$/ {print $1; exit}'
)

printf '%s\n' "$WANAKU_NAMESPACE_ID"

Add the forward:

wanaku forwards add --plain --no-auth --host http://127.0.0.1:8080 \
  -N "$WANAKU_NAMESPACE_ID" \
  --service="http://127.0.0.1:3001/sse" \
  --name local-everything-sse

Then verify both the forward and the expanded tool catalog:

wanaku forwards list --plain --no-auth --host http://127.0.0.1:8080
wanaku tools list --plain --no-auth --host http://127.0.0.1:8080

In my run, wanaku forwards list showed:

name                 address
local-everything-sse http://127.0.0.1:3001/sse

And wanaku tools list showed the imported currency tool plus remote MCP tools such as:

• echo

• get-sum

• get-env

• get-tiny-image

• trigger-long-running-operation

• simulate-research-query

That is the fully working bridge: one Wanaku router, one local HTTP toolset, one external SSE MCP server, one merged catalog.

What This Proves

At this point, we have all up and running.

• Wanaku starts locally and serves a healthy router endpoint

• Wanaku imports a normal HTTP toolset cleanly

• Wanaku connects to a separate MCP server over SSE

• Wanaku imports remote MCP tools into its own catalog under a namespace

That is enough to show the routing model is real.

What Broke in My Run

Lessons learned so you don’t have to.

Filesystem Server Through Supergateway

I started with this bridge:

npx -y supergateway \
  --stdio "npx -y @modelcontextprotocol/server-filesystem $HOME/wanaku-demo" \
  --port 8001 \
  --baseUrl http://127.0.0.1:8001

That looks reasonable. It did not survive the live run.

What happened:

• Wanaku connected to the upstream SSE endpoint

• The filesystem server asked for roots/list

• Wanaku replied with an empty roots list

• The filesystem server exited with No valid root directories provided by client

• On the next connection attempt, Supergateway crashed with Already connected to a transport

So this did not happen for this article.

Memory Server Through Supergateway

I also tried the official memory server behind Supergateway:

npx -y supergateway \
  --stdio "npx -y @modelcontextprotocol/server-memory" \
  --port 8002 \
  --baseUrl http://127.0.0.1:8002

That failed differently. Wanaku probed resources/list, the server replied with Method not found, and Wanaku treated the upstream as unavailable.

That tells me the current forward path is more demanding than “any MCP server behind an SSE bridge.” I have not looked into this deeper to figure out who’s fault this is. But maybe I find some time and find out more.

The Primitives, With the Current Rough Edges

The original idea I had for this article wanted to show tools and forwards as if they were equally smooth in local mode. They were not.

Tools

Tools are the cleanest part of the current local flow. The imported currency toolset worked immediately:

wanaku tools import --plain --no-auth --host http://127.0.0.1:8080 \
  https://raw.githubusercontent.com/wanaku-ai/wanaku-toolsets/refs/heads/main/toolsets/currency.json

If your backend already has an OpenAPI document, the next command to look at is:

wanaku tools generate http://petstore3.swagger.io/api/v3/openapi.yaml

That is where Wanaku stops looking like a demo wrapper and starts looking like a migration path.

Forwards

Forwards are the part that gives Wanaku real architectural value. They also have the sharpest edges right now.

The native SSE bridge to the Everything server worked:

wanaku forwards add --plain --no-auth --host http://127.0.0.1:8080 \
  -N "$WANAKU_NAMESPACE_ID" \
  --service="http://127.0.0.1:3001/sse" \
  --name local-everything-sse

The stdio-to-SSE bridges I tried were much less forgiving.

Why This Architecture Still Makes Sense

Even with the rough edges, the core idea still holds up.

Your agent does not need direct knowledge of every backend, every tool definition, every upstream MCP server, and every transport quirk. It needs one stable place to connect. Wanaku turns the rest into catalog management. Add a tool. Forward another MCP server. Keep the client side simpler.

The Apache Camel base matters too. Today the working example is an HTTP tool plus an upstream SSE MCP server. Tomorrow it can be a real Camel route, a Salesforce integration, or an internal API you already describe with OpenAPI. The routing model still makes sense even when a few local-mode features are rough.

What to Try Next

• Stay on Java 21 for now unless you want to carry the Java 25 JVM workaround.

• Start with an upstream server that already speaks SSE if you want the cleanest first forward.

• Treat stdio-to-SSE bridges as something to validate case by case, especially when the upstream server depends on roots or does not implement resources/list.

• Revisit file resources and prompt registration after Wanaku local mode exposes the right providers and the current prompt persistence error is fixed.

Subscribe now

The details change from company to company. Sometimes it is a custom framework that exists because one person got bored with the boring one. Sometimes it is a build system with three hidden assumptions and one person who knows the ritual. Sometimes it is a beautiful abstraction that looks like serious engineering until the first bug report arrives and nobody can explain how data moves through it.

Smart was fine. Smart helped.

The failure was unshared design.

The architecture lived in one head. The code fit the person writing it, not the people who had to change it later. Code review stopped helping because nobody wanted to say they did not understand the impressive part. Eventually the person left, changed teams, lost interest, or became too busy to explain the system they had created.

Then the team discovered the actual architecture document: production.

AI agents are making that old problem cheap, fast, and repeatable.

As I have written in AI Coding in Real Systems: Code Is Cheap. Software Isn’t, the code arrives faster now. The ownership, verification, and integration costs still arrive on the team’s side of the ledger.

The New Rockstar Forgets Yesterday

A human rockstar, for all the damage one can do, at least has continuity. They remember the argument that produced the abstraction. They remember why the service boundary landed there. They remember which bug made them add the defensive layer everyone else now has to step around.

A continuous mind sat behind the mess. That did not make the code maintainable, but it at least gave the team one place to ask why the system looked like that.

An agent session has no durable ownership unless the team builds it. It can generate a clever solution in the morning, generate a different style of clever solution in the afternoon, and then confidently refactor both tomorrow from a partial view of the repository.

The old rockstar developer created a private architecture.

The AI version creates a series of private architectures, one chat at a time.

Each individual change can look reasonable. The tests pass. The code compiles. The explanation sounds calm. The agent used a known pattern, maybe even a good one. But the system slowly collects incompatible ideas:

• two ways to model errors

• three ways to configure clients

• a new helper layer around a framework that already had the helper

• a validation style from one tutorial

• a reactive pattern in one endpoint because the model saw it somewhere

• a security check that is correct locally and meaningless in the real request path

None of those has to be catastrophic alone. That is why the failure mode is hard to see early. The cost shows up when the next developer has to change the system and finds five small architectures pretending to be one application.

This is not a model-quality complaint per se. Better models will reduce some of the local mistakes. They will not remove the need for a coherent engineering style.

That is also why broad “language support” claims are not the solution here. What Code LLM Vendors Usually Mean by “Language Support” splits that into a ladder: syntax is one thing, framework fit is another, and workflow support is where serious teams start caring.

Consistency Is Operational Memory

Developers like to talk about consistency as if it were polish. Naming, layout, dependency choices, exception patterns, logging, tests, configuration, HTTP client usage. These sound like small matters until a team is under pressure.

Under pressure, consistency becomes operational memory.

If every Quarkus resource handles failures the same way, you know where to look. If every service uses typed configuration the same way, you know how a setting reaches runtime. If tests follow a predictable shape, you can add one without rediscovering the local habits. If logging uses one library and one style, incident review is less annoying than it already has to be.

The boring shape of a codebase is part of how a team thinks together.

AI agents do not naturally respect that shape. They respect the context they see, the instructions they receive, the examples closest to the current task, and the statistical gravity of all the code they were trained on. That means they can be fast, helpful, and slightly too willing to solve the problem they inferred instead of the problem this system has.

• If the repo has no visible rules, the agent invents a style.

• If the prompt has no boundaries, the agent expands the task.

• If the review only checks whether the output works, the architecture drifts.

That is not the agent being malicious or stupid. It is doing what we asked badly enough.

The Agent Is Not a Teammate

I use agents. I like them. Hand-written code has no moral purity here. Plenty of terrible code was typed by humans with full confidence and a mechanical keyboard.

But teams get into trouble when they treat the agent like a teammate and then forget the parts of teamwork that make software survivable.

A teammate carries memory. A teammate can be responsible for a design after the merge. A teammate can sit in the incident review and explain the trade-off. A teammate can learn the local codebase over years and internalize what the team does not want repeated.

An agent can help with all of that, but it does not own any of it.

Ownership still lands on the humans.

That changes how I want teams to use these tools. The agent should not act like a replacement senior engineer with unlimited permission to reshape the codebase. It should work inside clear boundaries:

• clear task scope

• visible local conventions

• small changes

• tests that describe the expected behavior

• review that checks integration, not only syntax

• a human who can explain why the final version belongs in this system

That slows the tool down enough for the team to stay with it. Some slowness is not waste. Some slowness is the part where understanding catches up with output.

The Senior Job Moves Toward Taste and Control

Senior engineering judgment moves upstream and sideways in the agent era.

Less time typing boilerplate can be good. I am happy to spend less of my life writing the tenth shape of the same DTO mapping. But if implementation gets cheaper, the important question changes from “can we build it?” to “what shape should it have, and how do we keep that shape over time?”

That is architecture. Not the slide-deck version. The daily version.

• Which framework feature should we use before adding our own layer?

• Which abstraction is too clever for this problem?

• Which generated test proves behavior, and which one only proves that the mock was configured?

• Which local convention should be written down because the agent keeps missing it?

• Which change should be split because the diff is now too large for meaningful review?

That last question has its own version in Give Coding Agents a Change Budget. A change can be technically valid and still too large for a human reviewer to own in one pass.

That work can feel less glamorous than watching a tool generate a full feature in one pass. It is also the work that keeps a team from drowning in plausible code.

AI raises the value of taste because it lowers the cost of producing tasteless volume.

Taste here does not mean personal preference with seniority attached to it. It means choosing the plain thing that fits the system, rejecting the impressive thing that does not, and explaining the difference without turning review into a debate club.

Code Nobody Understands Is Still Debt

One of the more dangerous habits in agent-assisted development is accepting code because the agent can explain it.

That is a low bar.

The agent can explain almost anything after the fact. The explanation may even be accurate. But a generated explanation is not the same as team understanding. If the humans cannot maintain the code without reopening the chat and asking the same system to interpret itself, the team has taken on a new dependency.

Sometimes that dependency is fine. We already depend on tools. We depend on compilers, IDEs, search, docs, static analysis, and people with better memory than ours. But depending on an agent to make sense of architecture the team never understood is a bad trade.

It creates a codebase that can only be maintained through the same mechanism that made it confusing. A team can live there for a while, but it is an expensive place to maintain software.

This is also where The Spec Trap comes back in. If the intent never became concrete enough for a human or a test to check, the implementation can pass the written prompt and still miss the system.

The healthier pattern is simple: use the agent to accelerate work the team can still own.

Let it draft the boring parts. Let it find similar code. Let it write the first test. Let it propose a refactor. Let it explain an unfamiliar dependency. Let it handle mechanical edits where the rules are clear. You can even brainstorm implementation ideas and evaluate their pros- and cons.

Then slow down at the boundary where the code changes the architecture, the runtime behavior, the security model, or the team’s future review burden.

That boundary is where the rockstar problem starts.

Write Down the Local Rules

Banning agents would age badly and waste a lot of useful leverage.

The practical answer is to make the team’s engineering taste visible enough that an agent can work inside it.

That means plain files with plain names. AGENTS.md. CONTRIBUTING.md. Local coding rules. Architecture decision records when the decision matters. Examples of the pattern you want copied. Tests that fail when the agent improvises in the wrong place. Small prompts that tell the agent the shape of the task before it starts inventing a better world.

This can look like process until it saves a review. It is context engineering for codebases. The more operational version of that idea is AGENTS.md: Give Your Coding Agent an Operating Manual.

If you do not give the agent a local operating model, it will bring one from somewhere else. Usually several. Usually with unjustified confidence.

The same applies to review. Generated code needs a fit check before it gets a cleverness check. The first question is whether the change belongs.

• Does it follow the framework’s standard path?

• Does it use the same configuration model as the rest of the service?

• Does it add an abstraction because the system needs one, or because the model has seen that shape often?

• Can the next developer change this without asking an LLM to narrate the maze?

If the answer is no, the code still needs work, even if every test is green.

For the repository side of this, I would pair that operating manual with Coding Agent Guardrails: Treat Every Commit as Untrusted. Prompt discipline helps, but the repo still needs hooks, CI, protected paths, and merge boundaries that can say no.

Keep the Code Boring Enough to Survive

The old rockstar developer left behind code that made the rest of the team feel slower and less informed.

The AI version can do the same thing at a much larger scale. It can make everyone feel productive while the codebase becomes harder to reason about. It can make architecture drift look like velocity. It can make review feel unnecessary because the output is clean, formatted, and accompanied by a confident explanation.

That is the trap.

Generated code should still be team code. It should follow the same conventions, carry the same tests, fit the same architecture, and be understood by the people who will wake up when it breaks.

Agents are useful. Speed is useful. Fast drafts are useful.

That is the part the rockstar developer never optimized for.

It is the part we cannot outsource now.

Subscribe now

Have we seen this failure before?

The answer is often somewhere in an old incident report, a Slack thread, a ticket, or a runbook. The text is there. The problem is shape. A new stack trace does not match the old one character for character. The service moved packages, the line numbers changed, and the error message gained one extra adjective because software enjoys small acts of drama.

Keyword search helps when the same words repeat. It misses cases where the failure looks similar but the text is not identical. Full AI retrieval can help too, but it brings model keys, embedding providers, latency, and a different operational story. Also we all are tired of all of this somedays. So for this article I want the smaller tool: turn a Java incident into a vector, store it in Qdrant, and ask for the nearest old failures.

No LLM is required or even used. We use deterministic feature hashing in plain Java. That makes the vectors inspectable and repeatable. Qdrant gives us similarity search, payload filters, Dev Services, and a normal Quarkus client.

What We Build

Incident Vector Search is a Quarkus service with three endpoints:

• POST /incidents/seed loads five example incidents

• POST /incidents indexes one incident

• POST /incidents/search searches for similar incidents

The app stores each incident as a Qdrant point: one vector plus payload. The vector captures the rough shape of the failure. The payload keeps the fields humans still care about: service, environment, exception type, message, resolution, and incident URL.

Two Qdrant details matter early:

• A collection has a fixed vector size and distance metric

• A point ID must be a 64-bit unsigned integer or a UUID

That second detail is easy to miss. Our business IDs look like INC-1001, so we keep those in payload and use deterministic UUIDs as Qdrant point IDs.

What You Need

You need a normal Quarkus setup and a container runtime for Qdrant Dev Services.

• JDK 25

• Quarkus CLI

• Podman or another Docker-compatible container runtime

• curl and jq

• About two ☕️

This article uses Quarkus 3.37.0, Java 25, io.quarkiverse.qdrant:quarkus-qdrant:0.1.0, and Qdrant v1.18-unprivileged for Dev Services.

Create The Project

Create the base app and follow along or grab the repository from my Github:

quarkus create app dev.mainthread:incident-vector-search \
  --extension=rest-jackson,smallrye-health \
  --java=25 \
  --no-code

Use these extensions:

• quarkus-rest-jackson: JSON REST endpoints

• quarkus-smallrye-health: readiness checks, including the Qdrant extension check

Add Qdrant and validation to pom.xml:

<dependency>
    <groupId>io.quarkiverse.qdrant</groupId>
    <artifactId>quarkus-qdrant</artifactId>
    <version>0.1.0</version>
</dependency>
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-hibernate-validator</artifactId>
</dependency>

quarkus-qdrant gives us an injectable QdrantClient. Hibernate Validator keeps bad incident documents out of the application code path. A missing stack trace should fail as HTTP 400, not as a weak vector that pollutes the index.

Configure Qdrant Dev Services

Create src/main/resources/application.properties:

quarkus.qdrant.devservices.qdrant-image-name=docker.io/qdrant/qdrant:v1.18-unprivileged
quarkus.qdrant.devservices.collections.incidents.vector-size=384
quarkus.qdrant.devservices.collections.incidents.distance=Cosine

quarkus.smallrye-health.root-path=/q/health

Qdrant starts automatically in dev and test mode. The extension also creates the incidents collection before the app starts.

vector-size=384 is arbitrary but fixed. Every vector we write must have exactly 384 values. Change it later and old points no longer fit the collection. That is not a warning for Qdrant specifically; it is how vector collections work.

distance=Cosine compares direction more than magnitude. Since we normalize every vector to length one, cosine similarity is a good fit for this example.

Define The Incident Shape

Start with one request record:

package dev.mainthread.incidents;

import java.util.List;

import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotEmpty;
import jakarta.validation.constraints.Size;

public record IncidentInput(
        String id,
        @NotBlank String service,
        @NotBlank String environment,
        @NotBlank String exceptionType,
        @NotBlank String message,
        @NotEmpty @Size(max = 20) List<@NotBlank String> stackTrace,
        String resolvedBy,
        String incidentUrl) {
}

There is no database entity here. Qdrant stores the vector and payload. The record is the HTTP boundary and the indexing input.

The stack trace limit is intentionally small. We only need the top frames for this kind of matching. If you let clients send 5,000 frames, they will. Then you get worse vectors and a free denial-of-service shape. Very generous of them.

The search request wraps an incident and adds search controls:

package dev.mainthread.incidents;

import jakarta.validation.Valid;
import jakarta.validation.constraints.DecimalMax;
import jakarta.validation.constraints.DecimalMin;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotNull;

public record SimilarIncidentRequest(
        @NotNull @Valid IncidentInput incident,
        @Min(1) @Max(20) Integer limit,
        @DecimalMin("0.0") @DecimalMax("1.0") Float minScore,
        String filterService,
        String filterEnvironment,
        Boolean onlyResolved) {
}

The filters are not part of the vector. They stay explicit because they are business constraints. A checkout failure may look like a billing failure at the stack-trace level. If the operator asks for checkout incidents in prod, Qdrant should only search that slice.

Add the response records too:

package dev.mainthread.incidents;

public record IndexedIncident(String id, String pointId, String collection, int dimensions) {
}

package dev.mainthread.incidents;

public record IncidentMatch(
        String id,
        float score,
        String service,
        String environment,
        String exceptionType,
        String message,
        String resolvedBy,
        String incidentUrl) {
}

package dev.mainthread.incidents;

import java.util.List;

public record SearchResponse(int count, List<IncidentMatch> matches) {
}

package dev.mainthread.incidents;

import java.util.List;

public record SeedResponse(int indexed, List<String> ids) {
}

Now the boundary is clear. The API speaks in incident IDs like INC-1001; Qdrant receives UUID point IDs behind the service.

Turn Incidents Into Vectors

Feature hashing is the small trick in this app. We take tokens from the incident, hash each token into one of 384 buckets, add weighted counts, and normalize the result.

This is not a semantic embedding. It will not learn that “database pool exhausted” and “Agroal acquisition timeout” mean related things unless the tokens overlap. That is fine for this article. We want a simple vectorizer where every score is explainable from the input.

Create IncidentVectorizer:

package dev.mainthread.incidents;

import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
class IncidentVectorizer {

    static final int DIMENSIONS = 384;

    private static final Pattern TOKEN = Pattern.compile("[a-z0-9]+");

    List<Float> vectorForPoint(IncidentInput incident) {
        float[] values = vectorForSearch(incident);
        List<Float> boxed = new ArrayList<>(values.length);
        for (float value : values) {
            boxed.add(value);
        }
        return boxed;
    }

    float[] vectorForSearch(IncidentInput incident) {
        float[] vector = new float[DIMENSIONS];
        addWeighted(vector, "service:" + incident.service(), 5);
        addWeighted(vector, "environment:" + incident.environment(), 2);
        addWeighted(vector, "exception:" + incident.exceptionType(), 6);
        addText(vector, incident.exceptionType(), 3);
        addText(vector, incident.message(), 2);

        for (String frame : incident.stackTrace()) {
            addWeighted(vector, "frame:" + frame, 3);
            addText(vector, frame, 1);
        }

        normalize(vector);
        return vector;
    }

    double cosine(float[] left, float[] right) {
        double dot = 0;
        double leftNorm = 0;
        double rightNorm = 0;
        for (int i = 0; i < left.length; i++) {
            dot += left[i] * right[i];
            leftNorm += left[i] * left[i];
            rightNorm += right[i] * right[i];
        }
        if (leftNorm == 0 || rightNorm == 0) {
            return 0;
        }
        return dot / (Math.sqrt(leftNorm) * Math.sqrt(rightNorm));
    }

    private void addText(float[] vector, String text, int weight) {
        if (text == null || text.isBlank()) {
            return;
        }

        Matcher matcher = TOKEN.matcher(text.toLowerCase(Locale.ROOT));
        while (matcher.find()) {
            addWeighted(vector, matcher.group(), weight);
        }
    }

    private void addWeighted(float[] vector, String rawToken, int weight) {
        if (rawToken == null || rawToken.isBlank()) {
            return;
        }

        String token = rawToken.toLowerCase(Locale.ROOT).trim();
        int bucket = Math.floorMod(token.hashCode(), vector.length);
        vector[bucket] += weight;
    }

    private void normalize(float[] vector) {
        double length = 0;
        for (float value : vector) {
            length += value * value;
        }
        if (length == 0) {
            return;
        }

        float norm = (float) Math.sqrt(length);
        for (int i = 0; i < vector.length; i++) {
            vector[i] = vector[i] / norm;
        }
    }
}

Notice the weights. exceptionType and exact top frames matter more than general message words. service matters too, but we still keep service as a filter later. The vector says “these incidents have a similar failure shape.” The filter says “only search the operational slice I asked for.”

Before you continue, predict one thing: what happens if we remove normalization?

Longer stack traces start to score higher because they add more token counts. Normalization keeps the vector length stable, so the score is about overlap and direction rather than document size.

Add Seed Incidents

Create a small fixture class. These are realistic enough to make the first search interesting:

package dev.mainthread.incidents;

import java.util.List;

final class IncidentFixtures {

    private IncidentFixtures() {
    }

    static List<IncidentInput> examples() {
        return List.of(
                new IncidentInput(
                        "INC-1001",
                        "checkout-service",
                        "prod",
                        "java.lang.NullPointerException",
                        "Cannot invoke DiscountPolicy.percentage because policy is null while applying coupon",
                        List.of(
                                "dev.mainthread.checkout.CartPriceCalculator.applyDiscount(CartPriceCalculator.java:84)",
                                "dev.mainthread.checkout.CheckoutService.priceCart(CheckoutService.java:47)",
                                "dev.mainthread.checkout.CheckoutResource.pay(CheckoutResource.java:31)"),
                        "Guard missing coupon policy before discount calculation",
                        "https://runbooks.example.com/incidents/INC-1001"),
                new IncidentInput(
                        "INC-1002",
                        "billing-service",
                        "prod",
                        "java.sql.SQLTransientConnectionException",
                        "Timed out waiting for database connection from the invoice pool",
                        List.of(
                                "dev.mainthread.billing.InvoiceRepository.findOpenInvoices(InvoiceRepository.java:118)",
                                "dev.mainthread.billing.InvoiceBatch.closeCurrentPeriod(InvoiceBatch.java:55)",
                                "io.agroal.pool.ConnectionPool.handlerFromSharedCache(ConnectionPool.java:321)"),
                        "Increase pool timeout and split invoice close job into smaller batches",
                        "https://runbooks.example.com/incidents/INC-1002"),
                new IncidentInput(
                        "INC-1003",
                        "checkout-service",
                        "prod",
                        "java.util.concurrent.TimeoutException",
                        "Tax service call exceeded the 800 ms client timeout during payment authorization",
                        List.of(
                                "dev.mainthread.checkout.TaxClient.calculate(TaxClient.java:62)",
                                "dev.mainthread.checkout.CheckoutService.priceCart(CheckoutService.java:52)",
                                "dev.mainthread.checkout.CheckoutResource.pay(CheckoutResource.java:31)"),
                        "Cache tax jurisdiction lookups and fail payment before authorization",
                        "https://runbooks.example.com/incidents/INC-1003"),
                new IncidentInput(
                        "INC-1004",
                        "inventory-service",
                        "staging",
                        "jakarta.persistence.OptimisticLockException",
                        "Concurrent stock reservation updated the same SKU version",
                        List.of(
                                "dev.mainthread.inventory.StockRepository.reserve(StockRepository.java:77)",
                                "dev.mainthread.inventory.ReservationService.reserveCart(ReservationService.java:41)",
                                "dev.mainthread.inventory.InventoryResource.reserve(InventoryResource.java:29)"),
                        "Retry stock reservation once and return conflict after the second collision",
                        "https://runbooks.example.com/incidents/INC-1004"),
                new IncidentInput(
                        "INC-1005",
                        "search-service",
                        "prod",
                        "com.fasterxml.jackson.databind.JsonMappingException",
                        "Cannot deserialize shipment filter because status contains an unknown enum value",
                        List.of(
                                "dev.mainthread.search.ShipmentSearchResource.search(ShipmentSearchResource.java:37)",
                                "com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3895)",
                                "io.quarkus.resteasy.reactive.jackson.runtime.serialisers.ServerJacksonMessageBodyReader.readFrom(ServerJacksonMessageBodyReader.java:92)"),
                        "Reject unknown status values at the API boundary",
                        "https://runbooks.example.com/incidents/INC-1005"));
    }
}

The sample set has two checkout incidents. One is a NullPointerException, the other is a timeout. That gives the search a real choice: same service is not enough. Similar failure shape should win.

Store Incidents In Qdrant

Now wrap the Qdrant client in one application service:

package dev.mainthread.incidents;

import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;

import io.quarkiverse.qdrant.runtime.QdrantClient;
import io.quarkiverse.qdrant.runtime.model.PointStruct;
import io.quarkiverse.qdrant.runtime.model.ScoredPoint;

@ApplicationScoped
class IncidentArchive {

    private static final String COLLECTION = "incidents";
    private static final int DEFAULT_LIMIT = 5;
    private static final float DEFAULT_MIN_SCORE = 0.68f;

    private final QdrantClient qdrant;
    private final IncidentVectorizer vectorizer;
    private volatile boolean collectionChecked;

    @Inject
    IncidentArchive(QdrantClient qdrant, IncidentVectorizer vectorizer) {
        this.qdrant = qdrant;
        this.vectorizer = vectorizer;
    }

    IndexedIncident index(IncidentInput incident) {
        ensureCollection();
        String id = incidentId(incident);
        String pointId = pointIdFor(incident);
        qdrant.upsert(COLLECTION)
                .point(new PointStruct(pointId, vectorizer.vectorForPoint(incident), payloadFor(id, incident)))
                .execute();
        return new IndexedIncident(id, pointId, COLLECTION, IncidentVectorizer.DIMENSIONS);
    }

    SeedResponse seed(List<IncidentInput> incidents) {
        ensureCollection();
        List<String> ids = new ArrayList<>(incidents.size());
        List<PointStruct> points = new ArrayList<>(incidents.size());

        for (IncidentInput incident : incidents) {
            String id = incidentId(incident);
            String pointId = pointIdFor(incident);
            ids.add(id);
            points.add(new PointStruct(pointId, vectorizer.vectorForPoint(incident), payloadFor(id, incident)));
        }

        qdrant.upsert(COLLECTION).points(points).execute();
        return new SeedResponse(points.size(), ids);
    }

    SearchResponse search(SimilarIncidentRequest request) {
        ensureCollection();
        int limit = request.limit() == null ? DEFAULT_LIMIT : request.limit();
        float minScore = request.minScore() == null ? DEFAULT_MIN_SCORE : request.minScore();

        List<ScoredPoint> points = qdrant.search(COLLECTION)
                .vector(vectorizer.vectorForSearch(request.incident()))
                .limit(limit)
                .scoreThreshold(minScore)
                .withPayload(true)
                .withVector(false)
                .filter(filterFor(request))
                .execute();

        List<IncidentMatch> matches = points.stream()
                .map(IncidentArchive::matchFrom)
                .toList();
        return new SearchResponse(matches.size(), matches);
    }

    private synchronized void ensureCollection() {
        if (collectionChecked) {
            return;
        }
        if (!qdrant.listCollections().contains(COLLECTION)) {
            qdrant.createCollection(COLLECTION)
                    .vectorSize(IncidentVectorizer.DIMENSIONS)
                    .distance("Cosine")
                    .execute();
        }
        collectionChecked = true;
    }

    private static String incidentId(IncidentInput incident) {
        if (incident.id() != null && !incident.id().isBlank()) {
            return incident.id();
        }
        return pointIdFor(incident);
    }

    private static String pointIdFor(IncidentInput incident) {
        String source = incident.service() + "|" + incident.environment() + "|" + incident.exceptionType() + "|"
                + incident.message() + "|" + incident.stackTrace();
        if (incident.id() != null && !incident.id().isBlank()) {
            source = incident.id() + "|" + source;
        }
        return UUID.nameUUIDFromBytes(source.getBytes(StandardCharsets.UTF_8)).toString();
    }

    private static Map<String, Object> payloadFor(String id, IncidentInput incident) {
        Map<String, Object> payload = new LinkedHashMap<>();
        payload.put("id", id);
        payload.put("pointId", pointIdFor(incident));
        payload.put("service", incident.service());
        payload.put("environment", incident.environment());
        payload.put("exceptionType", incident.exceptionType());
        payload.put("message", incident.message());
        payload.put("stackTrace", incident.stackTrace());
        payload.put("resolved", incident.resolvedBy() != null && !incident.resolvedBy().isBlank());
        putIfPresent(payload, "resolvedBy", incident.resolvedBy());
        putIfPresent(payload, "incidentUrl", incident.incidentUrl());
        return payload;
    }

    private static Map<String, Object> filterFor(SimilarIncidentRequest request) {
        List<Map<String, Object>> must = new ArrayList<>();
        addMatch(must, "service", request.filterService());
        addMatch(must, "environment", request.filterEnvironment());
        if (Boolean.TRUE.equals(request.onlyResolved())) {
            addMatch(must, "resolved", true);
        }
        if (must.isEmpty()) {
            return null;
        }
        return Map.of("must", must);
    }

    private static void addMatch(List<Map<String, Object>> must, String key, Object value) {
        if (value == null) {
            return;
        }
        if (value instanceof String text && text.isBlank()) {
            return;
        }
        must.add(Map.of("key", key, "match", Map.of("value", value)));
    }

    private static void putIfPresent(Map<String, Object> payload, String key, String value) {
        if (value != null && !value.isBlank()) {
            payload.put(key, value);
        }
    }

    private static IncidentMatch matchFrom(ScoredPoint point) {
        Map<String, Object> payload = point.getPayload();
        return new IncidentMatch(
                valueOrFallback(payload, "id", point.getId()),
                point.getScore(),
                stringValue(payload, "service"),
                stringValue(payload, "environment"),
                stringValue(payload, "exceptionType"),
                stringValue(payload, "message"),
                stringValue(payload, "resolvedBy"),
                stringValue(payload, "incidentUrl"));
    }

    private static String stringValue(Map<String, Object> payload, String key) {
        Object value = payload == null ? null : payload.get(key);
        return value == null ? null : value.toString();
    }

    private static String valueOrFallback(Map<String, Object> payload, String key, String fallback) {
        String value = stringValue(payload, key);
        return value == null ? fallback : value;
    }
}

There are three decisions in this class worth keeping visible.

First, ensureCollection() checks the collection once. Dev Services already creates it for local runs, but the app can still create it when pointed at a fresh Qdrant instance. In production I would move collection creation to provisioning or migration code. Runtime creation is convenient for this hands-on path, not a full operations plan.

Second, pointIdFor() generates a deterministic UUID. Qdrant point loading is idempotent, so re-seeding the same incident overwrites the same point. That is exactly what you want when the input comes from a queue or retrying job.

Third, filterFor() builds Qdrant’s payload filter as a Map. The Quarkiverse client gives us a fluent search API and accepts raw filter JSON shape. For this article that is enough: match service, environment, and resolved status.

Expose The REST Endpoints

The resource stays thin:

package dev.mainthread.incidents;

import jakarta.inject.Inject;
import jakarta.validation.Valid;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/incidents")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class IncidentResource {

    private final IncidentArchive archive;

    @Inject
    IncidentResource(IncidentArchive archive) {
        this.archive = archive;
    }

    @POST
    public IndexedIncident index(@Valid IncidentInput incident) {
        return archive.index(incident);
    }

    @POST
    @Path("/search")
    public SearchResponse search(@Valid SimilarIncidentRequest request) {
        return archive.search(request);
    }

    @POST
    @Path("/seed")
    @Consumes(MediaType.WILDCARD)
    public SeedResponse seed() {
        return archive.seed(IncidentFixtures.examples());
    }
}

@Valid is the useful part here. It makes Quarkus reject incomplete incident documents before the vectorizer sees them. The seed endpoint accepts an empty POST because it has no request body. Without the method-level @Consumes(MediaType.WILDCARD), some clients send a default form content type and get a 415. That is a small test-discovered annoyance, so we remove it.

Run It

Start dev mode:

./mvnw quarkus:dev

On first run, Qdrant Dev Services pulls the Qdrant image and starts a container. After the app starts, seed the archive:

curl -s -X POST http://localhost:8080/incidents/seed | jq 

Expected output:

{
  "ids": [
    "INC-1001",
    "INC-1002",
    "INC-1003",
    "INC-1004",
    "INC-1005"
  ],
  "indexed": 5
}

Now search with a new checkout failure:

curl -s -X POST http://localhost:8080/incidents/search \
  -H "Content-Type: application/json" \
  -d '{
    "incident": {
      "service": "checkout-service",
      "environment": "prod",
      "exceptionType": "java.lang.NullPointerException",
      "message": "Cannot invoke DiscountPolicy.percentage because policy is null while pricing cart",
      "stackTrace": [
        "dev.mainthread.checkout.CartPriceCalculator.applyDiscount(CartPriceCalculator.java:91)",
        "dev.mainthread.checkout.CheckoutService.priceCart(CheckoutService.java:47)",
        "dev.mainthread.checkout.CheckoutResource.pay(CheckoutResource.java:31)"
      ]
    },
    "limit": 3,
    "minScore": 0.60,
    "filterService": "checkout-service",
    "filterEnvironment": "prod",
    "onlyResolved": true
  }' | jq .

Expected shape:

{
  "count": 1,
  "matches": [
    {
      "environment": "prod",
      "exceptionType": "java.lang.NullPointerException",
      "id": "INC-1001",
      "incidentUrl": "https://runbooks.example.com/incidents/INC-1001",
      "message": "Cannot invoke DiscountPolicy.percentage because policy is null while applying coupon",
      "resolvedBy": "Guard missing coupon policy before discount calculation",
      "score": 0.93308544,
      "service": "checkout-service"
    }
  ]
}

The exact score can move if you change the vectorizer weights or sample data. The first ID should stay INC-1001. That is the behavior we care about.

Now try the same search without the service and environment filters. You may still get the right result first, but Qdrant is free to compare against all services. Add the filters back and the search becomes the shape an operator expects: similar incident, same service, same environment, already resolved.

That is the part of Qdrant I like for this use case. Vector search handles fuzzy similarity. Payload filtering keeps operational policy explicit.

Check Readiness

Add the health extension and the Qdrant extension registers a readiness check automatically:

curl -s http://localhost:8080/q/health/ready | jq .

Expected shape:

{
  "status": "UP",
  "checks": [
    {
      "name": "Qdrant REST Client health check",
      "status": "UP"
    }
  ]
}

If Qdrant is down, readiness should fail. That is the right failure mode for this app. Serving incident search without the vector store would give operators an empty answer at the worst possible time.

For an external Qdrant instance, configure the client like this:

quarkus.qdrant.host=qdrant.example.com
quarkus.qdrant.port=6333
quarkus.qdrant.api-key=${QDRANT_API_KEY}
quarkus.qdrant.use-tls=true

Setting quarkus.qdrant.host disables Dev Services. Local runs get a container. Production gets the configured service.

Test The Behavior

The unit test checks the vectorizer before Qdrant enters the picture:

package dev.mainthread.incidents;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.List;

import org.junit.jupiter.api.Test;

class IncidentVectorizerTest {

    private final IncidentVectorizer vectorizer = new IncidentVectorizer();

    @Test
    void similarCheckoutFailuresScoreHigherThanUnrelatedBillingFailures() {
        IncidentInput query = new IncidentInput(
                null,
                "checkout-service",
                "prod",
                "java.lang.NullPointerException",
                "Cannot invoke DiscountPolicy.percentage because policy is null",
                List.of(
                        "dev.mainthread.checkout.CartPriceCalculator.applyDiscount(CartPriceCalculator.java:91)",
                        "dev.mainthread.checkout.CheckoutService.priceCart(CheckoutService.java:47)"),
                null,
                null);

        IncidentInput checkout = IncidentFixtures.examples().get(0);
        IncidentInput billing = IncidentFixtures.examples().get(1);

        double checkoutScore = vectorizer.cosine(vectorizer.vectorForSearch(query), vectorizer.vectorForSearch(checkout));
        double billingScore = vectorizer.cosine(vectorizer.vectorForSearch(query), vectorizer.vectorForSearch(billing));

        assertTrue(checkoutScore > 0.75, "Expected close checkout match but got " + checkoutScore);
        assertTrue(checkoutScore > billingScore + 0.35,
                "Expected checkout score " + checkoutScore + " to beat billing score " + billingScore);
    }

    @Test
    void vectorsUseTheConfiguredCollectionDimension() {
        float[] vector = vectorizer.vectorForSearch(IncidentFixtures.examples().get(0));

        assertEquals(384, vector.length);
    }
}

The resource test starts Quarkus and Qdrant Dev Services:

package dev.mainthread.incidents;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;

@QuarkusTest
class IncidentResourceTest {

    @Test
    void findsSimilarResolvedCheckoutIncident() {
        given()
                .when()
                .post("/incidents/seed")
                .then()
                .statusCode(200)
                .body("indexed", equalTo(5));

        given()
                .contentType("application/json")
                .body("""
                        {
                          "incident": {
                            "service": "checkout-service",
                            "environment": "prod",
                            "exceptionType": "java.lang.NullPointerException",
                            "message": "Cannot invoke DiscountPolicy.percentage because policy is null while pricing cart",
                            "stackTrace": [
                              "dev.mainthread.checkout.CartPriceCalculator.applyDiscount(CartPriceCalculator.java:91)",
                              "dev.mainthread.checkout.CheckoutService.priceCart(CheckoutService.java:47)",
                              "dev.mainthread.checkout.CheckoutResource.pay(CheckoutResource.java:31)"
                            ]
                          },
                          "limit": 3,
                          "minScore": 0.60,
                          "filterService": "checkout-service",
                          "filterEnvironment": "prod",
                          "onlyResolved": true
                        }
                        """)
                .when()
                .post("/incidents/search")
                .then()
                .statusCode(200)
                .body("count", greaterThan(0))
                .body("matches[0].id", equalTo("INC-1001"))
                .body("matches[0].service", equalTo("checkout-service"));
    }

    @Test
    void rejectsIncidentsWithoutStackFrames() {
        given()
                .contentType("application/json")
                .body("""
                        {
                          "service": "checkout-service",
                          "environment": "prod",
                          "exceptionType": "java.lang.NullPointerException",
                          "message": "policy is null",
                          "stackTrace": []
                        }
                        """)
                .when()
                .post("/incidents")
                .then()
                .statusCode(400);
    }
}

Run the tests:

./mvnw test

Expected result:

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

These tests caught two real issues while building the sample: the empty seed POST needed looser content negotiation, and Qdrant point IDs needed UUIDs instead of INC-1001. That is the reason to keep integration tests in a demo. They find the boring edges before readers do.

Production Edges

This app is small, but it creates real production questions.

Vector quality starts with the feature design. Our hash vectorizer is deterministic and cheap, but it only captures token overlap. If you need semantic similarity, swap the vectorizer for an embedding model and keep the IncidentArchive boundary. The rest of the application should not care where the numbers came from.

Collection management should move out of request handling. The ensureCollection() method keeps the tutorial self-contained. Production systems should create collections, payload indexes, and distance settings through provisioning, migrations, or platform automation.

Payload filters need indexes at scale. Qdrant can filter by payload fields directly, and the Qdrant filtering guide recommends payload indexes for fields you filter on often. For this example the dataset has five points. For real incident history, create indexes for fields like service, environment, and resolved.

Scores are application policy. minScore=0.68 is a starting point for this deterministic vectorizer and sample data. Change the vectorizer and you must re-check the threshold. I would treat it like any relevance setting: measure on real incident pairs before trusting it during an outage.

Readiness should fail when Qdrant fails. An empty incident search is worse than a clear dependency failure. If your fallback is “show keyword-only results,” make that explicit and observable. Do not let a broken vector store look like no matching incidents exist.

Conclusion

We built a Quarkus service that stores Java incident reports as Qdrant vectors, searches by failure shape, and still keeps service and environment constraints explicit through payload filters. Just remember: vectors do not have to mean “AI app.” Sometimes they are just a practical index for data that is similar without being equal.

Subscribe now

ThreadLocal is the usual shortcut. It works until the request flow changes. A filter runs too late, a background task forgets cleanup, or a refactor crosses a boundary where the tenant should have been explicit.

Quarkus already covers two big pieces. OIDC multitenancy picks the right authentication tenant. Hibernate ORM multitenancy isolates data. The missing part is the boring middle: resolve one tenant id from the request, keep it in request state, and let the rest of the app stop caring.

The preview Quarkus Multitenancy extension fills that gap. It gives you a small resolution API, a request-scoped TenantContext, HTTP resolvers for headers, JWT claims, cookies, and path segments, plus an ORM bridge that lets Hibernate route by the tenant already in that context.

We build a small fictional SaaS service called VaultBoard. It has two tenants, acme and globex, each backed by its own PostgreSQL database. We expose one REST API, create records for both tenants through the same endpoint, and prove that each tenant only sees its own data. After that, we switch the same app to JWT resolution and then add a custom host-based resolver.

What we build

• A Quarkus REST API with GET /api/dashboards, POST /api/dashboards, and GET /api/dashboards/tenant

• Two tenant databases, acme and globex, started automatically by Quarkus Dev Services

• Header-based tenant resolution with X-Tenant

• Tenant-aware Hibernate ORM routing with no tenantId parameter in the service layer

• A JWT-based tenant slice verified with a dedicated test profile

• A custom host-based resolver verified with a dedicated test profile

What you need

You need a normal Java and Quarkus setup plus a working container runtime. The version set here is Java 25, Quarkus 3.37.0, and Quarkus Multitenancy 0.1.0.

• JDK 25

• Quarkus CLI

• Docker or Podman

• curl

• About 45 minutes

Create the project

Use the CLI to create the application without starter code and follow along or grab the code from my Github repository:

quarkus create app io.mainthread:vaultboard \
  --package-name=io.mainthread.vaultboard \
  --extension='rest-jackson,hibernate-orm-panache,jdbc-postgresql' \
  --java=25 \
  --no-code

The Quarkus CLI keeps the platform-managed dependencies aligned. The multitenancy extension is outside the platform, so add it and the Flyway extension manually to pom.xml:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-flyway</artifactId>
</dependency>
<dependency>
    <groupId>io.quarkiverse.multitenancy</groupId>
    <artifactId>quarkus-multitenancy-http</artifactId>
    <version>0.1.0</version>
</dependency>
<dependency>
    <groupId>io.quarkiverse.multitenancy</groupId>
    <artifactId>quarkus-multitenancy-orm</artifactId>
    <version>0.1.0</version>
</dependency>

The two multitenancy artifacts handle different concerns:

• quarkus-multitenancy-http resolves the tenant from the incoming request and stores it in TenantContext

• quarkus-multitenancy-orm bridges that resolved tenant into Hibernate ORM

One detail is easy to miss if you saw older examples of this extension. The coordinates are now io.quarkiverse.multitenancy:*, not the earlier pre-Quarkiverse group. That is one good reason to pin the preview version explicitly.

Let Dev Services own the databases

As soon as the named datasources exist and jdbc-postgresql is on the classpath, Quarkus Dev Services starts the PostgreSQL containers for us. No Compose file. No JDBC URLs to configure by hand.

Put this in src/main/resources/application.properties:

quarkus.hibernate-orm.datasource=__bootstrap
quarkus.hibernate-orm.schema-management.strategy=none
quarkus.hibernate-orm.multitenant=DATABASE
quarkus.hibernate-orm.log.sql=true
quarkus.hibernate-orm.validate-in-dev-mode=false

quarkus.datasource.devservices.enabled=false
quarkus.datasource.__bootstrap.db-kind=postgresql
quarkus.datasource.__bootstrap.devservices.db-name=bootstrap

quarkus.datasource.acme.db-kind=postgresql
quarkus.datasource.acme.devservices.db-name=acme

quarkus.datasource.globex.db-kind=postgresql
quarkus.datasource.globex.devservices.db-name=globex

quarkus.flyway.acme.locations=classpath:db/migration
quarkus.flyway.acme.migrate-at-start=true
quarkus.flyway.globex.locations=classpath:db/migration
quarkus.flyway.globex.migrate-at-start=true

quarkus.multi-tenant.http.enabled=true
quarkus.multi-tenant.http.strategy=header
quarkus.multi-tenant.http.header-name=X-Tenant

quarkus.datasource.devservices.enabled=false disables Dev Services only for the unused default datasource. The named datasources still start their own PostgreSQL containers. I set this so Quarkus does not invent a default datasource we do not want.

quarkus.datasource.__bootstrap.* defines the bootstrap datasource. The ORM bridge uses it before a request-scoped tenant exists. If you skip it, Hibernate has no safe datasource to use during startup.

quarkus.multi-tenant.http.strategy=header keeps this slice in header mode. The preview extension still defaults to header,jwt,cookie. If you leave the property unset, jwt stays in the chain, and the app starts asking for JWT verification settings even though this demo is still on headers.

quarkus.flyway.<tenant>.migrate-at-start=true runs Flyway migrations on each tenant datasource when the app boots. Both tenants point at the same classpath:db/migration location because they share the same table structure. Hibernate’s own schema management does not support DATABASE multitenancy, so Flyway handles it instead.

Model the data

Create the entity first:

package io.mainthread.vaultboard.dashboard;

import java.math.BigDecimal;

import io.quarkus.hibernate.orm.panache.PanacheEntity;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Table;

@Entity
@Table(name = "dashboards")
public class Dashboard extends PanacheEntity {

    @Column(nullable = false)
    public String name;

    @Column(name = "owner_email", nullable = false)
    public String ownerEmail;

    @Column(name = "monthly_budget", nullable = false, precision = 12, scale = 2)
    public BigDecimal monthlyBudget;
}

There is no tenant field on the row model. Each tenant gets its own datasource, so the entity stays clean.

Create the request type next:

package io.mainthread.vaultboard.dashboard;

import java.math.BigDecimal;

public record CreateDashboardRequest(
        String name,
        String ownerEmail,
        BigDecimal monthlyBudget) {
}

Keep the service layer clean

DashboardService is small on purpose:

package io.mainthread.vaultboard.dashboard;

import java.util.List;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.transaction.Transactional;

@ApplicationScoped
public class DashboardService {

    public List<Dashboard> listAll() {
        return Dashboard.listAll();
    }

    @Transactional
    public Dashboard create(CreateDashboardRequest request) {
        Dashboard dashboard = new Dashboard();
        dashboard.name = request.name();
        dashboard.ownerEmail = request.ownerEmail();
        dashboard.monthlyBudget = request.monthlyBudget();
        dashboard.persist();
        return dashboard;
    }
}

The service never asks which tenant is active. No tenantId parameter. No context lookup. Hibernate already knows which datasource to use.

The resource follows the same pattern:

package io.mainthread.vaultboard.dashboard;

import java.util.List;
import java.util.Map;

import io.quarkiverse.multitenancy.core.runtime.context.TenantContext;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/api/dashboards")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class DashboardResource {

    private final DashboardService dashboardService;
    private final TenantContext tenantContext;

    public DashboardResource(DashboardService dashboardService, TenantContext tenantContext) {
        this.dashboardService = dashboardService;
        this.tenantContext = tenantContext;
    }

    @GET
    public List<Dashboard> list() {
        return dashboardService.listAll();
    }

    @POST
    public Dashboard create(CreateDashboardRequest request) {
        return dashboardService.create(request);
    }

    @GET
    @Path("/tenant")
    public Map<String, String> currentTenant() {
        return Map.of("tenant", tenantContext.getTenantId().orElse("missing"));
    }
}

/tenant is a cheap probe. It shows what the resolver stored before we involve the database.

Migrate the tenant schemas with Flyway

Hibernate’s built-in schema management (drop-and-create, update) does not run against tenant datasources in DATABASE mode. It only targets the single datasource Hibernate is pointed at, which is __bootstrap here. The Quarkus documentation recommends Flyway per named datasource instead.

Both tenants share the same table structure, so one migration file is enough. Create src/main/resources/db/migration/V1__create_dashboards.sql:

CREATE SEQUENCE IF NOT EXISTS dashboards_SEQ START WITH 1 INCREMENT BY 50;

CREATE TABLE IF NOT EXISTS dashboards (
    monthly_budget NUMERIC(12,2) NOT NULL,
    id BIGINT NOT NULL,
    name VARCHAR(255) NOT NULL,
    owner_email VARCHAR(255) NOT NULL,
    PRIMARY KEY (id)
);

The Flyway properties we added earlier point both acme and globex at this same classpath:db/migration location and run the migration on startup. No Java code needed.

Run the header-based slice

The first worked slice should prove the full path: resolve the tenant from a header, store it in TenantContext, route Hibernate to the right datasource, and keep each tenant’s rows separate.

Start the app in one terminal:

./mvnw quarkus:dev

On the first boot, Dev Services starts three PostgreSQL containers: __bootstrap, acme, and globex. Keep that terminal running. Open a second terminal in the same project directory for the requests below.

1. Confirm that the resolver sees the tenant header and that the database starts empty.

curl -H "X-Tenant: acme" http://localhost:8080/api/dashboards/tenant
curl -H "X-Tenant: globex" http://localhost:8080/api/dashboards/tenant
curl -H "X-Tenant: acme" http://localhost:8080/api/dashboards

Expected output:

{"tenant":"acme"}
{"tenant":"globex"}
[]

/tenant proves the HTTP resolver is working. The empty list matters too. It tells you there is no stale data hiding in the tenant database before the real demo starts.

2. Create one dashboard for acme.

curl -X POST \
  -H "X-Tenant: acme" \
  -H "Content-Type: application/json" \
  -d '{"name":"ARR","ownerEmail":"alice@acme.example","monthlyBudget":120000.00}' \
  http://localhost:8080/api/dashboards

Expected output:

{"id":1,"name":"ARR","ownerEmail":"alice@acme.example","monthlyBudget":120000.00}

3. Create one dashboard for globex.

curl -X POST \
  -H "X-Tenant: globex" \
  -H "Content-Type: application/json" \
  -d '{"name":"Cash Flow","ownerEmail":"finops@globex.example","monthlyBudget":98000.00}' \
  http://localhost:8080/api/dashboards

Expected output:

{"id":1,"name":"Cash Flow","ownerEmail":"finops@globex.example","monthlyBudget":98000.00}

Before the next step, predict what each tenant should see. The right answer is one row per tenant, not a shared list with two rows.

4. Read the data back through the same endpoint.

curl -H "X-Tenant: acme" http://localhost:8080/api/dashboards
curl -H "X-Tenant: globex" http://localhost:8080/api/dashboards

Expected output on a fresh start:

[{"id":1,"name":"ARR","ownerEmail":"alice@acme.example","monthlyBudget":120000.00}]
[{"id":1,"name":"Cash Flow","ownerEmail":"finops@globex.example","monthlyBudget":98000.00}]

Both rows start at id=1. That is normal because each tenant has its own database and its own sequence. If one response shows both rows, tenant resolution or datasource routing is wrong.

5. Check the failure path for header mode.

curl -i http://localhost:8080/api/dashboards

Expected output starts with:

HTTP/1.1 400 Bad Request

That error tells you the header-based resolver chain is active. In this mode, the request must include X-Tenant.

Add a repeatable header-mode test

I want the first slice under test before I add more resolution modes.

Dev Services keeps the tenant containers alive for the whole test run, so I add one helper that clears both tenant databases before and after each test. Outside HTTP there is no incoming request to populate TenantContext, so the helper sets the tenant directly:

package io.mainthread.vaultboard.support;

import java.util.List;

import io.mainthread.vaultboard.dashboard.Dashboard;
import io.quarkiverse.multitenancy.core.runtime.context.TenantContext;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.transaction.Transactional;

@ApplicationScoped
public class TenantDataCleaner {

    private final TenantContext tenantContext;

    public TenantDataCleaner(TenantContext tenantContext) {
        this.tenantContext = tenantContext;
    }

    @Transactional
    public void clearAll() {
        for (String tenant : List.of("acme", "globex")) {
            tenantContext.setTenantId(tenant);
            Dashboard.deleteAll();
        }
        tenantContext.clear();
    }
}

Then add the test class:

package io.mainthread.vaultboard.dashboard;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasSize;

import java.math.BigDecimal;

import io.mainthread.vaultboard.support.TenantDataCleaner;
import jakarta.inject.Inject;

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;

@QuarkusTest
class DashboardResourceTest {

    @Inject
    TenantDataCleaner tenantDataCleaner;

    @BeforeEach
    void resetBeforeEachTest() {
        tenantDataCleaner.clearAll();
    }

    @AfterEach
    void cleanup() {
        tenantDataCleaner.clearAll();
    }

    @Test
    void acmeDoesNotSeeGlobexData() {
        given()
                .header("X-Tenant", "acme")
                .contentType("application/json")
                .body(new CreateDashboardRequest("ARR", "alice@acme.example", new BigDecimal("120000.00")))
                .when().post("/api/dashboards")
                .then()
                .statusCode(200);

        given()
                .header("X-Tenant", "globex")
                .contentType("application/json")
                .body(new CreateDashboardRequest("Cash Flow", "finops@globex.example", new BigDecimal("98000.00")))
                .when().post("/api/dashboards")
                .then()
                .statusCode(200);

        given()
                .header("X-Tenant", "acme")
                .when().get("/api/dashboards")
                .then()
                .statusCode(200)
                .body("$", hasSize(1))
                .body("[0].ownerEmail", equalTo("alice@acme.example"));

        given()
                .header("X-Tenant", "globex")
                .when().get("/api/dashboards")
                .then()
                .statusCode(200)
                .body("$", hasSize(1))
                .body("[0].ownerEmail", equalTo("finops@globex.example"));
    }

    @Test
    void missingHeaderFailsFastInHeaderMode() {
        given()
                .when().get("/api/dashboards")
                .then()
                .statusCode(400);
    }
}

That second test matters. In header mode, the ORM module adds a filter that rejects requests without X-Tenant.

Why the resolution API matters

The contract matters more than any single resolver. The built-in header resolver is one implementation. TenantResolver no longer returns Optional<String>. It returns a sealed TenantResolution with three outcomes:

• Resolved means a resolver found a usable tenant id, so the chain stops

• NotApplicable means this resolver had nothing to work with, so the next strategy may try

• Rejected means input was present but invalid, so the request must stop with HTTP 401

A missing header and a bad JWT are different cases. Missing input may allow fallback. Present but invalid input should stop the request. The extension makes that rule explicit in the API.

Add JWT resolution without rewriting the main config

I leave the application itself in header mode. For the JWT slice, I switch strategies only inside a test profile. That keeps the first slice stable while we add another one.

Add the JWT dependencies:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-smallrye-jwt</artifactId>
</dependency>
<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-smallrye-jwt-build</artifactId>
</dependency>

Generate a dev-only key pair and place both files under src/main/resources:

openssl genrsa -out src/main/resources/privateKey.pem 2048
openssl rsa -pubout -in src/main/resources/privateKey.pem -out src/main/resources/publicKey.pem

Now add the profile:

package io.mainthread.vaultboard.dashboard;

import java.util.Map;

import io.quarkus.test.junit.QuarkusTestProfile;

public class JwtTenantTestProfile implements QuarkusTestProfile {

    @Override
    public Map<String, String> getConfigOverrides() {
        return Map.of(
                "quarkus.multi-tenant.http.strategy", "jwt",
                "quarkus.multi-tenant.orm.header-filter.enabled", "false",
                "mp.jwt.verify.publickey.location", "publicKey.pem",
                "mp.jwt.verify.publickey.algorithm", "RS256",
                "mp.jwt.verify.issuer", "https://auth.vaultboard.example",
                "smallrye.jwt.sign.key.location", "privateKey.pem");
    }
}

quarkus.multi-tenant.orm.header-filter.enabled=false disables the ORM-side header filter. That filter is right for header mode, but in JWT mode it keeps asking for X-Tenant.

Then add the test:

package io.mainthread.vaultboard.dashboard;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;

import java.time.Duration;

import io.smallrye.jwt.build.Jwt;
import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.TestProfile;

@QuarkusTest
@TestProfile(JwtTenantTestProfile.class)
class JwtTenantResolutionTest {

    @Test
    void resolvesTenantFromVerifiedJwtClaim() {
        String token = Jwt.upn("alice")
                .issuer("https://auth.vaultboard.example")
                .claim("tenant", "acme")
                .expiresIn(Duration.ofMinutes(15))
                .sign();

        given()
                .header("Authorization", "Bearer " + token)
                .when().get("/api/dashboards/tenant")
                .then()
                .statusCode(200)
                .body("tenant", equalTo("acme"));
    }

    @Test
    void missingTenantClaimRejectsRequest() {
        String token = Jwt.upn("alice")
                .issuer("https://auth.vaultboard.example")
                .expiresIn(Duration.ofMinutes(15))
                .sign();

        given()
                .header("Authorization", "Bearer " + token)
                .when().get("/api/dashboards/tenant")
                .then()
                .statusCode(401);
    }
}

This is the behavior I want. A verified token with a tenant claim resolves the tenant. A verified token without that claim gets a 401.

Add a host-based resolver

JWT is one input source. The custom resolver API lets you add your own.

This resolver reads the tenant from the subdomain and rejects unknown values:

package io.mainthread.vaultboard.tenant;

import java.util.Locale;
import java.util.Optional;
import java.util.Set;

import io.quarkiverse.multitenancy.core.runtime.api.TenantResolution;
import io.quarkiverse.multitenancy.core.runtime.api.TenantResolutionContext;
import io.quarkiverse.multitenancy.core.runtime.api.TenantResolver;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.ws.rs.container.ContainerRequestContext;

@ApplicationScoped
public class HostTenantResolver implements TenantResolver {

    private static final Set<String> KNOWN_TENANTS = Set.of("acme", "globex");

    @Override
    public TenantResolution resolve(TenantResolutionContext context) {
        Optional<ContainerRequestContext> request = context.get(ContainerRequestContext.class);
        if (request.isEmpty()) {
            return TenantResolution.notApplicable();
        }

        String hostHeader = request.get().getHeaderString("Host");
        if (hostHeader == null || hostHeader.isBlank()) {
            return TenantResolution.notApplicable();
        }

        String authority = hostHeader.trim().toLowerCase(Locale.ROOT);
        String host = authority.split(":", 2)[0];
        if (host.equals("localhost") || host.equals("127.0.0.1")) {
            return TenantResolution.notApplicable();
        }

        int firstDot = host.indexOf('.');
        if (firstDot < 1) {
            return TenantResolution.notApplicable();
        }

        String tenant = host.substring(0, firstDot);
        if (!KNOWN_TENANTS.contains(tenant)) {
            return TenantResolution.rejected("Unknown tenant host: " + tenant);
        }

        return TenantResolution.resolved(tenant);
    }
}

The localhost branch matters. Without it, normal local traffic starts behaving like a broken multitenant DNS setup.

Custom resolvers run before the built-in strategy chain, so the host slice only needs one test profile change:

package io.mainthread.vaultboard.dashboard;

import java.util.Map;

import io.quarkus.test.junit.QuarkusTestProfile;

public class HostTenantTestProfile implements QuarkusTestProfile {

    @Override
    public Map<String, String> getConfigOverrides() {
        return Map.of(
                "quarkus.multi-tenant.orm.header-filter.enabled", "false");
    }
}

The test stays small:

package io.mainthread.vaultboard.dashboard;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.TestProfile;

@QuarkusTest
@TestProfile(HostTenantTestProfile.class)
class HostTenantResolutionTest {

    @Test
    void resolvesTenantFromHostHeader() {
        given()
                .header("Host", "acme.vaultboard.example")
                .when().get("/api/dashboards/tenant")
                .then()
                .statusCode(200)
                .body("tenant", equalTo("acme"));
    }

    @Test
    void rejectsUnknownTenantHost() {
        given()
                .header("Host", "unknown.vaultboard.example")
                .when().get("/api/dashboards/tenant")
                .then()
                .statusCode(401);
    }
}

At this point the same TenantContext supports three inputs: header, JWT, and a custom host resolver.

What I would change for production

Before I would call this production-ready, I would change three things:

• Map external tenant ids to an allowed datasource map instead of trusting raw headers, claims, or hostnames directly

• Keep quarkus.multi-tenant.http.strategy explicit on every entry point instead of relying on the default chain

• Leave the ORM header filter enabled only for header mode, and disable it for JWT, host, path, or cookie-driven flows

Prove it

I run the slices one by one while building:

./mvnw test -Dtest=DashboardResourceTest
./mvnw test -Dtest=JwtTenantResolutionTest
./mvnw test -Dtest=HostTenantResolutionTest

Each test covers one resolution mode:

• Header mode proves tenant-aware CRUD and the ORM-side header filter

• JWT mode proves claim-based tenant resolution and request rejection

• Host mode proves a custom resolver can participate without rewriting the rest of the app

Result

This is the shape I want for multi-tenant code. Resolve the tenant once, store it in TenantContext, and let domain code talk about dashboards, invoices, and users. Tenant routing stays at the edge unless there is a real reason to pull it inward.

Subscribe now

Did the parser read the file?

Did the schema validate?

Did the tests pass?

Did the rendered artifact contain the expected objects?

Did the human approve the decision?

That pause turns a model claim into something the workflow can inspect. Reliable agent systems do not try to make the language model deterministic. They put the language model around deterministic islands: scripts, schemas, parsers, type systems, linters, tests, CI gates, and approval points. The model can reason, draft, route, and repair. The islands own the facts.

A large language model is good at working with incomplete context. It can infer intent from a vague issue, connect a failing test to the likely source file, or translate a human comment into a patch plan. That flexibility is why we use it. It can also talk itself into a clean answer while the change is still wrong, so it should not be the final judge of correctness.

Better prompts help, but they do not create contracts

A better prompt can make the model more consistent. It can name the goal, define style, list allowed tools, and remind the agent to run checks before it claims success. All of that helps.

But a prompt is still an instruction to a probabilistic system. It changes the distribution of likely answers. It does not create a hard guarantee.

If the model says “I validated the deck,” that sentence is cheap. If a script opened the generated PPTX, inspected the slide XML, rendered each slide to PNG, counted unresolved placeholders, checked overlap warnings, and returned a structured report, that is a different thing. The first one is a claim. The second one is evidence.

This is the mistake I see in many early agent setups. The team keeps adding instructions:

• be careful

• verify the result

• do not hallucinate

• make sure the output is valid

• double-check before responding

Those lines help a little. They can reduce sloppy behavior. They still leave the core problem open: no definition of “valid,” no validator run, and no gate that blocks a bad artifact from moving forward.

Reliability needs something outside the model to say yes or no.

What a deterministic island is

A deterministic island is a step in the workflow where truth is owned by something more constrained than the language model.

This is an engineering contract, not mathematics. A test can be flaky. A linter can have a bug. A parser can change behavior between versions. But the step still has a contract: same input, same environment, same version, same result. When it fails, it should fail in a way another program can read.

Examples:

• A JSON parser either parses the file or reports a line and column.

• A schema validator accepts or rejects a payload against explicit fields and types.

• A type checker says whether the code satisfies the language rules.

• A unit test checks a behavior and returns pass or fail.

• A linter catches formatting, unsafe patterns, or broken references.

• A renderer exports an artifact and exposes layout warnings.

• A CI gate runs the checks in a clean environment.

• A human approval step decides whether the change should ship.

A human approval point can be an island too, as long as the workflow is clear about what the human owns. Humans are good at deciding intent, taste, risk, policy, and business meaning. They are not the right checksum algorithm for 27 generated files.

The model moves between these islands. It reads the evidence from one step, decides what to try next, and hands the next concrete task to another constrained system.

The simple loop

The loop is simple:

The model sits between checkpoints. It handles ambiguity, chooses a path, and converts failure output into the next attempt. Every factual claim goes through a narrower system before the workflow treats it as true.

A concrete example: generated presentation artifacts

Presentation generation is a good stress test because it mixes content, layout, binary files, brand rules, and visual judgment. It is also easy to fake success. A model can say “the slides look clean” without ever opening the final deck. That is not validation.

A stronger workflow starts by inspecting the source deck with a real parser. It reads every slide, extracts text boxes, images, tables, placeholders, fonts, dimensions, and object identifiers. The model can reason over that inventory, but it should not invent the inventory.

Then the workflow creates a structured map:

• output slide 1 reuses source slide 3

• this text box is rewritten

• this table is filled

• this placeholder is deleted

• this logo stays untouched

• no new objects are allowed on this slide

That map is a deterministic island if it is schema-validated. Required fields are present. Slide numbers are sequential. Source slide references exist. Edit targets resolve to real objects. If the map says to edit shape-17 and there is no shape-17, the workflow stops early.

After that, a generator duplicates the selected source slides and edits inherited objects in place. Again, the model can choose the content and explain why a source slide fits. The script owns the actual mutation.

The final check renders every slide and inspects the exported file. It looks for unresolved placeholders, broken page markers, missing required assets, text clipping, overlap warnings, and layout drift. A good check does not say “looks good.” It says something like:

{
  "status": "fail",
  "checks": {
    "renderedSlides": 12,
    "unresolvedPlaceholders": 2,
    "overlapWarnings": 0,
    "missingRequiredAssets": 0
  },
  "issues": [
    {
      "code": "unresolved-placeholder",
      "slide": 7,
      "objectId": "title-placeholder",
      "message": "Inherited title placeholder is empty."
    }
  ]
}

Now the agent has something it can use. It can go back to slide 7, locate the inherited title placeholder, and fix the real object. It does not have to stare at a screenshot and guess where the presentation format might hide a structural placeholder. That detail belongs to the checker.

The same pattern works outside slides.

Parsing files: let parsers own structure

Agents are good at reading text. They are much less reliable when you ask them to infer structure that a parser can provide exactly.

If the input is JSON, parse JSON. If it is XML, use an XML parser. If it is Java, use the compiler, an AST parser, or the IDE refactoring engine when available. If it is a zip-based document format, inspect the archive and the internal manifests. If it is Markdown with front matter, parse the front matter as data before editing prose.

The model can decide that a publishing pack probably contains social copy, images, tags, and scheduling notes. A parser should say which fields exist and where they live. The model can suggest that a Java class needs a rename. The refactoring tool should find symbol references. The model can guess that a deck has a title placeholder. The presentation inspector should return the placeholder ID.

This is beginner-friendly in practice because it removes mystery. The agent is no longer “looking at the project.” It is calling a tool that returns named facts.

Schemas and typed outputs make the next step safer

Structured output is one of the simplest ways to make agent workflows less fragile.

Free text makes the next step interpret intent again:

I think slide 3 should become the opening slide and we should update the headline.

A typed shape gives the workflow something to validate:

{
  "outputSlide": 1,
  "sourceSlide": 3,
  "role": "opening",
  "edits": [
    {
      "targetId": "shape-title",
      "action": "rewrite",
      "text": "Quarterly rollout plan"
    }
  ]
}

The schema can require outputSlide, sourceSlide, role, and edits. It can restrict action to known values. It can reject an edit without a target. It can reject a slide number that does not exist.

A schema cannot know whether “Quarterly rollout plan” is the right headline. It can still keep the workflow out of impossible states. That removes a large class of failures before the model gets another chance to improvise.

Typed outputs also make review easier. Instead of reading a long agent explanation and trying to infer what will happen, the human can inspect a concise plan:

• which files change

• which artifacts are generated

• which channels are targeted

• which checks will run

• which decision needs approval

The model can still write the explanation. The machine path gets data.

Good tool responses for agents

Tools built for humans often return a nice sentence and a wall of logs. Tools built for agents need a different shape.

A good response should include:

• status: success, fail, or blocked

• stable error codes plus a short prose message

• exact file paths, line numbers, object IDs, slide numbers, test names, or rule IDs

• counts that describe the scope of the check

• the command, version, or rule set used

• a short human summary

• structured issue records the model can act on

• retry guidance when the failure is environmental

• a path to full logs or raw output when the payload is too large

For example, a security scan needs the rule ID, file, line, matched snippet, severity, and remediation hint. A publishing tool needs the channel ID, post ID, scheduled timestamp, timezone, and preview URL if one exists. A test runner should separate compilation failure, test failure, timeout, and infrastructure failure. Those are different next steps.

Vague errors make agents guess. Guessing is exactly what the deterministic island was supposed to reduce.

Refactoring: let the model propose, let the tools verify

Refactoring shows the boundary well.

An agent can read a codebase and decide that CustomerManager should become CustomerService, that a method should move, or that repeated code should become a helper. That judgment combines naming, intent, local style, and trade-offs. The toolchain should handle the mechanical proof.

The mechanical change should lean on deterministic tools:

• symbol-aware rename when the language server supports it

• AST-based transforms for repeated code shapes

• compiler checks for type errors

• unit tests for behavior

• formatting and import sorting

• dependency analysis when public APIs move

The agent can write the first patch, run the checks, read failures, and repair. The compiler owns type truth. The tests own behavior truth inside their scope. The linter owns the style rules it knows. CI owns the clean-room replay.

This is also why “the agent said it reviewed the code” is a weak signal. A review without a diff, tests, static checks, and clear findings is just commentary.

Publishing: approval should be about intent

Publishing workflows are another good example because they combine mechanical details and human judgment.

An agent can draft social copy from an article. It can create variants for different channels. It can select a short hook, pull a canonical URL, and suggest a schedule. Those are good jobs for a model.

The mechanical checks should be deterministic:

• parse the article metadata

• extract the canonical URL

• verify the image path exists

• inspect the Open Graph preview

• look up channel IDs from the provider because prose names drift

• check for duplicate scheduled posts

• validate that scheduled times are in the future and in the right timezone

• enforce character limits

• return the exact post IDs after scheduling

The human should approve the message, channel choice, timing, and risk. The human should not have to manually re-check whether the Mastodon post exceeded a character limit, whether the LinkedIn channel ID was copied correctly, or whether the URL resolves.

That is the point of the workflow. Human attention is expensive. Spend it on decisions.

Tests, linters, hooks, and CI turn guesses into evidence

The model can say “this should fix it.” Tests decide whether the behavior changed the way you expected.

Local checks catch fast mistakes. A linter catches a broken link, a missing field, a forbidden API, or an unsafe pattern before review. A pre-commit hook catches problems before they leave the workstation. CI runs the same checks in a clean environment, which catches missing files, hidden local state, version drift, and the usual “works on my machine” problem.

None of these checks prove the system is correct. They prove narrower claims:

• this file parses

• this generated artifact can be opened

• this schema is valid

• this code compiles

• this test still passes

• this forbidden pattern is absent

• this package has no known critical vulnerability under the configured scanner

That list is still worth a lot. Reliable systems are built from many narrow claims with clear owners.

Security checks need their own islands

Security is a bad place to trust a model’s confidence.

An agent can help explain a security finding, suggest a safer API, or update code to pass a rule. But the detection step should be deterministic where possible:

• secret scanners for tokens and credentials

• static analysis for risky code patterns

• dependency scanners for vulnerable packages

• policy checks for container images or infrastructure changes

• permission checks before tools touch external systems

• approval gates for deploys, publishes, and destructive actions

The reviewer still owns exceptions and policy calls. The scanner should still find the pattern every time.

Common failure modes

The pattern is simple. The failures are also simple.

Fake validation

The agent says it validated the result, but no validator ran. Or the validator only checked that a file exists. Or the check looked at the generated source but not the exported artifact. This is common with binary outputs, generated docs, and UI screenshots.

The fix is to validate the thing the user will receive.

Hand-edited generated output

Someone fixes the generated PPTX, PDF, OpenAPI file, or code output by hand after the generator runs. The artifact looks right once, but the workflow is now split. The next run overwrites the fix or produces a different result.

If a manual edit is necessary, make it an input, update the generator, or record it as an explicit patch step with a check after it.

Skipped checks

Small changes are where people skip checks because the change feels obvious. Agents learn the same bad habit if the workflow lets them. A one-line publishing change can post to the wrong channel. A tiny refactor can break serialization. A harmless deck edit can leave an invisible placeholder behind.

Make the cheap checks automatic. Save human exceptions for the expensive ones.

Too many tools

A giant tool menu can make an agent worse. It chooses the wrong tool, calls tools in the wrong order, or spends time exploring options that should not exist for the task.

Good workflows expose fewer tools with clearer contracts. A tool named inspect_presentation that returns slide objects is easier to use than a general “run any office command” tool with no clear contract.

Vague errors

Validation failed is barely an error. The agent now has to infer the cause, and it may repair the wrong thing.

Errors should name the failed check, the target, the evidence, and the next action. unresolved-placeholder on slide 7 object title-placeholder is something an agent can fix.

The real pattern

The model is the reasoning layer. It handles ambiguity, proposes changes, explains failures, and decides what to try next. Deterministic islands are the truth layer. They parse, validate, generate, test, scan, and gate.

Good workflows move between the two deliberately.

Keep the model away from jobs we already have tools for: compiling, parsing, linting, scanning, rendering, and replaying checks in CI. The engineering task is to make those tools available at the right points, with outputs the model can read and humans can trust.

That is how agent systems become reliable enough for real projects. The probabilistic reasoning stays. It gets a map, a set of checkpoints, and a few hard places where guessing stops.

Subscribe now

After ./mvnw package, the file that matters in this example is:

target/quarkus-run-spdx.json

This is where the article splits from my earlier CycloneDX SBOM walkthrough. That piece was about Quarkus distribution SBOMs, dependency SBOMs, and the CycloneDX workflow around them. This one starts when security, legal, procurement, or an OSPO asks for SPDX instead. Same supply-chain conversation, different document model, and much more weight on license metadata and exchange with downstream tooling.

SPDX is not just another SBOM file format. It is an ISO standard, specifically ISO/IEC 5962:2021. It is also where many teams first hit machine-readable license expressions, package identity, and relationship data that somebody outside the application team will read later. That part matters. A scanner wants a file. A lawyer wants declared licenses. A platform team wants identifiers and relationships they can compare across releases. A customer questionnaire wants proof that your answer came from a repeatable build instead of a copied spreadsheet and a confident shrug.

The Quarkiverse quarkus-spdx project gives Quarkus a native path into that world. We build a small REST service, generate an SPDX 3.0.1 SBOM during packaging, inspect the JSON-LD structure Quarkus writes, follow a declared license relationship for one dependency, and check the result with the SPDX ntia-conformance-checker. After that, we do a short SPDX 2.3 pass because older tools and internal workflows still care about tag-value output.

What You Need

This article uses Quarkus 3.27.2, quarkus-spdx 0.0.1, and Java 21. The Quarkus version stays pinned here because the current quarkus-spdx release was built against that line, and the article should describe a setup that packages cleanly.

• Java 25 installed

• Quarkus CLI installed

• Maven, or the Maven wrapper Quarkus generates

• jq for reading the SPDX JSON output

• Python 3.10 or newer if you want to run sbomcheck

• Basic Quarkus REST knowledge

• About two ☕️

Three terms matter before the code:

SPDX - a standard model for software metadata, not only a bag of package names. The current specs are SPDX 3.0.1 and the older but still common SPDX 2.3.

JSON-LD - the JSON-LD serialization used by SPDX 3. The file still looks like JSON, but the @context and graph model matter because relationships and element types are first-class citizens.

License expression - the SPDX syntax for license statements such as MIT, Apache-2.0, or composite forms like Apache-2.0 OR GPL-2.0-only. SPDX makes those expressions portable. Your internal approval policy still has to decide what to do with them.

Create the Project

Create the application and follow along or check out my github repository:

quarkus create app io.mainthread:license-ledger \
  --extension='rest-jackson,io.quarkiverse.spdx:quarkus-spdx-v3:0.0.1' \
  --platform-bom=io.quarkus:quarkus-bom:3.27.2 \
  --java=25 \
  --no-code
cd license-ledger

Use these extensions:

• rest-jackson - JSON REST endpoints

• io.quarkiverse.spdx:quarkus-spdx-v3:0.0.1 - SPDX 3.0.1 SBOM generation during the Quarkus build

SPDX is not a platform-managed Quarkus core extension, so the version should be easy to verify later.

We also add one plain Maven dependency so the generated SBOM has something easy to recognize and inspect later. Add this to pom.xml inside <dependencies>:

<dependency>
    <groupId>com.github.package-url</groupId>
    <artifactId>packageurl-java</artifactId>
    <version>1.5.0</version>
</dependency>

packageurl-java is a good fit here because it is small, it is directly relevant to SBOM work, and its Maven Central entry clearly states the MIT license. That gives us a clean license relationship to inspect later.

Add a Small Service Worth Packaging

The service stays small. We only need enough application code that the packaged Quarkus distribution contains our own artifact plus one non-Quarkus dependency we can follow through the SPDX graph.

Create src/main/java/io/mainthread/licenseledger/ComponentRequest.java:

package io.mainthread.licenseledger;

public record ComponentRequest(
        String supplier,
        String groupId,
        String artifactId,
        String version,
        String licenseExpression) {
}

Create src/main/java/io/mainthread/licenseledger/ComponentReport.java:

package io.mainthread.licenseledger;

public record ComponentReport(
        String supplier,
        String coordinate,
        String purl,
        String licenseExpression,
        String decision,
        String note) {
}

The request stays small. The response returns the fields we care about here: coordinate, package URL, license expression, and the decision the service made.

Now add the service in src/main/java/io/mainthread/licenseledger/ComponentCatalogService.java:

package io.mainthread.licenseledger;

import java.util.List;
import java.util.Set;
import java.util.TreeMap;

import com.github.packageurl.MalformedPackageURLException;
import com.github.packageurl.PackageURL;

import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class ComponentCatalogService {

    private static final Set<String> APPROVED_LICENSES = Set.of(
            "Apache-2.0",
            "BSD-2-Clause",
            "BSD-3-Clause",
            "MIT");

    private static final Set<String> REVIEW_LICENSES = Set.of(
            "EPL-2.0",
            "LGPL-2.1-only",
            "LGPL-2.1-or-later",
            "MPL-2.0");

    public List<ComponentReport> sampleComponents() {
        return List.of(
                review(new ComponentRequest(
                        "Quarkus",
                        "io.quarkus",
                        "quarkus-rest-jackson",
                        "3.27.2",
                        "Apache-2.0")),
                review(new ComponentRequest(
                        "Package URL",
                        "com.github.package-url",
                        "packageurl-java",
                        "1.5.0",
                        "MIT")),
                review(new ComponentRequest(
                        "Example Vendor",
                        "org.example",
                        "legacy-reports",
                        "2.4.1",
                        "GPL-2.0-only")));
    }

    public ComponentReport review(ComponentRequest request) {
        String coordinate = request.groupId() + ":" + request.artifactId() + ":" + request.version();
        String licenseExpression = normalize(request.licenseExpression());
        String decision = decisionFor(licenseExpression);
        String note = noteFor(licenseExpression, decision);

        return new ComponentReport(
                normalize(request.supplier()),
                coordinate,
                buildPurl(request),
                licenseExpression,
                decision,
                note);
    }

    private String buildPurl(ComponentRequest request) {
        try {
            return new PackageURL(
                    "maven",
                    request.groupId(),
                    request.artifactId(),
                    request.version(),
                    new TreeMap<>(java.util.Map.of("type", "jar")),
                    null).canonicalize();
        } catch (MalformedPackageURLException e) {
            throw new IllegalArgumentException("Invalid Maven coordinates for SPDX demo", e);
        }
    }

    private String decisionFor(String licenseExpression) {
        if (licenseExpression.contains(" OR ") || licenseExpression.contains(" WITH ")) {
            return "manual-review";
        }
        if (APPROVED_LICENSES.contains(licenseExpression)) {
            return "approved";
        }
        if (REVIEW_LICENSES.contains(licenseExpression)) {
            return "manual-review";
        }
        if (licenseExpression.contains("GPL") || licenseExpression.contains("AGPL")) {
            return "blocked";
        }
        return "manual-review";
    }

    private String noteFor(String licenseExpression, String decision) {
        if ("approved".equals(decision)) {
            return "Known SPDX identifier with a policy rule that can pass automatically.";
        }
        if ("blocked".equals(decision)) {
            return "Copyleft licenses are not rejected by SPDX, but this demo policy sends them to a hard stop.";
        }
        if (licenseExpression.contains(" OR ") || licenseExpression.contains(" WITH ")) {
            return "The SBOM can keep this SPDX expression exactly. A human still needs to decide which branch is acceptable.";
        }
        return "The SBOM stays useful, but this license still needs a reviewer before it reaches production.";
    }

    private String normalize(String value) {
        return value == null ? "" : value.trim();
    }
}

This service does two things that matter for the article. It creates a PURL we can mention without pretending SPDX invented package identity, and it treats the SPDX license expression as data instead of reducing it to a copied string in a report. That is where the operational value starts. SPDX gives you a portable expression. Your approval policy decides how much automation you trust.

The policy is intentionally boring. MIT passes, GPL-2.0-only blocks, and composite expressions such as Apache-2.0 OR GPL-2.0-only go to manual review. That split matches how these discussions usually happen in real teams. A machine can tell you the expression. A human still decides whether the choice inside that expression is acceptable for a given release.

Expose the service over HTTP in src/main/java/io/mainthread/licenseledger/LicenseLedgerResource.java:

package io.mainthread.licenseledger;

import java.util.List;

import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/components")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class LicenseLedgerResource {

    private final ComponentCatalogService service;

    public LicenseLedgerResource(ComponentCatalogService service) {
        this.service = service;
    }

    @GET
    @Path("/demo")
    public List<ComponentReport> demo() {
        return service.sampleComponents();
    }

    @POST
    @Path("/review")
    public ComponentReport review(ComponentRequest request) {
        return service.review(request);
    }
}

We keep the REST layer thin here on purpose. The resource exists so the project is a normal Quarkus service. The interesting work stays in the service, where we can test the decision logic without needing HTTP for every branch.

Add Tests Before Packaging

Create src/test/java/io/mainthread/licenseledger/ComponentCatalogServiceTest.java:

package io.mainthread.licenseledger;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;

import org.junit.jupiter.api.Test;

class ComponentCatalogServiceTest {

    private final ComponentCatalogService service = new ComponentCatalogService();

    @Test
    void shouldApproveKnownLicense() {
        ComponentReport report = service.review(new ComponentRequest(
                "Package URL",
                "com.github.package-url",
                "packageurl-java",
                "1.5.0",
                "MIT"));

        assertEquals("approved", report.decision());
        assertEquals("pkg:maven/com.github.package-url/packageurl-java@1.5.0?type=jar", report.purl());
    }

    @Test
    void shouldEscalateCompositeLicenseExpression() {
        ComponentReport report = service.review(new ComponentRequest(
                "Example Vendor",
                "org.example",
                "dual-licensed-lib",
                "2.0.0",
                "Apache-2.0 OR GPL-2.0-only"));

        assertEquals("manual-review", report.decision());
        assertTrue(report.note().contains("human"));
    }

    @Test
    void shouldBlockCopyleftLicense() {
        ComponentReport report = service.review(new ComponentRequest(
                "Example Vendor",
                "org.example",
                "legacy-reports",
                "2.4.1",
                "GPL-2.0-only"));

        assertEquals("blocked", report.decision());
        assertTrue(report.note().contains("hard stop"));
    }
}

Add the HTTP test in src/test/java/io/mainthread/licenseledger/LicenseLedgerResourceTest.java:

package io.mainthread.licenseledger;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasSize;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;

@QuarkusTest
class LicenseLedgerResourceTest {

    @Test
    void shouldReturnSampleComponents() {
        given()
                .when().get("/components/demo")
                .then()
                .statusCode(200)
                .body("$", hasSize(3))
                .body("[0].decision", equalTo("approved"))
                .body("[2].decision", equalTo("blocked"));
    }

    @Test
    void shouldReviewSubmittedComponent() {
        given()
                .contentType("application/json")
                .body("""
                        {
                          "supplier": "Package URL",
                          "groupId": "com.github.package-url",
                          "artifactId": "packageurl-java",
                          "version": "1.5.0",
                          "licenseExpression": "MIT"
                        }
                        """)
                .when().post("/components/review")
                .then()
                .statusCode(200)
                .body("supplier", equalTo("Package URL"))
                .body("decision", equalTo("approved"))
                .body("purl", equalTo("pkg:maven/com.github.package-url/packageurl-java@1.5.0?type=jar"));
    }
}

The unit test proves the license-policy branches. The @QuarkusTest proves the HTTP path and JSON mapping. That is enough for this example. We are not testing Quarkus itself, and we are not pretending a license policy engine needs a hundred lines of ceremonial coverage before it becomes credible.

Package the Application

Build the application:

./mvnw package

On a Java 21 runtime, the command should finish cleanly. On my workstation with Java 25 installed globally, the build still succeeds, but Quarkus test runs emit an extra JBoss Threads warning about --add-opens java.base/java.lang=ALL-UNNAMED. That is one more reason the article stays on Java 21.

Look at the top of target/:

ls target

You should see at least these entries:

license-ledger-1.0.0-SNAPSHOT.jar
quarkus-app
quarkus-run-spdx.json

That output already tells us something important. The current quarkus-spdx-v3 release writes a file named quarkus-run-spdx.json. The content is SPDX 3.0.1 JSON-LD, but the filename is still .json, not .jsonld.

Read the SPDX 3 Root

Start with the document node:

jq '.["@graph"][] | select(.type == "SpdxDocument") | {name, rootElement, profileConformance}' \
  target/quarkus-run-spdx.json

Expected output:

{
  "name": "quarkus-run",
  "rootElement": [
    "https://spdx.org/spdxdocs/quarkus-run-.../SBOM"
  ],
  "profileConformance": [
    "software",
    "core"
  ]
}

That is already a different mental model from the SPDX 2 JSON many people remember. The document points at an SBOM element inside a graph, and the graph uses typed nodes such as software_Sbom, software_Package, and Relationship.

Now inspect the packaged root artifact:

jq '.["@graph"][] | select((.spdxId? // "") | endswith("/SPDXRef-Package-quarkus-run.jar")) | {name, comment, hashes: [.verifiedUsing[].algorithm]}' \
  target/quarkus-run-spdx.json

Expected shape:

{
  "name": "quarkus-run.jar",
  "comment": "Distribution path: quarkus-run.jar",
  "hashes": [
    "sha1",
    "sha256",
    "sha512",
    "sha3_256",
    "sha3_512",
    "sha384",
    "sha3_384",
    "md5"
  ]
}

This is a good place to stop and look carefully. The root is the runnable Quarkus distribution, not your Maven coordinates in isolation. That should feel familiar if you read the CycloneDX article, but the representation is different. In SPDX 3, the distribution artifact is a software_Package node with hashes and relationship edges around it.

Your own application package is still present as a separate node:

jq '.["@graph"][] | select(.type == "software_Package" and ((.spdxId? // "") | contains("io.mainthread-license-ledger-1.0.0-SNAPSHOT"))) | {name, spdxId}' \
  target/quarkus-run-spdx.json

Expected output:

{
  "name": "io.mainthread:license-ledger",
  "spdxId": "https://spdx.org/spdxdocs/quarkus-run-.../SPDXRef-Package-io.mainthread-license-ledger-1.0.0-SNAPSHOT"
}

That split matters in practice. One node is the runnable package you ship. Another node is your application artifact as part of that build graph. People often compress those into “the app” in conversation. SPDX does not.

Follow a Declared License Relationship

The graph gets more interesting when you follow a dependency to its declared license. Use this query for packageurl-java:

jq '.["@graph"] as $g
  | ($g[] | select(.type == "software_Package" and ((.spdxId? // "") | contains("com.github.package-url-packageurl-java-1.5.0"))) | .spdxId) as $pkg
  | ($g[] | select(.relationshipType? == "hasDeclaredLicense" and .from == $pkg) | .to[0]) as $license
  | $g[] | select(.spdxId? == $license)
  | {type, expression: .simplelicensing_licenseExpression}' \
  target/quarkus-run-spdx.json

Expected output:

{
  "type": "simplelicensing_LicenseExpression",
  "expression": "MIT"
}

This is where the topic stops being abstract for me. Once you can point to a license expression and say, “this is what our tooling can see,” the whole SBOM discussion gets more concrete. SPDX is strong here because license data is part of the model, not an awkward extra field somebody added later.

It also shows why SPDX and CycloneDX are not interchangeable just because both are SBOM formats. CycloneDX often feels closer to scanner workflows. SPDX often becomes more relevant when legal review, procurement, and standardized license expressions enter the picture. You may still want both.

Check the Result with SPDX Tooling

JSON syntax is not enough. Run an SPDX-oriented checker:

python3 -m venv .sbomcheck
.sbomcheck/bin/pip install ntia-conformance-checker
.sbomcheck/bin/sbomcheck --sbom-spec spdx3 --comply ntia target/quarkus-run-spdx.json

Expected output:

2021 NTIA SBOM Minimum Elements Conformance Results

Conformant: False

Requirement                                    | Status
-------------------------------------------------------
All component names provided?                  | True
All component versions provided?               | False
All component identifiers provided?            | True
All component suppliers provided?              | False
SBOM author name provided?                     | True
SBOM creation timestamp provided?              | True
Dependency relationships provided?             | True

Not a result anybody would put on a marketing slide but the generated SBOM is real. It carries relationships, identifiers, hashes, and declared license links. It still does not satisfy every NTIA minimum element out of the box for every component. The NTIA minimum-elements work, the SPDX NTIA mapping guide, and the SPDX guidance on standards and regulation are the right references here. That honest result is still better than the usual “file exists, therefore compliance is done” fiction.

If you want the 2024 CISA framing check instead:

.sbomcheck/bin/sbomcheck --sbom-spec spdx3 --comply fsct3-min target/quarkus-run-spdx.json

On this sample, that also fails because some versions, suppliers, and concluded licenses are missing. Again, useful result. It tells you where the generated document helps and where your process still needs more data.

When SPDX 2.3 Still Wins

SPDX 3 is the more interesting model. SPDX 2.3 is still the safer answer for older tools and teams that want tag-value output.

Swap the dependency in pom.xml:

<dependency>
    <groupId>io.quarkiverse.spdx</groupId>
    <artifactId>quarkus-spdx-v2</artifactId>
    <version>0.0.1</version>
</dependency>

Add this to src/main/resources/application.properties:

quarkus.spdx.format=all

Build again:

./mvnw package

Now target/ includes:

quarkus-run-spdx.json
quarkus-run-spdx.spdx

The JSON file is SPDX 2.3 JSON. The .spdx file is tag-value. Its first lines look like this:

SPDXVersion: SPDX-2.3
DataLicense: CC0-1.0
DocumentNamespace: https://spdx.org/spdxdocs/quarkus-run-...
DocumentName: quarkus-run
SPDXID: SPDXRef-DOCUMENT

This is the clearest practical split:

• quarkus-spdx-v3 gives you SPDX 3.0.1 JSON-LD with a graph model and explicit typed relationships

• quarkus-spdx-v2 gives you SPDX 2.3 plus optional tag-value output, which older SPDX tooling still understands more easily

If you already know the downstream consumer wants SPDX 2.3 or tag-value, use v2 and move on. If you want the newer model and can handle JSON-LD, v3 is the more interesting path.

SPDX vs CycloneDX in Quarkus

Here is the short version, because I already spent a full article on CycloneDX:

CycloneDX is the better starting point when your main audience is scanner and CI tooling, and when you want the Quarkus-specific split between distribution SBOM and dependency SBOM that the built-in extension already gives you. If that is your problem, read Create Your First Quarkus SBOM with CycloneDX.

SPDX gets more interesting when you need standardized license expressions, SPDX-native exchange, and a document model that legal or OSPO workflows already expect. It is also where standards and regulation discussions tend to point more often. Keep the SPDX overview, the current specifications page, and the SPDX 3.0.1 spec open while you decide.

In real organizations, “pick one forever” is often the wrong question. One format may feed scanners better. Another may fit compliance and disclosure workflows better. If your release process has to satisfy both groups, generating both is less painful than arguing about which department owns reality.

Conclusion

We built a small Quarkus service, generated an SPDX SBOM with quarkus-spdx, inspected the SPDX 3.0.1 JSON-LD graph, followed a declared license expression for one dependency, and checked the result against NTIA and CISA minimum expectations. The useful mental shift is this: an SPDX file is not only an inventory. It is a graph of software elements, identities, relationships, and licenses that other teams will read long after your build finished.

Subscribe now

A refund request can look simple on first read:

if amount is small and the receipt exists, approve it
if the receipt is missing, deny it
otherwise ask a human

That is a policy decision. It belongs in boring, testable code. If your team already uses a rules engine, that policy can live there too.

The awkward part starts after the policy says “ask a human.” Now the system has to emit a review request, remember which workflow instance is waiting, accept a callback later, resume the exact request, record the final decision, and leave enough evidence for the next person who has to debug it. A decision table does not want that job. A workflow engine does.

This is where Quarkus Flow fits. It is a lightweight workflow engine for Quarkus based on the CNCF Serverless Workflow specification. Version 0.11.0, adds enough new pieces that it deserves a different article than another “call three services in order” walkthrough: runner images, better idempotency and correlation guidance, runtime-configurable messaging and observability pieces, Quartz scheduling, @ScheduleOn for agentic workflows, and gRPC channel routing.

We will build RefundDesk, a small Quarkus service that shows the boundary:

• A plain Java policy decides whether a refund is approved, denied, or routed to manual review

• A Quarkus Flow workflow coordinates the process around that decision

• CloudEvents carry the manual-review request and callback

• The workflow pauses and resumes by correlation data

• Tests prove the policy and the HTTP path

No AI is needed for the main path. That is intentional. A workflow engine is already useful before the model shows up and starts making confident little messes. But, yes, Quarkus Flow also works with LangChain4j and the agentic modules. But this will be another article soon hopefully.

What You Need

This article uses Quarkus 3.37.0, Quarkus Flow 0.11.0, Java 25, and Podman.

• Java 25 installed

• Quarkus CLI installed

• Podman available for Kafka Dev Services and the Runner image

• Basic Quarkus REST and CDI knowledge

• About two ☕️ (and yes, you can also take another beverage depending on time of the day and mood.)

Three definitions before we start to code:

Policy - the deterministic decision logic. In this article, it is a Java CDI bean. It could also be Drools, Easy Rules, or a remote decision service.

Workflow - the process that decides which step runs next and what state must survive across waiting, retries, and callbacks.

Correlation - the data that lets an incoming callback wake the right waiting workflow instance. In Quarkus Flow event workflows, the flowinstanceid CloudEvent extension is the important one.

Create the Project

Create the application and follow along or grab the source code from my repository. :

quarkus create app dev.mainthread:refunddesk-flow \
  --extension='io.quarkiverse.flow:quarkus-flow:0.11.0,io.quarkiverse.flow:quarkus-flow-mvstore:0.11.0,rest-jackson,messaging-kafka,smallrye-openapi,quarkus-micrometer-registry-prometheus' \
  --java=25 \
  --no-code
cd refunddesk-flow

Use these extensions:

• quarkus-flow: the Java DSL and workflow runtime

• quarkus-flow-mvstore: local file-backed workflow persistence

• rest-jackson: HTTP endpoints and JSON serialization

• quarkus-micrometer-registry-prometheus - has to be included when Quarkus Flow metrics are enabled.

• messaging-kafka: the Reactive Messaging bridge that connects Flow to Kafka topics

• smallrye-openapi: a quick way to inspect the REST API while testing

The important detail is that Flow is scaffolded by the Quarkus CLI. Do not paste a second Flow dependency block into the POM. The command above asks the CLI to add both Flow artifacts while the project is created.

I am using the full Quarkiverse coordinates because, when I tested this, the catalog short name did not resolve to the 0.11.0 artifact. Once the catalog catches up, --extension='quarkus-flow,...' is the nicer shape. Until then, pinning io.quarkiverse.flow:quarkus-flow:0.11.0 keeps the tutorial reproducible.

quarkus-flow-mvstore gives this local tutorial a file-backed persistence store so waiting workflow state can survive a JVM restart. MVStore is a good local fit. For production, you will usually move to Redis or JPA, depending on how your platform already stores operational state.

Define the Refund Model

The policy and workflow need a small domain model. Keep it explicit. If the records are too loose, the workflow becomes a bag of maps and the policy test stops telling you much.

Create src/main/java/dev/mainthread/refunddesk/DecisionOutcome.java:

package dev.mainthread.refunddesk;

public enum DecisionOutcome {
    APPROVED,
    DENIED,
    MANUAL_REVIEW
}

Create src/main/java/dev/mainthread/refunddesk/RefundRequest.java:

package dev.mainthread.refunddesk;

import java.math.BigDecimal;

public record RefundRequest(
        String refundId,
        String customerId,
        BigDecimal amount,
        boolean receiptPresent,
        int accountAgeDays,
        int chargebackCount) {
}

Create src/main/java/dev/mainthread/refunddesk/RefundDecision.java:

package dev.mainthread.refunddesk;

public record RefundDecision(DecisionOutcome outcome, String reason) {
}

Create src/main/java/dev/mainthread/refunddesk/RefundCase.java:

package dev.mainthread.refunddesk;

public record RefundCase(RefundRequest request, RefundDecision decision) {
}

Create src/main/java/dev/mainthread/refunddesk/ReviewDecision.java:

package dev.mainthread.refunddesk;

public record ReviewDecision(
        String refundId,
        DecisionOutcome outcome,
        String reviewer,
        String note) {
}

Create src/main/java/dev/mainthread/refunddesk/RefundResult.java:

package dev.mainthread.refunddesk;

public record RefundResult(
        String refundId,
        DecisionOutcome outcome,
        String reason,
        String reviewer) {
}

Create src/main/java/dev/mainthread/refunddesk/SubmitRefundResponse.java:

package dev.mainthread.refunddesk;

public record SubmitRefundResponse(String refundId, String workflowInstanceId) {
}

These records separate facts, policy output, review output, and final state. That keeps the policy easy to test and the workflow easier to read. A rules engine would use the same kind of boundary: facts in, decision out.

Write the Policy as Plain Java

The policy is deliberately small. The goal is to show where the rule belongs, not to pretend three predicates are a credit-risk platform.

Create src/main/java/dev/mainthread/refunddesk/RefundPolicy.java:

package dev.mainthread.refunddesk;

import java.math.BigDecimal;

import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class RefundPolicy {

    private static final BigDecimal AUTO_APPROVE_LIMIT = new BigDecimal("50.00");
    private static final BigDecimal MANUAL_REVIEW_LIMIT = new BigDecimal("300.00");

    public RefundCase evaluate(RefundRequest request) {
        if (!request.receiptPresent()) {
            return new RefundCase(request,
                    new RefundDecision(DecisionOutcome.DENIED, "receipt is missing"));
        }

        if (request.amount().compareTo(AUTO_APPROVE_LIMIT) <= 0 && request.chargebackCount() == 0) {
            return new RefundCase(request,
                    new RefundDecision(DecisionOutcome.APPROVED, "small refund with clean history"));
        }

        if (request.amount().compareTo(MANUAL_REVIEW_LIMIT) >= 0
                || request.accountAgeDays() < 30
                || request.chargebackCount() > 1) {
            return new RefundCase(request,
                    new RefundDecision(DecisionOutcome.MANUAL_REVIEW, "manual review required"));
        }

        return new RefundCase(request,
                new RefundDecision(DecisionOutcome.APPROVED, "standard refund approved"));
    }
}

This class has no workflow dependency. That is the point. You can unit test it without Kafka, Flow, REST, or a running Quarkus application.

If your organization already has a rules engine, this class is the seam where it would go. The workflow only cares that evaluate() returns a RefundCase, whether the decision came from Drools, a DMN service, or a switch statement.

Store Decisions

We need a small store so the HTTP API can show the final result after the workflow finishes. This uses memory because the tutorial is about workflow state, not database schema design. In a real refund system, this would be a database table with a uniqueness constraint on refundId.

Create src/main/java/dev/mainthread/refunddesk/DecisionStore.java:

package dev.mainthread.refunddesk;

import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;

import jakarta.enterprise.context.ApplicationScoped;

@ApplicationScoped
public class DecisionStore {

    private final ConcurrentMap<String, RefundResult> results = new ConcurrentHashMap<>();

    public void recordAutomatic(RefundCase refundCase) {
        RefundRequest request = refundCase.request();
        RefundDecision decision = refundCase.decision();

        results.put(request.refundId(),
                new RefundResult(request.refundId(), decision.outcome(), decision.reason(), "policy"));
    }

    public void recordManualReview(ReviewDecision review) {
        if (DecisionOutcome.MANUAL_REVIEW.equals(review.outcome())) {
            throw new IllegalArgumentException("A reviewer must approve or deny the refund");
        }

        results.put(review.refundId(),
                new RefundResult(review.refundId(), review.outcome(), review.note(), review.reviewer()));
    }

    public Optional<RefundResult> find(String refundId) {
        return Optional.ofNullable(results.get(refundId));
    }
}

There is one production warning worth saying here, and only here: this store is not an idempotency boundary. A real refund system should enforce one final decision per refundId in the database. Quarkus Flow can propagate correlation data, but the receiver of a side effect still owns deduplication.

The Quarkus Flow idempotency and correlation guide is clear about the limit: Flow gives every workflow instance a unique identity and propagates correlation metadata for HTTP and CloudEvents. Exactly-once behavior still belongs at the receiver, with stable business keys and storage constraints. Anything claiming otherwise usually hides the invoice in another room.

Build the Workflow

Now we can write the orchestration. It has five moves:

1. Run the policy

2. Branch on the policy result

3. Record automatic decisions

4. Emit a review request for manual cases

5. Wait for a review callback and record it

Create src/main/java/dev/mainthread/refunddesk/RefundWorkflow.java:

package dev.mainthread.refunddesk;

import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.consume;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.consumed;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.emitJson;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.function;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.listen;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.switchWhenOrElse;
import static io.serverlessworkflow.fluent.func.dsl.FuncDSL.toOne;

import jakarta.enterprise.context.ApplicationScoped;

import io.quarkiverse.flow.Flow;
import io.serverlessworkflow.api.types.FlowDirectiveEnum;
import io.serverlessworkflow.api.types.Workflow;
import io.serverlessworkflow.fluent.func.FuncWorkflowBuilder;

@ApplicationScoped
public class RefundWorkflow extends Flow {

    private final RefundPolicy policy;
    private final DecisionStore decisions;

    public RefundWorkflow(RefundPolicy policy, DecisionStore decisions) {
        this.policy = policy;
        this.decisions = decisions;
    }

    @Override
    public Workflow descriptor() {
        return FuncWorkflowBuilder.workflow("refund-review", "refunddesk", "1.0.0")
                .tasks(
                        function("evaluatePolicy", policy::evaluate, RefundRequest.class),
                        switchWhenOrElse(this::needsManualReview, "requestManualReview", "finishAutomatic",
                                RefundCase.class),
                        emitJson("requestManualReview", "refund.review.required", RefundCase.class),
                        listen("waitForReview",
                                toOne(consumed("refund.review.completed").extensionByInstanceId("flowinstanceid"))),
                        consume("finishManualReview", decisions::recordManualReview, ReviewDecision.class)
                                .then(FlowDirectiveEnum.END),
                        consume("finishAutomatic", decisions::recordAutomatic, RefundCase.class)
                                .then(FlowDirectiveEnum.END))
                .build();
    }

    private boolean needsManualReview(RefundCase refundCase) {
        return DecisionOutcome.MANUAL_REVIEW.equals(refundCase.decision().outcome());
    }
}

This is where the workflow actually happens.

function("evaluatePolicy", ...) calls normal Java code. The result becomes the workflow data for the next step. switchWhenOrElse(...) decides which task name runs next. Automatic decisions jump to finishAutomatic. Manual-review decisions jump to requestManualReview.

emitJson(...) publishes a structured CloudEvent to the Flow outbound channel. In a real system, a reviewer UI or back-office queue would consume that event. Then listen(...) pauses the workflow until a refund.review.completed CloudEvent reaches the engine.

The correlation part is in this line:

toOne(consumed("refund.review.completed").extensionByInstanceId("flowinstanceid"))

Flow will only resume this waiting workflow when the callback event has the matching flowinstanceid CloudEvent extension. That is the difference between “some review happened” and “the review for this workflow instance happened.”

The two then(FlowDirectiveEnum.END) calls are boring and easy to miss. After a manual review is recorded, the workflow must end. After an automatic decision is recorded, it must end. If you remove those endings, you have built a confusing process with excellent syntax highlighting.

Expose the Workflow Over HTTP

The REST resource starts workflow instances, reads final decisions, and turns reviewer callbacks into CloudEvents.

Create src/main/java/dev/mainthread/refunddesk/RefundResource.java:

package dev.mainthread.refunddesk;

import static jakarta.ws.rs.core.MediaType.APPLICATION_JSON;

import java.net.URI;
import java.util.Map;
import java.util.UUID;

import org.eclipse.microprofile.reactive.messaging.Channel;
import org.eclipse.microprofile.reactive.messaging.Emitter;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

import io.cloudevents.core.builder.CloudEventBuilder;
import io.cloudevents.core.provider.EventFormatProvider;
import io.cloudevents.jackson.JsonFormat;
import io.serverlessworkflow.impl.WorkflowInstance;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.Response;
import jakarta.ws.rs.core.Response.Status;

@Path("/refunds")
@ApplicationScoped
@Consumes(APPLICATION_JSON)
@Produces(APPLICATION_JSON)
public class RefundResource {

    private static final JsonFormat CE_JSON = (JsonFormat) EventFormatProvider.getInstance()
            .resolveFormat(JsonFormat.CONTENT_TYPE);

    @Inject
    RefundWorkflow workflow;

    @Inject
    DecisionStore decisions;

    @Inject
    ObjectMapper objectMapper;

    @Inject
    @Channel("flow-in-outgoing")
    Emitter<byte[]> flowIn;

    @POST
    public Response submit(RefundRequest request) {
        WorkflowInstance instance = workflow.instance(request);
        instance.start();

        return Response.accepted(new SubmitRefundResponse(request.refundId(), instance.id())).build();
    }

    @GET
    @Path("/{refundId}")
    public Response result(@PathParam("refundId") String refundId) {
        return decisions.find(refundId)
                .map(result -> Response.ok(result).build())
                .orElseGet(() -> Response.status(Status.NOT_FOUND).build());
    }

    @POST
    @Path("/{refundId}/review/{instanceId}")
    public Response review(@PathParam("refundId") String refundId,
            @PathParam("instanceId") String instanceId,
            ReviewDecision review) throws JsonProcessingException {

        if (!refundId.equals(review.refundId())) {
            return Response.status(Status.BAD_REQUEST)
                    .entity(Map.of("error", "path refundId and review refundId must match"))
                    .build();
        }

        byte[] body = objectMapper.writeValueAsBytes(review);
        byte[] event = CE_JSON.serialize(CloudEventBuilder.v1()
                .withId(UUID.randomUUID().toString())
                .withSource(URI.create("urn:refunddesk:review-api"))
                .withType("refund.review.completed")
                .withDataContentType(APPLICATION_JSON)
                .withExtension("flowinstanceid", instanceId)
                .withData(body)
                .build());

        flowIn.send(event);
        return Response.accepted().build();
    }
}

The callback endpoint writes a structured CloudEvent as JSON bytes. Notice the flowinstanceid extension. That value is the routing key for the waiting workflow instance.

The business key still matters. We keep refundId in the URL and in the event data because your domain system should know which refund was reviewed. The workflow instance ID wakes the process. The refund ID protects the business record.

Also notice that the callback writes to a channel named flow-in-outgoing, not directly to flow-in. In the configuration below, flow-in-outgoing is an outgoing producer that writes to the same Kafka topic that Flow consumes as flow-in. Keeping the app producer and Flow consumer as separate Reactive Messaging channels makes the wiring easier to reason about.

Configure Messaging, Persistence, and Tracing

Configure the application in src/main/resources/application.properties:

quarkus.application.name=refunddesk-flow

quarkus.flow.messaging.defaults-enabled=true

mp.messaging.incoming.flow-in.connector=smallrye-kafka
mp.messaging.incoming.flow-in.topic=refunddesk-flow-in
mp.messaging.incoming.flow-in.value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
mp.messaging.incoming.flow-in.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
mp.messaging.incoming.flow-in.auto.offset.reset=earliest

mp.messaging.outgoing.flow-out.connector=smallrye-kafka
mp.messaging.outgoing.flow-out.topic=refunddesk-flow-out
mp.messaging.outgoing.flow-out.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
mp.messaging.outgoing.flow-out.key.serializer=org.apache.kafka.common.serialization.StringSerializer

mp.messaging.outgoing.flow-in-outgoing.connector=smallrye-kafka
mp.messaging.outgoing.flow-in-outgoing.topic=refunddesk-flow-in
mp.messaging.outgoing.flow-in-outgoing.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
mp.messaging.outgoing.flow-in-outgoing.key.serializer=org.apache.kafka.common.serialization.StringSerializer

quarkus.flow.persistence.mvstore.db-path=target/refunddesk-flow.mv.db

%dev.quarkus.flow.tracing.enabled=true
%test.quarkus.flow.tracing.enabled=true

quarkus.flow.messaging.defaults-enabled=true turns on the Flow messaging bridge. Flow reads structured CloudEvents from flow-in and publishes workflow events to flow-out. Because messaging-kafka is on the classpath, Quarkus Dev Services starts Kafka for local development when no broker is configured.

The flow-in-outgoing channel writes to the same refunddesk-flow-in topic. That lets the REST callback endpoint wake the waiting workflow through the same external event path a real reviewer UI would use.

The MVStore path keeps workflow checkpoints in target/. This is fine for a tutorial and local testing. For production, put workflow state in a durable store outside the application filesystem.

Tracing is enabled in dev and test so you can see which workflow and task ran. Quarkus Flow’s tracing docs also show how to emit MDC fields for log correlation. The short version: use the workflow instance ID when you search logs. It is the thread that ties the process together after the process stops being one HTTP request.

Run the App

Start dev mode:

./mvnw quarkus:dev

Submit a small refund:

curl -i -X POST http://localhost:8080/refunds \
  -H 'Content-Type: application/json' \
  -d '{
    "refundId": "refund-1001",
    "customerId": "customer-7",
    "amount": 42.00,
    "receiptPresent": true,
    "accountAgeDays": 120,
    "chargebackCount": 0
  }'

Expected response:

{
  "refundId": "refund-1001",
  "workflowInstanceId": "01J..."
}

The endpoint always returns 202 Accepted because the workflow may finish immediately or may wait for a later event. That stable contract is easier for clients. The result lives behind GET /refunds/{refundId}.

Check the result:

curl http://localhost:8080/refunds/refund-1001

Expected response:

{
  "refundId": "refund-1001",
  "outcome": "APPROVED",
  "reason": "small refund with clean history",
  "reviewer": "policy"
}

Try a denied refund:

curl -i -X POST http://localhost:8080/refunds \
  -H 'Content-Type: application/json' \
  -d '{
    "refundId": "refund-1002",
    "customerId": "customer-8",
    "amount": 19.99,
    "receiptPresent": false,
    "accountAgeDays": 400,
    "chargebackCount": 0
  }'

curl http://localhost:8080/refunds/refund-1002

Expected result:

{
  "refundId": "refund-1002",
  "outcome": "DENIED",
  "reason": "receipt is missing",
  "reviewer": "policy"
}

Now submit a refund that needs a person:

curl -i -X POST http://localhost:8080/refunds \
  -H 'Content-Type: application/json' \
  -d '{
    "refundId": "refund-1003",
    "customerId": "customer-9",
    "amount": 450.00,
    "receiptPresent": true,
    "accountAgeDays": 15,
    "chargebackCount": 2
  }'

Keep the workflowInstanceId from the response. Before the review arrives, the result is not ready:

curl -i http://localhost:8080/refunds/refund-1003

Expected status:

HTTP/1.1 404 Not Found

For the manual curl path, wait until the workflow has reached the review step before sending the callback. In dev mode you will see requestManualReview complete and waitForReview start in the log. A real reviewer UI would not guess this timing; it would consume the refund.review.required event from flow-out, store its flowinstanceid, and use that value for the callback.

Send the review callback. Replace 01J... with the workflow instance ID returned by the submit call:

curl -i -X POST http://localhost:8080/refunds/refund-1003/review/01J... \
  -H 'Content-Type: application/json' \
  -d '{
    "refundId": "refund-1003",
    "outcome": "APPROVED",
    "reviewer": "alex",
    "note": "receipt and order history checked"
  }'

Check the result again:

curl http://localhost:8080/refunds/refund-1003

Expected response:

{
  "refundId": "refund-1003",
  "outcome": "APPROVED",
  "reason": "receipt and order history checked",
  "reviewer": "alex"
}

That is the workflow shape a rules engine does not cover by itself. The policy decision was one task. The process around the decision crossed an event boundary, waited, resumed, and recorded the final state.

Test the Policy First

Test the policy outside Quarkus. This is a normal unit test, and that is a feature.

Create src/test/java/dev/mainthread/refunddesk/RefundPolicyTest.java:

package dev.mainthread.refunddesk;

import static org.junit.jupiter.api.Assertions.assertEquals;

import java.math.BigDecimal;

import org.junit.jupiter.api.Test;

class RefundPolicyTest {

    private final RefundPolicy policy = new RefundPolicy();

    @Test
    void approvesSmallCleanRefund() {
        RefundCase refundCase = policy.evaluate(new RefundRequest(
                "refund-2001",
                "customer-1",
                new BigDecimal("25.00"),
                true,
                120,
                0));

        assertEquals(DecisionOutcome.APPROVED, refundCase.decision().outcome());
        assertEquals("small refund with clean history", refundCase.decision().reason());
    }

    @Test
    void deniesMissingReceipt() {
        RefundCase refundCase = policy.evaluate(new RefundRequest(
                "refund-2002",
                "customer-2",
                new BigDecimal("25.00"),
                false,
                120,
                0));

        assertEquals(DecisionOutcome.DENIED, refundCase.decision().outcome());
    }

    @Test
    void routesSuspiciousRefundToManualReview() {
        RefundCase refundCase = policy.evaluate(new RefundRequest(
                "refund-2003",
                "customer-3",
                new BigDecimal("450.00"),
                true,
                10,
                2));

        assertEquals(DecisionOutcome.MANUAL_REVIEW, refundCase.decision().outcome());
    }
}

Run it:

./mvnw test -Dtest=RefundPolicyTest

This test does not prove the workflow. It proves the decision logic. That separation keeps the article honest: a rules engine would sit at the same boundary.

Test the Workflow Through HTTP

Now test the process. This test starts the app, submits refunds, reads the emitted review request from flow-out, sends a review callback, and polls for the final result.

Add two test dependencies to pom.xml:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-test-kafka-companion</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.awaitility</groupId>
    <artifactId>awaitility</artifactId>
    <scope>test</scope>
</dependency>

Create src/test/resources/application.properties:

quarkus.flow.messaging.defaults-enabled=true

mp.messaging.incoming.flow-in.auto.offset.reset=earliest

quarkus.flow.persistence.mvstore.db-path=target/refunddesk-flow-test.mv.db
quarkus.flow.persistence.auto-restore=false
quarkus.flow.tracing.enabled=true

Create src/test/java/dev/mainthread/refunddesk/RefundResourceTest.java:

package dev.mainthread.refunddesk;

import static io.restassured.RestAssured.given;
import static java.time.Duration.ofSeconds;
import static org.awaitility.Awaitility.await;
import static org.hamcrest.Matchers.equalTo;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.fail;

import java.math.BigDecimal;
import java.util.concurrent.atomic.AtomicReference;

import jakarta.inject.Inject;

import org.junit.jupiter.api.Test;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;

import com.fasterxml.jackson.databind.ObjectMapper;

import io.cloudevents.CloudEvent;
import io.cloudevents.core.provider.EventFormatProvider;
import io.cloudevents.jackson.JsonFormat;
import io.quarkus.test.common.QuarkusTestResource;
import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.kafka.InjectKafkaCompanion;
import io.quarkus.test.kafka.KafkaCompanionResource;
import io.restassured.http.ContentType;
import io.restassured.response.Response;
import io.smallrye.reactive.messaging.kafka.companion.ConsumerTask;
import io.smallrye.reactive.messaging.kafka.companion.KafkaCompanion;

@QuarkusTest
@QuarkusTestResource(KafkaCompanionResource.class)
class RefundResourceTest {

    private static final JsonFormat CE_JSON = (JsonFormat) EventFormatProvider.getInstance()
            .resolveFormat(JsonFormat.CONTENT_TYPE);

    @InjectKafkaCompanion
    KafkaCompanion companion;

    @Inject
    ObjectMapper objectMapper;

    @Test
    void autoApprovesSmallRefund() {
        given()
                .contentType(ContentType.JSON)
                .body(new RefundRequest("refund-test-1", "customer-1", new BigDecimal("42.00"), true, 120, 0))
                .when()
                .post("/refunds")
                .then()
                .statusCode(202);

        Response result = waitForResult("refund-test-1");

        result.then()
                .statusCode(200)
                .body("outcome", equalTo("APPROVED"))
                .body("reviewer", equalTo("policy"));
    }

    @Test
    void resumesManualReviewFromCallbackEvent() throws Exception {
        ConsumerTask<Object, Object> flowOut = companion
                .consumeWithDeserializers(StringDeserializer.class, ByteArrayDeserializer.class)
                .fromTopics("refunddesk-flow-out");

        given()
                .contentType(ContentType.JSON)
                .body(new RefundRequest("refund-test-2", "customer-2", new BigDecimal("450.00"), true, 10, 2))
                .when()
                .post("/refunds")
                .then()
                .statusCode(202);

        CloudEvent reviewRequired = waitForReviewRequiredEvent(flowOut);
        assertEquals("refund.review.required", reviewRequired.getType());
        assertNotNull(reviewRequired.getExtension("flowinstanceid"));

        RefundCase refundCase = objectMapper.readValue(reviewRequired.getData().toBytes(), RefundCase.class);
        assertEquals("refund-test-2", refundCase.request().refundId());

        given()
                .contentType(ContentType.JSON)
                .body(new ReviewDecision("refund-test-2", DecisionOutcome.APPROVED, "test-reviewer", "checked"))
                .when()
                .post("/refunds/{refundId}/review/{instanceId}",
                        "refund-test-2",
                        reviewRequired.getExtension("flowinstanceid").toString())
                .then()
                .statusCode(202);

        Response result = waitForResult("refund-test-2");

        result.then()
                .statusCode(200)
                .body("outcome", equalTo("APPROVED"))
                .body("reviewer", equalTo("test-reviewer"));

        flowOut.close();
    }

    @Test
    void rejectsMismatchedReviewCallback() {
        given()
                .contentType(ContentType.JSON)
                .body(new ReviewDecision("other-refund", DecisionOutcome.APPROVED, "test-reviewer", "checked"))
                .when()
                .post("/refunds/{refundId}/review/{instanceId}",
                        "refund-test-3",
                        "01JFAKEINSTANCE")
                .then()
                .statusCode(400)
                .body("error", equalTo("path refundId and review refundId must match"));
    }

    private Response waitForResult(String refundId) {
        for (int attempt = 0; attempt < 50; attempt++) {
            Response response = given()
                    .accept(ContentType.JSON)
                    .when()
                    .get("/refunds/{refundId}", refundId);

            if (response.statusCode() == 200) {
                return response;
            }

            sleep();
        }

        fail("Timed out waiting for refund result " + refundId);
        return null;
    }

    private void sleep() {
        try {
            Thread.sleep(100);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IllegalStateException(e);
        }
    }

    private CloudEvent waitForReviewRequiredEvent(ConsumerTask<Object, Object> flowOut) {
        AtomicReference<CloudEvent> reviewRequired = new AtomicReference<>();

        await().atMost(ofSeconds(10)).untilAsserted(() -> {
            flowOut.stream().forEach(record -> {
                CloudEvent event = CE_JSON.deserialize((byte[]) record.value());
                if ("refund.review.required".equals(event.getType())) {
                    reviewRequired.set(event);
                }
            });
            assertNotNull(reviewRequired.get());
        });

        return reviewRequired.get();
    }
}

Run all tests:

./mvnw test

The manual-review test deliberately consumes refund.review.required before it sends the callback. That order matters. The event proves the workflow reached the wait point and gives the test the same flowinstanceid a reviewer UI would store with its work item.

What Flow Guarantees, and What It Does Not

The workflow now waits and resumes, but the boundary is still important.

Quarkus Flow assigns each workflow instance a unique ID. It can propagate that ID through HTTP headers and CloudEvent extension attributes. In our callback event, flowinstanceid tells Flow which waiting instance to resume.

Flow treats independent workflow instances as independent work. If the same refund.requested event arrives twice and you start two workflows, you have two workflow instances. Your application still needs a business idempotency key such as refundId and a database constraint that makes duplicate final decisions impossible.

Flow also gives you at-least-once task execution, not exactly-once execution. If a task calls an external service and the network fails at the wrong moment, the receiver must be safe to call again. Use stable business keys for operations such as payments, refunds, ledger writes, and emails.

That may sound disappointing until you compare it with the alternative: hiding the same distributed-systems problem behind a rule called approveRefund. At least here the boundary is visible.

Make the Workflow Survive a Restart

With quarkus-flow-mvstore on the classpath and the MVStore path configured, Flow checkpoints state after completed tasks and paused steps. You can test the practical effect:

1. Start the app with ./mvnw quarkus:dev

2. Submit refund-1003, the manual-review case

3. Stop dev mode with Ctrl+C

4. Start dev mode again

5. Send the review callback using the same workflowInstanceId from the curl response, or the flowinstanceid from flow-out if you are testing through a reviewer event consumer

6. Read GET /refunds/refund-1003

The waiting workflow should resume because the workflow state was persisted. The DecisionStore in this article is still in memory, so it only records the final result after the callback arrives. If you want the submitted refund record itself to survive restarts, put the domain state in a database too.

For production, the Quarkus Flow persistence guide gives three provider paths:

• Redis for distributed workloads

• JPA for relational databases

• MVStore for local development, tests, and single-node deployments

Pick one persistence provider. Running several persistence implementations in the same app is a good way to make future-you read source code with bad coffee.

Use the Runner When the Workflow Is Configuration

The Java DSL is the better fit for RefundDesk because the workflow calls CDI beans and domain code. Quarkus Flow 0.11.0 also ships pre-built Runner images for a different case: you have workflow definitions as YAML or JSON and want to run them as configuration.

Use the Runner when a platform team wants to mount workflows from GitOps, ConfigMaps, or another external workflow authoring path. Use the Java DSL when the workflow is tightly coupled to Quarkus beans and you want compile-time help.

Create a small policy-only workflow file under a new local directory:

mkdir -p workflows

Create workflows/refund-policy.yaml:

document:
  dsl: "1.0.0"
  namespace: refunddesk
  name: refund-policy
  version: "1.0.0"
input:
  schema:
    format: json
    document:
      type: object
      required:
        - amount
        - receiptPresent
        - chargebackCount
      properties:
        amount:
          type: number
        receiptPresent:
          type: boolean
        chargebackCount:
          type: integer
do:
  - route:
      switch:
        - missingReceipt:
            when: .receiptPresent == false
            then: deny
        - smallClean:
            when: .amount <= 50 and .chargebackCount == 0
            then: approve
        - default:
            then: manualReview
  - approve:
      set:
        outcome: APPROVED
        reason: small refund with clean history
      then: exit
  - deny:
      set:
        outcome: DENIED
        reason: receipt is missing
      then: exit
  - manualReview:
      set:
        outcome: MANUAL_REVIEW
        reason: manual review required
      then: exit

Run the minimal Runner image:

podman run --rm \
  --name refunddesk-runner \
  -p 8081:8080 \
  -v "$PWD/workflows:/deployments/workflows:ro" \
  quay.io/quarkiverse/quarkus-flow-runner:0.11.0-minimal

In another terminal, execute the workflow:

curl -X POST 'http://localhost:8081/q/flow/exec/refunddesk/refund-policy/1.0.0?wait=true' \
  -H 'Content-Type: application/json' \
  -d '{
    "amount": 42.00,
    "receiptPresent": true,
    "chargebackCount": 0
  }'

Expected output:

{
  "instanceId": "01KW65RE8KKQPW7FT6CQY7QS5Y",
  "status": "COMPLETED",
  "startedAt": "2026-06-28T03:53:36.274777328Z",
  "completedAt": "2026-06-28T03:53:36.306582193Z",
  "workflowOutput": {
    "reason": "small refund with clean history",
    "outcome": "APPROVED"
  }
}

This Runner version is intentionally smaller than the Java app. It shows the release feature without pretending YAML is always the right abstraction. The Runner can execute workflow definitions mounted at runtime. The Java app can call your CDI beans, keep policy code in normal Java, use tests, and participate in the rest of your Quarkus application.

That split is useful. Some workflows are platform configuration. Some workflows are application code. Treating both as the same thing usually makes one team unhappy.

Where @ScheduleOn Fits

Quarkus Flow 0.11.0 also adds @ScheduleOn for agentic LangChain4j workflows. That is not part of RefundDesk because this example does not need agents.

The feature matters when an agentic method should wake from an event, a cron schedule, or a fixed interval. For example, a nightly refund-anomaly reviewer could run every hour, fetch suspicious cases, and produce a report. The Quarkus Flow @ScheduleOn guide documents three trigger styles:

• event for a CloudEvent type

• cron for a Unix cron expression

• every for an ISO 8601 duration

The same design rule applies: use the agent for the task that needs language or fuzzy reasoning. Use the workflow for the process that needs state, waiting, and evidence.

Production Notes That Matter

RefundDesk is small, but it creates real production questions.

Use a database for final decisions. The ConcurrentHashMap keeps the tutorial readable, but the business boundary belongs in storage with a unique key on refundId.

Use idempotency keys at receivers. If recordManualReview() became a real refund transaction, the downstream payment or ledger API should reject duplicate refundId operations or return the already-recorded result.

Keep workflow state outside the pod. MVStore is fine for local testing. Use Redis, JPA, or the production Runner variants when workflow state must survive node drains, pod replacement, and rolling updates.

Treat flow-out as part of the contract. The review UI should consume refund.review.required, store the flowinstanceid, and send it back on refund.review.completed. Guessing instance IDs from an HTTP response is a tutorial convenience.

Use structured logs or tracing before the first incident. Quarkus Flow can emit workflow and task lifecycle data. Searching by workflow instance ID is much better than stitching together five log lines by timestamp and hope.

Use Quartz for clustered schedules. The normal scheduler is enough for simple standalone deployments. The Quarkus Flow Quartz scheduler docs call out persistent Quartz for clustered scheduled workflows.

Conclusion

RefundDesk keeps the policy small and lets the workflow own the process around it: branching, event emission, waiting, correlation, and final recording. That is the mental model I would keep: rules decide from facts, workflows coordinate what happens over time.

Subscribe now

With the new name Quarkus Data, future relational database work starts with the quarkus-data-hibernate extension. The Quarkus announcement positions it as an umbrella for data access in Quarkus, built around the Jakarta Data direction and the Hibernate processor. I like the new name. “Panache” only meant something to you, if you’d already knew what it was all about. “Data” is what people search for when they are starting a service and need a database.

The API is still experimental in Quarkus 3.37.0, so keep that in mind. Names can still move. But the direction is clear enough to build with, test, and judge by behavior. The team is still looking for feedback about it. If you find something, make sure to report and let the team know.

We will build CatalogBoard, a small product catalog API. It is deliberately plain: products have a SKU, category, stock count, reorder point, and a discontinued flag. That gives us enough room to show generated finder methods, HQL queries checked at build time, managed updates, one explicit stateless update, pagination, validation, and tests.

When we are done, the API can create products, list them by category, search by name, flag low stock, adjust inventory, and hide discontinued products.

Prerequisites

You need a recent Quarkus setup and a container runtime for PostgreSQL Dev Services. The example uses Quarkus 3.37.0 and Java 21.

• Java 25

• Quarkus CLI

• Podman or another Testcontainers-compatible runtime

• Basic Jakarta Persistence and REST knowledge

• About three ☕️

Quarkus Data Hibernate is experimental in this release. That works for a tutorial and for early project exploration. For a production service, put that status in the decision record so nobody has to rediscover it during an upgrade.

Create The Project

Create the app and follow along or grab the code from my repository:

quarkus create app io.mainthread.catalogboard:catalog-board \
  --extension=quarkus-data-hibernate,rest-jackson,jdbc-postgresql,hibernate-validator \
  --no-code
cd catalog-board

Use these extensions:

• quarkus-data-hibernate - the new Quarkus Data Hibernate entry point for relational data access

• rest-jackson - JSON REST endpoints

• jdbc-postgresql - PostgreSQL connectivity and Dev Services

• hibernate-validator - validation on request bodies and query parameters

The generated project already has the platform-managed dependencies. Quarkus Data Hibernate also needs the Hibernate annotation processor so the repository implementations and query checks happen during compilation. Add this to the maven-compiler-plugin configuration in pom.xml:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>${compiler-plugin.version}</version>
    <configuration>
        <parameters>true</parameters>
        <annotationProcessorPathsUseDepMgmt>true</annotationProcessorPathsUseDepMgmt>
        <annotationProcessorPaths>
            <path>
                <groupId>org.hibernate.orm</groupId>
                <artifactId>hibernate-processor</artifactId>
            </path>
        </annotationProcessorPaths>
    </configuration>
</plugin>

The annotationProcessorPathsUseDepMgmt flag lets the Quarkus BOM manage the processor version. Without the processor, the repository interfaces stay as interfaces. The generated implementation never appears.

Configure PostgreSQL

Add src/main/resources/application.properties:

quarkus.datasource.db-kind=postgresql

%dev.quarkus.datasource.devservices.image-name=docker.io/library/postgres:17
%test.quarkus.datasource.devservices.image-name=docker.io/library/postgres:17

%dev.quarkus.hibernate-orm.schema-management.strategy=drop-and-create
%test.quarkus.hibernate-orm.schema-management.strategy=drop-and-create
%prod.quarkus.hibernate-orm.schema-management.strategy=validate

%prod.quarkus.datasource.username=${DB_USERNAME}
%prod.quarkus.datasource.password=${DB_PASSWORD}
%prod.quarkus.datasource.jdbc.url=${DB_JDBC_URL}

quarkus.hibernate-orm.log.sql=false

Dev and test mode use PostgreSQL Dev Services. The schema is recreated on startup so the tutorial stays repeatable. Production uses validate because the application should not mutate real tables at startup. Use Flyway or Liquibase there.

The db-kind property is the one setting we keep outside a profile. It tells Quarkus which datasource to prepare even before a real URL exists.

Add The Entity And Repositories

Create src/main/java/io/mainthread/catalogboard/Product.java:

package io.mainthread.catalogboard;

import java.util.List;
import java.util.Optional;

import org.hibernate.annotations.processing.Find;
import org.hibernate.annotations.processing.HQL;

import io.quarkus.hibernate.panache.PanacheEntity;
import io.quarkus.hibernate.panache.PanacheRepository;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.Table;
import jakarta.persistence.UniqueConstraint;

@Entity
@Table(name = "products", uniqueConstraints = @UniqueConstraint(columnNames = "sku"))
public class Product extends PanacheEntity {

    @Column(nullable = false, length = 32)
    public String sku;

    @Column(nullable = false, length = 120)
    public String name;

    @Column(nullable = false, length = 80)
    public String category;

    @Column(nullable = false)
    public int stock;

    @Column(nullable = false)
    public int reorderPoint;

    @Column(nullable = false)
    public boolean discontinued;

    public boolean needsRestock() {
        return !discontinued && stock <= reorderPoint;
    }

    public interface Repo extends PanacheRepository<Product> {

        @Find
        Optional<Product> findBySku(String sku);

        @HQL("where category = :category and discontinued = false order by name")
        List<Product> findAvailableByCategory(String category);

        @HQL("where name like :pattern and discontinued = false order by name")
        List<Product> searchByName(String pattern);

        @HQL("where stock <= reorderPoint and discontinued = false order by stock, sku")
        List<Product> findLowStock();

        @HQL("delete from Product where discontinued = true")
        long deleteDiscontinued();
    }

    public interface InventoryRepo extends PanacheRepository.Stateless<Product, Long> {

        @Find
        Optional<Product> findBySku(String sku);

        @HQL("update Product set stock = stock + :delta where sku = :sku and discontinued = false")
        long changeStock(String sku, int delta);
    }
}

This file shows the core Quarkus Data model. The entity is ordinary Jakarta Persistence plus the Quarkus Data PanacheEntity base type. The nested repository interfaces are where the generated code appears.

@Find derives a query from the method name and parameters. findBySku(String sku) becomes a query against the sku field. Returning Optional<Product> is intentional. If the method returns Product, absence becomes a NoResultException. For a REST lookup, Optional gives the resource a cleaner boundary.

@HQL gives us explicit queries while still letting the Hibernate processor check the entity name, field names, syntax, and parameter names during compilation. Change reorderPoint to reorderLevel in the entity and forget to update the query, and the build catches it.

The second repository is stateless. Managed Hibernate tracks entity changes and flushes dirty state at transaction boundaries. A stateless repository does less tracking, so write operations stay explicit. changeStock is an update query because adjusting inventory is a command. That path should not depend on “load, mutate, hope somebody remembered the flush.”

Quarkus Data also supports Jakarta Data annotations such as jakarta.data.repository.Find, Query, and Delete. The current Quarkus Data Hibernate guide shows both styles. I use the Hibernate annotations here because we are writing HQL and staying close to Hibernate ORM.

Add Request And Response Types

Keep JSON payloads away from the entity. The entity is the persistence model. The REST API can stay smaller.

Create ProductCreateRequest.java:

package io.mainthread.catalogboard;

import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Size;

public record ProductCreateRequest(
        @NotBlank @Size(max = 32) String sku,
        @NotBlank @Size(max = 120) String name,
        @NotBlank @Size(max = 80) String category,
        @Min(0) int stock,
        @Min(0) int reorderPoint) {
}

Create ProductUpdateRequest.java:

package io.mainthread.catalogboard;

import jakarta.validation.constraints.Min;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Size;

public record ProductUpdateRequest(
        @NotBlank @Size(max = 120) String name,
        @NotBlank @Size(max = 80) String category,
        @Min(0) int reorderPoint) {
}

Create StockAdjustment.java:

package io.mainthread.catalogboard;

import jakarta.validation.constraints.Min;

public record StockAdjustment(@Min(-1000) int delta) {
}

Create ProductResponse.java:

package io.mainthread.catalogboard;

public record ProductResponse(
        Long id,
        String sku,
        String name,
        String category,
        int stock,
        int reorderPoint,
        boolean discontinued,
        boolean needsRestock) {

    static ProductResponse from(Product product) {
        return new ProductResponse(
                product.id,
                product.sku,
                product.name,
                product.category,
                product.stock,
                product.reorderPoint,
                product.discontinued,
                product.needsRestock());
    }
}

The create request includes sku; the update request does not. That prevents clients from changing the natural lookup key through a metadata update. You can pick a different boundary in your system, but it should be a deliberate choice.

Add The REST Resource

Create src/main/java/io/mainthread/catalogboard/ProductResource.java:

package io.mainthread.catalogboard;

import java.net.URI;
import java.util.List;

import io.quarkus.hibernate.panache.blocking.PanacheBlockingQuery;
import jakarta.data.Order;
import jakarta.data.Sort;
import jakarta.inject.Inject;
import jakarta.transaction.Transactional;
import jakarta.validation.Valid;
import jakarta.validation.constraints.Max;
import jakarta.validation.constraints.Min;
import jakarta.ws.rs.BadRequestException;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.DELETE;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.NotFoundException;
import jakarta.ws.rs.PATCH;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.PUT;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;

@Path("/products")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class ProductResource {

    @Inject
    Product.Repo repo;

    @Inject
    Product.InventoryRepo inventoryRepo;

    @GET
    public List<ProductResponse> list(
            @QueryParam("category") String category,
            @QueryParam("page") @DefaultValue("0") @Min(0) int page,
            @QueryParam("size") @DefaultValue("20") @Min(1) @Max(100) int size) {
        PanacheBlockingQuery<Product> query = category == null || category.isBlank()
                ? repo.find("discontinued = false", Order.by(Sort.asc("name")))
                : repo.find("category = ?1 and discontinued = false", Order.by(Sort.asc("name")), category);

        return query.page(page, size)
                .list()
                .stream()
                .map(ProductResponse::from)
                .toList();
    }

    @GET
    @Path("/{sku}")
    public ProductResponse get(@PathParam("sku") String sku) {
        return ProductResponse.from(requireProduct(sku));
    }

    @GET
    @Path("/search")
    public List<ProductResponse> search(@QueryParam("q") @DefaultValue("") String query) {
        if (query.isBlank()) {
            return List.of();
        }
        return repo.searchByName("%" + query + "%")
                .stream()
                .map(ProductResponse::from)
                .toList();
    }

    @GET
    @Path("/low-stock")
    public List<ProductResponse> lowStock() {
        return repo.findLowStock()
                .stream()
                .map(ProductResponse::from)
                .toList();
    }

    @POST
    @Transactional
    public Response create(@Valid ProductCreateRequest request) {
        if (repo.findBySku(request.sku()).isPresent()) {
            throw new BadRequestException("SKU already exists: " + request.sku());
        }

        Product product = new Product();
        product.sku = request.sku();
        product.name = request.name();
        product.category = request.category();
        product.stock = request.stock();
        product.reorderPoint = request.reorderPoint();
        repo.persist(product);

        return Response.created(URI.create("/products/" + product.sku))
                .entity(ProductResponse.from(product))
                .build();
    }

    @PUT
    @Path("/{sku}")
    @Transactional
    public ProductResponse update(@PathParam("sku") String sku, @Valid ProductUpdateRequest request) {
        Product product = requireProduct(sku);
        product.name = request.name();
        product.category = request.category();
        product.reorderPoint = request.reorderPoint();
        return ProductResponse.from(product);
    }

    @PATCH
    @Path("/{sku}/stock")
    @Transactional
    public ProductResponse changeStock(@PathParam("sku") String sku, @Valid StockAdjustment adjustment) {
        long updated = inventoryRepo.changeStock(sku, adjustment.delta());
        if (updated == 0) {
            throw new NotFoundException("Active product not found: " + sku);
        }
        return ProductResponse.from(inventoryRepo.findBySku(sku)
                .orElseThrow(() -> new NotFoundException("Product not found after stock change: " + sku)));
    }

    @DELETE
    @Path("/{sku}")
    @Transactional
    public Response discontinue(@PathParam("sku") String sku) {
        Product product = requireProduct(sku);
        product.discontinued = true;
        return Response.noContent().build();
    }

    @DELETE
    @Path("/maintenance/discontinued")
    @Transactional
    public Response deleteDiscontinued() {
        long deleted = repo.deleteDiscontinued();
        return Response.ok().entity(new DeletedProducts(deleted)).build();
    }

    private Product requireProduct(String sku) {
        Product product = repo.findBySku(sku)
                .orElseThrow(() -> new NotFoundException("Product not found: " + sku));
        if (product.discontinued) {
            throw new NotFoundException("Product not found: " + sku);
        }
        return product;
    }

    public record DeletedProducts(long deleted) {
    }
}

This class uses two different styles on purpose.

Create, update, and discontinue use the managed repository. Inside a transaction, Hibernate tracks the entity. When update changes name, category, and reorderPoint, there is no explicit repo.update(product) because the entity is already managed.

changeStock uses the stateless repository. That method does not load the product first. It runs one update query and returns the updated row afterward. If the update count is zero, the product is missing or discontinued. That gives the endpoint a direct 404 path and keeps the command explicit.

The list endpoint uses jakarta.data.Order and Sort with Panache query paging. Quarkus Data also has REST integration for Jakarta Data types such as PageRequest, Order, Sort, and Limit when quarkus-data-hibernate and Quarkus REST are both present. The official guide shows a PageRequest and Order<T> endpoint shape for that case. Here I keep page and size as plain query parameters because I want the paging mechanics to stay visible.

Try The API

Start dev mode:

./mvnw quarkus:dev

Create a product:

curl -s -X POST http://localhost:8080/products \
  -H 'Content-Type: application/json' \
  -d '{
    "sku": "SKU-100",
    "name": "Field Notebook",
    "category": "stationery",
    "stock": 8,
    "reorderPoint": 3
  }' | jq

Expected shape:

{
  "category": "stationery",
  "discontinued": false,
  "id": 1,
  "name": "Field Notebook",
  "needsRestock": false,
  "reorderPoint": 3,
  "sku": "SKU-100",
  "stock": 8
}

Adjust stock:

curl -s -X PATCH http://localhost:8080/products/SKU-100/stock \
  -H 'Content-Type: application/json' \
  -d '{"delta": -6}' | jq

The product now needs restocking:

{
  "sku": "SKU-100",
  "stock": 2,
  "reorderPoint": 3,
  "needsRestock": true
}

The real response includes all fields. The shortened output above shows the part you want to check.

List low-stock products:

curl -s http://localhost:8080/products/low-stock | jq

Search by name:

curl -s 'http://localhost:8080/products/search?q=Notebook' | jq

Discontinue the product:

curl -i -X DELETE http://localhost:8080/products/SKU-100

After that, GET /products/SKU-100 returns 404 because the API hides discontinued products from the active catalog.

Prove It With Tests

Create src/test/java/io/mainthread/catalogboard/ProductResourceTest.java:

package io.mainthread.catalogboard;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.not;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.restassured.http.ContentType;

@QuarkusTest
class ProductResourceTest {

    @Test
    void createsAndReadsProduct() {
        String sku = "SKU-" + System.nanoTime();

        given()
                .contentType(ContentType.JSON)
                .body("""
                        {
                          "sku": "%s",
                          "name": "Field Notebook",
                          "category": "stationery",
                          "stock": 8,
                          "reorderPoint": 3
                        }
                        """.formatted(sku))
                .when()
                .post("/products")
                .then()
                .statusCode(201)
                .body("sku", equalTo(sku))
                .body("needsRestock", equalTo(false));

        given()
                .when()
                .get("/products/{sku}", sku)
                .then()
                .statusCode(200)
                .body("name", equalTo("Field Notebook"))
                .body("category", equalTo("stationery"));
    }

    @Test
    void listsProductsByCategoryWithPagination() {
        String category = "category-" + System.nanoTime();
        createProduct("SKU-A-" + System.nanoTime(), "Alpha Binder", category, 4, 2);
        createProduct("SKU-B-" + System.nanoTime(), "Beta Binder", category, 4, 2);

        given()
                .queryParam("category", category)
                .queryParam("page", 0)
                .queryParam("size", 1)
                .when()
                .get("/products")
                .then()
                .statusCode(200)
                .body("", hasSize(1))
                .body("[0].name", equalTo("Alpha Binder"));
    }

    @Test
    void changesStockThroughStatelessRepositoryMethod() {
        String sku = "SKU-STOCK-" + System.nanoTime();
        createProduct(sku, "Shelf Label", "warehouse", 2, 5);

        given()
                .contentType(ContentType.JSON)
                .body("""
                        {
                          "delta": 4
                        }
                        """)
                .when()
                .patch("/products/{sku}/stock", sku)
                .then()
                .statusCode(200)
                .body("stock", equalTo(6))
                .body("needsRestock", equalTo(false));
    }

    @Test
    void findsLowStockProducts() {
        String sku = "SKU-LOW-" + System.nanoTime();
        createProduct(sku, "Packing Tape", "warehouse", 1, 5);

        given()
                .when()
                .get("/products/low-stock")
                .then()
                .statusCode(200)
                .body("sku", hasItem(sku))
                .body("findAll { it.sku == '%s' }.size()".formatted(sku), greaterThanOrEqualTo(1));
    }

    @Test
    void hidesDiscontinuedProductsFromCatalog() {
        String sku = "SKU-DISC-" + System.nanoTime();
        createProduct(sku, "Legacy Marker", "stationery", 4, 2);

        given()
                .when()
                .delete("/products/{sku}", sku)
                .then()
                .statusCode(204);

        given()
                .when()
                .get("/products/{sku}", sku)
                .then()
                .statusCode(404);

        given()
                .when()
                .get("/products/search?q=Legacy")
                .then()
                .statusCode(200)
                .body("sku", not(hasItem(sku)));
    }

    @Test
    void rejectsInvalidCreateRequest() {
        given()
                .contentType(ContentType.JSON)
                .body("""
                        {
                          "sku": "",
                          "name": "",
                          "category": "stationery",
                          "stock": -1,
                          "reorderPoint": 0
                        }
                        """)
                .when()
                .post("/products")
                .then()
                .statusCode(400);
    }

    private void createProduct(String sku, String name, String category, int stock, int reorderPoint) {
        given()
                .contentType(ContentType.JSON)
                .body("""
                        {
                          "sku": "%s",
                          "name": "%s",
                          "category": "%s",
                          "stock": %d,
                          "reorderPoint": %d
                        }
                        """.formatted(sku, name, category, stock, reorderPoint))
                .when()
                .post("/products")
                .then()
                .statusCode(201);
    }
}

Run the tests:

./mvnw test

Expected result:

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

The tests check behavior, not just startup. Product creation goes through validation and persistence. Category listing proves sorting and paging. The stock test proves the stateless update method. The discontinued test proves the API hides soft-deleted products from reads and search.

Before you move on, make one deliberate mistake: change reorderPoint in Product to reorderLevel and compile. The findLowStock HQL query still points at reorderPoint, so the processor should fail the build. That is exactly what you want. A broken query should fail before the service boots.

What Changed From Classic Panache

Classic Panache is still available. The Quarkus team says Panache 1 is not going away. Existing services do not need a panic migration, which is good because teams rarely do their best architecture work in a panic.

For new code, Quarkus Data Hibernate is the direction of travel.

The important changes are:

• A clearer extension name - quarkus-data-hibernate is the relational data entry point

• Generated repositories - interfaces are implemented at build time

• Checked queries - @Find and @HQL methods are validated against the entity model

• Managed and stateless choices - dirty checking when you want it, explicit writes when you do not

• Blocking and reactive variants - reactive support uses extra dependencies, but the model is designed to cover both

• Jakarta Data alignment - Quarkus Data can use Jakarta Data annotations and types where they fit

• REST integration - Jakarta Data paging and sorting types can be populated from HTTP query parameters

This sample uses blocking Hibernate ORM because it is the path most Java readers should learn first. Reactive is there too, but that’s another article in the furture.

Production Notes

The extension is experimental in Quarkus 3.37.0. You should expect some API movement.

Do not carry drop-and-create into production. It deletes data on startup. In real environments, let Flyway or Liquibase own schema changes and keep Hibernate ORM on validate.

Keep transaction boundaries at the service or resource boundary. In this small app the resource is the boundary. In a larger service, I would move the write methods into an application service and keep the REST class thinner.

Be careful with repository security annotations. The current guide notes that annotations on a repository type do not secure inherited methods. Put security on the resource or service method that represents the real operation, then add repository method annotations only where they are directly declared and tested.

Finally, @HQL is not a license to build strings from user input. Use parameters, as changeStock does. The compiler can check your query shape. It cannot fix a bad authorization rule.

Conclusion

Quarkus Data Hibernate gives Quarkus a clearer database entry point and a better repository story: generated methods, checked queries, managed writes where they help, and explicit stateless operations where they are clearer. The name changed from Panache Next, but the important part is the build-time contract around the data layer.

Subscribe now

It says “format this Markdown,” “review this PR,” or “write slides in our style,” and then leaves the agent to guess the rest from the conversation. That may work once. It may even look impressive if the task is simple. But it breaks the moment the workflow has local rules, fragile output, approvals, generated files, or a quality gate that must run before a human wastes time reading the result.

That is where skills become interesting.

I think of a skill as something stronger than a prompt library. A prompt library stores things you ask often. A skill stores how a repeated task should be done, where the agent should look, which scripts or tools make the work reliable, and which checks decide whether the result is ready.

That difference sounds small until you put it next to real developer work. A pull request summary is easy. A pull request summary that reads the right diff, respects protected paths, identifies test gaps, refuses to invent risk, and leaves the reviewer with a smaller decision is a workflow. A release note is easy. A release note that verifies tickets, changelog fragments, generated docs, and version bumps is a workflow. A slide deck is easy to ask for. A deck that uses the right template, stays inside text limits, runs brand checks, asks for approval before build, and fails QA on placeholder text is automation.

Skills are where that workflow knowledge lives.

The Small Definition

A skill is a reusable, named bundle of procedural knowledge for an agent.

Most systems I use have the same basic shape:

• a required instruction file, usually Markdown

• metadata such as name and description

• optional reference files for deeper guidance

• optional scripts for deterministic work

• optional assets such as templates, images, schemas, examples, or sample outputs

The description is the routing label. The agent sees many available skills, but it should not load all of them into every task. The description tells the agent when this skill applies. After that, the full body can give the workflow.

That gives skills their first practical property: progressive disclosure.

The agent does not need your whole release process when the user asks about a failing unit test. It needs the name and description of the release skill so it can decide that this is not the task. If the user does ask for release prep, the agent loads the full workflow. If that workflow points to references/versioning.md, scripts/check-release.py, or a release template, the agent reads or runs those only when needed.

That matters because context is not free. Even with large windows, stuffing every rule, reference, and example into the system prompt makes the agent worse at selection. It also makes maintenance painful. A good skill is loaded at the moment of use.

The mental model is simple:

user request
  -> skill metadata helps route the task
  -> SKILL.md gives the workflow
  -> references add deeper context only when needed
  -> scripts and assets make fragile steps repeatable
  -> hooks, tests, and CI decide whether the result is acceptable

The skill is one part of the system. It is the agent-facing contract for one kind of work.

How Skills Load

The exact mechanics vary by agent host, but the common pattern is stable.

First, the agent sees a lightweight catalog: skill names, descriptions, and sometimes locations or metadata. This catalog is enough for routing. It should stay small because it sits next to the user request, system instructions, project rules, and any other active context.

Then the agent decides whether a skill applies. A good description makes that decision cheap:

description: Use when preparing a pull request summary from a branch diff,
including changed behavior, test evidence, risky files, and reviewer notes.

That is much better than:

description: Helps with PRs.

The first version tells the agent when to load the skill. The second version only names a general topic.

After activation, the agent reads the full SKILL.md. This is where the workflow lives: steps, guardrails, stop conditions, verification, and links to supporting material.

References, scripts, and assets are the last layer. The skill should tell the agent when to read a reference, when to execute a script, and when to use an asset as output material. A reference file is context. A script is executable behavior. An asset is usually something the final artifact uses.

This loading model is why skill authors should keep the main file lean. Put the routing signal in the description. Put the core workflow in SKILL.md. Put long domain details, schemas, examples, and brand rules in references. Put deterministic work in scripts. Put templates and reusable media in assets.

What Belongs in a Skill

Put information in a skill when it is both reusable and task-specific.

That rules out a lot of noise.

Always-on repository policy belongs in AGENTS.md, project rules, or whatever your agent host uses for global instructions. “Use Podman for containers” and “do not commit secrets” are baseline rules. The agent should see them before any specific workflow starts.

A skill is narrower. It answers a question like:

• How do we prepare this repository’s pull requests?

• How do we run a bounded refactor?

• How do we publish a release note?

• How do we generate this kind of customer deck?

• How do we triage a dependency finding?

• How do we review architecture boundaries?

• How do we write and verify one type of documentation artifact?

The scope should be small enough that the skill can be tested. If you cannot tell whether the skill improved behavior, it is probably too vague.

This is a decent skeleton:

---
name: bounded-refactor
description: Use when making a small refactor that must preserve behavior, keep the diff narrow, and finish with tests plus a reviewer-facing summary.
---

Use this skill for refactors where the user names a specific class,
package, function, or behavior boundary.

Workflow:

1. Read the target code and the nearest tests before editing.
2. State the intended behavior boundary in one paragraph.
3. Make the smallest edit that achieves the requested change.
4. Run the narrowest relevant test first.
5. Run the broader test command only when the narrow test passes.
6. Summarize the diff as behavior preserved, implementation changed, and risk left.

Guardrails:

- Do not rename public APIs unless the user asked for that.
- Do not move files across module boundaries without calling it out first.
- Treat generated files as outputs, not source, unless this repo says otherwise.
- Stop if the first test failure is unrelated and explain the blocker.

This will not win a prose award. Good. It is a working contract.

The workflow tells the agent what to do. The guardrails tell it where to stop. The description tells it when to load the skill. That is already better than a prompt that says “please refactor carefully.”

The Automation Gradient

Skills start paying off when they connect instructions to deterministic pieces.

The gradient looks like this.

Prompt - one-time instruction inside a chat. Good for exploration. Weak for repeatability.

Command - a convenient entry point. Good for invocation. Often too small to hold the full workflow.

Skill - reusable workflow with routing metadata, steps, references, guardrails, and expected output.

Skill with scripts - the agent delegates fragile or repetitive operations to code. Good for parsing, validation, transformation, rendering, and checks.

Skill with hooks - deterministic checks fire at known events, regardless of whether the agent remembered to be careful.

Skill with CI gates - the repository repeats the load-bearing checks at the merge boundary.

That last part is where many teams stop too early. They write a SKILL.md, call it automation, and then ask the agent to remember the entire quality loop. Work with consequences needs a stronger loop.

If a task produces a deck, a report, generated source, release notes, an API client, or a migration patch, the skill should usually point to a script or a check. The agent is good at interpretation, selection, and repair. Counting characters in 40 slides by hand is a job for code.

For example, a deck-building skill should not say:

Make sure the deck follows our brand.

It should say something closer to:

1. Write the deck spec as JSON under `decks/<name>/<name>.json`.
2. Run `scripts/generate_outline.py --spec <spec> --fix`.
3. Resolve every `[OVER]` and `[LAYOUT]` warning before asking for approval.
4. After approval, run `scripts/build_from_template.py`.
5. Run `scripts/qa_deck.py`.
6. Do not call the deck ready until automatic QA passes and a visual review has happened.

Now the skill has moved past style advice. It describes a production path.

Skills and Hooks Solve Different Problems

Hooks are easy to confuse with skills because both shape agent behavior. They live at different layers.

A skill tells the agent how to do a task.

A hook runs when something happens.

The hook can be a Git hook, a pre-tool or post-tool hook in an agent host, a pre-commit scanner, a post-edit formatter, a pre-push policy check, or a CI job. The names vary by product. The pattern does not.

The part worth keeping is this: a hook should run even when the agent forgets the policy.

If a repository has a pre-commit hook that rejects secrets, the agent can be clever, tired, confused, or overconfident. The hook still runs. If it fails, the agent sees the output and can repair the diff. That is a much better loop than a sentence in a prompt that says “be careful with secrets.”

Skills and hooks work best when they share the same underlying checks.

skills/release-prep/SKILL.md
  -> tells the agent to update changelog fragments, bump versions, and run checks

scripts/check-release.py
  -> validates changelog, versions, generated docs, and forbidden placeholders

.githooks/pre-commit
  -> runs `scripts/check-release.py --staged`

.github/workflows/release-check.yml
  -> runs `scripts/check-release.py --all`

The skill gives the path. The hook gives fast feedback. CI gives the merge boundary.

The same idea applies to code:

• The skill tells the agent how to make a bounded refactor.

• The pre-commit hook rejects broad generated changes, secrets, formatting drift, or obvious policy violations.

• The CI gate runs the full build, architecture tests, dependency policy, and security scanners.

This is where agent automation starts to feel less like a chat trick. The agent is inside a control loop. It can still make mistakes, but the workflow gives it a way to see the mistake before a reviewer becomes the first real gate.

Guardrails Are Part of the Skill

Guardrails are stronger than “be careful” warnings. They are decisions about what the agent may do, what it must verify, and where human approval belongs.

A practical skill has at least a few explicit boundaries.

Scope - Which tasks should activate this skill, and which should not?

Inputs - Which files, commands, tickets, specs, or user-provided artifacts are the source of truth?

Outputs - What should exist at the end: a diff, a JSON spec, a deck, a report, a test, a PR summary?

Tool boundaries - Which tools are allowed or expected? Which tools are off limits for this workflow?

Stop conditions - When should the agent pause and ask, or report a blocker instead of improvising?

Approval points - Which step needs a human decision before generation, posting, publishing, or deployment continues?

Verification - Which commands, scripts, tests, or QA checks decide whether the work is ready?

You can encode these as prose, but the more fragile the task is, the more you should move toward scripts and gates.

For a developer workflow, I like guardrails like these:

Guardrails:

- Keep the diff inside the requested package unless the user approves a wider change.
- Treat changes to build files, CI workflows, security config, and suppression files as sensitive.
- Run the narrow test first. If it fails, fix that before running broader tests.
- Do not add broad suppressions such as `// nosemgrep`, `@SuppressWarnings`, or exclusion files without calling them out.
- If a generated file changes, run the generator again instead of editing the output by hand.
- If the tool that validates this workflow cannot run, report the workflow as blocked.

The last line matters. A failed check is different from “probably fine.” If the skill depends on a validator and the validator cannot run, the task did not pass. That may sound strict. It is also how you avoid turning QA into decoration.

QA Gates Make Skills Honest

Skills need their own QA.

That starts with the skill file itself. Check the front matter. Check links. Check reference paths. Check for placeholder text. Check that the description is specific enough to route. If the skill points to scripts, run those scripts. If the skill produces artifacts, inspect the artifacts.

Then test the behavior.

Do not only test the prompt that perfectly matches the skill. Test the near misses too.

• A request that should activate the skill

• A request that sounds similar but should not activate it

• A request with missing input

• A request with conflicting constraints

• A request that should stop for approval

• A request where the validator fails

This matters more for shared skills. A project skill used by one repository can evolve quickly. A global or team skill used across many repos becomes infrastructure. If it drifts, every user gets a slightly wrong assistant.

QA also needs output checks. A skill that generates Markdown can run a Markdown linter, link checker, or site build. A skill that changes Java can run focused tests, architecture checks, formatters, and dependency policy. A skill that creates slide decks can run text-limit checks, placeholder checks, brand checks, and a visual review step.

The skill should make these checks part of normal work. The hook or CI gate should make the high-risk ones hard to skip.

Developer Workflows That Deserve Skills

The best developer skills are boring in a good way. They encode the work that happens often, has local rules, and wastes time when done wrong.

Pull request preparation - Read the branch diff against the right base, summarize behavior changes, call out tests, list risky files, and propose a title. This is a good beginner skill because the source of truth is local and the output is easy to review.

Bounded refactor - Keep the change inside a named scope, preserve behavior, run tests in order, and report what moved. This is much better than asking an agent to “clean up” code.

Test gap analysis - Read a change, inspect nearby tests, identify missing assertions, and add only the tests that match the behavior. Agents often write tests that prove the implementation instead of the contract, so the skill needs to keep the contract visible.

Dependency and security triage - Pull scanner output, map findings to direct or transitive dependencies, check whether the vulnerable code path is reachable, and recommend patch, suppress, or accept-with-owner. The skill should not decide risk alone, but it can make the review smaller.

Architecture boundary review - Inspect package/module dependencies and turn stable findings into tests, for example with architecture rules. The skill should separate confirmed violations from suspicious edges that need human judgment.

Framework migration slice - Run a dry-run migration tool, inspect the patch, apply one slice, run tests, and stop before the diff becomes a rewrite of the whole system.

Release prep - Update changelog, versions, generated docs, release notes, and checks. This almost always deserves scripts because version text and generated files drift easily.

Documentation publishing - Read the canonical source, generate social copy, check links, verify image assets, and keep the copy inside platform limits. The work is simple, but it is easy to make one small wrong post at 8:08 in the morning and then spend the day looking at it.

Incident or runbook support - Gather logs, current status, recent deploys, and known rollback steps. The guardrail needs teeth here: the skill should distinguish read-only diagnosis from commands that change production.

These are plain skills. They are also exactly where teams lose time. Good automation usually starts with repeated annoyance, not with a grand architecture.

What Should Stay Out

Some things belong somewhere else.

A one-off answer does not need a skill. If the workflow is only needed once, write the prompt and move on.

Broad personality or tone preferences should usually live in always-on instructions or editing skills with a very clear scope. Do not hide repo-wide writing style inside a release skill.

Secrets belong outside skills. A skill can say where to find a secret through the approved mechanism. It should not contain the secret.

Credentials, deployment tokens, production URLs, and customer data need external controls. A skill can describe the safe path. It should not become a private vault with Markdown syntax highlighting.

Large reference manuals should not be pasted into SKILL.md. Put them in reference files and tell the agent when to read them. If a reference is huge, include search hints. The agent should not read 80 pages of API docs because one small task needed the JSON schema.

Finally, avoid all-purpose skills. “Use when coding in this repository” is a project instruction. A skill that always applies has become background policy.

The Special Cases

Skills get interesting around the edges.

Project, global, and team skills

A project skill lives with one repository. This is the default I prefer for developer workflows because the skill is reviewed with the code it affects. The team pulls the repo and gets the same workflow.

A global skill lives on one developer’s machine and follows them across workspaces. That is good for personal routines: standup notes, personal writing checks, local review style, or small helper workflows.

A team skill repository or packaged skill artifact makes sense when the same workflow should apply to many repos. At that point, versioning matters. You want release notes, compatibility notes, and a way to roll out changes without quietly changing every agent behavior overnight.

The sharing model changes ownership. A project skill belongs to that repo. A team skill belongs to the platform or enablement group that ships it.

Permissions and execution

Many agent hosts separate instruction loading from tool permission. A skill may tell the agent which command to run, but the shell, filesystem, browser, external API, or posting action may still need explicit approval.

That is a good boundary. The skill should name the commands and external calls it expects, and it should say what happens when the host refuses them.

For example:

This workflow needs read access to `publishing/<slug>/publishing-pack.md`,
execute access for `scripts/check-social-copy.py`, and Buffer access for
scheduling. If any of those permissions are missing, stop and report which
step is blocked.

The skill explains the workflow. The host still decides what the agent may do in this task.

Skill collisions

If two skills have similar names or descriptions, routing gets soft. The agent may load the wrong one or load both and mix the instructions into a bad compromise.

Make descriptions discriminating. “Use for release work” is weak. “Use when preparing a Java service release from changelog fragments, version files, and generated API docs” is better.

Collisions are also a reason to keep project-specific rules near the project. A repo-specific skill should usually win over a global habit.

Skills with assets

Assets change the task from “write text” to “produce an artifact from a known base.”

Templates, slide masters, icon catalogs, image masks, OpenAPI examples, Terraform module snippets, legal boilerplate, or approved diagrams all belong here when the output must match a concrete format.

The skill should tell the agent whether to read the asset, copy it, transform it, or pass it to a script. File names are too weak as workflow instructions.

Skills with scripts

Scripts are where you move work that should be independent of language-model memory.

Use scripts for:

• parsing structured files

• rendering or generating artifacts

• counting text limits

• validating schemas

• checking links

• comparing generated output with source

• extracting project facts

• running repeatable transformations

The skill should explain how to run the script, what output matters, and which failures are blockers. The script should do the deterministic part. The agent should interpret, fix, and explain.

Multiple skills in one task

Sometimes one task really needs more than one skill. A publishing workflow may need an article voice skill, a publishing-pack skill, and a social-scheduling skill. That is fine if each skill owns a clear phase.

The danger is instruction conflict. One skill says “rewrite copy only when asked.” Another says “always improve copy before scheduling.” That conflict will show up as surprising behavior.

The better fix is clear ownership:

• voice skill changes prose

• publishing-pack skill creates assets

• scheduling skill posts exact approved copy

The scheduling skill should explicitly say that it posts approved copy and only rewrites when the user asks. That small line can save a lot of accidental creativity.

Human approval in the middle

Some skills should pause.

Decks, release posts, external publishing, dependency suppressions, migrations, and destructive operations all need approval points. The skill should name those points directly.

For example:

After generating the outline, stop and ask for approval before building the final artifact.
After finding dependency suppressions, stop and list each suppression before editing policy files.
Before posting to external channels, verify the target account and show the exact copy.

This is still automation. Good automation has brakes.

The Difference Between a Skill and a Tool

An MCP tool, CLI command, or local script does work.

A skill tells the agent how and when to use that work.

This distinction matters because teams often wire many tools into an agent and then wonder why behavior gets worse. A tool catalog gives the agent capability. The workflow still needs judgment.

If the agent has 50 tools and no skill, it must infer the workflow from tool names and conversation context. That works for simple calls. It fails when the right path is “read the spec, validate the account, check duplicates, ask for approval, then call the posting tool.”

The skill carries that order.

The tool should be narrow and honest. The skill should provide the procedure around it. Hooks and CI should verify the parts that need enforcement.

A Practical Skill Design Test

When I review a skill, I ask a few simple questions.

Can the description decide routing without reading the body?

Can a teammate understand the workflow in two minutes?

Does the skill say where the source of truth lives?

Does it tell the agent where to stop?

Does it point to scripts for work that should be deterministic?

Does it say how to verify the result?

Can we test activation, rejection, failure, and output quality?

Does it reduce human review work, or does it only produce nicer-sounding output?

That last question carries the weight. Skills should reduce wrong turns, not make the agent more theatrical.

A Small Example With Hooks

Imagine a team has a recurring workflow: update generated API docs after OpenAPI changes.

A weak instruction says:

Remember to update the API docs after changing OpenAPI.

The skill version gives the agent a path:

---
name: api-doc-refresh
description: Use when OpenAPI files, REST resources, or generated API documentation change.
---

Workflow:

1. Detect changed OpenAPI source files and REST resource files.
2. Run `scripts/generate-api-docs.sh`.
3. Run `scripts/check-api-docs.sh`.
4. If generated docs changed, include them in the diff.
5. If the check fails, fix the source or generator input. Do not hand-edit generated docs.

Guardrails:

- Treat generated documentation as output.
- Do not edit files under `docs/api/generated/` by hand.
- If the generator is missing or fails, report the task as blocked.

Then the repository adds a hook:

#!/usr/bin/env bash
set -euo pipefail

scripts/check-api-docs.sh --staged

And CI runs:

scripts/generate-api-docs.sh
scripts/check-api-docs.sh --all

Now the workflow has three layers. The skill guides the agent. The hook catches local drift. CI protects the merge boundary.

The agent can still get it wrong. But it gets a clear error, a script name, and a repair path. Agents handle that mistake loop well.

Skills Are a Maintenance Surface

Once a skill changes generated code, publishes artifacts, schedules posts, edits release files, or shapes review output, it is part of your engineering system.

Treat it that way.

Review skill changes. Lint skill files. Test them with realistic prompts. Version shared skills. Keep references current. Delete stale examples. Pin scripts and tool versions where drift would hurt. Put skill-owned checks into hooks or CI when the output matters.

Also measure behavior, not vibes. Did the skill reduce retry turns? Did it keep the diff smaller? Did it catch missing tests? Did it stop posting to the wrong channel? Did reviewers spend less time reconstructing intent? Did bad outputs fail earlier?

That is a better discussion than “the agent sounded more confident.”

Confidence is cheap. A smaller, verified workflow is better.

Where I Would Start

For a software team starting from zero, I would skip the giant skill library.

Start with one repeated task that already annoys people and has a clear quality gate.

Good first choices:

• PR preparation for one repository

• bounded refactor for one module

• release-note generation with changelog checks

• dependency finding triage with scanner output

• documentation publishing with link checks

• generated API docs refresh

Write the smallest skill that makes the task repeatable. Add one script if the task needs deterministic validation. Add a hook if the mistake is cheap to catch before commit. Add CI if the mistake must not land.

Then run it on real work. The first version will be incomplete. That is fine. The failures worth keeping will tell you what the skill forgot.

This is the part I like about skills. A bad prompt just disappoints you. A bad skill can be improved as an artifact. You can review it, patch it, test it, and ship the next version.

The Actual Point

Agent skills are ordinary files with unusual leverage. They do not replace tests, hooks, CI, or human judgment. They put repeatable workflow knowledge where the agent can load it at the right moment.

Used well, they turn “please be careful” into a path:

read the right source
make the narrow change
run the deterministic check
respect the approval point
fail closed when validation fails
report the result in the shape humans need

That is the automation worth caring about.

The beginner mistake is to make a skill that tells the agent what output should look like. The better version tells the agent how the work moves through the system, which tools own which facts, and what must be true before the result is ready.

At that point the Markdown file is no longer just Markdown. It is a small contract between the human, the agent, and the repository.

Then the cutting starts. System prompt. Tool descriptions. Old examples. Comments. Variable names in injected snippets. Sometimes even the task brief gets squeezed until it reads like somebody was charged by the character.

I understand the reflex. Context is not free. In The GitHub MCP Server Can Burn 17k Tokens Before You Ask a Real Question, I measured one obvious leak: wide tool catalogs. A large MCP surface can burn a noticeable part of the window before the agent reads a single file. The second leak is payload. Raw diffs, full documents, and replayed transcripts can eat the same budget a minute later.

Those two cuts help a lot. Narrow the tool surface. Narrow the payload.

The trouble starts after that. Teams keep cutting, but the next cuts land on the wrong material.

The Expensive Mistake Starts After the Useful Cuts

Once the broad tool catalog is gone and the raw payload is under control, what remains is usually the part that carries meaning: the task description, the few-shot examples, the names in the code, the explanation of local rules, the one weird business constraint nobody would guess from reading the repository.

That is the part people start compressing.

A recent paper, Beyond Human-Readable: Rethinking Software Engineering Conventions for the Agentic Development Era, gives this mistake a methodology. The paper argues for semantic density, which means how much useful meaning each token carries. In its log-format experiment, aggressive compression reduced input tokens by 17 percent and still increased total session cost by 67 percent. The model had less text to read and more work to do.

That result makes sense to me. If you replace clear names with abbreviations, strip context until it becomes ambiguous, or turn a precise task brief into terse prompt folklore, the work does not disappear. The model still has to recover the missing meaning. Now it pays for that recovery with extra reasoning, extra retries, and extra correction loops.

OpenAI’s reasoning guide says this pretty clear: reasoning tokens still use context-window space and are billed as output. So when a compressed prompt forces the model to reconstruct what you used to state clearly, the bill moved. It did not shrink.

I think of that as the compression tax. You saved on the visible input side and paid the difference somewhere harder to see.

This Is the Same Accounting Bug Again

Thiskeeps coming back to the same pattern over and over. We measure the easy thing and ignore the expensive thing.

In Stop Measuring AI by How Much You Use It, I argued that token burn is heat. It is measurable activity. It is not proof of useful work. In AI Coding Break-Even: Cheap Tokens, Expensive Software, I made the same point at delivery scale: the real cost shows up in review, repair, security, rollout, and the hours after the model claims success. In The Spec Trap, the cheap-looking move was underspecifying the job and pushing the ambiguity downstream.

Token minimization makes the same mistake at prompt scale.

A smaller prompt can be cheaper. A smaller ambiguous prompt often is not. It can produce longer output, more retries, worse tool choices, and more human cleanup. The dashboard still shows fewer input tokens, which is nice if your goal is to win an argument in a screenshot. The real system pays elsewhere.

Vendors Already Know This

One reason I do not take “just make the prompt shorter” very seriously is that the platform vendors keep shipping features that solve a different problem.

OpenAI’s prompt caching guide tells you to keep repeated content in a stable prefix so cached prefixes can cut cost and latency. Gemini’s context caching docs split the same idea into implicit and explicit caching. Anthropic’s guide to managing tool context says large toolsets should use tool search so the full catalog stays out of the window until the model asks for it. Anthropic’s context editing docs says: context is finite, returns diminish, and irrelevant content degrades model focus.

Even if they are all burning through a lot of experimentation money right now, the fact that they all have this as tool category is a solid hint that this did not happen by accident.

Repeated context should be cached. Large tool catalogs should be loaded late. Stale results should be trimmed. Old conversation debris should stop competing with the current task. The pattern is clear enough: cut repetition, cut stale context, cut broad catalogs, and keep the remaining signal readable.

The Target Is Semantic Density

Aim for the densest prompt you can build without cutting away meaning.

That changes the operating rule.

A wide tool catalog is low density because most of the tools are irrelevant on this turn. A full raw diff is low density when the agent only needs the touched files that affect one behavior. A replayed transcript is low density when three earlier summaries and two failed attempts are still sitting in the window long after they stopped helping.

Clear task instructions, good names, and the local rule that explains a weird edge case all earn their place. Those tokens pull their weight.

The point is simple: remove noise first. Preserve signal. If you keep cutting after that, you start deleting the map and asking the model to redraw it from memory.

The split is easier to see when you draw the window as layers instead of one big token number.

Two cuts are selection work. The last cut is where teams start deleting meaning instead of noise.

Caveman Is a Useful Counterexample

This is also why juliusbrussee/caveman is interesting and not a contradiction.

The project is not mainly trying to compress the part I want to protect. It is mostly compressing the agent’s speaking style. The README claims about 65 percent average output reduction across ten prompts, and it is explicit about the boundary: output gets smaller, reasoning tokens do not. That lines up with the difference this article is trying to draw.

If a model wastes tokens on filler, politeness, repeated restatement, and long glue text, shorter output can help both budget and quality. The answer gets faster to read. Less narration means fewer places for the important instructions to hide. That is a different move from stripping context out of the task brief until the model has to guess what you meant.

caveman also pushes on the input side in narrower ways. The repo includes caveman-compress for memory files and a middleware layer that shrinks MCP tool descriptions. That can be a good trade if the compression preserves the load-bearing content. The project README says code, URLs, paths, and error strings stay exact. That is the only version of compression I think makes sense: cut the wrapper, keep the meaning.

So I would use caveman as a qualifier. Compression helps when it removes low-value phrasing around stable meaning. Compression hurts when it removes the meaning itself. Same word, different target, very different outcome.

What I Would Do Instead

If I had to reduce this to a small operating rule, it would look like this:

1. Cache what repeats. Stable instructions, examples, and other shared prefixes should not be resent at full price on every turn.

2. Retrieve narrow evidence. Give the model the sections, files, or snippets that answer the current question. Keep the archive available, not injected by default.

3. Load tools on demand and trim old tool chatter. Large tool menus and stale tool results are both taxes.

4. Keep the remaining context clear. Write the task brief plainly. Keep real names. State the local constraints in full sentences. Ask for a bounded artifact when a decision object or structured result is enough.

That is token discipline.

Clear Text Still Wins

Some teams still get stuck. They see a long prompt and assume the length itself is the problem. Sometimes it is. Often the real problem is that the prompt is carrying the wrong material. I am not saying that we should resurface the prompt engineering hype. But also not forget about what good prompts look like.

A clear system prompt with stable rules can be cached. A clear task brief can prevent three correction loops. A clear tool description can stop a bad call before it happens. A clear variable or field name can save the model from inventing a meaning that was never there.

One line to remember:

Cut noise hard. Keep meaning intact.

That is a better default than “make the prompt shorter” because it matches how the bill really moves through an agent system. The catalog tax is real. The payload tax is real. The compression tax is real too, and it is the one teams keep charging themselves.

The model is rarely confused because you used full words. It gets lost when the window is full of stale junk and the remaining signal arrives half-compressed. That is a context problem. It needs selection, structure, and a little less obsession with the screenshot-friendly part of the meter.

Subscribe now

In a booking application, “delete this user” is never just one DELETE endpoint. It touches active bookings, foreign keys, historical reads, cache state, and the uncomfortable question every senior team eventually has to answer: is this really a hard delete, or is it deactivation, anonymization, or something in between?

That is why I like the workflow Alex Soto shows in the latest video. He does not start by asking IBM Bob to write the feature. He starts by asking it to map the regression impact before implementation. That is a much better use of an assistant.

Use the Agent Before the Feature Code

I still think the basic rule is simple: use AI to go faster on work you understand already. If you hand core business logic to a model and hope for the best, you are outsourcing the part you still need to own.

Impact analysis is different. Here the agent helps you read the shape of the system, collect likely break points, and build a regression plan before the feature branch gets noisy. In this case, “shift-left” just means you do the regression thinking before you touch the code that will trigger it.

That matters because deletion flows are where business rules stop looking like CRUD. One missing constraint check and you get orphaned rows, broken history screens, or a second-order production issue that only shows up after cleanup jobs or cache refreshes run. Efficient. Also bad.

Why IBM Bob Fits This Job

It’s not IBM Bob’s magic that makes this work: It is context.

Bob runs against the workspace, so it can inspect the actual resources, entities, and services in the application. That gives it a real basis for the regression plan. It is reading the model you are about to change, not guessing from a paragraph of pasted code.

That is the difference. A browser chat can still help, but a workspace-aware tool can follow relationships across the codebase. For something like user unregistration, that is where the interesting bugs live.

The Prompt Is Small on Purpose

Alex starts with a constrained prompt:

“Act as a tester. Identify the areas of the system that should be included in regression testing after changes are made to add the unregistration of a user. Don’t implement anything.”

Don't implement anything keeps the session focused on analysis. Once an assistant starts writing code too early, it tends to collapse design, impact analysis, and implementation into one optimistic blob. That is fine for demos. It is much less fine when a deletion rule can invalidate half your booking flow.

What a Good Regression Plan Should Surface

Because Bob can inspect the workspace, it can point at actual system boundaries that deserve tests before the first delete path is implemented. In this example, the report highlights the kind of things I would expect a senior developer to check immediately:

• Active booking rules. What happens if the user still has a live reservation, an open payment flow, or another stateful process attached to the account?

• Database integrity. Do deletes cascade, fail fast, orphan records, or silently leave history in a shape the rest of the system no longer understands?

• Historical read paths. Can past bookings, reports, and audit views still load after the user record changes or disappears?

• Cache and derived state. If user data is cached, indexed, or copied into another read model, what gets invalidated and when?

• API behavior. What does the endpoint return when the user has already been unregistered, cannot be removed, or does not exist anymore?

This is where the workflow earns its keep. The assistant is not replacing engineering judgment. It is helping you build the list you will review anyway, only faster and with fewer blind spots.

The Senior Java Questions Start After the First Report

A regression plan is a good start. It is still only a start.

If I were taking this feature forward, I would want the team to answer a few more questions before implementation:

• Is “unregister” really delete? In many systems the right answer is soft delete, anonymization, or status change because legal retention, reconciliation, or audit rules still apply.

• Where is the transaction boundary? If one part of the removal fails, what rolls back, and what can be left half-done?

• What is the concurrency story? If one request unregisters the user while another creates or confirms a booking, which invariant wins?

• What happens on the second call? Idempotence matters here. A repeated delete should not create fresh damage.

• Which tests belong at which level? Some checks belong close to the domain model. Others need @QuarkusTest, HTTP assertions, persistence wiring, or a packaged-app check with @QuarkusIntegrationTest.

That last point is worth calling out. A lot of teams mix regression planning with test implementation planning and then get a giant bag of tests with no clear boundary. I would rather make that mapping explicit up front.

Turn the Report Into Proof

The report is useful because it changes what we test first.

For a change like this, I would usually split the work into a few layers:

• business-rule tests around the unregister decision itself

• persistence tests for relationship handling and transaction rollback

• @QuarkusTest coverage for the HTTP and CDI wiring

• packaged-app checks with @QuarkusIntegrationTest when the deployment shape matters

After that, continuous feedback matters more than one big test run at the end. Quarkus continuous testing is a good fit here because the regression plan will usually touch multiple paths, and you want quick confirmation while the model is still moving.

Coverage still matters, but later. First make sure the dangerous branches are tested. Then measure what you actually covered.

Further Reading

• Quarkus: Testing Your Application for @QuarkusTest, transactions in tests, mocks, and @QuarkusIntegrationTest.

• Quarkus: Continuous Testing if you want fast feedback while the unregister flow is still changing.

• JUnit User Guide for the current programming model, extension model, and test organization options once the regression plan becomes a real suite.

• JaCoCo Coverage Counters if you want branch, instruction, and complexity numbers to reflect the paths that actually matter.

• IBM Bob if you want to try the workspace-aware analysis workflow Alex uses in the video.

If your unregister flow goes through JPA or Hibernate relationships, read the deletion and orphan-handling rules in the ORM documentation for the exact version you run. That is one of those areas where small mapping details decide whether cleanup is correct or catastrophic.

Why I Like This Pattern

I like this workflow because it uses the assistant where assistants are actually useful: reading a codebase, building a candidate regression map, and giving you a sharper review checklist before implementation starts.

You still need to decide the delete semantics. You still need to own the transaction model. You still need to write the tests that prove the feature is safe.

But you do not need to build the first regression checklist from memory every single time. Let the tool help there. Save your energy for the decisions that still need an engineer.

Subscribe now