Build Zero-Trust Quarkus Services Without Guessing the Boundaries

A hands-on LoanFlow tutorial with mTLS, Keycloak, service tokens, and the edge checks that should not leak into internal services.

The ugly part of zero-trust is rarely the first application.properties file. It is the moment an internal service starts accepting calls it should never have seen, and nobody can tell whether the caller was a real service, a stolen token, or a policy bug in the edge API.

I do not find security tutorials very useful when they jump from TLS to Keycloak to a couple of annotations and call that a design. The part that matters is whether you can answer three separate questions cleanly:

• Is this channel trustworthy?

• Which service is calling me?

• Where do business rules actually live?

So that is what we do here.

What we build

We build LoanFlow: three Quarkus services on JDK 25 with app-managed mTLS, OIDC service tokens on outbound REST clients, and branch-level policy at the public edge.

• loan-service — public edge API. Accepts bearer tokens for human users, enforces branch ownership, orchestrates downstream calls.

• credit-service — internal only. Requires HTTPS with a client certificate and a bearer token with permission credit_check_run.

• document-service — internal only. Same transport rules; requires permission document_write.

By the end you can prove:

• A Berlin loan officer reads and submits their own loan

• A Hamburg officer gets 403 on a Berlin loan

• A direct call to credit-service without a client certificate fails at TLS

• A call with mTLS but no bearer token gets 403

• A second submit of the same loan returns 409

Prerequisites

You need a normal local Java setup plus the usual tools for certificates and test traffic. Use whatever Quarkus CLI version you already have installed. It pins the platform BOM in each generated pom.xml, so this tutorial does not depend on you matching one exact CLI release.

• JDK 25

• Quarkus CLI

• Podman (Dev Services starts Keycloak in a Podman container)

• OpenSSL, keytool, curl, jq

• Familiarity with JAX-RS and OIDC bearer tokens

• About ☕️☕️☕️

Create the workspace root and the shared support directories first:

mkdir -p loanflow-zero-trust/{infrastructure,scripts}
cd loanflow-zero-trust

The three service directories get created by the Quarkus commands in the next step. Copy infrastructure and scripts from my Github repository.

Project setup

Create the three applications from the repo root. Run each command once:

quarkus create app com.mainthread.loanflow:loan-service \
  --extension='rest-jackson,rest-client-jackson,oidc,rest-client-oidc-filter,tls-registry,smallrye-health' \
  --java=25 --no-code

quarkus create app com.mainthread.loanflow:credit-service \
  --extension='rest-jackson,oidc,tls-registry,smallrye-health' \
  --java=25 --no-code

quarkus create app com.mainthread.loanflow:document-service \
  --extension='rest-jackson,oidc,tls-registry,smallrye-health' \
  --java=25 --no-code

Extensions and why they matter:

• rest-jackson — Quarkus REST with JSON for edge and internal APIs

• rest-client-jackson + rest-client-oidc-filter — typed outbound clients that attach service tokens automatically (loan-service only)

• oidc — bearer token validation on every secured endpoint

• tls-registry — named TLS configurations for edge HTTPS, internal mTLS servers, and internal mTLS clients

• smallrye-health — liveness probes when you deploy this later

Add test dependencies in each module POM: rest-assured and quarkus-test-security-oidc (test scope). Internal services also need quarkus-test-oidc-server for OidcWiremockTestResource.

Architecture on purpose

Transport — Internal hops use app-managed mTLS through the TLS registry. We are not using a service mesh here because I want the transport layer to stay visible in config you can grep.

Service identity — loan-service acquires its own token with the OIDC client and REST client filter. I would rather let Quarkus do that work than hand-roll Authorization headers and rediscover the annoying parts ourselves.

Business authorization — User-specific rules stay in loan-service. credit-service and document-service never decide whether Alice from Berlin can touch loan LN-100. They decide whether the caller is an allowed internal service with the right permission. A client_credentials token does not represent a human loan officer, and pretending otherwise is how these demos get fuzzy fast.

Out of scope for this demo — service mesh, OPA, Keycloak authorization services, and propagating the end-user token downstream. If credit-service later needs user-level policy, you are in token propagation or token exchange territory. That is a different tutorial.

Generate certificates

Run ./scripts/generate-certs.sh from the repo root. It creates a local CA, one certificate per service, and a shared truststore under infrastructure/certs/.

Each service directory gets tls.key, tls.crt, and keystore.p12. The shared truststore contains only the CA. PKCS12 password is changeit for local dev.

Keycloak with Dev Services

Copy infrastructure/keycloak/loanflow-realm.json into each module as src/main/resources/loanflow-realm.json. Quarkus Dev Services for Keycloak starts Keycloak in a Podman container when you run quarkus:dev, imports that realm, and wires quarkus.oidc.auth-server-url for you. That saves us from turning local setup into a side quest.

Leave quarkus.oidc.auth-server-url unset in dev mode, because setting it disables Dev Services. Pin the host port so the curl examples stay stable:

quarkus.keycloak.devservices.realm-name=loanflow
quarkus.keycloak.devservices.realm-path=loanflow-realm.json
quarkus.keycloak.devservices.port=8180

quarkus.oidc.application-type=service

All three services carry the same Dev Services block. Quarkus shares one Keycloak container between dev-mode processes on the same machine. Start loan-service first if you want a predictable boot order, but any of the three can bring Keycloak up.

The realm defines:

• Users — alice/alice (Berlin, loan_officer), bob/bob (Hamburg, loan_officer), admin/admin (loan_admin)

• loanflow-cli — confidential client with direct access grants for local password-grant token retrieval (loanflow-cli-secret)

• loan-service — confidential client with service account; default scopes credit_check_run and document_write

Realm roles (not Keycloak Groups) map to @RolesAllowed. The loanflow-cli client uses the same pattern as the dpop-demo Keycloak realm: roles in defaultClientScopes, with a full roles client scope definition that adds realm_access.roles to the access token. The separate branch client scope adds the branch claim. Service token scopes map to @PermissionsAllowed on internal endpoints.

In the Keycloak admin UI, an empty Groups list is expected. Check Clients → loanflow-cli → Client scopes → Default for roles and branch, and Users → alice → Role mapping → Realm roles for loan_officer.

The password grant is here as a local test shortcut, nothing more. I would not turn that into the real browser story.

Implement loan-service

loan-service is the policy enforcement point. It knows users, branches, and loan state.

Seed data in an in-memory repository:

• LN-100, branch berlin, status DRAFT

• LN-200, branch hamburg, status DRAFT

• LN-300, branch berlin, status SUBMITTED

Expose:

• GET /api/loans/{loanId}

• POST /api/loans/{loanId}/submit

Coarse access uses @RolesAllowed({"loan_officer", "loan_admin"}). Keycloak puts realm roles in realm_access.roles; tell Quarkus where to find them:

quarkus.oidc.roles.role-claim-path=realm_access/roles

Fine-grained branch rules live in LoanAccessPolicy. Read branch from the JWT — it does not land on SecurityIdentity automatically:

@ApplicationScoped
public class LoanAccessPolicy {

    private final CallerContext callerContext;

    public LoanAccessPolicy(CallerContext callerContext) {
        this.callerContext = callerContext;
    }

    public void checkCanRead(LoanApplication loan) {
        if (callerContext.hasRole("loan_admin")) {
            return;
        }

        String branch = callerContext.branch();
        if (!callerContext.hasRole("loan_officer") || !loan.branch().equals(branch)) {
            throw new ForbiddenException();
        }
    }

    public void checkCanSubmit(LoanApplication loan) {
        checkCanRead(loan);
        if (loan.status() != LoanStatus.DRAFT) {
            throw new WebApplicationException("Loan is not in DRAFT status", 409);
        }
    }
}

CallerContext is @RequestScoped and reads branch from JsonWebToken at runtime (falling back to SecurityIdentity attributes in tests). ForbiddenAccessMapper turns branch denials into 403 with a small JSON body so curl output is obvious.

LoanApplicationService.submit() runs in order: load loan → checkCanSubmit → call credit-service → call document-service → persist SUBMITTED only after both succeed.

The outbound REST client attaches a service token automatically:

@Path("/internal/credit-checks")
@RegisterRestClient(configKey = "credit-service")
@OidcClientFilter("internal-calls")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public interface CreditServiceClient {

    @POST
    CreditCheckResponse run(CreditCheckRequest request);
}

Create DocumentServiceClient the same way with configKey = "document-service" and path /internal/documents.

Implement credit-service and document-service

Internal services stay small on purpose. Once these endpoints start owning business rules too, the boundary gets muddy fast. Here they validate transport and permission, then do one job.

@Path("/internal/credit-checks")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
public class CreditResource {

    @Inject
    CreditDecisionService creditDecisionService;

    @POST
    @PermissionsAllowed("credit_check_run")
    public CreditCheckResponse run(CreditCheckRequest request) {
        return creditDecisionService.run(request);
    }
}

document-service mirrors this with @PermissionsAllowed("document_write") on POST /internal/documents. Credit scoring is deterministic fake logic — this tutorial is about security boundaries, not bureau accuracy.

Wire the configuration

This is the part I usually check first in security demos, because it is where hand-wavy examples turn back into engineering. Every property below matters.

loan-service

Edge HTTPS without client certificates from human callers:

quarkus.http.ssl-port=8443
quarkus.http.insecure-requests=disabled
quarkus.http.tls-configuration-name=edge

quarkus.tls.edge.key-store.p12.path=../infrastructure/certs/loan-service/keystore.p12
quarkus.tls.edge.key-store.p12.password=changeit

quarkus.tls.internal-client.key-store.p12.path=../infrastructure/certs/loan-service/keystore.p12
quarkus.tls.internal-client.key-store.p12.password=changeit
quarkus.tls.internal-client.trust-store.p12.path=../infrastructure/certs/truststore.p12
quarkus.tls.internal-client.trust-store.p12.password=changeit

quarkus.oidc.application-type=service

quarkus.keycloak.devservices.realm-name=loanflow
quarkus.keycloak.devservices.realm-path=loanflow-realm.json
quarkus.keycloak.devservices.port=8180

quarkus.oidc-client.internal-calls.auth-server-url=${quarkus.oidc.auth-server-url}
quarkus.oidc-client.internal-calls.client-id=loan-service
quarkus.oidc-client.internal-calls.credentials.secret=loan-service-secret
quarkus.oidc-client.internal-calls.grant.type=client
quarkus.oidc-client.internal-calls.early-tokens-acquisition=false

quarkus.rest-client.credit-service.url=https://localhost:8444
quarkus.rest-client.credit-service.tls-configuration-name=internal-client

quarkus.rest-client.document-service.url=https://localhost:8445
quarkus.rest-client.document-service.tls-configuration-name=internal-client

What breaks when this is wrong:

• Missing edge TLS name → no HTTPS on 8443

• Wrong truststore on internal-client → PKIX path building failed on downstream calls

• Setting quarkus.oidc.auth-server-url in dev → Dev Services never starts Keycloak

• Missing OIDC client internal-calls → REST client cannot fetch a service token

• early-tokens-acquisition=true → short-lived tokens may expire before the first downstream call

credit-service and document-service

Internal only — require client certificates:

quarkus.http.ssl-port=8444
quarkus.http.insecure-requests=disabled
quarkus.http.tls-configuration-name=internal-server
quarkus.http.ssl.client-auth=REQUIRED

quarkus.tls.internal-server.key-store.p12.path=../infrastructure/certs/credit-service/keystore.p12
quarkus.tls.internal-server.key-store.p12.password=changeit
quarkus.tls.internal-server.trust-store.p12.path=../infrastructure/certs/truststore.p12
quarkus.tls.internal-server.trust-store.p12.password=changeit

quarkus.oidc.application-type=service

quarkus.keycloak.devservices.realm-name=loanflow
quarkus.keycloak.devservices.realm-path=loanflow-realm.json
quarkus.keycloak.devservices.port=8180

Use port 8445 and document-service cert paths for document-service.

What breaks:

• Missing client-auth=REQUIRED → anyone with a valid token reaches the internal API over TLS

• Truststore missing the CA → valid client certificates fail during handshake

• Hard-coded quarkus.oidc.auth-server-url in dev → Dev Services disabled; token validation fails if nothing listens on that URL

Run the system

After the certificates exist, start each service in its own terminal. The first quarkus:dev process starts Keycloak in Podman; the others attach to the shared container:

cd loan-service && ./mvnw quarkus:dev
cd credit-service && ./mvnw quarkus:dev
cd document-service && ./mvnw quarkus:dev

Wait until the startup log shows Keycloak listening on port 8180 before running token curl commands.

HTTPS URLs:

• loan-service — https://localhost:8443

• credit-service — https://localhost:8444

• document-service — https://localhost:8445

Prove it

Run these commands from the loanflow-zero-trust/ repo root — the certificate paths are relative to that directory. Your shell prompt should not be ~ unless the repo happens to live there.

cd /path/to/loanflow-zero-trust

export CA=infrastructure/certs/ca/ca.crt
export LOAN_CERT=infrastructure/certs/loan-service

Get a user token for Alice (local test shortcut):

export USER_TOKEN=$(
  curl -sf http://localhost:8180/realms/loanflow/protocol/openid-connect/token \
    --user loanflow-cli:loanflow-cli-secret \
    -H 'content-type: application/x-www-form-urlencoded' \
    -d 'username=alice&password=alice&grant_type=password' | jq -r '.access_token'
)

echo "$USER_TOKEN" | awk -F. '{print $2}' | python3 -c "import sys,base64,json; p=sys.stdin.read().strip(); p+=('='*(-len(p)%4)); print(json.dumps(json.loads(base64.urlsafe_b64decode(p)), indent=2))"

test -n "$USER_TOKEN" && test "$USER_TOKEN" != "null"

If test fails, Keycloak is not ready or the realm import missed loanflow-cli. Re-check the Dev Services startup log.

Read a Berlin loan through the edge:

curl -i --cacert "$CA" \
  -H "Authorization: Bearer $USER_TOKEN" \
  https://localhost:8443/api/loans/LN-100

Expected: HTTP/2 200 with the loan JSON.

Same call with Bob’s token against LN-100:

export BOB_TOKEN=$(
  curl -sf http://localhost:8180/realms/loanflow/protocol/openid-connect/token \
    --user loanflow-cli:loanflow-cli-secret \
    -H 'content-type: application/x-www-form-urlencoded' \
    -d 'username=bob&password=bob&grant_type=password' | jq -r '.access_token'
)

test -n "$BOB_TOKEN" && test "$BOB_TOKEN" != "null"

curl -i --cacert "$CA" \
  -H "Authorization: Bearer $BOB_TOKEN" \
  https://localhost:8443/api/loans/LN-100

Expected: HTTP/2 403 with {"error":"access_denied"}. Quarkus often returns an empty body on 403; this demo maps branch denials to a small JSON payload so the status is obvious in the terminal. The loan-service log should still show Loan access denied ... caller=bob callerBranch=hamburg loanBranch=berlin.

If you see 401 instead, the token is missing, expired, or stale — re-run the export BOB_TOKEN=... block above. Branch policy only runs after OIDC accepts the bearer token.

Prove the internal API is really internal. Fetch a service token:

export SERVICE_TOKEN=$(
  curl -sf http://localhost:8180/realms/loanflow/protocol/openid-connect/token \
    --user loan-service:loan-service-secret \
    -H 'content-type: application/x-www-form-urlencoded' \
    -d 'grant_type=client_credentials' | jq -r '.access_token'
)

Call credit-service without a client certificate — expect TLS handshake failure:

curl --cacert "$CA" \
  -H "Authorization: Bearer $SERVICE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"loanId":"LN-100","applicantId":"alice"}' \
  https://localhost:8444/internal/credit-checks

Call with the loan-service certificate but no bearer token — expect 403:

curl -i --cacert "$CA" \
  --cert "$LOAN_CERT/tls.crt" \
  --key "$LOAN_CERT/tls.key" \
  -H 'content-type: application/json' \
  -d '{"loanId":"LN-100","applicantId":"alice"}' \
  https://localhost:8444/internal/credit-checks

The client certificate satisfies mTLS, but without a bearer token the caller has no OIDC permission — Quarkus returns 403. Plain HTTP tests in @QuarkusTest (no client cert) typically show 401 instead.

Call with mTLS and the service token — expect 200:

curl --cacert "$CA" \
  --cert "$LOAN_CERT/tls.crt" \
  --key "$LOAN_CERT/tls.key" \
  -H "Authorization: Bearer $SERVICE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"loanId":"LN-100","applicantId":"alice"}' \
  https://localhost:8444/internal/credit-checks

Drive the full flow:

curl --cacert "$CA" \
  -X POST \
  -H "Authorization: Bearer $USER_TOKEN" \
  https://localhost:8443/api/loans/LN-100/submit

Expected: 200, credit band in the response, audit document stored, second submit returns 409.

Watch the logs

Run each service in its own terminal. After a successful submit, you should see the trust chain play out across three consoles — user token at the edge, service token on internal hops:

loan-service     Loan submit loanId=LN-100 by alice branch=berlin
loan-service     Calling credit-service for loanId=LN-100
credit-service   Credit check loanId=LN-100 band=D caller=service-account-internal-calls
loan-service     Credit band D for loanId=LN-100
loan-service     Calling document-service for loanId=LN-100
document-service Stored audit document loanId=LN-100 branch=berlin creditBand=D caller=service-account-internal-calls
loan-service     Loan submitted loanId=LN-100 creditBand=D

Denied requests log too — branch mismatch on read returns 403 with a Loan access denied line; a second submit on the same loan returns 409 with Loan submit rejected.

The services log principals and loan ids only. They never print bearer tokens or JWT bodies.

Or run ./scripts/smoke-test.sh from the repo root once all three services are up — it walks every failure path in order.

Tests worth keeping

Module tests rot fast when someone renames a claim or changes a port, so keep at least:

• loan-service — LoanAccessPolicyTest (unit), LoanResourceSecurityTest (@QuarkusTest + @TestSecurity for branch/admin/401)

• credit-service / document-service — @QuarkusTest with OidcWiremockTestResource proving 401 without a token

For bearer-token tests on internal services, I would rather use Wiremock OIDC than drag Keycloak into every ./mvnw test run. For end-to-end transport behavior, the smoke script is still worth more than ten paragraphs of architecture confidence.

Troubleshooting

401 from loan-service when you expected branch 403 — the bearer token is missing, expired, or invalid. Fetch a fresh token and confirm test -n "$BOB_TOKEN". Branch policy runs only after OIDC validation succeeds.

curl prints nothing — Quarkus often returns an empty body on 401/403. Add -i to see the status line. A 403 with a valid token usually means the access token is missing realm roles. Decode the token and confirm it contains realm_access.roles with loan_officer:

echo "$USER_TOKEN" | awk -F. '{print $2}' | python3 -c "import sys,base64,json; p=sys.stdin.read().strip(); p+=('='*(-len(p)%4)); print(json.dumps(json.loads(base64.urlsafe_b64decode(p)), indent=2))"

After changing loanflow-realm.json, Dev Services does not overwrite an existing realm. Stop all services, run ./scripts/reset-keycloak.sh, start loan-service first, then fetch a fresh token.

curl: (77) error setting certificate verify locations — you are not in the loanflow-zero-trust/ repo root, or ./scripts/generate-certs.sh has not been run. cd to the repo and confirm infrastructure/certs/ca/ca.crt exists. The smoke script resolves paths automatically; hand-typed curl commands need the same working directory.

PKIX path building failed — truststore missing the CA, or REST client points at the wrong TLS bucket.

TLS handshake fails before the request is logged — usually good news. client-auth=REQUIRED is doing its job; client certificate missing or untrusted.

401 from internal services — bearer token missing, expired, wrong issuer, or OIDC metadata unreachable.

403 from internal services — mTLS succeeded but bearer token missing or token valid without the permission required by @PermissionsAllowed.

403 from loan-service — business rule fired. Check branch claim, roles, and seeded loan branch.

Enable OIDC trace logging when token validation is unclear:

quarkus.log.category."io.quarkus.oidc.runtime.OidcProvider".level=TRACE
quarkus.log.category."io.quarkus.oidc.runtime.OidcProvider".min-level=TRACE

Make it survive production

This demo is deliberately local and small. Production needs a few extra moves:

• Replace the local CA with real internal PKI or a cert-manager pipeline

• Move secrets out of application.properties into a supported secret source

• Add quarkus.ssl.native=true for services that make HTTPS calls in native mode

• Rotate certificates and client secrets on a schedule that exists outside human memory

• Add correlation IDs so a rejected downstream request traces across services

• Consider audience enforcement or token exchange if downstream services later need user context

I would not move branch policy into internal services just because the local demo works. That is where these systems start getting weird.

Close the loop

In a microservice system, zero-trust is only useful when each trust decision is visible and testable.

In this design, the channel is protected with mTLS, the internal caller proves itself with a service token, and user-level policy stays where the loan aggregate actually lives. When a request fails, you can tell whether it failed at transport, service identity, or business policy. That is the part I care about, because it turns security from vibes into something you can actually debug.