Quarkus Goblin: Learn Chaos Engineering on One REST Call

Build a quote API that calls inventory, inject latency and HTTP 503s, and watch timeout, retry, and fallback take over.

Every month, I check the list of new Quarkus extensions. This time, Goblin caught my eye in the August list.

Chaos engineering. Experimental. A project called Goblin. Three things I like. Let’s test it.

I started the application and got this:

WARN  Chaos engineering active: 100% of REST requests subject to assault
      (latency=true, exception=false, httpStatus=false, dependencyDegradation=false)

Quarkus Goblin supplies failures at the JAX-RS boundary. It can delay a request, throw an exception, return a chosen HTTP status, or return a fixed dependency-degradation response. Package, annotation, and percentage filters confine the experiment to selected endpoints. That saves the rest of the application from joining the exercise without explicit consent and help us to narrow the tests.

Goblin intercepts an incoming REST request before the resource method runs. Putting @Timeout and @Fallback on that resource leaves the injected failure outside those interceptors. Goblin fails first, so the experiment evaluates the caller.

We need a real HTTP boundary. A public quote resource calls an internal inventory resource through a Quarkus REST client. Both run in one JVM to keep the example compact. Package targeting limits Goblin to the inventory resource, which leaves the public endpoint available while the nested request slows down or fails.

What We Are Building

The finished application has this request path:

curl
  -> GET /quotes/{sku}
  -> InventoryGateway (@Timeout, @Retry, @Fallback)
  -> Quarkus REST Client
  -> Goblin request filter
  -> GET /internal/inventory/{sku}

With chaos inactive, the quote contains live inventory. Latency above the gateway timeout or a forced 503 produces an HTTP 200 quote marked as fallback data. We will start with deterministic failures, then change the experiment through the Dev UI to cover exceptions, degradation, combined assaults, and percentage targeting.

Before You Start

The extension catalog labels Goblin as experimental, so plan for API and configuration changes after this first release.

• JDK 25 on PATH

• The Quarkus CLI

• curl

• About ☕️☕️

I used JDK 25 for this tutorial. The extension catalog currently lists Java 17 as requirement but the Goblin 0.0.1 build requires and targets Java 25. At least we don’t need a container runtime or anything else this time.

Create the Project

Create the application or start from the ready made project in my Github repository:

quarkus create app com.themainthread:quarkus-goblin-resilience \
  -P io.quarkus.platform:quarkus-bom:3.38.3 \
  --java=25 \
  --no-code \
  --extensions=rest-jackson,rest-client-jackson,smallrye-fault-tolerance
cd quarkus-goblin-resilience

quarkus-rest-jackson exposes the two JSON endpoints. quarkus-rest-client-jackson makes the loopback inventory call and maps its JSON. quarkus-smallrye-fault-tolerance provides @Timeout, @Retry, and @Fallback.

The Quarkus platform BOM leaves Goblin unmanaged, so pin the 0.0.1 release in pom.xml. Add the version to <properties>:

<goblin.version>0.0.1</goblin.version>

Then add the extension and RestAssured to <dependencies>:

<dependency>
    <groupId>io.quarkiverse.goblin</groupId>
    <artifactId>quarkus-goblin</artifactId>
    <version>${goblin.version}</version>
</dependency>
<dependency>
    <groupId>io.rest-assured</groupId>
    <artifactId>rest-assured</artifactId>
    <scope>test</scope>
</dependency>

Build the HTTP Boundary

The inventory endpoint is super simple. Goblin adds the failure behavior without changing this class.

Create src/main/java/com/themainthread/goblin/inventory/InventorySnapshot.java:

package com.themainthread.goblin.inventory;

public record InventorySnapshot(String sku, int available, boolean expressEligible, String source) {
}

Create src/main/java/com/themainthread/goblin/inventory/InventoryResource.java:

package com.themainthread.goblin.inventory;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/internal/inventory")
@Produces(MediaType.APPLICATION_JSON)
public class InventoryResource {

    @GET
    @Path("/{sku}")
    public InventorySnapshot inventory(@PathParam("sku") String sku) {
        return new InventorySnapshot(sku, 17, true, "live");
    }
}

When this method runs, source is always live. An exception or HTTP-status assault stops the request inside Goblin’s filter, preventing resource execution and state changes. Latency adds its sleep before the method. A caller timeout can leave that server-side request running, so the resource may execute after the fallback has returned.

Now describe the same endpoint as a REST client. Create src/main/java/com/themainthread/goblin/inventory/InventoryClient.java:

package com.themainthread.goblin.inventory;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

@Path("/internal/inventory")
@Produces(MediaType.APPLICATION_JSON)
@RegisterRestClient(configKey = "inventory")
public interface InventoryClient {

    @GET
    @Path("/{sku}")
    InventorySnapshot inventory(@PathParam("sku") String sku);
}

This loopback crosses real HTTP. The client opens a connection to the application, and the new request passes through Goblin’s JAX-RS filter before InventoryResource can run. A multi-service test would point the same interface at a second Quarkus process with Goblin installed there.

Put the Resilience Policy Around the Client

Create src/main/java/com/themainthread/goblin/inventory/InventoryGateway.java:

package com.themainthread.goblin.inventory;

import java.time.temporal.ChronoUnit;

import jakarta.enterprise.context.ApplicationScoped;

import org.eclipse.microprofile.faulttolerance.Fallback;
import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.faulttolerance.Timeout;
import org.eclipse.microprofile.rest.client.inject.RestClient;

@ApplicationScoped
public class InventoryGateway {

    private final InventoryClient client;

    public InventoryGateway(@RestClient InventoryClient client) {
        this.client = client;
    }

    @Timeout(value = 750, unit = ChronoUnit.MILLIS)
    @Retry(maxRetries = 1, delay = 50, jitter = 0, delayUnit = ChronoUnit.MILLIS)
    @Fallback(fallbackMethod = "fallbackInventory")
    public InventorySnapshot inventory(String sku) {
        return client.inventory(sku);
    }

    InventorySnapshot fallbackInventory(String sku) {
        return new InventorySnapshot(sku, 0, false, "fallback");
    }
}

The timeout applies to each attempt. One retry gives a transient failure a second chance while capping downstream traffic at two calls. jitter=0 fixes the retry delay at 50 milliseconds for this small lab. After two failures, the fallback returns zero available items and removes express eligibility because the system has no confirmed stock.

The source field exposes the degradation to callers and tests. A production API might put that signal in response metadata or telemetry. Keep fallback data distinct from a confirmed result wherever the signal lives.

Expose the Public Quote

Create src/main/java/com/themainthread/goblin/api/QuoteResponse.java:

package com.themainthread.goblin.api;

public record QuoteResponse(String sku, String service, int available, String source) {
}

Create src/main/java/com/themainthread/goblin/api/QuoteResource.java:

package com.themainthread.goblin.api;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

import com.themainthread.goblin.inventory.InventoryGateway;
import com.themainthread.goblin.inventory.InventorySnapshot;

@Path("/quotes")
@Produces(MediaType.APPLICATION_JSON)
public class QuoteResource {

    private final InventoryGateway inventoryGateway;

    public QuoteResource(InventoryGateway inventoryGateway) {
        this.inventoryGateway = inventoryGateway;
    }

    @GET
    @Path("/{sku}")
    public QuoteResponse quote(@PathParam("sku") String sku) {
        InventorySnapshot inventory = inventoryGateway.inventory(sku);
        String service = inventory.expressEligible() ? "EXPRESS" : "STANDARD";
        return new QuoteResponse(sku, service, inventory.available(), inventory.source());
    }
}

The package boundary makes targeting explicit. QuoteResource lives under com.themainthread.goblin.api; the resource we want to break lives under com.themainthread.goblin.inventory. Goblin targets resource packages by prefix, so this separation gives us a stable public control path and one failure target.

Configure a Deterministic First Experiment

Replace src/main/resources/application.properties with:

quarkus.http.port=8080
quarkus.http.test-port=8081

quarkus.rest-client.inventory.url=http://127.0.0.1:${quarkus.http.port}
%test.quarkus.rest-client.inventory.url=http://127.0.0.1:${quarkus.http.test-port}

%dev.quarkus.goblin.enabled=true
%dev.quarkus.goblin.assault.type=LATENCY
%dev.quarkus.goblin.assault.latency.min-milliseconds=1200
%dev.quarkus.goblin.assault.latency.max-milliseconds=1500
%dev.quarkus.goblin.target.level=100
%dev.quarkus.goblin.target.include-packages=com.themainthread.goblin.inventory

%test.quarkus.goblin.enabled=true
%test.quarkus.goblin.assault.type=HTTP_STATUS
%test.quarkus.goblin.assault.http-status.code=503
%test.quarkus.goblin.assault.http-status.message=Inventory unavailable (Goblin test)
%test.quarkus.goblin.target.level=100
%test.quarkus.goblin.target.include-packages=com.themainthread.goblin.inventory

The dev profile starts with 1.2–1.5 seconds of latency, safely above the 750-millisecond timeout. The package prefix keeps Goblin away from /quotes, so the client can observe the inventory failure and run its fallback.

Goblin can also exclude package prefixes and fully qualified annotation names. Annotation filtering stops at the REST resource method and its declaring class. Annotations on a service such as InventoryGateway sit outside that lookup. For this caller-side fault-tolerance test, target the resource package.

The test profile replaces wall-clock assertions with a fixed 503. Timing assertions often only report a busy CI runner. We want a stable business condition.

Static properties choose one assault type at startup. In dev mode, Goblin copies those values into mutable in-memory configuration; the Dev UI can then enable several types at once. Changes apply to the next request and disappear when the process restarts.

Run the Latency Assault

Start the application:

./mvnw quarkus:dev

Goblin announces the initial experiment:

WARN  Chaos engineering active: 100% of REST requests subject to assault
      (latency=true, exception=false, httpStatus=false, dependencyDegradation=false)

Before running the request, predict which endpoint will be slow. Goblin targets the nested inventory call, so the public endpoint can still answer. Call the quote endpoint:

curl -s -w '\nstatus=%{http_code} time=%{time_total}s\n' \
  http://localhost:8080/quotes/sku-1

The time trick measures this response:

{"available":0,"service":"STANDARD","sku":"sku-1","source":"fallback"}
status=200 time=1.570161s

The exact time changes because Goblin chooses a random delay and the gateway retries once. The stable facts are the 200 status and source=fallback. Each inventory attempt exceeds 750 milliseconds, so the public request completes after two failed attempts plus the retry delay.

Call the target directly:

curl -s -w '\nstatus=%{http_code} time=%{time_total}s\n' \
  http://localhost:8080/internal/inventory/sku-1

A direct caller has no timeout policy, so it waits for Goblin and eventually receives the live response:

{"available":17,"expressEligible":true,"sku":"sku-1","source":"live"}
status=200 time=1.381317s

The two responses show the boundary. Goblin creates the slow server response; the caller’s policy turns that delay into a timeout, a retry, or a fallback.

Change the Failure from the Dev UI

Open http://localhost:8080/q/dev and open the Goblin Chaos Dashboard. The 0.0.1 dashboard has a master activation control, four assault toggles, type-specific settings, and the target percentage. It also writes every change to the application log.

Start by clicking Deactivate and call /quotes/sku-1 again. The response now contains the live values:

{"available":17,"service":"EXPRESS","sku":"sku-1","source":"live"}

Activate Goblin again and try the remaining assault types one at a time:

1. Disable Latency, enable HTTP Status, set the code to 503, and save it. The REST client converts the 503 response into a failure, the gateway retries once, and the quote returns source=fallback.

2. Disable HTTP Status, enable Exception, and use java.lang.RuntimeException with a recognizable message. Goblin throws before InventoryResource.inventory() runs; the outer gateway again owns the recovery decision.

3. Disable Exception and enable Dependency Degradation. This is the short form of an unavailable dependency: a fixed 503 with Dependency unavailable (Goblin chaos) as its body.

4. Enable Latency together with HTTP Status. Goblin delays first and then aborts with the chosen status, which models a dependency that wastes most of the timeout budget before failing.

Latency can precede one terminal result. Exception has priority over HTTP status and dependency degradation, so a thrown exception ends processing before either later assault can run.

In 0.0.1, the exception class setting contains a cast that deserves some attention. The implementation constructs the configured class and casts it to RuntimeException; a failed cast produces a plain RuntimeException with the configured message. Choose an unchecked exception class when the policy depends on its exact type.

Add intermittent failures last

After the deterministic case passes, disable latency, leave the HTTP-status assault enabled, set Target Level to 25, and call the inventory endpoint directly:

for request in {1..20}; do
  curl -s -o /dev/null -w '%{http_code}\n' \
    http://localhost:8080/internal/inventory/sku-1
done | sort | uniq -c

A typical short run contains both statuses, perhaps 15 responses with 200 and five with 503. Those counts are observations only. Goblin makes an independent random decision for each inbound inventory request.

Now account for the retry. The public quote reaches fallback when both inventory attempts receive an assault. With a 25% target level and independent selection, the fallback probability becomes 0.25 × 0.25 = 6.25%. Retries reduce the failure rate observed at the outer boundary, which is why the percentage experiment comes after the deterministic case.

Open Assault History to see which resource methods Goblin affected. Version 0.0.1 keeps the latest 1,000 records in memory and shows the time, method, and assault type. Clear the history before each experiment so every row belongs to the current configuration.

Turn the Experiment into a Test

Goblin also activates in Quarkus test launch mode. Create src/test/java/com/themainthread/goblin/api/QuoteResourceTest.java:

package com.themainthread.goblin.api;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;

@QuarkusTest
class QuoteResourceTest {

    @Test
    void returnsFallbackQuoteWhenGoblinBreaksInventory() {
        given()
                .when().get("/quotes/sku-1")
                .then()
                .statusCode(200)
                .body("sku", equalTo("sku-1"))
                .body("service", equalTo("STANDARD"))
                .body("available", equalTo(0))
                .body("source", equalTo("fallback"));
    }

    @Test
    void targetsTheInternalResourceDirectly() {
        given()
                .when().get("/internal/inventory/sku-1")
                .then()
                .statusCode(503)
                .body(equalTo("Inventory unavailable (Goblin test)"));
    }
}

Run the suite:

./mvnw test

The expected result is deterministic:

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The second test proves that Goblin creates the internal 503. The first checks the full path: the public endpoint remains reachable, its REST client encounters that 503, and the fault-tolerance policy produces conservative fallback data.

Goblin 0.0.1 Boundaries

Goblin injects faults at the HTTP layer. Pod termination, network partitions, database corruption, connection-pool exhaustion, and process pauses require a cluster-level or infrastructure chaos tool.

This demo retries an idempotent GET. A write needs an idempotency key or another deduplication boundary before it can share this retry policy. The first attempt may commit after the caller starts its retry, producing two writes from one latency experiment.

Latency calls Thread.sleep() inside the JAX-RS filter. This produces blocked requests and caller timeouts while consuming server request capacity. Keep the experiment bounded, watch the request pool, and separate injector load from the dependency behavior under test.

Target selection uses string-prefix matching on the resource class’s package. com.themainthread.goblin.inventory also matches its subpackages. Keep the prefix specific, start at 100 percent, verify the targeted method in Assault History, and introduce probability after that check.

The dashboard state and history belong to one JVM and disappear on restart. Use them as development evidence; an audit trail needs durable storage. Each process also keeps separate state, and version 0.0.1 supports dev and test experiments.

Verify the Production Boundary

Stop dev mode, package the application, and run it normally:

./mvnw package -DskipTests
java -jar target/quarkus-app/quarkus-run.jar

Call the inventory endpoint from another terminal:

curl -s -w '\nstatus=%{http_code} time=%{time_total}s\n' \
  http://localhost:8080/internal/inventory/sku-1

The response is live and returns at baseline speed:

{"available":17,"expressEligible":true,"sku":"sku-1","source":"live"}
status=200 time=0.101425s

The 0.0.1 build step activates its engine only for development and test launch modes.

I checked the packaged application because the 0.0.1 guide promises physical removal. The runtime JAR remains under target/quarkus-app/lib/main, and Quarkus lists goblin among installed production features. Launch-mode gating keeps the assault engine inactive.

Conclusion

We built a caller-and-server failure boundary in one Quarkus process, injected latency and HTTP failures before the internal resource ran, and proved that the caller returns honest degraded data through its retry, timeout, and fallback policy. Use Goblin in dev/test cycles with deterministic targeting first, keep the recovery assertion with the caller, and test infrastructure failures separately.