matchbox.health blog

Monitoring Matchbox with OpenTelemetry

2026-10-19T00:00:00Z

If you run Matchbox as a shared FHIR validation service, sooner or later someone will ask how many resources you validate, why some $validate calls are so slow, or whether the server is running out of memory. Recent Matchbox releases can answer these questions themselves by exporting metrics, traces and logs with OpenTelemetry. In this tutorial, you'll turn that on, send the data to Traceway (a simple and standalone monitoring platform) and set up a dashboard and alerts for your Matchbox server.

You'll need Docker with Docker Compose, and a shell with curl and openssl. You don't need any prior experience with OpenTelemetry or Traceway.

What Matchbox measures

Matchbox produces three kinds of telemetry. Metrics are numbers tracked over time, like the number of validations per minute. Traces record a single request step by step: each step, such as a database query, is a span with its own duration. Logs are the messages Matchbox writes as it works, like the ones you see in its console.

Within metrics, we distinguish three types of data:

Counters are running totals that only ever go up, like the number of validations since Matchbox started. What you usually chart is how fast they grow, e.g. validations per second.
Timers measure how long something takes, each time it happens. A timer also counts the events it measures, so matchbox.validation.duration gives you both the number of validations and their total, average and maximum duration.
Gauges hold a current value that can go up or down, like the heap memory in use or the number of cached engines. Matchbox reads the value each time it exports metrics.

Matchbox gets most of this from the OpenTelemetry Spring Boot starter, a library that adds OpenTelemetry to Spring Boot applications. On top of that, Matchbox adds a few metrics of its own.

Spring Boot metrics

Matchbox is a Spring Boot application, and Spring Boot already measures a lot about itself through Micrometer, its built-in metrics library. When otel.instrumentation.micrometer.enabled=true is set, the starter forwards all of these metrics to OpenTelemetry under the same names. The most useful ones for a Matchbox server are:

jvm.memory.used (gauge): memory in use, in bytes, per memory pool. The area tag separates heap from nonheap memory, and id names the pool. jvm.memory.max gives the matching limits.
jvm.memory.usage.after.gc (gauge): the share of long-lived heap memory still in use after the last garbage collection, from 0 to 1. This is the best single sign that Matchbox is running out of memory.
jvm.gc.pause (timer): garbage-collection pauses. During a pause, the JVM stops all other threads, so requests have to wait.
process.cpu.usage and system.cpu.usage (gauges): the CPU used by Matchbox and by the whole host, from 0 to 1.
jdbc.connections.active, jdbc.connections.idle and jdbc.connections.max (gauges): connections in the database connection pool. If active stays at max, requests are waiting for a free connection. In Matchbox, the pool is only reported under these names, not under HikariCP's own hikaricp.connections.*.
http.server.requests (timer): incoming HTTP requests, tagged with method, status and uri. HAPI FHIR serves the FHIR API through its own servlet instead of Spring MVC, so Spring can't tell the FHIR endpoints apart: they all end up under uri=UNKNOWN. Only Spring's own endpoints, such as Actuator's, get their real path.
logback.events (counter): log events, tagged by level. A sudden rise in error events is often the first sign that something is wrong.

Spring Boot also reports thread counts, Tomcat sessions, free disk space and more. To see the complete list on your server, call /matchboxv3/actuator/metrics.

The starter also takes a few measurements of its own, named after OpenTelemetry's conventions. These are exported as soon as OpenTelemetry is on, even without the Micrometer setting:

http.server.request.duration (timer): incoming HTTP requests, tagged with method and status code. Like http.server.requests, it counts the FHIR requests but doesn't break them down by endpoint. For validation times, matchbox.validation.duration is more precise.
JVM metrics such as jvm.memory.used, jvm.gc.duration, jvm.thread.count and jvm.cpu.time. They overlap with Micrometer's JVM metrics, and two of them even share a name (jvm.memory.used and jvm.memory.committed), but with different tags: jvm.memory.type and jvm.memory.pool.name instead of area and id. A filter on area only matches Micrometer's version. If you don't need both, turn off the starter's version with otel.instrumentation.runtime-telemetry.enabled=false.

Matchbox's own metrics

The following metrics can be enabled by setting otel.instrumentation.micrometer.enabled=true in your configuration.

matchbox.validation.count (counter): number of FHIR resources validated, through either the FHIR $validate operation or the Gazelle validation API (used by IHE's Gazelle test tools).
matchbox.validation.duration (timer): how long each validation took.
matchbox.transformation.count (counter): number of resources transformed with StructureMap/$transform.
matchbox.engines.cached.permanent.number and matchbox.engines.cached.transient.number (gauges): how many validation engines Matchbox holds in memory. An engine is a validator loaded with a particular set of ImplementationGuides (IGs). Matchbox caches engines so it doesn't rebuild them for every request. Permanent engines stay cached; transient engines expire automatically after an hour.
matchbox.engines.creating.number (gauge): how many engines Matchbox is building right now.
matchbox.igs.number (gauge): number of installed IGs.
matchbox.engines.fetch.cache.size (gauge, from Matchbox v5.0.0): the sum of all resources stored in the engines' fetch cache, that is configurable with MatchboxEngine.setFetchCacheMaxSize(int).

Traces

The starter traces every incoming HTTP request, along with the database queries Matchbox runs while handling it. Each request becomes one span, with attributes such as the URL path (url.path), the query parameters (url.query, e.g. the profile to validate against) and the HTTP status. HAPI FHIR does its own routing instead of Spring MVC, so these spans carry the concrete URL path but no route template. In Traceway, that means one endpoint per path: one for POST /matchboxv3/fhir/Patient/$validate, another for POST /matchboxv3/fhir/Observation/$validate, and so on.

Calls to the terminology server and package downloads aren't traced, because the validator makes them with HTTP clients that the starter doesn't instrument. Their time counts toward the request's duration, but you can't see it separately. Matchbox doesn't add spans of its own yet either, so a trace tells you which validations are slow, not why.

Database queries that run outside a request, at startup or in HAPI FHIR's background jobs, are exported too, but Traceway drops them because they don't belong to any endpoint.

Logs

Matchbox also sends its logs. The starter hooks into Logback, the logging library Matchbox uses, and exports every log event as soon as OpenTelemetry is on, with no further setting. Matchbox still writes the same lines to its console, so docker compose logs keeps working.

Each record carries its level, its message and the name of the logger that wrote it, e.g. ch.ahdis.matchbox.engine.MatchboxEngine. Only what Logback lets through is exported: INFO and above by default. The levels you set under logging.level apply to the console and the export alike. Expect a burst of a couple of hundred records at startup.

Logs written while Matchbox handles a request carry that request's trace ID, which ties them to the request. For a $validate request, these are lines such as "Using profile for validation …" and "finished validation …", plus any error, for example when the requested IG isn't installed. They also tell you whether Matchbox reused a cached engine ("Using cached validate engine for …") or had to build one ("Creating new validate engine for …"), which the trace alone can't show. In Traceway, these lines appear in the Logs card of the endpoint's detail page, under This Trace. All logs, whether they're part of a trace or not, including the startup ones, are on the project's Logs page.

Gen-AI metrics (optional)

Matchbox can ask an LLM to analyze failed validations ("AI-assisted validation"). If you use that feature, Matchbox also records how many tokens each LLM call used (gen_ai.client.token.usage) and how long it took (gen_ai.client.operation.duration), broken down by provider, model, and input vs. output tokens. To set up AI-assisted validation, see the Validation with AI tutorial.

Where the data goes

OpenTelemetry (OTel) is an open standard for collecting metrics, traces and logs, and most monitoring tools support it. Matchbox sends its data over OTLP, OpenTelemetry's network protocol.

In this tutorial, that data goes to Traceway, an MIT-licensed observability platform. We picked it because its configuration is minimal, and deployment is easy: Traceway is a single container image; it receives OTLP directly, stores everything in SQLite, DuckDB or PostgreSQL, and has dashboards and alerting built in. This makes it a good solution for experimentation with telemetry. The usual self-hosted stack is an OpenTelemetry Collector plus one store per kind of data, such as Prometheus for metrics, Jaeger or Tempo for traces and Loki for logs, with Grafana on top for dashboards and alerts. That's several services to set up and wire together, but once that is done, they are much more powerful.

Nothing on the Matchbox side depends on Traceway, though. Matchbox only speaks standard OpenTelemetry, so this setup works the same with any backend that accepts OTLP, whether that's your own Grafana stack or a hosted service such as Grafana Cloud, Honeycomb or Dynatrace. To switch, you change the endpoint and the credentials. Matchbox sends the same metrics, traces and logs either way, although some backends adapt the names to their own conventions: Prometheus, for example, replaces the dots with underscores by default. What does differ from tool to tool is how you build dashboards and alerts, so treat the Traceway screens in Steps 4 and 5 as a guide to what to watch, not as exact instructions.

Step 1: Start Traceway and create a project

Matchbox needs a Traceway project token before it can send anything, so start Traceway first.

In an empty directory, create a docker-compose.yml file:

yaml

services:
  traceway:
    image: ghcr.io/tracewayapp/traceway:sqlite
    ports: ["80:80"]
    volumes: ["./traceway-data:/data"]
    environment:
      JWT_SECRET: ${JWT_SECRET}
      # The default cache is sized for machines with 16 GB+
      SQLITE_CACHE_SIZE_MB: "64"

This is Traceway's single-container image, which stores everything in SQLite under ./traceway-data. JWT_SECRET signs Traceway's login sessions and must be at least 32 characters long. Generate one into a .env file, which Docker Compose reads automatically, and start Traceway:

bash

echo "JWT_SECRET=$(openssl rand -hex 32)" > .env
docker compose up -d traceway

Then set up Traceway in your browser:

Open http://localhost/register and create an account. The first account becomes the owner of the organization.
In the header, open the project selector and select Add Project.
Name the project Matchbox, choose the OpenTelemetry framework, and select New Project.
Traceway shows the project's token. Add it to your .env file:

bash

echo "TRACEWAY_TOKEN=<your project token>" >> .env

If you lose the token, you can find it again on the project's Connection page.

Step 2: Enable telemetry in Matchbox

Telemetry is off by default. The stock application.yaml contains:

yaml

otel:
  sdk:
    disabled: true

On top of that, Matchbox only registers its own metrics when otel.instrumentation.micrometer.enabled is true. So nothing gets exported until you opt in.

Add Matchbox to your docker-compose.yml, under services:

yaml

  matchbox:
    image: europe-west6-docker.pkg.dev/ahdis-ch/ahdis/matchbox:latest
    ports: ["8080:8080"]
    environment:
      OTEL_SDK_DISABLED: "false"
      OTEL_SERVICE_NAME: matchbox
      OTEL_EXPORTER_OTLP_ENDPOINT: http://traceway/api/otel
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer ${TRACEWAY_TOKEN}"
      OTEL_INSTRUMENTATION_MICROMETER_ENABLED: "true"
      OTEL_METRIC_EXPORT_INTERVAL: "15000"

Here's what each setting does:

OTEL_SDK_DISABLED=false turns on OpenTelemetry. From then on, Matchbox exports traces and logs, with no further setting.
OTEL_SERVICE_NAME is the name Traceway shows for this server.
OTEL_EXPORTER_OTLP_ENDPOINT points at Traceway. It must end in /api/otel: without /api, Traceway's web interface answers instead, and the data silently disappears.
OTEL_EXPORTER_OTLP_HEADERS sends the project token, so Traceway knows which project the data belongs to.
OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true forwards Spring Boot's metrics and turns on the Matchbox-specific ones.
OTEL_METRIC_EXPORT_INTERVAL=15000 sends metrics every 15 seconds instead of every 60, so you see data sooner.

Start Matchbox:

bash

docker compose up -d matchbox

If no data shows up in Traceway a minute after Matchbox has started, check docker compose logs matchbox for export errors. A 401 means the token is missing or wrong.

Outside of this tutorial, you'll probably keep these settings in an override file instead. Spring Boot merges it over the defaults; with Docker, mount it at /config/application.yaml. Each environment variable above maps to a key in that file:

yaml

otel:
  sdk:
    disabled: false
  service:
    name: matchbox
  exporter:
    otlp:
      endpoint: https://traceway.example.com/api/otel
      headers:
        Authorization: Bearer <your project token>
  instrumentation:
    micrometer:
      enabled: true
  metric:
    export:
      interval: 15000 # milliseconds

A few notes:

Metrics only. Traces and logs are exported as soon as OpenTelemetry is on. To turn them off, add OTEL_TRACES_EXPORTER=none and OTEL_LOGS_EXPORTER=none (otel.traces.exporter: none and otel.logs.exporter: none in the file).
Sampling. By default, Matchbox sends every trace. On a busy server, you can keep only a share of them by adding OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 (keeps 10 %). Keep in mind that Traceway computes endpoint latency and error rates from the traces it receives, so with sampling, those numbers are based on a subset of requests.
Actuator works without OpenTelemetry. Spring Boot Actuator, Spring Boot's built-in management endpoints, serves /matchboxv3/actuator/metrics even with otel.sdk.disabled=true. That's handy for a quick check. The settings above add the export to Traceway and the Matchbox-specific metrics.
Context path. All Actuator endpoints live under the Matchbox context path, e.g. http://localhost:8080/matchboxv3/actuator/health.

Step 3: Generate some load

Matchbox needs an ImplementationGuide to validate against. These examples use hl7.fhir.r4.core, which is always installed.

bash

MB=http://localhost:8080/matchboxv3
PROFILE=http://hl7.org/fhir/StructureDefinition/Patient

# A valid Patient
curl -s -X POST "$MB/fhir/Patient/\$validate?profile=$PROFILE" \
  -H 'Content-Type: application/fhir+json' \
  -d '{"resourceType":"Patient","name":[{"family":"Test"}],"gender":"other"}'

# An invalid one — wrong data type for name
curl -s -X POST "$MB/fhir/Patient/\$validate" \
  -H 'Content-Type: application/fhir+json' \
  -d '{"resourceType":"Patient","name":"Test"}'

The backslash in \$validate stops the shell from treating $validate as a variable.

Two requests won't make an interesting graph, so send a couple of hundred more:

bash

for i in $(seq 1 200); do
  curl -s -o /dev/null -X POST "$MB/fhir/Patient/\$validate" \
    -H 'Content-Type: application/fhir+json' \
    -d '{"resourceType":"Patient","name":"Test"}'
done

For a heavier, sustained load, the Matchbox repository also includes a JMeter test plan in its jmeter/ directory.

Step 4: Read the results

4a. Actuator, for a quick look

bash

curl -s $MB/actuator/metrics/matchbox.validation.duration

json

{
  "name": "matchbox.validation.duration",
  "description": "Duration of FHIR resource validation by Matchbox",
  "baseUnit": "seconds",
  "measurements": [
    { "statistic": "COUNT", "value": 28 },
    { "statistic": "TOTAL_TIME", "value": 34.953 },
    { "statistic": "MAX", "value": 20.833 }
  ]
}

So far, Matchbox has run 28 validations, taking 35 seconds in total, and the slowest one took almost 21 seconds.

bash

curl -s $MB/actuator/metrics/matchbox.engines.cached.permanent.number

json

{
  "name": "matchbox.engines.cached.permanent.number",
  "description": "Number of cached immutable Matchbox engines in the server",
  "baseUnit": "engines",
  "measurements": [ { "statistic": "VALUE", "value": 3 } ]
}

That confirms Matchbox is recording metrics. Now switch to Traceway at http://localhost.

4b. Latency and errors: the Endpoints page

Traceway turns incoming HTTP requests into endpoints, named after their method and route, and shows latency percentiles, throughput and error rate for each one. Endpoints are ranked by an impact score that combines latency, errors and volume, so the ones that hurt the most come first.

Open Endpoints in the Matchbox project and find the $validate endpoints, one per resource type, such as POST /matchboxv3/fhir/Patient/$validate. This is where you get validation percentiles. The p95, for example, is the time within which 95 % of validations finish. Traceway computes percentiles from traces, not from metrics, which is why the dashboard below only shows average durations.

4c. Traces: one request in detail

Click one of the $validate endpoints to open its detail page. The Spans waterfall shows a single request step by step: the top line is the whole request, and the indented lines are the database queries Matchbox ran while handling it, labelled with their SQL.

For most validations, the waterfall is a single line:

POST /matchboxv3/fhir/Observation/$validate   296 ms

Matchbox found everything it needed in a cached engine, so it didn't query the database. When a validation has to look something up in the package database, for instance to find the IG that defines the requested profile, the queries appear underneath:

POST /matchboxv3/fhir/Patient/$validate                       9.4 ms
├─ select npvre1_0.pid, … from npm_package_ver_res … join …   0.1 ms
└─ select npvre1_0.pid, … from npm_package_ver_res … join …   0.1 ms

In the first example, most of the 296 ms went to checking a LOINC code on tx.fhir.org, but nothing in the trace says so: the terminology call is hidden inside the request's duration. Validating the same resource a second time took 18 ms, because Matchbox caches terminology results.

So use traces to find the slow requests, and their url.query attribute to see which profile they were validated against. If the same validation is slow the first time and fast afterwards, Matchbox was either building a validation engine or waiting for the terminology server, and caches both. The Logs card below the waterfall tells you which: it shows whether the request created a new engine.

4d. A Matchbox dashboard

Traceway shows metrics in dashboards made of widgets. You configure each widget in the interface by picking a metric, an aggregation such as avg or last, and optional tag filters or a group-by.

Three things to know about how Matchbox's metrics look in Traceway:

Names keep their dots, e.g. matchbox.validation.count.
Counters arrive as running totals. Use the rate aggregation to turn them into "per second".
Timers arrive as histograms, which Traceway reduces to <name>.avg (average duration) and <name>.count (number of measurements). Traceway doesn't keep the histogram buckets, so for percentiles, use the Endpoints page.

Open the dashboard picker (⌘K, or Ctrl K on Windows and Linux), create a dashboard for the Matchbox project, and add these widgets:

Traffic
- Validations per second: line chart of matchbox.validation.count, aggregation rate.
- Transformations per second: line chart of matchbox.transformation.count, aggregation rate.
Latency
- Average validation time: line chart of matchbox.validation.duration.avg, aggregation avg.
Capacity
- Heap used: stacked area of jvm.memory.used, filtered on area = heap, grouped by id (one band per memory pool).
- GC pauses: line chart of jvm.gc.pause.avg, aggregation avg.
- Database connections in use: line chart of jdbc.connections.active, aggregation last.
Matchbox internals
- Cached engines: stacked area of matchbox.engines.cached.permanent.number and matchbox.engines.cached.transient.number, aggregation last. This one needs JSON, see below.
- Installed IGs: single value of matchbox.igs.number, aggregation last.

The widget editor only lets you pick one metric per widget. A widget can hold several sources, though, and a stacked area piles them on top of each other. To add the cached-engines widget, open the dashboard tab's menu, select Edit as JSON, and add this entry to the widgets array:

json

{
  "title": "Cached engines",
  "widgetType": "stacked_area",
  "config": {
    "sources": [
      { "type": "metric", "name": "matchbox.engines.cached.permanent.number", "aggregation": "last", "label": "Permanent" },
      { "type": "metric", "name": "matchbox.engines.cached.transient.number", "aggregation": "last", "label": "Transient" }
    ]
  }
}

Traceway generates the widget's id when you save. The label fields name the two bands.

Here's the result of such a dashboard. Traceway's dashboards are simpler than Grafana's. The rate aggregation, for instance, is always per second, which is too fine for events as rare as validations. Still, the dashboard gives a good overview of the Matchbox server.

4e. Gen-AI usage

If AI-assisted validation is enabled, each failed $validate that triggers an LLM analysis also records token usage. The token metric is a histogram too, so Traceway shows it as two metrics:

gen_ai.client.token.usage.avg: average tokens per LLM call. Group by gen_ai.token.type to split input and output tokens.
gen_ai.client.token.usage.count: number of LLM calls. Use rate for calls per second.

Average tokens per call times the number of calls, times your provider's per-token price, gives you an estimate of what AI-assisted validation costs you. For LLM latency, chart gen_ai.client.operation.duration.avg.

Step 5: Alerts worth having

In Traceway, you create alert rules in the interface, under Alerts. Each rule sends its notifications to one channel, such as email, Slack or a webhook, so set up a channel first. These three rules cover the basics; adjust the thresholds to your traffic:

Server errors. Type Error Rate, with a threshold of 5 %. This counts server errors (HTTP 5xx) on all endpoints.
Heap pressure. Type Metric Threshold, on jvm.memory.usage.after.gc, with aggregation max, operator gt, threshold 0.9 and a lookback of 15 minutes. This metric is the share of long-lived heap memory still in use after garbage collection, from 0 to 1, so it fires when the JVM is close to running out of memory.
Matchbox has gone quiet. Type No Data, data type endpoints, with 30 silence minutes. It fires when Traceway hasn't seen a single request from Matchbox for 30 minutes, which usually means the service is down or unreachable.

Keep an eye on the engine-cache gauges too, either on the dashboard or with a Metric Threshold rule. If matchbox.engines.cached.transient.number keeps climbing while heap usage rises, Matchbox is creating session engines faster than they expire. In that case, look at the cache expiry time and the onlyOneEngine setting.

Wrap-up

Matchbox's telemetry stays quiet until you ask for it. Two settings turn it on:

otel.sdk.disabled=false starts OpenTelemetry and the export, traces and logs included.
otel.instrumentation.micrometer.enabled=true forwards metrics and enables the Matchbox-specific ones.

If you only want metrics, otel.traces.exporter=none and otel.logs.exporter=none turn traces and logs off.

To send everything to Traceway, point the exporter at Traceway's /api/otel endpoint and pass your project token in the Authorization header. The rest is standard OpenTelemetry, so moving to another backend later only means changing those two settings.

With that in place, the questions from the start of this post become dashboard widgets and alerts: how much you validate and transform and how fast, how the engine cache behaves, how much memory headroom is left, which requests are slow, and how many LLM tokens AI validation uses.

References

Matchbox on GitHub: see MatchboxMetricsConfig, MatchboxMetrics and application.yaml
Running Matchbox in Docker
Tutorial: Validation with AI
Traceway: sending OpenTelemetry data, the SQLite container, dashboards and alerts
OpenTelemetry Spring Boot starter
OpenTelemetry SDK configuration (env vars)
Spring Boot Actuator metrics

Matchbox: from 12 GB to 4 GB – 24 hours with an AI agent and a test setup

2026-09-26T00:00:00Z

In 24 hours, Matchbox went from a fixed 12 GB heap to a 4 GB container for a typical validation server. The work was done by an AI agent, Claude Code with Opus 5.5: it ran the builds, the load tests and the heap dump analysis, and proposed and implemented the optimizations, while a human stayed in the loop for every decision. This article describes how we got there, and why a reproducible load test mattered as much as the agent.

Matchbox and why validation matters

FHIR®, the HL7 standard for healthcare interoperability, leaves a lot of freedom in how data is represented. To guarantee the correctness of an exchange like a lab report or a patient summary, FHIR resources are validated against the rules published in FHIR implementation guides (IGs): which fields are required, which codes are allowed, which cardinalities apply.

Matchbox is our open-source FHIR validation and mapping tooling. It wraps the official HL7 FHIR validator in a server with its own FHIR API and a web GUI, supports multiple IGs at once, and also offers validation through an MCP service. We started in August 2020 and have published 98 releases since then. It runs in production infrastructure (e.g. the Swiss notification of laboratory results, ch-elm), in test platforms such as IHE Gazelle at IHE connectathons and projectathons, and in integration pipelines.

Memory became the bottleneck

FHIR IGs tend to get bigger: to stay aligned with international developments, they depend on European and international IGs, and each new terminology dependency makes validation hungrier for memory. Initially, we assumed this was a natural evolution and let the container grow. Then we were asked why an existing validation setup needed four times the memory within one year. In addition, several issues and pull requests from the community pointed to memory problems:

Achraf (@achrafachkari) found that the human-readable HTML narrative of every loaded resource stayed in memory (#566).
Valentin (@reva) measured better throughput with a smaller heap than the image's default of 12 GB and asked for configurable JVM options (#594, #595).
We ourselves hit an OutOfMemoryError on our memory-constrained test instance, while the health check still reported the server as up (#457).

In release 4.1.14 we had already added memory metrics to Matchbox, but observing the problem was not enough. Fortunately, we also had test setups for load testing.

The test setup: reproduce before you optimize

The test setups were mostly used to reproduce concurrency issues in validation: Apache JMeter sends 8,000 validations of a ch-elm lab report (4 threads) to a Matchbox container and asks the server for its memory use after each one. The results were trustworthy, but the comparison was manual on my development machine, and reruns were difficult to compare because the machine load could differ between runs.

So we first made the measurements reproducible. A script starts a fresh container and records the startup time, the first and the second validation, and the live heap. All runs happened on the same idle machine. To check servers with several IGs, a second JMeter test validates examples from different IGs, each of which gets its own validation engine. For a validator, a faster answer is worthless if it is a different answer, so every run also compared the validation results.

Working with an AI agent

Extending the test setup and the analysis were all done with Claude Code and Opus 5.5 (high effort) within VS Code. My job was to set the direction, review the findings and make every decision. What Claude Opus 5.5 did:

Bisecting across releases. It pulled or rebuilt ten ch-elm images of the last nine months (Matchbox 4.0.16 to 4.1.18), ran the same load test on each, and found in which release validation became slower and in which it became faster again, to get a clearer view of the problem.
Profiling and heap dumps. With Java Flight Recorder it recorded which code paths parsed resources. With Eclipse Memory Analyzer it analyzed heap dumps in headless mode: which objects keep the memory alive, per package and per validation engine.
Reading upstream code, recommending and implementing optimizations. It compared Matchbox with the official HL7 validator in org.hl7.fhir.core, changed the code, ran the tests and repeated the measurements until the numbers and the validation results matched.

In 24 hours this added up to about 35 load and startup test runs, 17 Docker image builds, 10 Matchbox versions compared and 9 heap dumps analyzed.

The culprits in memory consumption

The heap dumps showed where the memory went. For ch-elm, 44,650 conformance resources (profiles, value sets, code systems) were loaded, all parsed into Java objects, consuming about 1 GB. The IG and its dependencies alone pull in seven versions of the HL7 terminology package. The main culprits and their fixes:

Leftover package files (−171 MB). For a handful of small files, a helper kept a reference to the whole package it came from. So all raw files of hl7.fhir.r4.core and hl7.fhir.uv.xver-r5.r4 (210 MB) stayed in memory. It now keeps only the file it needs, which lowers the live heap by 171 MB.
Identical strings (−15%, about 220 MB). The many package versions contain a lot of identical text. Java's string deduplication (-XX:+UseStringDeduplication) lets the garbage collector share it, at no measurable cost. It is now on by default.
Everything parsed up front (−330 MB). The official validator loads resources lazily: it registers each resource with some metadata and parses it only when a validation needs it. Matchbox had never adopted this for its package loading. Now the terminology resources (code systems, value sets, naming systems, concept maps) are loaded lazily. Matchbox also pins the terminology resources of the FHIR core to their core versions; this step now reads the versions from the metadata instead of parsing every resource.
Unused definitions of the database layer (−40 MB). The search indexing of the underlying HAPI FHIR server loaded all 649 FHIR core profiles for a type analysis it never uses.

The agent discovered all four of these issues and presented approaches to solve them. Not every idea worked. Loading all resources lazily made the first validation take 2.4 instead of 0.85 seconds, because the validator walks through all profiles on first use anyway. So profiles are parsed at startup, like in the official validator, and terminology on demand.

Results for a single IG

For ch-elm, a typical production setup with one IG, the ch-elm image 1.15.3 (based on Matchbox 4.1.18) needs 660 MB of heap. That is less than a third of image 1.13.1, which still ran with 3 GB, and a fifth of image 1.14.1, which needs 3.4 GB and no longer fits into 3 GB. It validates three times faster than 1.13.1, and as fast as 1.15.2. Compared with 1.15.2, the memory drops by more than half and the server starts about 25% faster. Startup is still slower than with 1.13.1; as each image changes both the Matchbox and the ch-elm version, the two effects can't be separated here.

Matchbox release (ch-elm image)	4.0.16 (1.13.1)	4.1.9 (1.14.1)	4.1.17 (1.15.2)	4.1.18 (1.15.3)
Heap limit of the image	3 GB	12 GB	12 GB	70% of the container memory
Live heap after 8,000 validations	2,300 MB	3,420 MB	1,560 MB	660 MB
Server ready after	36–37 s	43–46 s	65–67 s	48–49 s
Validation under load (median of 8,000)	327 ms	250 ms	104 ms	108 ms
Runs the load test in a 3 GB heap	Yes	No (live heap 3.4 GB)	Yes	Yes, also in 1 GB

Matchbox creates one validation engine per IG/validation parameters, so that the rules of one IG cannot interfere with another. The heap dumps showed that, apart from the FHIR core and the base terminology, each engine loaded all its packages again. Swiss IGs share most of their dependencies, so with twelve IGs, 133,000 resource objects existed more than once: 2.3 GB of duplicates. Now a cache shares the loaded packages between the engines. It keeps only weak references, so a package is freed when the last engine using it is dropped after an hour without use.

We measured this with with-preload, a sample configuration of Matchbox that preloads 12 Swiss IGs with their dependencies (55 packages). With the new default in a container with a 4 GB limit (a heap of 2.8 GB), all 12 engines ran 400 validations without failures and used 3.5 GB of the container's memory.

with-preload: 12 Swiss IGs, 55 packages	Release 4.1.9	Release 4.1.17	Release 4.1.18
Heap limit of the image	12 GB	12 GB	70% of the container memory
Live heap with all 12 engines	10,630 MB	4,756 MB	1,187 MB
Memory held by the 12 IG engines alone	8,123 MB	3,465 MB	177 MB
Duplicated between engines	n.a.	2,278 MB	1 MB
Creating an IG engine (median)	19 s	48 s	16 s

What changes for you as a Matchbox user

The changes are released in Matchbox 4.1.18 and the ch-elm image 1.15.3:

A new memory default. The Docker image no longer sets a fixed 12 GB heap, which could even exceed the memory limit of a container. It now uses 70% of the container's memory limit (-XX:MaxRAMPercentage=70).
4 GB is enough for a typical setup. We recommend a memory limit of 4 GB for the container, also for multiple IGs. A single IG such as ch-elm runs comfortably with 3 GB.
Configurable JVM options. The options can be overridden with the JDK_JAVA_OPTIONS environment variable (#594). Setting it replaces the default, so include a heap setting, -XX:+ExitOnOutOfMemoryError and -XX:+UseStringDeduplication.
A restart instead of a zombie. On the first OutOfMemoryError, the server now exits (-XX:+ExitOnOutOfMemoryError) instead of continuing in an undefined state while the health check reports it as up, so that Docker or Kubernetes can restart it (#457).
Faster engines. Compared with 4.1.17, the server starts about 25% faster and creates IG engines three times faster.

bash

docker run -d --name matchbox -p 8080:8080 -m 4g europe-west6-docker.pkg.dev/ahdis-ch/ahdis/matchbox:latest

Lessons: a day instead of weeks with an agent

I could have done almost all of this myself. I'm familiar with load testing, bisecting releases, profiling and heap dump analysis, but less so with tools such as JFR or Eclipse Memory Analyzer. Even after we noticed the memory problem, I could not find the time: each step means waiting for a 30-minute test run, writing heap dump queries or reading code in a large upstream project.

Opus 5.5 took over exactly these steps: it ran them continuously, one after another and in parallel, in one session of 1 million tokens. I let it write a whole runbook and continued in a new session the next day, to verify that the setup can be reused and this is not a one-shot optimization.

The agent found features in the core libraries that we had missed, and proposed new approaches, such as sharing the loaded FHIR packages between the engines. I had not seen these qualities in agentic coding before. Opus 4.7 was the first model that could handle the Matchbox code, but Opus 5.5 is another leap: it can be used for software engineering, not just coding.

The reproducible test setup made this possible. Every proposal of the agent was measured against the same load test on the same machine, and every run checked that the validation results stayed the same. Every pull request goes through our Matchbox engine and server tests, which check specific validation features, and our integration tests, which check that the examples of the IGs we are interested in are still validated correctly. Without that, we could neither have trusted the improvements nor rejected the wrong ideas.

And a human in the loop stays essential. The agent proposed, measured and implemented, but not every proposal was right. Loading only the newest version of each terminology package would have saved 400 MB, but the IGs depend on specific versions, and validating against just the latest terminology version can change the results. Loading all resources lazily saved memory but made the first validation three times slower. I knew that multiple package versions are needed, which trade-offs are acceptable, and when a proposal was wrong.

In one day we delivered a Matchbox release that needs a fraction of the memory. But it needs the community to raise awareness and bring urgency to the problems, so thanks again to Achraf (@achrafachkari), Valentin (@reva) and especially my colleague Quentin (@qligier), co-maintainer of Matchbox, who has to review all my pull requests co-authored with Claude Code.

Disclaimer: As with the code, I used Claude to help me with this blog article, but I read and refined every sentence.

matchbox.health blog

Monitoring Matchbox with OpenTelemetry

What Matchbox measures

Spring Boot metrics

Matchbox's own metrics

Traces

Logs

Gen-AI metrics (optional)

Where the data goes

Step 1: Start Traceway and create a project

Step 2: Enable telemetry in Matchbox

Step 3: Generate some load

Step 4: Read the results

4a. Actuator, for a quick look

4b. Latency and errors: the Endpoints page

4c. Traces: one request in detail

4d. A Matchbox dashboard

4e. Gen-AI usage

Step 5: Alerts worth having

Wrap-up

References

Matchbox: from 12 GB to 4 GB – 24 hours with an AI agent and a test setup

Matchbox and why validation matters

Memory became the bottleneck

The test setup: reproduce before you optimize

Working with an AI agent

The culprits in memory consumption

Results for a single IG

Several IGs: share what engines have in common

What changes for you as a Matchbox user

Lessons: a day instead of weeks with an agent