ZB Field Notes

Spring events that survive a crash

Spring events that survive a crash

Somebody signs in to my site, and I get a one-line email about it. A trivial feature — until you ask what happens when the mail server is unreachable for ninety seconds.

The first version of that code is the version most of us write: an @Async method that sends the message and logs anything that goes wrong. It is fire-and-forget, and the “forget” half does more work than it looks. The business record is safely committed. The email is gone. What remains is a WARN line nobody reads, and nowhere in the system is there any record that a message is still owed.

Making that email durable is a small, well-understood pattern — a transactional outbox — and Spring already has the pieces. Here is the mechanism, the table it leans on, and the handful of details that decide whether it actually works.

Four ways to listen, and what each one promises

Spring has more listener flavours than most people use, and they differ in precisely the property that matters here: what survives a failure.

ListenerRuns whenOwn transaction?On failure
@EventListenerat publish, caller threadjoins the caller’spublisher rolls back too
@EventListener + @Asyncat publish, another threadnonesilently lost
@TransactionalEventListenerafter commitnorecoverable
… + @Async + REQUIRES_NEWafter commit, another threadyesrecoverable

A plain @EventListener is not unreliable, it is coupled: it runs inside the publisher’s transaction, so if it throws, the whole transaction rolls back with it. That is a real guarantee, just a limiting one.

The second row is the one worth staring at. @Async on a plain @EventListener fires the moment you call publishEvent — on another thread, before your transaction commits. The listener can query the database and fail to find the row it was just told about. If you have that combination anywhere, it is a live race, not a style preference.

Only AFTER_COMMIT makes “the publisher succeeded but the listener did not” a state the system can even represent. And that state is exactly what durability has to record.

Where the durability actually comes from

Spring Modulith’s event publication registry adds one move to the above. When you publish an event inside a transaction, it identifies every @TransactionalEventListener that matches and writes a row per listener into an outbox table — in that same transaction.

Diagram: a single transaction containing an INSERT into the business table and an INSERT into event_publication, followed by the after-commit listener that re-reads the record and sends the mail.
The business fact and the intent to notify are written by the same transaction, so they share its fate.

Then each listener is wrapped: the row is marked as processing, the listener runs, and it is marked complete only if the listener returns normally. Throw, and the row simply stays uncompleted — a durable, queryable record that a message is still owed.

That is the whole trick, and it is the same trick as a Debezium-style outbox, minus the change-data-capture pipeline.

One table carries the promise

CREATE TABLE event_publication (
  id                     UUID NOT NULL,
  listener_id            TEXT NOT NULL,
  event_type             TEXT NOT NULL,
  serialized_event       TEXT NOT NULL,
  publication_date       TIMESTAMPTZ NOT NULL,
  completion_date        TIMESTAMPTZ,
  status                 TEXT,
  completion_attempts    INT,
  last_resubmission_date TIMESTAMPTZ,
  PRIMARY KEY (id));

Four columns do the real work. listener_id means the unit of tracking is a listener, not an event — two listeners on one event get two independent fates. serialized_event is your payload as JSON, which is a design constraint worth respecting: everything you put in the event is now persisted outside your domain tables, with its own retention question. I publish an identifier and let the listener re-read the record, which keeps personal data out of the outbox entirely. That is safe precisely because the listener runs after commit.

completion_date is the honest signal — NULL means still owed. And completion_attempts is what any retry cap has to be expressed against.

If migrations are the source of truth in your project, own this table with one: the library will happily create it for you, and two mechanisms creating the same table is a problem you only discover in production.

The lifecycle of a row

Lifecycle diagram: PUBLISHED to PROCESSING to COMPLETED, with a branch from PROCESSING to FAILED when the listener throws.
A publication has two terminal states, and only one of them is the end of the story.

Note the asymmetry. A listener that throws is caught and marked failed on the way past. A JVM that dies mid-listener marks nothing at all — that row is stranded in PROCESSING, and only a staleness sweep will ever free it. If you care about crash recovery and not just SMTP hiccups, that sweep is not optional.

Nothing retries on its own

This surprises people: there is no max-attempts setting. The registry is deliberately a record, not a scheduler. It tracks attempts and exposes an API; deciding when and how often to resubmit is left to you.

@Scheduled(fixedDelay = 5, timeUnit = TimeUnit.MINUTES)
void retry() {
    failedPublications.resubmit(ResubmissionOptions.defaults()
        .withFilter(p -> p.getCompletionAttempts() < 3)
        .withMinAge(Duration.ofMinutes(1))
        .withBatchSize(50));
}

Three knobs, three different jobs. withFilter is your retry cap — expressed as which rows stay eligible rather than as a loop counter, which also means rows that exhaust it stop being selected and quietly become a dead-letter queue. withMinAge is backoff, and it doubles as protection against resending something that is merely still in flight. withBatchSize bounds the blast radius per pass.

The payoff is that recovery happens in the running process. The mail server dies at 14:02 and comes back at 14:07; the pass at 14:10 delivers. Nobody restarts anything, nobody logs into a box.

One flag to leave alone: the option to republish outstanding events on restart. It takes no resubmission options, so it ignores your attempt cap entirely — and if your deploys are frequent, that quietly undoes the limit you just set.

Four ways an outbox quietly does nothing

Four failure modes: no publishing transaction, a listener that catches, selecting on a status label, and half-configured staleness.
The uncomfortable part: none of these produce an error. They produce a green build.

The first is the sharpest. @Transactional is applied by a proxy, so on a private method — or any method called through this — it is ignored without a word. No transaction means @TransactionalEventListener never fires at all, and the outbox stays empty while everything appears to work.

The second is the one you write by habit. A catch (Exception) around the send means the listener always returns normally, so every publication is marked complete whether or not the message went out — a beautifully maintained log of successes that are not successes.

Testing it without fooling yourself

Two things make these tests lie. A test annotated @Transactional rolls back, so it never commits, so an after-commit listener never runs and every assertion passes vacuously — the test that proves an outbox has to commit for real and clean up after itself. And sequencing on a status column races the listener; wait for the send attempt, then assert the row survived it.

The only test that really matters is the round trip: break the mail server, trigger the event, confirm the row survives, restore the mail server, and confirm delivery on the next pass — without restarting anything. Everything before that step is plumbing.

What I would keep

A plain @EventListener is coupled, not unreliable; the risk arrives the moment you go asynchronous. The outbox row has to be written by your transaction — that single fact is the durability guarantee. A listener that throws is a feature, and every catch-all is a message you will never know you lost. And recording is not recovering: something of yours has to schedule the retry, and cap it.

Worth adding that none of this requires adopting a whole architectural framework. The event registry ships as a few standalone artifacts; you can take the outbox and leave the module model, the package restructuring and the architecture tests entirely alone.