The circuit breaker is one of the most well-known resilience patterns in distributed systems. It wraps calls to a dependency and, after a configurable failure threshold is reached, it “opens” — failing fast instead of letting requests pile up waiting for a service that’s already down.
Most engineers understand the concept. Far fewer have instrumented it properly. And that gap shows up at the worst possible time: during an incident, when you need to know whether your circuit breaker is the cause of degradation or your last line of defense against it.
How the Circuit Breaker Works
The pattern has three states:
- CLOSED: Normal operation. Requests flow through. Failures are counted.
- OPEN: Failure threshold exceeded. Requests fail immediately without calling the dependency.
- HALF-OPEN: A probe state. A limited number of requests are allowed through to test if the dependency recovered.
The state machine drives the behavior. What most teams forget to instrument is the transitions — not just the current state.
A Minimal Implementation in Kotlin
import java.time.Instant
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.atomic.AtomicReference
enum class CircuitState { CLOSED, OPEN, HALF_OPEN }
data class CircuitBreakerConfig(
val failureThreshold: Int = 5,
val successThreshold: Int = 2,
val openDurationMs: Long = 10_000,
)
class CircuitBreaker(
val name: String,
private val config: CircuitBreakerConfig = CircuitBreakerConfig(),
) {
private val state = AtomicReference(CircuitState.CLOSED)
private val failureCount = AtomicInteger(0)
private val successCount = AtomicInteger(0)
private var openedAt: Instant? = null
fun <T> execute(block: () -> T): T {
return when (currentState()) {
CircuitState.OPEN -> throw CircuitOpenException("Circuit '$name' is OPEN")
else -> {
try {
val result = block()
onSuccess()
result
} catch (e: Exception) {
onFailure()
throw e
}
}
}
}
private fun currentState(): CircuitState {
val s = state.get()
if (s == CircuitState.OPEN) {
val elapsed = Instant.now().toEpochMilli() - (openedAt?.toEpochMilli() ?: 0)
if (elapsed >= config.openDurationMs) {
transitionTo(CircuitState.HALF_OPEN)
return CircuitState.HALF_OPEN
}
}
return s
}
private fun onSuccess() {
if (state.get() == CircuitState.HALF_OPEN) {
if (successCount.incrementAndGet() >= config.successThreshold) {
successCount.set(0); failureCount.set(0)
transitionTo(CircuitState.CLOSED)
}
} else {
failureCount.set(0)
}
}
private fun onFailure() {
if (state.get() == CircuitState.HALF_OPEN) {
transitionTo(CircuitState.OPEN)
} else if (failureCount.incrementAndGet() >= config.failureThreshold) {
transitionTo(CircuitState.OPEN)
}
}
private fun transitionTo(next: CircuitState) {
val previous = state.getAndSet(next)
if (previous != next && next == CircuitState.OPEN) openedAt = Instant.now()
}
}
class CircuitOpenException(message: String) : RuntimeException(message)This works, but it’s completely dark. When the circuit opens in production you’ll see errors — but not why it opened, how long it’s been open, or how many requests it short-circuited.
Adding Metrics
Metrics are the foundation. You need to know at any point:
- What state each circuit is in
- How many state transitions have occurred
- How many calls succeeded, failed, or were rejected (fast-failed)
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.common.Attributes
import io.opentelemetry.api.metrics.LongCounter
import io.opentelemetry.api.metrics.ObservableLongGauge
class ObservableCircuitBreaker(
val name: String,
private val config: CircuitBreakerConfig = CircuitBreakerConfig(),
) {
private val state = AtomicReference(CircuitState.CLOSED)
private val failureCount = AtomicInteger(0)
private val successCount = AtomicInteger(0)
private var openedAt: Instant? = null
private val meter = GlobalOpenTelemetry.getMeter("circuit-breaker")
private val callCounter: LongCounter = meter
.counterBuilder("circuit_breaker.calls.total")
.setDescription("Total calls through the circuit breaker, by outcome")
.build()
private val transitionCounter: LongCounter = meter
.counterBuilder("circuit_breaker.transitions.total")
.setDescription("Number of state transitions")
.build()
// Gauge: 0=CLOSED, 1=OPEN, 2=HALF_OPEN
private val stateGauge: ObservableLongGauge = meter
.gaugeBuilder("circuit_breaker.state")
.setDescription("Current state of the circuit breaker")
.ofLongs()
.buildWithCallback { measurement ->
val v = when (state.get()) {
CircuitState.CLOSED -> 0L
CircuitState.OPEN -> 1L
CircuitState.HALF_OPEN -> 2L
}
measurement.record(v, Attributes.builder()
.put("circuit_breaker.name", name).build())
}
fun <T> execute(block: () -> T): T {
val current = currentState()
return if (current == CircuitState.OPEN) {
callCounter.add(1, attrsWithOutcome("rejected"))
throw CircuitOpenException("Circuit '$name' is OPEN")
} else {
try {
val result = block()
callCounter.add(1, attrsWithOutcome("success"))
onSuccess()
result
} catch (e: CircuitOpenException) { throw e }
catch (e: Exception) {
callCounter.add(1, attrsWithOutcome("failure"))
onFailure()
throw e
}
}
}
private fun transitionTo(next: CircuitState) {
val previous = state.getAndSet(next)
if (previous != next) {
if (next == CircuitState.OPEN) openedAt = Instant.now()
transitionCounter.add(1, Attributes.builder()
.put("circuit_breaker.name", name)
.put("circuit_breaker.previous_state", previous.name)
.put("circuit_breaker.new_state", next.name)
.build())
}
}
private fun attrsWithOutcome(outcome: String) = Attributes.builder()
.put("circuit_breaker.name", name)
.put("outcome", outcome)
.build()
// currentState / onSuccess / onFailure same as before
}With these three metrics you can build a dashboard showing: state per circuit, rejection rate, and transition rate. The most important alert: fire if circuit_breaker.state == 1 (OPEN) for longer than openDurationMs plus a buffer — it means the dependency never recovered.
Adding Tracing
Metrics tell you what happened. Traces tell you which requests were affected.
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.SpanKind
import io.opentelemetry.api.trace.StatusCode
private val tracer = GlobalOpenTelemetry.getTracer("circuit-breaker")
fun <T> execute(block: () -> T): T {
val currentState = currentState()
val span = tracer.spanBuilder("circuit_breaker.execute")
.setSpanKind(SpanKind.INTERNAL)
.setAttribute("circuit_breaker.name", name)
.setAttribute("circuit_breaker.state", currentState.name)
.startSpan()
return span.makeCurrent().use {
try {
if (currentState == CircuitState.OPEN) {
callCounter.add(1, attrsWithOutcome("rejected"))
span.setAttribute("circuit_breaker.rejected", true)
span.setStatus(StatusCode.ERROR, "Circuit is OPEN")
throw CircuitOpenException("Circuit '$name' is OPEN")
}
val result = block()
callCounter.add(1, attrsWithOutcome("success"))
span.setStatus(StatusCode.OK)
onSuccess()
result
} catch (e: CircuitOpenException) { throw e }
catch (e: Exception) {
callCounter.add(1, attrsWithOutcome("failure"))
span.recordException(e)
span.setStatus(StatusCode.ERROR)
onFailure()
throw e
} finally {
span.end()
}
}
}The circuit_breaker.state attribute on each span is especially powerful. In Jaeger or Tempo, filter for circuit_breaker.state = "OPEN" to see the exact blast radius of an outage: which services were calling the circuit, at what rate, and for how long.
Logging State Transitions
Logs give you the narrative — the precise moment the circuit tripped, with business context.
import org.slf4j.LoggerFactory
private val log = LoggerFactory.getLogger(ObservableCircuitBreaker::class.java)
private fun transitionTo(next: CircuitState) {
val previous = state.getAndSet(next)
if (previous != next) {
if (next == CircuitState.OPEN) openedAt = Instant.now()
when (next) {
CircuitState.OPEN -> log.warn(
"Circuit breaker opened. name={} failures={} threshold={}",
name, failureCount.get(), config.failureThreshold
)
CircuitState.HALF_OPEN -> log.info(
"Circuit breaker probing recovery. name={} open_duration_ms={}",
name, Instant.now().toEpochMilli() - (openedAt?.toEpochMilli() ?: 0)
)
CircuitState.CLOSED -> log.info(
"Circuit breaker closed — dependency recovered. name={}",
name
)
}
// emit metric...
}
}Use structured logging rather than string interpolation so you can query your log aggregator for all events from a specific circuit over time without parsing strings.
Using Resilience4j and Augmenting Its Observability
Most teams use Resilience4j rather than a homegrown circuit breaker. Its built-in Micrometer integration covers the basics, but state transition logging is sparse. Add your own listener:
import io.github.resilience4j.circuitbreaker.CircuitBreaker
import io.github.resilience4j.circuitbreaker.CircuitBreaker.State
fun CircuitBreaker.registerOtelObservability() {
val meter = GlobalOpenTelemetry.getMeter("resilience4j.circuit-breaker")
val log = LoggerFactory.getLogger("circuit-breaker.${this.name}")
meter.gaugeBuilder("resilience4j_circuit_breaker_state")
.ofLongs()
.buildWithCallback { m ->
val v = when (this.state) {
State.CLOSED -> 0L; State.OPEN -> 1L
State.HALF_OPEN -> 2L; else -> -1L
}
m.record(v, Attributes.builder().put("name", this.name).build())
}
this.eventPublisher.onStateTransition { event ->
log.warn(
"Circuit breaker transition. name={} from={} to={} failure_rate={} buffered={}",
this.name,
event.stateTransition.fromState,
event.stateTransition.toState,
this.metrics.failureRate,
this.metrics.numberOfBufferedCalls,
)
}
}Call circuitBreaker.registerOtelObservability() once after creating each breaker.
What Good Dashboards Look Like
With this instrumentation in place, a useful dashboard has:
Row 1 — Current health
- State per circuit (stat panel, color-coded: green/red/yellow)
- Rejection rate over the last 5 minutes
- Time since last transition
Row 2 — Trends
circuit_breaker.calls.totalsplit byoutcome(success, failure, rejected) over timecircuit_breaker.transitions.totalsplit by transition type — a circuit flapping between OPEN and HALF-OPEN every 30 seconds signals the dependency isn’t truly recovering
Row 3 — Correlation
- Layer circuit state transitions on top of your service’s error rate and p99 latency graphs. This immediately answers: “did our error rate spike because the circuit opened, or did the circuit open because of the spike?”
The Questions You Should Be Able to Answer
After adding this observability, you should be able to answer all of the following from dashboards and traces alone:
- Which circuit breakers are currently OPEN?
- How long has a given circuit been open?
- What failure rate triggered the transition?
- How many requests were fast-failed while the circuit was open?
- Did the circuit recover, or is it still flapping?
- Which upstream services were calling a circuit that opened?
If you can’t answer any of these today, start with metrics — they’re the fastest win. Then add structured log events for transitions, and finally span attributes so you can correlate circuit breaker state with individual request traces.
The circuit breaker protects your system. Observability protects your ability to understand it.