Published on

Monitoring cache stats using OpenTelemetry Metrics

This article explains how to use opentelemetry-go Metrics API to collect metrics, for example, go-redis/cache stats.


What is OpenTelemetry

OpenTelemetry is a vendor-neutral API for distributed traces and metrics. It specifies how to collect and send telemetry data to backend platforms. It means that you can instrument your application once and then add or change vendors (tracing backends) as required. OpenTelemetry is available for most programming languages and provides tracing interoperability across different languages and environments.

Getting started with OpenTelemetry Metrics

To get started with metrics, you need a MeterProvider which provides access to Meters:

import "go.opentelemetry.io/otel/metric/global"

// Meter can be a global/package variable.
var Meter = metric.Must(global.Meter("app_or_package_name"))

Using the meter, you can create instruments and use them to measure operations. The simplest Counter instrument looks like this:

import "go.opentelemetry.io/otel/metric"

counter := Meter.NewInt64Counter("app_or_package_name.component1.counter1",
	metric.WithDescription("Optional metric description"),
    metric.WithUnit(""), // optional metric unit
)

counter.Add(ctx, 1)
counter.Add(ctx, 10)

You can find more examples at GitHub.

Cache stats

Our Redis-based cache keeps stats about hits and misses in the following struct:

type Stats struct {
	Hits   uint64
	Misses uint64
}

You can get the current stats with:

stats := cache.Stats()
fmt.Println("hits", stats.Hits)
fmt.Println("misses", stats.Misses)

Monitoring cache stats

To start monitoring our cache, we can create a separate instrument for each struct field. Here we are using CounterObserver instrument which periodically calls a function to gather stats.

import "go.opentelemetry.io/otel/metric"

func MonitorCache(cache *cache.Cache, meter metric.MeterMust) {
	var hitsCounter, missesCounter metric.Int64SumObserver

	batchObserver := xotel.Meter.NewBatchObserver(
        // This func is periodically called by OpenTelemetry to collect data.
		func(ctx context.Context, result metric.BatchObserverResult) {
			stats := cache.Stats()

			result.Observe(nil,
				hitsCounter.Observation(int64(stats.Hits)),
				missesCounter.Observation(int64(stats.Misses)),
			)
		})

	hitsCounter = batchObserver.NewInt64SumObserver("cache.hits")
	missesCounter = batchObserver.NewInt64SumObserver("cache.misses")
}

Using the instruments above we get access to the following metrics:

  • cache.hits - number of cache hits.
  • cache.misses - number of cache hits.
  • cache.hits + cache.misses - number of cache requests.
  • cache.hits / (cache.hits + cache.misses) - cache hit rate.

Metric attributes

The code above works well enough, but what if we want to add another metric:

type Stats struct {
    Hits   uint64
    Misses uint64
+    Errors uint64
}

We could add another instrument to observe Errors field, but then we also need to update our math:

  • cache.hits + cache.misses + cache.errors - number of cache requests.
  • cache.hits / (cache.hits + cache.misses + cache.errors) - cache hit rate.

Can we do better? Yes, using a single instrument and metric attributes:

import (
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric"
)

func MonitorCache(cache *cache.Cache, meter metric.MeterMust) {
	var counter metric.Int64SumObserver

    // Arbitrary key/value labels.
	hits := []attribute.KeyValue{attribute.String("type", "hits")}
	misses := []attribute.KeyValue{attribute.String("type", "misses")}
	errors := []attribute.KeyValue{attribute.String("type", "errors")}

	batchObserver := xotel.Meter.NewBatchObserver(
		func(ctx context.Context, result metric.BatchObserverResult) {
			stats := cache.Stats()

			result.Observe(hits, counter.Observation(int64(stats.Hits)))
			result.Observe(misses, counter.Observation(int64(stats.Misses)))
			result.Observe(errors, counter.Observation(int64(stats.Errors)))
		})

	counter = batchObserver.NewInt64SumObserver("cache.stats")
}

Our new math looks like this and does not require changes when you add new stats:

  • cache.stats - number of cache requests.
  • filter(cache.stats, type = "hits") - number of cache hits.
  • filter(cache.stats, type = "misses") - number of cache misses.
  • filter(cache.stats, type = "hits") / cache.stats - cache hit rate.

As a bonus, you can easily visualize all available metrics using grouping by type attribute:

cache.stats | group by type

Cache metrics

What's next

Next, you can learn about the available metric instruments and try to instrument your code.