Holistic Observability

The Wikipedia definition of observability “…a measure of how well internal states of a system can be inferred by knowledge of its external outputs” means being able to determine what a system was doing only by looking at any outputs available from the system. We should be able to answer the question “Why did these users have problems at this particular time” without getting the users to retry against a test system with a debugger attached.While observability is essential to any system, modern cloud-native distributed architectures dramatically increase the complexity of the system in exchange for flexibility. Compared to a more traditional monolith running on a single server, modern architectures have far too many moving parts for any one person to be able to hold in their head. They also make use of a wide variety of tooling and techniques such as Infrastructure as Code, Machine Learning and a polyglot of programming languages.It is critical to employ observability so we can comprehend both what the system is doing now, such as when there’s an outage in production, and to understand what the system was doing in the past, such as when trying to understand reports of problems from users after the fact.Within computing, there are three main pillars of observability: logging, metrics and traces.

Logging

The pillar that is familiar to most people is logging. Logs are text-based, timestamp records of events happening over time – ideally in some sort of structure to allow information within the event log to more easily be parsed and extracted. What data gets logged will typically vary by level – with user-driven actions getting logged at information level and access, authorisation or system failures getting logged at error level.Logging is simple to get started with, but you can quickly run into problems with both the retention of the logs and the discovery of information from those logs. This is all the more difficult when different parts of the system run on multiple machines and means the logs have to be pieced together to create a single coherent narrative.

{"@timestamp":"2020-10-09T07:39:28.609Z","@version":"1","message":"Initializing Spring DispatcherServlet 'dispatcherServlet'","logger_name":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]","thread_name":"http-nio-8080-exec-1","level":"INFO","level_value":20000}{"@timestamp":"2020-10-09T07:39:28.627Z","@version":"1","message":"Initializing Servlet 'dispatcherServlet'","logger_name":"org.springframework.web.servlet.DispatcherServlet","thread_name":"http-nio-8080-exec-1","level":"INFO","level_value":20000}

Example of structured logging (JSON Format)

Metrics

A pillar that is familiar with some people is that of metrics. Metrics are numerical representations of the state of the system which should be recorded and stored, so they can be analysed over time – such as the rate of change of a metric over the last five minutes or predicting a value based on what happened last year. The metrics can also be used to trigger alerts – providing helpful warnings of impending failure, rather than waiting for the failure to occur. The metrics a system exposes will be values that it is able to measure at the time. Calculations such as moving averages or 95th percentiles should be performed by the system displaying the metrics as it allows for the greatest flexibility of use with the metrics.Precisely what the system should expose as metrics will depend on what kind of system it is. There are broadly two ideas about what to expose:

• USE – typically aimed towards infrastructure as it refers to metrics that don’t make sense with most services
  • Utilisation – what proportion of the resource is busy performing work
  • Saturation – how much extra work the resource needs to perform but can’t be serviced
  • Errors – Number of error events
• RED – aimed more towards services, a subset of the Four Golden Signals from Google’s Site Reliability Engineering book
  • Rate – the number of requests handled by the service per second
  • Error – the number of failed requests handled per second
  • Duration – the amount of time required to process each request

To reduce the cognitive workload around the management of the monitoring and alerting around metrics, it is best to have standardisation around the naming of metrics – making it easier to apply monitoring across many services easily.This doesn’t mean that other metrics – such as garbage collection statistics – are useless, but that they will typically be only of importance after the fact rather than monitoring the current state of a service.

Example of CPU usage metrics using Prometheus