How to Manage Blameless SLOs

Getting Started

Once you have the SLO components together, you need to look at managing your SLO environment using your identified Best Practices. To help support that effort, this guide describes the following actions:

  • Creating
  • Adding
  • Editing
  • Deleting
note

It is important to check your associations within the SLO components as you will not be able to delete or edit some of these components if they are inter-associated.

As a new User to SLOs

To help you, as a new user, Blameless provides you with the SLO Wizard to help guide you through the process. You start with the User Journey.

Start by launching the SLO Manager. Blameless opens to the User Journey Landing page. Next, click on “+New Journey”. The SLO Wizard will walk you through the process, and you can follow that process via the guide icon at the top of the page or by clicking the on “Next” button.

SLO Feature Nav Bar
note

You can create a User Journey and leave it blank as a placeholder for future population.

You can continue to the section “Working with the SLO Wizard” for a high level description of the feature.

For detailed instructions regarding the New User Journey and the SLO Wizard, refer to the Building a New SLO

As an Experienced User of SLOs

As an experienced user, you are probably familiar enough with the process to not need the SLO Wizard to create more SLIs, but it is certainly there for you to use to create new user journeys and add new SLOs to user journeys. You can continue on via the section, “Launching the SLO Manager”.

Working via the SLO Wizard

An SLO requires the following:

  • Create the User Journey
  • Create the SLI
  • Create the Error Budget Policy
  • Create the SLO
  • Set the Thresholds
note

The best practice for User Journey analysis is collaboration across teams and groups to collect the journey information.

Managing SLOs

Best Practice states you follow a consistent format for your naming conventions, with the goal of being able to look at the name of the SLO component and understand their goal. The challenge will be a generic enough name while allowing you to associate the SLO across multiple queries.

Adding SLOs to a User Journey

Adding an SLO to an existing User Journey occurs via the SLO Wizard, regardless of whether you are a new user or a seasoned SLO user. The SLO Wizard walks you through the steps, showing your progress at the top of the window.

Adding an SLO to an existing User Journey, requires the following:

  1. Select a User Journey
  2. Click the “Add SLO” button to start the SLO Wizard
  3. Select an existing SLI or create a new SLI
  4. Configure the SLO (reliability target, relative to the selected SLI type)
  5. Select an Error Budget Policy (optional)

Once these items are complete, you need to save the SLO.

note

The best practice for User Journey analysis is collaboration across teams and groups to collect the journey information (e.g. engineering, product, site reliability engineers, customer success, etc.).

note

For this example, we are using a Latency SLI to create the following SLO. An SLI must be selected or created first to create an SLO against that SLI.

  1. Click on the User Journeys option in the upper left corner of the window.
  2. Click on the desired User Journey. A new window opens. The User Journey window will contain a number of options:
  • A status for the User Journey
  • A list of any existing SLOs associated with the User Journey
  • A summary of the User Journey
  • Options to add a new SLO or edit an existing SLO if desired.
  • Click on the Existing SLOCard to open it
  • Click on the “+ ADD SLO” button to create a new one
Create a New SLO window
  1. Click on the “+ Add SLO” button. A new "SLI" window opens.
note

The SLI window appears because you must have at least one SLI to create an SLO.

Create a New SLO--determine SLI
  1. Select the SLI option you desire:

    a. Select an existing SLI from the list provided by the Blameless application from the drop-down. b. Enter a name for the new SLI.

note

Best practices suggest something that reflects the User Journey it is associated with.

For example: Login Latency for 95% percentile.

note

You can also search for a specific SLI from the available list using the provided search window under the drop-down.

  1. Associate the SLO with an Error Budget Policy beneath the SLO Name field. When you have selected a valid existing SLI or complete a valid new SLI, the SLI window opens and the “Next” button at the bottom of the screen becomes active.
Create a New SLO--SLI assigned
  1. Click the “Next” button. The SLO Details window opens.
Create a New SLO-assign Error Budget
  1. Enter the SLO Name (a required field).
  2. Specify the time percentage you need to meet the SLI (required field).
  3. Specify the Latency (second or milliseconds).
note

Depending on the percentage you enter, the “Total Error Budget” values in Days, Hours Minutes, and Seconds will change accordingly .

  1. Specify the Threshold Violation Operator (greater than, less than, etc.)
  2. Select the SLO Status type (required field).
note

“Active” and “Testing” options will affect your Error Budget (decrease your budget) while “Development” will not.

  1. Select an Error Budget to associate to the SLO.
  2. Click the “Save” button.
  3. Click the “Finish” Button to Complete. Blameless returns you to the User Journey Step. The new SLO should now appear in the User Journey window as an option.

When the SLO kicks off, it will (currently) connect to the selected Data source and start digesting data for the previous 28 day time window to measure against the SLO(s) that are activated.

note

Be aware that this may take some time to "crunch the numbers" once it has started.

The SLI status is reported depending on the status. For example:

SLI StatusIcon typeTool tip
In ProgressSpinning wheelThis SLI is currently fetching the latest data from your APM.
Backfill completedGreen circle checkmarkSuccessfully fetched latest data from your APM.
ErrorRed circle exclamation point“Error while fetching…”.
No incoming dataTBDFuture Feature

Examples regarding these status icons appears in the following:

Existing SLO components
note

The Error message will be similar to the sample image, based on the type of error and explanation available.

When it is done, Blameless will generate a chart of the data it has digested, based on the parameters set and the Error Budget policy values, and display it below the SLO list on the Step.

SLO Results window with charts

You will note within the Details window you have several icons identifying actions you can apply to the elements in the window. These are identified and the action defined in the following table.

IconTypeAction
“...”Drop-downEdit SLO
Recalculate Error Budget
Delete SLO
“+”ActionAdd SLO / SLI
Pencil iconActionEdit the associated field
“X”ActionClose Details Window

Editing SLOs

  1. Click on the User Journey Option in the upper left corner of the window.
  2. Click on the desired User Journey. A new window opens. The User Journey window will contain a number of options:
  • A status for the User Journey
  • A list of any existing SLOs associated with the User Journey
  • A summary of the User Journey
  • Options to add a new SLO or edit an existing SLO if desired
  1. Click on the Existing SLO Card to open it. The SLO Details window opens.
Create a New SLO-assign Error Budget
  1. Edit the fields you wish to change.
  2. Click on the “Save” Button.
  3. Click on the “Finish” Button.
note

Depending on the percentage you enter, the “Total Error Budget” values in Days, Hours Minutes, and Seconds will change accordingly .

note

“Active” and “Testing” options will affect your Error Budget (decrease your budget) while “Development” will not.

When the SLO kicks off, it will (currently) connect to the selected Data source and start digesting data for the previous 28 day time window to measure against the SLO(s) that are activated.

note

Be aware that this may take some time to "crunch the numbers" once it has started.

When it is done, Blameless will generate a chart of the data it has digested, based on the parameters set and the Error Budget policy values, and display it below the SLO list on the landing page.

SLO Results window with charts

Deleting SLOs

note

You can delete an SLO even if it is associated with an SLI or Error Budget Policy.

  1. If you have an SLO to delete: Click on the three dots icon in the SLO Title header.
  2. Select the Delete (trash can) icon.
  3. Blameless will confirm (and warn you) the planned deletion.
SLO Deletion warning
  1. Click the “Delete” button to confirm.

For More Information

For instructions regarding the creation, configuration, and use of User Journeys, Error Budgets, SLOs, and SLIs, refer to the following SLO references:

Getting started with Blameless SLOs

Building a New SLO

Creating Error Budget Policies

Managing your SLOs and detailed instructions:

Understanding your SLOs

Refer to the Google SRE Handbook for more information regarding Site Reliability Engineering.