Incident Management

Blameless' incident management UI facilitates all aspect of an incident workflow to help you reduce time to resolution, maximize efficiency during mission critical times, and ensure that all relevant data is captured throughout the process.

Before starting with Incident Management, we recommend you first read through our quickstart guide and understand the various configurations available in Settings.

Getting Started

Navigate over to the Blameless UI, and from the Incident list page, find the incident you want to manage.

Incident Overview

The incident page is setup to provide you with the capabilities to manage all aspects of your incident:

  • The top section provides an overview of the incident metadata
  • The side panel focuses on incident information such as summary, type, timings, tags, and team for quicker access and readability
  • The main panel focuses on core incident usability around events, tasks, follow-up actions and communications
Incident Details page - Overview

Incident Header

Within the incident header you can view and adjust crucial incident properties and metadata. The incident header shows the following incident metadata and properties:

  • Incident severity
  • Incident status
  • Tasks (pending and completed)
  • Follow-up actions (pending and completed)
  • Incident commander
  • Incident war room and video bridge
  • Top level JIRA ticket for the incident (if you have JIRA integration enabled)
  • A link to the incident postmortem
  • An actions menu in the top right for additional actions regarding the incident itself. For example, see deleting an incident
Incident Details page - Header

Changing Severity

Incident severity can be changed at any time via the severity dropdown in the incident header:

Incident Details page - Change Severity

You can also change the severity at any time using the Blameless Bot commands.

Changing the incident severity will also create a new event in the incident timeline to capture the time of severity change.

note

Changing the severity of an incident will not cause it to inherit the settings of that new severity and type. The settings for an incident are determined by the severity and type when the incident is created.

Changing Status

Blameless incidents currently have a status flow of:

Investigating -> Identified -> Monitoring -> Resolved

There is also an additional Duplicated status that can be used to flag an incident as a duplicate. Alternatively, in the event of a duplicate incident, you may want to simply delete it depending on the context.

Incident status can be changed at any time via the status dropdown in the incident header, and users can jump ahead to any phase as desired, unless there are blocking required tasks for that phase.

Depending on the phase the incident is being transitioned into, new tasks will become available in the Tasks tab and via Blameless Bot as per your configuration in Settings.

Incident Details page - Change Status

You can also change the status at any time using the Blameless Bot commands.

Changing the incident status will also create a new event in the incident timeline to capture the time of status change.

Deleting an incident

An incident can be deleted by clicking on the actions menu in the top right of the incident header and selecting the Delete Incident option:

Incident Details page - Delete Incident

You can also an incident using the Blameless Bot commands.

Incident Sidebar

Within the incident sidebar, you can view incident information such as summary, type, timings, tags, and team.

Incident Details page - Sidebar

Changing Incident Type

Incident type can be changed at any time via the type dropdown in the incident sidebar:

Incident Details page - Change Type

You can also change the type at any time using the Blameless Bot commands.

Changing the incident type will also create a new event in the incident timeline to capture the time of type change.

note

Changing the type of an incident will not cause it to inherit the settings of that new type and severity. The settings for an incident are determined by the type and type when the incident is created.

Working With Tags

Tags provide a powerful mechanism for you to capture and label metadata for an incident which can later be viewed in Reliability Insights reports and queries.

Tag categories can be configured per incident type and severity from Settings.

Incident Details page - Change Tags

Tags support an optional list of predefined tag values. This allows you to lock down a list of tag options to a predefined list in order to improve data curation, consistency, and reporting.

Adding a tag value to a category's list of predefined tags restricts selectable options for this category. If a category has no predefined values, the input values will remain freeform.

For tag categories which contain predefined values, their tag input will now be a multi-select dropdown. For tag categories which aren't locked down to predefined values, their input remains as freeform input.

Incident Details page - Incident Tags

You can also change tags at any time using the Blameless Bot commands.

Incident Details page - Incident Tags BotIncident Details page - Incident Tags Bot

Team

Members of the incident team are shown in the team section. Members that have been assigned to roles will be pinned at the top of the list, followed by the remaining particpant members.

Clicking on the edit icon allows you to reassign roles, and remove team members.

Incident Details page - TeamIncident Details page - Edit Team

You can also assign roles at any time using the Blameless Bot commands.

Event Timeline

As an incident progresses through its lifecycle, Blameless will automatically capture events and add them to the incident timeline. You can easily see the progression of an incident and key events along the way.

Incident Details page - Event timeline

Manually Capturing Events

Events may also be manually captured by highlighting important messages in the war room chat.

Adding Custom Events

Custom events may be added by clicking on the Add Event button in the top right of the Events tab.

Incident Details page - Add event

From here, users can input an event payload in text form, or as a file, and specify the time of the event.

Editing, Copying and Deleting Events

Certain events in the timeline can be edited, copied or deleted. Hover over these events in the incident timeline, and a menu with more actions will appear where you can perform additional actions.

Incident Details page - Event actions
OperationEvent types permittedDescription
CopyChat highlights
File uploads
Image uploads
Link
Custom events
Copies the event as text, or asset url depending on the event type
DeleteChat highlights
File uploads
Image uploads
Link
Custom events
Deletes the event from the timeline
EditCustom EventsAllows the user to edit the event

Event Types

Event TypeCapture TypeDescriptionExample
Start of Customer ImpactAutomaticEvent marking the start of customer impact. This event is defaulted to the incident creation timestamp, however can be edited as needed.
Incident CreatedAutomaticEvent marking the time of incident creation, captured automatically when you start an incident.
Chat HighlightManualCaptured when chat text is manually highlighted during an incident.
Role AssignmentAutomaticCaptured when a role is assigned.
Postmortem State ChangeAutomaticCaptured automatically first when an incident is started, and upon every subsequent Postmortem state change.
Image Highlight/UploadManual & AutomaticCaptured when an image is manually highlighted during an incident, or automatically when an image is uploaded.
FIle UploadAutomaticCaptured automatically when a file is uploaded.
Swimlane EventAutomaticCaptured automatically when a swimlane is opened, modified, or closed.
End of Customer ImpactAutomaticEvent marking the end of customer impact. This event is defaulted to the incident resolution timestamp, however can be edited as needed.
Status ChangeAutomaticCaptured automatically when incident status is changed.
Severity ChangeAutomaticCaptured automatically when incident severity is changed.
Incident Type ChangeAutomaticCaptured automatically when incident type is changed.
Custom EventAutomatic capture via manually added eventCaptured when a custom event is added.
Pager EventAutomaticCaptured automatically when triggering a pager during an incident.
Link EventAutomatic capture via manually added eventCaptured when a custom event contains a link.

Tasks

Tasks provide a mechanism by which you can ensure that certain items are completed or suggested during an incident lifecycle. These tasks can be assigned to each phase of the incident, as well as to a specific role or owner.

As the incident progress through each subsequent phase, the tasks for that phase will become available.

Incident Details page - Tasks Overview

Configuring Tasks

Tasks can be configured per incident type and severity.

To configure tasks, from the main navigation go to Settings then:

  • Expand Incident Workflows from the menu to find Incident Types
  • Click into an Incident Type to manage its settings at each severity level
  • Scroll down to the Task settings
Incident Details page - Task Settings

Here tasks can be customized per incident phase, be assigned to an incident role, set as reccurring, or add an informational link. Tasks can also be reordered within a given phase by dragging with the drag handle in the left margin.

Required Tasks

Users can require specified tasks to be completed during an incident workflow, to help ensure teams don’t overlook a specific task(s) before transitioning to another phase of the incident management process.

  • From Task settings, for to the tasks you'd like to require, click on the asterisk to set it as required
note

When a task is marked as required, the asterisk will be highlighted in red

Incident Details page - Required Tasks SettingsIncident Details page - Required Tasks Bot

For incidents where the state is either Investigating, Identified, or Monitoring, users will be unable to transition the incident to a subsequent phase where required tasks have not yet been marked as complete.

Incident Details page - Required Task Transition ErrorIncident Details page - Required Task Transition Error Bot

For required tasks that aren't complete while an incident is in the Resolved state, users will be unable to publish the corresponding Postmortem, leaving the channel unarchived.

note

Incidents transitioned to Duplicate status do not enforce required tasks

Interacting with Tasks

From the Tasks panel, users can interact with Tasks:

  • Complete a task by clicking on the completion circle in the left margin
  • Edit a task by hovering over a row and clicking on the edit icon
  • Duplicate a task by hovering over a row and clicking on the duplicate icon
  • Delete a task by hovering over a row and clicking on the delete icon
  • Filter tasks using the toolbar filters

Bulk interactions are also available by clicking on the Bulk Select button in the Tasks toolbar. From this mode, you may bulk delete or bulk toggle task completions.

You can also interact with tasks at any time using the Blameless Bot commands.

Adding New Tasks

From the Tasks panel, new tasks can be added during an incident by clicking the Add Task button:

Incident Details page - Add Task

Follow-up Actions

Follow-up actions can be added at any point during an incident. They can be added using:

  • the #todo or /blameless followup commands (see Blameless Bot commands)
  • via the Follow-up Actions tab on the Incident page, by clicking the Add Action button or the more actions menu
Incident Details page - Follow-up ActionsIncident Details page - Follow-up Actions

When adding a new action, you will be prompted to fill out standard fields such as:

  • Summary
  • Assignee
  • Type
note

If the JIRA integration is enabled, additional fields will also be shown, such as Project and Issue type.

Incident Details page - Add Action JIRA

Communications

The Communications workflow allows you to easily keep peers, stakeholders and public audiences in the loop as an incident progresses. Draft, preview and approve multiple communications easily via a streamlined workflow and publish across a variety of mediums including Slack, email, SMS, and Statuspage.

Message content can be curated as needed or crafted as a subset of the current incident data available.

Internal Communications Workflow

The internal communications workflow allow you to update audiences via Slack, email and SMS via a standard draft and publish workflow.

Incident Details page - Internal Comms Lead

External Communications Workflow

The external communications workflow allow you to update audiences via Slack, email, SMS and other external channels such as Statuspage via a draft, approval, and publish workflow.

Incident Details page - External Comms Lead