Machine down alerts people actually act on

You set alerts so the right person hears about a stop by alerting only on stops that outlast each machine's normal pauses, at two levels. A warning time goes to whoever can restart the machine, and a longer critical time goes to one on-call technician, each on one channel. Then you count alerts per person per shift, and prune every rule people ignore.

This guide is for maintenance leads and plant managers setting up stop alerts on a cell or a whole floor. It works one machine's stop history into two thresholds, routes each level, and shows how to measure fatigue.

Which stops deserve an alert?#

A stop deserves an alert when it has outlasted the machine's normal pauses and a named person can do something about it now. Every other stop belongs in the downtime record, where you read it at the weekly review.

An alert comes from an event rule: a condition on one machine, paired with the channels that carry it, as the event rule glossary entry puts it. The rule is only as good as the condition. For example, a rule that fires on every stop over 2 minutes on a Haas VF-2 reports each tool change and first-article check, and the cell lead mutes it by Friday.

Short stops need a rule, not a person: the short-stop rule labels everything under the cutoff, so nobody is paged for a 90-second jam. Stops that need a part or a technician become maintenance work, as which stops become work orders sets out.

How do you set the warning and critical times?#

Set both times from the machine's own stop history: the warning just above its longest routine stop, and the critical time just past the point where stops the first responder can clear have ended. A plant-wide number fails because a lathe with 8-minute changeovers and a press with 25-minute die changes need different times.

Use four weeks of stops, so every product mix and shift pattern shows up at least once. Split the stops into routine stops, the pauses that belong to the job, and unplanned stops, everything else.

With no history yet, start the warning 5 minutes above the machine's usual changeover and the critical time 30 minutes after the warning. Replace both with numbers from the history after four weeks.

You don't need a third level. Process-safety guidance from the UK Health and Safety Executive recommends about three alarm priorities, each based on what happens if nobody responds.^[1] For a stopped machine, warning and critical cover both consequences, and the third level is the downtime report.

Who gets each level, and over which channel?#

Each level goes to one audience over one channel: the warning to the person who can restart the machine, the critical alert to the person who can repair it. Sending both levels to everyone teaches everyone to ignore both.

Each channel suits a different urgency:

• Phone push reaches one person on the floor within seconds, and it needs the app installed with notifications allowed.
• SMS reaches a phone with no app and interrupts whatever the person is doing. Keep it for the one level that can't wait.
• A chat channel, such as a Slack or Microsoft Teams channel, reaches the whole shift at once and shows who answered. Post to the channel the shift already reads, not a new alerts channel nobody opens.
• Email reaches anyone with an address, including supervisors and managers who don't carry the app, and leaves a written record of every stop. Send it alongside push or SMS, or on its own to the people who need to know a machine is down.

An alert that reaches the right person in 2 minutes takes the longest wait out of a breakdown, as cutting how long each stop lasts works through.

How do shifts and quiet hours change alerts?#

Everyone subscribed to a machine's alert receives it, so choose subscribers and channels that suit the shifts the machine runs.

• Send warnings where the floor looks. A chat channel the floor reads on every shift reaches whoever is working. Subscribe the people who cover the cell to push.
• Keep the critical list short. Subscribe the technician who is on call, and update the list when on-call changes hands.
• Go quiet outside scheduled production. A machine that is stopped because nobody is scheduled on it isn't down. Turn its rule off when it is idle by design, instead of asking people to mute their phones.
• Keep critical alerts on for lights-out runs. A machine running unattended overnight has nobody beside it, so the critical alert to the on-call technician is the only thing watching it.
• Hand over open stops at shift change. The outgoing lead names every machine still down, and the incoming lead owns those alerts from then on.

How do you measure alert fatigue?#

You measure alert fatigue with two numbers: alerts per person per shift, and the share of alerts someone acted on. The first tells you about the load. The second tells you whether the load is worth carrying.

The cost shows up in the alerts that matter. In the last 11 minutes before the 1994 explosion at the Texaco Milford Haven refinery, two operators had to recognize, acknowledge and act on 275 alarms.^[1] The HSE now sets a long-term average of no more than one alarm every ten minutes in normal operation, for operators whose job is answering alarms.^[1] Google's site reliability engineers write that frequent low-priority alerts cause serious alerts to be "treated with less attention than necessary".^[2]

Count an alert as acted on when someone went to the machine or made a call because of it. The stop reason is a fair proxy: an alert on a stop later classified as a changeover or a tool change was noise.

For example, a cell lead covering 12 machines like the Haas VF-2 above receives 12 × 3 = 36 alerts a shift with a 5-minute warning. With a 15-minute warning, the lead receives 12 × 17 ÷ 40 = 5.1 alerts a shift.

The critical alert in the example fires on 4 stops in 40 shifts, 0.1 a shift for the machine. Across 12 machines, the on-call technician gets 48 ÷ 40 = 1.2 critical alerts a shift, under the limit Google's teams set for paging events: fewer than 2 in a 12-hour on-call shift.^[2]

Prune alerts every month#

Pruning takes an hour a month, and it decides whether next month's alerts get read. Google's monitoring chapter puts it in one line: "Every page should be actionable."^[3]

1. Count last month's alerts from the stop record

   Every stop longer than the warning time sent a warning, and every stop longer than the critical time sent a critical alert. Take last month's stops for each machine from the downtime report, with their lengths and reasons, and keep the ones past the warning time.

2. Mark each alert acted on or noise

   Read it from the stop reason. An alert on a stop classified as routine work, such as a changeover or a tool change, is noise.

3. Compute both numbers per rule and per person

   A rule below 50% acted on, or a person above their target, goes on this month's list.

4. Raise the threshold on noisy rules

   Rerun the stop-history method on that machine with the latest four weeks. Changeovers often get longer when the product mix changes.

5. Move passive recipients off push and SMS

   Someone who reads alerts but never acts wants to know, not to respond. Take them off push and SMS, and send them the alert by email or the daily summary instead.

6. Delete rules nobody would miss

   Ask the question Google asks of every paging rule: "Will I ever be able to ignore this alert, knowing it's benign?"^[3] If the answer is yes, the rule goes.

The HSE's first quick fix is the same: remove or review alarms with no defined operator response.^[1]

Set Machine Down alerts in IoTFlows#

In IoTFlows, each machine carries its own Machine Down rule at a warning time and a critical time, in minutes, so the Haas VF-2 can warn at 15 and the press beside it at 30. Each level goes by email, push or SMS to its own subscriber list, or to a Slack, Microsoft Teams or Discord channel through a chat integration. Machine Up follows when the machine runs again, and Log keeps each event on the machine's Health tab. For the monthly prune, the Downtimes report downloads as a CSV with one row per stop, its length and its reason, so the stops past each machine's warning time are the alerts it sent. The same grid carries rules on health score, shock and temperature.

The Machine Down settings cover both thresholds. See every channel on integrations.

RNPSA halved its maintenance response time, as RNPSA’s story describes. How to cut maintenance response time covers that workflow.

Set up stop alerts on one cell this week#

1. Pull four weeks of stops for each machine in the cell, and split them into routine and unplanned.
2. Set each machine's warning just above its longest routine stop, rounded up to the next 5 minutes.
3. Set the critical time where stops the first responder clears have ended.
4. Send warnings to the cell lead by push or email, or to the chat channel the floor already reads.
5. Send critical alerts to the on-call technician by SMS, and update the list when on-call changes hands.
6. Turn rules off on machines that are idle by design.
7. Stop one machine on purpose and confirm each alert arrives, as the alert quickstart does.
8. After a month, count alerts per person per shift and the share acted on, and prune.