BEMODEM - Cmdr - Mobile Checkpoint Reboot – Incident details

All systems operational

Mobile Checkpoint Reboot

Resolved
Partial outage
Started over 3 years agoLasted 1 minute

Affected

Synthetics Platform

Partial outage from 9:01 PM to 9:02 PM

Primary Synthetics

Partial outage from 9:01 PM to 9:02 PM

Updates
  • Resolved
    Resolved

    The reboot at the root of this <1min outage was to finalize a pending security update and being a critical patch, the pre-existing rules to force a graceful shutdown when rebooting were not honored. This interrupted a few of our test scripts mid-execution causing timeouts which were reported as potential customer outages.

    Once the above failure was reported, our 2nd and 3rd layer synthetics rechecked (as designed) and confirmed there was no issue so any alerts that did go out would have been resolved in one minute or less.

    We're adding logic to prevent a false alert like this in the future but, can say for sure that the issue has been resolved.

    We apologize for any beats your heart may have skipped as a result of this false alert.

  • Monitoring
    Monitoring

    One of the checkpoints in use for mobile testing in our Primary Synthetics group was abruptly rebooted, interrupting a small subset of in-flight synthetic tests and triggering warnings for customers with more aggressive alerting rules in place. Usually, these false alerts are avoided through the use of maintenance windows, synthetic replications, and our 2nd and 3rd layer synthetic groups.

    This reboot took less than 40 seconds and everything is again working as expected but, we're investigating further before considering the matter resolved.