BEMODEM - Cmdr - Outage at AWS California and Oregon – Incident details

All systems operational

Outage at AWS California and Oregon

Resolved
Major outage
Started about 3 years agoLasted about 1 hour

Affected

Synthetics Platform

Major outage from 3:52 PM to 4:38 PM

Primary Synthetics

Major outage from 3:52 PM to 4:38 PM

Incidents Platform

Major outage from 3:52 PM to 4:38 PM

Management Suite

Major outage from 3:52 PM to 4:38 PM

Updates
  • Resolved
    Resolved

    The issue has been resolved by Amazon and the cited root cause is internet connectivity at their US-WEST-2 facility. We'll monitor closely and as mentioned in an earlier update, leave our secondary synthetics layer as primary until we see that the primary layer is stable for a period of 24hrs or, suffer a similar outage requiring a fail over to our 3rd layer.

    This decision does not impact your coverage as a customer or performance of our scripts - our systems are designed with this functionality to keep you covered during exactly this scenario.

    We've also confirmed PagerDuty is back up and running for those using it as their central incident management tool.

  • Identified
    Identified

    During a closer look, we've determined out incident management platform is partially impacted. Though we're still able to escalate, assign, and acknowledge incidents, some users are unable to open the PagerDuty App or website.

    If you use a different incident management platform, you're unimpacted. Those using PagerDuty will still receive text, email, and phone call notifications but push notifications are currently unavailable.

    We believe this to be tied to the same AWS outage impacting our primary tier of synthetics and will advise once we've seen improvement.

    If your organization is using a Cmdr Status page from BEMODEM, this is still working as designed and will accurately reflect and alert on your system health.

  • Investigating
    Investigating

    We are currently investigating an incident impacting some AWS locations used in our primary layer of synthetics. some customer may have seen false alerts (which have all since been commented or resolved since triggered depending on your tools) but, most would have been spared the false alert when our systems cut-over to the secondary layer of synthetics 4mins after the incident began.

    The secondary layer is currently running predominantly on AWS Ohio and Azure Virginia and has been without incident for 90+ days so we plan to leave this running for the next 24hrs, allowing AWS time to figure out the issue on their end and stabilize before we switch back. This should be a seamless transition for all with no performance impact.