Continued TFN Inbound Issues

Resolved
Resolved
After 2 weeks, 6 days, 14 hours, and 8 minutes

Monitoring update: In light of the fact that the telecom vendor has seen continuous restoring of services after the DDOS attack and has not changed the status since November 23, 2021, Plum Voice is marking this event as Resolved. In the event there is a status update, our team will also post those here.

Recovering
After 2 weeks, 6 days, 1 hour, and 27 minutes

Monitoring update: Status remains constant- the telecom vendor continues to observe restored services. The Plum team is continuing to monitor.

Recovering
After 2 weeks, 5 days, 14 hours, and 58 minutes

Monitoring update: Our partner telecom vendor's status remains the same- they continue to observe restored services. The Plum team is continuing to monitor as well.

Recovering
After 2 weeks, 5 days, 1 hour, and 34 minutes

Monitoring update: Status remains constant- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 4 days, 14 hours, and 27 minutes

Monitoring update: Status remains constant- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 4 days, 1 hour, and 32 minutes

Monitoring update: Status remains the same- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 3 days, 13 hours, and 44 minutes

Monitoring update: Status remains the same- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 3 days, 1 hour, and 33 minutes

Monitoring update: Status remains the same- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 2 days, 14 hours, and 20 minutes

Monitoring update: Status remains the same- Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 2 days, 1 hour, and 32 minutes

Monitoring update: Our partner telecom vendor continues to see restored services. We are continuing to monitor.

Recovering
After 2 weeks, 1 day, 14 hours, and 24 minutes

Monitoring update: Our partner telecom vendor continues to see restored services and we are continuing to monitor.

Recovering
After 2 weeks, 20 hours, and 27 minutes

Our partner telecom vendor continues to see restored services and we are continuing to monitor.

Recovering
After 2 weeks, 14 hours, and 20 minutes

Status remains the same- telecom vendor continues to see restored services following the earlier DDoS attack on their network. Plum Voice and vendor continue to monitor.

Recovering
After 1 week, 6 days, 23 hours, and 37 minutes

Status remains the same- telecom vendor continues to see restored services following the earlier DDoS attack on their network. Plum Voice continues to monitor.

Recovering
After 1 week, 6 days, 20 hours, and 45 minutes

The telecom vendor continues to see restored services following an earlier DDoS attack on their network. We are continuing to monitor.

Recovering
After 1 week, 6 days, 17 hours, and 47 minutes

The telecom vendor is experiencing restored services from an earlier DDoS attack on their network. We are continuing to monitor.

Recovering
After 1 week, 5 days, 22 hours, and 37 minutes

The root cause of the issue has been identified by the involved telecom vendor. Remediation is still in progress and we continue to monitor.

Thank you.

Recovering
After 1 week, 5 days, 19 hours, and 47 minutes

Our team is continuing to monitor the situation and is coordinating with the appropriate parties at the impacted telecom vendor.

Recovering
After 1 week, 5 days, 18 hours, and 23 minutes

A renewed DDoS attack on a major telecom vendor is actively being mitigated by the telecom vendor's team. We are monitoring the situation.

Recovering
After 1 week, 5 days, 17 hours, and 12 minutes

We are currently observing intermittent call failures on toll-free PCI and non-PCI numbers. Our team is investigating and escalating with the appropriate resources. We appreciate your patience as we continue to look into this. Please direct any and all questions to support@plumvoice.com. Thank you.

Recovering
After 1 week, 3 days, 5 hours, and 52 minutes

Our team is seeing service restored to normal. All testing is successful and monitoring has cleared. We are continuing to monitor further at this time.

Identified
After 1 week, 3 days, 4 hours, and 47 minutes

Monitoring has alerted us to intermittent issues affecting PCI TFN numbers. This is currently escalated and already under review with our vendors. Thank you for your patience while this is being investigated further.

Identified
After 1 week, 22 hours, and 24 minutes

DDoS attacks are actively being mitigated outside of the Plum Voice network. No discernible service impairments are present. This information is being provided as a precaution and we are continuing to monitor system performance.

Recovering
After 1 week, 18 hours, and 3 minutes

We are observing a restoration of normal call connections and audio quality. We are continuing to monitor for changes.

Identified
After 1 week, 16 hours, and 22 minutes

Audio quality has increased. Calls are connecting successfully.

Identified
After 1 week, 16 hours, and 20 minutes

We've received alerts of call quality and connection issues. Certain calls are not connecting. Other calls exhibit degraded audio quality. These symptoms are in line with previous DDoS attacks. Additional context on these attacks can be found here:

https://portal.plumvoice.com/hc/en-us/community/posts/4410061201307-Telecom-Impact-Announcement

We appreciate your patience and understanding.

Recovering
After 4 days, 16 hours, and 59 minutes

We've received word that all impacted carriers have resolved issues stemming from a previous DDoS attack. At this time we are monitoring from our end to help ensure no additional service disruption is present. We are leaving this incident open but in a recovering status as there are no observable impairments at this time.

Identified
After 4 days, 2 hours, and 18 minutes

Plum Voice infrastructure and applications are functioning normally. All DDoS attacks are not native to our platform. These attacks are limited to certain carriers and prevent calls from being passed through to the Plum Voice platform. Testing and monitoring are still underway.

Identified
After 4 days, 1 hour, and 26 minutes

Current testing indicates that a portion of calls are not being completed. This is potentially an extension of previous waves of DDoS attacks.

Recovering
After 3 days, 21 hours, and 35 minutes

All internal testing points to the DDoS remediation attempts as being successful. We are continuing to test and monitor until the final all clear is given. No observable impairments are occurring at this time.

Identified
After 3 days, 5 hours, and 25 minutes

Remediation steps are still underway and additional steps have been taken to improve audio quality. We are continuing to monitor.

Identified
After 2 days, 21 hours, and 36 minutes

Our team is continuing to test and follow along with all current issues. Migration of services to prevent further DDoS attacks is still underway. At this time it is possible to experience failed calls or degraded audio quality until the migration is finalized and issues are completely resolved. We appreciate your patience and understanding.

Identified
After 2 days, 20 hours, and 6 minutes

We are continually testing and monitoring. At this time, we are experiencing ~ 20% call failure.

Identified
After 2 days, 19 hours, and 15 minutes

Call quality issues persist alongside a certain percentage of calls not connecting. Issues are intermittent and we continue to test and monitor.

Identified
After 2 days, 19 hours, and 10 minutes

Calls are connecting as expected. There is continued degradation in audio call quality.

Identified
After 2 days, 18 hours, and 34 minutes

Remediation steps have proven effective. Note: calls will complete with degraded voice quality.

Identified
After 2 days, 18 hours, and 26 minutes

DDoS remediation is currently underway as another wave of DDoS attacks has taken effect. The potential for failed calls exist and we are monitoring the situation.

Recovering
After 1 day, 18 hours, and 50 minutes

All issues are reported as being resolved and DDoS mitigation remediation has been successful. We are continuing to monitor the situation before formally flagging this as resolved.

Recovering
After 16 hours and 59 minutes

Our partner carrier has engaged remediation experts to prevent additional attacks from re-occuring. We have not experienced any additional issues but we continuing to monitor as this incident continues.

Recovering
After 2 hours and 2 minutes

We are receiving early indications that normal functionality has been restored. We are continuing to monitor for signs of additional health.

Identified
After 1 hour and 11 minutes

We are continuing to monitor the situation. Thank you.

Identified
After 12 minutes

The core of the issue has been identified and it is not native to the Plum Voice infrastructure. A distributed denial-of-service (DDOS) attack is impacting a major carrier within the telecom space.

Investigating

We've received reports of continued TFN inbound issues. We will continue to monitor.

Began at: