Graph showing campus internet activity

During the evening of April 7th, a piece of hardware at the ISP that provides our internet connection failed. Because we have redundancy built into our connection, the internet remained reachable from campus. That is until the ISP attempted to replace the failed hardware at about 2:20pm on April 8th.
For some reason the replacement of the failed hardware caused an issue with our redundancy, taking both connections offline. Still waiting to hear the ISP’s explanation for this.
About 40 minutes later, the ISP was able to resolve the issue, and get our connection back up. However, it was not fully functional. For some reason the redundant connections were only partially up, causing some traffic to fail.
Around 4:30pm the connections were reset, and traffic was finally fully back to normal.
This was the first time we have had a complete internet connection outage in a very, very, very long time.