January 14, 2025: Outage with media1 CDN services

Date: 1/14/2025

Impact: Cloud Streaming Portal and Cloud API customers are facing 5XX errors accessing thumbnails, trailers, and other assets from the media1-swank-com.azureedge.net and media1.swank.com endpoints for our media asset CDN.  Placeholder images display on the Cloud Streaming portals' UI.

Content playback is not impacted

Root Cause: Azure's migration of media1-swank-com from a deprecated CDN service to a new service created unexpected disruptions of content delivery from that new service.

Resolution: Asset delivery via API and Cloud Web UI is restored with changing the media1-swank-com domain to a new secondary CDN service previously established.  No changes required on the customer end.  Slightly slower loading delays will remain for a few hours while the assets propagate across the CDN.

Timeline:

• [8:00 am]: Customers reporting missing assets and placeholder images from the CDN.
• [9:30 am]: Service appears to be partially restored, with assets beginning to return as available.
• [10:30 am]: CDN service returns to unavailable status.  Team begins investigating the redirect to a secondary CDN, media2-swank-com
• [11:30 am]:  Team switched media1-swank-com endpoint to a new CDN endpoint to restore asset delivery.  New certificates need to propagate over the CDN, as well as assets to the local endpoints.  Cloud Streaming portals are displaying all available assets as expected, though with some longer load times that should continue to improve.

Lessons Learned:

• Cloud Streaming Portals' UI remained operational and fallback to a default image worked as desired.
• Customers were able to continue content streaming without interruption. 
• Team investigating next strategy for the publicity asset CDN to improve ability to limit disruption impact.

Next Steps:

• Team is continuing to monitor the restored media1-swank-com as the assets continue to propagate.