February 24, 2024: Legacy PlayReady DRM Service Outage

Post-mortem: Legacy PlayReady DRM Service Outage

Date: 2/24/2024

Impact: This incident caused an outage for both OneView and the Legacy PlayReady DRM service.

Root Cause: A typographical error (typo) was identified in a connection string value that was deployed in preparation to migrating a supported database to SQL 2019.   This typo prevented the affected services from establishing necessary connections to the database before and after the migration.

Resolution: The typo was corrected, and connectivity was restored for both services.

Timeline:

• [8:00 am]: SQL Migration work begins.  Swank Technical team reviewing logs of affected services.
• [8:15 am]: Errors related to Legacy PlayReady DRM found (prior and after migration).
• [8:30 am]: Error traced to typographical error in connection string and repaired.   Service affected confirmed.
• [8:35 am]: Legacy PlayReady DRM service issues reported to Swank Digital Support.
• [9:00 am]: Affected customer reports back that service is working at their location.

Lessons Learned:

• Highlighted a gap in our Legacy monitoring for this service.

Next Steps:

• Add additional monitoring to the legacy service that includes database calls