February 28, 2024: License Server / Keyprovider Service unable to process requests

Post-mortem: License Server / Keyprovider Service unable to process requests

Date: 2/28/2024 (6:02 pm – 6:22 pm CST)

Impact: This incident caused an outage for the following license server services:

• wvlsmod.swankmp.net
• fairplay.swankmp.net
• playready.swankmp.net
• marlinproxy.swankmp.net

Root Cause:  There was a brief service interruption in the primary environment due to an unexpected issue with the virtual infrastructure supporting the compute service. This resulted in the keyprovider service being unable to process key requests.  Our engineering team quickly identified the problem and initiated a failover to the secondary site to maintain service continuity while they investigated the root cause. The issue was found to be isolated to a single host, which was then rebooted. Additionally, the keyprovider application layer was restarted. These actions resolved the problem, and processing was successfully returned to the primary environment.

Resolution: Restarting of a control node in our virtual infrastructure.

Timeline:

• [6:02 pm]: Alerting starts from unhealthy status of license services
• [6:10 pm]: Dispatching to engineering
• [6:20 pm]: Failover to secondary site initiated
• [6:49 pm]: Hosting issue resolved
• [6:53 pm]: Failback to primary site completed