WSUS SUP causes high CPU and clients fail updates scan

NOTE: The configuration suggestions I mention in this post won’t fix the underlying issue. Depending on the size of your environment they may be enough to get things working for you again. Microsoft is currently working on releasing a hotfix that I have tested and found to resolve this problem

 

Microsoft have released the WSUS server hotfix, details here: https://blogs.technet.microsoft.com/configurationmgr/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

NOTE2: It turns out there is a new issue from the August 2017 updates that “clears” the update history on a computer that will trigger a full client scan again. This will also cause high load on your WSUS server, although for slightly different reasons, however the suggestions here and the coming updates will help to resolve the load issue from that problem as well.

Microsoft have updated the August cumulative updates to resolve this issue, details here: https://support.microsoft.com/en-us/help/4039396/windows-10-update-kb4039396

 

NOTE3: Microsoft has now published some additional official guidance here: https://blogs.technet.microsoft.com/askcore/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

This issue is one I first encountered on only a couple of our WSUS servers (2 or 3 of 15 servers) last year in November 2016 after the new cumulative update process was introduced for patching. At first I assumed it to be a failure on my part to do more regular cleanup, or a result of the recent upgrade to ConfigMgr 1610, or an “end of year” rush of activity on the network. This isn’t unusual for the environment I currently manage (Education with approx. 370,000+ devices)

At first I looked at server bottlenecks (we run everything in VMWare) and even SQL DB corruptions. I tried doing WSUS resets, even recreating the database (this is a last resort in a large environment). I then thought maybe it was a Server 2012 WSUS issue as we had other Server 2012 related cases open with Microsoft. To test I rebuilt one server as 2012R2, but the problems persisted. Given it was only happening on a couple of server I assumed it was an issue with those servers in particular and didn’t suspect a larger issue.

Over the Christmas holidays things went quite, so there was nothing more I could do until school returned the following February.

Then everything basically exploded.

The first patch cycle we ran saw the WSUS server rocket to 100% CPU and stay there. Nothing I did could stop this reoccurring. I found ways to bring things under control for a few hours at a time. Endpoint definitions started falling behind because clients couldn’t scan for updates. Then it started happening on a couple more of the servers. At this point I conceded defeat and called in Microsoft. Unfortunately it was another 6 months before they finally identified it was a “function” of WSUS causing the grief and not the configuration or size of our environment.

The Problem

The most obvious symptoms will be clients failing to scan for updates and the WSUS server CPU (w3wp.exe) going very high. Some clients get through, many will fail. The main cause will be Windows 10 clients and the way WSUS has to process the Cumulative Updates.

When a client performs a scan, WSUS will generate an XML response to the client with the update metadata. This will vary in size depending on what products and categories you are syncing in WSUS and what updates you have that are not declined or expired.

When WSUS generates the XML for a Windows 10 device it may trigger a state causing the WSUS IIS worker process to consume all available CPU which can lead to other issues on the WSUS server. Given time (minutes) it may complete the request and continue, other times it just never finishes.

Configuration Manager clients have a “special” feature that plain WSUS clients don’t have. When a request to scan fails they will retry 4 times before waiting to try again the next day. WSUS clients don’t retry. This has the effect of amplifying the issues as clients hit the server multiple times.

Symptoms

These are based on my own observations and have proven a reliable way to see the issue starting to happen. I now have a live perfmon windows connecting to each WSUS server watching the counters described below so I can identify the problem starting almost immediately.

• CPU on WSUS server will hit 100% (if no IIS throttling enabled)
• Clients fail when connecting to WSUS to perform update scan
  • Clients report timeout in windowsupdate log/events
  • ConfigMgr causes clients to retry 4 times when an error occurs which will increase the load on the system and make it worse
• Built-in ConfigMgr report for last scan states will show increasing number of clients failing or pending retry while “Completed” count drops
  • Software Updates – D Scan > Scan 2 – Last scan states by site
  • 
• WSUS AppPool worker process (w3wp.exe) shows very high CPU usage
  • Specifically related to the Clientwebservice web app
• Perfmon
  • Counter: Processor > _Total > %Processor Time
    • Shows high CPU usage
  • Counter: Web Service > WSUS Administration > Current Anonymous Users
    • Indications of internal thread processing by WSUS
    • A very high (>50) for extended periods or rapidly increasing value indicates internal processing is not completing
  • Counter: Web Services > WSUS Administration > Current Connections
    • Shows total external clients making connections to WSUS
  • Current Anonymous Users shows a “sawtooth” pattern as the count climbs and is reset by the AppPool hitting limits and recycling. Note: Current Anonymous Users refers to internal processes within the IIS WSUS apps and is not related to authentication of incoming client connections
  • “Normal” processing will rarely show Average Current Anonymous Users greater than single digits
    • 
      • Blue – IIS current connections (set to limit to 1200 here)
      • Brown – CPU with throttling enabled at 70%
      • Black – Current Anonymous Users climbing to nearly 600 at peak)
  • Counter: WSUS: Client Web Service > spgetcorexml > Average execution time
  • Counter: WSUS: Client Web Service > spgetcorexml > Cache hit ratio
  • Counter: WSUS: Client Web Service > spgetcorexml > Cache size
  • “Bad Server”
    • 
    • Cache Size (black) will vary widely as the system attempts to process the client requests to generate XML responses
    • Cache hit Ratio (blue) also varies wildly as the system is unable to find optimisations
    • Average Execution Time varies or shows constantly high values
  • “Good Server”
  • 
    • Cache Size (black) will reach optimum level and remain fairly consistent
    • Cache Hit Ratio (blue) will be near 1.00 showing very good optimisation when processing client requests
    • Average Execution time will be near 0
• Problem related to cumulative updates
  • WSUS server is generating an XML file to reply to client scan requests and generating that file is creating massive load on the worker process
  • In some cases leaving it to run high will eventually resolve, more often it never stopped until the process was terminated

Remediation

NOTE: These suggestions will not fix or prevent the actual issue, but might help keep things running until clients catch up and settle down again. Results will vary depending on the size of your environment

• Throttle worker process (Do this first to get CPU under control so server will be responsive again
  • Create new Application Pool (Copy settings on current WsusPool)
  • Name: ClientWebService
  • Queue Length: 2000
  • Identity: NetworkService
  • Limit: Server 2012=70000, Server 2012R2=70
  • Maximum Worker Processes: 2
  • Assign the WSUS ClientWebService web app to the new AppPool
    • 
  • Restart IIS to give everything a fresh start
  • In theory, if a client request comes in that triggers a worker process to be consumed, the remaining worker process(es) can continue servicing new client requests
  • When a “bad” request is being processed you will see one (or more) of the w3wp.exe processes using CPU. If process remains high for a long time terminate w3wp.exe process and it will recycle while leaving the remaining worker process running
  • 
    • Current Worker Processes can be seen under IIS > Server node > Worker Processes (showing 4 worker processes for ClientWebService here)
  • Task Manager will show w3wp.exe processes and Process ID’s can be matched against IIS
    • 
• Private memory limit
  • Set to 0 for unlimited, or as large as you are comfortable with if server is shared
  • Minimum of 8192000 (8GB) is recommended but adjust to suit environment
• Add additional Server memory and CPU’s
  • In a virtual environment this will be easier and may help
• Clean up WSUS
  • While this did not make any difference for us in this case, it is still good hygiene 🙂
  • Decline previous cumulative updates that are superseded by more recently approved ones
  • Decline any superseded updates where a newer update is approved
  • Don’t sync updates in WSUS for products you don’t need (reduce the overall size of the update metadata catalogue)
  • Check only the language(s) required have been enabled on the WSUS server
  • Check this post, and the articles it references for additional clean up activity “Remove obsolete/old updates“
• Reduce ConfigMgr client update scan cycles. Remember to restore these once everything is working again
  • This is to reduce the number of “hits” the WSUS server will need to deal with and help reduce retry backlogs
  • Reduce the “check for definition updates” client settings (2 days)
  • Reduce the check for updates agent schedule (7 days)
  • Reduce the updates reevaluation agent schedule (7 days)
• Set IIS limits to 50 (this is an arbitrarily low number and not chosen for any specific reason other than it seems to work most times)
  • 
  • Slowly increase in increments of 25 a few minutes after CPU returns to normal
  • Each time the value is increased it will trigger a recycle of the AppPool
  • This can help to get back up to full function, but it only takes a couple of machines making catalogue requests to break it again
  • During the period of IIS connections being limited, clients will receive a message from the IIS server saying it is unavailable still causing clients to retry, however the IIS/WSUS server itself is spared needing to attempt processing as many clients connections

There is a hotfix from Microsoft on the way that will resolve this. Currently estimated to be released late September if all goes well.