Microsoft Windows Servicing Stack Updates (SSU) – A Rant

Copied from my twitter rant here: https://twitter.com/ip1/status/1039668421615529985

May contain edits and be added to at a later time

A rant about service stack updates (SSU) and why Microsoft really needs to work something better out for this:
When the major monthly cumulative update requires the SSU as a pre-req, it creates major pain points with managed deployment. If you are running “unmanaged” and just let clients go to Microsoft directly to get whatever is applicable, then you probably aren’t worried about this.

WSUS Delete Obsolete Updates

NOTE: Usual warnings apply. Do a backup before making any changes. If you are unsure about anything in the post then ask or look for more information or help before attempting it.

Over time WSUS will accumulate update metadata that can create performance issues for clients. In large environments this can be quite an issue.

There is a script Microsoft often provides during Premier Support calls to cleanup this update metadata, however there are a few issues:

• The query can take a *really* long time to run if there are a lot of updates to cleanup. In some cases it can take *days*
• You need to stop all the WSUS services while it runs
• If it fails for whatever reason, it will have to start all over because it doesn’t commit the changes until it completes successfully
• While it runs, the TEMPDB and Transaction logs will grow quite significantly until the data is committed
• It gives no useful information on progress

There is a TechNet article (This is essential reading and has LOTS of important stuff) and a Forum Post where an improved version was written that gave progress of the cleanup, however it didn’t address the temp/transaction growth issues or the time issues. To this end I have applied my very rudimentary SQL scripting skills.

WSUS SUP causes high CPU and clients fail updates scan

NOTE: The configuration suggestions I mention in this post won’t fix the underlying issue. Depending on the size of your environment they may be enough to get things working for you again. Microsoft is currently working on releasing a hotfix that I have tested and found to resolve this problem

 

Microsoft have released the WSUS server hotfix, details here: https://blogs.technet.microsoft.com/configurationmgr/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

NOTE2: It turns out there is a new issue from the August 2017 updates that “clears” the update history on a computer that will trigger a full client scan again. This will also cause high load on your WSUS server, although for slightly different reasons, however the suggestions here and the coming updates will help to resolve the load issue from that problem as well.

Microsoft have updated the August cumulative updates to resolve this issue, details here: https://support.microsoft.com/en-us/help/4039396/windows-10-update-kb4039396

 

NOTE3: Microsoft has now published some additional official guidance here: https://blogs.technet.microsoft.com/askcore/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

This issue is one I first encountered on only a couple of our WSUS servers (2 or 3 of 15 servers) last year in November 2016 after the new cumulative update process was introduced for patching. At first I assumed it to be a failure on my part to do more regular cleanup, or a result of the recent upgrade to ConfigMgr 1610, or an “end of year” rush of activity on the network. This isn’t unusual for the environment I currently manage (Education with approx. 370,000+ devices)

At first I looked at server bottlenecks (we run everything in VMWare) and even SQL DB corruptions. I tried doing WSUS resets, even recreating the database (this is a last resort in a large environment). I then thought maybe it was a Server 2012 WSUS issue as we had other Server 2012 related cases open with Microsoft. To test I rebuilt one server as 2012R2, but the problems persisted. Given it was only happening on a couple of server I assumed it was an issue with those servers in particular and didn’t suspect a larger issue.

Over the Christmas holidays things went quite, so there was nothing more I could do until school returned the following February.

Then everything basically exploded.

The first patch cycle we ran saw the WSUS server rocket to 100% CPU and stay there. Nothing I did could stop this reoccurring. I found ways to bring things under control for a few hours at a time. Endpoint definitions started falling behind because clients couldn’t scan for updates. Then it started happening on a couple more of the servers. At this point I conceded defeat and called in Microsoft. Unfortunately it was another 6 months before they finally identified it was a “function” of WSUS causing the grief and not the configuration or size of our environment.

The Problem

The most obvious symptoms will be clients failing to scan for updates and the WSUS server CPU (w3wp.exe) going very high. Some clients get through, many will fail. The main cause will be Windows 10 clients and the way WSUS has to process the Cumulative Updates.

Configuration Manager Update source content being deleted

[Place holder post – FULL POST write up coming shortly]

https://social.technet.microsoft.com/Forums/en-US/3ac88d59-d1b5-4826-a44d-bb94bfd80f66/configmgr-2012-update-source-content-deleted?forum=configmanagergeneral#39c42c3c-d3af-4362-b097-09c022adc86b

An issue where some source folder content of Windows Update deployment packages is deleted for no apparent reason.

Currently waiting for a response from the Microsoft ConfigMgr Product team on if they feel this is an issue worthy of a product update or just a TechNet article warning of the impact

Summary: issues relating to Update deployment packages that are migrated between ConfigMgr hierarchies that share the same source folder location. Automated “orphan cleanup” process will delete content from the folder if one of the hierarchies doesn’t need it, even it the other one still does.

Windows Update KB2982791 MS14-045 causing bluescreen

I don’t have a fix of my own for this, but I am seeing a lot of people finding my site through various search engines from my previous Windows RT fix. This is not an issue I have experienced and at the moment is not something I am looking into.

If you need help then the following Microsoft article has some details on a fix.

http://support.microsoft.com/kb/2982791

If you know of another site with more “friendly” instructions, let me know and I’ll link it here. If this problem persists, or if I encounter it myself I will look to post my own fix if nothing else appears from Microsoft.

Create a “dummy” Distribution Point Group for testing

When working with new packages, in particular very large ones, or when performing an action with a wizard that insists on adding distribution points (e.g. Software Update download) I find it helpful to use a “dummy” Distribution Point Group.

The reason for this is simple. For example, I want to create a new Software Updates deployment package that will contain a lot of updates, and using the wizard it requires me to select a target to distribute this new package to. I plan to add other updates or make further changes so I don’t really want (or need) it to be sent to an actual DP just yet. This is especially true when creating temporary update source content (another issue for another post).

To work around this, I just create a “Blank”, or testing Distribution Point Group with no members. This DP Group is perfectly fine to use in the wizard and allows the package source to be created and downloaded without then needing to wait for it to be sent to the DP.

Microsoft Updates deployment with Configuration Manager – Braindump

This post is going to be more of a “brain-dump” post of thoughts and ideas around the process of using ConfigMgr to deploy Microsoft updates in your environment. There are already numerous “guides” on the net already, so why?

This may or may not be a long read. It will just be my thoughts at this stage, with later posts (possibly) going into details of the pros and cons of different approaches and issues encountered. All feedback or experiences you would like to share are most welcome, and I’ll incorporate new ideas and points to consider into the post

Windows Update failing with “Proxy authentication required” 0x8024401B errors

This issue isn’t specifically related to Configuration Manager or Windows Updates, but it does seem that Windows Updates itself could behave better to prevent this.

When Windows Updates attempts to connect to the WSUS server (or SUP), you will see the WINDOWSUPDATE.LOG show the client attempting to open the client.asmx file from the WSUS server. The lines following show this connection failing with “SyncUpdates failure, error = 0x8024401B, soap client error = 10, soap error code = 0, HTTP status code = 407” followed by several more 0x8024401b errors.

Endpoint Protection Log locations

Another “reminder where stuff is” post. This time for the Endpoint Protection logs. These should be the same for pretty much any version as far as I know, but I’m looking specifically at System Centre Endpoint Protection (SCEP) included as part of Config Manager 2012.

Log locations:

• %allusersprofile%\Microsoft\Microsoft Antimalware\Support—Log files specific for the antimalware service
• %allusersprofile%\Microsoft\Microsoft Security Client\Support—Log files specific for the SCEP client software
• %windir%\WindowsUpdate.log—Windows Update log files, which include information about definition updates
• %windir%\CCM\Logs\EndpointProtectionagent.log – Shows Endpoint version and policies applied
• %windir%\temp\MpCmdRun.log – Activity when performing scans and signature updates
• %windir%\temp\MpSigStub.log – Update progress for signature and Engine updates

References:

http://technet.microsoft.com/en-us/library/gg477022.aspx

Windows RT 8.1 upgrade fails with Boot Configuration error

UPDATE 19 August 2014 regarding KB14-045 bluescreens:

Although I don’t have a fix, there seems to be a lot of people finding this post while searching about the recent bluescreen issue caused by Microsoft Updates KB14-045. I have started a new post about this issue here:

https://kickthatcomputer.wordpress.com/2014/08/19/windows-update-kb2982791-ms14-045-causing-bluescreen/

---

Microsoft have now released the Full Recovery Image for WinRT (3.7GB download). You can get it here: http://www.microsoft.com/surface/en-us/support/warranty-service-and-recovery/surface-rt-startup-error-0xc000000d?lc=1033

It uses the same steps to complete the upgrade with the last part being to recreate the full recovery volume. Some news sites are saying the MS image will revert you to 8.0, it does not. It will complete the 8.1 upgrade and then the extra steps are to recreate the on-device recovery volume with the 8.0 image which only applies if you rollback sometime in the future.

First point – How annoying is it that you can only update Windows RT from the App Store, although probably not a drama if you only have one tablet. Slightly annoying if you have two. Both need to download the update, so that’s 2x the several GB needed. If you happen to be an enterprise/business that decided to buy several of them (does such a entity exist?) then it would appear you are about to have a ton of downloads happening unless you boost your proxy cache size.

And now for todays story. I updated one Surface RT with no drama. It took a long time but I just let it run overnight and it was all good the next morning. The other tablet, not so good.

Recovery
Your PC needs to be repaired
The Boot Configuration Data file is missing some required information
File: \BCD
Error code: 0xc000000d

Yay.

Although it takes a while, and might be a bit tricky, it does seem to be fairly easy to recover from. (UPDATE: I have now received multiple confirmations that this works, and no data is lost!)