ConfigMgr Client missing Health Eval Task

I found a couple of servers that were reporting they had failed the Configuration Manager Client Health Evaluation. No problem, how about I just manually run the Health Eval scheduled task…
Ah, there’s your problem. “Computer says NO”

Checking the clients CCMEVALTASK.LOG shows lots of lovely red for “Failed to create client evaluation task” and “Failed to delete task Configuration Manager Health Evaluation (0x80070002)”. I already know this is going to be similar to other schedule task creation problems.

WSUS SUP causes high CPU and clients fail updates scan

NOTE: The configuration suggestions I mention in this post won’t fix the underlying issue. Depending on the size of your environment they may be enough to get things working for you again. Microsoft is currently working on releasing a hotfix that I have tested and found to resolve this problem

 

Microsoft have released the WSUS server hotfix, details here: https://blogs.technet.microsoft.com/configurationmgr/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

NOTE2: It turns out there is a new issue from the August 2017 updates that “clears” the update history on a computer that will trigger a full client scan again. This will also cause high load on your WSUS server, although for slightly different reasons, however the suggestions here and the coming updates will help to resolve the load issue from that problem as well.

Microsoft have updated the August cumulative updates to resolve this issue, details here: https://support.microsoft.com/en-us/help/4039396/windows-10-update-kb4039396

 

NOTE3: Microsoft has now published some additional official guidance here: https://blogs.technet.microsoft.com/askcore/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

This issue is one I first encountered on only a couple of our WSUS servers (2 or 3 of 15 servers) last year in November 2016 after the new cumulative update process was introduced for patching. At first I assumed it to be a failure on my part to do more regular cleanup, or a result of the recent upgrade to ConfigMgr 1610, or an “end of year” rush of activity on the network. This isn’t unusual for the environment I currently manage (Education with approx. 370,000+ devices)

At first I looked at server bottlenecks (we run everything in VMWare) and even SQL DB corruptions. I tried doing WSUS resets, even recreating the database (this is a last resort in a large environment). I then thought maybe it was a Server 2012 WSUS issue as we had other Server 2012 related cases open with Microsoft. To test I rebuilt one server as 2012R2, but the problems persisted. Given it was only happening on a couple of server I assumed it was an issue with those servers in particular and didn’t suspect a larger issue.

Over the Christmas holidays things went quite, so there was nothing more I could do until school returned the following February.

Then everything basically exploded.

The first patch cycle we ran saw the WSUS server rocket to 100% CPU and stay there. Nothing I did could stop this reoccurring. I found ways to bring things under control for a few hours at a time. Endpoint definitions started falling behind because clients couldn’t scan for updates. Then it started happening on a couple more of the servers. At this point I conceded defeat and called in Microsoft. Unfortunately it was another 6 months before they finally identified it was a “function” of WSUS causing the grief and not the configuration or size of our environment.

The Problem

The most obvious symptoms will be clients failing to scan for updates and the WSUS server CPU (w3wp.exe) going very high. Some clients get through, many will fail. The main cause will be Windows 10 clients and the way WSUS has to process the Cumulative Updates.

Configuration Manager 1610 adventures

Yes, we skipped 1602 and went to 1606 in our Dev environment, but due to various change freezes, conflicts with other projects and change management delays we have decided we will be going from 1511 direct to 1610 in Prod.

The Dev upgrade went OK (using the “FastRing” powershell script) however it was then announced there were some additional bugs found and a newer 1610 installed was released. The updates though have not as yet been posted to update the people that had run the original 1610 install…

Locating Orchestrator Runbooks using bad credentials

A problem recently encountered was causing major headaches. There was a runbook somewhere in the system that had an action running with the security credentials of a user that had left sometime ago. Their account had recently been disabled and the Orchestrator was logging thousands of errors and causing the Orchestrator database to grow at a massive rate. OBJECTINSTANCEDATA was growing at thousands of rows a second and hit 50 million rows and 16GB in size after only a few days.

Configuration Manager 1602 adventures

Each new update of ConfigMgr I will start a new entry for anything specific to that version I want to test or make notes of.

ConfigMgr object locked for editing

Often if editing something like a Boot Image or Task Sequence and you experience a console crash, then next time you try to edit that object you may see a message saying it is currently locked for editing, giving only a choice to wait or open as Read-Only.

Some quick (unsupported) database edits can clear the lock, or there is a Powershell command that does the trick as well. The Powershell is a bit trickier, but is the “proper” supported method.

PowerShell: How to unlock objects in SCCM

Powershell Unlock-CMObject: https://technet.microsoft.com/en-us/library/jj821915(v=sc.20).aspx

and more info on how SEDO works: http://blogs.technet.com/b/sudheesn/archive/2012/10/28/sedo-serialized-editing-of-distributed-objects-configmgr-2012.aspx

 

Configuration Manager 1511 adventures

I am starting work on now upgrading have upgraded our ConfigMgr2012 R2 hierarchies to the new “Configuration Manager with no more version numbers” build 1511. There are roughly 120+ site servers in the central hierarchy (across 2 CAS hierarchies), so hopefully things go well. For the most part things went quite well, no major issues at all.

I’ll keep this page to note any special gotchas that are outside the various step-by-step guides already out there, and also some easy quick link references.

Configuration Manager Replication queue backlog clearing

I recently experienced an issue with a very large number of ConfigMgr 2007 package updates (400) to a large number of sites (1700). It turns out there was already a distribution job that was “stuck” in the queue and when the large update went out it resulted in a massive backlog. The end result being there were over 1.3 million files in the Replication Manager inbox that just weren’t being processed, and the number was increasing.

The only option in this sort of situation is to stop the services, move the files out of the inbox, let normal inbox processing resume and then copy the files back in a block at a time. In this case doing this manually wasn’t an option due to the number of files, so I resorted to a quick script based on the one found here: https://tricksntreats.wordpress.com/2010/05/30/sccm-backlog-fighting/

ConfigMgr client constantly reinstalling every 5 hours

I first had this happen on just one of my servers. It was annoying and was causing massive numbers of WMI entries in the event logs each time it happened. I tried to just stop and disable the ConfigMgr client, but it would be “reactivated” again by something. Recently at a client site I found it happening an a large number… so time to investigate.

I checked there were no push jobs running, but there was *something* causing it to repeat. Some digging later and I finally found the reason.

There is a scheduled task created by CCMSETUP to retry if it isn’t able to install correctly on the first attempt. Under some circumstances, this task isn’t cleaned up and so the reinstall keeps happening

How to update ConfigMgr Package and Application source paths

When you need to change the location that you have been saving all your package source files, you will then need to update all the existing packages and applications to point to that new location.

(Note: See here to read what happens when you do update all your source locations)

This is fairly simple for Packages, but is a bit more complicated for Applications