Archive for the ‘Configuration Manager’ Category

NOTE: Usual warnings apply. Do a backup before making any changes. If you are unsure about anything in the post then ask or look for more information or help before attempting it.

Over time WSUS will accumulate update metadata that can create performance issues for clients. In large environments this can be quite an issue.

There is a script Microsoft often provides during Premier Support calls to cleanup this update metadata, however there are a few issues:

  • The query can take a *really* long time to run if there are a lot of updates to cleanup. In some cases it can take *days*
  • You need to stop all the WSUS services while it runs
  • If it fails for whatever reason, it will have to start all over because it doesn’t commit the changes until it completes successfully
  • While it runs, the TEMPDB and Transaction logs will grow quite significantly until the data is committed
  • It gives no useful information on progress

There is a TechNet article (This is essential reading and has LOTS of important stuff) and a Forum Post where an improved version was written that gave progress of the cleanup, however it didn’t address the temp/transaction growth issues or the time issues. To this end I have applied my very rudimentary SQL scripting skills.

(more…)

Advertisements

NOTE: The configuration suggestions I mention in this post won’t fix the underlying issue. Depending on the size of your environment they may be enough to get things working for you again. Microsoft is currently working on releasing a hotfix that I have tested and found to resolve this problem

 

Microsoft have released the WSUS server hotfix, details here: https://blogs.technet.microsoft.com/configurationmgr/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

NOTE2: It turns out there is a new issue from the August 2017 updates that “clears” the update history on a computer that will trigger a full client scan again. This will also cause high load on your WSUS server, although for slightly different reasons, however the suggestions here and the coming updates will help to resolve the load issue from that problem as well.

Microsoft have updated the August cumulative updates to resolve this issue, details here: https://support.microsoft.com/en-us/help/4039396/windows-10-update-kb4039396

 

NOTE3: Microsoft has now published some additional official guidance here: https://blogs.technet.microsoft.com/askcore/2017/08/18/high-cpuhigh-memory-in-wsus-following-update-tuesdays/

This issue is one I first encountered on only a couple of our WSUS servers (2 or 3 of 15 servers) last year in November 2016 after the new cumulative update process was introduced for patching. At first I assumed it to be a failure on my part to do more regular cleanup, or a result of the recent upgrade to ConfigMgr 1610, or an “end of year” rush of activity on the network. This isn’t unusual for the environment I currently manage (Education with approx. 370,000+ devices)

At first I looked at server bottlenecks (we run everything in VMWare) and even SQL DB corruptions. I tried doing WSUS resets, even recreating the database (this is a last resort in a large environment). I then thought maybe it was a Server 2012 WSUS issue as we had other Server 2012 related cases open with Microsoft. To test I rebuilt one server as 2012R2, but the problems persisted. Given it was only happening on a couple of server I assumed it was an issue with those servers in particular and didn’t suspect a larger issue.

Over the Christmas holidays things went quite, so there was nothing more I could do until school returned the following February.

Then everything basically exploded.

The first patch cycle we ran saw the WSUS server rocket to 100% CPU and stay there. Nothing I did could stop this reoccurring. I found ways to bring things under control for a few hours at a time. Endpoint definitions started falling behind because clients couldn’t scan for updates. Then it started happening on a couple more of the servers. At this point I conceded defeat and called in Microsoft. Unfortunately it was another 6 months before they finally identified it was a “function” of WSUS causing the grief and not the configuration or size of our environment.

The Problem

The most obvious symptoms will be clients failing to scan for updates and the WSUS server CPU (w3wp.exe) going very high. Some clients get through, many will fail. The main cause will be Windows 10 clients and the way WSUS has to process the Cumulative Updates.

(more…)

Yes, we skipped 1602 and went to 1606 in our Dev environment, but due to various change freezes, conflicts with other projects and change management delays we have decided we will be going from 1511 direct to 1610 in Prod.

The Dev upgrade went OK (using the “FastRing” powershell script) however it was then announced there were some additional bugs found and a newer 1610 installed was released. The updates though have not as yet been posted to update the people that had run the original 1610 install…

(more…)

Each new update of ConfigMgr I will start a new entry for anything specific to that version I want to test or make notes of.

(more…)

Often if editing something like a Boot Image or Task Sequence and you experience a console crash, then next time you try to edit that object you may see a message saying it is currently locked for editing, giving only a choice to wait or open as Read-Only.

Some quick (unsupported) database edits can clear the lock, or there is a Powershell command that does the trick as well. The Powershell is a bit trickier, but is the “proper” supported method.

https://configmonkey.wordpress.com/2015/03/27/powershell-how-to-unlock-objects-in-sccm/

Powershell Unlock-CMObject: https://technet.microsoft.com/en-us/library/jj821915(v=sc.20).aspx

and more info on how SEDO works: http://blogs.technet.com/b/sudheesn/archive/2012/10/28/sedo-serialized-editing-of-distributed-objects-configmgr-2012.aspx

 

I am starting work on now upgrading have upgraded our ConfigMgr2012 R2 hierarchies to the new “Configuration Manager with no more version numbers” build 1511. There are roughly 120+ site servers in the central hierarchy (across 2 CAS hierarchies), so hopefully things go well. For the most part things went quite well, no major issues at all.

I’ll keep this page to note any special gotchas that are outside the various step-by-step guides already out there, and also some easy quick link references.

(more…)

This will be a quick reference page for links to build numbers for versions of various products

Configuration Manager 2012

Exchange 2007-2010

Exchange 2013

SQL Server

Windows Server and Workstation – Need better reference with build numbers

I recently experienced an issue with a very large number of ConfigMgr 2007 package updates (400) to a large number of sites (1700). It turns out there was already a distribution job that was “stuck” in the queue and when the large update went out it resulted in a massive backlog. The end result being there were over 1.3 million files in the Replication Manager inbox that just weren’t being processed, and the number was increasing.

The only option in this sort of situation is to stop the services, move the files out of the inbox, let normal inbox processing resume and then copy the files back in a block at a time. In this case doing this manually wasn’t an option due to the number of files, so I resorted to a quick script based on the one found here: https://tricksntreats.wordpress.com/2010/05/30/sccm-backlog-fighting/

(more…)

I first had this happen on just one of my servers. It was annoying and was causing massive numbers of WMI entries in the event logs each time it happened. I tried to just stop and disable the ConfigMgr client, but it would be “reactivated” again by something. Recently at a client site I found it happening an a large number… so time to investigate.

I checked there were no push jobs running, but there was *something* causing it to repeat. Some digging later and I finally found the reason.

There is a scheduled task created by CCMSETUP to retry if it isn’t able to install correctly on the first attempt. Under some circumstances, this task isn’t cleaned up and so the reinstall keeps happening

(more…)

When you need to change the location that you have been saving all your package source files, you will then need to update all the existing packages and applications to point to that new location.

(Note: See here to read what happens when you do update all your source locations)

This is fairly simple for Packages, but is a bit more complicated for Applications (more…)