Locating Orchestrator Runbooks using bad credentials

A problem recently encountered was causing major headaches. There was a runbook somewhere in the system that had an action running with the security credentials of a user that had left sometime ago. Their account had recently been disabled and the Orchestrator was logging thousands of errors and causing the Orchestrator database to grow at a massive rate. OBJECTINSTANCEDATA was growing at thousands of rows a second and hit 50 million rows and 16GB in size after only a few days.

Our standard practice is that any runbooks needing specific permissions should have a service account created to avoid just this situation. Obviously this one was done as a “test” and didn’t get updated with a proper account when it became production.

In this case I was seeing event entries in Orchestrator, and the Security event logs on the runbook server were also showing the failed login attempts, but there was no obvious way to connect that back to which runbook was actually causing it.

SOLUTION

The method that finally cracked it was using the Security Event Viewer entries to get the Process ID of the PolicyModule.exe and then looking up that PID in the Orchestrator database

The Caller Process ID is the magic value. It is presented in hexadecimal, but the Windows calculator or a search engine can convert it to decimal for you. In this case 0xdc4 = 3524

Now we can query the Orchestrator database to get the runbook causing the issue

SELECT     POLICYINSTANCES.ProcessID, [Microsoft.SystemCenter.Orchestrator].Runbooks.Name, [Microsoft.SystemCenter.Orchestrator].Runbooks.Description, 
                      [Microsoft.SystemCenter.Orchestrator].Runbooks.Path, POLICYINSTANCES.TimeStarted, ACTIONSERVERS.Computer
FROM         POLICYINSTANCES INNER JOIN
                      [Microsoft.SystemCenter.Orchestrator].Runbooks ON POLICYINSTANCES.PolicyID = [Microsoft.SystemCenter.Orchestrator].Runbooks.Id INNER JOIN
                      ACTIONSERVERS ON POLICYINSTANCES.ActionServer = ACTIONSERVERS.UniqueID
WHERE     (POLICYINSTANCES.ProcessID = 3524)

This will give you a list of any runbooks that have run under this process ID, the server they ran on and the time. You can match this up to your event log entries if there are more than one.

From there it’s just a matter of looking at the runbook, finding the offending action and then having a conversation with everyone else still creating runbooks about why we use service accounts.

If anyone knows of an easier way to work this, please let me know.