Windows Server upgrade followed by quick and dirty Exchange Repair

Posted: September 13, 2014 in Exchange, Solved
Tags: , , ,

NOTE: I am not an Exchange person by any stretch of the imagination, but I do run it at home for all my email as a partial learning exercise, and so get to encounter all sorts of issues I really think I’d rather live without. On that note, this post is here really for my future reference. If anyone else uses this it is at your own risk. Running Exchange 2013 on Windows Server 2012. I wanted to upgrade to Server 2012R2. I was already running Exchange 2013 CU6 so it was supported for 2012R2 at least. There is a *small* note on the Exchange pre-reqs page saying something about upgrading the OS not being supported when Exchange is installed. HA! http://technet.microsoft.com/en-us/library/bb691354(v=exchg.150).aspx#prereq

You can't upgrade Windows when Exchange is installed on the server.

I figured I’d give it a go anyway. I mean, like, what’s the worse that could happen, right? A quick Hyper-V Checkpoint (snapshot) created, checked last nights backups were good and away we go! After the upgrade, the worst happened😦 The OS upgrade went fine, and Exchange seemed to startup. Outlook could connect but not actual send email and nothing was coming in. The Exchange event logs showed that a few minutes after starting the Mailbox was being dismounted, then a few minutes later trying to remount and failing again.

Active Manager failed to mount database Mailbox Database 0352275541 on server [redacted]. Error: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionDatabaseError: Unable to mount database. (hr=0x80004005, ec=1108)
the copy of database 'Mailbox Database 0352275541' on this server was unexpectedly dismounted. The error returned by failover was "There is only one copy of this mailbox database (Mailbox Database 0352275541). Automatic recovery is not available.". For more specific information about the failures, consult the event log on the server for other "ExchangeStoreDb" events.
the copy of database 'Mailbox Database 0352275541' on this server encountered an error during the mount operation. For more information, consult the Event log on the server for "ExchangeStoreDb" or "MSExchangeRepl" events. The mount operation will be tried again automatically.
Microsoft Exchange Information Store worker process (16260) has encountered a fatal database exception (Microsoft.Isam.Esent.Interop.EsentPrimaryIndexCorruptedException: Primary index is corrupt. The database must be defragmented or the table deleted.
Information Store - Mailbox Database 0352275541 (16260) Mailbox Database 0352275541: Database 'E:\Exchange2013\Mailbox\Mailbox Database 0352275541\Mailbox Database 0352275541.edb': The primary index 'GlobalsPK' of table 'Globals' is out of date with sorting libraries. If used in this state (i.e. not rebuilt), it may appear corrupt or get further corrupted. If there is no later event showing the index being rebuilt, then please defragment the database to rebuild the index.

Bummer

SOLUTION

Well, taking shortcuts always has its risks and rewards, so a few quick Eseutils later everything seems to be good to go. Phew!

  1. Stop the Microsoft Exchange Transport Services
    • Or even stop and disable the Microsoft Exchange Active Directory Topology service to make sure nothing starts up while doing this
  2. Open a CMD prompt and go to wherever your Mailbox file is located. eg. E:\Exchange2013\Mailbox\Mailbox Database 0352275541
  3. ESEUTIL /r E00
    • To ensure a clean shutdown state
    • If things are really messed up then might need a ESEUTIL /p “Mailbox Database 0352275541.edb”
  4. ESEUTIL /d “Mailbox Database 0352275541.edb”
    • Performs a defrag, which also cleans up the indexes and works by making a new “copy” of the database
  5. ESEUTIL /g “Mailbox Database 0352275541.edb”
    • Do an integrity check to make sure everything is all nice and happy

Best thing to do from here is restart the server and if all is well things should all happy again. I probably wouldn’t recommend doing an upgrade like this if it’s part of a job you actually get paid money to do for someone else. Do you really need that kind of stress?

And once it all looks good…. do a backup!

UPDATE 14 June 2016: Another thing to do after the repair

The Microsoft Exchange team now have a policy in place where they will only support Exchange systems where the database has zero repairs performed against it. This is due to ongoing inconsistincies that can exist after a repair.

http://blogs.technet.com/b/exchange/archive/2015/05/01/new-support-policy-for-repaired-exchange-databases.aspx

The (simple) solution for this is to create a new mailbox database, then migrate all your mailboxes into the new database. This can be done quite quickly and easily through the EAC or using powershell

This may seem drastic, but for small environments at least it’s not such a big deal. Larger environment will require some planning, but I would assume larger environments are already employing people who know how to handle doing this anyway.

Comments
  1. garypendlebury says:

    As well as repairing the mailbox database, I had to perform a similar action against the SenderReputation database (
    C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\SenderReputation\pasettings.edb), which was also corrupt. The Transport service wouldn’t start without this repair.

    • Scott says:

      Thanks for the tip. I was not aware or had not encountered that. Do you have more info or a link about doing this?

      • garypendlebury says:

        I performed similar repair steps to those suggested for the mailbox database against the SenderReputation database:

        cd /d “%ExchangeInstallPath%\TransportRoles\data\SenderReputation\”
        eseutil /r trn
        eseutil /d “pasettings.edb”
        eseutil /g “pasettings.edb”

        Problem solved!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s