First, I’ll admit it. I got stupid. I was messing around where I did not belong and as a result, I took out the console access for our pre-production management group.

With that being said, I’ll make two disclaimers here. First, don’t freakin TOUCH your backend SCOM database without first doing a full backup!!! I cannot stress that enough! Second, unless you really know what you are doing, don’t TOUCH your backend database!

So, I created a management pack for the Exchange folks and sealed it. I imported it into the SCOM environment and life was good. They now have a sealed MP for their Exchange customizations and an Exchange overrides MP for their overrides. Life is good. In pre-prod where we develop and test our MP’s, I tried to delete the signed MP and import the unsigned MP so we could make changes and tune, etc. but it did not go so well because the unsigned overrides MP wouldn’t recognize the unsigned MP, only the signed MP.

So, while troubleshooting this, I was in the managementpacks table in the database and found the MPKeyToken field for my management pack was null. I tried to give it the value from the overrides MP. (update managementpack set mpkeytoken='4a6a6cefaeaaa8a0' where mpfriendlyname='My Exchange MP')

Once I did this, the console failed to open. I recycled the SDK, config and health services and I rebooted the RMS and SQL Servers with no luck. After a while on Google, an email from the domain admins complaining that they could not get into the console, and some serious thought on what I did, I decided to return the value above to null. I did that, recycled the services and it was still toast.

The error in the event log was

Event Type:    Error
Event Source:    OpsMgr SDK Client
Event Category:    None
Event ID:    27000
Date:        8/12/2009
Time:        9:27:57 AM
User:        N/A
Computer:    MOM-MGMT-04
Description:
The OpsMgr SDK Client threw an unknown exception while processing a type cache refresh notification.
The cache was not updated successfully: 
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
   at System.ThrowHelper.ThrowKeyNotFoundException()
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.EnterpriseManagement.Configuration.ManagementPackElementConstructor.CreateMultipleManagementPack(IList`1 dataAccessResult, ManagementGroup managementGroup)
   at Microsoft.EnterpriseManagement.Configuration.TypeSpaceCacheBaseWithManagementPacks.RefreshManagmentPackCache(ReadOnlyCollection`1 typeCacheResultSets)
   at Microsoft.EnterpriseManagement.Configuration.TypeSpaceCache.Refresh(Boolean forceDatabaseSync, ManualResetEvent acquireCacheUpdateLockEvent)
   at Microsoft.EnterpriseManagement.ManagementGroupInternal.RefreshTypeCacheInternal(Boolean forceDatabaseSync, ManualResetEvent acquireCacheUpdateLockEvent)
   at Microsoft.EnterpriseManagement.ManagementGroup.RefreshTypeCacheNotForce(Object eventToSignalOnLockAcquisition)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

One suggestion from Google was to restore the DB but since I did not bother to back it up and we don’t generally do nightly backups of the database in any environment other than production, a database restore was not an option.

After digging around for a while I found the ManagementPackReferences table. When I did a query in that table for all references to my Exchange MP, I found 20 records.

select * from managementpack where mpfriendlyname='My Exchange MP' gave me the management pack ID (GUID). From there I ran this query: select * from managementpackreferences where managementpackidreffedby='36AB2E36-2942-80B6-C7C2-5E229528369A' which showed me the 20 records referring to my custom Exchange MP. I deleted all 20 records and that restored my console.

Once I had the console back, I did a re-import of my custom MP just to restore any damaged or orphaned records to the database. This resolved the issue.

PLEASE UNDERSTAND! I did this without knowing the ramifications of my actions against a NON production system. I do NOT recommend you do this in production unless you have no other choice. I don’t know or pretend to understand what the ramifications were or will be. I did this based entirely on my understanding of how data SHOULD flow in a logical database design. Had Microsoft not named the tables properly or had they done something oddly in the database design, I could easily have screwed the whole thing up to where I’d have to rebuild the environment from scratch. You can do this at your OWN RISK! DO NOT BLAME ME IF THIS DESTROYS YOUR DATA IN ANY WAY!!!!

~Ron