SCCM 2012 Database Replication “Link Failed”

Okay, I guess sometimes there’s a real meaning of “patch testing” before throwing it into the production or production-like environments. Microsoft patches have been mostly unproblematic but every now and then, they tend to show their effective nature. I had one similar case this past Friday when I installed “bunch of patches” in my SCCM 2012 SP1 Secondary sites and didn't bother to examine further and went off for the weekend. I know, you are probably wondering, duh! don’t you test your patches or confirm once installed, right? Well, sometimes you will lose the focus of “what to test” and want to deal with it when real problems starts and besides, this was a production-like environment (test/dev) that I could easily troubleshoot if things go south.

Come Monday morning, as part of my routine, I check the Monitoring area of the console and under Database Replication, I found the secondary sites links are failing to replicate from the parent. I look at the replication status and the last time it successfully synchronized was right before I installed and rebooted the sites for patching. Running the Replication Link Analyzer was little help as it didn't show anything major other than check the rcmctrl.log (Replication Configuration Monitor) file for related issues.

The asynchronous command finished with return message: [A .NET Framework error occurred during execution of user-defined routine or aggregate "spDRSActivation": ~~System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnection' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnectionFactory' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlPerformanceCounters' threw an exception. ---> System.MethodAccessException: Attempt by method 'System.Configuration.TypeUtil.CreateInstanceRestricted(System.Type, System.Type)' to access method 'System.Diagnostics.SwitchElementsCollection..ctor()' failed. ---> System.Security.SecurityException: Request failed.~~System.Security.SecurityException: ~~   at System.Security.CodeAccessSecurityEngine.ThrowSecurityException(RuntimeAssembly asm, PermissionSet granted, PermissionSet refused, RuntimeMethodHandleInternal rmh, SecurityAction action, Object demand, IPermission permThatFailed)~~   at System.Security.CodeAccessSecurityEngine.CheckSetHelper(PermissionSet grants, PermissionSet refused, PermissionSet demands, RuntimeMethodHandleInternal rmh, Object assemblyOrString, SecurityAction action, Boolean throwException)~~   at System.Security.PermissionListSet.CheckSetDemandWithModification(PermissionSet pset, PermissionSet& alteredDemandSet, RuntimeMethodHandleInternal rmh)~~   at System.Security.PermissionListSet.CheckSetDemand(PermissionSet pset, RuntimeMethodHandleInternal rmh)~~   at System.Security.PermissionListSet.DemandFlagsOrGrantSet(Int32 flags, PermissionSet grantSet)~~   at System.Security.CodeAccessSecurityEngine.ReflectionTargetDemandHelper(Int32 permission, PermissionSet targetGrant)~~System.TypeInitializationException: ~~   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()~~   at Microsoft.ConfigurationManager.DataReplicationService.DatabaseOperation.ExecuteNonQuery(String commandText)~~   at Microsoft.ConfigurationManager.DataReplicationService.MessageHandl...].    SMS_REPLICATION_CONFIGURATION_MONITOR    7/22/2013 9:48:20 AM    5800 (0x16A8)

So I found this error message on the rcmctrl.log file and instantly saw it was a .NET Framework related issue. It didn't take much for me to go back and look at the list of patches I installed and sure enough, 22 patches that I installed related with .NET Framework. Sigh! Which one is the culprit now? I started looking at each KB at Microsoft’s support website for their description. KB2840628 came as interesting because under its “known issues” it said -

Microsoft .NET Framework 4 applications that rely on a partial trust host may encounter errors in certain cases. The SQL Server CLR (SQLCLR) feature in Microsoft SQL Server 2012 uses such a partial trust host. (This feature allows for stored procedures to be written by using managed code.) Therefore, applications that are built by using SQL Server 2012 may be affected.

Well, SCCM 2012 SP1 secondary sites use SQL Server Express 2012 (I even upgraded them to SQL Server 2012 SP1), so this KB can be related with my issue. Uninstalled this KB from the affected machine and restarted MSSQL$CONFIGMGRSEC as well as SMS_EXECUTIVE and SMS_SITE_COMPONENT_MANAGER services. Sure enough, all things started to be working again and rcmctrl.log file didn't throw any error messages as well as Database Replication node reported as “link active” state.

Updates:

• If only I would have searched on the web, I would have probably bumped up against this thread at myITforum where Brian Mason stumbled upon the same issue. Additionally, SCCM Team Blog also has this issue reported and a workaround solution where one of them happens to be uninstalling the patch, among several. 
• I also discovered that after the KB uninstall, you just have to restart the SQL Service (MSSQL$CONFIGMGRSEC) and that should start the replication back again, so NO need to restart other services.