How to shoot yourself in the foot with a config patch…

Sitecore config patches are great, right? We (should) all be using them to ensure that our changes in configuration don’t get stomped on when we upgrade, or install new modules. But like any bit of technology, they can sometimes cause problems. Here’s an example of one I saw recently, in the hope it can save others from similar issues:

The issue

A colleague reported that a server in a load-balanced production cluster running Sitecore 8.0 had been taken down for a scheduled reboot, but had failed to come back up afterwards. Instead of serving pages, it was now throwing 500 errors for many requests. The Windows Log was filling up with errors like this:

Exception information: 
    Exception type: InvalidOperationException 
    Exception message: Tracker.Current is not initialized
   at Sitecore.Analytics.Pipelines.StartAnalytics.StartTracking.Process(PipelineArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Mvc.Analytics.Pipelines.MvcEvents.RequestBegin.StartTracking.Process(RequestBeginArgs args)
   at (Object , Object[] )
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
   at Sitecore.Mvc.Pipelines.PipelineService.RunPipeline[TArgs](String pipelineName, TArgs args)
   at Sitecore.Mvc.Routing.RouteHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

That’s not an exception in the custom code for the website – it’s Sitecore complaining that something is wrong with the configuration for analytics. Googling for the error brings up a few hits mostly discussing how some specific configuration for “what DNS name is analytics using for this site” can cause this error.

But on the broken server, those analytics config settings were correct.

Cue head scratching…

The cause

Quite by chance, while zipping up config files to drag back to my dev machine for a big “diff all the files” exercise, the colleague who had reported the issue noticed a typo in a configuration patch file. Edited to focus on the issue, it was something like:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>

    <sites>
      <site patch:before="*[@name='website']"
          name="some-other-site"
          hostName="the-analytics-host-name"  
          -- load of other attributes we can ignore -- />
    </sits>

  </sitecore>
</configuration>

Somehow the closing tag for the custom <site/> definitions had been mucked up, making the XML in the patch file invalid. Since the host name was not correctly bound requests were falling back to a different <site/> entry, and hence the host name in the analytics wasn’t correct. And the tracker goes boom

Initially, this confused me quite a bit. Surely an invalid configuration file should prevent the site starting? I was expecting to get a 500 error for failing to parse the dodgy XML. So as a test, I applied a similar dodgy patch to a vanilla instance of Sitecore 8. I got the following in the log files:

...
 460 09:23:38 ERROR Could not load configuration file: C:\Inetpub\wwwroot\my-host-name\Website\App_Config\Include\zz_test\broken.config: System.Xml.XmlException: The 'sites' start tag on line 4 position 6 does not match the end tag of 'sits'. Line 8, position 7.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag)
   at System.Xml.XmlTextReaderImpl.ParseEndElement()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at Sitecore.Xml.Patch.XmlReaderSource.<GetChildren>d__4.MoveNext()
   at Sitecore.Xml.Patch.XmlPatchUtils.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
   at Sitecore.Xml.Patch.XmlPatchUtils.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
   at Sitecore.Xml.Patch.XmlPatchUtils.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
   at Sitecore.Configuration.ConfigPatcher.ApplyPatch(TextReader patch, String sourceName)
   at Sitecore.Configuration.ConfigPatcher.ApplyPatch(String filename)
   at Sitecore.Configuration.Factory.LoadAutoIncludeFiles(ConfigPatcher patcher, String folder)
...

So while this is recorded as an error int the log, it does not stop startup. Sitecore just ignores the dodgy config patch file and carries on regardless.

That left me wondering if this behaviour changed at some point, or whether I’ve always just had a wrong expectation of the software’s behaviour here. That’s something to do some digging about when I have some free time. So probably never then… 😉

But I think the key things to remember here are:

  • Make sure you validate that your patch files are actually valid XML, because stuff can go subtly wrong if they’re not.
  • When spelunking config on production servers, your default answer to Notepad asking “do you want to save your changes” should be “no”, because 99% of the time you really did not want to change it.
Advertisements

4 thoughts on “How to shoot yourself in the foot with a config patch…

  1. Hi Jeremy,

    Thanks for this share,

    I am having a very similar error and stack trace after deploying my Sitecore project to live. baring in mind it is fully functional on my local env. solution builds and deploys fine however when I publish the site to live I get this error on the page when trying to load it:

    Could not resolve type name: Unicorn.Remote.Processor.UnicornRemotePipelineProcessor (method: Sitecore.Configuration.Factory.CreateFromTypeName(XmlNode configNode, String[] parameters, Boolean assert)).

    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

    Exception Details: System.Exception: Could not resolve type name: Unicorn.Remote.Processor.UnicornRemotePipelineProcessor (method: Sitecore.Configuration.Factory.CreateFromTypeName(XmlNode configNode, String[] parameters, Boolean assert)).

    Source Error:

    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

    Stack Trace:
    [Exception: Could not resolve type name: Unicorn.Remote.Processor.UnicornRemotePipelineProcessor (method: Sitecore.Configuration.Factory.CreateFromTypeName(XmlNode configNode, String[] parameters, Boolean assert)).]
    Sitecore.Diagnostics.Error.Raise(String error, String method) +108
    Sitecore.Configuration.Factory.CreateType(XmlNode configNode, String[] parameters, Boolean assert) +230
    Sitecore.Configuration.Factory.CreateFromTypeName(XmlNode configNode, String[] parameters, Boolean assert) +41
    Sitecore.Configuration.Factory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper) +100
    Sitecore.Configuration.Factory.CreateObject(XmlNode configNode, Boolean assert) +39
    Sitecore.Pipelines.CorePipelineFactory.GetObjectFromType(String type, XmlNode processorNode) +66
    Sitecore.Pipelines.CorePipelineFactory.GetProcessorObject(XmlNode processorNode) +86
    Sitecore.Pipelines.CoreProcessor.GetMethod(Object[] parameters) +129
    Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args) +352
    Sitecore.Pipelines.CorePipeline.Run(String pipelineName, PipelineArgs args, String pipelineDomain, Boolean failIfNotExists) +158
    Sitecore.Nexus.Web.HttpModule.’(Object , EventArgs ) +473
    System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +141
    System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +69

    Looking in the logs it seems complaining about the file `App_Config\Include\Unicorn\Unicorn.config:` However I verified its XML and nothing seems to be wrong with it. Actually both configs on my local and on live should be identical as I disabled any transforms I have. Below is the error I see in the logs:

    5320 16:41:19 ERROR Could not load configuration file: C:\Website\App_Config\Include\Unicorn\Unicorn.config: System.InvalidOperationException: parent
    at Sitecore.Diagnostics.Assert.IsNotNull(Object value, String message)
    at Sitecore.Xml.Patch.XmlPatchHelper.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
    at Sitecore.Xml.Patch.XmlPatchHelper.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
    at Sitecore.Xml.Patch.XmlPatchHelper.MergeChildren(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns, Boolean targetWasInserted)
    at Sitecore.Xml.Patch.XmlPatchHelper.MergeNodes(XmlNode target, IXmlElement patch, XmlPatchNamespaces ns)
    at Sitecore.Configuration.ConfigPatcher.ApplyPatch(TextReader patch, String sourceName)
    at Sitecore.Configuration.ConfigPatcher.ApplyPatch(String filename)
    at Sitecore.Configuration.ConfigReader.LoadAutoIncludeFiles(ConfigPatcher patcher, String folder)

    • I have to say I’m not an expert with Unicorn. Kam Figy is the guy to go to for expertise there – since he built it 😉

      However, based on those messages, I’d suggest you try looking at the following:

      1) Do you have the same version of Sitecore on your dev and live sites? Different versions can have different config file schemas (which can break patch files) or different binaries (which can break code).
      2) Does the production server have all the binaries and config files Unicorn requires deployed to it? If something is missing it will cause errors.
      3) Could something in the deployment process have messed up the config files in some way? Accidentally replacing windows line endings with unix ones? Messing up the XML somehow as per the post above?
      4) Is there a config difference between dev and live that means the config file deployed for Unicorn isn’t valid for production? Different database names perhaps? Some missing service or settings that Unicorn relies on?

      I can’t really say whether any of those is the underlying cause, but it’s somewhere to start at least…

      • Hi Jeremy,

        Really appreciate your response above. Yes it turned out to be the 4th possibility as you explained. One of Unicorn’s config files was being deployed to Live where it shouldn’t be as it was disabled by default. problem solved when I deleted that config file namely Unicorn.Remote.Config.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s