Monday, 25 February 2013

Citrix Storefront - Failing To Propagate Changes, Sync and Slow Consol

Storefront is now becoming more and more common and we see deployments happening now in large enterprise all the time. Once of my customers was having major issues with their roll-out last week. They were setting up HA pairs of SF servers load-balanced VIA Netscaler. Unfortunately all was not going to plan and they were seeing all types of apparently random issues including:

  • Slow Consol Response / Loading
  • Failing to Sync Changes between SF servers
  • Failing to Propagate changes between SF servers
We could see many errors in the windows logs and from the SF Trace logs we could see the following: 
Citrix.DeliveryServices.PeerResolver Error: 0 : [GetReplicationKeys] Error obtaining Key information was System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://xxxxxxsf1/Citrix/PeerResolverService/Replication that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace:
   at System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan timeout)
   at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecurityUtils.OpenCommunicationObject(ICommunicationObject obj, TimeSpan timeout)
   at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
   at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
   at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

...also we were seeing errors and popups such as:

Looking at the above we suspected some time-out  issues with the Wallet Service  or some .Net components so we Set the following on the SF servers VIA powershell to help speed up the running of our scripts:

Add-PSSnapin Citrix.DeliveryServices.Framework.Commands
Set-DSAssemblyVerification $false 

We also increased the amount of time we allow services to start using the information at to set the ServicesPipeTimeout  to 60000 (60 Seconds)

We had limited success at this point and although actions seemed to progress the whole set up was still slow and defiantly not inspiring any confidence. We also seemed to have had some limited issues with propagation.

Next I decided to focus on the slow consol loading as this would be easily diagnosed and curing the slowness here could have an impact on the other slowness we experienced. Taking a procmon of the consol launch I could see network traffic out to Verisign. Realising now that the Storefront servers had no internet access we quickly established that the checking of Citrix signed DLL's was causing huge delays.

To this end we set 2 options:
  • Disabled Certification Revokation Checking on the StoreFront Servers.
    • In Internet Explorer –> Tools –> Internet Options –> Advanced tab 
    • In the Security section, uncheck or clear the box for two options mentioned below
    • Publisher’s certificate revocation 
    • Server certificate revocation

  • Create both of the the following files:
    • C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Aspnet.config
    • C:\Windows\System32\mmc.exe.config
 Place the following inside each of them:
<?xml version="1.0" encoding="utf-8"?>
<generatePublisherEvidence enabled="false" />
With the above set the consol began to load immediately and all issues we had seen with Propagating and syncing the StoreFront servers disappeared. I would recommend that the SF servers be left with internet access (as I'd imagine they would be sitting in your DMZ anyway?) but if this is not an option, be aware of cert checking and disable it using the above. 

Please use the comments section below for any questions/queries and I'll do my best to help.

No comments:

Post a Comment