We recently added a new Edge server to a pre-existing Lync 2013 Edge pool in our primary site. For purposed of this article the problem edge server is named “lyncedge2.pipe2text.com”. No matter what we tried we could not get the system to replicate. The most difficult part of trying to figure this out is that there was almost no indication in any of the Lync Server event logs that there was any issue. This article will guide you thru all the steps we went thru to try to figure out the issue.
Determining replication status
To determine what is replicating and what is not, run the following command. Scroll down to the output below for lynccedge2.pipe2text.com and you will notice the UpToDate value is False and LastStatusReport is blank
PS C:Userspipe2text> Get-CsManagementStoreReplicationStatus
UpToDate : True
ReplicaFqdn : lyncfe1.pipe2text.com
LastStatusReport : 3/18/2014 10:29:55 AM
LastUpdateCreation : 3/18/2014 10:29:49 AM
ProductVersion : 5.0.8308.0
UpToDate : True
ReplicaFqdn : lynccedge1.pipe2text.com
LastStatusReport : 3/18/2014 10:29:59 AM
LastUpdateCreation : 3/18/2014 10:29:49 AM
ProductVersion : 5.0.8308.0
UpToDate : True
ReplicaFqdn : lyncfe2.pipe2text.com
LastStatusReport : 3/18/2014 10:29:54 AM
LastUpdateCreation : 3/18/2014 10:29:49 AM
ProductVersion : 5.0.8308.0
UpToDate : False
ReplicaFqdn : lyncedge2.pipe2text.com
LastStatusReport :
LastUpdateCreation : 3/18/2014 10:29:49 AM
ProductVersion : 5.0.8308.0
UpToDate : False
ReplicaFqdn : lyncedge2.pipe2text.com
LastStatusReport :
LastUpdateCreation : 3/18/2014 10:29:49 AM
ProductVersion : 5.0.8308.0
Force Replication
Ran the following command from one of the Front End Servers
PS C:\Usersp2tuser> Invoke-CsManagementStoreReplication
We watched the eventlogs on the CMS Master server we ran this from. All event logs that related to “replication” came back clean no errors. So we ran Get-CsManagementStoreReplicationStatus again:
UpToDate : False
ReplicaFqdn : lyncedge2.pipe2text.com
LastStatusReport :
LastUpdateCreation : 3/18/2014 10:49:37 AM
ProductVersion : 5.0.8308.0
Still showing “False”
Check Topology
Went back to the Topology Builder and took note of the name that was entered when we added the server to the topology. It was spelled correctly and all lower case. Logged back into the server and checked the computer name, also all in lower case. Then verified all the FQDN’s in the certificate we applied. Again all in lower case. I stress this point because we found articles that indicated the names specified in the topology had to match the server name, even the case. Not sure it would’ve mattered in the end but everything lined up.
lyncedge2.pipe2text.com
Domain Suffix
As the Edge server is not joined to the domain a domain suffix doesn’t get applied. Went back into the Computer Name settings on the Edge Server to double check we had manually added this in. If you specify the fqdn in topology builder but the computer name is just lyncedge2 without the pipe2text.com you will have an issue. This will also cause an issue when you run “Setup or Remove Lync Server Components”
Firewall
Generally when I’ve had a Lync issue between the Front End and Edge Servers its firewall related. So we ran a test from every Front End server to lyncedge2.pipe2text.com
telnet lyncedge2.pipe2text.com 4443
Connected successfully from each Front End Server. No issues there
Services
Verified all of the services on the Front End Servers were started. Restarted the “Replicator” Services on all FE servers
Verified all of the services on the Edge Servers were started. Restarted the “Replicator” Services on all Edge servers
In addition the Edge server was rebooted. Still not working
Certificates
Verified all certificates were valid and not expired and applied correctly via the Deployment Wizard – Request, Install, or Assign Certificates console.
All certificates applied were correct and matched exactly what was installed on our other Edge server which was replicating successfully.
Re-apply and Re-install
Just for the sake of trying something we re-published the Lync Topology. It completed successfully. Then we exported the Topology from one of the FE servers and imported it into the broken Edge Server again.
The command to export the topology file from your Front End Server is the following. Change the directory to your liking.
Export-CsConfiguration -FileName c:\lynctopologyexport.zip
Then we re-ran “Setup or Remove Lync Server Components”
No Issues with the re-install
Re-ran Get-CsManagementStoreReplicationStatus and got the following:
UpToDate : False
ReplicaFqdn : lyncedge2.pipe2text.com
LastStatusReport :
LastUpdateCreation : 3/18/2014 3:45:22 PM
ProductVersion : 5.0.8308.0
Invoked replication again.
Still showing “False”
Logging
At this point everything is configured properly and has been verified. We decided to run some additional Logging/Tracing tests on the Edge server. We installed Microsoft Lync Server 2013 Debugging Tools.
http://www.microsoft.com/en-us/download/details.aspx?id=35453
Opened up the Logging Tool and selected “XDS_Replica_Replicator”, chose all for “Level” and put a check in “All Flags
Clicked on start logging and ran the Invoke-CsManagementStoreReplication command from the one of our FE servers. Waited about a minute or so and stopped the logging. Clicked on Analyze Log Files.
Typed the word certificate into the search box and got the following results.
TL_ERROR(TF_COMPONENT) [0]197C.1A7C::03/18/2014-21:36:30.903.0000000d (XDS_Replica_Replicator,ReplicationWebService.ValidateMessageSender:replicationwebservice.cs(213))(0000000000456E68)Invalid certificate presented by remote source lync1.pipe2text.com in cluster lyncpool.pipe2text.com. Details: No match.
So we’ve narrowed it down to a certificate issue as you can see from the logs. So to figure out the cause we proceeded to do the following:
-Checked the certificate on lync1.pipe2text.com and it was valid.
-Opened up the certificate stores on both systems.
-On lyncedge2.pipe2text.com we had GeoTrust certificates
-On lync1.pipe2text.com we had used Verisign certificates (hmmm)
-On lync1.pipe2text.com there were Verisign and Geotrust Intermediate and Root Certs.
-On lyncedge2.pipe2text.com there were only Geotrust Intermediate and Root Certs. No Symantec Intermediate and Root Certs
**Certificate chain cannot be verified**
-Exported the Verisign Intermediate and Root Certs certificates from lync1.pipe2text.com
-Imported them into the Intermediate and Root Cert stores on lyncedge2.pipe2text.com
-Restarted the Lync Replication services on the FE Servers and the Edge Server
– Re-ran Get-CsManagementStoreReplicationStatus and got the following:
UpToDate : True
ReplicaFqdn : lyncedge2.pipe2text.com
LastStatusReport : 3/18/2014 5:19:14 AM
LastUpdateCreation : 3/18/2014 5:19:14 AM
ProductVersion : 5.0.8308.0