Lync Phone Edition deployment – notes from the field

Lync Phone Deployment is pretty straight forward, and the mystery of DHCP options is pretty well documented on both Technet article and in Jeff Schertz’s excellent blog post on the matter. I have done a few of them, and they have all gone smoothly. Yet, a recent deployment really had me “rolling up my sleeves” to get it working. Here are some of my experiences, and what to look out for when setting it up.

I realised, by finishing up this post that is was much longer than I intended. For the impatient Reader, I will rather start by summing up a short list of findings. If you find something that might require more in-Depth Reading, be my guest and delve into the rest of the story!

Things to verify With Lync Phone Edition deployment:

1. Verify that you have a proper time Source/NTP server providing service within Your network
2. Verify that DHCP options have been set up properly, through manual Control and using Utilities provided (DHCPutil -emulateclient etc)
3. Verify DNS records needed for Devices
4. Verify that Certificate Provisioning web service is working, and not being broken by Trusted Root Cert store bug

Here are a more detailed description of the bullet Points:

Network Time Protocol (NTP)

Computer Clients within the domain will by default sync their time from a Domain Controller (DC), where the PDC Emulator is on top of the time hierarchy. Devices on the other hand is depending on you to provide a proper time Source through DHCP and/or DNS.

Jeff Schertz mentions in his blog how you make sure that the DC is actually running the w32time service, by running the following command:
w32tm /Query /configuration

My advice is; do not stop there. Make sure that other servers or computers, Lync Front End for instance, is able to sync time from the server in question. By running the command:
w32tm /stripchart /computer:<NTP server FQDN> /samples:5 /dataonly

This will make sure that firewall (port 123) is not being blocked, and that the time service is replying to requests. I encountered the 0x800705B4 error, indicating that there is no response from the server.

I would also make sure that the authorative time server is properly configured With external time Sources. While internal domain computers/servers will function just fine without, this might prove an issue With non-domain computers and Devices. In my case I discovered that the PDC Emulator did not function well at all, and rather had another DC take the role as authorative NTP peer within the domain.

Make sure that internal servers apply to this change, running:
w32tm /Query /peers

This will output a list With the active peers the server/computer is syncing With.

DCHP Scope configuration

For Devices to properly sync With the time Source (which you by now have double checked is functioning), DHCP options offered upon the Device registering on the network is crucial:

• Option 004: Time server. This should point to the NTP Source server IP.
• Option 042: NTP Servers. Provide one or more IP addresses of functional NTP time servers.

Furthermore, the Option 119 – DNS Search List – may or may not be applicable for Your environment, as explained in Jeff Schertz’ blog. Adding it will do no harm, though.

The most important part is adding Option 43 and 120. These will point to the FQDN of Your Lync Server pool, and the Certificate Provisioning service running on the Web Services pool. The configuration needed for this is generated by running the DHCPutil.exe Application, located on Lync server under %ProgramFiles%\Common Files\Microsoft Lync Server 201x\DHCPUtil.exe (x representing the difference between Lync server 2010 and 2013 respectively).

Although most documentation refers to running this command on the Lync server (when generating output for 32 bit DHCP servers), it might as well be run on any 64 bit computer in Your domain. If Your DHCP server is 64 bit, you can actually run it directly on that server. If not, take the output generated from this util and paste it as parameters for the DHCPconfigscript.bat (copied from the same path as the DHCPUtil.exe, and to the DHCP 32 bit server).

When this is Complete, make sure that all the options have been properly updated to Your DHCP Server Options section. If you had DHCP management open by the time you ran the DHCP util/script, you may have to restart it to see the changes. As these are Server options, they are also added to all subnets/Scopes you have set up. I have experienced customers trying to manually add e.g. Option 120 to their Scopes, and this will override the Server option so that it is not Distributed.

Running DCHPUtil.exe -Emulateclient from a 64 bit computer in Your domain will test the DHCP server, and verify the options returned. Make sure these are a match With the ones you have set up!

DNS records

For time sync, Devices will look for a DNS record _ntp._udp.<SIP domain> pointing to a NTP server, port 123. If this is missing, it will fallback to time.windows.com according to Microsoft docs (I have also seen time.nist.gov in WireShark logs).

Records pointing to Your Front End pool goes without saying, as these are necessary for all Clients – not only Devices.

While Devices will receive updates uploaded to the Front End pool once logged in (through in-band provisioning), a Device that is Connected to the network for the first time and has never logged in can take advantage of the ucupdates.<SIP domain> and ucupdates-r2.<SIP domain> for Lync and OCS 2007R2 respectively. This will Direct the Device to the web service offering Device updates, so it can obtain one before even having registered With the server.

Certificates

All Clients and Devices for Lync rely heavily on certificates. These are used for securing traffic (SIP/TLS signalling and SRTP media), but also for authentication. The TLS-DSK is a Client certificate allowing the Client or Device using it as a “token” for later registrations With the Lync server. As most Devices do not allow for username/password input, the phone number (or extension) in combination With the user PIN is used for both recognizing the correct user and authentication. Following this, the TLS-DSK is then sent to the Device.

I encountered a problem where the Device seemed to get all the DHCP options, as well as getting proper time sync (which can be devastating for Devices) – but still the phone would complain about wrong username/password. Running Powershell test cmdlet
Test-CsPhoneBootStrap -PhoneOrExt "12345678" -PIN "1234"
would Return a success.

Through thorough logging (luckily the customer had a switch With MONITOR port, allowing for PC With WireShark intercepting traffic) and Front End OCSlogger, we finally identified the problem being related to Trusted Root Cert error.

This is evident from the following, not so obvious SIP stack Message:

Also, reviewing Event Viewer for SChannel errors in the System log will indicate the same:

This error, although I had never seen it in conjunction With Devices before, is due to a December 2012 Windows update intended for Clients – but erroneously released for Server 2008R2 through Windows Update and WSUS.

It makes the Trusted Root store on the server contain some 350 certificates (a lot of 3rd party ones). When the Device tries to get a Client Certificate, this list of trusted certs is “too much” and the process breaks.

The fix is to disable that the long list of certs is being sent through Registry editing. Also, you might as well just remove all the unnecessary 3rd party certs in the Trusted Root store, by editing Registry as well. Although these fixes refer to the need of the Lync server being rebooted, just an IIS restart will suffice (most will hesitate to have their Lync server restarted though working hours, whereas an IIS restart is more “gentle”).

After enduring all these errors, Lync Phone Edition is finally ready to work as intended – really easy to log on to.

Hopefully you will not need to endure the same problems, although you have endured this long blog post – hopefully it will be helpful to you!

Advertisement