Office 365 Hybrid Configuration Wizard for Exchange 2010 free/busy bug

Recently we used the new Office 365 Hybrid Configuration Wizard for Exchange 2010.

The Office 365 Hybrid Configuration wizard has been updated to support Exchange 2010. This new wizard comes with the following advantages:

  • An updated user experience that simplifies the hybrid configuration process
  • The error handling experience allows for simple remediation of issues, meaning you can actually read and understand the error
  • Fixes for HCW can happen quickly and are no longer tied to the on-premises product release cycle
  • Inefficient code that caused the HCW to take hours to run has been completely reworked and now you should be in and out in minutes

The on-premises Exchange 2010 environment was updated to the latest SP and CU and had one multi-role hybrid server. For the new Office 365 HCW please read this article published by the Exchange product team.

The HCW ran smoothly without big issues and completed succesfully. However when testing the created functionality we noticed free/busy didn’t work from on-prem users to cloud users. The free/busy worked like a charm vice-versa.

When troubleshooting further we ran the following two PowerShell cmd-lets to test the created OrganizationFederation on the on-premises Exchange Organization.

Get-FederationInformation -DomainName contoso.mail.onmicrosoft.com

Test-OrganizationRelationship -Identity “On-premises to O365 – 9a4a2b73-ebb9-4925-91ec-3b4dae60
6805” -UserIdentity clouduser@contoso.com 

Both cmd-lets resulted in this generic error message:

WARNING: An unexpected error has occurred and a Watson dump is being generated:

After running the Test-OrganizationRelationship cmd-let with the -Verbose parameter we noticed all steps and iterations done and exactly where the cmd-let failed. The cmd-let failed on the EWS call to our tenant at this step:

Test-OrganizationRelationship : Calling the Microsoft Exchange Autodiscover service for the remote federation information.

Further down the pipeline we noticed this explicit error message which contained valuable troubleshooting information.

Test-OrganizationRelationship : The Microsoft Exchange Autodiscover service failed to be called at ‘https://autodiscover.outlook.com/autodiscover/autodiscover.svc’ because the following error occurred:
Exception:
Microsoft.Exchange.SoapWebClient.GetFederationInformationException: Discovery for domain contoso.mail.onmicrosoft.com failed. —>
System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection
could be made because the target machine actively refused it 132.245.226.24:443
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket,
IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
— End of inner exception stack trace —

After comparing the current OrganizationRelationship (see image below)  with other clients hybrid configurations we noticed a difference on the TargetAutodiscoverEpr memberobject of the OrganizationRelationship object.

26-4-2016 10-48-56.png

At the OrganizationRelationship which was created by the new Office 365 Hybrid Configuration Wizard for Exchange 2010 the TargetAutodiscoverEpr was configured at the https://autodiscover.outlook.com/autodiscover/autodiscover.svc/WSSecurity namespace. At the other clients environment we used the built-in HCW from the Exchange 2010 EMC. And the namespace of the TargetAutodiscoverEpr was different. It had https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity configured and worked like a charm.

Now when we found the dissimilarity in both configurations we contacted Microsoft. We told Microsoft Suppoort that we used the brand-new Office 365 Hybrid Configuration Wizard for Exchange 2010 and showed them the other clients environment. Microsoft proposed to change the namespace on the environment to https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity and test free/busy calls. After changing the value to the correct namespace the free/busy availability information worked!

Microsoft is further investigating this behaviour but to me it looks a lot like a bug in the new Office 365 HCW for Exchange 2010 environments. So please be aware for this behaviour when running the newly published HCW on Exchange 2010 environments…

Cheers!

Hybrid Configuration Wizard : Exception=The remote server returned an error: (407) Proxy Authentication Required

A while ago we were facing some issues when running the Exchange 2013 Hybrid Configuration Wizard (HCW) for Exchange Online. As it is recommended to bypass proxy servers for most of the Office 365 services. This is absolutely necessary for Exchange Online in a hybrid scenario.

When running the  HCW it actually runs a large series of PowerShell commandlets which you develop by configuring the New-HybridConfiguration cmd-let with all the required parameters. Once configured it actually goes through these eleven phases:

  1. Creation of Hybrid Configuration Object (New-HybridConfiguration)
  2. Check Tenant Prerequisites
  3. Upgrading Hybrid Configuration from Exchange 2013
  4. Check Prerequisites
  5. Configure Recipient Settings
  6. Creating Organization Relationship
  7. Configure Free/Busy Settings
  8. Configure Mail Flow
  9. Configure MRS Proxy Settings
  10. Configure IntraOrganization Connector
  11. Configure OAuth

Now when we were facing this issue we ended up getting stuck at phase 6 which is configuring the Organization Relationship. One which is rather complex. The Hybrid Configuration Wizard threw this error:

ERROR: System.Management.Automation.RemoteException: Federation information could not be received from the external organization.

ERROR : Subtask NeedsConfiguration execution failed: Configure Organization Relationship
Exchange was unable to communicate with the autodiscover endpoint for your Office 365 tenant. This is typically an outbound http access configuration issue. If you are using a proxy server for outbound communication, verify that Exchange is configured to use it via the “Get-ExchangeServer –InternetWebProxy” cmdlet. Use the “Set-ExchangeServer –InternetWebProxy” cmdlet to configure if needed.

The client did have a proxy and reverse proxy within their infrastructure and as the solution and technical design required the traffic from the Hybrid Exchange server should have direct route to the Internet, so it bypasses any proxyserver. I was able to double-check this with the Network Administrator and all was configured correctly as written in the technical design.

When we digged through the logging on the Hybrid server which is located in the D:\Program Files\Exchange Server\Logging\Update-HybridConfiguration directory. We found an error message:

Exception=The remote server returned an error: (407) Proxy Authentication Required

This error was thrown after running the Get-FederationInformation cmd-let and pointed the cause to a proxy server, or at least a proxy setting. After reading several TechNet articles we found out that the commands run by the HCW are run under the context of “Local System”. As such, these commands are subject to the proxy settings of the “Local System” user profile and not my administrator profile settings.

The default value of “Automatically Detect Settings” in the Internet Options is always “Enabled” and is configured on per unique user. So this configuration also applies to “Local System”. This default setting, combined with the client’s PAC file, the HCW was directing “Local System” to use the proxy server.

Resolution:
To fix this you have to download a tool like PsExec which can run Internet Explorer under the context of “Local System”. Once you are running IE under the local system user simply disable the setting and save changes. Run the following cmd-let:

psexec.exe -i -s -d “C:\Program Files\Internet Explorer\iexplore.exe” in CMD with administrative priviliges.

psexec_cmndlet

This workarround allowed me to bypass the proxy settings in the PAC-file the client used and succesfully complete the Exchange Hybrid Configuration Wizard. Have fun running the HCW! 🙂