Office 365 Hybrid Configuration Wizard for Exchange 2010 free/busy bug

Recently we used the new Office 365 Hybrid Configuration Wizard for Exchange 2010.

The Office 365 Hybrid Configuration wizard has been updated to support Exchange 2010. This new wizard comes with the following advantages:

  • An updated user experience that simplifies the hybrid configuration process
  • The error handling experience allows for simple remediation of issues, meaning you can actually read and understand the error
  • Fixes for HCW can happen quickly and are no longer tied to the on-premises product release cycle
  • Inefficient code that caused the HCW to take hours to run has been completely reworked and now you should be in and out in minutes

The on-premises Exchange 2010 environment was updated to the latest SP and CU and had one multi-role hybrid server. For the new Office 365 HCW please read this article published by the Exchange product team.

The HCW ran smoothly without big issues and completed succesfully. However when testing the created functionality we noticed free/busy didn’t work from on-prem users to cloud users. The free/busy worked like a charm vice-versa.

When troubleshooting further we ran the following two PowerShell cmd-lets to test the created OrganizationFederation on the on-premises Exchange Organization.

Get-FederationInformation -DomainName contoso.mail.onmicrosoft.com

Test-OrganizationRelationship -Identity “On-premises to O365 – 9a4a2b73-ebb9-4925-91ec-3b4dae60
6805” -UserIdentity clouduser@contoso.com 

Both cmd-lets resulted in this generic error message:

WARNING: An unexpected error has occurred and a Watson dump is being generated:

After running the Test-OrganizationRelationship cmd-let with the -Verbose parameter we noticed all steps and iterations done and exactly where the cmd-let failed. The cmd-let failed on the EWS call to our tenant at this step:

Test-OrganizationRelationship : Calling the Microsoft Exchange Autodiscover service for the remote federation information.

Further down the pipeline we noticed this explicit error message which contained valuable troubleshooting information.

Test-OrganizationRelationship : The Microsoft Exchange Autodiscover service failed to be called at ‘https://autodiscover.outlook.com/autodiscover/autodiscover.svc’ because the following error occurred:
Exception:
Microsoft.Exchange.SoapWebClient.GetFederationInformationException: Discovery for domain contoso.mail.onmicrosoft.com failed. —>
System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection
could be made because the target machine actively refused it 132.245.226.24:443
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket,
IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
— End of inner exception stack trace —

After comparing the current OrganizationRelationship (see image below)  with other clients hybrid configurations we noticed a difference on the TargetAutodiscoverEpr memberobject of the OrganizationRelationship object.

26-4-2016 10-48-56.png

At the OrganizationRelationship which was created by the new Office 365 Hybrid Configuration Wizard for Exchange 2010 the TargetAutodiscoverEpr was configured at the https://autodiscover.outlook.com/autodiscover/autodiscover.svc/WSSecurity namespace. At the other clients environment we used the built-in HCW from the Exchange 2010 EMC. And the namespace of the TargetAutodiscoverEpr was different. It had https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity configured and worked like a charm.

Now when we found the dissimilarity in both configurations we contacted Microsoft. We told Microsoft Suppoort that we used the brand-new Office 365 Hybrid Configuration Wizard for Exchange 2010 and showed them the other clients environment. Microsoft proposed to change the namespace on the environment to https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity and test free/busy calls. After changing the value to the correct namespace the free/busy availability information worked!

Microsoft is further investigating this behaviour but to me it looks a lot like a bug in the new Office 365 HCW for Exchange 2010 environments. So please be aware for this behaviour when running the newly published HCW on Exchange 2010 environments…

Cheers!

Add Office 365: Exchange Online e-mail address on every mailbox with PowerShell

When carrying out hybrid Exchange deployments to Office 365: Exchange Online one of the challenges I commonly face is the disablement of the e-mail address policy on stacks of mailboxes. To handle mailflow in a hybrid scenario every mailbox needs an e-mail routing address matching the tenant e-mail domainname (contoso.mail.onmicrosoft.com). Now this requirement is covered by running the Hybrid Configuration Wizard in stage 8: configuring mailflow. This phase simply adds the new e-mail routing address to the “Default Policy”.

This new configuration automatically adds the new e-mail address on every mailbox which has the e-mail address policy enabled on it. Now this is where the challenge appears. What to do when a lot of mailboxes within your Exchange organisation do not have the e-mail address policy enabled?

The easiest fix is enabling the e-mail address policy on every mailbox with a simple one-liner of PowerShell right? Well in a lot cases this solution doesn’t suffice because this would mess up naming conventions. A lot of IT departments knowingly disabled the policy on mailboxes over the years to configure the desired e-mail address by hand for their end-users.

To automatically configure or add the new e-mail address on every mailbox without enabling the e-mail address policy I use this great PowerShell script.

<#
.SYNOPSIS
The Add-O365MailAddress script is used to add your Office 365 e-mail address on every mailbox within your Exchange Organisation.

.DESCRIPTION
The Add-O365MailAddress script is used to add a new Office 365 e-mail address on every mailboxes where it is currently not present.

Make sure script execution is set to unrestricted by running “Set-ExecutionPolicy -ExecutionPolicy unrestricted -Force”

Please note that this script is only tested on Windows Server 2008 R2 and higher servers which have the Exchange Management Shell installed.
#>

$mailbox = Get-Mailbox -ResultSize Unlimited -Filter {EmailAddresses -notlike “*@contoso.mail.onmicrosoft.com*”}
foreach ($user in $mailbox) {

$alias=$user.alias

$email=$alias + “@contoso.mail.onmicrosoft.com”

Set-mailbox $user.Identity -EmailAddresses @{add=$email}
}

Export-Csv -Path c:\temp\mailboxes_output.csv

 

DOWNLOAD HERE

The script iterates trough every mailbox in the Exchange Organisation. Every mailbox which does not have a present Office 365 routing address (contoso.mail.onmicrosoft) is put in the pipeline. Next the script will add the new e-mail address on every selected mailbox which is required for for Office 365 hybrid mailflow.

Hybrid Configuration Wizard : Exception=The remote server returned an error: (407) Proxy Authentication Required

A while ago we were facing some issues when running the Exchange 2013 Hybrid Configuration Wizard (HCW) for Exchange Online. As it is recommended to bypass proxy servers for most of the Office 365 services. This is absolutely necessary for Exchange Online in a hybrid scenario.

When running the  HCW it actually runs a large series of PowerShell commandlets which you develop by configuring the New-HybridConfiguration cmd-let with all the required parameters. Once configured it actually goes through these eleven phases:

  1. Creation of Hybrid Configuration Object (New-HybridConfiguration)
  2. Check Tenant Prerequisites
  3. Upgrading Hybrid Configuration from Exchange 2013
  4. Check Prerequisites
  5. Configure Recipient Settings
  6. Creating Organization Relationship
  7. Configure Free/Busy Settings
  8. Configure Mail Flow
  9. Configure MRS Proxy Settings
  10. Configure IntraOrganization Connector
  11. Configure OAuth

Now when we were facing this issue we ended up getting stuck at phase 6 which is configuring the Organization Relationship. One which is rather complex. The Hybrid Configuration Wizard threw this error:

ERROR: System.Management.Automation.RemoteException: Federation information could not be received from the external organization.

ERROR : Subtask NeedsConfiguration execution failed: Configure Organization Relationship
Exchange was unable to communicate with the autodiscover endpoint for your Office 365 tenant. This is typically an outbound http access configuration issue. If you are using a proxy server for outbound communication, verify that Exchange is configured to use it via the “Get-ExchangeServer –InternetWebProxy” cmdlet. Use the “Set-ExchangeServer –InternetWebProxy” cmdlet to configure if needed.

The client did have a proxy and reverse proxy within their infrastructure and as the solution and technical design required the traffic from the Hybrid Exchange server should have direct route to the Internet, so it bypasses any proxyserver. I was able to double-check this with the Network Administrator and all was configured correctly as written in the technical design.

When we digged through the logging on the Hybrid server which is located in the D:\Program Files\Exchange Server\Logging\Update-HybridConfiguration directory. We found an error message:

Exception=The remote server returned an error: (407) Proxy Authentication Required

This error was thrown after running the Get-FederationInformation cmd-let and pointed the cause to a proxy server, or at least a proxy setting. After reading several TechNet articles we found out that the commands run by the HCW are run under the context of “Local System”. As such, these commands are subject to the proxy settings of the “Local System” user profile and not my administrator profile settings.

The default value of “Automatically Detect Settings” in the Internet Options is always “Enabled” and is configured on per unique user. So this configuration also applies to “Local System”. This default setting, combined with the client’s PAC file, the HCW was directing “Local System” to use the proxy server.

Resolution:
To fix this you have to download a tool like PsExec which can run Internet Explorer under the context of “Local System”. Once you are running IE under the local system user simply disable the setting and save changes. Run the following cmd-let:

psexec.exe -i -s -d “C:\Program Files\Internet Explorer\iexplore.exe” in CMD with administrative priviliges.

psexec_cmndlet

This workarround allowed me to bypass the proxy settings in the PAC-file the client used and succesfully complete the Exchange Hybrid Configuration Wizard. Have fun running the HCW! 🙂