Hybrid Configuration Wizard : Exception=The remote server returned an error: (407) Proxy Authentication Required

A while ago we were facing some issues when running the Exchange 2013 Hybrid Configuration Wizard (HCW) for Exchange Online. As it is recommended to bypass proxy servers for most of the Office 365 services. This is absolutely necessary for Exchange Online in a hybrid scenario.

When running the  HCW it actually runs a large series of PowerShell commandlets which you develop by configuring the New-HybridConfiguration cmd-let with all the required parameters. Once configured it actually goes through these eleven phases:

  1. Creation of Hybrid Configuration Object (New-HybridConfiguration)
  2. Check Tenant Prerequisites
  3. Upgrading Hybrid Configuration from Exchange 2013
  4. Check Prerequisites
  5. Configure Recipient Settings
  6. Creating Organization Relationship
  7. Configure Free/Busy Settings
  8. Configure Mail Flow
  9. Configure MRS Proxy Settings
  10. Configure IntraOrganization Connector
  11. Configure OAuth

Now when we were facing this issue we ended up getting stuck at phase 6 which is configuring the Organization Relationship. One which is rather complex. The Hybrid Configuration Wizard threw this error:

ERROR: System.Management.Automation.RemoteException: Federation information could not be received from the external organization.

ERROR : Subtask NeedsConfiguration execution failed: Configure Organization Relationship
Exchange was unable to communicate with the autodiscover endpoint for your Office 365 tenant. This is typically an outbound http access configuration issue. If you are using a proxy server for outbound communication, verify that Exchange is configured to use it via the “Get-ExchangeServer –InternetWebProxy” cmdlet. Use the “Set-ExchangeServer –InternetWebProxy” cmdlet to configure if needed.

The client did have a proxy and reverse proxy within their infrastructure and as the solution and technical design required the traffic from the Hybrid Exchange server should have direct route to the Internet, so it bypasses any proxyserver. I was able to double-check this with the Network Administrator and all was configured correctly as written in the technical design.

When we digged through the logging on the Hybrid server which is located in the D:\Program Files\Exchange Server\Logging\Update-HybridConfiguration directory. We found an error message:

Exception=The remote server returned an error: (407) Proxy Authentication Required

This error was thrown after running the Get-FederationInformation cmd-let and pointed the cause to a proxy server, or at least a proxy setting. After reading several TechNet articles we found out that the commands run by the HCW are run under the context of “Local System”. As such, these commands are subject to the proxy settings of the “Local System” user profile and not my administrator profile settings.

The default value of “Automatically Detect Settings” in the Internet Options is always “Enabled” and is configured on per unique user. So this configuration also applies to “Local System”. This default setting, combined with the client’s PAC file, the HCW was directing “Local System” to use the proxy server.

Resolution:
To fix this you have to download a tool like PsExec which can run Internet Explorer under the context of “Local System”. Once you are running IE under the local system user simply disable the setting and save changes. Run the following cmd-let:

psexec.exe -i -s -d “C:\Program Files\Internet Explorer\iexplore.exe” in CMD with administrative priviliges.

psexec_cmndlet

This workarround allowed me to bypass the proxy settings in the PAC-file the client used and succesfully complete the Exchange Hybrid Configuration Wizard. Have fun running the HCW! 🙂

Import distribution groups to Office 365: Exchange Online with PowerShell

Today we faced an issue where a client needed to migrate their GroupWise distribution groups to Office 365. Since there is no easy way doing this we developed a PowerShell script to automate this proces. Well, actually it are two scripts.

The scripts are divided in one, creating the distribution groups and part two is adding the members to the newly created groups. First we have to gather the input for creating the distribution groups in Office 365. For this I only used the required attributes for creating a distribution group.

$import = Import-Csv -Path “C:\temp\Create-DG.csv” -Delimiter “;”
foreach ($item in $import) {
New-DistributionGroup -Name $item.Name -DisplayName $item.DisplayName -PrimarySmtpAddress $item.PrimairyEmailAddress -Type $item.Type
Export-Csv -Path “C:\temp\New-DistributionGroup_LogFile_$(get-date -Format ddMMyyyy).csv”
}

Once the distribution groups are created we can head on adding the members to them using the PowerShell script below.

$import = Import-Csv -Path “C:\temp\Add-DGMembers.csv” -Delimiter “;”
foreach ($item in $import) {
Add-DistributionGroupMember -Identity $item.GroupName -Member $item.UPN -Verbose
Out-File -FilePath “C:\temp\Add-DGMembers_LogFile_$(get-date -Format ddMMyyyy).log”
}

As we can see both the scripts import a .csv file containing the content. The .csv files and the PowerShell scripts can be downloaded below.

DOWNLOAD HERE

Please note that the distribution groups can be either created on-premises in Active Directory and then synced to Windows Azure Active Directory (Office 365) or either directly created in Office 365 with the help of remote PowerShell. Based on your infrastructure and migration scenario it can differ which is the best way to go. Functionally there will be no difference for Exchange Online. However in some hybrid scenarios it can be best to create the distribution groups on-premises and sync them to Office 36 with the help of Azure Active Directory Sync (DirSync).

Filter OU’s to be synchronized to Office 365 with Windows Azure Active Directory Sync

When performing Office 365 deployments for most companies DirSync comes in the picture. Or as Microsoft prefers to call it, Windows Azure Active Directory Sync. Since most of the IT environments become polluted when time flies by most of my client’s prefer to perform a limited synchronization with DirSync.

As said earlier, most IT environments become polluted by time. As Active Directory Users & Computers is one of the most heavily administered you can imagine how much dirt sneaks in which makes it very interesting for businesses. Therefore Microsoft has a way to perform a limited DirSync based on OU (Organizational Unit) hierarchy and selection. For this we have to digg our way into ForeFront Identity Manager.

For this we have to open the FIM 2010 console by navigating to the following directory “C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell”.

Here you will find a executable called miisclient.exe. Run this application with “Run as administrator” priviliges. Make sure the account which initiates the request is member of the “Domain Admins” and the “Enterpirse Admins” security groups within your ADS forest. It’s also required to be a member of the local security group “FIMSyncAdmins” on the DirSync server.

Please refer to the image illustrated here for the instructions listed below.
FIM_console_instructions1
1. Click on “Management Agents” in the top ribbon.
2. Right click on the “Active Directory Connector” management agent and select “Properties”.
3. Click on “Configure Directory Partitions” on the left hand side of the menu.
4. Click on “Containers” and fill in the credentials of the local ADS service account.
5. After filling in the credentials a windows opens with the AD forest hierarchy. Here you can select the required Organizational Unit’s. Please refer to the image illustrated below.
FIM_console_instructions2

After selecting the prefered OU’s select OK and save the configuration. Next is waiting for DirSync to kick of a fresh Delta Sync. If you can’t wait then fire up PowerShell on the DirSync server and run the following two cmd-lets:

1. Import-Module DirSync
2. Start-OnlineCoexistenceSync -FullSync

Hope this helps fellow IT Professionals deploying succesfull Office 365 product. And obviously it is best to configure this filtering before Windows Azure Directory Sync synchronizes for the first time.

Convert security groups to mail-enabled and universal for Office 365 with PowerShell

When carrying out projects for Enterprise clients I commonly face challenges with companies not meeting the system requirements for Office 365. One of the most commonly seen missing requirements are on the Identity and Access Management part of Office 365.

When migrating legacy Identity and Access Management infrastructures to Office 365 you quickly bump in to Microsoft’s Active Directory Services (ADS). To migrate this service to Windows Azure Active Directory – which is part of every Office 365 license – you can use the Windows Azure Active Directory Sync tool. Or as most IT Professionals know it “DirSync”, this is a special edition of FIM.

Now back to businness. To migrate legacy security groups to Windows Azure Active Directory, for products such as Exchange Online it is a requirement to have a GroupScope of Universal.(see image below)

Get-ADGroup-GroupScope

Since most companies still use Global security groups these need to be converted. Therefore I use a PowerShell script which automates this proces. For this script to work, import the ActiveDirectory module in PowerShell or run the script with Active Directory Module for Windows PowerShell.

Clear-Host

if((Get-Module | where {$_.Name -eq “ActiveDirectory”}) -eq $null){
Import-Module ActiveDirectory
}
$scriptPath = split-path -parent $MyInvocation.MyCommand.Definition
Set-Location $scriptPath
Write-Output “Output will be stored in ” (Get-Location)

$SeaBase = “DC=corp,DC=local”
$SeaVal = “CN=Mailbox_*”
$SeaScope = “Subtree”
$GrpList = “ADSecGrp.csv”
$UniGrpList = “Uni_ADSecGrp.csv”
$strLogFile = “ErrorLog.txt”
$DomainAdmin = Get-Credential

#Search for all Groups that are of type Security and scope is Global and starts with “Mailbox_”
$SecGrps = Get-ADGroup -SearchScope $SeaScope -SearchBase $SeaBase -Filter {GroupCategory -eq “Security” -and GroupScope -eq “Global”}

foreach ($secGrp in $SecGrps) {
try {
$DN = $secGrp | Where-Object {$_.DistinguishedName -like $SeaVal}
$DN | Export-Csv $GrpList -Append
} catch {
throw
Break
}
}

(Get-Content $GrpList | Select-Object -Skip 1) | Set-Content $GrpList

Write-Output “Check $GrpList to verify all exported security Groups are of type Global”
Write-Output “Press Y to continue”
$selection = read-host
if ($selection -eq “y” -or $selection -eq “Y”){
Write-Output “$GrpList CSV File Checked….”
foreach($G in Import-Csv $GrpList){
try {
$D = $G.DistinguishedName
Get-ADGroup -Identity $G.SID
Set-ADGroup -Identity $G.SID -GroupScope Universal -Credential $DomainAdmin
} catch {
$ErrorMessage = $_.Exception.Message
Write-Output “Error converting for $D ..`n Error Message : $ErrorMessage” | Add-Content $strLogFile
Throw
Break
}
$DN = Get-ADGroup -Identity $G.SID
$DN | Export-Csv $UniGrpList -Append
}
(Get-Content $UniGrpList | Select-Object -Skip 1) | Set-Content $UniGrpList
Write-Output “Check $UniGrpList to verify all modified security Groups are of type Universal”
}else{
Write-Output “Script Stopped by User” | Add-Content $strLogFile
Break
}

As you can see the script contains several variables. With these you can define the scope of OU’s or name convention for existing security groups. When running the PowerShell script it builds up a CSV-file called Uni_ADSecGrp.csv. When paused you can open and check the file to see if it contains the groups which you wish to convert. If so, you can insert “Y” to the script and it proceeds running.

After we have succesfully changed the GroupScopes to Universal we can carry on and use the second PowerShell script which mail-enables the security groups so they meet the requirements for Exchange Online. Besides the conversion to mail-enabled it also hides the groups from the Global Address List.

Run this script on one of the legacy Exchange servers with the use of the  Exchange Management Shell.

Clear-Host

if((Get-Module | where {$_.Name -eq “ActiveDirectory”}) -eq $null){
 Import-Module ActiveDirectory
}

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010
 $env:ExchangeInstallPath\bin\RemoteExchange.ps1
Connect-ExchangeServer -auto

Write-Output “Output will be stored in ” (Get-Location)

$GrpList = “Final_ADSecGrp.csv”
$strLogFile = “enableErrorLog.txt”
$log = “AfterLog.txt”
$ErrorLog = “ErrorLog.txt”

foreach($G in Import-Csv “Uni_ADSecGrp.csv”){

try {
Get-ADGroup -Identity $G.SID

Enable-DistributionGroup -Identity $G.DistinguishedName -Alias $G.Name
Set-DistributionGroup -Identity $G.DistinguishedName -HiddenFromAddressListsEnabled $true
Get-DistributionGroup -Identity $G.DistinguishedName | Add-Content $Log
$x = Get-DistributionGroup -Identity $G.DistinguishedName
if($x -ne $Null){
Write-Output $G.DistinguishedName
}else{
Write-Output $G.DistinguishedName | Add-Content $ErrorLog
}
} catch {
$ErrorMessage = $_.Exception.Message
Write-Output “Error Enabl-DistributionGroup for $G.DistinguishedName …..`nError Message : $ErrorMessage” | Add-Content $strLogFile
throw
Break
}

}

Once you have succesfully executed the second script you can add these objects to your Windows Azure Directory Sync cycle. Please be aware that when you convert the groups, the groups may not contain unsupported characters such as namespaces or & characters.

Best of luck to you all with carrying out succesfull Office 365 deployments. Before I publish the post, I have thank to my colleague and teammate Dev Chaudhari for working on the scripting!

Change explicit User Principal Names to match Office 365 domain suffix

#MICROSOFT #OFFICE365 #EMS #INTUNE #EXCHANGE #AZURE #IDENTITY

Most Enterprise Office 365 clients will use Active Directory Federation Services and Windows Azure Active Directory Sync, also known as DirSync for Single Sign-On functionality with Office 365.

Standard AD FS will use the on-premises UPN to access services in Office 365. Therefore Microsoft advises to make sure User Principal Name suffixes and e-mail adress suffixes match to make life simpler. Most ADS domains still use an internal routable domainname for instance contoso.local. However when implementing AD FS with Office 365 it permitted to have an “internet routable” domain suffix for instance contoso.com.

The best solution for this before implementing and running a DirSync is to create a new User Principal Name suffix to match your Office 365 domain suffix. This can be done by adding a new UPN suffix by going to:

  1. Open Active Directory Domains and Trusts
    2. Right click on the top level of the tree in…

View original post 116 more words

ULS log settings. What setting should I use?

I get this question surprisingly often: “what is the best setting for my SharePoint Farm diagnostic logging (ULS logs)?
Unfortunately; there is no best answer here because it really depends on how your farm is configured and, most importantly, how it is managed and runs.

So there isn’t an answer then?” Well.. there is, kinda, because there are guidelines! And knowing how to use these guidelines got me triggered to write up some advice.
The other day a customer called us and told us that their SharePoint environment did not work anymore. Their hosting company rebooted the server and it worked again, but could not explain the issue so they asked if we could help out. After logging in it didn’t take long to see what happened: the guidelines were not followed.

  • The ULS logs were on the system drive
  • The ULS logs were configured with Verbose logging

What we did here was move the ULS log file directory away from the system drive and configure the ULS logs to their default setting.

Basically this last one sentence should be your best practice in most SharePoint environments.
You should never have your ULS log files on your system drive. Yes, ULS is designed for knowing when a disk space issue is imminent and reduce logging when this happens, but issues can still occur in certain cases!
Furthermore never use Verbose when you are not actively troubleshooting one or more issues on a environment. If you do use it, remember to set the level back to what you need when you are done with troubleshooting.

So, back to the guidelines.
I’ve created an overview of when to use what setting. Use it the right way and you are well on your way to a properly managed SharePoint environment!

The default setting.
Use this in 90% of the SharePoint environments:

  • SharePoint 2007: STSADM -o SetLoggingLevel –Default
  • SharePoint 2010: (start the SharePoint Management Shell) Clear-SPLogLevel
  • SharePoint 2010: (start the SharePoint Management Shell) Clear-SPLogLevel

The-very-good-managed-SharePoint-environment setting (yeah, I just made that up).
Use this when you need almost no logging since you have very few to no issues most of the time:

  • SharePoint 2007: Central Admin, select “All” categories, and “Error”, “Error”.
  • SharePoint 2010: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Unexpected -EventSeverity Error
  • SharePoint 2013: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Unexpected -EventSeverity Error

The troubleshoot setting.
Use this when you need to troubleshoot an issue (remember to set the level back to what you need when you are done with troubleshooting)

  • SharePoint 2007: Open Central Admin > Operations > Diagnostic Logging. Then set ‘select a category’ to ‘All’ categories, set ‘Least critical event to report to the event log’ box value to ‘Warning’. Set ‘Least critical event to report to the trace log’ box value to ‘Verbose’.
  • SharePoint 2010: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Verbose
  • SharePoint 2013: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Verbose

There is more to this, of course, but if you obey and maintain these guidelines you can’t go wrong.

Remember that a good performing SharePoint environment starts with who manages it. So I’ll leave you with what I tell my customers who manage their SharePoint environment themselves:

don’t prepare your environment for SharePoint but make sure you are prepared for SharePoint.

Make sure you properly know how it works and users will love their environment!

 

Change explicit User Principal Names to match Office 365 domain suffix

Most Enterprise Office 365 clients will use Active Directory Federation Services and Windows Azure Active Directory Sync, also known as DirSync for Single Sign-On functionality with Office 365.

Standard AD FS will use the on-premises UPN to access services in Office 365. Therefore Microsoft advises to make sure User Principal Name suffixes and e-mail adress suffixes match to make life simpler. Most ADS domains still use an internal routable domainname for instance contoso.local. However when implementing AD FS with Office 365 it permitted to have an “internet routable” domain suffix for instance contoso.com.

The best solution for this before implementing and running a DirSync is to create a new User Principal Name suffix to match your Office 365 domain suffix. This can be done by adding a new UPN suffix by going to:

  1. Open Active Directory Domains and Trusts
    2. Right click on the top level of the tree in the left pane and select properties
    3. Add the new internet routable UPN suffix e.i. contoso.com

Next step is to change the explicit User Principal Names for all the designated user objects in Active Directory which you plan to synchronize to Office 365. The easiest way to do this is with the help of allmighty PowerShell.

I wrote this easy one-liner which allows you to target a specific OU in Active Directory. Don’t forget to use the Active Directory Module for PowerShell.

Get-ADUser -Filter * -SearchBase ‘OU=Users,DC=fabrikam,DC=local’ -Properties userPrincipalName | foreach { Set-ADUser $_ -UserPrincipalName “$($_.samaccountname)@contoso.com”}

After succesfully running the cmd-lets you will notice the new UPN suffix is configured for the targeted user objects in ADS.

GetSafeOrdinal FATAL ERROR Could not find column LastModifiedTime in PartitionProperties

Migrating from one SharePoint version to another can be harder then it seems at first glance. Thinking through your migration strategy and prerequisites is the first and foremost important step, but no migration is without issues, and no issue is without an error you nor Google has seen before.

In our case this issue happened when migrating from SharePoint 2010 to SharePoint 2013 and upgrading the User Profile Service Application. This caused the error:

GetSafeOrdinal: FATAL ERROR: Could not find column ‘LastModifiedTime’ in PartitionProperties. Please check if the Content and Federated Services farm are of compatible versions.

After rerouting our steps we found out that our customer had ordered and installed a SharePoint .iso in their native language and not in English, where the old SharePoint environment was in English but with a language pack installed in their native language. So where the environment seems Dutch on both versions when using Central Administration, basically these are actually two different languages. Migrating the UPS Sync database from one environment to another caused this error, but recreating the UPS Sync database (and keeping the migrated and upgraded UPS Social and UPS Profile databases) solved this error. All other databases could be migrated and upgraded but this particular one caused a little headache, but nothing is unsolvable – right? 🙂

SharePoint Online performance issue caused by deleted term store admin user objects

After working with Microsoft for over a month on a daily basis we found the cause of a serious performance issue within SharePoint Online. This issue affected several enterprise customers of ours.

This issue only occurs on sites which use the “managed navigation” functionality. In SharePoint, you can choose between two navigation methods: structural navigation or managed navigation. Structural navigation is based on site structure. Managed navigation is based on term sets. For further information read this article.

Rootcause symptoms which identify the cause:
When clients initiate a webrequest to the SharePoint Online site it takes 5 to 30 seconds untill the server returns data to the GET method. See this screenshot from Internet Exporer “Profiler”. 

IE_Profiler

Rootcause analysis:
After intensively troubleshooting several tenants we noticed strange behaviour the way the system communicates with the Managed Metadata Service Application. Since the sites are using managed navigation the content will be represented using a managed metadata term set. Because of this configuration EVERY FIRST WEBREQUEST is relies on your defined term sets and its configuration.

Solution:
Allright. Now that we have some understanding how the system relies on the Managed Metadata Service Application when using a managed navigation let’s proceed to implementing the solution.

The reason why the system “waits” for over 15 seconds on this specific part of SharePoint is caused by corrupt or missing user permissions on the database of the Service Application. As we explained “Managed navigation is using term store for its items, every time a request made to the term store, it will try to look for these users which takes some time until the process gives up.  Therefore causes the delay.”

1. Check if there are “present” Term Store Administrators defined in the Office365 portal by going to the SharePoint Admin center.
2. Select “term store” on the left hand side menu.
3. Check if you see any current users defined or SID’s i.e. s-1-5-21-2522053391-2242016340-1337919630-744690. See the image listed below.

Term_store_permissions
4. If you have identified these unknown SID’s on your environment OR identified other non system service-accounts please feel free to remove the permissions to instantly improve performance. 

Now.. if you are using AD FS and DirSync for providing single sign-on access to Office365 services then stick with me. If for instance while developing and building SharePoint solutions several users (developers or admins) have been granted the “Term Set Administrator” permission. At some point the project finishes and their account will be disabled or deleted at the on-premises Active Directory Services. Standard behaviour is that this deletion synchronizes to the cloud with DirSync. This process succesfully deletes the userobject on all front-end systems of Office365 but NOT correctly on all back-end systems in regarding to the Managed Metadata Service Application. In our case the user’s SID was still present on the Managed Metadata SQL database and therefore first webrequests got stuck on this explicit permission.

The only solution for this is contacting Microsoft Support and request them to delete the SID’s and/or old user objects on the database. Once deleted you will instantly notice great performance improvement on your environment!

Set Calendar Reviewer rights to all users in Office365

Most companies have policies which permit all users to share their calendars with everyone. There are several PowerShell scripts available online to configure this. However the tricky thing is most enterprises have different Outlook client languages. Because the Outlook client language determines the mailboxfolder attribute to it’s unique language. In this case it is set to the Dutch version “kalender” and the Engilsh version “calendar”.

  1. The fist thing which needs to be done is creating a mail-enabled universal securitygroup on-premises which contains all user objects. Make sure it syncs to Office365 with Windows Azure DirSync.
  2. Once the objects have been succesfully synchronized to the cloud run the script below to determine which language the calendar mailbox folder is:

    $GroupMembers = Get-DistributionGroupMember SG_CalendarReviewer
    $GroupMembers | foreach {
    try
    {
    $isNL = ((Get-MailboxRegionalConfiguration -Identity $_.PrimarySmtpAddress | where {$_.Language -eq “nl-NL”} | Measure).Count -eq 1)
    if ($isNL)
    {
    $identity = $_.PrimarySmtpAddress + “:\Kalender”
    }
    else
    {
    $identity = $_.PrimarySmtpAddress + “:\Calendar”
    }$permission = Get-MailboxFolderPermission -Identity $identity -ErrorAction SilentlyContinue | where {$_.User.toString() -eq “SG_CalendarReviewer”}
    if ($permission)
    {
    write-host “$identity has permission”
    }
    else
    {
    write-warning “$identity doest not have permission”
    } }
    catch
    {

    }
    }

  3. Once you have determined which language the mailboxfolder is for the calendar run this PowerShell script to grant all users in the created securitygroup with the ” Reviewer” on every mailbox belonging to each user object in the securitygroup. The PowerShell script listed below grants the access-right to all English calendars:

    $GroupMembers = Get-DistributionGroupMember SG_CalendarReviewer
    $GroupMembers | foreach {
    $identity = $_.PrimarySmtpAddress + “:\calendar”
    $permissionExists = ((Get-MailboxFolderPermission -Identity john.doe@domain.com:\calendar | where {$_.User.toString() -eq “SG_CalendarReviewer”} | measure).Count -eq 0)
    #Add-MailboxFolderPermission -Identity -User SG_CalendarReviewer -AccessRights Reviewer
    }

  4. Now change the mailboxfolderpermission attribute to the appropiate language and run the last script again.