Remove SMTP aliases and e-mail addresses from all mailboxes on Exchange 2007 with PowerShell

When migrating your on-premises Exchange organization to Office 365: Exchange Online a common challenge is removing old and unused SMTP aliasses or e-mail addresses.

It’s rather easy to create new SMTP addresses to a mailbox or recipient in Exchange via an e-mail address policy. However it can be a real pain in the ass removing specific e-mail addresses and namespaces on mailboxes. Since it is not allowed to have unaccepted domain SMTP addresses on a mailbox prior migrating to Exchange Online this is a core requirement and activity to be carried out.

Another challenge is the incapability of the version of the Exchange 2007 Management Shell to run this special hash table syntax which is only supported for Exchange 2010 SP1 or higher. Most of the PowerShell scripts I looked for use this particular syntax, making them useless for Exchange 2007.

For this example I used the domainname contoso.com to be removed from all User and Shared Mailboxes within the Exchange organization.

$alias = Get-Mailbox -ResultSize Unlimited | Where {$_.EmailAddresses -like ‘*contoso.com’}
$alias | Select-Object -property SamAccountName,UserPrincipalName,EmailAddresses,PrimarySmtpAddress | Export-Csv “mailboxes_with_contoso.com.csv”

foreach($L in $alias)
{
$x = $L
$y = $x.EmailAddresses | where{$_.AddressString -like ‘*@contoso.com’}
$z = $y.SmtpAddress
$T = $x.UserPrincipalName
Write-host $x.EmailAddresses
Write-host “*****”
if($z){
$x.EmailAddresses -= $z
$x | Set-mailbox
Write-Host “$z is removed”
Write-Host “Check”
Write-host $x.EmailAddresses
}else{
Write-Host “$T does not have a contoso.com SMTP Address.”
}
Write-host “####################################################”
#Return
}

This script will iterate through each mailbox, and look for an address containing the @contoso.com namespace. Any matches are removed. To use this code, just replace contoso.com with the e-mail domain you want to remove. The script also contains some logging for you to digg to if you’re up for it.

In my scenario this was required for removing the unvalidated Office 365 domains prior to migrating mailboxes. When you have succesfully executed the script don’t forget to kick off a Full DirSync to make sure the changes are synced to Windows Azure Active Directory.

Best of luck to you all with carrying out succesfull Office 365 deployments. Before I publish the post, I have thank to my colleague and teammate Dev Chaudhari for working on the scripting!

SharePoint User Profile Service Application: Exception while trying to migrate account. The user does not exist or is not unique.

A while ago I was facing an issue with NetBIOS names when configuring the SharePoint Server User Profile Service Application. In this particular scenario the NetBIOS name was different from the domainname. For instance NetBIOS namespace would be CON\username and the domainname would be CONTOSO\username. The errors which were thrown on the applicationserver:

The extensible extension returned an unsupported error.
The stack trace is:

“System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.AggregateException: One or more errors occurred. —> Microsoft.Office.Server.UserProfiles.UserProfileException: Exception while trying to migrate account ‘CON\Username’ to ‘CONTOSO\Username’. —> Microsoft.SharePoint.SPException: The user does not exist or is not unique. —> System.Runtime.InteropServices.COMException: The user does not exist or is not unique.<nativehr>0x81020054</nativehr><nativestack></nativestack>
at Microsoft.SharePoint.Library.SPRequestInternalClass.SelectSitesAndUserInfoForMigration (Object& pvarContentDsns, Object& pvarContentIds, String bstrOldLogin, String bstrNewLogin, String bstrFullUserKey, Boolean bEnforceSidHistory, Guid guidSubscriptionId, String& pbstrNewLogin, Byte[]& ppsaOldSid, Byte[]& ppsaNewSid, Object& pvarSidHistory, Object& pvarSiteIds)at Microsoft.SharePoint.Library.SPRequest.SelectSitesAndUserInfoForMigration (Object& pvarContentDsns, Object& pvarContentIds, String bstrOldLogin, String bstrNewLogin, String bstrFullUserKey, Boolean bEnforceSidHistory, Guid guidSubscriptionId, String& pbstrNewLogin, Byte[]& ppsaOldSid, Byte[]& ppsaNewSid, Object& pvarSidHistory, Object& pvarSiteIds)
— End of inner exception stack trace —
at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
at Microsoft.SharePoint.Library.SPRequest.SelectSitesAndUserInfoForMigration(Object& pvarContentDsns, Object& pvarContentIds, String bstrOldLogin, String bstrNewLogin, String bstrFullUserKey, Boolean bEnforceSidHistory, Guid guidSubscriptionId, String& pbstrNewLogin, Byte[]& ppsaOldSid, Byte[]& ppsaNewSid, Object& pvarSidHistory, Object& pvarSiteIds)
at Microsoft.SharePoint.Administration.SPFarm.MigrateUserOrGroup(Guid subscriptionId, String oldLogin, String newLogin, Boolean usersOnly, Boolean enforceSidHistory)
at Microsoft.SharePoint.Administration.SPFarm.MigrateUserAccount(Guid subscriptionId, String oldLogin, String newLogin, Boolean enforceSidHistory)
at Microsoft.Office.Server.UserProfiles.UserProfile.MigrateUser(String newAccountName, Boolean fullMigration)
— End of inner exception stack trace —

The default configuration and behaviour of the User Profile Service Application does not include the NetBIOS namespace. When you would address the Service Application with PowerShell with Get-SPServiceApplication you would see the property NetBIOSDomainNamesEnabled set to 0 which is disabled.

To enable the NetBIOS namespace for SharePoint you have to recreate the Synchronization Connection to ADDS in the User Profile Service Application. This is necessary since in the creation process of this connection the Configuration Naming Context (CNC) is written to the FIM configuration and SQL database behind it.

  1. Delete the existing faulty Synchronization Connection in the User Profile Service Application.
  2. Open the SharePoint 2013 Management Shell with administrative rights.
  3. Run Get-SPServiceApplication and copy the GUID of the User Profile Service Application.
  4. Run this script in with the correct GUID.

    $UserProfile = Get-SPServiceApplication –Id <GUID>

    $UserProfile.NetBIOSDomainNamesEnabled=1

    $UserProfile.Update()

  5. Create a new Synchronization Connection with Active Directory Domain Services.
  6. Run a new and first “Start Profile Synchronization”.

Once the sync has completed eveyones SAMAccountName should be including the correct NetBIOS namespace of CON\username.

Offboarding mailboxes back to on-premises Exchange with Office 365

Most companies use the built-in migration feature in the Office 365 Portal for onboarding mailboxes to the cloud. This has been recently upgraded with alot of new features such as batches and migration end-points etc.

However a logical question is, what about migrating mailboxes back to our on-premises Exchange organisation? Creating those end-points can sometimes be difficult. To make life easier I use this simple PowerShell script. Before you can run the script you first have to connect to Remote PowerShell. For that please follow the instructions described here.

$opcred = get-credential domain\domainadmin
Get-Mailbox -Identity username@contoso.com | New-MoveRequest -OutBound -RemoteTargetDatabase ‘Database01’ -RemoteHostName ‘hybrid.contoso.com’ -RemoteCredential $opcred -TargetDeliveryDomain ‘contoso.com’

As you can see we define several parameters. Most of them should make sense to you, however I want to highlight that in a hybrid scenario you must use the database on your hybrid server. The RemoteHostName parameter is your hybrid endpoint.

Hybrid Configuration Wizard : Exception=The remote server returned an error: (407) Proxy Authentication Required

A while ago we were facing some issues when running the Exchange 2013 Hybrid Configuration Wizard (HCW) for Exchange Online. As it is recommended to bypass proxy servers for most of the Office 365 services. This is absolutely necessary for Exchange Online in a hybrid scenario.

When running the  HCW it actually runs a large series of PowerShell commandlets which you develop by configuring the New-HybridConfiguration cmd-let with all the required parameters. Once configured it actually goes through these eleven phases:

  1. Creation of Hybrid Configuration Object (New-HybridConfiguration)
  2. Check Tenant Prerequisites
  3. Upgrading Hybrid Configuration from Exchange 2013
  4. Check Prerequisites
  5. Configure Recipient Settings
  6. Creating Organization Relationship
  7. Configure Free/Busy Settings
  8. Configure Mail Flow
  9. Configure MRS Proxy Settings
  10. Configure IntraOrganization Connector
  11. Configure OAuth

Now when we were facing this issue we ended up getting stuck at phase 6 which is configuring the Organization Relationship. One which is rather complex. The Hybrid Configuration Wizard threw this error:

ERROR: System.Management.Automation.RemoteException: Federation information could not be received from the external organization.

ERROR : Subtask NeedsConfiguration execution failed: Configure Organization Relationship
Exchange was unable to communicate with the autodiscover endpoint for your Office 365 tenant. This is typically an outbound http access configuration issue. If you are using a proxy server for outbound communication, verify that Exchange is configured to use it via the “Get-ExchangeServer –InternetWebProxy” cmdlet. Use the “Set-ExchangeServer –InternetWebProxy” cmdlet to configure if needed.

The client did have a proxy and reverse proxy within their infrastructure and as the solution and technical design required the traffic from the Hybrid Exchange server should have direct route to the Internet, so it bypasses any proxyserver. I was able to double-check this with the Network Administrator and all was configured correctly as written in the technical design.

When we digged through the logging on the Hybrid server which is located in the D:\Program Files\Exchange Server\Logging\Update-HybridConfiguration directory. We found an error message:

Exception=The remote server returned an error: (407) Proxy Authentication Required

This error was thrown after running the Get-FederationInformation cmd-let and pointed the cause to a proxy server, or at least a proxy setting. After reading several TechNet articles we found out that the commands run by the HCW are run under the context of “Local System”. As such, these commands are subject to the proxy settings of the “Local System” user profile and not my administrator profile settings.

The default value of “Automatically Detect Settings” in the Internet Options is always “Enabled” and is configured on per unique user. So this configuration also applies to “Local System”. This default setting, combined with the client’s PAC file, the HCW was directing “Local System” to use the proxy server.

Resolution:
To fix this you have to download a tool like PsExec which can run Internet Explorer under the context of “Local System”. Once you are running IE under the local system user simply disable the setting and save changes. Run the following cmd-let:

psexec.exe -i -s -d “C:\Program Files\Internet Explorer\iexplore.exe” in CMD with administrative priviliges.

psexec_cmndlet

This workarround allowed me to bypass the proxy settings in the PAC-file the client used and succesfully complete the Exchange Hybrid Configuration Wizard. Have fun running the HCW! 🙂

Import distribution groups to Office 365: Exchange Online with PowerShell

Today we faced an issue where a client needed to migrate their GroupWise distribution groups to Office 365. Since there is no easy way doing this we developed a PowerShell script to automate this proces. Well, actually it are two scripts.

The scripts are divided in one, creating the distribution groups and part two is adding the members to the newly created groups. First we have to gather the input for creating the distribution groups in Office 365. For this I only used the required attributes for creating a distribution group.

$import = Import-Csv -Path “C:\temp\Create-DG.csv” -Delimiter “;”
foreach ($item in $import) {
New-DistributionGroup -Name $item.Name -DisplayName $item.DisplayName -PrimarySmtpAddress $item.PrimairyEmailAddress -Type $item.Type
Export-Csv -Path “C:\temp\New-DistributionGroup_LogFile_$(get-date -Format ddMMyyyy).csv”
}

Once the distribution groups are created we can head on adding the members to them using the PowerShell script below.

$import = Import-Csv -Path “C:\temp\Add-DGMembers.csv” -Delimiter “;”
foreach ($item in $import) {
Add-DistributionGroupMember -Identity $item.GroupName -Member $item.UPN -Verbose
Out-File -FilePath “C:\temp\Add-DGMembers_LogFile_$(get-date -Format ddMMyyyy).log”
}

As we can see both the scripts import a .csv file containing the content. The .csv files and the PowerShell scripts can be downloaded below.

DOWNLOAD HERE

Please note that the distribution groups can be either created on-premises in Active Directory and then synced to Windows Azure Active Directory (Office 365) or either directly created in Office 365 with the help of remote PowerShell. Based on your infrastructure and migration scenario it can differ which is the best way to go. Functionally there will be no difference for Exchange Online. However in some hybrid scenarios it can be best to create the distribution groups on-premises and sync them to Office 36 with the help of Azure Active Directory Sync (DirSync).

Filter OU’s to be synchronized to Office 365 with Windows Azure Active Directory Sync

When performing Office 365 deployments for most companies DirSync comes in the picture. Or as Microsoft prefers to call it, Windows Azure Active Directory Sync. Since most of the IT environments become polluted when time flies by most of my client’s prefer to perform a limited synchronization with DirSync.

As said earlier, most IT environments become polluted by time. As Active Directory Users & Computers is one of the most heavily administered you can imagine how much dirt sneaks in which makes it very interesting for businesses. Therefore Microsoft has a way to perform a limited DirSync based on OU (Organizational Unit) hierarchy and selection. For this we have to digg our way into ForeFront Identity Manager.

For this we have to open the FIM 2010 console by navigating to the following directory “C:\Program Files\Windows Azure Active Directory Sync\SYNCBUS\Synchronization Service\UIShell”.

Here you will find a executable called miisclient.exe. Run this application with “Run as administrator” priviliges. Make sure the account which initiates the request is member of the “Domain Admins” and the “Enterpirse Admins” security groups within your ADS forest. It’s also required to be a member of the local security group “FIMSyncAdmins” on the DirSync server.

Please refer to the image illustrated here for the instructions listed below.
FIM_console_instructions1
1. Click on “Management Agents” in the top ribbon.
2. Right click on the “Active Directory Connector” management agent and select “Properties”.
3. Click on “Configure Directory Partitions” on the left hand side of the menu.
4. Click on “Containers” and fill in the credentials of the local ADS service account.
5. After filling in the credentials a windows opens with the AD forest hierarchy. Here you can select the required Organizational Unit’s. Please refer to the image illustrated below.
FIM_console_instructions2

After selecting the prefered OU’s select OK and save the configuration. Next is waiting for DirSync to kick of a fresh Delta Sync. If you can’t wait then fire up PowerShell on the DirSync server and run the following two cmd-lets:

1. Import-Module DirSync
2. Start-OnlineCoexistenceSync -FullSync

Hope this helps fellow IT Professionals deploying succesfull Office 365 product. And obviously it is best to configure this filtering before Windows Azure Directory Sync synchronizes for the first time.

Convert security groups to mail-enabled and universal for Office 365 with PowerShell

When carrying out projects for Enterprise clients I commonly face challenges with companies not meeting the system requirements for Office 365. One of the most commonly seen missing requirements are on the Identity and Access Management part of Office 365.

When migrating legacy Identity and Access Management infrastructures to Office 365 you quickly bump in to Microsoft’s Active Directory Services (ADS). To migrate this service to Windows Azure Active Directory – which is part of every Office 365 license – you can use the Windows Azure Active Directory Sync tool. Or as most IT Professionals know it “DirSync”, this is a special edition of FIM.

Now back to businness. To migrate legacy security groups to Windows Azure Active Directory, for products such as Exchange Online it is a requirement to have a GroupScope of Universal.(see image below)

Get-ADGroup-GroupScope

Since most companies still use Global security groups these need to be converted. Therefore I use a PowerShell script which automates this proces. For this script to work, import the ActiveDirectory module in PowerShell or run the script with Active Directory Module for Windows PowerShell.

Clear-Host

if((Get-Module | where {$_.Name -eq “ActiveDirectory”}) -eq $null){
Import-Module ActiveDirectory
}
$scriptPath = split-path -parent $MyInvocation.MyCommand.Definition
Set-Location $scriptPath
Write-Output “Output will be stored in ” (Get-Location)

$SeaBase = “DC=corp,DC=local”
$SeaVal = “CN=Mailbox_*”
$SeaScope = “Subtree”
$GrpList = “ADSecGrp.csv”
$UniGrpList = “Uni_ADSecGrp.csv”
$strLogFile = “ErrorLog.txt”
$DomainAdmin = Get-Credential

#Search for all Groups that are of type Security and scope is Global and starts with “Mailbox_”
$SecGrps = Get-ADGroup -SearchScope $SeaScope -SearchBase $SeaBase -Filter {GroupCategory -eq “Security” -and GroupScope -eq “Global”}

foreach ($secGrp in $SecGrps) {
try {
$DN = $secGrp | Where-Object {$_.DistinguishedName -like $SeaVal}
$DN | Export-Csv $GrpList -Append
} catch {
throw
Break
}
}

(Get-Content $GrpList | Select-Object -Skip 1) | Set-Content $GrpList

Write-Output “Check $GrpList to verify all exported security Groups are of type Global”
Write-Output “Press Y to continue”
$selection = read-host
if ($selection -eq “y” -or $selection -eq “Y”){
Write-Output “$GrpList CSV File Checked….”
foreach($G in Import-Csv $GrpList){
try {
$D = $G.DistinguishedName
Get-ADGroup -Identity $G.SID
Set-ADGroup -Identity $G.SID -GroupScope Universal -Credential $DomainAdmin
} catch {
$ErrorMessage = $_.Exception.Message
Write-Output “Error converting for $D ..`n Error Message : $ErrorMessage” | Add-Content $strLogFile
Throw
Break
}
$DN = Get-ADGroup -Identity $G.SID
$DN | Export-Csv $UniGrpList -Append
}
(Get-Content $UniGrpList | Select-Object -Skip 1) | Set-Content $UniGrpList
Write-Output “Check $UniGrpList to verify all modified security Groups are of type Universal”
}else{
Write-Output “Script Stopped by User” | Add-Content $strLogFile
Break
}

As you can see the script contains several variables. With these you can define the scope of OU’s or name convention for existing security groups. When running the PowerShell script it builds up a CSV-file called Uni_ADSecGrp.csv. When paused you can open and check the file to see if it contains the groups which you wish to convert. If so, you can insert “Y” to the script and it proceeds running.

After we have succesfully changed the GroupScopes to Universal we can carry on and use the second PowerShell script which mail-enables the security groups so they meet the requirements for Exchange Online. Besides the conversion to mail-enabled it also hides the groups from the Global Address List.

Run this script on one of the legacy Exchange servers with the use of the  Exchange Management Shell.

Clear-Host

if((Get-Module | where {$_.Name -eq “ActiveDirectory”}) -eq $null){
 Import-Module ActiveDirectory
}

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010
 $env:ExchangeInstallPath\bin\RemoteExchange.ps1
Connect-ExchangeServer -auto

Write-Output “Output will be stored in ” (Get-Location)

$GrpList = “Final_ADSecGrp.csv”
$strLogFile = “enableErrorLog.txt”
$log = “AfterLog.txt”
$ErrorLog = “ErrorLog.txt”

foreach($G in Import-Csv “Uni_ADSecGrp.csv”){

try {
Get-ADGroup -Identity $G.SID

Enable-DistributionGroup -Identity $G.DistinguishedName -Alias $G.Name
Set-DistributionGroup -Identity $G.DistinguishedName -HiddenFromAddressListsEnabled $true
Get-DistributionGroup -Identity $G.DistinguishedName | Add-Content $Log
$x = Get-DistributionGroup -Identity $G.DistinguishedName
if($x -ne $Null){
Write-Output $G.DistinguishedName
}else{
Write-Output $G.DistinguishedName | Add-Content $ErrorLog
}
} catch {
$ErrorMessage = $_.Exception.Message
Write-Output “Error Enabl-DistributionGroup for $G.DistinguishedName …..`nError Message : $ErrorMessage” | Add-Content $strLogFile
throw
Break
}

}

Once you have succesfully executed the second script you can add these objects to your Windows Azure Directory Sync cycle. Please be aware that when you convert the groups, the groups may not contain unsupported characters such as namespaces or & characters.

Best of luck to you all with carrying out succesfull Office 365 deployments. Before I publish the post, I have thank to my colleague and teammate Dev Chaudhari for working on the scripting!

ULS log settings. What setting should I use?

I get this question surprisingly often: “what is the best setting for my SharePoint Farm diagnostic logging (ULS logs)?
Unfortunately; there is no best answer here because it really depends on how your farm is configured and, most importantly, how it is managed and runs.

So there isn’t an answer then?” Well.. there is, kinda, because there are guidelines! And knowing how to use these guidelines got me triggered to write up some advice.
The other day a customer called us and told us that their SharePoint environment did not work anymore. Their hosting company rebooted the server and it worked again, but could not explain the issue so they asked if we could help out. After logging in it didn’t take long to see what happened: the guidelines were not followed.

  • The ULS logs were on the system drive
  • The ULS logs were configured with Verbose logging

What we did here was move the ULS log file directory away from the system drive and configure the ULS logs to their default setting.

Basically this last one sentence should be your best practice in most SharePoint environments.
You should never have your ULS log files on your system drive. Yes, ULS is designed for knowing when a disk space issue is imminent and reduce logging when this happens, but issues can still occur in certain cases!
Furthermore never use Verbose when you are not actively troubleshooting one or more issues on a environment. If you do use it, remember to set the level back to what you need when you are done with troubleshooting.

So, back to the guidelines.
I’ve created an overview of when to use what setting. Use it the right way and you are well on your way to a properly managed SharePoint environment!

The default setting.
Use this in 90% of the SharePoint environments:

  • SharePoint 2007: STSADM -o SetLoggingLevel –Default
  • SharePoint 2010: (start the SharePoint Management Shell) Clear-SPLogLevel
  • SharePoint 2010: (start the SharePoint Management Shell) Clear-SPLogLevel

The-very-good-managed-SharePoint-environment setting (yeah, I just made that up).
Use this when you need almost no logging since you have very few to no issues most of the time:

  • SharePoint 2007: Central Admin, select “All” categories, and “Error”, “Error”.
  • SharePoint 2010: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Unexpected -EventSeverity Error
  • SharePoint 2013: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Unexpected -EventSeverity Error

The troubleshoot setting.
Use this when you need to troubleshoot an issue (remember to set the level back to what you need when you are done with troubleshooting)

  • SharePoint 2007: Open Central Admin > Operations > Diagnostic Logging. Then set ‘select a category’ to ‘All’ categories, set ‘Least critical event to report to the event log’ box value to ‘Warning’. Set ‘Least critical event to report to the trace log’ box value to ‘Verbose’.
  • SharePoint 2010: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Verbose
  • SharePoint 2013: (SharePoint Management Shell) Set-SPLogLevel -TraceSeverity Verbose -EventSeverity Verbose

There is more to this, of course, but if you obey and maintain these guidelines you can’t go wrong.

Remember that a good performing SharePoint environment starts with who manages it. So I’ll leave you with what I tell my customers who manage their SharePoint environment themselves:

don’t prepare your environment for SharePoint but make sure you are prepared for SharePoint.

Make sure you properly know how it works and users will love their environment!