Test Layout

When you install your first Exchange Server 2013 or Exchange Server 2016 server, a certificate with the friendly name Microsoft Exchange Server Auth Certificate is created. This certificate is self-signed and used for OAuth authentication between applications such as Exchange Server and SharePoint. However, it is also used for hybrid deployments between on-premises Exchange Server and Exchange Online. This certificate is unique because it is installed on all of your Exchange servers. The subject for the certificate is "CN=Microsoft Exchange Server Auth Certificate" and does not contain any SAN names with references to specific servers. It also has a 5-year lifetime. Which is just long enough for everyone to forget about it. I suspect that this certificate is due to expire in many organizations soon. Today I got a call from an organization with the following symptoms: Outlook clients were slow to start Outlook clients were not displaying the user's calendar Outlook clients...

A key part of most Office 365 deployments is directory synchronization with on-premises Active Directory (AD). To maintain a link between individual object in AD and Office 365, one attribute in AD is defined as the source anchor. The source anchor acts as a unique identifier for each object so that you can change properties like UPN and have them replicate to the proper matching object in Office 365. Older versions of Azure AD Connect used the objectGUID attribute as the source anchor. In many cases, this is a good choice because it is an attribute that doesn't change. However, in complex configurations with multiple AD forests, this can cause an issue during migrations. When a user account is moved from one AD forest to another, it gets a new objectGUID in the new forest. Due to the new objectGUID , Azure AD Connect treats the user as a new user instead of an existing user. To avoid this issue, we need to use an attribute for source anchor that can be moved between forests. Micro...

This has been popping up for a while, but it's worth pointing out again. When you get a SAN/UCC certificate, the DNS name that you use for the subject (common name) also needs to be in the subject alternative names attribute. For example: Subject: webmail.contoso.com Subject alternative names webmail.contoso.com exch1.contoso.com exch2.contoso.com Autodiscover.contoso.com When you get your certificate from a public certificate authority (CA), most of them ensure that the subject is also added as a subject alternative name. However, some might not and you should watch for it. It's important to note that the Microsoft CA does not automatically add the subject to the list of subject alternative names. So, make sure that you do it as part of your certificate request for your internal CA. The reason you need to do this is because of how web browsers are processing subject alternative names. Web browsers are ignoring the subject if a list of subject alternative names is present. All ...

The script that I reference in this post can be downloaded here: GoDaddyDNSUpdatePublic.ps1.txt I love the concept of using Let's Encrypt for free SSL/TLS certificates. However, the short 90-day lifetime of the certificates is designed for automated renewal. In this blog post I'm going to show the steps required to script the use of GoDaddy for DNS verification. For the basic steps on how to get a SAN certificate by using Let's Encrypt and DNS verification by using Windows PowerShell, please see my previous blog post: Using Let's Encrypt Certificates for Exchange Server Let's Encrypt requires you to create an identifier for each DNS name that you want to include on a certificate. You need to validate each identifier to prove ownership of the domain. When you are using DNS validation, you need to create a TXT record in DNS for each identifier. Unfortunately (from an ease of user perspective), the validation for an identifier is only valid for 30 days. This means, wh...

In Office 365 or Exchange Server 2013/2016, you can use the administration console to view information about migration batches. To find out information about failing moves, you can view the details of the migration batch and then view the report for individual mailboxes. When you view the report for a mailbox a text file is downloaded for viewing. The report provides detailed information about how much data has been downloaded. Also, if there are errors, they are contained in the report. Unfortunately sometimes the errors are pretty generic. For example, one error I got recently was: Transient error TimeoutErrorTransientException has occurred. The system will retry (200/1300). Instructions on how to review the report: https://technet.microsoft.com/en-us/library/jj898491(v=exchg.150).aspx Since the error was happening often, we needed to get more information. Fortunately that detail is available, but not in that report. Instead, you need to use Windows PowerShell to view the move reques...

Had a call about an older Exchange 2010 server this morning. Users were having problems working with Outlook Web App. I saw a high volume of errors in that Application event log. This error was referring to both autodiscover and '/EWS/Exchange.asmx' and was appearing multiple times per minute. Event ID: 3 Source: System.ServiceModel 3.0.0.0 Category: WebHost WebHost failed to process a request.  Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/32001227  Exception: System.ServiceModel.ServiceActivationException: The service '/Autodiscover/autodiscover.xml' cannot be activated due to an exception during compilation.  The exception message is: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection. Parameter name: item. ---> System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in th...

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful. Error #1 WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened: WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long. My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after. Error #2 WARN...

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4. Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term). Here is the process we saw: Message requiring moderation sent. Approval request sent to moderator. Moderator approves request Approval request sent to moderator Moderator approves request Repeat request and approval process a few more or a lot more times. This process was happening even though we had not moved any mailboxes to Exchange 2016 yet.  When searching, there were very few references to this issue on the Internet or support foru...

Many organizations have configured a dynamic distribution group that includes all users with mailboxes. This Everyone group is used for sending out company notifications and the like. Because this group is dynamic, it's automatically updated as new mailboxes are added and removed. When you implement hybrid mode with Office 365, the dynamic group will include on-premises mailboxes but not cloud mailboxes. This is because cloud mailboxes are a different recipient type that is not included. Cloud mailboxes are MailUser recipients (an AD user with an email address but no mailbox). One way to fix this is to modify the dynamic distribution group to include MailUser recipients. The screenshot below has enabled Users with external e-mail addresses. This will include users with cloud mailboxes. This is the recipient filter for the dynamic distribution group: ((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(Name -like 'SystemMailbox{*...

The title for this post is a bit incomplete because there wasn't enough space for a full explanation. Here is the scenario I had... Before setting up hybrid mode for an Exchange 2010 organization, a user account was created in Office 365 with the same UPN as a user account in on-premises. The on-premises domain had been added to Office 365, but directory synchronization was not yet in place.  As part of testing, the user account had been give an Office 365 license, which then created an online Exchange mailbox. When directory synchronization was configured, the on-premises AD account was matched with the existing cloud user. However, the on-premises user has an on-premises mailbox and the cloud user has a cloud mailbox. When viewing the cloud user in Office 365, you could see the mailbox in the cloud. Because there was an existing cloud mailbox I couldn't move the on-premises mailbox up to Office 365. There was also no way to remove the cloud mailbox to allow the on-premises ma...

When you configure hybrid mode, calendar sharing is automatically configured between on-premises and cloud users. You don't need to configure anything. However, in some cases, you may need to perform an IISreset. On a recent project implementing implementing hybrid mode for an Exchange 2010 organization, we enabled hybrid mode and tested connectivity. Everything looked good for mail flow and mailbox moves. However, free/busy lookups failed from the cloud users to on-premises. When we tried to view the on-premises users' calendars the status was "No Information". When you're not sure what's going on, it's always a good idea to use the Remote Connectivity Analyzer to test things out. There is an Office 365 tab which includes a Free/Busy test. When I ran this test, it was successful, as shown in the figure below. However it still wasn't working. It's been a while since I've implemented Hybrid for Exchange 2010. With a quick search, I found this b...

Today I made the mistake of downloading and attempting to run the Office 365 Hybrid Deployment Wizard by using Firefox. When you do, the app starts to launch and the fails. When you click on the Details button to open the log file, you see this under the error summary: Deployment and application do not have matching security zones. Download the app from within IE and no problems. Link to download the wizard: https://aka.ms/HybridWizard UPDATE: Also worth noting that a similar error with the wizard not running occurs if the .application file extension is not associated with Internet Explorer. https://paulrobichaux.com/2017/06/30/office-365-hybrid-configuration-wizard-wont-launch/ 

When planning an Office 365 implementation, it is best practice to start by assuming that UPN for signing in to Office 365 should match the user email address. If you don't configure it this way, then users have two separate items (their UPN for signing in and their email address) that look very similar. In many cases users are confused by the similarity. If you are synchronizing  your on-premises Active Directory with Office 365 (in most cases you do) then you need to set the UPN for the on-premises user accounts with the correct UPN. The UPN from on-premises user accounts is synchronized to Office 365 to create the ID for signing in. Most organizations are not using the UPN on user accounts for authentication on-premises. The option has been there since Windows 2000, but most organizations still use the domainname\username format for authentication. However, you need to verify if any user accounts are using the UPN for authentication before making this change. At minimum, you sho...

When managing email addresses and domains in Exchange Server, old email addresses are never removed automatically. This is good because it ensures that email addresses on a mailbox are never accidentally lost. However, you may want to clean up old domains or address formats that are no longer in use. Some common scenarios where you might want to remove an old domain: An SMB deployment of Exchange Server where a .local domain was added as the first domain for email addresses. Old GroupWise addresses are left in place from an older migration. Obsolete domain left over from a company merger many years ago I often find that obsolete domains are identified when I run IDFix as part of preparing to migrate to Office 365. To simplify the removal of obsolete domains, I have created the following script. A few things to note: You need to set $RemovePattern to identify the domain to be removed. Any email addresses matching this pattern will be removed from proxyAddresses attribute in Active Direc...

Ran into a new issue yesterday related to installing Exchange 2010 SP3. I was called in to help when the initial upgrade attempt failed. The error during SP3 installation was: An unexpected error occured while modifying the forms authentication settings for path /LM/W3SVC/1. The error returned was 5506. A screenshot of the error is below: Doing a search didn't come up with much, but it did give this: https://social.technet.microsoft.com/Forums/exchange/en-US/32df8c03-1957-4e8c-8381-4dc4cf26aee9/error-when-installing-exchange-server-2010?forum=exchangesvrgenerallegacy That link seemed to indicate that it could be related to the SSL binding on the default web site in IIS. Taking a look at the SSL binding, it seemed to be missing the certificate assignment. However, when I tried to add the certificate I got a strange error about the session be closed. Ok then, since you won't let me add the SSL certificate to the binding by using IIS Manager, let's try with Exchange Admin Cons...

I haven't run into this yet, but it appears that in some cases, Hyper-V virtual machines running Exchange Server will hang when installing updates. Specifically this seems to occur when running updates for Hyper-V integration services. There are reports of KB3037623 specifically causing this issue. KB3037623 The fix is to: Disable the Exchange services Apply the update Reenable the Exchange services This blog posting provides detailed steps: http://blog.scng.si/exchange-server-vm-becomes-unresponsive-while-updating-hyper-v-integration-services/

Just recently ran into another issue related to certificate revocation list (CRL) verification. This time it was an Exchange 2010 organization that had been fine when we initially installed the certificates, but now in the Exchange Management Console was showing the currently assigned certificate with a red X and an error message indicating that the CRL for the certificate could not be verified. The certificate was still valid, but the Exchange server couldn't verify that it hadn't been revoked. No clients were affected by this issue. Viewing the certificate on a client accessing OWA showed as valid. Like many organizations, this organization has a proxy between the internal network and the Internet. For the Exchange Servers to verify the CRL, they need to download it from the source specified in the certificate. This had been working, so, what changed? It turns out that as part of troubleshooting connectivity to WSUS from the Exchange servers, the proxy configuration was remov...

I recently ran into an issue where a mailbox database content index was in a failed state after the Exchange 2010 server had a power outage. Not too big of a deal since it was in a remote site that was hosting only passive databases for disaster recovery. Normally for a passive copy, you run the following command and all is good: Update-MailboxDatabaseCopy -Identity DBName\ServerName -CatalogOnly However I got this error: A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while updating the search catalog files from server 'XXXXXXX'. Error: A transient exception from Exchange Search was encountered. Error: Catalog was not paused for indexing on database dbGUID . There were only a couple of search hits for this specific error about pausing indexing and they really only indicated it was an internal process as part of the catalog reseed. There were no useful errors in the event log. I tried a few different variati...