Exchange 2016 CU20 breaks Event logs

I don't post often, but when I have something that I need to get out there with regards to my subject matter, this is where I post it. In this instance it relates to Exchange 2016 CU20 and a little known bug.

I found this quite by accident, many months after successful installation of CU20. I encountered no issues at all as it went on, but when I needed to find something in my event logs, it transpired that they had stopped being populated from the point of CU20 installation.

The log I am referring to is known as a crimson log. When you open Event Viewer and go to Applications and Services Logs and expand that section - these are crimson logs. The one I refer to in this post is the MSExchange Management log.

This log was broken and not just on one server. When an analysis was done, it had broken on 10% of my servers (4 out of 40) - which meant that when MS Premier Support tried to replicate the issue, they were able to do so quite easily.

To cut to the chase: the log is not actually broken. Rather, entries are being diverted into the Application Log instead (where they are easily lost due to the noise). The reason for this is that during CU20, setup may incorrectly write a registry key which takes precedence as the registry is read from top to bottom and the new key appears before the crimson log entries. The key is:

HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\MSExchange CmdletLogs\

It should not be there. Because it is, entries from source MSExchange CmdletLogs are routed to the Application Log instead of the MSExchange Management crimson log, which basically goes dormant. Microsoft don't have this listed on their database of eff-ups, but that may be because (like me) nobody looks at this log until they actually need to.

The fix is to remove the invalid key and restart the server.

To identify the presence of the key on your Exchange servers, you can use the following EMS:

Get-ExchangeServer | %{$s = New-PSSession -ComputerName $_.name; Invoke-Command -Session $s{if(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\MSExchange CmdletLogs\' -ErrorAction SilentlyContinue){$env:computername}}; remove-pssession $s}

If it finds it, you'll get the server NetBIOS name listed.

The following PowerShell will remove the key on a local server:

Remove-item 'HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\MSExchange CmdletLogs\' -Force -Recurse

That could be compiled into a remote PowerShell command, but I like to mitigate risk of trashing everything by taking such actions on a local server, which needs to be paced into Exchange Management Mode and restarted anyway (the key being in CurrentControlSet means that it only takes effect when the Registry hive is read i.e. on startup).

After a restart, the original command should create an error which is logged to the Event log you just fixed:

Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\EventLog\Application\MSExchange CmdletLogs\' 

I don't know if this affects any other CUs on other Exchange versions as I am only currently working with Exchange 2016. I have implemented two further CUs since then and not experienced the same issue.

Y2K22

Tasked with recovering from the issue discovered in https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447 (Email Stuck in Exchange On-premises Transport Queues) on a per server basis, I came up with the following PowerShell.

It's all well and good to provide an automated method, but most of us can't blindly run such scripts and hope that everything goes okay. It didn't go okay in the first place, hence why the issue needs fixing.

Some background:

A break/fix was put in place in conjunction with MS Premier Support:

Get-MalwareFilteringServer | Set-MalwareFilteringServer -BypassFiltering $true

Get-ExchangeServer | where { ($_.IsHubTransportServer -eq "true")} | ForEach{ Invoke-Command -ComputerName $_.Name -ScriptBlock { Restart-Service msexchangetransport } }

I wasn't involved in that (I was busy seeing in the New Year), but I get the point. The Malware Filtering was bypassed as it is only one of many layers of protection in this particular environment and the least important one. It's possibly the same elsewhere for other on-premises Exchange customers.

My mission (and I chose to accept it, as that is what I get paid for), was to fix the underlying issue before re-enabling the Malware Filtering. I don't normally have a need to do anything with it, but quickly observed that it is a bit 'laggy' when commands are run with it. So I wrote the following, which uses the manual rectification method in a set of commands that I can easily interact with and adjust if necessary. They are working well for me now, so I am finding that I can run them all in one hit and just observe. The updating takes at least half an hour per server and the use of timestamps is because I use a start-transcript for all my PS command windows.

Run from an elevated PowerShell prompt.

get-date -f HH:mm

$serverfqdn = ([System.Net.Dns]::GetHostByName($env:computerName)).HostName

. $env:ExchangeInstallPath\bin\RemoteExchange.ps1

Connect-ExchangeServer -auto -AllowClobber

$currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())

if($currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) -eq $FALSE){write-host -backgroundcolor yellow -foregroundcolor red "PLEASE RE-RUN AS ADMINISTRATOR"; EXIT}

$p = Get-Process -Name updateservice -ErrorAction SilentlyContinue

if($p){Stop-Process $p}

$path1 = $exinstall + "FIP-FS\Data\Engines\amd64\Microsoft"

$path2 = $exinstall + "FIP-FS\Data\Engines\metadata"

Remove-Item $path1 -Recurse -Force

Remove-Item $path2 -Recurse -Force

New-Item $path2 -Type Directory | out-null

get-service MSExchangeTransport | stop-service

get-Service FMS | start-service

get-service MSExchangeTransport | start-service

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

cd $exscripts

$origcheck = (Get-EngineUpdateInformation).LastChecked

Get-EngineUpdateInformation

.\Update-MalwareFilteringServer.ps1 $serverfqdn

get-date -f HH:mm

while($origcheck -eq (Get-EngineUpdateInformation).LastChecked){sleep 15}

(Get-EngineUpdateInformation).UpdateStatus

get-date -f HH:mm

while((Get-EngineUpdateInformation).UpdateStatus -eq "UpdateInProgress"){sleep 60}; Get-EngineUpdateInformation

get-date -f HH:mm

There are a few things to note in this script.

1. A standard check that it is being run at an elevated prompt or stop you in your tracks

2. $exinstall allows for non-standard installation path. It is quite common

3. The while statements allow for the Malware update process to kick in. If it doesn't, you may have a proxy issue. Refer to the Exchange blog article to help you out on that one

4. Updating isn't quick. It takes between 30 and 60 minutes for me. But so far (touch wood) it has been successful every time.

5. This is just fixing the stuff under the cover. I will still need to re-enable the Malware Filtering service by setting the bypassfiltering to $false later

AMSI - Antimalware Scan Interface

Microsoft Exchange Server 2016 now supports integration with Windows Antimalware Scan Interface (AMSI). This feature enables an AMSI-capable antivirus or antimalware solution to scan content in HTTP requests that're sent to the Exchange Server. Additionally, it will block a malicious request before it's handled by Exchange. This was introduced with CU21 (also with Exchange Server 2019 CU10, or higher).

More information about AMSI is available at:

https://techcommunity.microsoft.com/t5/exchange-team-blog/more-about-amsi-integration-with-exchange-server/ba-p/2572371

It has been reported that in some environments users experience significant degradation in Outlook performance. I have only seen reports relating to third party Anti Virus products.

Whilst the feature is desirable to have, it may be that it has to be disabled until the performance issues are overcome.

The feature writes into AD at the following location:

CN=Setting Overrides, CN=Global Settings, CN=[your Exchange Org.], CN=Microsoft Exchange, CN=Services, CN=Configuration, DC=[your AD domain], DC=[your AD domain suffix]

So in ADSIEDIT:

Configuration > Services > Microsoft Exchange > [your Exchange Org.] > Global Settings > Setting Overrides

The override you can create can be a global or local (server) override. By not using the Server attribute (and therefore not defining it), you will be creating a Global override that applies to all applicable Exchange servers.

To create a Server Override for Server1 (needs to be performed on an upgraded server, e.g. Server1):

New-SettingOverride -Name "DisablingAMSIScan" -Component Cafe -Section HttpRequestFiltering -Parameters ("Enabled=False") -Reason "Testing" -Server Server1

Next, on Server1:

Get-ExchangeDiagnosticInfo -Process Microsoft.Exchange.Directory.TopologyService -Component VariantConfiguration -Argument Refresh

Finally (also on Server 1):

Restart-Service -Name W3SVC, WAS -Force

If you didn't set it as a local override in the first instance, you need to use the Add command and you need to specify the servers it applies to.

Add-SettingOverride DisablingAMSIScan -Server Server1, Server2

Once you have populated the servers attribute, you need to use the Set command to modify it. (I tried appending the attribute, unsuccessfully - as the property type is affected). So adding Server3:

Set-SettingOverride DisablingAMSIScan -Server Server1, Server2, Server3

Obviously, the post-config part could be done with remote PS:

$servers = @("Server1", "Server2", "Server3")

Foreach($server in $servers){$s = new-pssession -computername $server; invoke-command -session $s{Get-ExchangeDiagnosticInfo -Process Microsoft.Exchange.Directory.TopologyService -Component VariantConfiguration -Argument Refresh; Restart-Service -Name W3SVC, WAS -Force};Exit-PSSession}

Removing servers from the override is removing them from the defined servers and then running the recycling commands again. So to remove Server3:

Set-SettingOverride DisablingAMSIScan -Server Server1, Server2

Once again, the post-config part could be done with remote PS:

$servers = @("Server3")

Foreach($server in $servers){$s = new-pssession -computername $server; invoke-command -session $s{Get-ExchangeDiagnosticInfo -Process Microsoft.Exchange.Directory.TopologyService -Component VariantConfiguration -Argument Refresh; Restart-Service -Name W3SVC, WAS -Force};Exit-PSSession}

To remove the override completely, use the Remove-SettingOverride command.

Exchange Transport Rules and Regular expressions

 I haven't posted anything in a long time and I have chosen not to go back over what has already been posted to see if it is still valid. I expect readers to have a level of understanding and knowledge to know how to interpret what has been posted and determine if it is relevant. I use blogs like these just to point me in the right direction, which brings me to an item that Microsoft had published for Exchange 2010, but the link is now defunct (after they moved their library) and I couldn't find it anywhere else, except on another blog. So this is to bolster that blog in case it too vanishes.

When using PowerShell / Exchange Management Shell, certain characters are interpreted as scripting language (e.g. the $ symbol). When writing something to the screen for example, when a dollar sign is encountered, PS is looking for a variable. You can overcome this using the tilde character (`) immediately before any character you want to be displayed literally (it is located to the left of number one across the top of most keyboards). So, to get the following to display the way you intend it to:

write-host "The contents of $variable are"$variable

you would write:

write-host "The contents of `$variable are"$variable

It's a subtle difference. If you wanted to display a tilde, you'd put a tilde in front of it.

The same principle is not used in Transport Rules. When you want to match criteria in a rule, you need to use a different approach. Below is a copy of the content originally posted by Microsoft. It was in relation to Exchange 2010, but is still relevant in exchange 2016 at least. Note that the criteria you filter for is not case-sensitive so looking for 'External' is the same as 'external' and 'EXTERNAL' and even 'ExTeRnAL'.

Here is an example to match only 'External' encased in brackets and not 'external' elsewhere in the field being searched:

\(External\)

Pattern matching in Exchange Transport Rules

Pattern string	Description
\S	The \S pattern string matches any single character that's not a space.
\s	The \s pattern string matches any single white-space character.
\D	The \D pattern string matches any non-numeric digit.
\d	The \d pattern string matches any single numeric digit.
\w	The \w pattern string matches any single Unicode character categorized as a letter or decimal digit.
\W	The \W pattern string matches any single Unicode character not categorized as a letter or a decimal digit.
|	The pipe ( | ) character performs an OR function.
*	The asterisk ( * ) character matches zero or more instances of the previous character. For example, ab*c matches the following strings: ac, abc, abbbbc.
( )	Parentheses act as grouping delimiters. For example, a(bc)* matches the following strings: a, abc, abcbc, abcbcbc, and so on.
\	A backslash is used as an escaping character before a special character. Special characters are characters used in pattern strings: Backslash ( \ ) Pipe ( | ) Asterisk ( * ) Opening parenthesis ( ( ) Closing parenthesis ( ) ) Caret ( ^ ) Dollar sign ( $ ) For example, if you want to match a string that contains (525), you would type \(525\).
^	The caret ( ^ ) character indicates that the pattern string that follows the caret must exist at the start of the text string being matched. For example, ^fred@contoso matches fred@contoso.com and fred@contoso.co.uk but not alfred@contoso.com.
$	The dollar sign ( $ ) character indicates that the preceding pattern string must exist at the end of the text string being matched. For example, contoso.com$ matches adam@contoso.com and kim@research.contoso.com, but doesn't match kim@contoso.com.au

Snippets

 A few PowerShell snippets I use a bit...

# Show Services that are set to Automatic that are not running:

Get-WmiObject win32_service |?{ $_.StartMode -eq 'Auto' -and $_.State -ne 'Running'} | ft -auto

 # and a loop to start them:

while((Get-WmiObject win32_service |?{ $_.StartMode -eq 'Auto' -and $_.State -ne 'Running' }) -ne $null){Get-WmiObject win32_service |?{ $_.StartMode -eq 'Auto' -and $_.State -ne 'Running' } | %{start-service $_.name}}

 # check for missing updates pushed by SCCM (null return means no updates pending):

Get-WMIObject -Namespace root\CCM\ClientSDK -Class CCM_SoftwareUpdate -Filter ComplianceState=0

 # and install them and reboot:

$MissingUpdates = Get-WMIObject -Namespace root\CCM\ClientSDK -Class CCM_SoftwareUpdate -Filter ComplianceState=0

$MissingUpdatesReformatted = @($MissingUpdates | Foreach-Object {if($_.ComplianceState -eq 0){[WMI]$_.__PATH}})

$InstallReturn = Invoke-WMIMethod -Namespace root\CCM\ClientSDK –Class CCM_SoftwareUpdatesManager -Name InstallUpdates –ArgumentList (,$MissingUpdatesReformatted)

while((Get-WMIObject -Namespace root\CCM\ClientSDK -Class CCM_SoftwareUpdate -Filter ComplianceState=0 | ?{$_.EvaluationState -lt 8}) -ne $null){write-progress -Activity "Installing updates…" -Status "The server will reboot automatically when all updates have finished installing"; Sleep 60}

shutdown /r /t 60 /f

 # get mail users from AD:

Ipmo activedirectory

$mailusers = get-aduser -filter * -Properties mail,mailnickname,msExchRecipientTypeDetails

$mailusers.count

$path = $env:userprofile + "\Documents\"

$file1 = "mailusers.csv"

$file2 = "mailboxes.csv"

$mailusers | export-csv $path$file1 -NoTypeInformation

 # and mailboxes:

$mailusers | ?{$_.msExchRecipientTypeDetails -eq 1} | export-csv $path$file2 -NoTypeInformation

 # last boot

Gwmi win32_operatingsystem | select @{L=’Boot’;E={$_.ConverttoDateTime($_.lastbootuptime)}}

 

Restart Event IDs

I can never remember what these Event IDs are, so it's time to record them here.

Event ID	Description
1074	Logged when an app (ex: Windows Update) causes the system to restart, or when a user initiates a restart or shutdown.
6006	Logged when an app (ex: Windows Update) causes the system to restart, or when a user initiates a restart or shutdown.
6008	Logged as a dirty shutdown. It gives the message "The previous system shutdown at time on date was unexpected".

Here's an article elaborating on this:

https://www.tenforums.com/tutorials/78335-read-shutdown-logs-event-viewer-windows.html

To PowerShell it (https://devblogs.microsoft.com/scripting/powertip-get-the-last-boot-time-with-powershell/):

PS2.0:

Get-WmiObject win32_operatingsystem | select csname, @{LABEL=’LastBootUpTime’;EXPRESSION={$_.ConverttoDateTime($_.lastbootuptime)}}

PS3.0

Get-CimInstance -ClassName win32_operatingsystem | select csname, lastbootuptime

EWS & PowerShell

A bit of stuff using the EWS managed API, which can be downloaded here:
http://www.microsoft.com/en-us/download/details.aspx?id=35371

Note: Make sure the path to the EWS dll matches the install location of the EWS managed API. Change it if necessary.

This script searches folders in mailboxes under the Contacts folder, looking for 'Skype for Business contacts' and then deletes it. It is also doing a disable-ummailbox. As both actions are quite destructive, there's a warning for the person running the script.

The list of mailboxes to run this against was gathered with a get-aduser searchbase of specific OUs, extracting the Primary SMTP address and outputting that to a text file.


$inputfile = "mailboxlist.txt"

$MailboxList = get-content $inputfile
$MailboxListcount = $MailboxList.count

$CurrentDate = (Get-Date).ToString('MMM-dd-yyyy')

[string]$logfilepath = "SkypeForBus-DeletionsLog_" + $CurrentDate + ".csv"

Write-host -backgroundcolor yellow -foregroundcolor red "WARNING! This script will perform the following:"`n
write-host "Using the file $inputfile containing $MailboxListcount users"`n
write-host "Action 1: Disable-UMMailbox -Identity `$MailboxName -KeepProperties:`$True -Confirm:`$false"
write-host "Action 2: Delete the Skype for Business contacts folder (if found) under the Contacts folder"`n

$verify = Read-Host "If you wish to proceed, type Y"

if($verify -eq "Y"){

$dllpath = "C:\Program Files\Microsoft\Exchange\Web Services\2.0\Microsoft.Exchange.WebServices.dll"
    [void][Reflection.Assembly]::LoadFile($dllpath)
    $Service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP1)

$FoundAmount = 0
$log = @()
$T1 = "Processing "
$T2 = " ("
$T3 = " of "
$T4 = ")"
$counter = 0
$MailboxListCount = $MailboxList.count

$StartTime = Get-Date

Foreach ($MailboxName in $MailboxList)
{
$counter++

$Activity = $T1+$MailboxName+$T2+$counter+$T3+$MailboxListCount+$T4

$percent = (($counter/$MailboxListCount) * 100)

$percentstatus = [String]([System.Math]::Round($percent, 0)) + "`% complete"

Write-Progress -id 1 -PercentComplete $percent -Activity $activity -Status $percentstatus

$FoundFolder = $FALSE
$i = New-Object -TypeName PSObject
$i | Add-Member NoteProperty Email $MailboxName
if(Get-UMMailbox $MailboxName){$UMdisabled = "True"
    Disable-UMMailbox -Identity $MailboxName -KeepProperties:$True -Confirm:$false
$i | Add-Member NoteProperty UMdisabled "True"
}
else{
$i | Add-Member NoteProperty UMdisabled "NA"
}

#    $Service.AutodiscoverUrl($MailboxName,{$true})
    $service.Url= new-object Uri("https://outlook.uk2.practice.linklaters.net/EWS/Exchange.asmx")
    $RootFolderID = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Contacts,$MailboxName)
    $RootFolder = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($Service,$RootFolderID)
    $FolderView = New-Object Microsoft.Exchange.WebServices.Data.FolderView(100,0)
    $FolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Shallow
    $Response = $RootFolder.FindFolders($FolderView)
    

    ForEach ($Folder in $Response.Folders){
            if($folder.DisplayName -eq "Skype for Business contacts"){
        ++$FoundAmount
$FoundFolder = $True
#$folder.delete([Microsoft.Exchange.WebServices.Data.DeleteMode]::SoftDelete)
     #$folder.Empty([Microsoft.Exchange.WebServices.Data.DeleteMode]::SoftDelete, $true)
$i | Add-Member NoteProperty Folder $folder.DisplayName
$i | Add-Member NoteProperty Action "Deleted"
}
    }
    if($FoundFolder -eq $false){
$i | Add-Member NoteProperty Folder "NotFound"
$i | Add-Member NoteProperty Action "None"
        }
$log += $i
}

$log | export-csv $logfilepath -notypeinformation -append

$StopTime = Get-Date
$TimeDiff = New-TimeSpan -Start $StartTime -End $StopTime
Write-Host "Deleted"$FoundAmount "folders in"$TimeDiff " and saved log to: "$logfilepath
}

Exchange Database statistics

DB sizes with whitespace:

Get-MailboxDatabase -Status | sort name | select name,@{Name='DB Size (Gb)';Expression={$_.DatabaseSize.ToGb()}},@{Name='Available New Mbx Space Gb)';Expression={$_.AvailableNewMailboxSpace.ToGb()}} | ft -auto

Database usage stats:

$dbs = Get-mailboxdatabase | select Name,Server | sort Name

$collcount = $dbs.count

$T1 = "Processing "

$T2 = " ("

$T3 = " of "

$T4 = ")"

$counter = 0

$results=@()

foreach($db in $databases){

$counter++

$Activity = $T1+$T2+$counter+$T3+$collcount+$T4

$percent = (($counter/$userscount) * 100)

$percentstatus = [String]([System.Math]::Round($percent, 0)) + "`% complete"

Write-Progress -PercentComplete $percent -Activity $activity -Status $percentstatus

$SUS = Get-StoreUsageStatistics -Database $db.name

$i = New-Object -TypeName PSObject

$i | Add-Member NoteProperty Database $db.name

$i | Add-Member NoteProperty Server $db.server

$i | Add-Member NoteProperty DisplayName $SUS.DisplayName

$i | Add-Member NoteProperty DigestCategory $SUS.DigestCategory

$i | Add-Member NoteProperty TimeInServer $SUS.TimeInServer

$i | Add-Member NoteProperty TimeInServer $SUS.TimeInCPU

$i | Add-Member NoteProperty SampleId $SUS.SampleId

$results += $i

}

$results | sort Server, TimeInServer -descending, Database | export-csv database-statistics.csv -notypeinformation

Functions 1

############## function to add append to export-csv #########################

function Export-CSV {
[CmdletBinding(DefaultParameterSetName='Delimiter',
  SupportsShouldProcess=$true, ConfirmImpact='Medium')]
param(
 [Parameter(Mandatory=$true, ValueFromPipeline=$true,
           ValueFromPipelineByPropertyName=$true)]
 [System.Management.Automation.PSObject]
 ${InputObject},

 [Parameter(Mandatory=$true, Position=0)]
 [Alias('PSPath')]
 [System.String]
 ${Path},

 [Switch]
 ${Append},

 [Switch]
 ${Force},

 [Switch]
 ${NoClobber},

 [ValidateSet('Unicode','UTF7','UTF8','ASCII','UTF32',
                  'BigEndianUnicode','Default','OEM')]
 [System.String]
 ${Encoding},

 [Parameter(ParameterSetName='Delimiter', Position=1)]
 [ValidateNotNull()]
 [System.Char]
 ${Delimiter},

 [Parameter(ParameterSetName='UseCulture')]
 [Switch]
 ${UseCulture},

 [Alias('NTI')]
 [Switch]
 ${NoTypeInformation})

begin
{
 $AppendMode = $false

 try {
  $outBuffer = $null
  if ($PSBoundParameters.TryGetValue('OutBuffer', [ref]$outBuffer))
  {
      $PSBoundParameters['OutBuffer'] = 1
  }
  $wrappedCmd = $ExecutionContext.InvokeCommand.GetCommand('Export-Csv',
    [System.Management.Automation.CommandTypes]::Cmdlet)
        
 $scriptCmdPipeline = ''

if ($Append) {
  
  $PSBoundParameters.Remove('Append') | Out-Null
    
  if ($Path) {
   if (Test-Path $Path) {        
    # Need to construct new command line
    $AppendMode = $true
    
    if ($Encoding.Length -eq 0) {
     $Encoding = 'ASCII'
    }
    
    $scriptCmdPipeline += 'ConvertTo-Csv -NoTypeInformation '

    if ( $UseCulture ) {
     $scriptCmdPipeline += ' -UseCulture '
    }
    if ( $Delimiter ) {
     $scriptCmdPipeline += " -Delimiter '$Delimiter' "
    } 
    
    $scriptCmdPipeline += ' | Foreach-Object {$start=$true}'
    $scriptCmdPipeline += '{if ($start) {$start=$false} else {$_}} '
    
    $scriptCmdPipeline += " | Out-File -FilePath '$Path'"
    $scriptCmdPipeline += " -Encoding '$Encoding' -Append "
    
    if ($Force) {
     $scriptCmdPipeline += ' -Force'
    }

    if ($NoClobber) {
     $scriptCmdPipeline += ' -NoClobber'
    }   
   }
  }
 } 
  
$scriptCmd = {& $wrappedCmd @PSBoundParameters }

 if ( $AppendMode ) {
  $scriptCmd = $ExecutionContext.InvokeCommand.NewScriptBlock(
      $scriptCmdPipeline
    )
 } else {
  $scriptCmd = $ExecutionContext.InvokeCommand.NewScriptBlock(
      [string]$scriptCmd
    )
 }

$steppablePipeline = $scriptCmd.GetSteppablePipeline(
        $myInvocation.CommandOrigin)
 $steppablePipeline.Begin($PSCmdlet)

 } catch {
   throw
 }
    
}

process
{
  try {
      $steppablePipeline.Process($_)
  } catch {
      throw
  }
}

end
{
  try {
      $steppablePipeline.End()
  } catch {
      throw
  }
}
}

######################################################################

function folder-statistics ($mailbox){

if(!($mailbox)){write-host -backgroundcolor yellow -foregroundcolor red "You must specify mailbox"}

else{

$INC = read-host "Enter INC reference"

$CurrentDate = (Get-Date).ToString('_MMM-dd-yyyy_HH-mm-ss_')

$outfile = "c:\users\me\documents\Reports\folderstats_" + $mailbox + $CurrentDate + $INC + ".csv"

[INT]$reportvalue = Read-host "Specify folder size value to report above"

$newhash = @()

$mbsize = Get-MailboxStatistics $mailbox | select TotalItemSize

[String]$mbsizestring = $mbsize.TotalItemSize

$mbSizeMB = [math]::Round(((((($mbsizestring.split("(")[1])).replace(" bytes)","")).replace(",","")))/1048576)

$mbSizeGB = [math]::Round(($mbSizeMB/1024),2)

$folderstats = Get-MailboxFolderStatistics $mailbox | select identity,name,FolderPath,FolderType,ItemsInFolder,DeletedItemsInFolder,FolderSize,ItemsInFolderAndSubfolders,DeletedItemsInFolderAndSubfolders,FolderAndSubfolderSize

foreach($fs in $folderstats){

[string]$foldersize = $fs.foldersize

$FolderSizeMB = [math]::Round(((((($foldersize.split("(")[1])).replace(" bytes)","")).replace(",","")))/1048576)

$FolderSizeGB = [String][System.math]::Round(($FolderSizeMB/1024),2)

$percentofmailbox = (($FolderSizeMB/$mbSizeMB) * 100)

$percentrounded = [string]([System.Math]::Round($percentofmailbox,2))

[string]$identity=$fs.identity

$i = New-Object -TypeName PSObject

$i | Add-Member NoteProperty User $mailbox

$i | Add-Member NoteProperty TotalSize $mbSizeGB

$i | Add-Member NoteProperty Identity $identity.replace($mailbox,"")

$i | Add-Member NoteProperty Name $fs.name

$i | Add-Member NoteProperty GBFolderSize $FolderSizeGB

$i | Add-Member NoteProperty MBFolderSize $FolderSizeMB

$i | Add-Member NoteProperty PerCent $percentrounded

$i | Add-Member NoteProperty ItemsInFolder $fs.ItemsInFolder

$i | Add-Member NoteProperty DeletedItemsInFolder $fs.DeletedItemsInFolder

$i | Add-Member NoteProperty FolderPath $fs.FolderPath

if($FolderSizeMB -gt $reportvalue){

$newhash += $i

}

}

$newhash = $newhash | sort MBFolderSize -descending | export-csv $outfile -NoTypeInformation

start-process excel.exe $outfile

}

}

######################################################################

function enumerate-dl ($dl){

$users = Get-DistributionGroupMember $dl | ?{$_.RecipientType -like "Use*"} | select name | sort name

$groups = Get-DistributionGroupMember $dl | ?{$_.RecipientType -like "*Group"} | select name | sort name

if($groups){Write-Host -foregroundcolor yellow "Nested groups:"

$groups | ft -hidetableheaders}

else{Write-Host -foregroundcolor green "No nested groups found"}

if($users){Write-Host -foregroundcolor white "Users:"

$users | ft -hidetableheaders}

else{Write-Host -foregroundcolor yellow "No users found"}

}

######################################################################

function queues {

$Global:hubs = Get-TransportServer

foreach($hub in $hubs){get-queue -Server $hub | ?{$_.MessageCount -gt 1 -and $_.DeliveryType -notlike "shadow*"}}

}

######################################################################

PS profile variables & transcript

$Global:scripts = "C:\Users\me\Documents\scripts"
$Global:reports = "C:\Users\me\Documents\Reports"

[string]$date = get-date -format "_dd-MMM-yy_HH-mm-ss"

$basepath = $env:userprofile + "\documents\pstranscripts\"

if((test-path $basepath) -eq $false){New-item –ItemType directory –force $basepath}

$Global:filepath = $basepath + "pstranscript" + $date + ".txt"

start-transcript -Path $filepath

Snippets for PsObject

This one pulls PrimarySMTP from ProxyAddresses:

ipmo ActiveDirectory

$PAS = Get-ADUser -Filter * -properties ProxyAddresses | select -ExpandProperty ProxyAddresses | ?{$_ -clike "SMTP:*"}

# (can use -SearchBase with DN of OU or foreach through a list of users)

foreach($PA in $PAS){

if($PA){$smtp = $PA.split(":")[1]}

# (get just the e-mail address)

if($PA){
[string]$cn = "CN=" + $PA.samaccountname + ","
[string]$DN = $PA.DistinguishedName
$oudn = $DN.replace($cn,"")
$truncoudn = $oudn.replace(",DC=domain,DC=com","")
if($truncoudn -notlike "*Mail Enabled*"){

$i = New-Object -TypeName PSObject
$i | Add-Member NoteProperty SamAccountName $PA.samaccountname
$i | Add-Member NoteProperty TruncOU $truncoudn

$results += $i

}}}

O365 - resuming move requests

For resuming move requests outside of business hours

Picks up failed move requests too (if PAW is configured) - https://blogs.technet.microsoft.com/exchange/2017/12/01/paw-your-way-into-office-365-migrations/

EMS:

[string]$ExchOnline = "outlook.office365.com"
$exolusername = "#####@#####.onmicrosoft.com"
$exolpassword = Get-Content "C:\TEMP\EncryptedPasswords\#####.txt" | ConvertTo-SecureString  
$Excredential = new-object -typename System.Management.Automation.PSCredential -argumentlist $exolusername, $exolpassword
$EXOnlineSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://$ExchOnline/powershell-liveid/ -Credential $Excredential -Authentication Basic -AllowRedirection -ErrorAction Stop

$startresumetimeinput = Read-host "Specify the time to resume move requests (in hh:mm format)"
$now = [System.DateTime]::Now
$today = [System.DateTime]::Today
$tomorrow = ($today).AddDays(1)

$startresumetime = Get-date $startresumetimeinput
if($now -gt $startresumetime){$startresumetime = ($startresumetime).AddDays(1)}

write-host "`$startresumetime is $startresumetime"

while($startresumetime -gt $now){write-progress -Activity "Waiting to resume moverequests at $($startresumetime)" -Status "Checking every 5 minutes. Time last checked at $($now)"; sleep 300; $now = [System.DateTime]::Now}

$warningPreference = "SilentlyContinue"
Import-PSSession $EXOnlineSession -warningaction silentlycontinue  -erroraction silentlycontinue | Out-Null
$warningPreference = "Continue"

[System.DateTime]::Now

Write-Progress -Activity "Getting move requests..." -Status "(where status = Suspended or Failed/failing)"
$mrstr = Get-MoveRequest -ResultSize unlimited | ?{$_.Status -eq "Suspended" -or $_.Status -like "*ail*"} | select exchangeguid
[INT]$mrstrcount = ($mrstr).count
Write-host "MoveRequests to be resumed:"$mrstrcount
[INT]$counter = 0
foreach($mrtr in $mrstr){
$counter ++
Write-Progress -Activity "Resuming move requests..." -Status "$counter of $mrstrcount"
[string]$mrexguid = $mrtr.exchangeguid
Resume-MoveRequest $mrexguid -confirm:$false
}

[System.DateTime]::Now
Get-PsSession | Remove-PsSession

O365 - suspending move requests

For suspending move requests during business hours (only those below 94%)

Bear these in mind:

The script has to produce a move request statistics report which takes an hour per 1500 requests.
Suspending is performed at a rate of 6000 per hour.
Work backwards from the cutoff time (with an emergency window allowed for).
E.g. Move requests = 3000
Business hours start at 09:00
Emergency cutoff  = 08:00
Suspend time = 07:30
Start report time = 05:00 (allow an extra bit of time between report and suspend to be safe)

EMS:

[string]$ExchOnline = "outlook.office365.com"
$exolusername = "#####@#####.onmicrosoft.com"
$exolpassword = Get-Content "C:\TEMP\EncryptedPasswords\#####.txt" | ConvertTo-SecureString  
$Excredential = new-object -typename System.Management.Automation.PSCredential -argumentlist $exolusername, $exolpassword
$EXOnlineSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://$ExchOnline/powershell-liveid/ -Credential $Excredential -Authentication Basic -AllowRedirection -ErrorAction Stop

$startreporttimeinput = Read-host "Specify the time to run a report (in hh:mm format)"
$startsuspendtimeinput = Read-host "Also state what time mailbox moves should be suspended (in hh:mm format)"
$now = [System.DateTime]::Now
$today = [System.DateTime]::Today
$tomorrow = ($today).AddDays(1)

$startreporttime = Get-date $startreporttimeinput
$startsuspendtime = Get-date $startsuspendtimeinput
if($now -gt $startreporttime){$startreporttime = ($startreporttime).AddDays(1); $startsuspendtime = ($startsuspendtime).AddDays(1) }

write-host "`$startreporttime is $startreporttime"
write-host "`$startsuspendtime is $startsuspendtime"

while($startreporttime -gt $now){write-progress -Activity "Waiting to start report at $($startreporttime)" -Status "Time last checked at $($now)"; sleep 60; $now = [System.DateTime]::Now}

[string]$DateTime = (Get-Date).Tostring("dd-MM-yyyy_HHmmss")
[string]$FilePath = "C:\Temp\"
[string]$OutputFilename1 = [string]::Format($FilePath + "MoveRequest_Full_Report_{0}.csv",$DateTime)
[System.DateTime]::Now

$warningPreference = "SilentlyContinue"
Import-PSSession $EXOnlineSession -warningaction silentlycontinue  -erroraction silentlycontinue | Out-Null
$warningPreference = "Continue"

Write-host "Gathering all move request information..."
$mrs = Get-MoveRequest -ResultSize unlimited | ?{$_.Status -notlike "Complet*"}
[INT]$mrscount = ($mrs).count
$moveresults = @()
$remainingmoveresults = @()
[INT64]$totaldata = 0
[INT64]$remainingdatavol = 0
$allmovestats = @()
[int64]$alltransferreddata = 0
[INT]$counter = 0
[System.DateTime]::Now
Write-host "The move requests will now be analyzed for more information..."
foreach($mr in $mrs){
$counter ++
Write-Progress -Activity "Gathering move request statistics..." -Status "$counter of $mrscount"
[String]$ExchangeGUID = $mr.ExchangeGUID
$mrstats = Get-MoveRequestStatistics $ExchangeGUID
[int64]$TotalMailboxSizeBytes = $mrstats.TotalMailboxSize -replace "(^.*?\()|\D"
[INT64]$TotalMailboxSizeMB = $TotalMailboxSizeBytes /1048576
[INT64]$TotalMailboxSizeGB = $TotalMailboxSizeMB /1024
[INT64]$TotalMailboxSizeGBrounded =[math]::Round($TotalMailboxSizeGB,2)
[int64]$TransferredBytes = $mrstats.BytesTransferred -replace "(^.*?\()|\D"
[INT64]$TransferredMB = $TransferredBytes /1048576
[INT64]$TransferredGB = $TransferredMB /1024
[INT64]$TransferredGBrounded =[math]::Round($TransferredGB,2)
#$RemainingGBtoTransfer = $TotalMailboxSizeGB - $TransferredGB
#if($RemainingGBtoTransfer -lt 0){$RemainingGBtoTransfer = 0}
#[INT64]$RemainingGBtoTransferrounded =[math]::Round($RemainingGBtoTransfer,2)
[INT]$TotalMailboxItemCount = $mrstats.TotalMailboxItemCount
[INT]$ItemsTransferred = $mrstats.ItemsTransferred
[INT]$ItemsRemaining = (($TotalMailboxItemCount) - ($ItemsTransferred))
if($ItemsRemaining -lt 0){$ItemsRemaining = 0}
$i = New-Object -TypeName PSObject
$i | Add-Member -MemberType NoteProperty -Name Alias -Value $mrstats.Alias
$i | Add-Member -MemberType NoteProperty -Name PrimarySMTPAddress -Value $mrstats.Alias
$i | Add-Member -MemberType NoteProperty -Name BatchName -Value $mrstats.BatchName
$i | Add-Member -MemberType NoteProperty -Name TotalMailboxSizeGB -Value $TotalMailboxSizeGBrounded
$i | Add-Member -MemberType NoteProperty -Name TransferredGB -Value $TransferredGBrounded
$i | Add-Member -MemberType NoteProperty -Name PercentComplete -Value $mrstats.PercentComplete
$i | Add-Member -MemberType NoteProperty -Name TotalMailboxItemCount -Value $TotalMailboxItemCount
$i | Add-Member -MemberType NoteProperty -Name ItemsTransferred -Value $ItemsTransferred
$i | Add-Member -MemberType NoteProperty -Name ItemsRemaining -Value $ItemsRemaining
$i | Add-Member -MemberType NoteProperty -Name OverallDuration -Value $mrstats.OverallDuration
$i | Add-Member -MemberType NoteProperty -Name TotalInProgressDuration -Value $mrstats.TotalInProgressDuration
$i | Add-Member -MemberType NoteProperty -Name TotalSuspendedDuration -Value $mrstats.TotalSuspendedDuration
$i | Add-Member -MemberType NoteProperty -Name ExchangeGuid -Value $mrstats.ExchangeGuid
$moveresults += $i
$totaldata = (($totaldata) + ($TotalMailboxSizeGB))
[INT64]$alltransferreddata = (($alltransferreddata) + ($TransferredGB))
}
Write-host "Analysis complete. Report saved to"$OutputFilename1
[System.DateTime]::Now

Get-PsSession | Remove-PsSession

$moveresults | sort PercentComplete | export-csv $OutputFilename1 -NoTypeInformation
$moverequeststosuspend = $moveresults | ?{$_.PercentComplete -ne 95}
[INT]$moverequeststosuspendCount = ($moverequeststosuspend).Count

$timenow = [System.DateTime]::Now
while($startsuspendtime -gt $now){write-progress -Activity "Waiting to suspend $moverequeststosuspendCount move requests at $($startsuspendtime)" -Status "Checking once per minute. Time last checked at $($now)"; sleep 60; $now = [System.DateTime]::Now}

write-progress -Activity "Waiting to suspend $moverequeststosuspendCount move requests at $($startsuspendtime)" -Status "Completed" -Completed

[System.DateTime]::Now

$warningPreference = "SilentlyContinue"
Import-PSSession $EXOnlineSession -warningaction silentlycontinue  -erroraction silentlycontinue | Out-Null
$warningPreference = "Continue"

[INT]$counter = 0
Write-host "Suspending $moverequeststosuspendCount move requests..."
foreach($moverequesttosuspend in $moverequeststosuspend){
$counter ++
Write-Progress -Activity "Suspending $moverequeststosuspendCount move requests..." -Status "$counter of $moverequeststosuspendCount"
[String]$ExchangeGUID2 = $moverequesttosuspend.ExchangeGUID
Suspend-MoveRequest $ExchangeGUID2 -confirm:$false
}
[System.DateTime]::Now
Get-PsSession | Remove-PsSession