Restarting after a power outage

When servers start on their own after a power outage

The shutdown is usually carefully planned, the startup rarely. Whether servers and VMs run again on their own after a power outage depends on when power returns:

Case C is more common than you would think. The shutdown deliberately triggers with a reserve so that every VM shuts down safely. If the outage lasts 20 minutes and the UPS would have held for 30, the servers are off while the UPS keeps running. From the servers' point of view there was no power outage, so they do not switch back on.

Power on when AC returns, in BIOS or BMC

For case B to work, every server must be set to switch on when power is applied. The setting lives in the BIOS or in the management controller (BMC):

Be careful with “last state” (Last State, Restore): after an orderly shutdown the last state was “off”, so the server stays off. For servers that go through a UPS shutdown, “always on” is the right choice.

Use the delay for the order: storage systems immediately, Hyper-V hosts one to two minutes later. It also spreads the inrush current when many devices share one UPS.

Case C: letting the UPS switch its output off

There are two ways to solve case C. The first: the UPS switches its output off once all servers are down and back on when power returns. Case C then becomes case B.

1. NUT — the primary NUT client shuts down as the last system and tells the UPS to switch its output off after a delay and back on when power returns. That is NUT's standard sequence, provided the driver supports it for your UPS.
2. UPS network card — the cards have settings for when the output comes back after a shutdown: a return delay and a minimum battery charge. The minimum charge prevents servers from starting on an almost empty battery and going down hard at the next outage.

A switched-off output also hits everything that was not shut down, though: switches, firewall, phone system, NAS. Check what is connected to the UPS before letting it switch the output. The UPS Hyper-V Shutdown Monitor leaves the UPS output on. For case C you use the second way with it.

Case C: powering servers on via the BMC

The second way: after power returns, the servers are switched on through their management controller, by hand in the web interface or by script. With Redfish this works across vendors on current servers (PowerShell 7):

$cred = Get-Credential   # BMC account allowed to power the server on
$body = @{ ResetType = 'On' } | ConvertTo-Json

# System ID: Dell iDRAC "System.Embedded.1", HPE iLO and Lenovo XCC usually "1"
# -SkipCertificateCheck only for a self-signed BMC certificate
Invoke-RestMethod -Method Post -Credential $cred -Authentication Basic -SkipCertificateCheck `
  -Uri 'https://bmc-hv01/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset' `
  -ContentType 'application/json' -Body $body

Older servers without Redfish can be powered on via IPMI:

# Password from the IPMI_PASSWORD environment variable, not on the command line
ipmitool -I lanplus -H bmc-hv01 -U admin -E chassis power on

The script has to run on a device that survived the outage, such as a small management computer or a server on a separate power supply. Wake-on-LAN is an alternative, but not every server supports it from the powered-off state, and it must be enabled in the BIOS and on the network card.

If the UPS Hyper-V Shutdown Monitor keeps running after the shutdown, it reports the return of power by email and, from the Pro edition, via Teams or webhook. A webhook can trigger such a power-on script.

The order in which everything starts again

1. Network — switches and firewall. They usually start by themselves as soon as power is applied.
2. Storage systems — SAN or NAS, until the volumes are ready. Hosts that start earlier will not find their VM storage.
3. Hyper-V hosts — staggered using the power-on delay in BIOS or BMC.
4. Domain controllers — as the first VMs, so sign-in and DNS are available for all others.
5. Database and mail servers, then file and application servers.

Hyper-V starts the VMs through the automatic start action. There is a common trap here: the default “Automatically start if it was running when the service stopped” (StartIfRunning) does not apply when a tool shuts down the VMs before the host. When the Hyper-V service stopped, the VMs were already off, and they stay off. For VMs that should run again after a UPS shutdown, set Start with a delay:

Set-VM -Name "DC01"   -AutomaticStartAction Start -AutomaticStartDelay 0
Set-VM -Name "SQL01"  -AutomaticStartAction Start -AutomaticStartDelay 120
Set-VM -Name "APP01"  -AutomaticStartAction Start -AutomaticStartDelay 240
Set-VM -Name "TEST01" -AutomaticStartAction Nothing

# Check
Get-VM | Select-Object Name, State, AutomaticStartAction, AutomaticStartDelay

In failover clusters the cluster starts the VMs by their priority; the VM's start action plays no role there. Details are in the guide Hyper-V failover cluster power outage, the shutdown order in the guide Shutting down Hyper-V VMs in a power outage.

Restart checklist

1. BIOS or BMC — every server set to “always on”, not “last state”, with a staggered delay.
2. Case C solved — the UPS switches its output off and on again, or a script or a person powers the servers on via the BMC.
3. Minimum charge — the UPS only restores its output once the battery is sufficiently charged.
4. VM start action — Start with a delay instead of StartIfRunning, test VMs set to Nothing.
5. Storage before hosts — the SAN or NAS is ready before the hosts start their VMs.
6. Test in a maintenance window — unplug the UPS, wait for the shutdown, restore power before the battery is empty and check that everything runs again.

Frequently asked questions