rabin.blog

Fix Intel e1000e “Detected Hardware Unit Hang” on Proxmox

How an Intel NIC Hang Took Down My Homelab – and How I Fixed It

This incident happened on a Dell OptiPlex Micro running Proxmox VE. The machine is my always-on Node02 host and carries critical home infrastructure:

  • Home Assistant
  • AdGuard Home
  • Unbound DNS
  • MQTT
  • Zigbee2MQTT
  • Other LAN support services

At first, this looked like a disaster spread across half the house. That host is the backbone for the living room stack. It runs Home Assistant and several supporting services, so when its network dies, the failure can look like everything is broken at once. That is what makes these incidents so sneaky – the real problem is hidden behind a long chain of symptoms.

The hardware in question is a Dell OptiPlex 7070 Micro using an Intel I219-LM NIC with the e1000e driver. During the incident, the kernel repeatedly logged:

e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang:

That message matters because it is not just a harmless warning; it means the NIC hardware or driver has wedged hard enough that the interface can stop behaving normally. If your host depends on that NIC for DNS, automation, and service reachability, the whole homelab can appear to collapse.

When the Dell system appeared to hang, the impact looked much bigger than a single server issue:

  • Smart devices stopped behaving normally. As they depend on Home Assistant.
  • Smart-device schedules did not run.
  • DNS via AdGuard Home and Unbound stopped working. The browser fails to work on other machines.
  • Home Assistant and automation services became unreachable.

The visible symptom was “the system hung,” but the important detail was that many services failed together because they shared one network path through the Proxmox host.

Assessment: It Was a Network Driver Hang, Not Just “The Server Froze”

The key kernel log signature was:

e1000e 0000:00:1f.6 eno2: Detected Hardware Unit Hang

The affected machine and NIC details:

HostProxmox Node02 / pve-node02
HardwareDell OptiPlex Micro class system
NICIntel I219-LM
Linux drivere1000e
Interfaceeno2
Failure signatureDetected Hardware Unit Hang

This type of failure can make the host look dead from the network even if the kernel is still running. That explains why Home Assistant, DNS, MQTT, and Zigbee-related services all disappeared together.

Useful diagnosis commands

journalctl --list-boots
journalctl -b -k --no-pager | grep -Ei 'e1000e|Detected Hardware Unit Hang|eno2'
ethtool -i eno2
ethtool -k eno2
ethtool -S eno2 | grep -Ei 'err|drop|timeout|restart|reset|collision|crc|fifo|carrier|watchdog'

If you see repeated Detected Hardware Unit Hang messages for e1000e, treat the NIC/driver path as a prime suspect.

Solution Summary: Proxmox Intel e1000e Hardware Unit Hang Fix

The solution has two layers:

  1. Primary mitigation: disable problematic offload features on the Intel e1000e interface using ethtool, and make that persistent with a systemd service.
  2. Safety net: add a systemd watchdog timer that checks recent kernel logs for Detected Hardware Unit Hang, rate-limits recovery attempts, resets eno2 only when needed, and reapplies the offload mitigation after a reset.

This is intentionally not just a cron job. The systemd timer gives better logging, service status, timer visibility, and failure semantics.

Fix Part 1: Disable Intel e1000e Offloads Persistently

The first fix is to disable NIC offloads that are commonly involved in Intel e1000e hardware-unit hang issues.

Create the systemd service

cat >/etc/systemd/system/disable-nic-offload-eno2.service <<'EOF'
[Unit]
Description=Disable NIC offloading for Intel e1000e interface eno2
After=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/sbin/ethtool -K eno2 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

Enable it

systemctl daemon-reload
systemctl enable --now disable-nic-offload-eno2.service

Verify it

systemctl is-enabled disable-nic-offload-eno2.service
systemctl is-active disable-nic-offload-eno2.service
ethtool -k eno2 | grep -E 'tcp-segmentation-offload|generic-segmentation-offload|generic-receive-offload|rx-checksumming|tx-checksumming|scatter-gather|rx-vlan-offload|tx-vlan-offload'

On the active Node02 host, the verified offload state is:

rx-checksumming: off
tx-checksumming: off
scatter-gather: off
    tx-scatter-gather: off
    tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off

Fix Part 2: Add a systemd Watchdog for Self-Recovery

The offload service is the main mitigation. The watchdog is a second layer: if the NIC still logs a fresh Detected Hardware Unit Hang and the host is still alive, it tries to reset eno2.

Important: this is the exact logic currently active on Node02. It uses a 5-minute journal lookback, not 2 minutes, because the systemd timer runs every 2 minutes with AccuracySec=30s. A strict 2-minute lookback could miss an event if the timer fires late.

Create the watchdog script

cat >/usr/local/sbin/e1000e-eno2-watchdog <<'EOF'
#!/usr/bin/env bash
set -u
set -o pipefail

IFACE="eno2"
LOOKBACK="5 minutes ago"
LOCK="/run/e1000e-${IFACE}-watchdog.lock"
STAMP="/run/e1000e-${IFACE}-last-reset"
MIN_RESET_INTERVAL_SECONDS=600

exec 9>"$LOCK"
flock -n 9 || exit 0

if ! journalctl -k --since "$LOOKBACK" --no-pager \
  | grep -Eqi "e1000e .*${IFACE}: Detected Hardware Unit Hang|Detected Hardware Unit Hang"; then
  exit 0
fi

now="$(date +%s)"
last=0
[ -f "$STAMP" ] && last="$(cat "$STAMP" 2>/dev/null || echo 0)"

if [ $((now - last)) -lt "$MIN_RESET_INTERVAL_SECONDS" ]; then
  logger -t e1000e-watchdog "hardware hang detected on ${IFACE}, but reset suppressed by rate limit"
  exit 0
fi

logger -t e1000e-watchdog "hardware hang detected on ${IFACE}; resetting NIC"

if ! ip link set "$IFACE" down; then
  logger -t e1000e-watchdog "failed to bring ${IFACE} down; NIC reset incomplete"
  exit 1
fi

sleep 2

if ! ip link set "$IFACE" up; then
  logger -t e1000e-watchdog "failed to bring ${IFACE} up; NIC reset incomplete"
  exit 1
fi

# Re-apply the offload mitigation after link reset, just in case.
ethtool -K "$IFACE" gso off gro off tso off tx off rx off rxvlan off txvlan off sg off 2>/dev/null || \
  logger -t e1000e-watchdog "warning: unable to re-apply offload mitigation on ${IFACE}"

date +%s >"$STAMP"

logger -t e1000e-watchdog "NIC reset completed for ${IFACE}"
EOF

chmod +x /usr/local/sbin/e1000e-eno2-watchdog

Create the watchdog service

cat >/etc/systemd/system/e1000e-eno2-watchdog.service <<'EOF'
[Unit]
Description=Watch for Intel e1000e hardware hangs and reset eno2
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/e1000e-eno2-watchdog
EOF

Create the watchdog timer

cat >/etc/systemd/system/e1000e-eno2-watchdog.timer <<'EOF'
[Unit]
Description=Run e1000e eno2 watchdog every 2 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=2min
AccuracySec=30s
Unit=e1000e-eno2-watchdog.service

[Install]
WantedBy=timers.target
EOF

Enable the timer

systemctl daemon-reload
systemctl enable --now e1000e-eno2-watchdog.timer

Why the script exits non-zero on reset failure

If ip link set eno2 down or ip link set eno2 up fails, the watchdog must not pretend recovery succeeded. The script exits with status 1 before writing the reset stamp. That prevents a failed reset from being hidden by the 10-minute rate limit.

Why I Did Not Use the Cron One-Liner

A suggested cron approach looked like this:

*/2 * * * * dmesg -T --since "2 min ago" 2>/dev/null | grep -qi "hardware unit hang" && \
    ip link set eno2 down && sleep 2 && ip link set eno2 up

This can work as a quick test, but I do not recommend it for a critical Proxmox host running DNS and smart-home services.

Problems with the cron approach

  • No robust service status: systemd gives systemctl status, exit codes, and timer visibility.
  • No lock by default: overlapping cron runs are possible unless you add flock.
  • No rate limit by default: a recurring log message could flap the interface repeatedly.
  • Too narrow timing window: a strict 2-minute lookback can miss an event if the scheduler runs late.
  • Weak failure semantics: if reset fails, cron can hide the failure unless explicitly logged and checked.
  • No offload reapply: the one-liner brings the link back but does not reapply the ethtool -K mitigation.
  • Harder monitoring: systemd timer and service logs are easier to inspect with journalctl.

The systemd version keeps the same recovery idea but makes it operationally safer.

Verification Commands

Use these commands after installing the fix:

systemctl status disable-nic-offload-eno2.service --no-pager -l
systemctl status e1000e-eno2-watchdog.timer --no-pager -l
systemctl list-timers e1000e-eno2-watchdog.timer --no-pager
systemctl status e1000e-eno2-watchdog.service --no-pager -l

ethtool -k eno2 | grep -E 'tcp-segmentation-offload|generic-segmentation-offload|generic-receive-offload|rx-checksumming|tx-checksumming|scatter-gather|rx-vlan-offload|tx-vlan-offload'

journalctl -k --since "24 hours ago" --no-pager | grep -i "hardware unit hang" || true
journalctl -t e1000e-watchdog --since "24 hours ago" --no-pager || true

Current verified Node02 state

At the time of finalizing this post, Node02 reported:

disable-nic-offload-eno2.service: enabled, active
e1000e-eno2-watchdog.timer: enabled, active
watchdog next run: scheduled every 2 minutes
selected eno2 offloads: off
recent hardware unit hangs: none in last 24 hours
watchdog action logs: no entries

No watchdog action logs is healthy here. It means the timer is running, but it has not found a fresh NIC hang to recover from.

Optional External Monitoring with Uptime Kuma

The watchdog is local to the Proxmox host. For better visibility, monitor the host from another machine outside the failure domain. In my setup, Uptime Kuma runs on a separate mini PC.

Suggested Uptime Kuma monitor tree:

Node02 / Living Room Critical Stack
├── Node02 host ping: 192.168.XX.2
├── Node02 Proxmox UI: https://192.168.XX.2:8006
├── Home Assistant HTTP: http://<ha-ip>:8123
├── AdGuard DNS query: @<adguard-ip> google.com A
├── MQTT TCP: <mqtt-ip>:1883
├── Zigbee2MQTT UI: http://<z2m-ip>:8080
└── Router/gateway ping: 192.168.XX.1

Keep the router/gateway check as an independent baseline. If the router is up but Node02, Home Assistant, AdGuard, MQTT, and Zigbee2MQTT fail together, suspect the Node02 host or NIC path.

Final Notes and Escalation Path

This fix is a practical mitigation, not a guarantee that the Intel NIC can never hang again.

If Detected Hardware Unit Hang returns repeatedly even after offloads are disabled, I would escalate in this order:

  1. Verify offloads are still disabled after reboot or kernel updates.
  2. Check cable and switch port.
  3. Update or test a different Proxmox kernel during a maintenance window.
  4. Move critical services or management traffic to a different NIC path, such as USB Ethernet or another host.
  5. Consider replacing the host if the integrated NIC remains unreliable.

Bottom line: the most searchable solution for this class of issue is: disable Intel e1000e offloads with ethtool -K, persist it with systemd, and use a rate-limited systemd watchdog timer instead of a raw cron job for emergency NIC reset recovery.

Comments

Loading comments...