In days of yore (Tue, 16 Apr 2024), Sirius thus quoth:

In days of yore (Mon, 15 Apr 2024), Jamie thus quoth:

So  there is a very nasty bug in the e1000e network card
driver.

Doing some reading turned up a Proxmox thread about the issues with these
Intel NICs.

https://forum.proxmox.com/threads/e1000-driver-hang.58284/page-10

May be worth scanning that thread and applying some of their solutions to
this problem.

--
Kind regards,

/S

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In days of yore (Mon, 15 Apr 2024), Jamie thus quoth:

So  there is a very nasty bug in the e1000e network card
driver.

https://www.intel.com/content/www/us/en/support/articles/000005480/ethernet-products.html
notes that MSI interrupts may be problematic on some systems. Worth
digging into whether that is an issue on this system of yours.

I am not sure Debian can resolve this problem with the driver, but
upstream kernel folks might. Unsure whether Intel still helps maintain
this driver as it is quite old (I dealt with support issues on this driver
some 15-16 years ago).

The Intel page states this is upstream kernel only at this point, so going
to SourceForge for their out-of-tree driver is no longer an option.

I am running Debian 12 Bookworm.

You will get the message "Detected Hardware Unit Hang" and then
the network card just stops working.

[snip]

This is a gigabit network card as I said it is a built in NIC I believe it
is an Intel NIC.

It is an Intel NIC. Most of the NIC drivers beginning with an 'e' followed
by numbers are Intel as far as I know. These NICs were very common as
on-board NICs in OEM systems as Intel provided them in large volumes. They
are not the best, but they usually do their job.

[snip]

This seems to happen when you are actually pushing a bit of traffic
though it not a lot but just even a little bit.  It isn't network overload
or anything I am barely doing anything really but it will do this.

If it is a hardware hang, it may be the NIC firmware getting its knickers
in a twist, and that is not something the kernel or the driver can do much about.

I have already tried  the following

ethtool -K eth1 tx off rx off
ethtool -K eth1 tso off gso off
ethtool -K eth1 gso off gro off tso off tx off rx off rxvlan off txvlan
off sg off

All worthwhile things to try. You can also try reducing the RX buffers
from the default 4096 to 2048 if you are not running a lot of traffic. It
might not help, but worth trying.

I have disabled all power management in the bios as well including the one for ASPM

I added the following to grub

pcie_aspm=off e1000e.SmartPowerDownEnable=0

This is in /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_aspm=off e1000e.SmartPowerDownEnable=0"

Good thinking about power management. :)

Then I did an update-grub as well.

None of this has worked in fixing this problem.  I am still getting the
same issue.

Best bet at this point would be to scout the Linux Kernel Mailing List
archives to see if anyone else have run into the same problems, and then reviewing the kernel maintainers list to find someone that works on the
e1000e driver to strike up a direct dialogue with them.

Can you please fix this issue this is a really nasty problem with Debian
12 (Bookworm)

I am seeing this being reported back in Kernel 5.3.x but i am not seeing any reports for 6.1.x about this issue.

Debian Bug report logs - #945912
Kernel 5.3 e100e Detected Hardware Unit Hang https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945912

If it has been reported before and is still present now, one of two things
is likely true.
1) the problem was intermittent and could not be reliably reproduced in
order to debug and resolve
2) the problem was related to the hardware itself, and there was no way
to fix it in either driver or firmware

It has been known to happen that drivers implement workarounds for issues
in the hardware itself, so that hardware bugs do not get tripped (or are tripped less often).

Please reply back and confirm that you got this email and that you are looking into this problem please.

To state the obvious, I am not a kernel maintainer for Debian and do not
speak on behalf of the Debian project.

I work for a Linux company you may have heard of and have done so for
almost eighteen years, a decade of which was in support. 15 years ago, I
know exactly who I would have gone to to look into this problem, but he
now works for Broadcom and probably has forgotten all about the
e1000/e1000e drivers.

Upstream driver maintainer would be the best bet IMHO. If this driver is community support only (i.e. if Intel no longer participates in driver maintenance), I would say that all bets are off.

With only one datapoint - your system and your NIC, it is not possible to
rule out that the NIC itself is bad. :-/

-- This email message, including any attachments, is for the intended recipient(s) only and may contain information that is privileged, confidential and/or exempt from disclosure under applicable law. If you
have received this message in error, or are obviously not one of the
intended recipients, please immediately notify the sender by reply email
and delete this email message, including any attachments. All
information in this email including any attachment(s) is to be kept in
strict confidence and is not to be released to anyone without my prior written consent.

You may want to discard these blurbs when posting to a mailing list.

--
Kind regards,

/S

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

It has been known to happen that drivers implement workarounds for issues
in the hardware itself, so that hardware bugs do not get tripped (or are tripped less often).

🙂

You make it sound like it's a rare occurrence, but it's actually
quite common. Most of it is discrete so you'll rarely be exposed to it,
but `grep bugs /proc/cpuinfo` is one of the places where you can see it
being somewhat documented.


Stefan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Apr 16, 2024 at 09:05:29AM -0400, Stefan Monnier wrote:

It has been known to happen that drivers implement workarounds for issues in the hardware itself, so that hardware bugs do not get tripped (or are tripped less often).

🙂

You make it sound like it's a rare occurrence, but it's actually
quite common. Most of it is discrete so you'll rarely be exposed to it,
but `grep bugs /proc/cpuinfo` is one of the places where you can see it
being somewhat documented.

One might argue that a driver's whole raison d'être /is/ to work around hardware bugs. But then, perhaps I'm a cynic ;-)

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCZh54ogAKCRAFyCz1etHa RvSEAJ0X+HPvt3H28gk53YTQpn07rsJt4ACfZhFM0lHUHpbpwH2vqZvuy24vZdg=
=zmZR
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

In days of yore (Tue, 16 Apr 2024), Jamie thus quoth:

Look this is a kernel bug and Debian needs to
fix this! Don't give me any of this crap about upstream
this is a bug with the Debian Kernel!

Pay attention, because I am now in Support Mode as a former Principal
Technical Account Manager for Red Hat.


No, this is not necessarily a kernel bug. It can be a hardware bug and it
is plausible it can not be solved with a driver work-around.

You are hitting a problem and you want someone else to fix it for you. The answer may simply be that you need to replace the NIC with something else.

FWIW, I have these Intel NICs in my two NUCs and they are functioning
fine. With Debian 12.5 and the latest updates.

$ lspci -v -s 00:1f.6
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection I219-V (rev 21)
DeviceName: LAN
Subsystem: Intel Corporation Ethernet Connection I219-V
Flags: bus master, fast devsel, latency 0, IRQ 123, IOMMU group 7
Memory at df100000 (32-bit, non-prefetchable) [size=128K]
Capabilities: <access denied>
Kernel driver in use: e1000e
Kernel modules: e1000e

The revision of the NIC may determine whether you have *hardware* problems
or not.

This needs to be fixed!

Quick answer: replace the NIC. And do some groundwork to determine if the
NIC you replace it with has issues you should be aware of or not.

I have already tried disabling the offloads and it does
not work.

The specific offloads seemed to be the CRC related ones.

# ethtool -k eno1
Features for eno1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]

Note: when you disable these, throughput can drop sharply.

The other setting suggested was to hike the TX ringbuffer.

# ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX: 4096
RX Mini: n/a
RX Jumbo: n/a
TX: 4096
Current hardware settings:
RX: 256
RX Mini: n/a
RX Jumbo: n/a
TX: 256
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
TCP data split: n/a

# ethtool -G eno1 tx 2048 rx 2048
# ethtool -g eno1
Ring parameters for eno1:
Pre-set maximums:
RX: 4096
RX Mini: n/a
RX Jumbo: n/a
TX: 4096
Current hardware settings:
RX: 2048
RX Mini: n/a
RX Jumbo: n/a
TX: 2048
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
TCP data split: n/a

The reason the ringbuffers are important is that the kernel and the OS can construct packets faster in bursts than the NIC can handle, so the OS can
queue up packets in the ringbuffer and the NIC can asynchronously pick the packets from the buffer and send them across the wire. If the ringbuffers
are set too small, they will overflow and you will get overflow errors.

# ethtool -S eno1
NIC statistics:
rx_packets: 24463
tx_packets: 6358
rx_bytes: 3093199
tx_bytes: 669733
rx_broadcast: 8044
tx_broadcast: 9
rx_multicast: 10434
tx_multicast: 2510
rx_errors: 0
tx_errors: 0
tx_dropped: 0 <<<< If buffers are set too small, this increases
multicast: 10434
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 9
rx_flow_control_xoff: 9
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_csum_offload_good: 8539 <<<< If you have issues with checksum
rx_csum_offload_errors: 0 <<<< offload, check these
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
rx_hwtstamp_cleared: 0
uncorr_ecc_errors: 0
corr_ecc_errors: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0

It isn't the cable either I have tried different cables it
still happens! This is an issue with the Kernel module for
the e1000e NIC card.

Excellent data-point, you have ruled out whether the cable is faulty or
not. But your assumption that this is the kernel module that is broken
is still faulty.

Provably, I am running the same type of NIC (albeit a different revision)
with the same driver and I do not observe any issues. Thus, leveraging
Occam's Razor, it follows that scrutinising your particular NIC is in
order.

This is a bug with the kernel that needs to be fixed in Debian!

That is not certain. Please refrain from making such statements. You have
a problem and you want it fixed, but you are deaf to what you might need
to consider to get the problem resolved. So you may end up living with
this problem for longer than you have to.

I have already replaced it but this bug needs to be fixed
by the Debian kernel team!

Replaced *what*? The NIC? What did you replace it with? The cable?

Troubleshooting tips:
1) Remain calm. When agitated, mistakes happens.
2) Document every step.
3) Change one thing at a time
4) Record the outcome of the changes
5) Go back to 3) until you can form a hypothesis

You have tried some things and drawn a conclusion, yet that conclusion may
be incorrect. Be open to the fact you may be wrong and that the answer
could be something other than what you think/want it to be.

--
Kind regards,

/S

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)