Network crashes under heavy load

[Nathen]

For some reason my server stops responding to network traffic (shares
go offline, no response to SSH or ping, etc) after heavy load -
transferring large amounts of data via Samba or running several iperf
benchmarks causes it, however the system still responds to the power
button so shuts down when it's pressed. I'm using an Intel D510MO
board with Realtek 8111DL network controller which I'm guessing has
something to do with it, I could be wrong. I think it might be a bug
but I thought it best to ask here first.
Thanks. :)

[Andrei Popescu]

Post by Nathen
For some reason my server stops responding to network traffic (shares
go offline, no response to SSH or ping, etc) after heavy load -
transferring large amounts of data via Samba or running several iperf
benchmarks causes it, however the system still responds to the power
button so shuts down when it's pressed. I'm using an Intel D510MO
board with Realtek 8111DL network controller which I'm guessing has
something to do with it, I could be wrong. I think it might be a bug
but I thought it best to ask here first.
Thanks. :)

...

Anything interesting in the logs (/var/log/syslog)?

Do you have a different adapter (preferably different chipset) to test
with?

Are you sure it's not the switch (if you have any)?

You might also try to replace the cable(s), you never know.

Regards,
Andrei

[Ron Johnson]

Post by Nathen
For some reason my server stops responding to network traffic (shares
go offline, no response to SSH or ping, etc) after heavy load -
transferring large amounts of data via Samba or running several iperf

How heavy is "heavy"?

Post by Nathen
benchmarks causes it, however the system still responds to the power
button so shuts down when it's pressed. I'm using an Intel D510MO
board with Realtek 8111DL network controller which I'm guessing has

Yeah, probably.

Post by Nathen
something to do with it, I could be wrong. I think it might be a bug
but I thought it best to ask here first.

Intel NICs seem to be very popular on servers, and are very well
supported under Linux.

[Jaime Di Cristina]

Post by Nathen
For some reason my server stops responding to network traffic (shares
go offline, no response to SSH or ping, etc) after heavy load -
transferring large amounts of data via Samba or running several iperf
benchmarks causes it, however the system still responds to the power
button so shuts down when it's pressed. I'm using an Intel D510MO
board with Realtek 8111DL network controller which I'm guessing has
something to do with it, I could be wrong. I think it might be a bug
but I thought it best to ask here first.
Thanks. :)

...

Hello:

Are you running the latest BIOS version? Check this thread:
http://bbs.archlinux.org/viewtopic.php?pid=771174 . A poster there
said that network problems went away after updating the BIOS to
version 0210. Also there are some vague reports of network problems
in the newegg.com reviews for this board. One user says that the
problems can be solved by "manually compiling drivers from Realtek".

I hope that you can solve the problem and would appreciate if you
shared your results.

Jaime

[Nathen]

Thanks for replying.
To answer your questions - I couldn't find anything unusual in the
logs, the first message around the time of the crash was the shutdown
message when I pressed the power button.
I have a PCI realtek card I could try with, I think it's a different
chipset so I'll try with that, no switch and a different cable when I
get home later.

Sorry, I should have been more specific - it usually happens after
copying tens of GB but it seems random because it survives more
sometimes. For example I was backing up a number of large files (about
40GB total) which caused a crash the first time I tried, halfway
through copying a file but everything copied fine after a reboot. Also
running iperf about ten times caused another crash but again this
seems variable.
An Intel NIC would usually be my first choice but since this board has
no PCIe slots I'm hesitant to use a PCI NIC if it's going to limit
network bandwidth.

I hope that answers all your questions and I'll try updating BIOS,
drivers and run those tests when I get home.
Thanks

[Alexander Samad]

[snip]

Post by Nathen
An Intel NIC would usually be my first choice but since this board has
no PCIe slots I'm hesitant to use a PCI NIC if it's going to limit
network bandwidth.

have you tried getting the realtek driver and compiling it, on some of
my earlier boards, the nic was loaded by an in line kernel module
but it didn't work properly.... I had to down load the realtek and compile it.

Post by Nathen
I hope that answers all your questions and I'll try updating BIOS,
drivers and run those tests when I get home.
Thanks

[snip]

[Davide Mirtillo]

Post by Alexander Samad
[snip]

Post by Nathen
An Intel NIC would usually be my first choice but since this board has
no PCIe slots I'm hesitant to use a PCI NIC if it's going to limit
network bandwidth.

have you tried getting the realtek driver and compiling it, on some of
my earlier boards, the nic was loaded by an in line kernel module
but it didn't work properly.... I had to down load the realtek and compile it.

I was experiencing weird issues with my NIC aswell, check if you have
the r8169 kernel module loaded, if so, that might be the cause of your
crashes.

Get the proper drivers for your NIC on the realtek website, compile
them against your kernel, run depmod, blacklist the old module, update
initramfs and you shouldn't have any more trouble.

[Stan Hoeppner]

Post by Nathen
An Intel NIC would usually be my first choice but since this board has
no PCIe slots I'm hesitant to use a PCI NIC if it's going to limit
network bandwidth.

Hint: a standard PCI 32bit/33MHz PCI bus can transfer 132MB/s, which is
slightly greater than the throughput of a single GigE NIC which is a little
over 100MB/s. If the NIC is the only device on that bus, then you have no
worries, and likewise if the bus is shared with a low bandwidth PCI device or two.

What motherboard is this again and which RAID card? Apologies if you already
posted it and I missed it.
--
Stan

[Nathen]

I posted this earlier but it's not appeared on the list for some reason:

Sorry about the delay I didn't have a chance to try anything yesterday.
Anyway I've just tried a direct connection, updating BIOS, compiling and
installing the latest drivers and it's still the same, if not worse - it
seems I can crash it with as little as two iperf runs now. I'm still
relatively inexperienced with Linux though so I might be doing something
wrong. I haven't managed to find that PCI card yet but I'll keep looking
or try to borrow one from a friend or something.

I'm using an Intel Atom D510MO board and no RAID.

[Stan Hoeppner]

Post by Nathen
Sorry about the delay I didn't have a chance to try anything yesterday.
Anyway I've just tried a direct connection, updating BIOS, compiling and
installing the latest drivers and it's still the same, if not worse - it
seems I can crash it with as little as two iperf runs now. I'm still
relatively inexperienced with Linux though so I might be doing something
wrong. I haven't managed to find that PCI card yet but I'll keep looking
or try to borrow one from a friend or something.
I'm using an Intel Atom D510MO board and no RAID.

...

~$30 + shipping

http://www.newegg.com/Product/Product.aspx?Item=N82E16833106121
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106122
(2nd one has short bracket for low profile cases, functionally identical)

I don't use anything but Intel NICs with Linux. I've had such great luck with
them and the cost is low enough there's no legitimate reason to even try
competing products. These cards simply work, and work well. Totally
compatible, totally reliable. Did I mention they're fast? They push wire
speed all day long if the rest of the system is up to the task.

[Nathen]

I've just found and installed that Realtek PCI card I was talking
about - it's an r8169 chipset card and so far I've not been able to
crash it but as I feared, throughput is much lower, varying between
about 580 and 650 Mbps although I'm not sure whether that's due to the
PCI bus or the card itself?

[Stan Hoeppner]

Post by Nathen
I've just found and installed that Realtek PCI card I was talking
about - it's an r8169 chipset card and so far I've not been able to
crash it but as I feared, throughput is much lower, varying between
about 580 and 650 Mbps although I'm not sure whether that's due to the
PCI bus or the card itself?

The lowish performance is likely due to the 8169 chip itself. Realtek low end
chips are somewhat notorious for not being able to reach wire speed. This is
probably why the 8169 based cards are only ~$10 USD compared to ~$30 USD for
the Intel NICs, which always hit near wire speed. Sometimes you get what you
pay for. Like I said, this is why I always use Intel NICs.

[Nathen]

I thought that might be the case. I think I've finally learned my
lesson about Realtek NICs anyway, I ordered a PCI Intel NIC earlier
today so I'll see how that goes.
Thanks very much for your help everyone. :)