Disk drive zero-fill benchmarks for various synchronization methods and block sizes

[David Christensen]

debian-user:

I had the need to zero-fill a Seagate Constellation ES.2 3 TB SATA
magnetic hard disk drive using an Intel DQ67SW motherboard, Intel Core
i7-2600S processor, 8 GB dual-channel memory, StarTech PEXSAT32 adapter,
and Debian:

2024-09-08 08:05:02 ***@bullseye-bios
~/hardware/seagate/ST33000650NS/REDACTED
# cat /etc/debian_version ; uname -a
11.10
Linux bullseye-bios 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1
(2024-08-10) x86_64 GNU/Linux


When done, I ran some benchmarks to compare sequential write performance
for various synchronization methods and block sizes:

1. Using dd(1) with option oflag=sync:

2024-09-08 09:19:27 ***@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 'oflag=sync' |
egrep 'time|GiB'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.3181 s, 147 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.39608 s, 145 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.48021 s, 144 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.92566 s, 135 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=4M count=256
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 8.97546 s, 120 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.3814 s, 69.8 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K
oflag=sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 40.8931 s, 26.3 MB/s

2. Using dd(1) with sync(1):

2024-09-08 09:19:38 ***@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 '&& sync' |
egrep 'time|GiB'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1 && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.3975 s, 145 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4 && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21795 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16 && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21845 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64 && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.20049 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=4M count=256 && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.11058 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K && sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.12372 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K &&
sync
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.10426 s, 151 MB/s

3. Using dd(1) without explicit synchronization:

2024-09-08 09:20:45 ***@laalaa ~/hardware/seagate/ST33000650NS/REDACTED
$ tail -n 214 log.txt | perl -pe 's/^\s+//' | egrep -A 3 time | egrep
'time|GiB' | egrep -v -A 1 '(MB\/s|sync)$' | egrep -v '^--'
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1G count=1
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.30077 s, 147 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256M count=4
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.27323 s, 148 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=64M count=16
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.21564 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=16M count=64
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.17308 s, 150 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_Z294MZFA bs=4M count=256
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.20907 s, 149 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=1M count=1K
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.11303 s, 151 MB/s
# time dd if=/dev/zero
of=/dev/disk/by-id/ata-SEAGATE_ST33000650NS_REDACTED bs=256K count=4K
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.13726 s, 150 MB/s


Additional observations:

1. The Bash shell 'time' built-in command and the per-second/
per-transfer performance values displayed by dd(1) option
'status=progress' can show the interaction between dd(1) option
'oflag=sync', kernel I/O buffering, dd(1) write completion, and sync(1).

2. dd(1) does not exit until the final block has been written.


Conclusions:

1. dd(1) option 'oflag=sync' causes block size to affect write
performance -- increasing the block size increases the performance. The
effect is non-linear, with a knee at around 30 IOPS. For best
performance, block size should be tuned to produce 30 IOPS or less.

2. sync(1) is unnecessary.

3. Use dd(1) as-is, without 'oflag=sync' or sync(1).


David

[Anssi Saari]

Post by David Christensen
2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

[David Wright]

Post by Anssi Saari

Post by David Christensen
2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

I think you also need to read the other manpage, fsync(2).

Cheers,
David.

[Anssi Saari]

Post by David Wright

Post by Anssi Saari

Post by David Christensen
2. sync(1) is unnecessary.

Does it even do anything wrt device files? sync(2) says "sync() causes
all pending modifications to filesystem metadata and cached file data to
be written to the underlying filesystems."

I think you also need to read the other manpage, fsync(2).

Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

[Anssi Saari]

Post by Anssi Saari
Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful? I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable I found for example
https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem
and so arrived to the conclusion that the final close(2) call on a block
device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

[t***@tuxteam.de]

Post by Anssi Saari

Post by Anssi Saari
Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful? I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable [...]

Please, Anssi. You can do better than that.

Post by Anssi Saari
[...] I found for example
https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem
and so arrived to the conclusion that the final close(2) call on a block
device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

So I did an (admittedly very rough) experiment, and this seems to confirm the
above:

***@caliban:~$ time sudo dd if=/dev/zero of=/dev/sdb bs=1M ; time sync
dd: error writing '/dev/sdb': No space left on device
15268+0 records in
15267+0 records out
16008609792 bytes (16 GB, 15 GiB) copied, 3614.25 s, 4.4 MB/s

real 60m14.260s
user 0m0.005s
sys 0m0.007s

real 0m0.077s
user 0m0.002s
sys 0m0.000s

I interpret that this way: after 16G of dd (the stick is 16G) there isn't
a significant amount of buffers lying around waiting to be flushed to the
stick (it's a slow USB, as suggested by the 60m the dd took, so if there
were a lot of buffers, we'd see more than .077 sec for the sync).

The box has plenty of RAM (16G) for what it's doing (14G free), so there
is no particular memory pressure which would favour quick flushing of
buffers.

So yes, it seems when the dd is done, it's done.

Cheers

[Michael Stone]

Post by t***@tuxteam.de
dd: error writing '/dev/sdb': No space left on device
15268+0 records in
15267+0 records out
16008609792 bytes (16 GB, 15 GiB) copied, 3614.25 s, 4.4 MB/s
real 60m14.260s
user 0m0.005s
sys 0m0.007s
real 0m0.077s
user 0m0.002s
sys 0m0.000s
I interpret that this way: after 16G of dd (the stick is 16G) there isn't
a significant amount of buffers lying around waiting to be flushed to the
stick (it's a slow USB, as suggested by the 60m the dd took, so if there
were a lot of buffers, we'd see more than .077 sec for the sync).

It's an invalid test because you hit the end of the device; there's
nothing left to sync.

[t***@tuxteam.de]

[rough dd test]

It's an invalid test because you hit the end of the device; there's nothing
left to sync.

This doesn't make sense to me. The question was whether dd writes "raw" to
the device or "buffered" (meaning that its output ends up in OS buffers to
be flushed later). Since the whole thing (dd + OS) can't look into the
future, stuff will end in buffers to be flushed if the second is the case,
regardless on whether dd "hits the end of the device" or not.

Those buffers would have to be flushed, thus showing up in sync's time.
They do not, therefore I conclude that we have the first case.

What am I missing?

Cheers

[Jeffrey Walton]

On Fri, Sep 13, 2024 at 8:00 AM Anssi Saari

Post by Anssi Saari

Post by Anssi Saari
Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful? I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable I found for example
https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem
and so arrived to the conclusion that the final close(2) call on a block
device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

To add a datapoint...

My daily driver workstation is really fast with lots of RAM. It has
3.4 GHz cpu and 64 GB of RAM. I also set swappiness to a low value to
avoid spilling out of RAM.

I use a lot of SBC's/dev boards for testing. They usually use a
SDcard. The SDcard is really slow. The card can only provide 10 MB/s
or 30 MB/s write speeds. Some cards I use are so cheap they are even
slower.

If I dd an image from the workstation to the SDcard, it happens in
under a second. dd exits, and closes its file descriptors. Something
is obviously wrong since the image is 1 or 2 GB, and the SDcard write
speed is 10 MB/s or 30 MB/s.

The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

What I found is, I have to manually call sync to ensure the image is
written from cache to the SDcard. When I call `dd if=... of=/dev/sdd
&& sync`, the command takes 30 seconds or so to complete. The time is
spent in sync, not dd.

Based on my experience with lots of RAM and slow media, you have to
call sync to get the cache manager to write back to the disk.

Jeff

[David Christensen]

Post by Jeffrey Walton
To add a datapoint...
My daily driver workstation is really fast with lots of RAM. It has
3.4 GHz cpu and 64 GB of RAM. I also set swappiness to a low value
to avoid spilling out of RAM.
I use a lot of SBC's/dev boards for testing. They usually use a
SDcard. The SDcard is really slow. The card can only provide 10 MB/s
or 30 MB/s write speeds. Some cards I use are so cheap they are even
slower.
If I dd an image from the workstation to the SDcard, it happens in
under a second. dd exits, and closes its file descriptors. Something
is obviously wrong since the image is 1 or 2 GB, and the SDcard write
speed is 10 MB/s or 30 MB/s.
The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove",
I mean pop the card out of the card reader since the write has
supposedly finished.
What I found is, I have to manually call sync to ensure the image is
written from cache to the SDcard. When I call `dd if=... of=/dev/sdd
&& sync`, the command takes 30 seconds or so to complete. The time is
spent in sync, not dd.
Based on my experience with lots of RAM and slow media, you have to
call sync to get the cache manager to write back to the disk.

That makes me think there is a bug in the secure digital card firmware,
the Linux secure digital card device driver, and/or the Linux file
system driver (?).


Have you seen a dd(1) invocation work correctly on a SD card without any
synchronization options and/or trailing sync(1)?

Post by Jeffrey Walton
On the other hand, [running sync after dd] may not be necessary, but
it doesn't do any harm. If it is really unnecessary, it will probably
cost only a fraction of a second to do. And if it is actually
necessary, you should do it, no matter how long it takes.

+1


David

[Anssi Saari]

Post by Jeffrey Walton
The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

It'd be interesting to know if you have caching related kernel messages
about your SD cards when plugged in? I did have from my USB SD reader.

I remember I have a USB-SSD case which demonstrated this situation,
super fast writes to a block device. I may dig that up for some testing.

In my little experiments with media on hand I found only uncached writes
this weekend. Test devices were one old 1 GB SD card, one newish 64 GB
SDXC card, one old 4 GB USB stick, two SD card readers, one USB and one
PCI-connected built in to a laptop.

Interestingly with the old memory stick there's a kernel message which
says outright write caching is write through which seems to indicate
synchronous writes.

[3675744.043264] sd 6:0:0:0: [sdc] No Caching mode page found
[3675744.048781] sd 6:0:0:0: [sdc] Assuming drive cache: write through
[3675744.113885] sdc: sdc1
[3675744.139820] sd 6:0:0:0: [sdc] Attached SCSI removable disk

Writes to that stick were about 6 MB/s as reported by dd.

With my USB SD card reader there was a similar message:
[17740.817424] sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

Laptop's built in PCI-connected SD card reader didn't say anything about
cache.

Write speeds reported from dd were pretty much as expected, from a few
MB/s to a few dozen with the SDXC card.

[t***@tuxteam.de]

Post by Anssi Saari

Post by Jeffrey Walton
The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

Actually, dd doesn't "do" file system cache (unless you are dd'ing
to a file, of course).

Cheers

[Anssi Saari]

Post by t***@tuxteam.de

Post by Anssi Saari

Post by Jeffrey Walton
The file system cache is still holding the writes. If I remove the
SDcard and try to use it, the image is corrupt. When I say "remove", I
mean pop the card out of the card reader since the write has
supposedly finished.

All right. So in fact, block devices either have write back write
caching or they don't and apparently the device driver decides
that. hdparm may be able to control caching but I don't know it works
for USB devices.

You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

I think Jeffrey's post was pretty clear in that it was not about writing
to a file system. Notwithstanding the words "file system cache" in that
quote above which I think that was just a mistake.

[t***@tuxteam.de]

[...]

Post by Anssi Saari

Post by t***@tuxteam.de
You are talking device cache. Jeffrey is talking file system cache.
Two different (and independent) layers.

I think Jeffrey's post was pretty clear in that it was not about writing
to a file system. Notwithstanding the words "file system cache" in that
quote above which I think that was just a mistake.
[3675744.043264] sd 6:0:0:0: [sdc] No Caching mode page found

and so on talk device cache, not FS cache. To avoid confusion, it is
important to keep them apart.

(Still, if you yank the device out while its caches are dirty, you might
get corruption, depending on how those react to power loss).

Cheers

[Michael Stone]

Post by t***@tuxteam.de
Actually, dd doesn't "do" file system cache (unless you are dd'ing
to a file, of course).

Incorrect; linux block device accesses cache unless specifically asked
not to. This is very easy to demonstrate: run something like "dd
if=/dev/nvme0n1 bs=1M count=1000" twice, and you'll see that the second
run is significantly faster than the first. (Correct for a block device
you actually have, and this assumes that you have more than 1G of
unused RAM available for cache.)

[Rick Thomas]

On Fri, Sep 13, 2024, at 5:00 AM, Anssi Saari wrote:
...

Post by Anssi Saari
So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

On the other hand, it may not be necessary, but it doesn't do any harm. If it is really unnecessary, it will probably cost only a fraction of a second to do. And if it is actually necessary, you should do it, no matter how long it takes.

Rick

[David Wright]

Post by Anssi Saari

Post by Anssi Saari
Why do you think that? Which part of the fsync manpage explicitly covers
fsync's effect on device files? Share share, it's fair.

“fsync() transfers ("flushes") all modified in-core data of (i.e.,
modified buffer cache pages for) the file referred to by the file
descriptor fd to the disk device (or other permanent storage device)
so that all changed information can be retrieved even after the
system crashed or was rebooted. This includes writing through or
flushing a disk cache if present. The call blocks until the device
reports that the transfer has completed. It also flushes metadata
information associated with the file (see stat(2)).”

You just parroted a man page I had already read. Why did you think
that'd be helpful?

Um, because you asked me too, above.

Post by Anssi Saari
I asked the questions because the man page didn't
answer my questions. As you are apparemntly unable I found for example
https://unix.stackexchange.com/questions/473854/block-device-cache-v-s-a-filesystem

I see that I looked at that page back in May 2021 (the page has been
updated since then), when the list had a previous conversation on dd,
partly about blocksize, but in which we were told that we were only
meant to be using dd with tapes or when converting, say, EBCDIC to
ASCII, or whatever.

I've yet to find another tool that can give you progress when already
in progess.

I had landed on a different page from the above this time around,
https://stackoverflow.com/questions/58276098/fsync-with-raw-device-fd
but I couldn't determine a definitive answer there. There still seems
to be some disagreement in this thread.

Post by Anssi Saari
and so arrived to the conclusion that the final close(2) call on a block
device already flushes all buffers before returning. So the answer to
the question "is running sync needed after dd to block device" is
no. Someone else posted that too on this list recently, in another
thread.

Well, if you knew that already, you could have just posted your view
rather than "Share share, it's fair".

Of course, we all know that the time it takes to add ;sync
to a command line is hardly going to delay you more than a
suitably sized choice of blocksize, so much of this discussion
is moot in practical terms.

Cheers,
David.

[Michael Stone]

These two tests are identical, as dd has exited (and all the performance
information has been printed) before sync runs. The flag you want[1] is
conv=fsync which runs sync before exiting, thus making the stats reflect
the time taken to flush to disk.

Mike Stone

[1] insofar as you want to use dd at all; as I've said before, it's
almost never the best tool for the task unless you're trying to reblock
data from a mainframe 3490 tape to a unix 9 track or somesuch. I haven't
needed to do that in almost 25 years. In general cp will provide the
best possible performance using familiar parameters and without the need
to try to understand how to apply flags based on a 1960s-era mainframe
directive to a modern unix system. I suppose if you enjoy impressing
people with obscure command syntax (and if you know people who can be
impressed by same) dd is its own reward.

[Anssi Saari]

[1] insofar as you want to use dd at all ...

dd does conveniently provide progress and stats whereas cp does not.

[Michael Stone]

Post by Anssi Saari

[1] insofar as you want to use dd at all ...

dd does conveniently provide progress and stats whereas cp does not.

stats which are misleading without the proper arguments, so it's a bit
of a wash. This is actually a hard problem because what people expect as
the desired output varies considerably depending on the task at hand.
There are a number of tools which will do variations of this, without
the baggage of dd. (E.g., pv)