The command you're probably thinking of is sync.

- Nate

--
"The optimist proclaims that we live in the best of all
possible worlds. The pessimist fears this is true."
Web: https://www.n0nb.us
Projects: https://github.com/N0NB
GPG fingerprint: 82D6 4F6B 0E67 CD41 F689 BBA6 FB2C 5130 D55A 8819


-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQSC1k9rDmfNQfaJu6b7LFEw1VqIGQUCaI7BBQAKCRD7LFEw1VqI GbipAJ9rNBaoO8GGOkzEf27eBpIKnp8lPQCghchMjzuxqLqY2v3rm+w6DDw42X0=
=NQ9M
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 8/2/25 20:53, Nate Bargmann wrote:

The command you're probably thinking of is sync.

Thanks, I added sync to my bash notes. I had looked at man sync earlier
today and thought that's not it...

However it turned out to be a hardware issue. This was my first time
using that new flash drive. I discovered I get that "device offline
error" *before* unplugging the usb, whether or not I used sync. I tried
another usb flash drive and that one works right every time and I didn't
need to use sync. For whatever reason "Disk Image Writer" worked better
with the bad drive.


--

Titus Newswanger
Curtiss WI

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 8/2/25 18:33, Titus Newswanger wrote:

I recall reading somewhere how to send cached writes to disk using a
shell command before unplugging a usb flash drive but now I'm failing to
find it. Below follows why I think I need that:

Today I installed trixie, everything worked great except for a problem I
ran into preparing the installer media. I usually run:

default@debian:~$ sudo dd if=/home/default/debian-trixie-DI-rc2-amd64- netinst.iso of=/dev/sdf status=progress bs=1M
[sudo] password for default:
778+0 records in
778+0 records out
815792128 bytes (816 MB, 778 MiB) copied, 242.327 s, 3.4 MB/s default@debian:~$ cat /etc/debian_version
12.11
default@debian:~$ dd --version
dd (coreutils) 9.1

Strangely, it did not display status updates like it used to until after
it completed. The disk did not even show up in thunar until after I
unplugged then plugged it back into the usb port. Next strange thing was
when I mounted the finished boot media, the directory contents looked
normal, but the text files - readme, changelog, etc contents were garbled.

Attempted boot, grub came up with the usual options, I selected
graphical install (the default) then kernel panic, with something like "initrd contained junk"

SHA512SUM on debian-trixie-DI-rc2-amd64-netinst.iso matched so the
problem lies somewhere in my writing the media

retried with:

sudo dd if=/home/default/debian-trixie-DI-rc2-amd64-netinst.iso of=/dev/
sdf status=progress bs=1M conv=sync

same result

Next I performed some tests on my brand new very cheap usb 8GB flash
drive = all ok.

Finally, I wrote my iso file to this flash drive using "Disk Image
Writer" and everything worked like a charm. So now I wonder, what am I
doing wrong with dd? I recall having a similar failure maybe two years
ago. The rest of the time it just worked.

uh-oh here I found a clue... so I think this makes it seem like
something wasn't finished writing when I pulled it from the usb.

sudo dmesg
[ snip ]
[40798.311928] usb 2-1.2: new high-speed USB device number 27 using
xhci_hcd
[40798.416473] usb 2-1.2: New USB device found, idVendor=abcd, idProduct=1234, bcdDevice= 1.00
[40798.416491] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[40798.416500] usb 2-1.2: Product: UDisk
[40798.416508] usb 2-1.2: Manufacturer: General
[40798.416515] usb 2-1.2: SerialNumber: 2312182347284409360901
[40798.417989] usb-storage 2-1.2:1.0: USB Mass Storage device detected [40798.418491] scsi host7: usb-storage 2-1.2:1.0
[40799.444551] scsi 7:0:0:0: Direct-Access     General UDisk 5.00 PQ: 0 ANSI: 2
[40799.444763] sd 7:0:0:0: Attached scsi generic sg5 type 0
[40799.445225] sd 7:0:0:0: [sdf] 15728640 512-byte logical blocks: (8.05 GB/7.50 GiB)
[40799.445333] sd 7:0:0:0: [sdf] Write Protect is off
[40799.445335] sd 7:0:0:0: [sdf] Mode Sense: 0b 00 00 08
[40799.445453] sd 7:0:0:0: [sdf] No Caching mode page found
[40799.445455] sd 7:0:0:0: [sdf] Assuming drive cache: write through [40799.446941]  sdf: sdf1 sdf2
[40799.447110] sd 7:0:0:0: [sdf] Attached SCSI removable disk
[41167.136643] usb 2-1.2: reset high-speed USB device number 27 using xhci_hcd
[41167.236475] usb 2-1.2: device firmware changed
[41167.236574] usb 2-1.2: USB disconnect, device number 27
[41167.252396] blk_print_req_error: 1324 callbacks suppressed
[41167.252405] device offline error, dev sdf, sector 1384200 op 0x1:
(WRITE) flags 0x800 phys_seg 1 prio class 2
[41167.252423] buffer_io_error: 36796 callbacks suppressed
[41167.252426] Buffer I/O error on dev sdf, logical block 173025, lost
async page write
[41167.252625] device offline error, dev sdf, sector 1384216 op 0x1:
(WRITE) flags 0x104000 phys_seg 30 prio class 2
[41167.252635] Buffer I/O error on dev sdf, logical block 173027, lost
async page write
 [ deleted 7 similar lines ]
[41167.252669] Buffer I/O error on dev sdf, logical block 173035, lost
async page write
[41167.252707] device offline error, dev sdf, sector 1384456 op 0x1:
(WRITE) flags 0x100000 phys_seg 1 prio class 2
 [ deleted 6 similar lines ]
[41167.253272] device offline error, dev sdf, sector 1385296 op 0x1:
(WRITE) flags 0x100000 phys_seg 19 prio class 2
[41167.348915] ldm_validate_partition_table(): Disk read failed. [41167.348941] Dev sdf: unable to read RDB block 0
[41167.348961]  sdf: unable to read partition table
[41167.536322] usb 2-1.2: new high-speed USB device number 28 using
xhci_hcd
[41172.740486] usb 2-1.2: device descriptor read/64, error -110 [41172.948721] usb 2-1.2: New USB device found, idVendor=1e3d, idProduct=198a, bcdDevice= 1.00
[41172.948739] usb 2-1.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[41172.950112] usb-storage 2-1.2:1.0: USB Mass Storage device detected [41172.950452] scsi host7: usb-storage 2-1.2:1.0
[41194.784951] usb 2-1.2: reset high-speed USB device number 28 using xhci_hcd
[41480.585883] usb 2-1.2: USB disconnect, device number 28

Thank you for posting your console session. Accurate information helps.


Assuming your ISO is correct and that you have downloaded the
corresponding SHA512SUMS file, here is a console session from when I
burned a Debian 11.3.0 amd64 ISO to a USB flash drive:

1. Find the size of the ISO file in bytes -- size is 396361728 bytes:

2022-04-29 22:38:53 root@tinkywinky ~/hardware/adata/usb-flash-drive/***REDACTED***
# ll /home/dpchrist/samba/dpchrist/iso/debian/11.3.0/debian-11.3.0-amd64-netinst.iso

-rwxr-xr-x 1 dpchrist dpchrist 396361728 2022-04-28 21:05:05 /home/dpchrist/samba/dpchrist/iso/debian/11.3.0/debian-11.3.0-amd64-netinst.iso

2. Convert size into a convenient block size and count for use with
dd(1) -- block size is 1M (2**20 bytes) and count is 378 blocks:

2022-04-29 22:39:14 root@tinkywinky ~/hardware/adata/usb-flash-drive/***REDACTED***
# perl -e 'print 396361728/512/2048, "\n"'
378

3. Burn ISO to USB flash drive -- note bs, iflag, and oflag options.
Today, I do not believe the oflag "noatime" option is needed:

2022-04-29 22:39:25 root@tinkywinky ~/hardware/adata/usb-flash-drive/***REDACTED***
# time dd if=/home/dpchrist/samba/dpchrist/iso/debian/11.3.0/debian-11.3.0-amd64-netinst.iso
of=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_***REDACTED***-0\:0 bs=1M iflag=fullblock oflag=sync,noatime status=progress
394264576 bytes (394 MB, 376 MiB) copied, 76.1204 s, 5.2 MB/s
378+0 records in
378+0 records out
396361728 bytes (396 MB, 378 MiB) copied, 76.5701 s, 5.2 MB/s

real 1m16.582s
user 0m0.012s
sys 0m0.584s

4. Compute the SHA512 checksum of the blocks burned to the USB flash
drive -- note bs, count, and iflag options:

2022-04-29 22:43:56 root@tinkywinky ~/hardware/adata/usb-flash-drive/***REDACTED***
# time dd
if=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_***REDACTED***-0\:0 bs=1M count=378 iflag=fullblock | sha512sum
378+0 records in
378+0 records out
396361728 bytes (396 MB, 378 MiB) copied, 25.0641 s, 15.8 MB/s 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53
-

real 0m25.068s
user 0m3.468s
sys 0m0.720s

5. Verify the computed SHA256 checksum appears in the downloaded
SHA512SUMS file:

2022-04-29 22:44:58 root@tinkywinky ~/hardware/adata/usb-flash-drive/***REDACTED***
# grep 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53
/home/dpchrist/samba/dpchrist/iso/debian/11.3.0/SHA512SUMS 2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53
debian-11.3.0-amd64-netinst.iso


HTH,

David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 8/2/25 22:44, David Christensen wrote:

5. Verify the computed SHA256 checksum appears in the downloaded
SHA512SUMS file:

I've been meaning to learn how to sha512sum after it is written to disk.
Now I've got it. Here are my results:

on the older usb disk that worked, sha512sum matched

on the new faulty disk, after writing with dd, sha512sum did not match

on the same faulty disk, after writing with "Disk Image Writer" and I successfully installed trixie on a machine, I checked the usb disk and sha512sum did not match 🙁 I thought I was finished with that server.
Now I need to decide if I will start over or just leave it and hope for
the best. Considering it is a mini pc used as a headless print server to connect a little used usb only printer to my network, I think I will
just place a note in /etc/motd like:

This copy of Debian was installed with a broken installer, expect anything!

Thanks

--

Titus Newswanger
Curtiss WI

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, Aug 02, 2025 at 10:49:44PM -0500, Titus Newswanger wrote:

On 8/2/25 20:53, Nate Bargmann wrote:

The command you're probably thinking of is sync.

I always recommend to add "oflag=sync" to dd itself: this way it
syncs as it goes and you don't have to wait for a (potentially
long) time for sync to "come back".

Together with "status=progress" you get a visual feedback on how
things are going (without having to kill -USR1 the thing all the
time :)

So:

dd if=<mysource> of=<mydest> oflag=sync status=progress

When that's done, it's done (you can throw a sync after it to feel
better, but it will come back immediately anyway :-)

Cheers
--
tomás

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCaI75FgAKCRAFyCz1etHa RmesAJ9pFn+9Xun+DUP+DdztaU9CXJf5nACfVC5svGjq8pKAzu7uZQEPy/UOcBU=
=wfPj
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Titus Newswanger (HE12025-08-02):

Strangely, it did not display status updates like it used to until after it completed.

The rest has been explained, but not that. Let me.

First dd wrote all the data extremely fast. We can assume the input file
was recently downloaded and still in memory cache. And the output went
directly into memory cache for asynchronous writing. It all happened in
less than one second, so status=progress was not triggered even once.

That tell us your computer almost certainly has more than two gigaoctets
of memory, which nowadays is not that surprising.

Then dd closed its input file. Nothing much happened.

Then dd closed its output file. If it was a regular file, nothing much
would have happened. But this time it was a block device, and
furthermore a block device that nobody else kept open. In that case, the
kernel performs the equivalent of a fsync as part of the close: the
close returns only when the buffers have been written to the device.

Finally, dd printed its final statistics, taking the time for the close
into account, thus ruining the impressive speed of the reads and writes.

Note it is not a feature of dd, the same would have happened with cat or
any other writing program (I like pv to have a progress bar and
sometimes ETA estimate).

As Tomas pointed, with dd specifically you can use oflag=sync to have it
sync explicitly between each block, to get a better progress estimate.
Be sure to use a large block size or you will ruin performances.

Regards,

--
Nicolas George

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Titus Newswanger composed on 2025-08-02 23:50 (UTC-0500):

I've been meaning to learn how to sha512sum after it is written to disk.
Now I've got it. Here are my results:

on the older usb disk that worked, sha512sum matched

on the new faulty disk, after writing with dd, sha512sum did not match

on the same faulty disk, after writing with "Disk Image Writer" and I successfully installed trixie on a machine, I checked the usb disk and sha512sum did not match 🙁 I thought I was finished with that server.
Now I need to decide if I will start over or just leave it and hope for
the best. Considering it is a mini pc used as a headless print server to connect a little used usb only printer to my network, I think I will
just place a note in /etc/motd like:

This copy of Debian was installed with a broken installer, expect anything!

Next time, consider the alternative to dd that I use:

time ddrescue downloads/win10_22H2x64.iso /dev/sdh --force && sync

ddrescue output is verbose by default, and extremely rarely fails to write successfully even on a poor quality stick.
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.

Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun Aug 3, 2025 at 6:52 AM BST, tomas wrote:

I always recommend to add "oflag=sync" to dd itself: this way it
syncs as it goes and you don't have to wait for a (potentially
long) time for sync to "come back".

Together with "status=progress" you get a visual feedback on how
things are going (without having to kill -USR1 the thing all the
time :)

A bugbear of mine is people recommending `dd` and using it without
any customisation e.g. `dd if=iso of=/device`, which is harder to
type than `cp iso /device`, and runs slower.

Your suggested `dd` flags demonstrate why it can be a better choice
than simple `cp` for this use-case. Thanks!

--
Please do not CC me for listmail.

👱🏻 Jonathan Dowland
✎ jmtd@debian.org
🔗 https://jmtd.net

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Aug 03, 2025 at 11:58:59AM +0200, Nicolas George wrote:

[...]

As Tomas pointed, with dd specifically you can use oflag=sync to have it
sync explicitly between each block, to get a better progress estimate.
Be sure to use a large block size or you will ruin performances.

Oh, yes, forgot that one, thanks for mentioning. After some experimenting,
I settled with bs=4M (note that this is for USB 3, and for some specific computer and stick, some experimenting pays off): it somewhat "flats off" beyond that point.

The default, 512 bytes, is *definitely* too small: from there to 4M there
was a factor of roughly 8 in troughput for me.

So, as Nicolas says, throw in a "bs=4M" (or 1G or whatnot) into the mix.

Cheers
--
tomás

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCaI85aQAKCRAFyCz1etHa RkKeAJ4tNtBQUCMdzqtpAh8ELAnB0xPw/ACfSFAodwkmRGaF008MHfG5MaHC1Y4=
=hasC
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* On 2025 03 Aug 05:31 -0500, Jonathan Dowland wrote:

On Sun Aug 3, 2025 at 6:52 AM BST, tomas wrote:

I always recommend to add "oflag=sync" to dd itself: this way it
syncs as it goes and you don't have to wait for a (potentially
long) time for sync to "come back".

Together with "status=progress" you get a visual feedback on how
things are going (without having to kill -USR1 the thing all the
time :)

A bugbear of mine is people recommending `dd` and using it without
any customisation e.g. `dd if=iso of=/device`, which is harder to
type than `cp iso /device`, and runs slower.

Curiously, I recall trying "cp" in the past and not having satisfactory results.

Your suggested `dd` flags demonstrate why it can be a better choice
than simple `cp` for this use-case. Thanks!

Meanwhile "dd" has always worked for me. I'll have to remember Tomas' recommendation for "oflag=sync" for the next time I write an image,
though that might be a while.

- Nate

--
"The optimist proclaims that we live in the best of all
possible worlds. The pessimist fears this is true."
Web: https://www.n0nb.us
Projects: https://github.com/N0NB
GPG fingerprint: 82D6 4F6B 0E67 CD41 F689 BBA6 FB2C 5130 D55A 8819


-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQSC1k9rDmfNQfaJu6b7LFEw1VqIGQUCaI9LFQAKCRD7LFEw1VqI GUCEAJ9nHO1TtHrJnU+e9Zj5yUcBFJ1+cgCbBeas9H1DFzhEUvObjhYxY9UYzyo=
=gY+M
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, Aug 03, 2025 at 06:42:13AM -0500, Nate Bargmann wrote:

[...]

Meanwhile "dd" has always worked for me. I'll have to remember Tomas' recommendation for "oflag=sync" for the next time I write an image,
though that might be a while.

I usually just remember "there was a flag for that... hmmm..." and
then check my notes (or the man page).

Cheers
--
t

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQRp53liolZD6iXhAoIFyCz1etHaRgUCaI9NhAAKCRAFyCz1etHa RvaOAJwMo10eUcVGaI5Jao5rNpXpUj6BHgCfROU3SG8iCSRKj9PY/vf810yaUT0=
=mwMk
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)