1. Forum
  2. Usenet
  3. LINUX.DEBIAN.USER

  • Problems with big data transfer and partitions? - suggestion!

    From Hans@21:1/5 to All on Thu Jan 23 16:20:01 2025
    Hi folks,

    in the last weks there were several issues with data transfer from ext4 to exfat. Most cases wanted to be done, to transfer seceral terrabyte of date to
    a MS-Windows system.

    Thinking of it, IMO this is a bad choice. Olease let me explain:

    Besides to host tzerrabyte of important data on a windows system (which I
    never would do!), I see several issues when using harddrives for trasfer.

    First, you have to copy (let us say for example: 4 TB) via USB3 to a
    harddrive not in ext4. You may use rsync, but this will last very long.

    Then you will have to copy 4TB to the windows-system via usb3, whcih will last again long.

    So first bad thing: You have to copy twice and the copytime is twice, too.

    Second: USB has no correction in its transfer protocol, so there migjht be erros during tarnsfer. Yes, when using rsync they may be corrected.

    Third: the harddrive for transport may have physical or logical defects, which may cause problems.

    Fourth: exfat (needed or big files) does not have a journal like ext3 or ext4, so data may be going corrupt on the harddrive and could not be restored.

    Sixth: USB3 is (IMO) very slow against gigabit.

    All these should be thought of before one goes to work.

    A better solution IMO is, take the computer to the target host and connect it via gigabit network. Most notebooks and desktop computers and of course most modern servers already have gigabit network.

    Then connect to the target host and use rsync for tarnsfer. Yes, there is
    rsync available for windows, it is called grsync. Even it is not much secure,
    I see there no problem, because the target can be airgapped during transfer.

    The advantage of doing so:

    - gigabit is much faster than usb3, saves much time
    - you only have to copy the data once, saves more time
    - you have much lesser error sources
    - you do not to hassle with several partition formats
    - using grsync you have correction during transfer
    - all in all, you will have much less pain

    This came in my mind during the days and I wanted to share it with you, maybe it will be usefull for one or another.

    Last but not least: I may be completely wrong with my thoughts, so please
    don't blame me.

    Best regards

    Hans

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Stone@21:1/5 to Hans on Thu Jan 23 17:20:02 2025
    On Thu, Jan 23, 2025 at 04:16:29PM +0100, Hans wrote:
    Fourth: exfat (needed or big files) does not have a journal like ext3 or ext4, >so data may be going corrupt on the harddrive and could not be restored.

    That's not what a journal is for, and if the copy completes and the disk
    is unmounted the data is fine. The journal is there for cases when the filesystem isn't shut down properly, as in a loss of power or a system
    crash. Some implementations will journal data, but most only journal
    metadata. A metadata journal's main purpose is to avoid having to run
    fsck; the filesystem will end up in essentially the same state whether
    using a journal or using fsck, but the journal is much, much faster.

    Sixth: USB3 is (IMO) very slow against gigabit.

    This is incorrect: USB 3.0 is 5Gb/s, and various implmentations of USB
    3.1 or 3.2 can run at 10Gb/s or 20Gb/s. Copying a large amount of data
    via an external drive can be many times faster than copying via the
    network, depending on the speed of the drive and the speed of the
    network.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)