Split & Cat Large Files

We still live in a 32-bit world.  This mean if you deal with large file transfers, you will undoubtedly bump up against the 4 GB limitation (2^32) posed by legacy file systems and network infrastructure.  One way of skirting around the 4 GB limitation is to use the Unix commands split and cat to split up large files into smaller chunks.

For example, if you’re trying to save a 5 GB file (e.g. 5gb_largefile.iso) to a USB drive formatted for the FAT32 filesystem (max file size of 4 GB), you can split and save the file as 3 files of 2 GB or less with the following command:

split -b 2048m 5gb_largefile.iso 5gb_largefile.iso_

to yield the following files:

  • 5gb_largefile.iso_xaa (2 GB)
  • 5gb_largefile.iso_xab (2 GB)
  • 5gb_largefile.iso_xac (1 GB)

When you’re ready to utilize the file, you can combine the 3 files back into the original file with the following command:

cat 5gb_largefile.iso_xa* > /tmp/5gb_largefile.iso

In this case, you need to be sure to write the output to a directory (e.g. /tmp on a Unix machine) that does not have the 4 GB limit.

Advertisements