@Jasper 
Are you starting to (planning to) boot completely from usb drive?
Is this a UEFI secure boot computer? If so some additional things will need to be done.
If YES then preparing the USB drive using the tools and utilities from the LiveCD Fossapup64 will be the recommended way.
If NO it would be possible to boot and run the LiveCD and store and recall a system persistence folder or file on the USB drive. This save folder/file (the choice here depends on the usb's partitioning and formatting) will be looked for and loaded by the Fossapup64 LiveCD. Puppy Linux will guide you and set up the persistence mechanism.
For a USB stick that one can plug in and the computer will boot from instead of the internal HDD, first let's try to use the utility called FrugalPup which should get the contents of the LiveCD ISO file and create what is known as a "frugal install". Which is the recommended method to install a Puppy Linux. The "frugal" installation will bring all of Fossapup64's special features into play. The program can be started from the Menu->Setup->FrugalPup. Once this is running you will see the options. You can use the Fossapup64 LiveCD ISO file for all of the needed base files for Fosspup64 when that option selection comes up.
I personally would boot up Fossapup64 LiveCD then plug in the USB drive. Use Menu->System->GParted to set up a small FAT32 partition with the boot flag set, and an ext4 using the rest of the usb's storage space. I would open the Fossapup ISO file in a Rox file manager window and make a directory called Fossapup64 in the ext4 partition and copy the entire contents of the ISO file into that directory.
Then run Menu->Setup->Grub4Dos select the USB drive and install the boot loader and boot menu system.
All of this sounds like a lot but once you see it all in action it's not that complicated. I probably missed a bunch of details but this is round about how it will go.
It's worth it.......once you work with and use a quick, responsive OS like Fossapup64 it quickly becomes apparent why you would.