Pupsave-backup: Several Issues Identified (need testing of my PuppyHotSave that fixes issues)

[mow9902]

Using F96 CE (v3) frugal.
I previously posted about a couple of minor issues in pupsave-backup https://www.forum.puppylinux.com/viewtopic.php?t=8414 but since then I have noticed something more important - so created this new topic.

I believe there are 4 issues with the pupsave-backup script - 3 of these are easily fixed and I have created a solution for each of them. However, I am absolutely sure that my solution could be replaced with a more elegant solution by someone with greater scripting skill than I have.

The 4th issue is something for which I do not understand the cause, and therefore not something I can fix. It requires analysis by someone with a much greater level of knowledge than myself. I believe everyone who relies on these backups to restore their system when needed is at RISK!

Issue:

1. When the script is run an initial message "saving session" is displayed when the snapmergepuppy command is executed before the backup commences. On my system, this message remains displayed even after the backup is completed successfully. It should close.

   • I have made a change to remove this message once the backup actually starts.

1. When the initial GUI is launched it shows an "Information" section at the bottom which includes the name which will be given to the backup file which is to be created. The display of the name is truncated if the save file to be backed up has a long name.

   • I have made a change to allow double height for the display of this save file name, and it will wrap to fit.

1. The user can choose to also compress the backup file. But, if this option is chosen, the user selected backup location path is ignored, and the file is created in /mnt/home regardless of the selected location.

   • I have made a change to ensure the user selected backup location is used.

I have attached a copy of my amended pupsave-backup script. All changes made have been commented with ##mow9902.
This might be a 'thought starter' for an experienced scripter to make some improvements.

4. MOST IMPORTANT
The whole purpose of creating a backup is to be able to use it to recover your system in the event that the current save file/folder becomes corrupted for some reason.

I have tested this using a non-compressed backup folder from this script (and also creating a backup myself manually with tar) and all is good.

BUT - not the same with a compressed backup folder.
After many,many,many attempts using both this script, and also manual 'compression' commands from the terminal (including the use of tar, and mksquashfs), I have found that if I 'extract' the backup save folder from the compressed folder created, rename it, move it into /mnt/home and then select it when rebooting the system, the resulting system is SCREWED up. It boots ok, and on first glance appears normal - but it immediately becomes obvious there is some significant problems - quickly identified because there is no 'sound' icon in the system tray, and the firewall is turned off. Attempts to start pulse or the firewall produce errors. All of this is WAY beyond my ability to diagnose.

I have tried tar to a ext2 disk and a NTFS disk - both with with the same result.
I have tried first copying savefile to a ext2 disk and a NTFS disk - and then doing tar on the copy - same result.

I don't know if this is just a problem on my system, or if it applies to all systems - but for now I have to conclude that the system cannot be successfully restored from a compressed backup file.

I really need someone to see if they can reproduce this problem, and/or give me some clue as to what I might be doing wrong.
Has anyone recently tested by rebooting their system from a compressed backup folder/file?

I apologize for the rather long post. Thanks in anticipation.

---

17 Oct 2024 - Attempted some improvements
23 Oct 2024 - Minor change - create lock file in application directory instead of /tmp (which may differ on some systems)

[mow9902]

An update on the issues I identified in my original post.

I was hoping that someone with sufficient system knowledge and coding skill would take up the challenge and revise the pupsave-backup script; but seems that is not going to happen anytime soon.

So - I decided to write a replacement myself. I have attached the script, and would really appreciate some peer review and testing. (to use, just download and change the file extension)

My knowledge/skills are not nearly as good as I'd like them to be and I know that this offering is much less concise and efficient as what it should be. However, it is:
1. fully documented internally and very easy to read
2. self contained and relies only on some inbuilt gtkdialog options
3. provides all of the functionality of the original script - and a little more. Just to summarize:

Pupsave-backup (the original)

• automatically identifies the save file in use

• allows the user to specify a destination directory for the backup

• allows the user to choose whether or not the backup should be compressed, and which compression to use

BUT
(a) requires the destination disk to be ext2,3,4 (not anything else)
(b) if either compression type is selected, then the backup is directed to /mnt/home instead of the nominated directory
(c) testing on my system shows that if I restore a save folder from a compressed backup I am able to boot my system, but the system appears to be at least partly corrupted

PuppyHotSave (My alternative)

• automatically identifies the save folder or file in use

• allows the user to specify a working directory

• allows the user to specify a destination directory

• allows the user to choose whether or not the backup should be compressed - using mksquash only (ie a .SFS file is produced)

AND
(a) does NOT require the destination disk to be any particular format
(b) creates the temporary working files and the final backup files in the directories specified by the user
(c) testing on my system shows that if I restore a save folder or save file from a compressed backup the system boots and works normally.

So - I am now using this myself, and have simply:
(a) placed the script into my /root/my-applications/bin directory
(b) amended my ~/.jwmrc file to replace the original script with this new one in the menu

Looking forward to some feedback. Thanks
.
Download and rename by removing .txt from name.
Right click on file and select properties.
Give it exec permissions.
Click on file to run script.

Thanks for the latest feedback from bigpup - I really appreciate it.

Updates:
As suggested by bigpup I have made one small change to include the name of the save folder/file in the backup file name to make it easy to see which folder/file is contained within the backup.

I have attached an amended version. Like before,
1. download the file PHS-GUI-v3.txt
2. change the name to PHS-GUI
3. set the properties to executable
4. clicked on the script PHS-GUI

PHS-GUI-v3.txt

PHS-GUI v3

(22 KiB) Downloaded 34 times

[bigpup]

It is kind of a good idea to use lower case letters in the names of files.

Keeps from people having to remember it is uppercase, if they type the name in a terminal, to run the script.

Believe me, you will appreciate this when trying to find out if there are any errors, when running a programs exec or run script.

[bigpup]

Tried using the PHS-GUI running in F96-CE_3

Sure this is the modified one attached for download?

When I try to use it. I got these errors.

requires the destination disk to be ext2,3,4 (not anything else) Refused to use a fat32 formatted location.

Location I choose to be the working directory has 168GB free space.
But get this error.
.

Screenshot.jpg (7.93 KiB) Viewed 1416 times

.
The save file is only 4096 M in size.

Notice what that error message says it is 4095999 M

[Clarity]

Hi @bigpup

Dont know if this helps, but it says "Mb". That's the designation for Megabit...not Megabyte

[mow9902]

Tried using the PHS-GUI running in F96-CE_3

Sure this is the modified one attached for download?

When I try to use it. I got these errors.

requires the destination disk to be ext2,3,4 (not anything else) Refused to use a fat32 formatted location.

Location I choose to be the working directory has 168GB free space.
But get this error.
.
Screenshot.jpg
.
The save file is only 4096 M in size.

Notice what that error message says it is 4095999 M

@bigpup
I think you might have tried to execute the amended version of pupsave-backup and not this new rewritten script named PuppyHotSave which is run using PHS-GUI
After you download the file PHS-GUI.txt; rename it and remove the .txt extension; set properties to be 'executable'; and then run the script - it should look like this: (..obviously except for the parameters, which you change to suit your system)

PHS-GUI

PHS-GUI.png (56.41 KiB) Viewed 1395 times

[bigpup]

If you look again at my post.

I am using the PHS-GUI exactly as you say.

[mow9902]

I'm puzzled, because if you look at the screenshot in your earlier post you can see that the dialog box does not even have the name "PuppyHotSave" which is the name I use in all of the dialog boxes in my PHS-GUI script.
You can verify this by searching through the script yourself to see if you can find a dialog box with the title in your screenshot.

screenshot.png (52.84 KiB) Viewed 1326 times

I don't want to insult you - but are you sure you:
1. downloaded the file PHS-GUI.txt
2. changed the name to PHS-GUI
3. set the properties to executable
4. clicked on the script PHS-GUI

If you have done all of this, then can you please rename the copy of the script you download to "bigpup-copy.txt" and attach it to a reply which will enable me to see exactly what you are seeing.

[bigpup]

Asking questions is fine to make sure you fully understand.

Yes doing exactly what you say to do.

The PHS-GUI I have .

I think this is what generates the error message.
Notice the title in the code.

Line 274
# Issue message

Code: Select all

	Xdialog --icon /usr/share/pixmaps/error.xpm --wrap --title "Backup Save File" --msgbox " $msg Exiting" --wrap 50 5
	echo "Error - Insufficient Space on $bktemp"
	echo " "

also found this:
line 48

Code: Select all

	/usr/lib/gtkdialog/box_ok "Pupsave Backup" error "Puppy Hot Save Backup is already running."

.
.

bigpup copy PHS-GUI.txt

(19.46 KiB) Downloaded 30 times

.
.

[Sofiya]

i have no error

[mow9902]

Asking questions is fine to make sure you fully understand.

Yes doing exactly what you say to do.

The PHS-GUI I have .

I think this is what generates the error message.
Notice the title in the code.

Line 274
# Issue message

Code: Select all

	Xdialog --icon /usr/share/pixmaps/error.xpm --wrap --title "Backup Save File" --msgbox " $msg Exiting" --wrap 50 5
	echo "Error - Insufficient Space on $bktemp"
	echo " "

also found this:
line 48

Code: Select all

	/usr/lib/gtkdialog/box_ok "Pupsave Backup" error "Puppy Hot Save Backup is already running."

.
.
bigpup copy PHS-GUI.txt
.
.

Thank you - I stand corrected. I don't know why I did not see that myself. I will work on the fix tomorrow. Sorry for the back-and-forth.

[Chelsea80]

@ mow9902

Thankyou for PuppyHotSave -

Followed your instructions to the letter -

Not used it yet as I don't know if I HAVE to use:

Config:
Select Working Directory

I have no error -

PuppyHotSave.jpg (54.09 KiB) Viewed 1498 times

Edit -

I notice:

On mow9902 screenshot PuppyHotSave is next to SFS Symbol

On Sofiya screenshot PuppyHotSave is central

On Chelsea80 screenshot PuppyHotSave is next to SFS Symbol

Best regards

Chelsea80

[mow9902]

@bigpup @Chelsea80

Thanks for the feedback from bigpup and Chelsea80 - I really appreciate it.

I have made some changes to hopefully fix these issues.
Main changes:
(a) change formula for calculation of disk space and make clear that sizes reported are megabytes MB
(b) force use of compression option if destination disk is not a ext format
(c) make sure the title on all message windows is "PuppyHotSave Backup"

I have attached an amended version. Like before,
1. download the file PHS-GUI-v2.txt
2. change the name to PHS-GUI
3. set the properties to executable
4. clicked on the script PHS-GUI

Please try again.

PHS-GUI-v2.txt

v2

(21.49 KiB) Downloaded 44 times

[Chelsea80]

@ mow9902

Ok, downloaded Version 2 -

Very useful that you have now included "(b) force use of compression option if destination disk is not a ext format"

As most USB sticks come formatted as FAT this saves a re-format to ext -

I notice that when run it still says Version 1 and the title remains as PuppyHotSave -

Also I still don't know if I should use the Config: Select Working Directory or not -

It has obviously been included for a reason -

Is it purely a choice for functionality of RAM or a safety reason? -

If chosen where would that Working Directory be and what name would be used? -

Is the Working Directory automatically deleted when the programme is terminated before completion (by user or the script) or on completion of the programme?

I ask because within the script there are 'testing for' and 'verification of' sections for a Working Directory -

I guess, that like me, time for you is precious and 'a little and not so often' approach is taken -

With all that said I appreciate your efforts so far -

Best regards

Chelsea80

[mow9902]

@Chelsea80 - thanks for taking the time. My answers to your questions:

I notice that when run it still says Version 1 and the title remains as PuppyHotSave -

• It still says version 1 because at this point you (and hopefully some others) are reviewing for me. When any significant issues have been resolved then it will be the final version 1.

• Yes - the title is PuppyHotSave and that is what I intended it to be.

Also I still don't know if I should use the Config: Select Working Directory or not -

It has obviously been included for a reason -

Is it purely a choice for functionality of RAM or a safety reason? -

• The logic behind the "working directory" is that it is used as the interim destination directory, and then when the copy and (optional) compression is completed, the finalized backup is moved to the destination drive, and the "interim" copy is deleted from the working directory.
  I have done it this way because:

  • an interim directory would normally be on a local disk which may be faster and larger than the destination directory (which could be on a remote or networked attached disk) and this helps speed things up

  • the destination directory may not be an ext formatted disk, so, the backup copy and compression is always done in the working directory before the resultant .SFS file is moved to the destination directory.

If chosen where would that Working Directory be and what name would be used? -

• the working directory is simply any directory you wish to use on any mounted disk. Your choice.

Is the Working Directory automatically deleted when the programme is terminated before completion (by user or the script) or on completion of the programme?
I ask because within the script there are 'testing for' and 'verification of' sections for a Working Directory -

• No - since the working directory is a directory on your system which already exists, it is not deleted by my script.

• Correct - my script checks for the existence of a selected working directory and also makes sure that it is ext formatted

I hope this helps.

[Chelsea80]

@ mow9902

Thanks for the explanation, now I am clued up to the logic -

I was thinking of Config: Select Working Directory as a choice (use or not) -

Rather than it being a necessary part of the procedure -

Apparently there are different views on a Hot Save Backup (not your script) -

Refer to Topic > House Training > Users > How to use a backup save file?

So I guess it's horses for courses -

Best regards

Chelsea80

[bigpup]

@mow9902

We are all here to help.

You are the software developer and we are the testers.

This is normal Puppy software development.

Downloading your latest version now to give it a test.

Note:
It is best if the first post of this topic has the latest version as a download attachment.
You can always edit that first post.
If this topic turns into more than a few pages.
People are not going to spend the time looking thru pages of posts to find the latest version.
They will look at the first post of this topic and think what is in it for download is the latest version.

[bigpup]

Running F96-CE_4

PHS-GUI-v2

Seems to have made a backup of the save.

The file is named:
PHS-2023-04-22-23.32.sfs

OK.

But how do I know what save file it is a backup of?

Should not the name of the actual save file be in the name of the backup?

Something like:
PHS-fossapup64.2fs-2023-04-22-23.32.sfs

-----------------------------------------------------------------------------------------------------------------------------------------------------

When the programs main window was showing.

The information statements on what was being done were placed over it.

Would be best if the main window dropped to the tray and just the info statements showed until completed.

I moved the window and the stuff in it disappeared, because of interaction with the info statements display.

[mow9902]

Thanks bigpup

I have taken your suggestion of including the save folder/file name into the name of the backup file. Latest version now added to my original post (per your advice).

WRT to your suggestion about the "information" box overlaying the rest of the GUI - I'm not sure I can (or need) to change this.
I have included all of this information into this single script so that it is self contained. My coding skills are not good enough to make just the msg box display while the rest of the script is hidden.
Also - I can't really think of a situation where the user would need to have the "information" screen open while the backup is actually running. It's the sort of information you really only need to read once, either before or after the backup.

I'm happy to make the change if anyone can tell me how to do it.

thanks again

[mikewalsh]

@mow9902 :-

Although I'm strictly a copy/paste guy - I usually backup entire Puppy sub-directories from my main Puppy partition - I thought I'd take a quick look at this.

I have no idea if this is due to the somewhat convoluted set-up I have here - so many directories are sym-linked into other locations across the entire system! - but when attempting to make a backup in ANY Puppy (and these are ALL save-folders, OK?), despite having terabytes of available disk space AND the entire kennels - and attached drives - is formatted to ext3, I keep getting the messages that I haven't got enough disk space and that I'm using the wrong format?

Seems that perhaps there's still a few issues here that don't yet cover all "eventualities"...

I freely admit I'm no expert on this stuff; just trying to help out by giving some feedback here.

--------------------------------------

Also, to follow-up on Chelsea80's point about the GUI itself, not everything shows correctly for me until after a window re-size. This may be due to the fact that I run a globally implemented higher DPI setting; I have a 1920x1080 monitor here, but if I run Puppies in standard, OOTB trim I need to use a magnifying glass to see anything. Hence the need to make everything show larger..!

(I've found myself, when developing stuff, there's all sorts of 'edge' use-cases you have to take into account; just because something works on your own system is NO guarantee it will work for other folks to their satisfaction...nor that others will be 'happy' with the way you've implemented things. Being a 'dev' is a thankless task, believe me!)

As the old saw has it; "ye cannae please all the folks ALL of the time". There will always be those who are utterly convinced that you should have done everything in a totally different way..........and some of 'em get quite "vocal" with their fault-finding!

You soon find you need to develop quite broad shoulders....and to develop a "thick skin". Like a rhino's...

Mike.

[mow9902]

Thanks to those who have responded and tried it out.

It was something I put together for myself because the inbuilt Utilities > Pupsave Backup did not work for me.

This script I have built to replace it works flawlessly for me after more than 40 tests with both save folders and save files. I had hoped that by sharing it some others would also benefit.

I don't have the skill or 'system infrastructure' to take this any further - so this is where I will leave it - but before I do (just for curiosity) I'd like to have some idea why the check for disk format type is not working for you as it does for me.

@mikewalsh
This is the code I am using to perform that check: (let's assume we are looking at the working directory - and for this example it is /mnt/sda2)
(a) find the disk that this mount point is attached to
bktempmntdev=$( findmnt -n -o SOURCE --target "/mnt/sda2")

(b) find the format of that disk
fstype=$( findmnt "$bktempmntdev" | awk '{print $3}' | grep -v FSTYPE )

(c) look for the characters 'ext' in the first 3 positions of that output

If 'ext' if found I have judging it to be an ext format disk - otherwise not.

This is what it looks like on my system if you execute these commands in a terminal. What does it look like on yours?

fstype.png (37.4 KiB) Viewed 1265 times

[glene77is]

PuppyHotSave GUI v2
to mow9902
Thanks for trying your hand on this Application, "PuppyHotSave GUI v2 " labeled "PHS" .
Has been very interesting to read your code, your methods, your style.
You certainly have tried to be cover all issues involved
to backup "Frugal" saveDATA directory and "saveDATA.4fs" types of Puppy installations.
Have had much fun this week trying to re-engineer
to be a "Frugal/4fs" app, which I label "PCB" PupColdBackup.
PCB Handles both Frugal and 4fs, no questions or presets required.
and works on 22 Puppy distros.
...
Experimentally, using my "PCB" (Pup Cold Bak) ,
I have been able to cause the "nice -19' command to deliver a "data has been changed' message !
when running two compressions in quick sequence (almost two at a time), but not always.
As I interpret the results, Each instance of "PCB" talks to the other, because they are in the same "group"
( ie, same computer/server, sharing the same time ).
This observation is in line with the literature on "nice -19" which I have read..
This would violate the "non-interruptable" requirements for a "hot backup" protocol.
Further tests should be run using "ionice realtime", trying to totally prioritize / isolate the backup process.
There is always the "mv" command to rename the saveDATA entity, and execute your backup immediately,
thus dumping any interruptions to the bit-bucket, then requiring a reboot to restore the previous saveDATA. .
...
Current round of tests is with your "PHS" ( PuppyHotSave method) using the "copy-out / e2fsck / copy-back" method ,
( I suspect that in a networked environment
there would not be any 'safe' HotBackup ! possible
with everybody banging on the same permissions door.)
...
I set up a Puppy install in the file.4fs method for experiment with "PSH" .
Note: that approach is thrice longer (by the clock) and requires several extra GBytes RAM,
Tests on a 2GB computer with saveDATA.4fs crashed out , NO RAM left, locking/disabling the Puppy OS.
...
The thought came to mind that
you are using the multiple copy-out method , with e2fsck, and copy-back.
in order to capture the saveDATA quickly,
before a multi-user networked system might alter your saveDATA "during" capture of data.
( I suspect that in a multi-user network,
there would not be any 'safe' HotBackup !
with everybody banging on the same permissions door.)
***
Continuing:
Using only "cp" does not establish a "priority", so I used "nice -19" to introduce the "cp" et al.
Using your "PHS" , I have experimented with running a first PHS app and then starting a second PHS process,
both under "nice -19" control, and have noted that "nice -19" will try to re-prioritize
the first-vs-second PHS apps. Still, the two instances of "PHS" may contest/vie for permission to "Run Now!".
I noted that the 'first' running app may (possibly) give a message of 'data has been changed' ,
indicating that the 'second' "nice -19" app has truly interrupted the first running app.
Whether or not the "PHS" or "PCB" has been interupted , or possibly altered the data is not the issue,
What is to be noted is that the Puppy OS "detected" a "change in data" status,
( ie, the system possibly 'recognized' an "interrupt" in the system process).
***
Using my "PCB" approach depends on "nice -19" with tar.gz , on which I have read the literature.
"ionice realtime" will be used in some future test runs, as well as other "tar" options. .
I have opted to use the "nice -19" approach to give my app the "top priority" in a 'group' ,
This would require only a single group capture by "tar" and then a single compress by "gz".
I reasoned that this approach will be safe on my 'single-user' system (which is a single 'group').
where there are no privileged "interrupts" to my "nice -19" app.
ie, so, No one has networked privileged ability to interrupt any of my processes.
Almost true, but there are variations among my 22 running Puppy Distros.
I found that multiple instances of any program/app may contest/vie for permission to "Run Now!".
I suspect that the "cp" copy command , as implemented in your "PHS", is also subject to this constraint.
***
After 45 years, I am still finding interesting things happening in system code.
Just some more things i didn't know .
Point being, running several instances of a program/app may enable each to vie/contest for permission to "Run Now!" .
Thus one of the program/apps will generate a message of 'data has been changed'
which indicates that a "true" "Hot-Backup" may not be possible.
...
Some of my 22 running Puppy systems use base "/initrd/save/dev_save" , not "/mnt/home".
As I ran Your PHS app, across these many installations,
some runs with PHS failed to find that base address in the config file.
The error message was that the working subdir was "." , as if there was an empty variable.
So, I hand-entered "/mnt/home" in the config, and your app proceeded properly,
or I selected a more convoluted pathing to get "/mnt/home" .
I do not see any work-around, if this problem arises, ,
except to hand-enter the base address into the config, IF required.
But, this does not always correct the process, just 'usually'.
So, Xenial-8 and Tahr-6 refuse to run "PHS". But all 22 OS will run with the "tar.gz" method , file-Directory or file.4fs format.
***
But on this single-user workstation, I have the old "Tar.Gz" method to use, everywhere, all the time.
...
I experimented with NOT USING the "cd" command to repoint the system,
and quickly traced out where the system was pointing. Ugh! so obvious when tracing.
The "cd" from "/current" , to "/root" , and back to "/current" , is very necessary.
I could be wrong, but it makes sense that the compression code is designed to operate from "/root" .
...
So, that's "my" story/comment.
No suggestions for you.
You have done a good work developing things.
You are very capable of handling your own development.
...

[mow9902]

@glene77is

Thanks for the close examination of my app, and your experiments to make a better version. Some brief comments on a couple of the points you made:

You said:

Experimentally, using my "PCB" ,
I have been able to cause the "nice -19' command to deliver a "data has been changed' message !
when running two compressions in quick sequence (almost two at a time).
Each instance of "PCB" talks to the other, because they are in the same group
( ie, same computer, sharing the same time ).
This observation is in line with the literature on "nice -19" which I have read..
This would violate the "non-interruptable" requirements for a "hot backup" protocol.
Next round will be with the "copy out" method used in the PuppyHotSave method,
which should share the same computer and same time, thus being in the same 'group'.
( I suspect that in a network,
there would not be any 'safe' HotBackup ! possible
with everybody banging on the same permissions door.)

I had never thought of multiple users at the same time - so definitely not something I had catered for. I would be interested in what you eventually come up with - But I would think that for the vast majority of the time it would only be used by a single user at one time.

You said:

I set up a Puppy install in the file.4fs method for experiment with "PSH" .
Note: that approach is thrice longer (by the clock) and requires several extra GBytes RAM,
On a 2GB computer with saveDATA.4fs, it crashed out , NO RAM left, locking/disabling the Puppy OS.

Fair point and acknowledged. The available RAM and the size of the save folder/file is going to determine how viable the mksquashfs step is, and how long it is going to take to do the compression of a large save folder/file. My save folder is around 783M and with 8GB RAM it works fine for me. I cannot really test anything else.

FYI ..your comments came at a opportune time. I have just finished some changes (improvements I hope) to my code, so will post the latest to the first post as normal.

[glene77is]

to mow,
You kindly wrote "Thanks for the close examination of my app".
...
I decided NOT to use the original HotBackup method
"just because" it is included in all the Puppy Distros,
nor "just because" somebody wrote it ( and therefore "it must be a good method").
...
I did go all out to write "Cold-Tar-Backup" several years back, based on "tar.gz".
Too much code for anyone to peek into.
But I had great fun, nobody paying-me nor telling-me what to do!
...
I commonly use my "Puppy-Cold-Backup" ("PCB").
A minimalist program, only 400 lines of BASH,
having no GTK, no entries required, just run it.
...
This is my suggestion ( since you invited ).
(1) Run the Tar.Gz based backup, and forget about making a squash file,
(2) Since Tar will capture (a) "saveDIRECTORY" or (b) "file.4fs",
and will compress quicklly. (pk)Zip has been around since 1978, very reliable..
...
Of course, this will not be your own, personal approach,
but I think you are over-thinking this "backup-project" with diminishing results.
...
As to quick-dirty, I am toying with simply
(1) "mv" the saveDATA , to a 'new name' ,
(2) Zipping the newname data entity, to the HardDrive, where the original boot-time saveDATA was located .
(3) then rebooting so Puppy will restore from the boot-time saveDATA on your Hard-Disk. .
I am sure that sounds wierd/different, but it might loosen up your perspective
on possible back-up methods.
The idea of using "mv" needs some details worked out ...
...
But, using a minimalist program like "PCB" based on "tar.gz" will work predictably,
and will draw less "flaming" comments .
...
I do like your organization, your annotated headers to subroutines, and your attention to details.
...
I saw PuppyHotBackup some years ago, and it would not work on my system, so I re-wrote it,
and then I wondered "why?" would that program be included in all the Puppy Distros
if it did not work "out of the box" on alll the Puppy Distros ?.
I made comments to the authors, and some patches were put in place, but still the design seemed ??? questionable.
...
So, I had some fun , like only octogenarian engineers can do, and wrote my own, with an abundance of code and details/notes.
But , in the end, the "PCB" mini-app works really well, and I like running it with an "nice -19 xterm" lead-in ,
so I can watch my bits fly by ! on the xterm screen.
Now, PCB runs with the "ionice -c 1" parameter, which (by the book) is more isolated and more secure from interrupts.
...
BTW, if you thihk you like SquashFiles, then you should spend a year working in KNOPPIX !
Beautiful SquashFile implementation.
...
So, mow, tell me your thoughts, please.
Don't be bashful/timid just because I use lots of words.
Words are nearly free, but thoughts can be inspiring.
===
This AM, I wrote up a "PCB" using the "ionice -c 1 " program
to isolate/insulate my "PCB" from any interrupts ( so the manual reads ).
Seems to work fine, fending off any attempts at interrupts.
The "man" literature on "ionice" is very clear, and readable.
You should Google "linux bash ionice" and see for yourself.
I need to Lint and Upload , perhaps to a new topic , for general perusal/inspection.
===

[mow9902]

@glene77is

You said:

That is my suggestion, since you invited.
(1) Run the Tar.Gz based backup, and forget about making a squash file,
(2) Since Tar will capture (a) saveDIRECTORY and (b) file.4fs,
and will compress quicklly. (pk)Zip has been around since 1978.
Of course, this will not be your own, personal approach,
but I think you are over-thinking this "backup-project" with diminishing results.

For this purpose I have played with both Tar and Mksquashfs and found that they both do the job, and both take about the same time on my system, and both produce a compressed file close to the same size. I had to choose one of them - and decided to use mksquashfs and pipe the output to an xdialogue box for a GUI progress indicator. I could not work out how to do that using tar - but others will probably not have a problem doing that.

You said:

As to quick-dirty, I am toying with simply
(1) "mv" the saveDATA , to a 'new name' ,
(2) Zipping the newname data entity, to the HardDrive, where the original boot-time saveDATA was located .
(3) then rebooting so Puppy will restore from the boot-time saveDATA on your Hard-Disk. .

..mmm ..not sure about the 'move' approach - at least for my purposes. Are you sure that a 'move' operation while the system is running will not screw up the save folder/file? Anyway - why use 'move' - you could just use a 'copy' command couldn't you?

WRT to your use of 'ionice'. This sounds a bit of overkill for what I am doing for a simple backup.

[glene77is]

to mow,
I had not seen a scripted "progress bar" from "tar", previously. Good work, mow !.
I run "PCB-ion.sh" from Xterm, which displays a numbered line of data by "tar.gz", to Xterm,
( so I can trace operations more important than progress ) .
Each tar.gz numbered line is 1000 KB., just a minimal progress indicator, displayed in Xterm. .
-------
The "PCB-ion.sh" program automatically reads parameters from PUPSTATE, no user entry required.
The "PCB-ion.sh" sets up all required parameters.
The prep and call to Tar and Gz can be seen if you inspect the script.

ionice -c 1 tar -c -z -f ${destfile##*/} ${1##*/}/ --checkpoint=1000

If you have questions, then Google for "Linux BASH tar "
If you have questions, then Google for "Linux BASH tar gz"
If you have questions, then Google for "Linux BASH snapmergepuppy "
If you have questions, then Google for "Linux BASH nice "
If you have questions, then Google for "Linux BASH ionice "
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I use GEANY as the editor, and it has a built-in call to Xterm, very handy.
the call to run "PCB-ion.sh" is
xterm -e "${/path/path/PCB-ion.sh}"
only a few lines , all together.
The "PCB-ion-Xterm.sh is attached and is so small and simple.
In an editor, Just specify the fullly pathed name of your saveDATA entity.
I could have read this parameter from PUPSTATE, and pulled out the part I wanted, but I did not.
- - - - - - - - - - - - - - - - - - - - - - -
The "PCB-ion.sh" program automatically reads parameters from PUPSTATE, no user entry required.
and will backup a saveDIRECTORY or saveFILE.4fs entity, into a filename.gz
to your working subdir, where your saveDATA is located.
I kept everything in my Working Directory, all together.
------------------------------------
IF you look at the Install Puppy INSTALL program, which is the very first one you MUST run to INSTALL !,
THEN you will see that when you establish a saveDATA BackUp format [ Directory vs .4fs] ,
Barry Kaur's admonition is to choose the "saveDIRECTORY" .
That is the master's suggestion : { saveDIRECTORY } not { saveFILE.4fs } !
** Preferred ** is the word he used, I think.
...
Attached files are "PCB-ion.sh" and "PCB-ion-Xterm.sh" .
It will backup a saveDIRECTORY and a saveFile.4fs. with no user entry/selection required.
I have tested my "PCB-ion.sh" on 20 running Puppy Linux Distros, on my computer, and it runs as advertised.
If you have a problem, please let me know.
...
BTW, I have enjoyed reading through your very organized coding. You have obviously done lots of this before !
...

[glene77is]

to mow,

..mmm ..not sure about the 'move' approach - at least for my purposes. Are you sure that a 'move' operation while the system is running will not screw up the save folder/file?

The instant "mv" touches the old saveDATA file It Is Locked and GONE ! to become a NewName entity.

Anyway - why use 'move' - you could just use a 'copy' command couldn't you?

Don't worry about the "mv" method, I have not developed working code for that possible method.
"mv" renames the file in two clock cycles, and the old name is gone, can't be touched.
In puppy, the old-mv-name entity can be restored via "reboot".
The new name is a "static" newfile entity, and you can manipulate however you need.
The "cp" (copy) command takes a minute, during which time,
some competing program can compete/vie for Permission to Run Now!
This can interrupt your "cp" and maybe change some single byte of data.
That is why the author of the original Puppy Hot Save ( might be 2byte)
included the use of "e2fsck" to make sure the "cp" newfile has valid structure.
Don't worry about the "mv" method, I was just daydreaming anyway,
since I use the "nice -19" and "ionice -c 1 " methods,
which were written for the purpose of insuring security/validity of data.
To be clear, use of "mv" will not "screw up" the data, your target file entity is Locked and GONE !.
Don't worry about the "mv" method, I was just daydreaming.

***

"WRT to your use of 'ionice'. This sounds a bit of overkill for what I am doing for a simple backup."

***
The purpose of the program is to do a backup, securely,
on a a "HOT" running system. where changes/interrupts occur in a networked system. .
The saveDATA entity is not active, it is "static" , unless you "cp" to transfer data into a new file.
Or, unless you execute a "snapmergepuppy".
If you copy it to a new-name, then your new-name will 'instantly' become your "static"/inactive data file. .
...
Something you must understand, about all this Puppy Hot Save discourse,
[] the difficulties are about SECURITY from INTERRUPTS to the running backup program. []
The problem with a "HOT backup" is that some other running program on your system or network
may have higher priority and could interrupt your "copy" .
If you are networked, then some other dude could interrupt your "copy".
Interrupts like those are far fetched, unlikely,
BUT if they happen, then you can copy corrupted data.
...
Recall that the original program was titled "Puppy Hot Save",
and "ionice" has been designed to deal with competing priorities which can corrupt data.
At least that is what the manual and literature says (cross your fingers).
That is why the file.4fs method also runs "e2fsck" to check and correct
any data possibly corrupted during the simple "copy" function.
You can avoid all this stuff by running the "tar.gz" Unix method :
--- " ionice -c 1 tar -c -z -f ${destfile##*/} ${1##*/}/ --checkpoint=1000 " .
This is not "overkill" kinda, this is how a software engineer does his work.
===

PCB-ion-241206-1800-sh.zip

PupColdBackup script . zip

(4.71 KiB) Downloaded 1 time

===

[glene77is]

...
Attached files are "PCB-ion.sh" and "PCB-ion-Xterm.sh" .
It will backup a saveDIRECTORY and a saveFile.4fs. correctly, with no user entry/selection required.
I have tested my "PCB-ion.sh" on 20 running Puppy Linux Distros, on my computer, and it runs as advertised.
If you have a problem, please let me know.
Your "PHS" has been tested on 20 of the Puppy distros, usually running saveDATA/Directory , which is good news.
Runs are OK on a file.4fs setup, also. ... But, the file.4fs can cause RAM overrun problems .
But, your code looks good, and I expect that since you have 8GB RAM, things should be OK. Its just a "RAM" thing.
...
I just noticed that your "PHS" has enlarged my "saveDATA/Subdir" from 341M to 1098M, by calling e2fsck.....
I know that this is because of the "copy" duplicate and the e2fsk doing its own working backup, and not deleting properly.
e2fsck is a utility that examines ext2 , ext3 , and ext4 filesystems for errors, and attempts to repair them if possible.
e2fsck checks/repairs invalid superblock and bad group descriptors .
I also see several "PHS" output files with filesize=2070MB. Part of the method uses a duplicate of your saveDATA file.4fs .
Inspecting these several files, internally, I realize they will NOT restore the original saveDATA entity.

...
I have found several distros do things differently,
which cause my few problems when trying to run/test your "PSH" program.
For example, Noble Numbat uses an extra subdir in its /tmp system.
S15-64 releases its temp entity , so it cannot be "rm". Xterm will show the non-fatal error.
Xenial64 always shows as '.' is "PSH" (empty variable)
Quirky-April-7 refuses to run, but that is par for the course considering it is a Beta (not fully finished).

...
BTW, I have enjoyed reading through your very organized coding.
Turns my mind around a little bit !

...
...
This "simple" file is my suggestion : :

PCB-ion-241206-1800-sh.zip

PupColdBak_ion_241206-1800-sh.zip

(4.71 KiB) Downloaded 4 times

...
...

[mikewalsh]

Y'know, quite frankly I've never understood all the fuss surrounding the various ways to back-up Puppy (or 'parts' of Puppy). All I've ever done is a simple copy/paste to a different location.....usually external.

It's always worked flawlessly, and is endlessly reproducible whenever required. Why complicate what SHOULD be simple?

Mike.

[mow9902]

@glene77is

You said:

I have found several distros do things differently,
which cause my few problems problems when trying to run/test your "PSH" program.
For example, Noble Numbat uses an extra subdir in its /tmp system.
S15-64 releases its temp entity , so it cannot be "rm". Xterm shows the non-fatal error.
Xenial64 and Xenial-64 always shows as '.' (empty variable)

Fair point - I have made a very minor change so that the 'lock' file is saved into the app directory instead.

This is where I will now leave it. It works well for me - which is why I wrote it. See first post for updated version.