Rolling Releases and fallback

[geo_c]

I thought I'd put this here, even though it's a particular software issue that has caused me to consider the pitfalls of rolling releases.

I ran into an issue with all my KL's, Spectr, Airedale, Awesome, Bspwm, meaning the issue is really particular software that was updated.

The issue, though not critical for discussion in this section of the forum, is that apparently either a lilypond update or guile update broke lilypond in all these void based OS's.

So I rolled back to upper_changes in Airedale squashed in October and it all works again.

The particular error shows as an argument error in a lilypond init:

Code: Select all

Converting to `FractalNo01-MASTER.pdf'...
warning: `(gs -q -dNODISPLAY -dNOSAFER -dNOPAUSE -dBATCH -dAutoRotatePages=/None -dPrinted=false /tmp/lilypond-tmp-6701569)' failed (256)

/usr/share/lilypond/2.24.2/ly/init.ly:65:2: error: Guile signaled an error for the expression beginning here
#
 (let ((book-handler (if (defined? 'default-toplevel-book-handler)
Throw to key `ly-file-failed' with args `()'.

Manual/net searches don't yield much info. So that's something for which I'll be doing further investigation.

That issue aside, it does give me thought about rolling releases and the risk of breaking things with a system update, which can be quite a pain using daily drivers during busy times of the year.

Fortunately as a Pup user, I always have a static OS like F-96 handy.

This kind of thing also gives me more incentive to pursue more convenient backup/rollback methods.

A backup of a working upper_changes I suppose could be accomplished with a simple rsync or cp script in the install directory. It could require a reboot, but not necessarily booting into another OS. That approach though doesn't logically add up, in that to use one would have to load some kind of linux command kernel first, which might require a change to the w_init?

Using w_changes=RAM2 is probably the most sensible choice for creating backups on the fly, as those can be safely copied while in a running system.

Still, I don't generally prefer having to save changes as I go.

Just some thoughts I'm having, no real question or request here.

[geo_c]

Now it occurs to me what I've been overlooking as a handy way to rollback.

It's the multi-instance capability that I have yet to take advantage of.

I think that's going to be next on my list.

[williwaw]

this?

the wd_multi setup definitely provides great scope for alternative way of controlling rollbacks with restore points as you say.

[geo_c]

this?

the wd_multi setup definitely provides great scope for alternative way of controlling rollbacks with restore points as you say.

Yes that, exactly. I don't know why it seemed to slip off my thing-to-do list. I overlook things rather easily as my general attention is spread all over the map.

[geo_c]

this?

the wd_multi setup definitely provides great scope for alternative way of controlling rollbacks with restore points as you say.

HOLY CRAP @wiak this is completely cool!

I installed a 10-instance KLV-Airedale-RT (older version) that contains 5 squashed additional layers and a working upper_changes as instance 0, then went into into the next subseqent instances and peeled off a layer starting with the working upper_changes.

Then I created 5 boot stanzas for them, and now I can boot into rollbacks of my system builds back to the original date of install.

Freaking amazing.

There's gotta be a way to make this the standard install method in that the it would be possible to backup up the current upper_changes to a sequential instance, and then a script could be created to rollback to that instance, or something like that. Each instance would contain a previous upper_changes, and rolling back works something like having the working KL be instance1, while the squashed layers reside in instance0, then instance 2 and greater would contain sequential upper_changes that could be linked back into instance1.

I don't know, just throwing out ideas. I still can't fathom how one might get this choice at boot without having to add a lot of boot stanzas and how to do something like rename a layer or link that the current system is using.

However if it's possible at shutdown/reboot to rename an instance or multiple instances, then a single boot stanza might be possible and the choice to roll back could be executed on the way out, so that the next boot would reflect the rollback.

[williwaw]

the choice to roll back could be executed on the way out, so that the next boot would reflect the rollback.

so a script in a running instance would ask you ......

if you want to make a restore point or make a different multi the default boot
if restore point, run snapmerge puppy to build a new instance, then ask if you want the new instance to be the default boot instance

• if yes, make it the default

• if no, then let you select the default from the remaining choices

if make a different multi the default boot, then show you the other choices

[wiak]

Whilst I don't have time to write any such script myself at this time, I have now experimented with a simple change to the grub config functions that makes the resulting grub_config.txt instances files much easier to loop through and auto-parse. This will be useful for any person doing the likes of writing a post-processing script to auto-extract the grub2/UUID only stanza for each instance and assemble these or a subset of these in a new file.

Will become much easier once this simple modification is published, therefore, to automate grub config handling for the likes of using multi-instances as a restore mechanism.

The actual 'restore-point' utility script for doing so will still need to be written of course...

The current wd_multi generated installs will still work fine, though to get the easier to parse grub_config.txt files for each instance it will be necessary to re-run wd_grubconfig for each instance (using new wd_grubconfig script that will be supplied), but that won't be difficult and also will thus maintain backwards compatibility will already being used multi-installs. i.e. the new better parsed grub config won't break anything already existing...

[fredx181]

Here's a concept script for to create a timestamped restore point and/or restore to that point of time (tested on KLV-Airedale):

UPDATE 2024-02-24: Re-attached, changes:

• more error checking
• exit (with message) when booted with save file (only savefolder supported)
• exit (with message) when trying to restore twice (only once supported during a session), but creating restore points is unlimited
  (edit: some more info here:viewtopic.php?p=112680#p112680)
• .bak directories (previously original changes) showing also now in restore list
• dialog to add extra info (will show in history log) when creating restore point (pic below)
• button 'History log' added to the CREATE/RESTORE dialog

  restore-sys.gz

  Updated 2024-02-24 Remove fake .gz extension and make executable

  (6.74 KiB) Downloaded 11 times

  

  2024-02-24_13-39-37.png (34.22 KiB) Viewed 252 times

  

  2024-02-24_13-52-42.png (32.33 KiB) Viewed 245 times

  UPDATE 2023-11-21: Re-attached, has changes and improvements, see changes info here: viewtopic.php?p=104124#p104124
  It may be not finished yet, thinking of some extra's, e.g. create a log or any other way to add a personal description to the timestamped restore points (edit 2023-11-21, now creates restore_history log) . This for testing now...
  edit: also may need some more error checking.
  Not like discussed above (i.e. make it default in e.g. grub.cfg), the way it works is by creating a timestamped backup of the "upper_changes" folder and by choosing it make it the default (will be renamed to "upper_changes"), so then at next reboot it will be in use.
  This can be manually done of course too, but this script makes it easier (it's YAD GUI, not required to run from terminal, but I'd advise to do so, to be able to check for possible errors).

Screenshot(1).png (27.71 KiB) Viewed 1177 times

Note that if you are running the system with "save on demand" the changes cannot be saved to a created restore point afterwards, so first run save2flash to make the changes included before creating.

[geo_c]

Note that if you are running the system with "save on demand" the changes cannot be saved to a created restore point afterwards, so first run save2flash to make the changes included before creating.

So does this mean this script is designed to be running in direct save to changes mode? In other words it's not only for RAM2 mode. So I can run it from a system that's saving as changes are being made.

I'm getting confused with the terminology.

[rockedge]

@geo_c I think it means to make an accurate restore point snapshot of a current session while running in RAM2 mode, run save2flash before creating the restore point.

Because saving in RAM2 mode after the restore point snapshot creation like at system shutdown/reboot, none of the data of the current session will be in the restore point snapshot.

Might be something the script can do to check if running in RAM2 mode and offer a reminder or an option to run save2flash or not before creating the restore point.

[williwaw]

The current wd_multi generated installs will still work fine, though to get the easier to parse grub_config.txt files for each instance it will be necessary to re-run wd_grubconfig for each instance (using new wd_grubconfig script that will be supplied), but that won't be difficult and also will thus maintain backwards compatibility will already being used multi-installs. i.e. the new better parsed grub config won't break anything already existing...

can wd_grubconfig find a directory with a descriptive name that contains an upper_changes subfolder in addition to links to initrd and vmlimz,?
would it be able to write a grub entry with the descriptive or timestamped name in the title line?

Freds script would then only need to create a directory with a timestamp or description in it's name that contains the copied upper_changes (without a name change)

incorporating wd_grubconfig into the restore point process would add the capibility to choose at boot time in addition to the running the script prior to a reboot

[geo_c]

@geo_c I think it means to make an accurate restore point snapshot of a current session while running in RAM2 mode, run save2flash before creating the restore point.

Because saving in RAM2 mode after the restore point snapshot creation like at system shutdown/reboot, none of the data of the current session will be in the restore point snapshot.

Might be something the script can do to check if running in RAM2 mode and offer a reminder or an option to run save2flash or not before creating the restore point.

Yes, that all makes sense, but the way @fredx181 worded it, "if you are running in 'save on demand," makes it seem that the script will also work in convential install mode.

[rockedge]

@geo_c yes it works in regular mode The tests I have done are all on a regular frugal install on HDD and so far they have worked.

[fredx181]

@geo_c I think it means to make an accurate restore point snapshot of a current session while running in RAM2 mode, run save2flash before creating the restore point.

Because saving in RAM2 mode after the restore point snapshot creation like at system shutdown/reboot, none of the data of the current session will be in the restore point snapshot.
....

Yes, exactly, when running in RAM2 mode, if you want to include in the restore point the changes just made in current session, you'll need to run save2flash first before creating restore point.
The script works with both modes (RAM2 and regular), but with regular mode the current changes are already "overlayed" during a session.

Might be something the script can do to check if running in RAM2 mode and offer a reminder or an option to run save2flash or not before creating the restore point.

It does such a check already, good idea to add some (warning) message.

[geo_c]

The script works with both modes (RAM2 and regular), but with regular mode the current changes are already "overlayed" during a session.

Might be something the script can do to check if running in RAM2 mode and offer a reminder or an option to run save2flash or not before creating the restore point.

It does such a check already, good idea to add some (warning) message.

I'll start using the script.

Using this script in conjunction with multi-instance installs could be cool. Allowing for easier rollback points in each instance means not as many instances are needed in grub stanzas to freely experiment.

I thought about the idea of copying backups to other instances, which would work, but is risky in messing up the package manager of those instances if not done very carefully.

[fredx181]

Some improvements for the "restore-sys" script:
Re-attached here: viewtopic.php?p=103935#p103935

First: (should have mentioned earlier) after restoring to some point in time, a direct reboot is recommended because afterwards any change made to the system will not have any effect.
(this is because after restoring to the 'new' upper_changes folder it is NOT mounted yet during the session (the upper_changes_<timestamp>.bak folder (renamed from upper_changes) is still mounted, pity, seems not possible with overlayfs to remount to this new created upper_changes folder, anyway... I don't know how).

Changes:
- Added message in case running with boot option w_changes=RAM2, that save2flash is required to run first before creating a new restore point (if wanted to add changes from current session).
- Create a 'history' log file in the 'bootfrom' folder called 'restore_history', will add new line every time you create or restore, something like:
[At: 20 Nov 2023 12:35:35] Created: upper_changes_20231120123535 20 Nov 2023 12:35:35
[At: 21 Nov 2023 12:25:52] Restored: upper_changes_20231120123535 20 Nov 2023 12:35:35 Original upper_changes renamed to upper_changes_20231121122552.bak
(possibly edit as you wish to add e.g. personal description)
- Final message after restore process has more info and choice to "Reboot now" (recommended YES, see info above).

[rockedge]

@fredx181 Thanks for the great work! Once again you've built an important tool that will be included in the next KLV ISO's.

It's Formula One!

[geo_c]

@fredx181 Thanks for the great work! Once again you've built an important tool that will be included in the next KLV ISO's.

It's Formula One!

Yes @fredx181, this is exactly what was needed, in essence a "pupsave" mechanism for KL.

Now with multi-instance installs and rollback saves, things can get mighty confiusing! In a good way of course. Best to use a certain restraint before creating a plethora of versions of the same system.

[fredx181]

@fredx181 Thanks for the great work! Once again you've built an important tool that will be included in the next KLV ISO's.

It's Formula One!

Yes @fredx181, this is exactly what was needed, in essence a "pupsave" mechanism for KL.

Now with multi-instance installs and rollback saves, things can get mighty confiusing! In a good way of course. Best to use a certain restraint before creating a plethora of versions of the same system.

Thanks @rockedge @geo_c ! Including in ISO is fine by me but must say that I didn't test with all possible boot options, e.g. not with w_changes=RAM0 (no changes , so the script probably won't work) or when booting with a save-file (which I never tried on KL, TBH), ONLY with upper_changes savefolder it will work, I guess.

[wiak]

I am particularly looking forward to trying out this new facility once I really get back to FR/KL work. Alas, crazy busy filling a container for ship and volumes of paperwork prior to big family move. Hopefully, all will calm down for me around March 2024 - it is frustrating missing out on so much going on here right now.

[geo_c]

I haven't been busy on the forum or building KL's these days, as I've been busy building snare drums and attending to the music business.

Instead of building KL's I've been using them as daily drivers, utilizing @wiak's multi-instance script, and @fredx181's backup script.

As I said at the original post of this topic, a void update broke my much needed lilypond application, which got me motivated to setup multi-instance booting.

So here's an update on how it's going, and my strategy.

Basically, I created a multi-instance scenario using the full daily driver with the broken lilypond as instance-0, then created a version as a fallback to the last cleanly working system (links to broken layers removed) as instance-1, then created a similar version as instance-2.

My strategy:

instance0 - just keep using it and doing occassional Void system updates to see if the broken package gets fixed
instance1 - never boot it, just keep it there as a master copy to update after experimenting with the other two
instance2 - use this as a daily driver, as it contains a working lilypond package, but don't update it.

while doing this, I use @fredx181's backup script on instance-2 when making small changes to applications, adding files and tweaks I want to keep, but still not doing Void system updates.

So today I did a Void update on instance0, and it contained a lilypond update that fixed lilypond.

So strategy is paying off.

When I get back to building SR7s and all, these older versions might not matter as much, but it's been great to have all the versatility available.

In reality I have 10 instances setup, but I only have boot stanzas for these 3 to keep it simple and organized.

[fredx181]

*** Updated restore-sys script ***

Main goal was to add more error checking (to prevent weird things happening), e.g. cannot restore twice or more during a session, so now limited to only once.
But creating restore points is unlimited (i.e. make changes > create restore point , make again changes > create again etc....).
(and more improvements IMHO)

New attachment and detailed changes info: viewtopic.php?p=103935#p103935