universal package manager/maker

[wanderer]

hi all

this is just a question for the gurus
what are the ins and outs of a universal package manager/maker

that is a process that can take an app from various sources
and package it in a format for a particular distro

i have some idea of the complexity/impossibility of this task
and that it has been asked innumerable times
this question is just for discussion

wanderer

[wiak]

Let's think briefly and in simple terms about how a package comes about and how it is managed...

The package has source code and dependencies (being other software it needs present on a running system, some of which it is built to specially 'link' to; libraries etc).

The package gets compiled on a particular distro, which uses a particular version of a compiler and particular versions of the package's dependencies.

Each distro has its own way of naming and holding various pieces of meta data about the compiled package. For example, package name the distro uses for the package, version information, file sizes, where all the various files that make up the package are to be stored, details regarding the packages dependencies and so on... Such meta data might be stored in simple text files, or inside a database of some sort, and each distro will have its own way of doing that.

So you want to make a universal package manager that can re-package from one distro compiled package format to that of another distro... Basically, for each distro's package type you are interested you need to pull that distro's package apart to extract all that same type of information and bits and pieces (such as the package files themselves) and place them inside a new package container that is arranged in the form your target distribution requires.

Hence it is a potentially large task, and fraught with pitfalls. Much of the work will be translation/conversion from one package format to another: a different format to match how the target package manager stores all the information required about the package so that it is able to do such things as: install it, remove all it's pieces, provide info on the package via a query, dependency information... etc

And that last one, dependency info, is major problematic, because different distro's package differently in terms of what they include in their packages, so sometimes dependencies will be hard to maintain (some distros may include more libraries in the same single package name, for example). Moreover, there is the problem of dependencies of dependencies; different distros often as I say use different names for their packages so how to match them up???

In practice, it can therefore all be a bit of a hit or a miss situation. The more self-contained the required package is the more likely it can be re-packaged for a different distro with success. Older packages, can often be re-packaged for other distros since the libraries used are often backwards compatible (though not always...).

However, much of the time, such package translations seem to work pretty well (which is a testament to how flexible/compatible/standardised Linux system can be really), but it is a risk. A distro is designed very carefully such that its component parts are compatible - that cohesive distro design can be a bit of a (carefully crafted, configured, optimised) art, and using a package from another distro can become like throwing an alien spanner into the works. The package may work perfectly, or seem to work perfectly "most of the time". And there is also the problem regarding any different configurations used and any pre and post installation scripts provided by different distros.

The question could become: "Is it worth the risk"?

The alternative is that if the package is not available in the distro's own repos, then it would have to be compiled specially. That would be preferred. It is always best, I'd say, to use the official package manager for the repo the package is being installed from, but never good to mix package managers in the same distro since the alternative package managers are not designed to keep track of the what and where of another package manager storing its installed packages. The package manager database will thus be effectively corrupt if it thinks things are where they aren't or are present but known about.

Best, to me, is to use one package manager along with its own repos. Of course repos need to be large and extensive enough to satisfy a distro's users, so it is important for a distro to have an auto-compile-packaging system; i.e. a build system that can track packages for new releases and auto-compile and re-package them. Distro's like Debian, Ubuntu, and even Void Linux have exactly that.

But from a less rigorous perspective, the suck it and see approach certainly allows distros that have limited repos to add to what they have available. Not ideal, but as Puppy users can testament, taking a package and re-packaging it as a pet or an sfs for Puppy can often be made to work, with or without multiple-tweaks. And such efforts can be useful and fun. Writing the multi-distro capable package manager, however, is a big task for sure, and no matter how well it is done its users have to accept the pit-falls and flaws inherent with that approach.

EDIT: Probably worth adding that instead of trying to mix packages from different distros via some universal package manager, you can use a well-known Puppy (and Dog) Linux approach to alien packages: put them into some kind of portable package format, such as an sfs (which is itself read-only) that has been carefully crafted to not leave any traces (such as config/cached info) behind when it is unloaded. That approach does not ensure the package will work, but allows it to be very usefully used if it does work without danger in terms of messing up the overall system or corrupting the package manager database. So, yes, the sfs load/unload approach to OS design can be very useful. That approach can actually be used will full installs too, but is particularly matched to the way frugal installs work via overlay/aufs mechanisms.

[wanderer]

...

thank you very much wiak for your comprehensive answer

yes i am not thinking of anything big or complex or foolproof
which i think with the variation in distros is virtually impossible

more like a general formula
to convert/package an application into one that can be tried on your distro

and for me an sfs is perfect
but i would symlink it instead of using a union filesystem

one can both symlink a compressed package
and an uncompressed package
(for example residing in home or root)
and when you reboot the symlinks are all gone
or you can remove and add the package without rebooting
by renaming the package
so the symlinks can either find it or not

puppy has sfs and pets and there are debs etc
and i don't know what void or slitaz calls theirs
for me personally i need only about 20 or so packages
but the ability to try new stuff is im sure of interest to most of us

and the idea of a tiny core with extensions
is of course of great interest to me
so making the extensions is integral to the system

anyway
a useful and interesting topic

thanks again

wanderer

[wanderer]

...

[wanderer]

...

[wanderer]

...

[wanderer]

something strange is going on
my text is being arbitrarily put in code boxes
will ask rockedge to look into this

wanderer

solved by mm
remove all preceding spaces

[wanderer]

so my universal package manager

is to use uextract to extract it into a folder
and then squash it into an sfs file
(maybe rename it tcz for corepup/tinycore)

or leave it as an uncompressed folder
and symlink it using the copy command

sometimes it works

pretty advanced

wanderer

[mikeslr]

Hi wanderer,

Take a look at taersh's PaDS 1.1.7, viewtopic.php?p=6355#p6355. Currently, it is capable accepting a bunch of packages in .deb, .pet, .sfs, .tazpkg, .txz, and .tar.gz formats --they don't all have to be the same format-- dissembling them and combining their contained files and packaging them as an SFS. But with a little tweaking you could probably get it to 'spit out' a tcz.

If you retain the source files and have something like LIstDD, you can test the resulting package, obtain whatever may be missing, add it to the folder holding the source files and re-run PaDS. Alternatively, as I often have to do working with Ubuntu-compatible Puppies, you can mount + copy the built package to a folder, edit /usr/share/applications ...desktop (and maybe locate icons somewhere or a symlink 'on the path') then use the equivalent of dir2SFS/pet so that applications will appear where desired on the Menu.

[wanderer]

thanks mikesir

will check it out

wanderer