QEMU and File Sharing

[wiak]

Following my research on Plan 9 file sharing in qemu a week or so ago, I will now briefly provide you with a little bit qemu file sharing between host and guest 'magic'. I was so excited when I first found this and it proved to work (took me a while to piece it altogether and make it work the first time though).
DISCLAIMER: Obviously the following very powerful file sharing technology can be very dangerous depending what folders you share, but then again running as root user is dangerous in that sense anyway. Handle with care and at your own risk.

Once again, for convenience cutting and pasting below commands, it is handy to start a browser in your VM once it is booted so you can read this post and cut and post from it into your guest VM.

1. Prior to starting your virtual machine load the module 9p and dependencies on your host machine using terminal command:

Code: Select all

modprobe 9p

1. Let's say for argument you are about to boot a cdrom in qemu by going to the directory the cdrom is in. At the end of the qemu command you use, put a space followed by a \ and that lets you add another line of options. Add the following:

Code: Select all

-fsdev local,security_model=mapped,id=fsdev-fs0,multidevs=remap,path=. -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=fs0

Note the path=. above. That 'dot' after the equals sign means you are going to share the 'current directory' between the host and the guest VM. But understand that you can use any path=<PATH> in practice I think.

1. Once VM is booted. In that guest open a terminal and create a mount point for the shared directory as follows:

Code: Select all

mkdir /mnt/inst  # you could really call the mount point directory anything

1. Finally, you should be able to mount the shared directory in your guest VM using guest command:

Code: Select all

mount -t 9p -o trans=virtio fs0 /mnt/inst -oversion=9p2000.L,posixacl,msize=5000000,cache=mmap

In your guest VM you should now be able to view the host files inside (from where you started) inside that directory /mnt/inst
Furthermore you can use the files from there and also put new files in there if you want which will immediately also be on the host computer. i.e. You can share in both directions. So put away your ftp server clumsy techniques, and so on, this method may (at times) prove better for you.

NOTE: I don't know how 'secure' this Plan 9 file sharing method is when running qemu as root user. The method does however also work if you start your virtual machine as a normal user (though you'd have to be root user in the virtual machine guest to do the command needed there (one way being to first run the command "sudo bash" inside the virtual machine guest, assuming you weren't already root user and sudo is available for whoever you are there...). Apparently the methodology is pretty secure in that situation, but who cares if it works anyway eh? NOTE2: I haven't actually tried it yet with KLV as the host; I've only tried it with Zorin lite OS host and KLV VM guest, but I hope it will work fine with KLV host too.

EDIT: Okay, just confirmed that above also worked using KLV host with KLV guest.

EDIT (22July2024): Also for EasyOS here by @Frederico 01July2023 here:
Finishing Touched -> Sharing a folder with the host (requires EasyOS >= 5.4.3): viewtopic.php?p=92950#p92950

[wiak]

I discovered another way of getting raw access to a drive on my laptop, but I really don't know the details of what its bits and pieces mean, or if it is dangerous to use in any way, but I'll post the extra segment I used in my qemu line to give me that access. It appears to end up using /dev/vda; found I can even mount that again with wd_mount vda, though no real reason to do that and it isn't required here). Note, this has nothing to do with Plan 9 (9p) method:

Just added this extra bit:

Code: Select all

-drive file=/dev/nvme0n1p6,format=raw,if=virtio

EDIT2: NOTE - I am finding this method quite flaky - not reliable for 2-way file sharing in my tests. Kind of works but some odd results sending files back to host and not appearing to get there - more testing required. Maybe it is okay, but USE WITH CARE... The above Plan9 (9p) method seems reliable thus far though.

/dev/nvme0n1p6 is one of the partitions on my SSD nvme drive that stores my Zorin lit OS full installation. Obviously you would need to change that to /dev/whatever to match your system (e.g. /dev/sda3 or whatever)...

On my system, using KLV-AIredale as a host and booting a guest KLV-Airedale in Qemu, that extra part above added to my qemu boot line had the effect that when I opened the guest VM thunar I could see and access read/write my Zorin full install /dev/nvme0n1p6 partition (it was in the thunar side panel list of partitions that could be mounted/unmounted).

The reason I stumbled on this method is that I'm trying to frugal mount my full installed Zorin lite OS, via qemu, with KLV-Airedale host system using the magic FirstRib generic initrd.gz (specially modified to contain boot modules for Zorin lite OS); I already know I can do this on my physical machine by simply making a new directory, say FR_zorin, on the / of my partition /dev/nvme0n1p6 and then 'moving' the whole of / (except that FR_zorin empty directory) into FR_zorin. That is a super quick operation since no actual physical move of bytes takes place since just a 'move' on same physical partition). I can then boot that FR_zorin, via the zorin vmlinuz and FR initrd...

Since I'm in KLV-Airedale host, it suits me to also be able to run my Zorin lite full install distro as a guest virtual machine - hence my quickly scripting it using FR initrd into a FR-pseudo-full install type of setup... Wonderful really.

EDIT:
As for the 'magic'. It was in fact forum member @Duprate that called FR initrd a 'magic' initrd (he seems to use an old version of it a lot to boot all sorts of Puppy and Dog systems as a way of avoiding aufs); previous weedogit, now firstribit, script uses it to boot many a distrowatch-described mainline distro as a frugal install; I just borrowing his term 'magic', since the results of using it do sometimes seem like magic (even for me - especially this booting of full installed Zorin directory - albeit via that one move operation (later I just move it back out of FR_zorin subdir when wanting to reboot Zorin OS as normal full install - takes milliseconds only).
The truth is, it isn't difficult to boot a Linux system if you understand how it all fits together. As long as all the exact needed parts are in their exact correct places, a boot will be successful. A failed boot is simply something out of place (e.g. even a missing symlink to correct switch_root init script). Computers are perfectly fussy - one smallest slip and boot generally will fail. It is like a jigsaw where every piece has to be exactly correct and fit perfectly, then everything works no matter how you achieve the exact jigsaw pattern.

I was taught, when I once worked in a research group, that I should try and find a generic theory rather than just a one-off special case proof/example. Hence, when FR initrd was designed I purposively tried to make it generic, so could be used with various/most-any root filesystem, of any distro, and that it could use a mix of either or both sfs or normal directories for its addon layers and not have a limited number of fixed order layers (in practice it is limited to 99 layers, but other factors limit what is reasonable there and a one or two line simple init change can make it 999 layers!)..

So the the above 'make full install of Zorin" into a FR initrd frugal install, simply uses that generic flexibility: the overlayfs simply arranges all the layers (the uncompressed Zorin directory being one of them) in such a way that the final jigsaw is complete and correct, and in that state it is thus not magic at all that the overlay can be booted successfully - that was an intentional part of the philosophy behind FR initrd design; it wasn't just thrown together to allow one specific root filesystem assembly to work (which is the case with most all other distro initrd designs), it was designed as a, reasonably small and flexible/easy-to-modify, assemble-jigsaw generic tool to boot most anything - so the results sometimes seem like 'magic'.

[wiak]

hahaha... so I'm sure Plan 9 (9p) qemu file sharing between host and guest is probably safe. Not so sure about that other more direct -drive method, though maybe fine if careful... I'm not...!
So using -drive method in experiments to Qemu my underlying host main Zorin full install, using KLV-Airedale as host machine, I corrupted my Zorin full install and failed to get it to boot normally again... sigh (that's supposed to be used by me for family business use only, but I couldn't resist the 'experiment').

However, I don't know why people imagine it is difficult to re-install a full installed Linux. I just did it... didn't take long. Have a list of apps I wanted to then re-install in my cherrytree notebook. Did that, didn't take long. I even unsquashed 10gtkdialog sfs from KLV-Airedale and simply transplanted the utilities there to my new installed Zorin /usr/local/bin and so on. Hence my Zorin full install system is fully back to normal and probably benefited from the re-install (lots of rubbish I'd installed now gone).

Only issue I have with Zorin is that it is FocalFossa based, so apps a bit old in the tooth. I tend to seek out PPA's with up-to-date versions, but main issue is that I don't like using Firefox (Zorin supplies) but instead want Chromium. Alas, Ubuntu official only now supports Chromium as snaps, and I object to snapd and so on, hence another PPA search. Last time I went for Ungoogled-Chromium as an AppImage, but that had a problem for me: Google gmail worked fine, but Google Drive wouldn't let me download - that facility is essential here, so Ungoogled had to go. Instead I'm using PPA Chromium from here:

https://launchpad.net/~savoury1/+archiv ... u/chromium

Current version: 109.0.5414.74

Seems fine. I also always install Chromium in KLV-Airedale, but that's easier since Void Linux xbps provides a good Chromium package.

[wiak]

Finally, seem to have sorted out the grub config errors I made for KLV-Airedale rc8. It was very difficult to get this right, and so many modes to test, so no absolute guarantee, but these are for rc9 release when it comes. Please do not modify their contents, but if you see something wrong let me know for checking. Like I say this is not easy to get correct since some of the entries rely on rarely used flexibility of FirstRib initrd (so rarely used thus far that even I get it wrong...).

@rockedge:
Please therefore find attached all the boot/grub/ configs needed for when you release rc9.

In summary, all of Ventoy, SG2D, and Qemu can use save persistence to a partition labelled Persistence and puts it in directory Sessions.

Ventoy can also put save persistence onto its own bootfrom media as long as the Ventoy main data disk is Linux formatted and given label 'Ventoy' (again savefolders get stored in a Sessions directory there too).
Similarly for SG2D, but the SG2D data disk needs to be Linux formatted and given label 'SG2D'
and similarly for QEMU, qcow2 hard disk image, which needs to be Linux formatted and given label 'QEMU'
Note that there are other provided modes that do not need the Ventoy, SG2D, or QEMU disks labelled in that way (for example, if you choose instead to store savefolder on any partition elsewhere, anywhere attached to your system, named 'Persistence').

My brain is near evaporated now - very frazzled, so only lots of mode testing by others will reveal any errors I may have made still. However, before reporting 'failed boot' do please study what is required in terms of partition labels and so on for many of these offered modes to work in the first case. False negatives and positives in reports can waste a lot of time tracking down user config mistakes. Having said that, I certainly do make grub config mistakes in the isos (several such made by me in rc8 release grub configs as it turned out...).

[wiak]

Progress report on attempt to improve FR initrd slightly:

First attempt worked for normal frugal install RAM0 boot only. Alas, was a disastrous fail with all other modes! Flipping two solid days work had to be thrown in the bucket!

Took a few good bits out of that code attempt though, and thus far this second attempt is working in some early tests. Problem is the amount of modes/mechanisms that need tested - worst is testing anything to do with any kind of iso-booting since I have to keep rebuilding the 07 sfs, initrd, and the iso itself pretty much everytime something fails with that. I think I build the iso twenty times at least yesterday...

However, just before bed last night Ventoy with Persistence worked with new FR initrd, so this morning I was ambitious and tried a mode I haven't used for probably at least a year, so didn't expect it to work. I refer to kernel line w_altNN=..., which allows numbered addon sfs or uncompressed filesystems to be placed on different partition to the bootfrom partition and indeed anywhere on the filesystem. In fact is 'supposed' to work even if just a few of the NNaddons are placed in some other partition. Hopefully, the grub config stanza below also helps to illustrate how to use w_altNN argument. I had taken 10gtkdialogXXX.sfs and 01firmwareXXX.sfs and placed them on a different partition in a directory KLU-jamBITS:

menuentry "KLU-jam" {
insmod ext2
search --no-floppy --fs-uuid --set 7d34b88e-bc8e-4bb6-86cf-d619c3b545a9
linux /KLU-jam/vmlinuz w_bootfrom=UUID=7d34b88e-bc8e-4bb6-86cf-d619c3b545a9=/KLU-jam w_changes=LABEL=Persistence=/Sessions/KLU-jam-rc4 w_changes1=RAM2 w_altNN=UUID=424d8f42-e835-4111-9053-dd086b3d38e8=/KLU-jamBITS
initrd /KLU-jam/initrd.gz
}

And it booted fine; found the 10gtkdialog and the 01firmware. More luck than anything that initrd/init code still works after all the time that has passed without me finding time to retest that functionality until today.

So I 'think' I am getting near to publishing FR initrd ver 7.0.0 rc2 though soon as I have now said that I'll probably find some new serious error...

Obviously, last thing I want to do is recommend or publish any new FR initrd for general KL use since if faulty would break everything else for everyone else.

[wiak]

Some alternative file sharing methods for qemu, such as using samba or ftp can be found in this thread started by @vtpup:
"How to share a folder in qemu? (SOLVED)"
viewtopic.php?p=115233#p115233

and this thread by @Clarity with some detail about using SMB/samba:

"QEMU - Transfer files between the VM and your HOST"
viewtopic.php?p=79153#p79153

[wiak]

Above are just some old posts that are important (to me) that I wanted to make easier to find in this one topic area. I may add other ones later. Since I'm working with frugal2qemu at the moment viewtopic.php?p=126355#p126355, I'm planning to experiment again with 9p (Plan 9) file sharing method along with that. viewtopic.php?p=78844#p78844

Basically, I 'plan' to get save on demand (RAM2 mode or Pupmode13 for example) working 'safely' in frugal2qemu qemu boots via that host/guest disk sharing mechanism if I can.

A related thread on "QEMU and booting" is here: viewtopic.php?p=78785#p78785

I've locked this thread for now because I'm arranging it with particular posts I want without clutter of comments. However, if you know of any other interesting qemu-related posts/threads of interest that I may later want to include links to feel free to alert me about them in thread KL-Dev_work/KLV To-Do List: viewtopic.php?t=7515

Many thanks in advance.

[wiak]

I believe I 'might' indeed be able to use Plan 9 (9p) file sharing method to allow wd_changes=RAM2 save on demand to work with my otherwise 'read-only' boot of frugal installed distro into qemu.

Well, I did rather fail on my first attempt, but when I think about what I told the system to do, I told it to save2flash using rsync back into the frugal install directory, but that was a mistake on my part since it should of course be rsyncing back into the upper_changes persistence save folder. Erroneously telling it to rsync back to the main frugal install directory results of course in all the main directories (etc, home, usr, and so on) being written back into there (rather than into subdir upper_changes) and that indeed happened and of course rsync deleted all sorts of other on disk folders in there since, well, it was 'syncing'... oops

The great thing though is that there was no file corruption of any kind (just erroneous deletions... sigh) so if I get the upper_changes path correct on next attempt I am hopeful this will be fine... maybe...