Distributed Computing on LAN

[p310don]

Is there a way to use all PCs in a network to work on one task?

I often take videos of things, and the files straight out of the camera are huge. I have found that for archival, I can run the camera's mp4 file through QWinFF, which is an FFMPEG frontend and convert the file to an mp4 and it often takes up half the space, or less with virtually no difference in perceivable quality.

This process takes time.

I have numerous computers and laptops laying around. I have figured out I can split up my files into groups and get each computer to do a few file conversions across the network, which is pretty great.

What would be better for me (I feel) is to be able to sit at my desktop and use only one machine, but have all the PCs in the network working on the same task. This would be particularly handy if I have one big video to convert.

In my mind what I'm asking is if I'm using PC1, can I set up PC2 on the network to act like another CPU core?

[p310don]

Side story...

I often use BackSeatDriver to remote control my work computers to work from home. BSD works by creating a network tunnel (VNC) so that the work computer is seen as part of my local network.

I can use the work PC, which is at least twice as powerful as my home PC to convert my local files, over the network tunnel, faster than I can use my home PC that the files are stored on. It's pretty cool.

[jamesbond]

Not at the CPU level, not that I know of. But I know you can split the jobs and hand them to different machines (e.g 10 files and 5 laptops = 2 files per laptop).

For compiling, use distcc.

For other things, there is https://www.gnu.org/software/parallel/

Perhaps others can say more.

[Grey]

Side story...

Hello. This is a good topic. When I worked for a small local TV company, I was interested/engaged in this. But not now.

Right now I only know about Blender. 2 network renderers:
1. Built-in.
2. External.
In principle, it's not bad.

If you learn anything about video editing, then be sure to share it.

[Jasper]

@p310don

Do you have a dedicated graphics card that could offload the video encoding?

This would take the pressure off from your cpu.

Also, it will depend on the codec that you are using.

[p310don]

@Jasper I have a very old nvidia card in one computer, the rest are integrated intel graphics.

I think it is a GT210

I've never had success getting nvidia to do the work for me.

[Jasper]

@p310don

I did upload some driver applications for onboard Intel gfx cards (....i915) and from memory:

https://github.com/intel/media-driver

There is one specific tool which could be utilised by FFMpeg .... Quicksync

https://trac.ffmpeg.org/wiki/Hardware/QuickSync

Unfortunately the Nvidia GT210 does not have NVenc support

You can find a list here ( CTRL+F Nvenc ) :

https://en.wikipedia.org/wiki/List_of_N ... sing_units

If it is does hold you up with your work, frustrating etc .................. maybe look for a second-hand card?

Just make sure that your motherboard & PSU can support it.

Gamers are always trading in their old gfx cards for newer ones.

[p310don]

@Jasper I've gone down the rabbit hole of trying to get GPU assistance working on my intel boxes. Just frustrating when I don't know what I'm doing...

Alternatively, I've also looked at a new graphics card. Actually have looked before today. Notice my local PC store has GT710 on sales for $44, which should have GPU encoding enabled. Otherwise 1030's are $99. All pretty cheap.

Have you done video transcoding using GPU? I have read some forum posts that reckon it gives rubbish quality. I don't trust other forums as much as I trust this forum, so would be interested to see other's experience.

[Grey]

Alternatively, I've also looked at a new graphics card. Actually have looked before today. Notice my local PC store has GT710 on sales for $44, which should have GPU encoding enabled. Otherwise 1030's are $99. All pretty cheap.
Have you done video transcoding using GPU? I have read some forum posts that reckon it gives rubbish quality.

Since the day has not passed yet, I can still answer (do not pay attention).

I encoded and transcoded the video using the Handbrake transcoder (H265/HEVC).
1050(Ti) encodes and decodes perfectly. 1030 is a "trim" from 1050, but most likely everything will be the same, only slower. I have both video cards, but on the 1030 I only decode to mpv.

If we use Linux for such purposes, then except Nvidia - unfortunately - there is no alternative in the medium and cheap segment.

The quality is normal, unless you are a specialist in the film industry with perfect eyesight and will not get carried away with the settings.

ffmpeg can do all this, but I don't remember the standard or special version.

[Jasper]

@p310don

Yes, I have encoded videos from the VCD era to the present day.

Admittedly, using Windows (VirtualDub Mod etc) and not Linux to do so.

From TV/Satellite caps, ripping physical media, encoding from one format to another to be displayed on different devices eg DVD to console/handheld. Also, had purchased hardware PCI Mpeg2 Encoders.

Is the end product for commercial purposes or for personal use?

I did use to upload a lot of material to newsgroups (quicker then some of the Release Groups) and provided tutorials for others to follow.

I believe I still have a very very old torrent listed on Pirate Bay still under a pseudonym which can be linked in a round about way to his forum

As @Grey has stated Nvidia is the best option at the moment and cheaper than purchasing a newer cpu.

BTW If you are demuxing your content into AV streams and encoding to different formats, this could be done as individual tasks. Some allowing the best rig to handle only the Video.

[Jasper]

@p310don

Might be of interest to you?

SCUDA is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines.

https://github.com/kevmo314/scuda