Wednesday, October 11, 2006

Parallel xargs and faster ssh connection initiation

You learn something new every day, even with old friends such as ssh and xargs.

SSH: It turns out that it's possible to create a live "Master" connection to a specific remote host which then can be used by other ssh command executions to quickly open sub-channels to that host without going through the authentication process every time and without compromising on security.

Example:

$ ssh -fMN -o ControlPath=~/.ssh-control-sock remote-host
$ ssh -o ControlPath=~/.ssh-control-sock remote-host date

more about this in ssh(1) and ssh_config(5)

XARGS: In response to some Digg pointer to xjobs, which seems to be similar to xargs only it can run multiple jobs in parallel, someone pointed out to xargs' own -P (--max-procs) argument which does exactly the same. Just have to remember to limit the number of file names passed to each job using -n (--max-args), otherwise all the file names will be passed to a single job.

1 comment:

ole said...

Xargs have quite a few limitations when running jobs in parallel.
http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel

Consider using GNU Parallel http://www.gnu.org/software/parallel/ instead. Watch the intro video to GNU Parallel at http://www.youtube.com/watch?v=OpaiGYxkSuQ