[med-svn] [Git][med-team/parallel][upstream] New upstream version 20210822+ds
Andreas Tille (@tille)
gitlab at salsa.debian.org
Tue Aug 24 12:09:17 BST 2021
Andreas Tille pushed to branch upstream at Debian Med / parallel
Commits:
5101cc6e by Andreas Tille at 2021-08-24T11:10:58+02:00
New upstream version 20210822+ds
- - - - -
30 changed files:
- CITATION
- Makefile.am
- Makefile.in
- NEWS
- README
- configure
- configure.ac
- src/Makefile.am
- src/Makefile.in
- src/env_parallel
- src/env_parallel.ash
- src/env_parallel.bash
- src/env_parallel.dash
- src/env_parallel.ksh
- src/env_parallel.mksh
- src/env_parallel.sh
- src/env_parallel.zsh
- src/niceload
- src/parallel
- src/parallel.pod
- src/parallel_alternatives.pod
- src/parallel_book.pod
- src/parallel_tutorial.pod
- src/parcat
- src/parset
- src/parset.pod
- src/parsort
- src/sem
- src/sem.pod
- src/sql
Changes:
=====================================
CITATION
=====================================
@@ -19,3 +19,7 @@ Or:
DOI https://doi.org/10.5281/zenodo.11460
This helps funding further development.
+
+SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
+SPDX-License-Identifier: GFDL-1.3-or-later
+SPDX-License-Identifier: CC-BY-SA-4.0
=====================================
Makefile.am
=====================================
@@ -7,6 +7,9 @@ SUBDIRS = src
test:
cd testsuite; make 1
+web:
+ cd src; make web
+
upload:
eval `gpg-agent --daemon`
# make sure $YYYYMMDD is set
@@ -130,6 +133,7 @@ reuse:
(wd=`pwd`; \
cd /tmp/reuse; \
git clone "$$wd"; \
+ cd parallel; \
reuse lint || \
(echo Use; \
echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \
=====================================
Makefile.in
=====================================
@@ -758,6 +758,9 @@ uninstall-am:
test:
cd testsuite; make 1
+web:
+ cd src; make web
+
upload:
eval `gpg-agent --daemon`
# make sure $YYYYMMDD is set
@@ -881,6 +884,7 @@ reuse:
(wd=`pwd`; \
cd /tmp/reuse; \
git clone "$$wd"; \
+ cd parallel; \
reuse lint || \
(echo Use; \
echo ' reuse addheader --copyright="Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc." --license="GPL-3.0-or-later" source-files'; \
=====================================
NEWS
=====================================
@@ -1,3 +1,127 @@
+20210822
+
+New in this release:
+
+* --ctag/--ctagstring colors the tag in different colors for each job.
+
+* You can use unit prefixes (k, m, g, etc) with -n -N -L.
+
+* Bug fixes and man page updates.
+
+News about GNU Parallel:
+
+* Parallelising jobs with GNU parallel
+ https://blog.ronin.cloud/gnu-parallel/
+
+* Use multiple CPU Cores with your Linux commands - awk, sed, bzip2,
+ grep, wc, etc. https://cdmana.com/2021/07/20210728132344693t.html
+
+* How to execute commands in parallel in Linux
+ https://net2.com/how-to-execute-commands-in-parallel-in-linux/
+
+
+20210722
+
+New in this release:
+
+* --results no longer prints the result to standard output (stdout) as
+ voted in
+ https://lists.gnu.org/archive/html/parallel/2020-12/msg00003.html
+
+* parset supports associative arrays in bash, ksh, zsh.
+
+* Online HTML is now generated by Sphinx.
+
+* Bug fixes and man page updates.
+
+News about GNU Parallel:
+
+* Cleaning Up Scanned Documents with Open Source Tools
+ https://kaerumy.medium.com/cleaning-up-scanned-documents-with-open-source-tools-9d87e15305b
+
+
+20210622
+
+New in this release:
+
+* Bug fixes and man page updates.
+
+News about GNU Parallel:
+
+* How to use GNU Parallel
+ https://techtipbits.com/linux/how-to-use-gnu-parallel/
+
+* How to Speed Up Bash Scripts with Multithreading and GNU Parallel
+ https://adamtheautomator.com/how-to-speed-up-bash-scripts-with-multithreading-and-gnu-parallel/
+
+* Use Parallel to split by line
+ https://madflex.de/use-parallel-to-split-by-line/
+
+* Optimizing long batch processes or ETL by using buff/cache properly
+ II (parallelizing network operations)
+ http://www.elsotanillo.net/2021/06/optimizing-long-batch-processes-or-etl-by-using-buff-cache-properly-ii-parallelizing-network-operations/
+
+* Parallelization 3: GNU Parallel https://www.youtube.com/watch?v=Rl06WD60afA
+
+
+20210522
+
+New in this release:
+
+* --plus includes {%%regexp} and {##regexp}.
+
+* Bug fixes and man page updates.
+
+News about GNU Parallel:
+
+* Batch Calculate and Verify MD5 Checksum With GNU Parallel
+ https://omicx.cc/posts/2021-04-28-calculate-and-verify-md5-checksum-with-gnu-parallel/
+
+* HerrComp Gnu parallel, c++11 threads 2021 04 28
+ https://www.youtube.com/watch?v=wDd9F9nn0qA
+
+* Distributing embarrassingly parallel tasks GNU Parallel
+ https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/gnu-parallel/
+
+* Job Parallelization on Niagara
+ https://www.maryamdaryalal.com/post/job-parallelization-on-niagara
+
+* Use Parallel to split by line
+ https://madflex.de/use-parallel-to-split-by-line/
+
+* m1 multi-core batch convert with gpu parallel + ffmpeg
+ https://www.youtube.com/watch?v=hAuc0YsXv6A
+
+
+20210422
+
+New in this release:
+
+* Bug fixes and man page updates.
+
+News about GNU Parallel:
+
+* Bioinformatics tutorials - linux and shell advanced - parallel
+ https://www.youtube.com/watch?v=5leL8pyl0XA
+
+* GNU Parallel for quick gains
+ https://edbennett.github.io/high-performance-python/04-gnu-parallel/index.html
+
+* Processing Linux Commands in Parallel
+ https://www.baeldung.com/linux/processing-commands-in-parallel
+
+* GNU parallel
+ https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/running-your-jobs/gnu-parallel/
+
+* GNU Parallel 활용 가이드https://genoglobe.com/kribb/gnu_parallel
+
+* Parallel Grep and Awk
+ https://www.highonscience.com/blog/2021/03/21/parallel-grep/
+
+* Getting things done with shell scripting
+ https://doma.dev/blog/get-things-done-with-bash/
+
+
20210322
* Bug fixes and man page updates.
=====================================
README
=====================================
@@ -40,13 +40,13 @@ installation.
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
- $ sha1sum install.sh | grep 67bd7bc7dc20aff99eb8f1266574dadb
- 12345678 67bd7bc7 dc20aff9 9eb8f126 6574dadb
- $ md5sum install.sh | grep b7a15cdbb07fb6e11b0338577bc1780f
- b7a15cdb b07fb6e1 1b033857 7bc1780f
- $ sha512sum install.sh | grep 186000b62b66969d7506ca4f885e0c80e02a22444
- 6f25960b d4b90cf6 ba5b76de c1acdf39 f3d24249 72930394 a4164351 93a7668d
- 21ff9839 6f920be5 186000b6 2b66969d 7506ca4f 885e0c80 e02a2244 40e8a43f
+ $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
+ 12345678 883c667e 01eed62f 975ad28b 6d50e22a
+ $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
+ cc21b4c9 43fd03e9 3ae1ae49 e28573c0
+ $ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
+ 79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
+ fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
This will literally install faster than reading the rest of this
@@ -57,11 +57,11 @@ document.
Full installation of GNU Parallel is as simple as:
- wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2
- wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2.sig
- gpg parallel-20210322.tar.bz2.sig
- bzip2 -dc parallel-20210322.tar.bz2 | tar xvf -
- cd parallel-20210322
+ wget https://ftpmirror.gnu.org/parallel/parallel-20210822.tar.bz2
+ wget https://ftpmirror.gnu.org/parallel/parallel-20210822.tar.bz2.sig
+ gpg parallel-20210822.tar.bz2.sig
+ bzip2 -dc parallel-20210822.tar.bz2 | tar xvf -
+ cd parallel-20210822
./configure && make && sudo make install
@@ -70,11 +70,11 @@ Full installation of GNU Parallel is as simple as:
If you are not root you can add ~/bin to your path and install in
~/bin and ~/share:
- wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2
- wget https://ftpmirror.gnu.org/parallel/parallel-20210322.tar.bz2.sig
- gpg parallel-20210322.tar.bz2.sig
- bzip2 -dc parallel-20210322.tar.bz2 | tar xvf -
- cd parallel-20210322
+ wget https://ftpmirror.gnu.org/parallel/parallel-20210822.tar.bz2
+ wget https://ftpmirror.gnu.org/parallel/parallel-20210822.tar.bz2.sig
+ gpg parallel-20210822.tar.bz2.sig
+ bzip2 -dc parallel-20210822.tar.bz2 | tar xvf -
+ cd parallel-20210822
./configure --prefix=$HOME && make && make install
Or if your system lacks 'make' you can simply copy src/parallel
@@ -122,8 +122,8 @@ will love you for it.
When using programs that use GNU Parallel to process data for
publication please cite:
- Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').
- Zenodo. https://doi.org/10.5281/zenodo.4628277
+ Tange, O. (2021, August 22). GNU Parallel 20210822 ('Kabul').
+ Zenodo. https://doi.org/10.5281/zenodo.5233953
Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and
@@ -142,3 +142,6 @@ if you system has split up Perl into multiple packages then these are
the important ones:
opkg install perlbase-getopt perlbase-ipc procps-ng-ps perlbase-mime
+
+SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
+SPDX-License-Identifier: GPL-3.0-or-later
=====================================
configure
=====================================
@@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.69 for parallel 20210322.
+# Generated by GNU Autoconf 2.69 for parallel 20210822.
#
# Report bugs to <bug-parallel at gnu.org>.
#
@@ -579,8 +579,8 @@ MAKEFLAGS=
# Identity of this package.
PACKAGE_NAME='parallel'
PACKAGE_TARNAME='parallel'
-PACKAGE_VERSION='20210322'
-PACKAGE_STRING='parallel 20210322'
+PACKAGE_VERSION='20210822'
+PACKAGE_STRING='parallel 20210822'
PACKAGE_BUGREPORT='bug-parallel at gnu.org'
PACKAGE_URL=''
@@ -1214,7 +1214,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures parallel 20210322 to adapt to many kinds of systems.
+\`configure' configures parallel 20210822 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@@ -1281,7 +1281,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
- short | recursive ) echo "Configuration of parallel 20210322:";;
+ short | recursive ) echo "Configuration of parallel 20210822:";;
esac
cat <<\_ACEOF
@@ -1357,7 +1357,7 @@ fi
test -n "$ac_init_help" && exit $ac_status
if $ac_init_version; then
cat <<\_ACEOF
-parallel configure 20210322
+parallel configure 20210822
generated by GNU Autoconf 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
@@ -1374,7 +1374,7 @@ cat >config.log <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by parallel $as_me 20210322, which was
+It was created by parallel $as_me 20210822, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ $0 $@
@@ -2237,7 +2237,7 @@ fi
# Define the identity of the package.
PACKAGE='parallel'
- VERSION='20210322'
+ VERSION='20210822'
cat >>confdefs.h <<_ACEOF
@@ -2880,7 +2880,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
# report actual input values of CONFIG_FILES etc. instead of their
# values after options handling.
ac_log="
-This file was extended by parallel $as_me 20210322, which was
+This file was extended by parallel $as_me 20210822, which was
generated by GNU Autoconf 2.69. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@@ -2942,7 +2942,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
ac_cs_version="\\
-parallel config.status 20210322
+parallel config.status 20210822
configured by $0, generated by GNU Autoconf 2.69,
with options \\"\$ac_cs_config\\"
=====================================
configure.ac
=====================================
@@ -1,4 +1,4 @@
-AC_INIT([parallel], [20210322], [bug-parallel at gnu.org])
+AC_INIT([parallel], [20210822], [bug-parallel at gnu.org])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([
=====================================
src/Makefile.am
=====================================
@@ -1,3 +1,7 @@
+# SPDX-FileCopyrightText: 2002-2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
bin_SCRIPTS = parallel sql niceload parcat parset parsort \
env_parallel env_parallel.ash env_parallel.bash \
env_parallel.csh env_parallel.dash env_parallel.fish \
@@ -30,6 +34,12 @@ doc_DATA = parallel.html env_parallel.html sem.html sql.html \
parallel_cheat_bw.pdf
endif
+web: sphinx
+ true
+
+sphinx: *.rst
+ cd sphinx && make && cd ..
+
# Build documentation file if the tool to build exists.
# Otherwise: Use the distributed version
parallel.1: parallel.pod
=====================================
src/Makefile.in
=====================================
@@ -14,6 +14,10 @@
@SET_MAKE@
+# SPDX-FileCopyrightText: 2002-2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
VPATH = @srcdir@
am__is_gnu_make = { \
@@ -629,6 +633,12 @@ install-exec-hook:
rm "$(DESTDIR)$(bindir)"/sem || true
$(LN_S) parallel "$(DESTDIR)$(bindir)"/sem
+web: sphinx
+ true
+
+sphinx: *.rst
+ cd sphinx && make && cd ..
+
# Build documentation file if the tool to build exists.
# Otherwise: Use the distributed version
parallel.1: parallel.pod
=====================================
src/env_parallel
=====================================
@@ -17,6 +17,9 @@
# along with this program; if not, see <http://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA
+#
+# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
+# SPDX-License-Identifier: GPL-3.0-or-later
grepq() {
# grep -q for systems without -q
@@ -80,45 +83,45 @@ env_parallel only works if it is a function. Do the below and restart your shell
bash: Put this in $HOME/.bashrc: . `which env_parallel.bash`
E.g. by doing: echo '. `which env_parallel.bash`' >> $HOME/.bashrc
- Supports: aliases, functions, variables, arrays
-
-zsh: Put this in $HOME/.zshrc: . `which env_parallel.zsh`
- E.g. by doing: echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
- Supports: functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
fish: Put this in $HOME/.config/fish/config.fish:
. (which env_parallel.fish)
E.g. by doing:
echo '. (which env_parallel.fish)' >> $HOME/.config/fish/config.fish
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
ksh: Put this in $HOME/.kshrc: source `which env_parallel.ksh`
E.g. by doing: echo 'source `which env_parallel.ksh`' >> $HOME/.kshrc
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
mksh: Put this in $HOME/.mkshrc: source `which env_parallel.mksh`
E.g. by doing: echo 'source `which env_parallel.mksh`' >> $HOME/.mkshrc
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
pdksh: Put this in $HOME/.profile: source `which env_parallel.pdksh`
E.g. by doing: echo '. `which env_parallel.pdksh`' >> $HOME/.profile
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
+
+zsh: Put this in $HOME/.zshrc: . `which env_parallel.zsh`
+ E.g. by doing: echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
+ Supports: variables, functions, arrays
ash: Put this in $HOME/.profile: . `which env_parallel.ash`
E.g. by doing: echo '. `which env_parallel.ash`' >> $HOME/.profile
- Supports: aliases, variables
+ Supports: variables, aliases
dash: Put this in $HOME/.profile: . `which env_parallel.dash`
E.g. by doing: echo '. `which env_parallel.dash`' >> $HOME/.profile
- Supports: aliases, variables
+ Supports: variables, aliases
csh: Put this in $HOME/.cshrc: source `which env_parallel.csh`
E.g. by doing: echo 'source `which env_parallel.csh`' >> $HOME/.cshrc
- Supports: aliases, variables, arrays with no special chars
+ Supports: variables, aliases, arrays with no special chars
tcsh: Put this in $HOME/.tcshrc: source `which env_parallel.tcsh`
E.g. by doing: echo 'source `which env_parallel.tcsh`' >> $HOME/.tcshrc
- Supports: aliases, variables, arrays with no special chars
+ Supports: variables, aliases, arrays with no special chars
To install in all shells run:
=====================================
src/env_parallel.ash
=====================================
@@ -385,7 +385,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
=====================================
src/env_parallel.bash
=====================================
@@ -32,12 +32,14 @@ env_parallel() {
# env_parallel.bash
_names_of_ALIASES() {
- compgen -a
+ # No aliases will return false. This error should be ignored.
+ compgen -a || true
}
_bodies_of_ALIASES() {
local _i
- for _i in $@; do
- if [ $(alias $_i | wc -l) == 1 ] ; then
+ for _i in "$@"; do
+ # shellcheck disable=SC2046
+ if [ $(alias "$_i" | wc -l) == 1 ] ; then
true Alias is a single line. Good.
else
_warning_PAR "Alias '$_i' contains newline."
@@ -64,6 +66,7 @@ env_parallel() {
echo '(_|TIMEOUT|GROUPS|FUNCNAME|DIRSTACK|PIPESTATUS|USERNAME|BASHPID|BASH_[A-Z_]+)'
}
_ignore_READONLY() {
+ # shellcheck disable=SC1078,SC1079,SC2026
readonly | perl -e '@r = map {
chomp;
# sh on UnixWare: readonly TIMEOUT
@@ -195,15 +198,16 @@ env_parallel() {
END { exit not $exit }'
}
_warning_PAR() {
- echo "env_parallel: Warning: $@" >&2
+ echo "env_parallel: Warning: $*" >&2
}
_error_PAR() {
- echo "env_parallel: Error: $@" >&2
+ echo "env_parallel: Error: $*" >&2
}
# Bash is broken in version 3.2.25 and 4.2.39
# The crazy '[ "`...`" == "" ]' is needed for the same reason
if [ "`_which_PAR parallel`" == "" ]; then
+ # shellcheck disable=SC2016
_error_PAR 'parallel must be in $PATH.'
return 255
fi
@@ -225,7 +229,7 @@ env_parallel() {
(_names_of_ALIASES;
_names_of_FUNCTIONS;
_names_of_VARIABLES) |
- cat > $HOME/.parallel/ignored_vars
+ cat > "$HOME"/.parallel/ignored_vars
return 0
fi
@@ -365,13 +369,6 @@ _parset_main() {
# parset "var_a4 var_b4 var_c4" echo ::: {1..3}
# echo $var_c4
- _make_TEMP() {
- # mktemp does not exist on some OS
- perl -e 'use File::Temp qw(tempfile);
- $ENV{"TMPDIR"} ||= "/tmp";
- print((tempfile(DIR=>$ENV{"TMPDIR"}, TEMPLATE => "parXXXXX"))[1])'
- }
-
_parset_NAME="$1"
if [ "$_parset_NAME" = "" ] ; then
echo parset: Error: No destination variable given. >&2
@@ -387,7 +384,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@@ -402,45 +399,18 @@ _parset_main() {
return 255
fi
shift
- echo "$_parset_NAME" |
- perl -ne 'chomp;for (split /[, ]/) {
- # Allow: var_32 var[3]
- if(not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/) {
- print STDERR "parset: Error: $_ is an invalid variable name.\n";
- print STDERR "parset: Error: Variable names must be letter followed by letters or digits.\n";
- print STDERR "parset: Error: Usage:\n";
- print STDERR "parset: Error: parset varname GNU Parallel options and command\n";
- $exitval = 255;
- }
- }
- exit $exitval;
- ' || return 255
- _exit_FILE=`_make_TEMP`
- if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
- # $_parset_NAME contains , or space
- # Split on , or space to get the names
- eval "$(
- # Compute results into files
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- # var1=`cat tmpfile1; rm tmpfile1`
- # var2=`cat tmpfile2; rm tmpfile2`
- parallel --plain -q echo {2}='`cat {1}; rm {1}`' :::: - :::+ $(
- echo "$_parset_NAME" | perl -pe 's/,/ /g'
- )
- );
- "
+
+ # Bash: declare -A myassoc=( )
+ # Zsh: typeset -A myassoc=( )
+ # Ksh: typeset -A myassoc=( )
+ if (typeset -p "$_parset_NAME" 2>/dev/null; echo) |
+ perl -ne 'exit not (/^declare[^=]+-A|^typeset[^=]+-A/)' ; then
+ # This is an associative array
+ eval "`$_parset_PARALLEL_PRG -k --parset assoc,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
else
- # $_parset_NAME does not contain , or space
- # => $_parset_NAME is the name of the array to put data into
- # Supported in: bash zsh ksh mksh
- # Arrays do not work in: sh ash dash
- eval "$_parset_NAME=( $(
- # Compute results into files. Save exit value
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- perl -pe 'chop;$_="\"\`cat $_; rm $_\`\" "'
- ) )"
+ # This is a normal array or a list of variable names
+ eval "`$_parset_PARALLEL_PRG -k --parset var,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
fi
- unset _parset_NAME _parset_PARALLEL_PRG _parallel_exit_CODE
- # Unset _exit_FILE before return
- eval "unset _exit_FILE; return \`cat $_exit_FILE; rm $_exit_FILE\`"
}
=====================================
src/env_parallel.dash
=====================================
@@ -385,7 +385,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
=====================================
src/env_parallel.ksh
=====================================
@@ -69,8 +69,10 @@ env_parallel() {
# sh on UnixWare: readonly TIMEOUT
# ash: readonly var='val'
# ksh: var='val'
- s/^(readonly )?([^= ]*)(=.*|)$/$2/ or
+ # mksh: PIPESTATUS[0]
+ s/^(readonly )?([^=\[ ]*?)(\[\d+\])?(=.*|)$/$2/ or
# bash: declare -ar BASH_VERSINFO=([0]="4" [1]="4")
+ # zsh: typeset -r var='val'
s/^\S+\s+\S+\s+(\S[^=]*)(=.*|$)/$1/;
$_ } <>;
$vars = join "|",map { quotemeta $_ } @r;
@@ -214,7 +216,7 @@ env_parallel() {
(_names_of_ALIASES;
_names_of_FUNCTIONS;
_names_of_VARIABLES) |
- cat > $HOME/.parallel/ignored_vars
+ cat > "$HOME"/.parallel/ignored_vars
return 0
fi
@@ -346,13 +348,6 @@ _parset_main() {
# parset "var_a4 var_b4 var_c4" echo ::: {1..3}
# echo $var_c4
- _make_TEMP() {
- # mktemp does not exist on some OS
- perl -e 'use File::Temp qw(tempfile);
- $ENV{"TMPDIR"} ||= "/tmp";
- print((tempfile(DIR=>$ENV{"TMPDIR"}, TEMPLATE => "parXXXXX"))[1])'
- }
-
_parset_NAME="$1"
if [ "$_parset_NAME" = "" ] ; then
echo parset: Error: No destination variable given. >&2
@@ -368,7 +363,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@@ -383,45 +378,18 @@ _parset_main() {
return 255
fi
shift
- echo "$_parset_NAME" |
- perl -ne 'chomp;for (split /[, ]/) {
- # Allow: var_32 var[3]
- if(not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/) {
- print STDERR "parset: Error: $_ is an invalid variable name.\n";
- print STDERR "parset: Error: Variable names must be letter followed by letters or digits.\n";
- print STDERR "parset: Error: Usage:\n";
- print STDERR "parset: Error: parset varname GNU Parallel options and command\n";
- $exitval = 255;
- }
- }
- exit $exitval;
- ' || return 255
- _exit_FILE=`_make_TEMP`
- if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
- # $_parset_NAME contains , or space
- # Split on , or space to get the names
- eval "$(
- # Compute results into files
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- # var1=`cat tmpfile1; rm tmpfile1`
- # var2=`cat tmpfile2; rm tmpfile2`
- parallel -q echo {2}='`cat {1}; rm {1}`' :::: - :::+ $(
- echo "$_parset_NAME" | perl -pe 's/,/ /g'
- )
- );
- "
+
+ # Bash: declare -A myassoc=( )
+ # Zsh: typeset -A myassoc=( )
+ # Ksh: typeset -A myassoc=( )
+ if (typeset -p "$_parset_NAME" 2>/dev/null; echo) |
+ perl -ne 'exit not (/^declare[^=]+-A|^typeset[^=]+-A/)' ; then
+ # This is an associative array
+ eval "`$_parset_PARALLEL_PRG -k --parset assoc,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
else
- # $_parset_NAME does not contain , or space
- # => $_parset_NAME is the name of the array to put data into
- # Supported in: bash zsh ksh mksh
- # Arrays do not work in: sh ash dash
- eval "$_parset_NAME=( $(
- # Compute results into files. Save exit value
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- perl -pe 'chop;$_="\"\`cat $_; rm $_\`\" "'
- ) )"
+ # This is a normal array or a list of variable names
+ eval "`$_parset_PARALLEL_PRG -k --parset var,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
fi
- unset _parset_NAME _parset_PARALLEL_PRG _parallel_exit_CODE
- # Unset _exit_FILE before return
- eval "unset _exit_FILE; return \`cat $_exit_FILE; rm $_exit_FILE\`"
}
=====================================
src/env_parallel.mksh
=====================================
@@ -74,6 +74,7 @@ env_parallel() {
# mksh: PIPESTATUS[0]
s/^(readonly )?([^=\[ ]*?)(\[\d+\])?(=.*|)$/$2/ or
# bash: declare -ar BASH_VERSINFO=([0]="4" [1]="4")
+ # zsh: typeset -r var='val'
s/^\S+\s+\S+\s+(\S[^=]*)(=.*|$)/$1/;
$_ } <>;
$vars = join "|",map { quotemeta $_ } @r;
@@ -217,7 +218,7 @@ env_parallel() {
(_names_of_ALIASES;
_names_of_FUNCTIONS;
_names_of_VARIABLES) |
- cat > $HOME/.parallel/ignored_vars
+ cat > "$HOME"/.parallel/ignored_vars
return 0
fi
@@ -349,13 +350,6 @@ _parset_main() {
# parset "var_a4 var_b4 var_c4" echo ::: {1..3}
# echo $var_c4
- _make_TEMP() {
- # mktemp does not exist on some OS
- perl -e 'use File::Temp qw(tempfile);
- $ENV{"TMPDIR"} ||= "/tmp";
- print((tempfile(DIR=>$ENV{"TMPDIR"}, TEMPLATE => "parXXXXX"))[1])'
- }
-
_parset_NAME="$1"
if [ "$_parset_NAME" = "" ] ; then
echo parset: Error: No destination variable given. >&2
@@ -371,7 +365,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@@ -386,45 +380,18 @@ _parset_main() {
return 255
fi
shift
- echo "$_parset_NAME" |
- perl -ne 'chomp;for (split /[, ]/) {
- # Allow: var_32 var[3]
- if(not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/) {
- print STDERR "parset: Error: $_ is an invalid variable name.\n";
- print STDERR "parset: Error: Variable names must be letter followed by letters or digits.\n";
- print STDERR "parset: Error: Usage:\n";
- print STDERR "parset: Error: parset varname GNU Parallel options and command\n";
- $exitval = 255;
- }
- }
- exit $exitval;
- ' || return 255
- _exit_FILE=`_make_TEMP`
- if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
- # $_parset_NAME contains , or space
- # Split on , or space to get the names
- eval "$(
- # Compute results into files
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- # var1=`cat tmpfile1; rm tmpfile1`
- # var2=`cat tmpfile2; rm tmpfile2`
- parallel -q echo {2}='`cat {1}; rm {1}`' :::: - :::+ $(
- echo "$_parset_NAME" | perl -pe 's/,/ /g'
- )
- );
- "
+
+ # Bash: declare -A myassoc=( )
+ # Zsh: typeset -A myassoc=( )
+ # Ksh: typeset -A myassoc=( )
+ if (typeset -p "$_parset_NAME" 2>/dev/null; echo) |
+ perl -ne 'exit not (/^declare[^=]+-A|^typeset[^=]+-A/)' ; then
+ # This is an associative array
+ eval "`$_parset_PARALLEL_PRG -k --parset assoc,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
else
- # $_parset_NAME does not contain , or space
- # => $_parset_NAME is the name of the array to put data into
- # Supported in: bash zsh ksh mksh
- # Arrays do not work in: sh ash dash
- eval "$_parset_NAME=( $(
- # Compute results into files. Save exit value
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- perl -pe 'chop;$_="\"\`cat $_; rm $_\`\" "'
- ) )"
+ # This is a normal array or a list of variable names
+ eval "`$_parset_PARALLEL_PRG -k --parset var,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
fi
- unset _parset_NAME _parset_PARALLEL_PRG _parallel_exit_CODE
- # Unset _exit_FILE before return
- eval "unset _exit_FILE; return \`cat $_exit_FILE; rm $_exit_FILE\`"
}
=====================================
src/env_parallel.sh
=====================================
@@ -27,6 +27,7 @@
#
# SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
# SPDX-License-Identifier: GPL-3.0-or-later
+# shellcheck disable=SC2006
env_parallel() {
# env_parallel.sh
@@ -36,8 +37,8 @@ env_parallel() {
for _i in `alias 2>/dev/null | perl -ne 's/^alias //;s/^(\S+)=.*/$1/ && print' 2>/dev/null`; do
# Check if this name really is an alias
# or just part of a multiline alias definition
- if alias $_i >/dev/null 2>/dev/null; then
- echo $_i
+ if alias "$_i" >/dev/null 2>/dev/null; then
+ echo "$_i"
fi
done
}
@@ -47,7 +48,7 @@ env_parallel() {
# alias myalias='definition' (FreeBSD ash)
# so remove 'alias ' from first line
for _i in "$@"; do
- echo 'alias '"`alias $_i | perl -pe '1..1 and s/^alias //'`"
+ echo 'alias '"`alias "$_i" | perl -pe '1..1 and s/^alias //'`"
done
}
_names_of_maybe_FUNCTIONS() {
@@ -55,6 +56,7 @@ env_parallel() {
}
_names_of_FUNCTIONS() {
# myfunc is a function
+ # shellcheck disable=SC2046
LANG=C type `_names_of_maybe_FUNCTIONS` |
perl -ne '/^(\S+) is a function$/ and not $seen{$1}++ and print "$1\n"'
}
@@ -70,7 +72,7 @@ env_parallel() {
for _i in "$@"
do
perl -e 'print @ARGV' "$_i="
- eval echo \"\$$_i\" | perl -e '$/=undef; $a=<>; chop($a); print $a' |
+ eval echo "\"\$$_i\"" | perl -e '$/=undef; $a=<>; chop($a); print $a' |
perl -pe 's/[\002-\011\013-\032\\\#\?\`\(\)\{\}\[\]\^\*\<\=\>\~\|\; \"\!\$\&\202-\377]/\\$&/go;'"s/'/\\\'/g; s/[\n]/'\\n'/go;";
echo
done
@@ -80,6 +82,7 @@ env_parallel() {
echo '(_|TIMEOUT)'
}
_ignore_READONLY() {
+ # shellcheck disable=SC1078,SC1079,SC2026
readonly | perl -e '@r = map {
chomp;
# sh on UnixWare: readonly TIMEOUT
@@ -203,15 +206,16 @@ env_parallel() {
END { exit not $exit }'
}
_warning_PAR() {
- echo "env_parallel: Warning: $@" >&2
+ echo "env_parallel: Warning: $*" >&2
}
_error_PAR() {
- echo "env_parallel: Error: $@" >&2
+ echo "env_parallel: Error: $*" >&2
}
if _which_PAR parallel >/dev/null; then
true parallel found in path
else
+ # shellcheck disable=SC2016
_error_PAR 'parallel must be in $PATH.'
return 255
fi
@@ -231,7 +235,7 @@ env_parallel() {
(_names_of_ALIASES;
_names_of_FUNCTIONS;
_names_of_VARIABLES) |
- cat > $HOME/.parallel/ignored_vars
+ cat > "$HOME"/.parallel/ignored_vars
return 0
fi
@@ -310,6 +314,7 @@ env_parallel() {
unset _remove_bad_NAMES _grep_REGEXP
unset _prefix_PARALLEL_ENV
# Test if environment is too big
+ # shellcheck disable=SC2092
if `_which_PAR true` >/dev/null 2>/dev/null ; then
parallel "$@"
_parallel_exit_CODE=$?
@@ -385,7 +390,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@@ -417,12 +422,13 @@ _parset_main() {
if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
# $_parset_NAME contains , or space
# Split on , or space to get the names
+ # shellcheck disable=SC2016,SC2046
eval "`
# Compute results into files
($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
# var1= cat tmpfile1; rm tmpfile1
# var2= cat tmpfile2; rm tmpfile2
- parallel -q echo {2}='\`cat {1}; rm {1}\`' :::: - :::+ \`
+ parallel --plain -q echo '{2}=\`cat {1}; rm {1}\`' :::: - :::+ \`
echo "$_parset_NAME" |
perl -pe 's/,/ /g'
\`
=====================================
src/env_parallel.zsh
=====================================
@@ -208,7 +208,7 @@ env_parallel() {
(_names_of_ALIASES;
_names_of_FUNCTIONS;
_names_of_VARIABLES) |
- cat > $HOME/.parallel/ignored_vars
+ cat > "$HOME"/.parallel/ignored_vars
return 0
fi
@@ -340,13 +340,6 @@ _parset_main() {
# parset "var_a4 var_b4 var_c4" echo ::: {1..3}
# echo $var_c4
- _make_TEMP() {
- # mktemp does not exist on some OS
- perl -e 'use File::Temp qw(tempfile);
- $ENV{"TMPDIR"} ||= "/tmp";
- print((tempfile(DIR=>$ENV{"TMPDIR"}, TEMPLATE => "parXXXXX"))[1])'
- }
-
_parset_NAME="$1"
if [ "$_parset_NAME" = "" ] ; then
echo parset: Error: No destination variable given. >&2
@@ -362,7 +355,7 @@ _parset_main() {
return 255
fi
if [ "$_parset_NAME" = "--version" ] ; then
- echo "parset 20210322 (GNU parallel `parallel --minversion 1`)"
+ echo "parset 20210822 (GNU parallel `parallel --minversion 1`)"
echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software"
echo "Foundation, Inc."
echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>"
@@ -377,45 +370,18 @@ _parset_main() {
return 255
fi
shift
- echo "$_parset_NAME" |
- perl -ne 'chomp;for (split /[, ]/) {
- # Allow: var_32 var[3]
- if(not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/) {
- print STDERR "parset: Error: $_ is an invalid variable name.\n";
- print STDERR "parset: Error: Variable names must be letter followed by letters or digits.\n";
- print STDERR "parset: Error: Usage:\n";
- print STDERR "parset: Error: parset varname GNU Parallel options and command\n";
- $exitval = 255;
- }
- }
- exit $exitval;
- ' || return 255
- _exit_FILE=`_make_TEMP`
- if perl -e 'exit not grep /,| /, @ARGV' "$_parset_NAME" ; then
- # $_parset_NAME contains , or space
- # Split on , or space to get the names
- eval "$(
- # Compute results into files
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- # var1=`cat tmpfile1; rm tmpfile1`
- # var2=`cat tmpfile2; rm tmpfile2`
- parallel -q echo {2}='`cat {1}; rm {1}`' :::: - :::+ $(
- echo "$_parset_NAME" | perl -pe 's/,/ /g'
- )
- );
- "
+
+ # Bash: declare -A myassoc=( )
+ # Zsh: typeset -A myassoc=( )
+ # Ksh: typeset -A myassoc=( )
+ if (typeset -p "$_parset_NAME" 2>/dev/null; echo) |
+ perl -ne 'exit not (/^declare[^=]+-A|^typeset[^=]+-A/)' ; then
+ # This is an associative array
+ eval "`$_parset_PARALLEL_PRG -k --parset assoc,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
else
- # $_parset_NAME does not contain , or space
- # => $_parset_NAME is the name of the array to put data into
- # Supported in: bash zsh ksh mksh
- # Arrays do not work in: sh ash dash
- eval "$_parset_NAME=( $(
- # Compute results into files. Save exit value
- ($_parset_PARALLEL_PRG --files -k "$@"; echo $? > "$_exit_FILE") |
- perl -pe 'chop;$_="\"\`cat $_; rm $_\`\" "'
- ) )"
+ # This is a normal array or a list of variable names
+ eval "`$_parset_PARALLEL_PRG -k --parset var,"$_parset_NAME" "$@"`"
+ # The eval returns the function!
fi
- unset _parset_NAME _parset_PARALLEL_PRG _parallel_exit_CODE
- # Unset _exit_FILE before return
- eval "unset _exit_FILE; return \`cat $_exit_FILE; rm $_exit_FILE\`"
}
=====================================
src/niceload
=====================================
@@ -26,7 +26,7 @@
use strict;
use Getopt::Long;
$Global::progname="niceload";
-$Global::version = 20210322;
+$Global::version = 20210822;
Getopt::Long::Configure("bundling","require_order");
get_options_from_array(\@ARGV) || die_usage();
if($opt::version) {
=====================================
src/parallel
=====================================
@@ -14,7 +14,7 @@
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
-# along with this program; if not, see <http://www.gnu.org/licenses/>
+# along with this program; if not, see <https://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA
#
@@ -562,7 +562,8 @@ sub pipe_part_files(@) {
my ($file) = @_;
my $buf = "";
if(not -f $file and not -b $file) {
- ::error("$file is not a seekable file.");
+ ::error("--pipepart only works on seekable files, not streams/pipes.",
+ "$file is not a seekable file.");
::wait_and_exit(255);
}
my $header = find_header(\$buf,open_or_exit($file));
@@ -969,6 +970,7 @@ sub spreadstdin() {
my $header = find_header(\$buf,$in);
my $anything_written;
my $eof;
+ my $garbage_read;
sub read_block() {
# Read a --blocksize from STDIN
@@ -1023,23 +1025,37 @@ sub spreadstdin() {
# Pass records of N regexps
# -N => (start..*?end){n}
# -L -N => (start..*?end){n*l}
- my $read_n_lines = -1+
+ if(not $garbage_read) {
+ $garbage_read = 1;
+ if($buf !~ /^$recstart/o) {
+ # Buf does not start with $recstart => There is garbage.
+ # Make a single record of the garbage
+ if($buf =~
+ /(?s:^)(
+ (?:(?:(?!$recend$recstart)(?s:.))*?$recend)
+ )
+ # Followed by recstart
+ (?=$recstart)/mox and length $1 > 0) {
+ $anything_written +=
+ write_record_to_pipe($chunk_number++,\$header,\$buf,
+ $recstart,$recend,length $1);
+ shorten(\$buf,length $1);
+ }
+ }
+ }
+
+ my $n_records =
$Global::max_number_of_args * ($Global::max_lines || 1);
# (?!negative lookahead) is needed to avoid backtracking
# See: https://unix.stackexchange.com/questions/439356/
+ # (?s:.) = (.|[\n]) but faster
while($buf =~
- /(
- # Either recstart or at least one char from start
- ^(?: $recstart | .)
- # followed something
- (?:(?!$recend$recstart).)*?
- # and then recend
- $recend
- # Then n-1 times recstart.*recend
- (?:$recstart(?:(?!$recend$recstart).)*?$recend){$read_n_lines}
+ /(?s:^)(
+ # n more times recstart.*recend
+ (?:$recstart(?:(?!$recend$recstart)(?s:.))*?$recend){$n_records}
)
# Followed by recstart
- (?=$recstart)/osx) {
+ (?=$recstart)/mox and length $1 > 0) {
$anything_written +=
write_record_to_pipe($chunk_number++,\$header,\$buf,
$recstart,$recend,length $1);
@@ -1050,7 +1066,8 @@ sub spreadstdin() {
sub pass_regexp() {
# Find the last recend-recstart in $buf
$eof and return;
- if($buf =~ /^(.*$recend)$recstart.*?$/os) {
+ # (?s:.) = (.|[\n]) but faster
+ if($buf =~ /^((?s:.)*$recend)$recstart(?s:.)*?$/mox) {
$anything_written +=
write_record_to_pipe($chunk_number++,\$header,\$buf,
$recstart,$recend,length $1);
@@ -1230,20 +1247,18 @@ sub recstartrecend() {
$recend = $opt::recend;
if($opt::regexp and $recend eq '') {
# --regexp --recend ''
- $recend = '.';
+ $recend = '(?s:.)';
}
}
if($opt::regexp) {
+ # Do not allow /x comments - to avoid having to quote space
+ $recstart = "(?-x:".$recstart.")";
+ $recend = "(?-x:".$recend.")";
# If $recstart/$recend contains '|'
- # this should only apply to the regexp
+ # the | should only apply to the regexp
$recstart = "(?:".$recstart.")";
$recend = "(?:".$recend.")";
- # Quote # and space
- $recstart =~ s/#/\\#/g;
- $recend =~ s/#/\\#/g;
- $recstart =~ s/ /\\ /g;
- $recend =~ s/ /\\ /g;
} else {
# $recstart/$recend = printf strings (\n)
$recstart =~ s/\\([0rnt\'\"\\])/"qq|\\$1|"/gee;
@@ -1541,6 +1556,8 @@ sub options_hash() {
"nice=i" => \$opt::nice,
"tag" => \$opt::tag,
"tagstring|tag-string=s" => \$opt::tagstring,
+ "ctag" => \$opt::ctag,
+ "ctagstring|ctag-string=s" => \$opt::ctagstring,
"onall" => \$opt::onall,
"nonall" => \$opt::nonall,
"filter-hosts|filterhosts|filter-host" => \$opt::filter_hosts,
@@ -1603,18 +1620,18 @@ sub options_hash() {
# xargs-compatibility - implemented, man, testsuite
"max-procs|P=s" => \$opt::jobs,
"delimiter|d=s" => \$opt::d,
- "max-chars|s=i" => \$opt::max_chars,
+ "max-chars|s=s" => \$opt::max_chars,
"arg-file|a=s" => \@opt::a,
"no-run-if-empty|r" => \$opt::r,
"replace|i:s" => \$opt::i,
"E=s" => \$opt::eof,
"eof|e:s" => \$opt::eof,
- "max-args|maxargs|n=i" => \$opt::max_args,
- "max-replace-args|N=i" => \$opt::max_replace_args,
+ "max-args|maxargs|n=s" => \$opt::max_args,
+ "max-replace-args|N=s" => \$opt::max_replace_args,
"colsep|col-sep|C=s" => \$opt::colsep,
"csv"=> \$opt::csv,
"help|h" => \$opt::help,
- "L=f" => \$opt::L,
+ "L=s" => \$opt::L,
"max-lines|l:f" => \$opt::max_lines,
"interactive|p" => \$opt::interactive,
"verbose|t" => \$opt::verbose,
@@ -1658,6 +1675,7 @@ sub options_hash() {
"hgrp|hostgrp|hostgroup|hostgroups" => \$opt::hostgroups,
"embed" => \$opt::embed,
"filter=s" => \@opt::filter,
+ "parset=s" => \$opt::parset,
);
}
@@ -1703,6 +1721,40 @@ sub get_options_from_array($@) {
return $retval;
}
+sub parse_parset() {
+ $Global::progname = "parset";
+ @Global::parset_vars = split /[ ,]/, $opt::parset;
+ my $var_or_assoc = shift @Global::parset_vars;
+ # Legal names: var _v2ar arrayentry[2]
+ my @illegal = (grep { not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/ }
+ @Global::parset_vars);
+ if(@illegal) {
+ ::error
+ ("@illegal is an invalid variable name.",
+ "Variable names must be letter followed by letters or digits.",
+ "Usage:",
+ " parset varname GNU Parallel options and command");
+ wait_and_exit(255);
+ }
+ if($var_or_assoc eq "assoc") {
+ my $var = shift @Global::parset_vars;
+ print "$var=(";
+ $Global::parset = "assoc";
+ $Global::parset_endstring=")\n";
+ } elsif($var_or_assoc eq "var") {
+ if($#Global::parset_vars > 0) {
+ $Global::parset = "var";
+ } else {
+ my $var = shift @Global::parset_vars;
+ print "$var=(";
+ $Global::parset = "array";
+ $Global::parset_endstring=")\n";
+ }
+ } else {
+ ::die_bug("parset: unknown '$opt::parset'");
+ }
+}
+
sub parse_options(@) {
# Returns: N/A
init_globals();
@@ -1730,11 +1782,21 @@ sub parse_options(@) {
}
::debug("init","Global::shell $Global::shell\n");
$Global::cshell = $Global::shell =~ m:(/[-a-z]*)?csh:;
+ if(defined $opt::parset) { parse_parset(); }
if(defined $opt::X) { $Global::ContextReplace = 1; }
if(defined $opt::silent) { $Global::verbose = 0; }
if(defined $opt::null) { $/ = "\0"; }
if(defined $opt::d) { $/ = unquote_printf($opt::d) }
parse_replacement_string_options();
+ $opt::tag ||= $opt::ctag;
+ $opt::tagstring ||= $opt::ctagstring;
+ if(defined $opt::ctag or defined $opt::ctagstring) {
+ $Global::color = 1;
+ }
+ if(defined $opt::tag and not defined $opt::tagstring) {
+ # Default = {}
+ $opt::tagstring = $Global::parensleft.$Global::parensright;
+ }
if(defined $opt::tagstring) {
$opt::tagstring = unquote_printf($opt::tagstring);
if($opt::tagstring =~ /\Q$Global::parensleft\E.*\Q$Global::parensright\E/
@@ -1751,6 +1813,7 @@ sub parse_options(@) {
if(defined $opt::verbose) { $Global::stderr_verbose = 1; }
if(defined $opt::eof) { $Global::end_of_file_string = $opt::eof; }
if(defined $opt::max_args) {
+ $opt::max_args = multiply_binary_prefix($opt::max_args);
$Global::max_number_of_args = $opt::max_args;
}
if(defined $opt::blocktimeout) {
@@ -1806,6 +1869,9 @@ sub parse_options(@) {
if(defined $opt::max_line_length_allowed) {
print Limits::Command::real_max_length(),"\n"; wait_and_exit(0);
}
+ if(defined $opt::max_chars) {
+ $opt::max_chars = multiply_binary_prefix($opt::max_chars);
+ }
if(defined $opt::version) { version(); wait_and_exit(0); }
if(defined $opt::record_env) { record_env(); wait_and_exit(0); }
if(defined $opt::show_limits) { show_limits(); }
@@ -1939,10 +2005,14 @@ sub parse_options(@) {
$opt::max_lines = 1;
$opt::null = 1;
$/ = "\0";
- } elsif ($opt::max_lines == 0) {
- # If not given (or if 0 is given) => 1
- $opt::max_lines = 1;
+ } else {
+ $opt::max_lines = multiply_binary_prefix($opt::max_lines);
+ if ($opt::max_lines == 0) {
+ # If not given (or if 0 is given) => 1
+ $opt::max_lines = 1;
+ }
}
+
$Global::max_lines = $opt::max_lines;
if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args
@@ -1952,6 +2022,7 @@ sub parse_options(@) {
# Read more than one arg at a time (-L, -N)
if(defined $opt::L) {
+ $opt::L = multiply_binary_prefix($opt::L);
$Global::max_lines = $opt::L;
if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args
@@ -1959,6 +2030,7 @@ sub parse_options(@) {
}
}
if(defined $opt::max_replace_args) {
+ $opt::max_replace_args = multiply_binary_prefix($opt::max_replace_args);
$Global::max_number_of_args = $opt::max_replace_args;
$Global::ContextReplace = 1;
}
@@ -1967,10 +2039,6 @@ sub parse_options(@) {
not ($opt::xargs or $opt::m)) {
$Global::ContextReplace = 1;
}
- if(defined $opt::tag and not defined $opt::tagstring) {
- # Default = {}
- $opt::tagstring = $Global::parensleft.$Global::parensright;
- }
if(grep /^$Global::arg_sep\+?$|^$Global::arg_file_sep\+?$/o, @ARGV) {
# Deal with ::: :::+ :::: and ::::+
@ARGV = read_args_from_command_line();
@@ -2173,7 +2241,7 @@ sub check_invalid_option_combinations() {
sub init_globals() {
# Defaults:
- $Global::version = 20210322;
+ $Global::version = 20210822;
$Global::progname = 'parallel';
$::name = "GNU Parallel";
$Global::infinity = 2**31;
@@ -2228,10 +2296,20 @@ sub init_globals() {
'{:(\d+?)}' => 'substr($_,0,$$1) = ""',
# Bash ${a:2:3}
'{:(\d+?):(\d+?)}' => '$_ = substr($_,$$1,$$2);',
+ # echo {#z.*z.} ::: z.z.z.foo => z.foo
+ # echo {##z.*z.} ::: z.z.z.foo => foo
# Bash ${a#bc}
- '{#([^#}][^}]*?)}' => 's/^$$1//;',
+ '{#([^#}][^}]*?)}' =>
+ '$nongreedy=::make_regexp_ungreedy($$1);s/^$nongreedy(.*)/$1/;',
+ # Bash ${a##bc}
+ '{##([^#}][^}]*?)}' => 's/^$$1//;',
+ # echo {%.z.*z} ::: foo.z.z.z => foo.z
+ # echo {%%.z.*z} ::: foo.z.z.z => foo
# Bash ${a%def}
- '{%([^}]+?)}' => 's/$$1$//;',
+ '{%([^}]+?)}' =>
+ '$nongreedy=::make_regexp_ungreedy($$1);s/(.*)$nongreedy$/$1/;',
+ # Bash ${a%%def}
+ '{%%([^}]+?)}' => 's/$$1$//;',
# Bash ${a/def/ghi} ${a/def/}
'{/([^}]+?)/([^}]*?)}' => 's/$$1/$$2/;',
# Bash ${a^a}
@@ -2493,7 +2571,7 @@ sub open_joblog() {
# $opt::joblog
# $opt::results
# $Global::job_already_run
- # %Global::fd
+ # %Global::fh
my $append = 0;
if(($opt::resume or $opt::resume_failed)
and
@@ -2599,7 +2677,7 @@ sub open_joblog() {
} else {
if($opt::joblog eq "-") {
# Use STDOUT as joblog
- $Global::joblog = $Global::fd{1};
+ $Global::joblog = $Global::fh{1};
} elsif(not open($Global::joblog, ">", $opt::joblog)) {
# Overwrite the joblog
::error("Cannot write to --joblog $opt::joblog.");
@@ -2628,8 +2706,8 @@ sub open_json_csv() {
# by forcing all other output to /dev/null
open my $fd, ">", "/dev/null" or
::die_bug("Can't >/dev/null in csv: $!");
- $Global::fd{1} = $fd;
- $Global::fd{2} = $fd;
+ $Global::fh{1} = $fd;
+ $Global::fh{2} = $fd;
} elsif($Global::csvsep) {
if(not open($Global::csv_fh,">",$opt::results)) {
::error("Cannot open results file `$opt::results': ".
@@ -3108,7 +3186,7 @@ sub save_stdin_stdout_stderr() {
# Remember the original STDIN, STDOUT and STDERR
# and file descriptors opened by the shell (e.g. 3>/tmp/foo)
# Uses:
- # %Global::fd
+ # %Global::fh
# $Global::original_stderr
# $Global::original_stdin
# Returns: N/A
@@ -3123,7 +3201,7 @@ sub save_stdin_stdout_stderr() {
# 2-argument-open is used to be compatible with old perl 5.8.0
# bug #43570: Perl 5.8.0 creates 61 files
if(open($fh,">&=$fdno")) {
- $Global::fd{$fdno}=$fh;
+ $Global::fh{$fdno}=$fh;
}
}
open $Global::original_stderr, ">&", "STDERR" or
@@ -3139,7 +3217,7 @@ sub enough_file_handles() {
# another job
# Uses:
# $opt::ungroup
- # %Global::fd
+ # %Global::fh
# Returns:
# 1 if ungrouped (thus not needing extra filehandles)
# 0 if too few filehandles
@@ -3151,7 +3229,7 @@ sub enough_file_handles() {
# open3 uses 2 extra filehandles temporarily
# We need a filehandle for each redirected file descriptor
# (normally just STDOUT and STDERR)
- for my $i (1..(7+2+keys %Global::fd)) {
+ for my $i (1..(7+2+keys %Global::fh)) {
$enough_filehandles &&= open($fh{$i}, "<", "/dev/null");
}
for (values %fh) { close $_; }
@@ -3835,7 +3913,8 @@ sub progress() {
my $arg = $Global::newest_job ?
$Global::newest_job->{'commandline'}->
replace_placeholders(["\257<\257>"],0,0) : "";
- # These chars mess up display in the terminal
+ # These chars mess up display in the terminal in US-ASCII
+ # and in some combinations as UTF8 (e.g. ঐ ও ঔ ক 𐅪 𐅫 𐅬)
$arg =~ tr/[\011-\016\033\302-\365]//d;
my $eta_dhms = ::seconds_to_time_units($eta);
my $bar_text =
@@ -4788,6 +4867,8 @@ sub reaper() {
$job->set_exitstatus($? >> 8);
$job->set_exitsignal($? & 127);
}
+
+ debug("run", "\nseq ",$job->seq()," died (", $job->exitstatus(), ")");
if($Global::delayauto or $Global::sshdelayauto) {
if($job->exitstatus()) {
# Job failed: Increase delay (if $opt::(ssh)delay set)
@@ -4798,9 +4879,8 @@ sub reaper() {
$opt::delay &&= $opt::delay * 0.9;
$opt::sshdelay &&= $opt::sshdelay * 0.9;
}
+ debug("run", "delay:$opt::delay ssh:$opt::sshdelay ");
}
-
- debug("run", "seq ",$job->seq()," died (", $job->exitstatus(), ")");
$job->set_endtime(::now());
my $sshlogin = $job->sshlogin();
$sshlogin->dec_jobs_running();
@@ -4924,6 +5004,10 @@ sub wait_and_exit($) {
# Avoid: Warning: unable to close filehandle properly: No space
# left on device during global destruction.
$SIG{__WARN__} = sub {};
+ if($opt::parset) {
+ # Make the shell script return $error
+ print "$Global::parset_endstring\nreturn $error";
+ }
exit($error);
}
@@ -4968,8 +5052,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- " Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').",
- " Zenodo. https://doi.org/10.5281/zenodo.4628277",
+ " Tange, O. (2021, August 22). GNU Parallel 20210822 ('Kabul').",
+ " Zenodo. https://doi.org/10.5281/zenodo.5233953",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@@ -4999,8 +5083,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- " Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').",
- " Zenodo. https://doi.org/10.5281/zenodo.4628277",
+ " Tange, O. (2021, August 22). GNU Parallel 20210822 ('Kabul').",
+ " Zenodo. https://doi.org/10.5281/zenodo.5233953",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@@ -5076,8 +5160,8 @@ sub error(@) {
sub die_bug($) {
my $bugid = shift;
print STDERR
- ("$Global::progname: This should not happen. You have found a bug.\n",
- "Please contact <parallel\@gnu.org> and follow\n",
+ ("$Global::progname: This should not happen. You have found a bug. ",
+ "Please follow\n",
"https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS\n",
"\n",
"Include this in the report:\n",
@@ -5123,20 +5207,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- "\@software{tange_2021_4628277,",
+ "\@software{tange_2021_5233953,",
" author = {Tange, Ole},",
- " title = {GNU Parallel 20210322 ('2002-01-06')},",
- " month = Mar,",
- " year = 2020,",
+ " title = {GNU Parallel 20210822 ('Kabul')},",
+ " month = Aug,",
+ " year = 2021,",
" note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel",
" without changing them.}},",
" publisher = {Zenodo},",
- " doi = {10.5281/zenodo.4628277},",
- " url = {https://doi.org/10.5281/zenodo.4628277}",
+ " doi = {10.5281/zenodo.5233953},",
+ " url = {https://doi.org/10.5281/zenodo.5233953}",
"}",
"",
- "(Feel free to use \\nocite{tange_2021_4628277})",
+ "(Feel free to use \\nocite{tange_2021_5233953})",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@@ -5295,6 +5379,8 @@ env_parallel --session
env_parallel -k echo ::: Put your code here
parset p,y,c,h -k echo ::: Put your code here
echo $p $y $c $h
+echo You can also activate GNU Parallel for interactive use by:
+echo . "$0"
!;
} else {
::error("Cannot open $0");
@@ -5712,9 +5798,9 @@ sub which(@) {
# ash bash csh dash fdsh fish fizsh ksh ksh93 mksh pdksh
# posh rbash rc rush rzsh sash sh static-sh tcsh yash zsh
- my @shells = (qw(ash bash bsd-csh csh dash fdsh fish fizsh
- ksh ksh93 lksh mksh pdksh posh rbash rc rush rzsh sash sh
- static-sh tcsh yash zsh -sh -csh -bash),
+ my @shells = (qw(ash bash bsd-csh csh dash fdsh fish fizsh ksh
+ ksh93 lksh mksh pdksh posh rbash rc rush rzsh sash sh
+ static-sh tcsh yash zsh -sh -csh -bash),
'-sh (sh)' # sh on FreeBSD
);
# Can be formatted as:
@@ -5762,13 +5848,15 @@ sub which(@) {
if($shellpath = readlink "/proc/$testpid/exe") {
::debug("init","procpath $shellpath\n");
if($shellpath =~ m:/$shell$:o) {
- ::debug("init", "proc which ".$shellpath." => ");
+ ::debug("init",
+ "proc which ".$shellpath." => ");
return $shellpath;
}
}
}
::debug("init", "which ".$shellname." => ");
- $shellpath = (which($shellname,@{$fakename{$shellname}}))[0];
+ $shellpath = (which($shellname,
+ @{$fakename{$shellname}}))[0];
::debug("init", "shell path $shellpath\n");
return $shellpath;
}
@@ -5829,11 +5917,13 @@ sub which(@) {
if(not %pid_parentpid_cmd) {
# Filter for SysV-style `ps`
- my $sysv = q( ps -ef | perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
+ my $sysv = q( ps -ef |).
+ q(perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
q(s/^.{$s}//; print "@F[1,2] $_"' );
# Minix uses cols 2,3 and can have newlines in the command
# so lines not having numbers in cols 2,3 must be ignored
- my $minix = q( ps -ef | perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
+ my $minix = q( ps -ef |).
+ q(perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
q(s/^.{$s}// and $F[2]>0 and $F[3]>0 and print "@F[2,3] $_"' );
# BSD-style `ps`
my $bsd = q(ps -o pid,ppid,command -ax);
@@ -5861,7 +5951,8 @@ sub which(@) {
'syllable' => "echo ps not supported",
);
}
- $pid_parentpid_cmd{$^O} or ::die_bug("pid_parentpid_cmd for $^O missing");
+ $pid_parentpid_cmd{$^O} or
+ ::die_bug("pid_parentpid_cmd for $^O missing");
my (@pidtable,%parent_of,%children_of,%name_of);
# Table with pid -> children of pid
@@ -5913,6 +6004,37 @@ sub usleep($) {
select(undef, undef, undef, $ms/1000);
}
+sub make_regexp_ungreedy {
+ my $regexp = shift;
+
+ my $class_state = 0;
+ my $escape_state = 0;
+ my $found = 0;
+ my $ungreedy = "";
+ my $c;
+
+ for $c (split (//, $regexp)) {
+ if ($found) {
+ if($c ne "?") { $ungreedy .= "?"; }
+ $found = 0;
+ }
+ $ungreedy .= $c;
+
+ if ($escape_state) { $escape_state = 0; next; }
+ if ($c eq "\\") { $escape_state = 1; next; }
+ if ($c eq '[') { $class_state = 1; next; }
+ if ($class_state) {
+ if($c eq ']') { $class_state = 0; }
+ next;
+ }
+ # Quantifiers: + * {...}
+ if ($c =~ /[*}+]/) { $found = 1; }
+ }
+ if($found) { $ungreedy .= '?'; }
+ return $ungreedy;
+}
+
+
sub __KILLER_REAPER__() {}
sub reap_usleep() {
@@ -5968,8 +6090,12 @@ sub reap_usleep() {
$SIG{CHLD} = sub { kill "ALRM", $$ };
if($opt::delay) {
# The 0.004s is approximately the time it takes for one round
- usleep(1000*($Global::newest_starttime +
- $opt::delay - 0.004 - ::now()));
+ my $next_earliest_start =
+ $Global::newest_starttime + $opt::delay - 0.004;
+ my $remaining_ms = 1000 * ($next_earliest_start - ::now());
+ # The next job can only start at $next_earliest_start
+ # so sleep until then (but sleep at least $ms)
+ usleep(::max($ms,$remaining_ms));
} else {
usleep($ms);
}
@@ -5998,7 +6124,8 @@ sub kill_youngest_if_over_limit() {
push @{$jobs_of{$job->sshlogin()}}, $job;
}
for my $sshlogin (@sshlogins) {
- for my $job (sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}}) {
+ for my $job (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}}) {
if($sshlogin->limit() == 2) {
$job->kill();
last;
@@ -6028,7 +6155,8 @@ sub suspend_young_if_not_enough_mem() {
if($free < 2*$limit) {
# Suspend all jobs (resume some of them later)
map { $_->suspended() or $_->suspend(); } @{$jobs_of{$sshlogin}};
- my @jobs = sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}};
+ my @jobs = (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}});
# how many should be running?
# limit*1 => 1;
# limit*1.5 => 2;
@@ -6049,8 +6177,8 @@ sub suspend_young_if_not_enough_mem() {
for my $job (@{$jobs_of{$sshlogin}}) {
if($job->suspended()) {
$job->resume();
- ::debug("mem","\nResume ",$#{$jobs_of{$sshlogin}}+1, " jobs. Seq ",
- $job->seq(), " resumed ",
+ ::debug("mem","\nResume ",$#{$jobs_of{$sshlogin}}+1,
+ " jobs. Seq ", $job->seq(), " resumed ",
$sshlogin->memfree()," > ",2*$limit);
last;
}
@@ -6076,7 +6204,8 @@ sub kill_youngster_if_not_enough_mem() {
push @{$jobs_of{$job->sshlogin()}}, $job;
}
for my $sshlogin (@sshlogins) {
- for my $job (sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}}) {
+ for my $job (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}}) {
if($sshlogin->memfree() < $limit) {
::debug("mem","\n",map { $_->seq()." " }
(sort { $b->seq() <=> $a->seq() }
@@ -6101,17 +6230,17 @@ sub __DEBUGGING__() {}
sub debug(@) {
# Uses:
# $Global::debug
- # %Global::fd
+ # %Global::fh
# Returns: N/A
$Global::debug or return;
@_ = grep { defined $_ ? $_ : "" } @_;
if($Global::debug eq "all" or $Global::debug eq $_[0]) {
- if($Global::fd{1}) {
- # Original stdout was saved
- my $stdout = $Global::fd{1};
- print $stdout @_[1..$#_];
+ if($Global::fh{2}) {
+ # Original stderr was saved
+ my $stderr = $Global::fh{2};
+ print $stderr @_[1..$#_];
} else {
- print @_[1..$#_];
+ print STDERR @_[1..$#_];
}
}
}
@@ -6463,16 +6592,15 @@ sub limit($) {
limit=$1;
io_file=$2;
# Do the measurement in the background
- (tmp=$(tempfile);
+ ((tmp=$(tempfile);
LANG=C iostat -x 1 2 > $tmp;
- mv $tmp $io_file) &
+ mv $tmp $io_file) </dev/null >/dev/null & );
perl -e '-e $ARGV[0] or exit(1);
for(reverse <>) {
/Device/ and last;
/(\S+)$/ and $max = $max > $1 ? $max : $1; }
- exit ($max < '$limit')' $io_file;
+ exit ('$limit' < $max)' $io_file;
};
- export -f io;
io %s %s
!,
"mem" => q!
@@ -6515,6 +6643,7 @@ sub limit($) {
local %ENV = %env;
$ENV{'SSHLOGIN'} = $self->string();
system($Global::shell,"-c",$self->{'limitscript'});
+ #::qqx($self->{'limitscript'});
::debug("limit","limit `".$self->{'limitscript'}."` result ".($?>>8)."\n");
return $?>>8;
}
@@ -8004,15 +8133,15 @@ sub cleanup_cmd($$$) {
$dir .= $_."/";
unshift @rmdir, ::shell_quote_file($dir);
}
- my $rmdir = @rmdir ? "sh -c ".::Q("rmdir @rmdir 2>/dev/null;") : "";
+ my $rmdir = @rmdir ? "rmdir @rmdir 2>/dev/null;" : "";
if(defined $opt::workdir and $opt::workdir eq "...") {
- $rmdir .= ::Q("rm -rf " . ::shell_quote_file($workdir).';');
+ $rmdir .= "rm -rf " . ::shell_quote_file($workdir).';';
}
-
- $f = ::shell_quote_file($f);
+ my $rmf = "sh -c ".
+ ::Q("rm -f ".::shell_quote_file($f)." 2>/dev/null;".$rmdir);
my $sshcmd = $self->sshcommand();
my $serverlogin = $self->serverlogin();
- return "$sshcmd $serverlogin -- ".::Q("rm -f $f; $rmdir");
+ return "$sshcmd $serverlogin -- ".::Q("$rmf");
}
{
@@ -8402,8 +8531,8 @@ sub openoutputfiles($) {
}
} else {
# --ungroup
- open($outfhw,">&",$Global::fd{1}) || die;
- open($errfhw,">&",$Global::fd{2}) || die;
+ open($outfhw,">&",$Global::fh{1}) || die;
+ open($errfhw,">&",$Global::fh{2}) || die;
# File name must be empty as it will otherwise be printed
$outname = "";
$errname = "";
@@ -8542,7 +8671,7 @@ sub filter_through_compress($) {
# thus output file can then be removed by the decompressor.
my $wpid = open(my $fdw,"|-", "(echo > $comfile; ".
empty_input_wrapper($opt::compress_program).") >".
- $self->fh($fdno,'name')) || die $?;
+ ::Q($self->fh($fdno,'name'))) || die $?;
$self->set_fh($fdno,'w',$fdw);
$self->set_fh($fdno,'wpid',$wpid);
# Decompressor: open output; -s $comfile > 0: rm $comfile output;
@@ -8675,11 +8804,11 @@ sub remove_rec_sep($) {
my ($block_ref,$recstart,$recend) = @_;
# Remove record separator
if($opt::regexp) {
- $$block_ref =~ s/$recend$recstart//gos;
+ $$block_ref =~ s/$recend$recstart//gom;
$$block_ref =~ s/^$recstart//os;
$$block_ref =~ s/$recend$//os;
} else {
- $$block_ref =~ s/\Q$recend$recstart\E//gos;
+ $$block_ref =~ s/\Q$recend$recstart\E//gom;
$$block_ref =~ s/^\Q$recstart\E//os;
$$block_ref =~ s/\Q$recend\E$//os;
}
@@ -9247,12 +9376,13 @@ sub sshlogin_wrap($) {
exec $shell, "-c", ($bashfunc."@ARGV");
die "exec: $!\n";
}
+ my $parent = getppid;
do {
- # Parent is not init (ppid=1), so sshd is alive
+ # Parent pid is not changed, so sshd is alive
# Exponential sleep up to 1 sec
$s = $s < 1 ? 0.001 + $s * 1.03 : $s;
select(undef, undef, undef, $s);
- } until ($done || getppid == 1);
+ } until ($done || getppid != $parent);
if(not $done) {
# Kill as per --termseq
my @term_seq = split/,/,$termseq;
@@ -9501,6 +9631,7 @@ sub filter($) {
replace_placeholders(\@opt::filter,0,0)) {
$run &&= eval $eval;
}
+ $self->{'commandline'}{'skip'} ||= not $run;
}
return $run;
}
@@ -10047,8 +10178,10 @@ sub interactive_start($) {
::status("See output with: $ENV{'PARALLEL_TMUX'} -S $tmuxsocket attach");
}
$tmux = "sh -c '".
- $ENV{'PARALLEL_TMUX'}." -S $tmuxsocket new-session -s p$$ -d \"sleep .2\" >/dev/null 2>&1';" .
- $ENV{'PARALLEL_TMUX'}." -S $tmuxsocket new-window -t p$$ -n $title";
+ $ENV{'PARALLEL_TMUX'}.
+ " -S $tmuxsocket new-session -s p$$ -d \"sleep .2\" >/dev/null 2>&1';" .
+ $ENV{'PARALLEL_TMUX'}.
+ " -S $tmuxsocket new-window -t p$$ -n $title";
::debug("tmux", "title len:", $l_tit, " act ", $l_act, " max ",
$Limits::Command::line_max_len, " tot ",
@@ -10060,7 +10193,8 @@ sub interactive_start($) {
(
"(".$actual_command.');'.
# The triple print is needed - otherwise the testsuite fails
- q[ perl -e 'while($t++<3){ print $ARGV[0],"\n" }' $?h/$status >> ].$tmpfifo."&".
+ q[ perl -e 'while($t++<3){ print $ARGV[0],"\n" }' $?h/$status >> ]..
+ $tmpfifo."&".
"echo $title; echo \007Job finished at: `date`;sleep 10"
).
# Run outside tmux
@@ -10193,10 +10327,10 @@ sub print($) {
}
my $returnsize = $self->returnsize();
- for my $fdno (sort { $a <=> $b } keys %Global::fd) {
+ for my $fdno (sort { $a <=> $b } keys %Global::fh) {
# Sort by file descriptor numerically: 1,2,3,..,9,10,11
$fdno == 0 and next;
- my $out_fd = $Global::fd{$fdno};
+ my $out_fh = $Global::fh{$fdno};
my $in_fh = $self->fh($fdno,"r");
if(not $in_fh) {
if(not $Job::file_descriptor_warning_printed{$fdno}++) {
@@ -10207,15 +10341,15 @@ sub print($) {
::debug("print", "File descriptor $fdno (", $self->fh($fdno,"name"), "):\n");
if($opt::linebuffer) {
# Line buffered print out
- $self->print_linebuffer($fdno,$in_fh,$out_fd);
+ $self->print_linebuffer($fdno,$in_fh,$out_fh);
} elsif($opt::files) {
- $self->print_files($fdno,$in_fh,$out_fd);
- } elsif($opt::tag or defined $opt::tagstring) {
- $self->print_tag($fdno,$in_fh,$out_fd);
+ $self->print_files($fdno,$in_fh,$out_fh);
+ } elsif($opt::results) {
+ $self->print_results($fdno,$in_fh,$out_fh);
} else {
- $self->print_normal($fdno,$in_fh,$out_fd);
+ $self->print_normal($fdno,$in_fh,$out_fh);
}
- flush $out_fd;
+ flush $out_fh;
}
::debug("print", "<<joboutput\n");
if(defined $self->{'exitstatus'}
@@ -10246,7 +10380,8 @@ sub print($) {
sub jsonquote($) {
my $a = shift;
if(not $jsonmap{"\001"}) {
- map { $jsonmap{sprintf("%c",$_)} = sprintf '\u%04x', $_ } 0..31;
+ map { $jsonmap{sprintf("%c",$_)} =
+ sprintf '\u%04x', $_ } 0..31;
}
$a =~ s/\\/\\\\/g;
$a =~ s/\"/\\"/g;
@@ -10411,7 +10546,7 @@ sub print_files($) {
# $opt::linebuffer = Print ASAP
# Returns: N/A
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
# If the job is dead: close printing fh. Needed for --compress
close $self->fh($fdno,"w");
@@ -10433,7 +10568,7 @@ sub print_files($) {
::rm($self->fh($fdno,"unlink"));
}
} elsif($fdno == 1 and $self->fh($fdno,"name")) {
- print $out_fd $self->tag(),$self->fh($fdno,"name"),"\n";
+ print $out_fh $self->tag(),$self->fh($fdno,"name"),"\n";
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}},
$self->tag(), $self->fh($fdno,"name");
@@ -10446,7 +10581,7 @@ sub print_files($) {
sub print_linebuffer($) {
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
if(defined $self->{'exitstatus'}) {
# If the job is dead: close printing fh. Needed for --compress
close $self->fh($fdno,"w");
@@ -10465,7 +10600,7 @@ sub print_linebuffer($) {
if($opt::files or ($opt::results and not $Global::csvsep)) {
# Print filename
if($fdno == 1 and not $self->fh($fdno,"printed")) {
- print $out_fd $self->tag(),$self->fh($fdno,"name"),"\n";
+ print $out_fh $self->tag(),$self->fh($fdno,"name"),"\n";
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}}, $self->tag(),
$self->fh($fdno,"name"));
@@ -10494,19 +10629,23 @@ sub print_linebuffer($) {
my $tag = $self->tag();
unshift @$halfline_ref, $tag;
# TODO --recend that can be partially in @$halfline_ref
- substr($buf,0,$i-1) =~ s/(?<=[\n\r])(?=.|$)/$tag/gs;
+ substr($buf,0,$i-1) =~
+ s/(?<=[\n\r])(?=.|$)/$tag/gs;
# The length changed, so find the new ending pos
- $i = ::max((rindex($buf,"\n")+1), (rindex($buf,"\r")+1));
+ $i = ::max((rindex($buf,"\n")+1),
+ (rindex($buf,"\r")+1));
} else {
# Replace with freshly computed value of tag
unshift @$halfline_ref, $self->tag();
- substr($buf,0,$i-1) =~ s/(?<=[\n\r])(?=.|$)/$self->tag()/gse;
+ substr($buf,0,$i-1) =~
+ s/(?<=[\n\r])(?=.|$)/$self->tag()/gse;
# The length changed, so find the new ending pos
- $i = ::max((rindex($buf,"\n")+1), (rindex($buf,"\r")+1));
+ $i = ::max((rindex($buf,"\n")+1),
+ (rindex($buf,"\r")+1));
}
}
# Print the partial line (halfline) and the last half
- print $out_fd @$halfline_ref, substr($buf,0,$i);
+ print $out_fh @$halfline_ref, substr($buf,0,$i);
# Buffer in memory for SQL and CSV-output
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}},
@@ -10539,7 +10678,7 @@ sub print_linebuffer($) {
unshift @$halfline_ref, $self->tag();
}
# Print the partial line (halfline)
- print $out_fd @{$self->{'halfline'}{$fdno}};
+ print $out_fh @{$self->{'halfline'}{$fdno}};
# Buffer in memory for SQL and CSV-output
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}}, @$halfline_ref);
@@ -10562,15 +10701,11 @@ sub print_linebuffer($) {
}
}
-sub print_tag(@) {
- return print_normal(@_);
-}
-
sub free_ressources() {
my $self = shift;
if(not $opt::ungroup) {
my $fh;
- for my $fdno (sort { $a <=> $b } keys %Global::fd) {
+ for my $fdno (sort { $a <=> $b } keys %Global::fh) {
$fh = $self->fh($fdno,"w");
$fh and close $fh;
$fh = $self->fh($fdno,"r");
@@ -10579,9 +10714,54 @@ sub free_ressources() {
}
}
+sub print_parset($) {
+ # Wrap output with shell script code to set as variables
+ my $self = shift;
+ my ($fdno,$in_fh,$out_fh) = @_;
+ my $outputlength = 0;
+
+ ::debug("parset","print $Global::parset");
+ if($Global::parset eq "assoc") {
+ # eval "`echo 'declare -A myassoc; myassoc=(
+ # Each:
+ # [$'a\tb']=$'a\tb\tc ddd'
+ # End:
+ # )'`"
+ print '[',::Q($self->{'commandline'}->
+ replace_placeholders(["\257<\257>"],0,0)),']=';
+ } elsif($Global::parset eq "array") {
+ # eval "`echo 'myassoc=(
+ # Each:
+ # $'a\tb\tc ddd'
+ # End:
+ # )'`"
+ } elsif($Global::parset eq "var") {
+ # var=$'a\tb\tc ddd'
+ if(not @Global::parset_vars) {
+ ::error("Too few named destination variables");
+ ::wait_and_exit(255);
+ }
+ print shift @Global::parset_vars,"=";
+ }
+ local $/ = "\n";
+ my $tag = $self->tag();
+ my @out;
+ while(<$in_fh>) {
+ $outputlength += length $_;
+ # Tag lines with \r, too
+ $_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
+ push @out, $tag,$_;
+ }
+ # Remove last newline
+ # This often makes it easier to use the output in shell
+ @out and ${out[$#out]} =~ s/\n$//s;
+ print ::Q(join("", at out)),"\n";
+ return $outputlength;
+}
+
sub print_normal($) {
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
my $buf;
close $self->fh($fdno,"w");
if($? and $opt::compress) {
@@ -10594,7 +10774,9 @@ sub print_normal($) {
my $outputlength = 0;
my @output;
- if($opt::tag or $opt::tagstring) {
+ if($Global::parset and $fdno == 1) {
+ $outputlength += $self->print_parset($fdno,$in_fh,$out_fh);
+ } elsif($opt::tag or $opt::tagstring) {
# Read line by line
local $/ = "\n";
my $tag = $self->tag();
@@ -10602,14 +10784,15 @@ sub print_normal($) {
$outputlength += length $_;
# Tag lines with \r, too
$_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
- print $out_fd $tag,$_;
+ print $out_fh $tag,$_;
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}}, $tag, $_;
}
}
} else {
+ # Most efficient way of copying data from $in_fh to $out_fh
while(sysread($in_fh,$buf,131072)) {
- print $out_fd $buf;
+ print $out_fh $buf;
$outputlength += length $buf;
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}}, $buf;
@@ -10627,6 +10810,61 @@ sub print_normal($) {
}
}
+sub print_results($) {
+ my $self = shift;
+ my ($fdno,$in_fh,$out_fh) = @_;
+ my $buf;
+ close $self->fh($fdno,"w");
+ if($? and $opt::compress) {
+ ::error($opt::compress_program." failed.");
+ $self->set_exitstatus(255);
+ }
+ if(not $self->virgin()) {
+ seek $in_fh, 0, 0;
+ # $in_fh is now ready for reading at position 0
+ my $outputlength = 0;
+ my @output;
+
+ if($Global::membuffer) {
+ # Read data into membuffer
+ if($opt::tag or $opt::tagstring) {
+ # Read line by line
+ local $/ = "\n";
+ my $tag = $self->tag();
+ while(<$in_fh>) {
+ $outputlength += length $_;
+ # Tag lines with \r, too
+ $_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
+ push @{$self->{'output'}{$fdno}}, $tag, $_;
+ }
+ } else {
+ # Most efficient way of copying data from $in_fh to $out_fh
+ while(sysread($in_fh,$buf,131072)) {
+ $outputlength += length $buf;
+ push @{$self->{'output'}{$fdno}}, $buf;
+ }
+ }
+ } else {
+ # Not membuffer: No need to read the file
+ if($opt::compress) {
+ $outputlength = -1;
+ } else {
+ # Determine $outputlength = file length
+ seek($in_fh, 0, 2) || ::die_bug("cannot seek result");
+ $outputlength = tell($in_fh);
+ }
+ }
+ if($fdno == 1) {
+ $self->add_returnsize($outputlength);
+ }
+ close $in_fh;
+ if($? and $opt::compress) {
+ ::error($opt::decompress_program." failed.");
+ $self->set_exitstatus(255);
+ }
+ }
+}
+
sub print_joblog($) {
my $self = shift;
my $cmd;
@@ -10648,17 +10886,51 @@ sub print_joblog($) {
$self->set_job_in_joblog();
}
-sub tag($) {
- my $self = shift;
- if(not defined $self->{'tag'} or not $Global::cache_replacement_eval) {
- if($opt::tag or defined $opt::tagstring) {
- $self->{'tag'} = $self->{'commandline'}->
- replace_placeholders([$opt::tagstring],0,0)."\t";
- } else {
- $self->{'tag'} = "";
+{
+ my @color;
+
+ sub tag($) {
+ sub init_color() {
+ # color combinations that are readable: black/white text
+ # on colored background, but not white on yellow
+ @color =
+ # Force each color code to have the same length in chars
+ # This will make \t work as expected
+ ((map { [sprintf("%03d",$_),"000"] }
+ 6..7,9..11,13..15,40..51,75..87,113..123,147..159,
+ 171..231,249..254),
+ (map { [sprintf("%03d",$_),231] }
+ 1..9,12..13,16..45,52..81,88..116,124..151,153,
+ 160..180,182..185,187..189,196..214,232..252,
+ 255..254));
+ # reorder list so adjacent colors are dissimilar
+ # %7 and %17 were found experimentally
+ @color = @color[
+ sort { ($b%7 <=> $a%7) or ($a%17 <=> $b%17) } 0..$#color
+ ];
+ }
+ my $self = shift;
+ if(not defined $self->{'tag'} or not $Global::cache_replacement_eval) {
+ if($opt::tag or defined $opt::tagstring) {
+ if($Global::color) {
+ if(not @color) { init_color() }
+ # Choose a value based on the seq
+ my $col = @color[$self->seq() % ($#color+1)];
+ $self->{'tag'} = "\033[48;5;".$col->[0].
+ ";38;5;".$col->[1]."m".
+ ($self->{'commandline'}->
+ replace_placeholders([$opt::tagstring],0,0)).
+ "\033[00m\t";
+ } else {
+ $self->{'tag'} = $self->{'commandline'}->
+ replace_placeholders([$opt::tagstring],0,0)."\t";
+ }
+ } else {
+ $self->{'tag'} = "";
+ }
}
+ return $self->{'tag'};
}
- return $self->{'tag'};
}
sub hostgroups($) {
@@ -12658,21 +12930,27 @@ sub total_jobs() {
$Global::unquote_arg = 1;
}
sub yyyy_mm_dd_hh_mm_ss() {
+ # ISO8601 2038-01-19T03:14:08
::strftime("%Y-%m-%dT%H:%M:%S", localtime(time()));
}
sub yyyy_mm_dd_hh_mm() {
+ # ISO8601 2038-01-19T03:14
::strftime("%Y-%m-%dT%H:%M", localtime(time()));
}
sub yyyy_mm_dd() {
+ # ISO8601 2038-01-19
::strftime("%Y-%m-%d", localtime(time()));
}
sub yyyymmddhhmmss() {
+ # ISO8601 20380119031408
::strftime("%Y%m%d%H%M%S", localtime(time()));
}
sub yyyymmddhhmm() {
+ # ISO8601 203801190314
::strftime("%Y%m%d%H%M", localtime(time()));
}
sub yyyymmdd() {
+ # ISO8601 20380119
::strftime("%Y%m%d", localtime(time()));
}
@@ -13596,7 +13874,7 @@ sub main() {
save_stdin_stdout_stderr();
save_original_signal_handler();
parse_options();
- ::debug("init", "Open file descriptors: ", join(" ",keys %Global::fd), "\n");
+ ::debug("init", "Open file descriptors: ", join(" ",keys %Global::fh), "\n");
my $number_of_args;
if($Global::max_number_of_args) {
$number_of_args = $Global::max_number_of_args;
@@ -13742,3 +14020,4 @@ sub main() {
}
main();
+
=====================================
src/parallel.pod
=====================================
@@ -67,12 +67,12 @@ how-to, reference and explanation.
=head3 Tutorial
If you prefer reading a book buy B<GNU Parallel 2018> at
-http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
+https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
or download it at: https://doi.org/10.5281/zenodo.1146014 Read at
least chapter 1+2. It should take you less than 20 minutes.
Otherwise start by watching the intro videos for a quick introduction:
-http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
+https://youtube.com/playlist?list=PL284C9FF2488BC6D1
If you want to dive deeper: spend a couple of hours walking through
the tutorial (B<man parallel_tutorial>). Your command line will love
@@ -303,7 +303,7 @@ directory (if any) and extension removed.
To understand positional replacement strings see B<{>I<n>B<}>.
-=item B<{=>I<perl expression>B<=}> (beta testing)
+=item B<{=>I<perl expression>B<=}>
Replace with calculated I<perl expression>. B<$_> will contain the
same as B<{}>. After evaluating I<perl expression> B<$_> will be used
@@ -352,6 +352,10 @@ sequence number of job
the arguments
+=item Z<> B<skip()>
+
+skip this job (see also B<--filter>)
+
=item Z<> B<yyyy_mm_dd_hh_mm_ss()>
=item Z<> B<yyyy_mm_dd_hh_mm()>
@@ -604,10 +608,7 @@ alternatives in B<man parallel_alternatives>.
=item B<--block-size> I<size>
Size of block in bytes to read at a time. The I<size> can be postfixed
-with K, M, G, T, P, E, k, m, g, t, p, or e which would multiply the
-size with 1024, 1048576, 1073741824, 1099511627776, 1125899906842624,
-1152921504606846976, 1000, 1000000, 1000000000, 1000000000000,
-1000000000000000, or 1000000000000000000 respectively.
+with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
GNU B<parallel> tries to meet the block size but can be off by the
length of one record. For performance reasons I<size> should be bigger
@@ -668,12 +669,12 @@ files on the remote computer after processing is done.
--return {.}.bz2 --cleanup "zcat {} | bzip -9 >{.}.bz2"
With B<--transferfile {}> the file transferred to the remote computer
-will be removed on the remote computer. Directories created will not
-be removed - even if they are empty.
+will be removed on the remote computer. Directories on the remote
+computer containing the file will be removed if they are empty.
With B<--return> the file transferred from the remote computer will be
-removed on the remote computer. Directories created will not be
-removed - even if they are empty.
+removed on the remote computer. Directories on the remote
+computer containing the file will be removed if they are empty.
B<--cleanup> is ignored when not used with B<--transferfile> or
B<--return>.
@@ -697,7 +698,7 @@ B<--colsep> implies B<--trim rl>, which can be overridden with
B<--trim n>.
I<regexp> is a Perl Regular Expression:
-http://perldoc.perl.org/perlre.html
+https://perldoc.perl.org/perlre.html
=item B<--compress>
@@ -739,6 +740,16 @@ Even quoted newlines are parsed correctly:
When used with B<--pipe> only pass full CSV-records.
+=item B<--ctag> I<str> (alpha testing)
+
+Color tag. See B<--tag>.
+
+
+=item B<--ctagstring> I<str> (alpha testing)
+
+Color tagstring. See B<--tagstring>.
+
+
=item B<--delay> I<mytime>
Delay starting next job by I<mytime>. GNU B<parallel> will pause
@@ -873,7 +884,7 @@ Implies B<--pipe> unless B<--pipepart> is used.
See also: B<--cat>.
-=item B<--filter> I<filter> (beta testing)
+=item B<--filter> I<filter>
Only run jobs where I<filter> is true. I<filter> can contain
replacement strings and Perl code. Example:
@@ -1279,6 +1290,9 @@ continued on the next input line.
B<-L 0> means read one line, but insert 0 arguments on the command
line.
+I<recsize> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
+UNIT PREFIX).
+
Implies B<-X> unless B<-m>, B<--xargs>, or B<--pipe> is set.
@@ -1373,9 +1387,9 @@ mix. Compare:
See also: B<--group> B<--ungroup>
-=item B<--xapply> (beta testing)
+=item B<--xapply>
-=item B<--link> (beta testing)
+=item B<--link>
Link input sources. Read multiple input sources like B<xapply>. If
multiple input sources are given, one argument will be read from each
@@ -1433,10 +1447,7 @@ most likely do what is needed.
=item B<--memfree> I<size>
Minimum memory free when starting another job. The I<size> can be
-postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply
-the size with 1024, 1048576, 1073741824, 1099511627776,
-1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
-1000000000000000, respectively.
+postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
If the jobs take up very different amount of RAM, GNU B<parallel> will
only start as many as there is memory for. If less than I<size> bytes
@@ -1453,10 +1464,8 @@ See also: B<--memsuspend>
=item B<--memsuspend> I<size>
Suspend jobs when there is less than 2 * I<size> memory free. The
-I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which
-would multiply the size with 1024, 1048576, 1073741824, 1099511627776,
-1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or
-1000000000000000, respectively.
+I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
+UNIT PREFIX).
If the available memory falls below 2 * I<size>, GNU B<parallel>
will suspend some of the running jobs. If the available memory falls
@@ -1498,6 +1507,9 @@ GNU B<parallel> will exit.
B<-n 0> means read one argument, but insert 0 arguments on the command
line.
+I<max-args> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
+UNIT PREFIX).
+
Implies B<-X> unless B<-m> is set.
@@ -1519,6 +1531,9 @@ This will set the owner of the homedir to the user:
Implies B<-X> unless B<-m> or B<--pipe> is set.
+I<max-args> can be postfixed with K, M, G, T, P, k, m, g, t, or p (see
+UNIT PREFIX).
+
When used with B<--pipe> B<-N> is the number of records to read. This
is somewhat slower than B<--block>.
@@ -1606,14 +1621,16 @@ B<--pipepart> has a few limitations:
The file must be a normal file or a block device (technically it must
be seekable) and must be given using B<-a> or B<::::>. The file cannot
-be a pipe or a fifo as they are not seekable.
+be a pipe, a fifo, or a stream as they are not seekable.
If using a block device with lot of NUL bytes, remember to set
B<--recend ''>.
=item *
-Record counting (B<-N>) and line counting (B<-L>/B<-l>) do not work.
+Record counting (B<-N>) and line counting (B<-L>/B<-l>) do not
+work. Instead use B<--recstart> and B<--recend> to determine
+where records end.
=back
@@ -1635,9 +1652,9 @@ Activate additional replacement strings: {+/} {+.} {+..} {+...} {..}
B<{##}> is the total number of jobs to be run. It is incompatible with
B<-X>/B<-m>/B<--xargs>.
-B<{0%}> zero-padded jobslot. (alpha testing)
+B<{0%}> zero-padded jobslot.
-B<{0#}> zero-padded sequence number. (alpha testing)
+B<{0#}> zero-padded sequence number.
B<{choose_k}> is inspired by n choose k: Given a list of n elements,
choose k. k is the number of input sources and n is the number of
@@ -1658,9 +1675,11 @@ inspired by bash's parameter expansion:
{:-str} str if the value is empty
{:num} remove the first num characters
{:num1:num2} characters from num1 to num2
- {#str} remove prefix str
- {%str} remove postfix str
- {/str1/str2} replace str1 with str2
+ {#regexp} remove prefix regexp (non-greedy)
+ {##regexp} remove prefix regexp (greedy)
+ {%regexp} remove postfix regexp (non-greedy)
+ {%%regexp} remove postfix regexp (greedy)
+ {/regexp/str} replace regexp with str
{^str} uppercase str if found at the start
{^^str} uppercase str
{,str} lowercase str if found at the start
@@ -1811,7 +1830,8 @@ Swap activity is computed as (swap-in)*(swap-out) which in practice is
a good value: swapping out is not a problem, swapping in is not a
problem, but both swapping in and out usually indicates a problem.
-B<--memfree> may give better results, so try using that first.
+B<--memfree> and B<--memsuspend> may give better results, so try using
+those first.
=item B<--record-env>
@@ -1862,9 +1882,9 @@ it to the command.
Only used with B<--pipe>.
-=item B<--results> I<name>
+=item B<--results> I<name> (beta testing)
-=item B<--res> I<name>
+=item B<--res> I<name> (beta testing)
Save the output into files.
@@ -2132,6 +2152,8 @@ B<--return> is often used with B<--transferfile> and B<--cleanup>.
B<--return> is ignored when used with B<--sshlogin :> or when not used
with B<--sshlogin>.
+For details on transferring see B<--transferfile>.
+
=item B<--round-robin>
@@ -2245,6 +2267,9 @@ the argument strings. The largest allowed value is system-dependent,
and is calculated as the argument length limit for exec, less the size
of your environment. The default value is the maximum.
+I<max-chars> can be postfixed with K, M, G, T, P, k, m, g, t, or p
+(see UNIT PREFIX).
+
Implies B<-X> unless B<-m> is set.
@@ -2372,7 +2397,7 @@ On FreeBSD B<env> is needed:
https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
There are many limitations of shebang (#!) depending on your operating
-system. See details on http://www.in-ulm.de/~mascheck/various/shebang/
+system. See details on https://www.in-ulm.de/~mascheck/various/shebang/
=item B<--shebang-wrap>
@@ -2638,9 +2663,9 @@ Silent. The job to be run will not be printed. This is the default.
Can be reversed with B<-v>.
-=item B<--template> I<file>=I<repl> (beta testing)
+=item B<--template> I<file>=I<repl>
-=item B<--tmpl> I<file>=I<repl> (beta testing)
+=item B<--tmpl> I<file>=I<repl>
Copy I<file> to I<repl>. All replacement strings in the contents of
I<file> will be replaced. All replacement strings in the name I<repl>
@@ -2682,7 +2707,7 @@ to GNU B<parallel> giving each child its own process group, which is
then killed. Process groups are dependant on the tty.
-=item B<--tag>
+=item B<--tag> (alpha testing)
Tag lines with arguments. Each output line will be prepended with the
arguments and TAB (\t). When combined with B<--onall> or B<--nonall>
@@ -2690,8 +2715,10 @@ the lines will be prepended with the sshlogin instead.
B<--tag> is ignored when using B<-u>.
+B<--ctag> gives the tag a color.
+
-=item B<--tagstring> I<str>
+=item B<--tagstring> I<str> (alpha testing)
Tag lines with a string. Each output line will be prepended with
I<str> and TAB (\t). I<str> can contain replacement strings such as
@@ -2699,6 +2726,8 @@ B<{}>.
B<--tagstring> is ignored when using B<-u>, B<--onall>, and B<--nonall>.
+B<--ctagstring> gives the tag a color.
+
=item B<--tee>
@@ -2789,29 +2818,41 @@ Transfer files to remote computers. Shorthand for: B<--transferfile {}>.
B<--transferfile> is used with B<--sshlogin> to transfer files to the
remote computers. The files will be transferred using B<rsync> and
-will be put relative to the default work dir. If the path contains /./
-the remaining path will be relative to the work dir. E.g.
+will be put relative to the work dir (see B<--workdir>).
- echo foo/bar.txt | parallel --transferfile {} \
- --sshlogin server.example.com wc
+The I<filename> will normally contain a replacement string.
+
+If the path contains /./ the remaining path will be relative to the
+work dir (for details: see B<rsync>). If the work dir is
+B</home/user>, the transferring will be as follows:
+
+ /tmp/foo/bar => /tmp/foo/bar
+ tmp/foo/bar => /home/user/tmp/foo/bar
+ /tmp/./foo/bar => /home/user/foo/bar
+ tmp/./foo/bar => /home/user/foo/bar
+
+I<Examples>
This will transfer the file I<foo/bar.txt> to the computer
I<server.example.com> to the file I<$HOME/foo/bar.txt> before running
-B<wc foo/bar.txt> on I<server.example.com>.
+B<wc foo/bar.txt> on I<server.example.com>:
- echo /tmp/foo/bar.txt | parallel --transferfile {} \
+ echo foo/bar.txt | parallel --transferfile {} \
--sshlogin server.example.com wc
This will transfer the file I</tmp/foo/bar.txt> to the computer
I<server.example.com> to the file I</tmp/foo/bar.txt> before running
-B<wc /tmp/foo/bar.txt> on I<server.example.com>.
+B<wc /tmp/foo/bar.txt> on I<server.example.com>:
- echo /tmp/./foo/bar.txt | parallel --transferfile {} \
- --sshlogin server.example.com wc {= s:.*/./:./: =}
+ echo /tmp/foo/bar.txt | parallel --transferfile {} \
+ --sshlogin server.example.com wc
This will transfer the file I</tmp/foo/bar.txt> to the computer
I<server.example.com> to the file I<foo/bar.txt> before running
-B<wc ./foo/bar.txt> on I<server.example.com>.
+B<wc ./foo/bar.txt> on I<server.example.com>:
+
+ echo /tmp/./foo/bar.txt | parallel --transferfile {} \
+ --sshlogin server.example.com wc {= s:.*/\./:./: =}
B<--transferfile> is often used with B<--return> and B<--cleanup>. A
shorthand for B<--transferfile {}> is B<--transfer>.
@@ -2950,7 +2991,8 @@ Print the version GNU B<parallel> and exit.
=item B<--wd> I<mydir>
-Jobs will be run in the dir I<mydir>.
+Jobs will be run in the dir I<mydir>. The default is the current dir
+for the local machine, and the login dir for remote computers.
Files transferred using B<--transferfile> and B<--return> will be
relative to I<mydir> on remote computers.
@@ -3335,7 +3377,7 @@ To remove a string anywhere you can use regular expressions with
Let us assume a website stores images like:
- http://www.example.com/path/to/YYYYMMDD_##.jpg
+ https://www.example.com/path/to/YYYYMMDD_##.jpg
where YYYYMMDD is the date and ## is the number 01-24. This will
download images for the past 30 days:
@@ -3343,7 +3385,7 @@ download images for the past 30 days:
getit() {
date=$(date -d "today -$1 days" +%Y%m%d)
num=$2
- echo wget http://www.example.com/path/to/${date}_${num}.jpg
+ echo wget https://www.example.com/path/to/${date}_${num}.jpg
}
export -f getit
@@ -3871,13 +3913,13 @@ parallel.
To download byte 10000000-19999999 you can use B<curl>:
- curl -r 10000000-19999999 http://example.com/the/big/file >file.part
+ curl -r 10000000-19999999 https://example.com/the/big/file >file.part
To download a 1 GB file we need 100 10MB chunks downloaded and
combined in the correct order.
seq 0 99 | parallel -k curl -r \
- {}0000000-{}9999999 http://example.com/the/big/file > file
+ {}0000000-{}9999999 https://example.com/the/big/file > file
=head2 EXAMPLE: Parallel grep
@@ -4146,6 +4188,15 @@ the special short hand I<-S ..> can be used:
--trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
+=head2 EXAMPLE: Advanced file transfer
+
+Assume you have files in in/*, want them processed on server,
+and transferred back into /other/dir:
+
+ parallel -S server --trc /other/dir/./{/}.out \
+ cp {/} {/}.out ::: in/./*
+
+
=head2 EXAMPLE: Distributing work to local and remote computers
Convert *.mp3 to *.ogg running one process per CPU on local computer
@@ -4256,6 +4307,38 @@ If not all hosts are accessible through TOR:
See more B<ssh> tricks on https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts
+=head2 EXAMPLE: Use outrun instead of ssh
+
+B<outrun> lets you run a command on a remote server. B<outrun> sets up
+a connection to access files at the source server, and automatically
+transfers files. B<outrun> must be installed on the remote system.
+
+You can use B<outrun> in an sshlogin this way:
+
+ parallel -S 'outrun user at server eval' command
+
+
+=head2 EXAMPLE: Slurm cluster
+
+The Slurm Workload Manager is used in many clusters.
+
+Here is a simple example of using GNU B<parallel> to call B<srun>:
+
+ #!/bin/bash
+
+ #SBATCH --time 00:02:00
+ #SBATCH --ntasks=4
+ #SBATCH --job-name GnuParallelDemo
+ #SBATCH --output gnuparallel.out
+
+ module purge
+ module load gnu_parallel
+
+ my_parallel="parallel --delay .2 -j $SLURM_NTASKS"
+ my_srun="srun --export=all --exclusive -n1 --cpus-per-task=1 --cpu-bind=cores"
+ $my_parallel "$my_srun" echo This is job {} ::: {1..20}
+
+
=head2 EXAMPLE: Parallelizing rsync
B<rsync> is a great tool, but sometimes it will not fill up the
@@ -4451,6 +4534,31 @@ To call B<myprog> with the sequence as argument run:
'read a; echo Name: "$a"; myprog $(tr -d "\n")'
+=head2 EXAMPLE: Call program with interleaved FASTQ records
+
+FASTQ files have the format:
+
+ @M10991:61:000000000-A7EML:1:1101:14011:1001 1:N:0:28
+ CTCCTAGGTCGGCATGATGGGGGAAGGAGAGCATGGGAAGAAATGAGAGAGTAGCAAGG
+ +
+ #8BCCGGGGGFEFECFGGGGGGGGG@;FFGGGEG at FF<EE<@FFC,CEGCCGGFF<FGF
+
+Interleaved FASTQ starts with a line like these:
+
+ @HWUSI-EAS100R:6:73:941:1973#0/1
+ @EAS139:136:FC706VJ:2:2104:15343:197393 1:Y:18:ATCACG
+ @EAS139:136:FC706VJ:2:2104:15343:197393 1:N:18:1
+
+where '/1' and ' 1:' determines this is read 1.
+
+This will cut big.fq into one chunk per CPU core and pass it on
+stdin (standard input) to the program fastq-reader:
+
+ parallel --pipepart -a big.fq --block -1 --regexp \
+ --recend '\n' --recstart '@.*(/1| 1:.*)\n[A-Za-z\n\.~]' \
+ fastq-reader
+
+
=head2 EXAMPLE: Processing a big file using more CPUs
To process a big file or some output you can use B<--pipe> to split up
@@ -4795,6 +4903,18 @@ chunk border.
B<--group-by> can be combined with B<--round-robin> or B<--pipe-part>.
+
+=head1 UNIT PREFIX
+
+Many numerical arguments in GNU B<parallel> can be postfixed with K,
+M, G, T, P, k, m, g, t, or p which would multiply the number with
+1024, 1048576, 1073741824, 1099511627776, 1125899906842624, 1000,
+1000000, 1000000000, 1000000000000, or 1000000000000000, respectively.
+
+You can even give it as a math expression. E.g. 1000000 can be written
+as 1M-12*2.024*2k.
+
+
=head1 QUOTING
GNU B<parallel> is very liberal in quoting. You only need to quote
@@ -5347,7 +5467,7 @@ this.
=item *
The complete output of B<parallel --version>. If you are not running
-the latest released version (see http://ftp.gnu.org/gnu/parallel/) you
+the latest released version (see https://ftp.gnu.org/gnu/parallel/) you
should specify why you believe the problem is not fixed in that
version.
@@ -5384,17 +5504,19 @@ by others, the output might help them figure out the problem.
=item *
Whether you have watched the intro videos
-(http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), walked
+(https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1), walked
through the tutorial (man parallel_tutorial), and read the EXAMPLE
section in the man page (man parallel - search for EXAMPLE:).
=back
+=head2 Bug dependent on environment
+
If you suspect the error is dependent on your environment or
distribution, please see if you can reproduce the error on one of
these VirtualBox images:
-http://sourceforge.net/projects/virtualboximage/files/
-http://www.osboxes.org/virtualbox-images/
+https://sourceforge.net/projects/virtualboximage/files/
+https://www.osboxes.org/virtualbox-images/
Specifying the name of your distribution is not enough as you may have
installed software that is not in the VirtualBox images.
@@ -5467,7 +5589,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
+along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I
=====================================
src/parallel_alternatives.pod
=====================================
@@ -575,7 +575,7 @@ using GNU B<parallel>:
echo unzip 1.zip >> /var/run/my_named_pipe;
echo tar cf /backup/myhome.tar /home/me >> /var/run/my_named_pipe
-http://www.maier-komor.de/xjobs.html (Last checked: 2019-01)
+https://www.maier-komor.de/xjobs.html (Last checked: 2019-01)
=head2 DIFFERENCES BETWEEN prll AND GNU Parallel
@@ -692,8 +692,7 @@ using GNU B<parallel>:
11$ parallel '[ -f {} ] && echo {}' < List | ...
-https://web.archive.org/web/20160702211113/
-http://carrera.databits.net/~ksb/msrc/local/bin/xapply/xapply.html
+https://www.databits.net/~ksb/msrc/local/bin/xapply/xapply.html
=head2 DIFFERENCES BETWEEN AIX apply AND GNU Parallel
@@ -930,19 +929,19 @@ RELPATH
These can be simulated using GNU B<parallel> by putting this in
B<~/.parallel/config>:
- --rpl 'FULLPATH $_=Q($_);chomp($_=qx{readlink -f $_});'
- --rpl 'DIRNAME $_=Q(::dirname($_));chomp($_=qx{readlink -f $_});'
- --rpl 'BASENAME s:.*/::;s:\.[^/.]+$::;'
- --rpl 'EXT s:.*\.::'
- --rpl 'RELDIR $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
- s:\Q$c/\E::;$_=::dirname($_);'
- --rpl 'RELPATH $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
- s:\Q$c/\E::;'
+ --rpl 'FULLPATH $_=Q($_);chomp($_=qx{readlink -f $_});'
+ --rpl 'DIRNAME $_=Q(::dirname($_));chomp($_=qx{readlink -f $_});'
+ --rpl 'BASENAME s:.*/::;s:\.[^/.]+$::;'
+ --rpl 'EXT s:.*\.::'
+ --rpl 'RELDIR $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
+ s:\Q$c/\E::;$_=::dirname($_);'
+ --rpl 'RELPATH $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
+ s:\Q$c/\E::;'
B<ladon> deals badly with filenames containing " and newline, and it
fails for output larger than 200k:
- ladon '*' -- seq 36000 | wc
+ ladon '*' -- seq 36000 | wc
=head3 EXAMPLES FROM ladon MANUAL
@@ -1096,9 +1095,9 @@ https://github.com/reconquest/orgalorg
=head2 DIFFERENCES BETWEEN Rust parallel AND GNU Parallel
-Rust parallel focuses on speed. It is almost as fast as B<xargs>. It
-implements a few features from GNU B<parallel>, but lacks many
-functions. All these fail:
+Rust parallel focuses on speed. It is almost as fast as B<xargs>, but
+not as fast as B<parallel-bash>. It implements a few features from GNU
+B<parallel>, but lacks many functions. All these fail:
# Read arguments from file
parallel -a file echo
@@ -1321,7 +1320,7 @@ Contrary to B<rush> this also works if the value is complex like:
My brother's 12" records
-B<14. B<Preset variable> (`-v`), avoid repeatedly writing verbose replacement strings>
+B<14. Preset variable (`-v`), avoid repeatedly writing verbose replacement strings>
14$ # naive way
echo read_1.fq.gz | rush 'echo {:^_1} {:^_1}_2.fq.gz'
@@ -3162,9 +3161,21 @@ of ~0.05 ms/job compared to GNU B<parallel>'s ~3 ms/job). So if your
jobs are extremely short lived, and you can live with the quite
limited command, this may be useful.
-B<parallel-bash> will not start the first job, until it has read all
-input. The input can at most be 20935 lines and the lines cannot be
-all be empty.
+It works by making a queue for each process. Then the jobs are
+distributed to the queues in a round robin fashion. Finally the queues
+are started in parallel. This works fine, if you are lucky, but if
+not, all the long jobs may end up in the same queue, so you may see:
+
+ $ printf "%b\n" 1 1 1 4 1 1 1 4 1 1 1 4 |
+ time parallel -P4 sleep {}
+ (7 seconds)
+ $ printf "%b\n" 1 1 1 4 1 1 1 4 1 1 1 4 |
+ time ./parallel-bash.bash -p 4 -c sleep {}
+ (12 seconds)
+
+Because it uses bash lists, the total number of jobs is limited to
+167000..265000 depending on your environment. You get a segmentation
+fault, when you reach the limit.
Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend
running jobs.
@@ -3189,7 +3200,7 @@ running jobs.
4$ something | parallel -j 5 echo {} {}
https://reposhub.com/python/command-line-tools/Akianonymus-parallel-bash.html
-(Last checked: 2021-02)
+(Last checked: 2021-06)
=head2 DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel
@@ -3214,9 +3225,131 @@ https://github.com/themattrix/bash-concurrent
(Last checked: 2021-02)
-=head2 Todo
+=head2 DIFFERENCES BETWEEN spawntool AND GNU Parallel
+
+Summary (see legend above):
+
+=over
+
+=item I1 - - - - - -
+
+=item M1 - - - - M6
+
+=item - O2 O3 - O5 O6 - x x O10
+
+=item E1 - - - - - -
+
+=item - - - - - - - - -
+
+=item - -
+
+=back
+
+B<spawn> reads a full command line from stdin which it executes in
+parallel.
+
+
+http://code.google.com/p/spawntool/
+(Last checked: 2021-07)
+
+
+=head2 DIFFERENCES BETWEEN go-pssh AND GNU Parallel
+
+Summary (see legend above):
+
+=over
+
+=item - - - - - - -
+
+=item M1 - - - - -
+
+=item O1 - - - - - - x x O10
+
+=item E1 - - - - - -
+
+=item R1 R2 - - - R6 - - -
+
+=item - -
+
+=back
+
+B<go-pssh> does B<ssh> in parallel to multiple machines. It runs the
+same command on multiple machines similar to B<--nonall>.
+
+The hostnames must be given as IP-addresses (not as hostnames).
+
+Output is sent to stdout (standard output) if command is successful,
+and to stderr (standard error) if the command fails.
+
+=head3 EXAMPLES FROM go-pssh
+
+ 1$ go-pssh -l <ip>,<ip> -u <user> -p <port> -P <passwd> -c "<command>"
+
+ 1$ parallel -S 'sshpass -p <passwd> ssh -p <port> <user>@<ip>' \
+ --nonall "<command>"
+
+ 2$ go-pssh scp -f host.txt -u <user> -p <port> -P <password> \
+ -s /local/file_or_directory -d /remote/directory
+
+ 2$ parallel --nonall --slf host.txt \
+ --basefile /local/file_or_directory/./ --wd /remote/directory
+ --ssh 'sshpass -p <password> ssh -p <port> -l <user>' true
+
+ 3$ go-pssh scp -l <ip>,<ip> -u <user> -p <port> -P <password> \
+ -s /local/file_or_directory -d /remote/directory
+
+ 3$ parallel --nonall -S <ip>,<ip> \
+ --basefile /local/file_or_directory/./ --wd /remote/directory
+ --ssh 'sshpass -p <password> ssh -p <port> -l <user>' true
+
+https://github.com/xuchenCN/go-pssh
+(Last checked: 2021-07)
+
+
+=head2 DIFFERENCES BETWEEN go-parallel AND GNU Parallel
+
+Summary (see legend above):
+
+=over
+
+=item I1 I2 - - - - I7
+
+=item - - M3 - - M6
+
+=item - O2 O3 - O5 - - x x - O10
+
+=item E1 - - E4 - - -
+
+=item - - - - - - - - -
+
+=item - -
+
+=back
+
+B<go-parallel> uses Go templates for replacement strings. Quite
+similar to the I<{= perl expr =}> replacement string.
+
+=head3 EXAMPLES FROM go-parallel
+
+ 1$ go-parallel -a ./files.txt -t 'cp {{.Input}} {{.Input | dirname | dirname}}'
+
+ 1$ parallel -a ./files.txt cp {} '{= $_=::dirname(::dirname($_)) =}'
+
+ 2$ go-parallel -a ./files.txt -t 'mkdir -p {{.Input}} {{noExt .Input}}'
+
+ 2$ parallel -a ./files.txt echo mkdir -p {} {.}
+
+ 3$ go-parallel -a ./files.txt -t 'mkdir -p {{.Input}} {{.Input | basename | noExt}}'
+
+ 3$ parallel -a ./files.txt echo mkdir -p {} {/.}
https://github.com/mylanconnolly/parallel
+(Last checked: 2021-07)
+
+
+=head2 Todo
+
+http://code.google.com/p/push/ (cannot compile)
https://github.com/krashanoff/parallel
@@ -3234,42 +3367,16 @@ https://github.com/benoror/better-npm-run - not obvious how to use
https://github.com/bahmutov/with-package
-https://github.com/xuchenCN/go-pssh
-
https://github.com/flesler/parallel
https://github.com/Julian/Verge
-http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
+https://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html
-http://vicerveza.homeunix.net/~viric/soft/ts/
+https://vicerveza.homeunix.net/~viric/soft/ts/
https://github.com/chapmanjacobd/que
-https://github.com/ExpectationMax/simple_gpu_scheduler
- simple_gpu_scheduler --gpus 0 1 2 < gpu_commands.txt
- parallel -j3 --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' < gpu_commands.txt
-
- simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
- parallel --header : --shuf -j3 -v CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128
-
- simple_hypersearch "python3 train_dnn.py --lr {lr} --batch_size {bs}" --n-samples 5 -p lr 0.001 0.0005 0.0001 -p bs 32 64 128 | simple_gpu_scheduler --gpus 0,1,2
- parallel --header : --shuf CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1; seq() > 5 and skip() =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 128
-
- touch gpu.queue
- tail -f -n 0 gpu.queue | simple_gpu_scheduler --gpus 0,1,2 &
- echo "my_command_with | and stuff > logfile" >> gpu.queue
-
- touch gpu.queue
- tail -f -n 0 gpu.queue | parallel -j3 CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}' &
- # Needed to fill job slots once
- seq 3 | parallel echo true >> gpu.queue
- # Add jobs
- echo "my_command_with | and stuff > logfile" >> gpu.queue
- # Needed to flush output from completed jobs
- seq 3 | parallel echo true >> gpu.queue
-
-https://github.com/Overv/outrun#outrun
=head1 TESTING OTHER TOOLS
@@ -3533,7 +3640,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
+along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I
=====================================
src/parallel_book.pod
=====================================
@@ -108,7 +108,7 @@ job slots with B<-j>. Give B<-j> the number of jobs to run in
parallel:
parallel -j50 \
- wget http://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \
+ wget https://ftpmirror.gnu.org/parallel/parallel-{1}{2}22.tar.bz2 \
::: 2012 2013 2014 2015 2016 \
::: 01 02 03 04 05 06 07 08 09 10 11 12
=====================================
src/parallel_tutorial.pod
=====================================
@@ -14,11 +14,11 @@ real world.
=head2 Reader's guide
If you prefer reading a book buy B<GNU Parallel 2018> at
-http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
+https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html
or download it at: https://doi.org/10.5281/zenodo.1146014
Otherwise start by watching the intro videos for a quick introduction:
-http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
+https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Then browse through the B<EXAMPLE>s after the list of B<OPTIONS> in
B<man parallel> (Use B<LESS=+/EXAMPLE: man parallel>). That will give
@@ -3006,11 +3006,11 @@ Output:
GNU parallel 20210122
Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software
Foundation, Inc.
- License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
+ License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
- Web site: http://www.gnu.org/software/parallel
+ Web site: https://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
@@ -3045,7 +3045,7 @@ Output:
month = {Feb},
number = {1},
volume = {36},
- url = {http://www.gnu.org/s/parallel},
+ url = {https://www.gnu.org/s/parallel},
year = {2011},
pages = {42-47},
doi = {10.5281/zenodo.16303}
@@ -3121,7 +3121,7 @@ If you like GNU B<parallel>:
=item *
(Re-)walk through the tutorial if you have not done so in the past year
-(http://www.gnu.org/software/parallel/parallel_tutorial.html)
+(https://www.gnu.org/software/parallel/parallel_tutorial.html)
=item *
=====================================
src/parcat
=====================================
@@ -14,7 +14,7 @@
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
-# along with this program; if not, see <http://www.gnu.org/licenses/>
+# along with this program; if not, see <https://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA
#
=====================================
src/parset
=====================================
@@ -86,33 +86,33 @@ Do the below and restart your shell.
bash: Put this in $HOME/.bashrc: . `which env_parallel.bash`
E.g. by doing: echo '. `which env_parallel.bash`' >> $HOME/.bashrc
- Supports: aliases, functions, variables, arrays
-
-zsh: Put this in $HOME/.zshrc: . `which env_parallel.zsh`
- E.g. by doing: echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
- Supports: functions, variables, arrays
-
-fish: Unsupported
+ Supports: variables, aliases, functions, arrays
ksh: Put this in $HOME/.kshrc: source `which env_parallel.ksh`
E.g. by doing: echo 'source `which env_parallel.ksh`' >> $HOME/.kshrc
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
mksh: Put this in $HOME/.mkshrc: source `which env_parallel.mksh`
E.g. by doing: echo 'source `which env_parallel.mksh`' >> $HOME/.mkshrc
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
pdksh: Put this in $HOME/.profile: source `which env_parallel.pdksh`
E.g. by doing: echo '. `which env_parallel.pdksh`' >> $HOME/.profile
- Supports: aliases, functions, variables, arrays
+ Supports: variables, aliases, functions, arrays
+
+zsh: Put this in $HOME/.zshrc: . `which env_parallel.zsh`
+ E.g. by doing: echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
+ Supports: variables, functions, arrays
ash: Put this in $HOME/.profile: . `which env_parallel.ash`
E.g. by doing: echo '. `which env_parallel.ash`' >> $HOME/.profile
- Supports: aliases, variables
+ Supports: variables, aliases
dash: Put this in $HOME/.profile: . `which env_parallel.dash`
E.g. by doing: echo '. `which env_parallel.dash`' >> $HOME/.profile
- Supports: aliases, variables
+ Supports: variables, aliases
+
+fish: Unsupported
csh: Unsupported
=====================================
src/parset.pod
=====================================
@@ -29,12 +29,13 @@ The B<parset> and B<env_parset> functions are defined as part of
B<env_parallel>.
If I<variablename> is a single variable name, this will be treated as
-the destination variable and made into an array.
+the destination variable. If the variable is defined as an associative
+array (using B<typeset -A myassoc>), this will be used. Otherwise the
+variable will be made into a normal array.
If I<variablename> contains multiple names separated by ',' or space,
the names will be the destination variables. The number of names must
-be at least the number of jobs - otherwise some tmp files will not be
-cleaned up.
+be at least the number of jobs.
=head1 OPTIONS
@@ -65,6 +66,12 @@ Put output into vars B<$seq, $pwd, $ls>:
parset "${into_vars[*]}" ::: "seq 10" pwd ls
echo "$ls"
+Put output into associative array B<myassoc> (not supported for mksh):
+
+ typeset -A myassoc
+ parset myassoc seq ::: 4 5 ::: 6 7
+ echo "${myassoc[4 7]}"
+
The commands to run can be an array:
cmd=("echo first" "echo '<<joe \"double space\" cartoon>>'" "pwd")
@@ -76,8 +83,8 @@ B<parset> can read from stdin (standard input) if it is a file:
parset res echo < parallel_input_file
-but B<parset> can not be part of a pipe. In particular this means it
-cannot read from a pipe or write to a pipe:
+but B<parset> can I<not> be part of a pipe. In particular this means
+it cannot read from a pipe or write to a pipe:
seq 10 | parset res echo Does not work
@@ -102,6 +109,7 @@ or Bash/Zsh/Ksh process substitution:
echo "${res[1]}"
echo "${res[99]}"
+
=head3 Installation
Put this in the relevant B<$HOME/.bashrc> or B<$HOME/.zshenv> or B<$HOME/.kshrc>:
=====================================
src/parsort
=====================================
@@ -37,7 +37,8 @@ Sort stdin (standard input) numerically:
=head1 PERFORMANCE
-B<parsort> is faster on files, because these can be read in parallel.
+B<parsort> is faster on a file than on stdin (standard input), because
+different parts of a file can be read in parallel.
On a 48 core machine you should see a speedup of 3x over B<sort>.
@@ -121,7 +122,7 @@ GetOptions(
"help" => \$opt::dummy,
) || exit(255);
$Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1];
-$Global::version = 20210322;
+$Global::version = 20210822;
if($opt::version) { version(); exit 0; }
@Global::sortoptions =
shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]);
@@ -174,7 +175,8 @@ sub sort_stdin {
map { mkfifo($_,0600) } @fifos;
# This trick removes the fifo as soon as it is connected in the other end
# (rm fifo; ...) < fifo
- my @cmd = map { "(rm $_; sort @Global::sortoptions) < $_" } @fifos;
+ my @cmd = (map { "(rm $_; sort @Global::sortoptions) < $_" }
+ map { Q($_) } @fifos);
@cmd = merge(@cmd);
if(fork) {
} else {
@@ -196,11 +198,13 @@ sub tmpname {
my($tmpname);
if(not -w $ENV{'TMPDIR'}) {
if(not -e $ENV{'TMPDIR'}) {
- ::error("Tmpdir '$ENV{'TMPDIR'}' does not exist.","Try 'mkdir $ENV{'TMPDIR'}'");
+ ::error("Tmpdir '$ENV{'TMPDIR'}' does not exist.","Try 'mkdir ".
+ Q($ENV{'TMPDIR'})."'");
} else {
- ::error("Tmpdir '$ENV{'TMPDIR'}' is not writable.","Try 'chmod +w $ENV{'TMPDIR'}'");
+ ::error("Tmpdir '$ENV{'TMPDIR'}' is not writable.","Try 'chmod +w "..
+ Q($ENV{'TMPDIR'})."'");
}
- ::wait_and_exit(255);
+ exit(255);
}
do {
$tmpname = $ENV{'TMPDIR'}."/".$name.
@@ -332,6 +336,76 @@ sub Q($) {
}
+sub status(@) {
+ my @w = @_;
+ my $fh = $Global::status_fd || *STDERR;
+ print $fh map { ($_, "\n") } @w;
+ flush $fh;
+}
+
+sub status_no_nl(@) {
+ my @w = @_;
+ my $fh = $Global::status_fd || *STDERR;
+ print $fh @w;
+ flush $fh;
+}
+
+sub warning(@) {
+ my @w = @_;
+ my $prog = $Global::progname || "parsort";
+ status_no_nl(map { ($prog, ": Warning: ", $_, "\n"); } @w);
+}
+
+{
+ my %warnings;
+ sub warning_once(@) {
+ my @w = @_;
+ my $prog = $Global::progname || "parsort";
+ $warnings{@w}++ or
+ status_no_nl(map { ($prog, ": Warning: ", $_, "\n"); } @w);
+ }
+}
+
+sub error(@) {
+ my @w = @_;
+ my $prog = $Global::progname || "parsort";
+ status(map { ($prog.": Error: ". $_); } @w);
+}
+
+sub die_bug($) {
+ my $bugid = shift;
+ print STDERR
+ ("$Global::progname: This should not happen. You have found a bug. ",
+ "Please follow\n",
+ "https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS\n",
+ "\n",
+ "Include this in the report:\n",
+ "* The version number: $Global::version\n",
+ "* The bugid: $bugid\n",
+ "* The command line being run\n",
+ "* The files being read (put the files on a webserver if they are big)\n",
+ "\n",
+ "If you get the error on smaller/fewer files, please include those instead.\n");
+ exit(255);
+}
+
+sub version() {
+ # Returns: N/A
+ print join
+ ("\n",
+ "GNU $Global::progname $Global::version",
+ "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software",
+ "Foundation, Inc.",
+ "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>",
+ "This is free software: you are free to change and redistribute it.",
+ "GNU $Global::progname comes with no warranty.",
+ "",
+ "Web site: https://www.gnu.org/software/${Global::progname}\n",
+ "When using programs that use GNU Parallel to process data for publication",
+ "please cite as described in 'parallel --citation'.\n",
+ );
+}
+
if(@ARGV) {
sort_files(@ARGV);
} elsif(length $opt::files0_from) {
=====================================
src/sem
=====================================
@@ -14,7 +14,7 @@
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
-# along with this program; if not, see <http://www.gnu.org/licenses/>
+# along with this program; if not, see <https://www.gnu.org/licenses/>
# or write to the Free Software Foundation, Inc., 51 Franklin St,
# Fifth Floor, Boston, MA 02110-1301 USA
#
@@ -562,7 +562,8 @@ sub pipe_part_files(@) {
my ($file) = @_;
my $buf = "";
if(not -f $file and not -b $file) {
- ::error("$file is not a seekable file.");
+ ::error("--pipepart only works on seekable files, not streams/pipes.",
+ "$file is not a seekable file.");
::wait_and_exit(255);
}
my $header = find_header(\$buf,open_or_exit($file));
@@ -969,6 +970,7 @@ sub spreadstdin() {
my $header = find_header(\$buf,$in);
my $anything_written;
my $eof;
+ my $garbage_read;
sub read_block() {
# Read a --blocksize from STDIN
@@ -1023,23 +1025,37 @@ sub spreadstdin() {
# Pass records of N regexps
# -N => (start..*?end){n}
# -L -N => (start..*?end){n*l}
- my $read_n_lines = -1+
+ if(not $garbage_read) {
+ $garbage_read = 1;
+ if($buf !~ /^$recstart/o) {
+ # Buf does not start with $recstart => There is garbage.
+ # Make a single record of the garbage
+ if($buf =~
+ /(?s:^)(
+ (?:(?:(?!$recend$recstart)(?s:.))*?$recend)
+ )
+ # Followed by recstart
+ (?=$recstart)/mox and length $1 > 0) {
+ $anything_written +=
+ write_record_to_pipe($chunk_number++,\$header,\$buf,
+ $recstart,$recend,length $1);
+ shorten(\$buf,length $1);
+ }
+ }
+ }
+
+ my $n_records =
$Global::max_number_of_args * ($Global::max_lines || 1);
# (?!negative lookahead) is needed to avoid backtracking
# See: https://unix.stackexchange.com/questions/439356/
+ # (?s:.) = (.|[\n]) but faster
while($buf =~
- /(
- # Either recstart or at least one char from start
- ^(?: $recstart | .)
- # followed something
- (?:(?!$recend$recstart).)*?
- # and then recend
- $recend
- # Then n-1 times recstart.*recend
- (?:$recstart(?:(?!$recend$recstart).)*?$recend){$read_n_lines}
+ /(?s:^)(
+ # n more times recstart.*recend
+ (?:$recstart(?:(?!$recend$recstart)(?s:.))*?$recend){$n_records}
)
# Followed by recstart
- (?=$recstart)/osx) {
+ (?=$recstart)/mox and length $1 > 0) {
$anything_written +=
write_record_to_pipe($chunk_number++,\$header,\$buf,
$recstart,$recend,length $1);
@@ -1050,7 +1066,8 @@ sub spreadstdin() {
sub pass_regexp() {
# Find the last recend-recstart in $buf
$eof and return;
- if($buf =~ /^(.*$recend)$recstart.*?$/os) {
+ # (?s:.) = (.|[\n]) but faster
+ if($buf =~ /^((?s:.)*$recend)$recstart(?s:.)*?$/mox) {
$anything_written +=
write_record_to_pipe($chunk_number++,\$header,\$buf,
$recstart,$recend,length $1);
@@ -1230,20 +1247,18 @@ sub recstartrecend() {
$recend = $opt::recend;
if($opt::regexp and $recend eq '') {
# --regexp --recend ''
- $recend = '.';
+ $recend = '(?s:.)';
}
}
if($opt::regexp) {
+ # Do not allow /x comments - to avoid having to quote space
+ $recstart = "(?-x:".$recstart.")";
+ $recend = "(?-x:".$recend.")";
# If $recstart/$recend contains '|'
- # this should only apply to the regexp
+ # the | should only apply to the regexp
$recstart = "(?:".$recstart.")";
$recend = "(?:".$recend.")";
- # Quote # and space
- $recstart =~ s/#/\\#/g;
- $recend =~ s/#/\\#/g;
- $recstart =~ s/ /\\ /g;
- $recend =~ s/ /\\ /g;
} else {
# $recstart/$recend = printf strings (\n)
$recstart =~ s/\\([0rnt\'\"\\])/"qq|\\$1|"/gee;
@@ -1541,6 +1556,8 @@ sub options_hash() {
"nice=i" => \$opt::nice,
"tag" => \$opt::tag,
"tagstring|tag-string=s" => \$opt::tagstring,
+ "ctag" => \$opt::ctag,
+ "ctagstring|ctag-string=s" => \$opt::ctagstring,
"onall" => \$opt::onall,
"nonall" => \$opt::nonall,
"filter-hosts|filterhosts|filter-host" => \$opt::filter_hosts,
@@ -1603,18 +1620,18 @@ sub options_hash() {
# xargs-compatibility - implemented, man, testsuite
"max-procs|P=s" => \$opt::jobs,
"delimiter|d=s" => \$opt::d,
- "max-chars|s=i" => \$opt::max_chars,
+ "max-chars|s=s" => \$opt::max_chars,
"arg-file|a=s" => \@opt::a,
"no-run-if-empty|r" => \$opt::r,
"replace|i:s" => \$opt::i,
"E=s" => \$opt::eof,
"eof|e:s" => \$opt::eof,
- "max-args|maxargs|n=i" => \$opt::max_args,
- "max-replace-args|N=i" => \$opt::max_replace_args,
+ "max-args|maxargs|n=s" => \$opt::max_args,
+ "max-replace-args|N=s" => \$opt::max_replace_args,
"colsep|col-sep|C=s" => \$opt::colsep,
"csv"=> \$opt::csv,
"help|h" => \$opt::help,
- "L=f" => \$opt::L,
+ "L=s" => \$opt::L,
"max-lines|l:f" => \$opt::max_lines,
"interactive|p" => \$opt::interactive,
"verbose|t" => \$opt::verbose,
@@ -1658,6 +1675,7 @@ sub options_hash() {
"hgrp|hostgrp|hostgroup|hostgroups" => \$opt::hostgroups,
"embed" => \$opt::embed,
"filter=s" => \@opt::filter,
+ "parset=s" => \$opt::parset,
);
}
@@ -1703,6 +1721,40 @@ sub get_options_from_array($@) {
return $retval;
}
+sub parse_parset() {
+ $Global::progname = "parset";
+ @Global::parset_vars = split /[ ,]/, $opt::parset;
+ my $var_or_assoc = shift @Global::parset_vars;
+ # Legal names: var _v2ar arrayentry[2]
+ my @illegal = (grep { not /^[a-zA-Z_][a-zA-Z_0-9]*(\[\d+\])?$/ }
+ @Global::parset_vars);
+ if(@illegal) {
+ ::error
+ ("@illegal is an invalid variable name.",
+ "Variable names must be letter followed by letters or digits.",
+ "Usage:",
+ " parset varname GNU Parallel options and command");
+ wait_and_exit(255);
+ }
+ if($var_or_assoc eq "assoc") {
+ my $var = shift @Global::parset_vars;
+ print "$var=(";
+ $Global::parset = "assoc";
+ $Global::parset_endstring=")\n";
+ } elsif($var_or_assoc eq "var") {
+ if($#Global::parset_vars > 0) {
+ $Global::parset = "var";
+ } else {
+ my $var = shift @Global::parset_vars;
+ print "$var=(";
+ $Global::parset = "array";
+ $Global::parset_endstring=")\n";
+ }
+ } else {
+ ::die_bug("parset: unknown '$opt::parset'");
+ }
+}
+
sub parse_options(@) {
# Returns: N/A
init_globals();
@@ -1730,11 +1782,21 @@ sub parse_options(@) {
}
::debug("init","Global::shell $Global::shell\n");
$Global::cshell = $Global::shell =~ m:(/[-a-z]*)?csh:;
+ if(defined $opt::parset) { parse_parset(); }
if(defined $opt::X) { $Global::ContextReplace = 1; }
if(defined $opt::silent) { $Global::verbose = 0; }
if(defined $opt::null) { $/ = "\0"; }
if(defined $opt::d) { $/ = unquote_printf($opt::d) }
parse_replacement_string_options();
+ $opt::tag ||= $opt::ctag;
+ $opt::tagstring ||= $opt::ctagstring;
+ if(defined $opt::ctag or defined $opt::ctagstring) {
+ $Global::color = 1;
+ }
+ if(defined $opt::tag and not defined $opt::tagstring) {
+ # Default = {}
+ $opt::tagstring = $Global::parensleft.$Global::parensright;
+ }
if(defined $opt::tagstring) {
$opt::tagstring = unquote_printf($opt::tagstring);
if($opt::tagstring =~ /\Q$Global::parensleft\E.*\Q$Global::parensright\E/
@@ -1751,6 +1813,7 @@ sub parse_options(@) {
if(defined $opt::verbose) { $Global::stderr_verbose = 1; }
if(defined $opt::eof) { $Global::end_of_file_string = $opt::eof; }
if(defined $opt::max_args) {
+ $opt::max_args = multiply_binary_prefix($opt::max_args);
$Global::max_number_of_args = $opt::max_args;
}
if(defined $opt::blocktimeout) {
@@ -1806,6 +1869,9 @@ sub parse_options(@) {
if(defined $opt::max_line_length_allowed) {
print Limits::Command::real_max_length(),"\n"; wait_and_exit(0);
}
+ if(defined $opt::max_chars) {
+ $opt::max_chars = multiply_binary_prefix($opt::max_chars);
+ }
if(defined $opt::version) { version(); wait_and_exit(0); }
if(defined $opt::record_env) { record_env(); wait_and_exit(0); }
if(defined $opt::show_limits) { show_limits(); }
@@ -1939,10 +2005,14 @@ sub parse_options(@) {
$opt::max_lines = 1;
$opt::null = 1;
$/ = "\0";
- } elsif ($opt::max_lines == 0) {
- # If not given (or if 0 is given) => 1
- $opt::max_lines = 1;
+ } else {
+ $opt::max_lines = multiply_binary_prefix($opt::max_lines);
+ if ($opt::max_lines == 0) {
+ # If not given (or if 0 is given) => 1
+ $opt::max_lines = 1;
+ }
}
+
$Global::max_lines = $opt::max_lines;
if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args
@@ -1952,6 +2022,7 @@ sub parse_options(@) {
# Read more than one arg at a time (-L, -N)
if(defined $opt::L) {
+ $opt::L = multiply_binary_prefix($opt::L);
$Global::max_lines = $opt::L;
if(not $opt::pipe) {
# --pipe -L means length of record - not max_number_of_args
@@ -1959,6 +2030,7 @@ sub parse_options(@) {
}
}
if(defined $opt::max_replace_args) {
+ $opt::max_replace_args = multiply_binary_prefix($opt::max_replace_args);
$Global::max_number_of_args = $opt::max_replace_args;
$Global::ContextReplace = 1;
}
@@ -1967,10 +2039,6 @@ sub parse_options(@) {
not ($opt::xargs or $opt::m)) {
$Global::ContextReplace = 1;
}
- if(defined $opt::tag and not defined $opt::tagstring) {
- # Default = {}
- $opt::tagstring = $Global::parensleft.$Global::parensright;
- }
if(grep /^$Global::arg_sep\+?$|^$Global::arg_file_sep\+?$/o, @ARGV) {
# Deal with ::: :::+ :::: and ::::+
@ARGV = read_args_from_command_line();
@@ -2173,7 +2241,7 @@ sub check_invalid_option_combinations() {
sub init_globals() {
# Defaults:
- $Global::version = 20210322;
+ $Global::version = 20210822;
$Global::progname = 'parallel';
$::name = "GNU Parallel";
$Global::infinity = 2**31;
@@ -2228,10 +2296,20 @@ sub init_globals() {
'{:(\d+?)}' => 'substr($_,0,$$1) = ""',
# Bash ${a:2:3}
'{:(\d+?):(\d+?)}' => '$_ = substr($_,$$1,$$2);',
+ # echo {#z.*z.} ::: z.z.z.foo => z.foo
+ # echo {##z.*z.} ::: z.z.z.foo => foo
# Bash ${a#bc}
- '{#([^#}][^}]*?)}' => 's/^$$1//;',
+ '{#([^#}][^}]*?)}' =>
+ '$nongreedy=::make_regexp_ungreedy($$1);s/^$nongreedy(.*)/$1/;',
+ # Bash ${a##bc}
+ '{##([^#}][^}]*?)}' => 's/^$$1//;',
+ # echo {%.z.*z} ::: foo.z.z.z => foo.z
+ # echo {%%.z.*z} ::: foo.z.z.z => foo
# Bash ${a%def}
- '{%([^}]+?)}' => 's/$$1$//;',
+ '{%([^}]+?)}' =>
+ '$nongreedy=::make_regexp_ungreedy($$1);s/(.*)$nongreedy$/$1/;',
+ # Bash ${a%%def}
+ '{%%([^}]+?)}' => 's/$$1$//;',
# Bash ${a/def/ghi} ${a/def/}
'{/([^}]+?)/([^}]*?)}' => 's/$$1/$$2/;',
# Bash ${a^a}
@@ -2493,7 +2571,7 @@ sub open_joblog() {
# $opt::joblog
# $opt::results
# $Global::job_already_run
- # %Global::fd
+ # %Global::fh
my $append = 0;
if(($opt::resume or $opt::resume_failed)
and
@@ -2599,7 +2677,7 @@ sub open_joblog() {
} else {
if($opt::joblog eq "-") {
# Use STDOUT as joblog
- $Global::joblog = $Global::fd{1};
+ $Global::joblog = $Global::fh{1};
} elsif(not open($Global::joblog, ">", $opt::joblog)) {
# Overwrite the joblog
::error("Cannot write to --joblog $opt::joblog.");
@@ -2628,8 +2706,8 @@ sub open_json_csv() {
# by forcing all other output to /dev/null
open my $fd, ">", "/dev/null" or
::die_bug("Can't >/dev/null in csv: $!");
- $Global::fd{1} = $fd;
- $Global::fd{2} = $fd;
+ $Global::fh{1} = $fd;
+ $Global::fh{2} = $fd;
} elsif($Global::csvsep) {
if(not open($Global::csv_fh,">",$opt::results)) {
::error("Cannot open results file `$opt::results': ".
@@ -3108,7 +3186,7 @@ sub save_stdin_stdout_stderr() {
# Remember the original STDIN, STDOUT and STDERR
# and file descriptors opened by the shell (e.g. 3>/tmp/foo)
# Uses:
- # %Global::fd
+ # %Global::fh
# $Global::original_stderr
# $Global::original_stdin
# Returns: N/A
@@ -3123,7 +3201,7 @@ sub save_stdin_stdout_stderr() {
# 2-argument-open is used to be compatible with old perl 5.8.0
# bug #43570: Perl 5.8.0 creates 61 files
if(open($fh,">&=$fdno")) {
- $Global::fd{$fdno}=$fh;
+ $Global::fh{$fdno}=$fh;
}
}
open $Global::original_stderr, ">&", "STDERR" or
@@ -3139,7 +3217,7 @@ sub enough_file_handles() {
# another job
# Uses:
# $opt::ungroup
- # %Global::fd
+ # %Global::fh
# Returns:
# 1 if ungrouped (thus not needing extra filehandles)
# 0 if too few filehandles
@@ -3151,7 +3229,7 @@ sub enough_file_handles() {
# open3 uses 2 extra filehandles temporarily
# We need a filehandle for each redirected file descriptor
# (normally just STDOUT and STDERR)
- for my $i (1..(7+2+keys %Global::fd)) {
+ for my $i (1..(7+2+keys %Global::fh)) {
$enough_filehandles &&= open($fh{$i}, "<", "/dev/null");
}
for (values %fh) { close $_; }
@@ -3835,7 +3913,8 @@ sub progress() {
my $arg = $Global::newest_job ?
$Global::newest_job->{'commandline'}->
replace_placeholders(["\257<\257>"],0,0) : "";
- # These chars mess up display in the terminal
+ # These chars mess up display in the terminal in US-ASCII
+ # and in some combinations as UTF8 (e.g. ঐ ও ঔ ক 𐅪 𐅫 𐅬)
$arg =~ tr/[\011-\016\033\302-\365]//d;
my $eta_dhms = ::seconds_to_time_units($eta);
my $bar_text =
@@ -4788,6 +4867,8 @@ sub reaper() {
$job->set_exitstatus($? >> 8);
$job->set_exitsignal($? & 127);
}
+
+ debug("run", "\nseq ",$job->seq()," died (", $job->exitstatus(), ")");
if($Global::delayauto or $Global::sshdelayauto) {
if($job->exitstatus()) {
# Job failed: Increase delay (if $opt::(ssh)delay set)
@@ -4798,9 +4879,8 @@ sub reaper() {
$opt::delay &&= $opt::delay * 0.9;
$opt::sshdelay &&= $opt::sshdelay * 0.9;
}
+ debug("run", "delay:$opt::delay ssh:$opt::sshdelay ");
}
-
- debug("run", "seq ",$job->seq()," died (", $job->exitstatus(), ")");
$job->set_endtime(::now());
my $sshlogin = $job->sshlogin();
$sshlogin->dec_jobs_running();
@@ -4924,6 +5004,10 @@ sub wait_and_exit($) {
# Avoid: Warning: unable to close filehandle properly: No space
# left on device during global destruction.
$SIG{__WARN__} = sub {};
+ if($opt::parset) {
+ # Make the shell script return $error
+ print "$Global::parset_endstring\nreturn $error";
+ }
exit($error);
}
@@ -4968,8 +5052,8 @@ sub usage() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- " Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').",
- " Zenodo. https://doi.org/10.5281/zenodo.4628277",
+ " Tange, O. (2021, August 22). GNU Parallel 20210822 ('Kabul').",
+ " Zenodo. https://doi.org/10.5281/zenodo.5233953",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice
@@ -4999,8 +5083,8 @@ sub citation_notice() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- " Tange, O. (2021, March 22). GNU Parallel 20210322 ('2002-01-06').",
- " Zenodo. https://doi.org/10.5281/zenodo.4628277",
+ " Tange, O. (2021, August 22). GNU Parallel 20210822 ('Kabul').",
+ " Zenodo. https://doi.org/10.5281/zenodo.5233953",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@@ -5076,8 +5160,8 @@ sub error(@) {
sub die_bug($) {
my $bugid = shift;
print STDERR
- ("$Global::progname: This should not happen. You have found a bug.\n",
- "Please contact <parallel\@gnu.org> and follow\n",
+ ("$Global::progname: This should not happen. You have found a bug. ",
+ "Please follow\n",
"https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS\n",
"\n",
"Include this in the report:\n",
@@ -5123,20 +5207,20 @@ sub citation() {
"If you use programs that use GNU Parallel to process data for an article in a",
"scientific publication, please cite:",
"",
- "\@software{tange_2021_4628277,",
+ "\@software{tange_2021_5233953,",
" author = {Tange, Ole},",
- " title = {GNU Parallel 20210322 ('2002-01-06')},",
- " month = Mar,",
- " year = 2020,",
+ " title = {GNU Parallel 20210822 ('Kabul')},",
+ " month = Aug,",
+ " year = 2021,",
" note = {{GNU Parallel is a general parallelizer to run",
" multiple serial command line programs in parallel",
" without changing them.}},",
" publisher = {Zenodo},",
- " doi = {10.5281/zenodo.4628277},",
- " url = {https://doi.org/10.5281/zenodo.4628277}",
+ " doi = {10.5281/zenodo.5233953},",
+ " url = {https://doi.org/10.5281/zenodo.5233953}",
"}",
"",
- "(Feel free to use \\nocite{tange_2021_4628277})",
+ "(Feel free to use \\nocite{tange_2021_5233953})",
"",
# Before changing this line, please read
# https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and
@@ -5295,6 +5379,8 @@ env_parallel --session
env_parallel -k echo ::: Put your code here
parset p,y,c,h -k echo ::: Put your code here
echo $p $y $c $h
+echo You can also activate GNU Parallel for interactive use by:
+echo . "$0"
!;
} else {
::error("Cannot open $0");
@@ -5712,9 +5798,9 @@ sub which(@) {
# ash bash csh dash fdsh fish fizsh ksh ksh93 mksh pdksh
# posh rbash rc rush rzsh sash sh static-sh tcsh yash zsh
- my @shells = (qw(ash bash bsd-csh csh dash fdsh fish fizsh
- ksh ksh93 lksh mksh pdksh posh rbash rc rush rzsh sash sh
- static-sh tcsh yash zsh -sh -csh -bash),
+ my @shells = (qw(ash bash bsd-csh csh dash fdsh fish fizsh ksh
+ ksh93 lksh mksh pdksh posh rbash rc rush rzsh sash sh
+ static-sh tcsh yash zsh -sh -csh -bash),
'-sh (sh)' # sh on FreeBSD
);
# Can be formatted as:
@@ -5762,13 +5848,15 @@ sub which(@) {
if($shellpath = readlink "/proc/$testpid/exe") {
::debug("init","procpath $shellpath\n");
if($shellpath =~ m:/$shell$:o) {
- ::debug("init", "proc which ".$shellpath." => ");
+ ::debug("init",
+ "proc which ".$shellpath." => ");
return $shellpath;
}
}
}
::debug("init", "which ".$shellname." => ");
- $shellpath = (which($shellname,@{$fakename{$shellname}}))[0];
+ $shellpath = (which($shellname,
+ @{$fakename{$shellname}}))[0];
::debug("init", "shell path $shellpath\n");
return $shellpath;
}
@@ -5829,11 +5917,13 @@ sub which(@) {
if(not %pid_parentpid_cmd) {
# Filter for SysV-style `ps`
- my $sysv = q( ps -ef | perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
+ my $sysv = q( ps -ef |).
+ q(perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
q(s/^.{$s}//; print "@F[1,2] $_"' );
# Minix uses cols 2,3 and can have newlines in the command
# so lines not having numbers in cols 2,3 must be ignored
- my $minix = q( ps -ef | perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
+ my $minix = q( ps -ef |).
+ q(perl -ane '1..1 and /^(.*)CO?MM?A?N?D/ and $s=length $1;).
q(s/^.{$s}// and $F[2]>0 and $F[3]>0 and print "@F[2,3] $_"' );
# BSD-style `ps`
my $bsd = q(ps -o pid,ppid,command -ax);
@@ -5861,7 +5951,8 @@ sub which(@) {
'syllable' => "echo ps not supported",
);
}
- $pid_parentpid_cmd{$^O} or ::die_bug("pid_parentpid_cmd for $^O missing");
+ $pid_parentpid_cmd{$^O} or
+ ::die_bug("pid_parentpid_cmd for $^O missing");
my (@pidtable,%parent_of,%children_of,%name_of);
# Table with pid -> children of pid
@@ -5913,6 +6004,37 @@ sub usleep($) {
select(undef, undef, undef, $ms/1000);
}
+sub make_regexp_ungreedy {
+ my $regexp = shift;
+
+ my $class_state = 0;
+ my $escape_state = 0;
+ my $found = 0;
+ my $ungreedy = "";
+ my $c;
+
+ for $c (split (//, $regexp)) {
+ if ($found) {
+ if($c ne "?") { $ungreedy .= "?"; }
+ $found = 0;
+ }
+ $ungreedy .= $c;
+
+ if ($escape_state) { $escape_state = 0; next; }
+ if ($c eq "\\") { $escape_state = 1; next; }
+ if ($c eq '[') { $class_state = 1; next; }
+ if ($class_state) {
+ if($c eq ']') { $class_state = 0; }
+ next;
+ }
+ # Quantifiers: + * {...}
+ if ($c =~ /[*}+]/) { $found = 1; }
+ }
+ if($found) { $ungreedy .= '?'; }
+ return $ungreedy;
+}
+
+
sub __KILLER_REAPER__() {}
sub reap_usleep() {
@@ -5968,8 +6090,12 @@ sub reap_usleep() {
$SIG{CHLD} = sub { kill "ALRM", $$ };
if($opt::delay) {
# The 0.004s is approximately the time it takes for one round
- usleep(1000*($Global::newest_starttime +
- $opt::delay - 0.004 - ::now()));
+ my $next_earliest_start =
+ $Global::newest_starttime + $opt::delay - 0.004;
+ my $remaining_ms = 1000 * ($next_earliest_start - ::now());
+ # The next job can only start at $next_earliest_start
+ # so sleep until then (but sleep at least $ms)
+ usleep(::max($ms,$remaining_ms));
} else {
usleep($ms);
}
@@ -5998,7 +6124,8 @@ sub kill_youngest_if_over_limit() {
push @{$jobs_of{$job->sshlogin()}}, $job;
}
for my $sshlogin (@sshlogins) {
- for my $job (sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}}) {
+ for my $job (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}}) {
if($sshlogin->limit() == 2) {
$job->kill();
last;
@@ -6028,7 +6155,8 @@ sub suspend_young_if_not_enough_mem() {
if($free < 2*$limit) {
# Suspend all jobs (resume some of them later)
map { $_->suspended() or $_->suspend(); } @{$jobs_of{$sshlogin}};
- my @jobs = sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}};
+ my @jobs = (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}});
# how many should be running?
# limit*1 => 1;
# limit*1.5 => 2;
@@ -6049,8 +6177,8 @@ sub suspend_young_if_not_enough_mem() {
for my $job (@{$jobs_of{$sshlogin}}) {
if($job->suspended()) {
$job->resume();
- ::debug("mem","\nResume ",$#{$jobs_of{$sshlogin}}+1, " jobs. Seq ",
- $job->seq(), " resumed ",
+ ::debug("mem","\nResume ",$#{$jobs_of{$sshlogin}}+1,
+ " jobs. Seq ", $job->seq(), " resumed ",
$sshlogin->memfree()," > ",2*$limit);
last;
}
@@ -6076,7 +6204,8 @@ sub kill_youngster_if_not_enough_mem() {
push @{$jobs_of{$job->sshlogin()}}, $job;
}
for my $sshlogin (@sshlogins) {
- for my $job (sort { $b->seq() <=> $a->seq() } @{$jobs_of{$sshlogin}}) {
+ for my $job (sort { $b->seq() <=> $a->seq() }
+ @{$jobs_of{$sshlogin}}) {
if($sshlogin->memfree() < $limit) {
::debug("mem","\n",map { $_->seq()." " }
(sort { $b->seq() <=> $a->seq() }
@@ -6101,17 +6230,17 @@ sub __DEBUGGING__() {}
sub debug(@) {
# Uses:
# $Global::debug
- # %Global::fd
+ # %Global::fh
# Returns: N/A
$Global::debug or return;
@_ = grep { defined $_ ? $_ : "" } @_;
if($Global::debug eq "all" or $Global::debug eq $_[0]) {
- if($Global::fd{1}) {
- # Original stdout was saved
- my $stdout = $Global::fd{1};
- print $stdout @_[1..$#_];
+ if($Global::fh{2}) {
+ # Original stderr was saved
+ my $stderr = $Global::fh{2};
+ print $stderr @_[1..$#_];
} else {
- print @_[1..$#_];
+ print STDERR @_[1..$#_];
}
}
}
@@ -6463,16 +6592,15 @@ sub limit($) {
limit=$1;
io_file=$2;
# Do the measurement in the background
- (tmp=$(tempfile);
+ ((tmp=$(tempfile);
LANG=C iostat -x 1 2 > $tmp;
- mv $tmp $io_file) &
+ mv $tmp $io_file) </dev/null >/dev/null & );
perl -e '-e $ARGV[0] or exit(1);
for(reverse <>) {
/Device/ and last;
/(\S+)$/ and $max = $max > $1 ? $max : $1; }
- exit ($max < '$limit')' $io_file;
+ exit ('$limit' < $max)' $io_file;
};
- export -f io;
io %s %s
!,
"mem" => q!
@@ -6515,6 +6643,7 @@ sub limit($) {
local %ENV = %env;
$ENV{'SSHLOGIN'} = $self->string();
system($Global::shell,"-c",$self->{'limitscript'});
+ #::qqx($self->{'limitscript'});
::debug("limit","limit `".$self->{'limitscript'}."` result ".($?>>8)."\n");
return $?>>8;
}
@@ -8004,15 +8133,15 @@ sub cleanup_cmd($$$) {
$dir .= $_."/";
unshift @rmdir, ::shell_quote_file($dir);
}
- my $rmdir = @rmdir ? "sh -c ".::Q("rmdir @rmdir 2>/dev/null;") : "";
+ my $rmdir = @rmdir ? "rmdir @rmdir 2>/dev/null;" : "";
if(defined $opt::workdir and $opt::workdir eq "...") {
- $rmdir .= ::Q("rm -rf " . ::shell_quote_file($workdir).';');
+ $rmdir .= "rm -rf " . ::shell_quote_file($workdir).';';
}
-
- $f = ::shell_quote_file($f);
+ my $rmf = "sh -c ".
+ ::Q("rm -f ".::shell_quote_file($f)." 2>/dev/null;".$rmdir);
my $sshcmd = $self->sshcommand();
my $serverlogin = $self->serverlogin();
- return "$sshcmd $serverlogin -- ".::Q("rm -f $f; $rmdir");
+ return "$sshcmd $serverlogin -- ".::Q("$rmf");
}
{
@@ -8402,8 +8531,8 @@ sub openoutputfiles($) {
}
} else {
# --ungroup
- open($outfhw,">&",$Global::fd{1}) || die;
- open($errfhw,">&",$Global::fd{2}) || die;
+ open($outfhw,">&",$Global::fh{1}) || die;
+ open($errfhw,">&",$Global::fh{2}) || die;
# File name must be empty as it will otherwise be printed
$outname = "";
$errname = "";
@@ -8542,7 +8671,7 @@ sub filter_through_compress($) {
# thus output file can then be removed by the decompressor.
my $wpid = open(my $fdw,"|-", "(echo > $comfile; ".
empty_input_wrapper($opt::compress_program).") >".
- $self->fh($fdno,'name')) || die $?;
+ ::Q($self->fh($fdno,'name'))) || die $?;
$self->set_fh($fdno,'w',$fdw);
$self->set_fh($fdno,'wpid',$wpid);
# Decompressor: open output; -s $comfile > 0: rm $comfile output;
@@ -8675,11 +8804,11 @@ sub remove_rec_sep($) {
my ($block_ref,$recstart,$recend) = @_;
# Remove record separator
if($opt::regexp) {
- $$block_ref =~ s/$recend$recstart//gos;
+ $$block_ref =~ s/$recend$recstart//gom;
$$block_ref =~ s/^$recstart//os;
$$block_ref =~ s/$recend$//os;
} else {
- $$block_ref =~ s/\Q$recend$recstart\E//gos;
+ $$block_ref =~ s/\Q$recend$recstart\E//gom;
$$block_ref =~ s/^\Q$recstart\E//os;
$$block_ref =~ s/\Q$recend\E$//os;
}
@@ -9247,12 +9376,13 @@ sub sshlogin_wrap($) {
exec $shell, "-c", ($bashfunc."@ARGV");
die "exec: $!\n";
}
+ my $parent = getppid;
do {
- # Parent is not init (ppid=1), so sshd is alive
+ # Parent pid is not changed, so sshd is alive
# Exponential sleep up to 1 sec
$s = $s < 1 ? 0.001 + $s * 1.03 : $s;
select(undef, undef, undef, $s);
- } until ($done || getppid == 1);
+ } until ($done || getppid != $parent);
if(not $done) {
# Kill as per --termseq
my @term_seq = split/,/,$termseq;
@@ -9501,6 +9631,7 @@ sub filter($) {
replace_placeholders(\@opt::filter,0,0)) {
$run &&= eval $eval;
}
+ $self->{'commandline'}{'skip'} ||= not $run;
}
return $run;
}
@@ -10047,8 +10178,10 @@ sub interactive_start($) {
::status("See output with: $ENV{'PARALLEL_TMUX'} -S $tmuxsocket attach");
}
$tmux = "sh -c '".
- $ENV{'PARALLEL_TMUX'}." -S $tmuxsocket new-session -s p$$ -d \"sleep .2\" >/dev/null 2>&1';" .
- $ENV{'PARALLEL_TMUX'}." -S $tmuxsocket new-window -t p$$ -n $title";
+ $ENV{'PARALLEL_TMUX'}.
+ " -S $tmuxsocket new-session -s p$$ -d \"sleep .2\" >/dev/null 2>&1';" .
+ $ENV{'PARALLEL_TMUX'}.
+ " -S $tmuxsocket new-window -t p$$ -n $title";
::debug("tmux", "title len:", $l_tit, " act ", $l_act, " max ",
$Limits::Command::line_max_len, " tot ",
@@ -10060,7 +10193,8 @@ sub interactive_start($) {
(
"(".$actual_command.');'.
# The triple print is needed - otherwise the testsuite fails
- q[ perl -e 'while($t++<3){ print $ARGV[0],"\n" }' $?h/$status >> ].$tmpfifo."&".
+ q[ perl -e 'while($t++<3){ print $ARGV[0],"\n" }' $?h/$status >> ]..
+ $tmpfifo."&".
"echo $title; echo \007Job finished at: `date`;sleep 10"
).
# Run outside tmux
@@ -10193,10 +10327,10 @@ sub print($) {
}
my $returnsize = $self->returnsize();
- for my $fdno (sort { $a <=> $b } keys %Global::fd) {
+ for my $fdno (sort { $a <=> $b } keys %Global::fh) {
# Sort by file descriptor numerically: 1,2,3,..,9,10,11
$fdno == 0 and next;
- my $out_fd = $Global::fd{$fdno};
+ my $out_fh = $Global::fh{$fdno};
my $in_fh = $self->fh($fdno,"r");
if(not $in_fh) {
if(not $Job::file_descriptor_warning_printed{$fdno}++) {
@@ -10207,15 +10341,15 @@ sub print($) {
::debug("print", "File descriptor $fdno (", $self->fh($fdno,"name"), "):\n");
if($opt::linebuffer) {
# Line buffered print out
- $self->print_linebuffer($fdno,$in_fh,$out_fd);
+ $self->print_linebuffer($fdno,$in_fh,$out_fh);
} elsif($opt::files) {
- $self->print_files($fdno,$in_fh,$out_fd);
- } elsif($opt::tag or defined $opt::tagstring) {
- $self->print_tag($fdno,$in_fh,$out_fd);
+ $self->print_files($fdno,$in_fh,$out_fh);
+ } elsif($opt::results) {
+ $self->print_results($fdno,$in_fh,$out_fh);
} else {
- $self->print_normal($fdno,$in_fh,$out_fd);
+ $self->print_normal($fdno,$in_fh,$out_fh);
}
- flush $out_fd;
+ flush $out_fh;
}
::debug("print", "<<joboutput\n");
if(defined $self->{'exitstatus'}
@@ -10246,7 +10380,8 @@ sub print($) {
sub jsonquote($) {
my $a = shift;
if(not $jsonmap{"\001"}) {
- map { $jsonmap{sprintf("%c",$_)} = sprintf '\u%04x', $_ } 0..31;
+ map { $jsonmap{sprintf("%c",$_)} =
+ sprintf '\u%04x', $_ } 0..31;
}
$a =~ s/\\/\\\\/g;
$a =~ s/\"/\\"/g;
@@ -10411,7 +10546,7 @@ sub print_files($) {
# $opt::linebuffer = Print ASAP
# Returns: N/A
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
# If the job is dead: close printing fh. Needed for --compress
close $self->fh($fdno,"w");
@@ -10433,7 +10568,7 @@ sub print_files($) {
::rm($self->fh($fdno,"unlink"));
}
} elsif($fdno == 1 and $self->fh($fdno,"name")) {
- print $out_fd $self->tag(),$self->fh($fdno,"name"),"\n";
+ print $out_fh $self->tag(),$self->fh($fdno,"name"),"\n";
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}},
$self->tag(), $self->fh($fdno,"name");
@@ -10446,7 +10581,7 @@ sub print_files($) {
sub print_linebuffer($) {
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
if(defined $self->{'exitstatus'}) {
# If the job is dead: close printing fh. Needed for --compress
close $self->fh($fdno,"w");
@@ -10465,7 +10600,7 @@ sub print_linebuffer($) {
if($opt::files or ($opt::results and not $Global::csvsep)) {
# Print filename
if($fdno == 1 and not $self->fh($fdno,"printed")) {
- print $out_fd $self->tag(),$self->fh($fdno,"name"),"\n";
+ print $out_fh $self->tag(),$self->fh($fdno,"name"),"\n";
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}}, $self->tag(),
$self->fh($fdno,"name"));
@@ -10494,19 +10629,23 @@ sub print_linebuffer($) {
my $tag = $self->tag();
unshift @$halfline_ref, $tag;
# TODO --recend that can be partially in @$halfline_ref
- substr($buf,0,$i-1) =~ s/(?<=[\n\r])(?=.|$)/$tag/gs;
+ substr($buf,0,$i-1) =~
+ s/(?<=[\n\r])(?=.|$)/$tag/gs;
# The length changed, so find the new ending pos
- $i = ::max((rindex($buf,"\n")+1), (rindex($buf,"\r")+1));
+ $i = ::max((rindex($buf,"\n")+1),
+ (rindex($buf,"\r")+1));
} else {
# Replace with freshly computed value of tag
unshift @$halfline_ref, $self->tag();
- substr($buf,0,$i-1) =~ s/(?<=[\n\r])(?=.|$)/$self->tag()/gse;
+ substr($buf,0,$i-1) =~
+ s/(?<=[\n\r])(?=.|$)/$self->tag()/gse;
# The length changed, so find the new ending pos
- $i = ::max((rindex($buf,"\n")+1), (rindex($buf,"\r")+1));
+ $i = ::max((rindex($buf,"\n")+1),
+ (rindex($buf,"\r")+1));
}
}
# Print the partial line (halfline) and the last half
- print $out_fd @$halfline_ref, substr($buf,0,$i);
+ print $out_fh @$halfline_ref, substr($buf,0,$i);
# Buffer in memory for SQL and CSV-output
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}},
@@ -10539,7 +10678,7 @@ sub print_linebuffer($) {
unshift @$halfline_ref, $self->tag();
}
# Print the partial line (halfline)
- print $out_fd @{$self->{'halfline'}{$fdno}};
+ print $out_fh @{$self->{'halfline'}{$fdno}};
# Buffer in memory for SQL and CSV-output
if($Global::membuffer) {
push(@{$self->{'output'}{$fdno}}, @$halfline_ref);
@@ -10562,15 +10701,11 @@ sub print_linebuffer($) {
}
}
-sub print_tag(@) {
- return print_normal(@_);
-}
-
sub free_ressources() {
my $self = shift;
if(not $opt::ungroup) {
my $fh;
- for my $fdno (sort { $a <=> $b } keys %Global::fd) {
+ for my $fdno (sort { $a <=> $b } keys %Global::fh) {
$fh = $self->fh($fdno,"w");
$fh and close $fh;
$fh = $self->fh($fdno,"r");
@@ -10579,9 +10714,54 @@ sub free_ressources() {
}
}
+sub print_parset($) {
+ # Wrap output with shell script code to set as variables
+ my $self = shift;
+ my ($fdno,$in_fh,$out_fh) = @_;
+ my $outputlength = 0;
+
+ ::debug("parset","print $Global::parset");
+ if($Global::parset eq "assoc") {
+ # eval "`echo 'declare -A myassoc; myassoc=(
+ # Each:
+ # [$'a\tb']=$'a\tb\tc ddd'
+ # End:
+ # )'`"
+ print '[',::Q($self->{'commandline'}->
+ replace_placeholders(["\257<\257>"],0,0)),']=';
+ } elsif($Global::parset eq "array") {
+ # eval "`echo 'myassoc=(
+ # Each:
+ # $'a\tb\tc ddd'
+ # End:
+ # )'`"
+ } elsif($Global::parset eq "var") {
+ # var=$'a\tb\tc ddd'
+ if(not @Global::parset_vars) {
+ ::error("Too few named destination variables");
+ ::wait_and_exit(255);
+ }
+ print shift @Global::parset_vars,"=";
+ }
+ local $/ = "\n";
+ my $tag = $self->tag();
+ my @out;
+ while(<$in_fh>) {
+ $outputlength += length $_;
+ # Tag lines with \r, too
+ $_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
+ push @out, $tag,$_;
+ }
+ # Remove last newline
+ # This often makes it easier to use the output in shell
+ @out and ${out[$#out]} =~ s/\n$//s;
+ print ::Q(join("", at out)),"\n";
+ return $outputlength;
+}
+
sub print_normal($) {
my $self = shift;
- my ($fdno,$in_fh,$out_fd) = @_;
+ my ($fdno,$in_fh,$out_fh) = @_;
my $buf;
close $self->fh($fdno,"w");
if($? and $opt::compress) {
@@ -10594,7 +10774,9 @@ sub print_normal($) {
my $outputlength = 0;
my @output;
- if($opt::tag or $opt::tagstring) {
+ if($Global::parset and $fdno == 1) {
+ $outputlength += $self->print_parset($fdno,$in_fh,$out_fh);
+ } elsif($opt::tag or $opt::tagstring) {
# Read line by line
local $/ = "\n";
my $tag = $self->tag();
@@ -10602,14 +10784,15 @@ sub print_normal($) {
$outputlength += length $_;
# Tag lines with \r, too
$_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
- print $out_fd $tag,$_;
+ print $out_fh $tag,$_;
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}}, $tag, $_;
}
}
} else {
+ # Most efficient way of copying data from $in_fh to $out_fh
while(sysread($in_fh,$buf,131072)) {
- print $out_fd $buf;
+ print $out_fh $buf;
$outputlength += length $buf;
if($Global::membuffer) {
push @{$self->{'output'}{$fdno}}, $buf;
@@ -10627,6 +10810,61 @@ sub print_normal($) {
}
}
+sub print_results($) {
+ my $self = shift;
+ my ($fdno,$in_fh,$out_fh) = @_;
+ my $buf;
+ close $self->fh($fdno,"w");
+ if($? and $opt::compress) {
+ ::error($opt::compress_program." failed.");
+ $self->set_exitstatus(255);
+ }
+ if(not $self->virgin()) {
+ seek $in_fh, 0, 0;
+ # $in_fh is now ready for reading at position 0
+ my $outputlength = 0;
+ my @output;
+
+ if($Global::membuffer) {
+ # Read data into membuffer
+ if($opt::tag or $opt::tagstring) {
+ # Read line by line
+ local $/ = "\n";
+ my $tag = $self->tag();
+ while(<$in_fh>) {
+ $outputlength += length $_;
+ # Tag lines with \r, too
+ $_ =~ s/(?<=[\r])(?=.|$)/$tag/gs;
+ push @{$self->{'output'}{$fdno}}, $tag, $_;
+ }
+ } else {
+ # Most efficient way of copying data from $in_fh to $out_fh
+ while(sysread($in_fh,$buf,131072)) {
+ $outputlength += length $buf;
+ push @{$self->{'output'}{$fdno}}, $buf;
+ }
+ }
+ } else {
+ # Not membuffer: No need to read the file
+ if($opt::compress) {
+ $outputlength = -1;
+ } else {
+ # Determine $outputlength = file length
+ seek($in_fh, 0, 2) || ::die_bug("cannot seek result");
+ $outputlength = tell($in_fh);
+ }
+ }
+ if($fdno == 1) {
+ $self->add_returnsize($outputlength);
+ }
+ close $in_fh;
+ if($? and $opt::compress) {
+ ::error($opt::decompress_program." failed.");
+ $self->set_exitstatus(255);
+ }
+ }
+}
+
sub print_joblog($) {
my $self = shift;
my $cmd;
@@ -10648,17 +10886,51 @@ sub print_joblog($) {
$self->set_job_in_joblog();
}
-sub tag($) {
- my $self = shift;
- if(not defined $self->{'tag'} or not $Global::cache_replacement_eval) {
- if($opt::tag or defined $opt::tagstring) {
- $self->{'tag'} = $self->{'commandline'}->
- replace_placeholders([$opt::tagstring],0,0)."\t";
- } else {
- $self->{'tag'} = "";
+{
+ my @color;
+
+ sub tag($) {
+ sub init_color() {
+ # color combinations that are readable: black/white text
+ # on colored background, but not white on yellow
+ @color =
+ # Force each color code to have the same length in chars
+ # This will make \t work as expected
+ ((map { [sprintf("%03d",$_),"000"] }
+ 6..7,9..11,13..15,40..51,75..87,113..123,147..159,
+ 171..231,249..254),
+ (map { [sprintf("%03d",$_),231] }
+ 1..9,12..13,16..45,52..81,88..116,124..151,153,
+ 160..180,182..185,187..189,196..214,232..252,
+ 255..254));
+ # reorder list so adjacent colors are dissimilar
+ # %7 and %17 were found experimentally
+ @color = @color[
+ sort { ($b%7 <=> $a%7) or ($a%17 <=> $b%17) } 0..$#color
+ ];
+ }
+ my $self = shift;
+ if(not defined $self->{'tag'} or not $Global::cache_replacement_eval) {
+ if($opt::tag or defined $opt::tagstring) {
+ if($Global::color) {
+ if(not @color) { init_color() }
+ # Choose a value based on the seq
+ my $col = @color[$self->seq() % ($#color+1)];
+ $self->{'tag'} = "\033[48;5;".$col->[0].
+ ";38;5;".$col->[1]."m".
+ ($self->{'commandline'}->
+ replace_placeholders([$opt::tagstring],0,0)).
+ "\033[00m\t";
+ } else {
+ $self->{'tag'} = $self->{'commandline'}->
+ replace_placeholders([$opt::tagstring],0,0)."\t";
+ }
+ } else {
+ $self->{'tag'} = "";
+ }
}
+ return $self->{'tag'};
}
- return $self->{'tag'};
}
sub hostgroups($) {
@@ -12658,21 +12930,27 @@ sub total_jobs() {
$Global::unquote_arg = 1;
}
sub yyyy_mm_dd_hh_mm_ss() {
+ # ISO8601 2038-01-19T03:14:08
::strftime("%Y-%m-%dT%H:%M:%S", localtime(time()));
}
sub yyyy_mm_dd_hh_mm() {
+ # ISO8601 2038-01-19T03:14
::strftime("%Y-%m-%dT%H:%M", localtime(time()));
}
sub yyyy_mm_dd() {
+ # ISO8601 2038-01-19
::strftime("%Y-%m-%d", localtime(time()));
}
sub yyyymmddhhmmss() {
+ # ISO8601 20380119031408
::strftime("%Y%m%d%H%M%S", localtime(time()));
}
sub yyyymmddhhmm() {
+ # ISO8601 203801190314
::strftime("%Y%m%d%H%M", localtime(time()));
}
sub yyyymmdd() {
+ # ISO8601 20380119
::strftime("%Y%m%d", localtime(time()));
}
@@ -13596,7 +13874,7 @@ sub main() {
save_stdin_stdout_stderr();
save_original_signal_handler();
parse_options();
- ::debug("init", "Open file descriptors: ", join(" ",keys %Global::fd), "\n");
+ ::debug("init", "Open file descriptors: ", join(" ",keys %Global::fh), "\n");
my $number_of_args;
if($Global::max_number_of_args) {
$number_of_args = $Global::max_number_of_args;
@@ -13742,3 +14020,4 @@ sub main() {
}
main();
+
=====================================
src/sem.pod
=====================================
@@ -265,7 +265,7 @@ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
-along with this program. If not, see <http://www.gnu.org/licenses/>.
+along with this program. If not, see <https://www.gnu.org/licenses/>.
=head2 Documentation license I
=====================================
src/sql
=====================================
@@ -600,7 +600,7 @@ $Global::Initfile && unlink $Global::Initfile;
exit ($err);
sub parse_options {
- $Global::version = 20210322;
+ $Global::version = 20210822;
$Global::progname = 'sql';
# This must be done first as this may exec myself
View it on GitLab: https://salsa.debian.org/med-team/parallel/-/commit/5101cc6ed80f548428f20202b87fb39582ae10da
--
View it on GitLab: https://salsa.debian.org/med-team/parallel/-/commit/5101cc6ed80f548428f20202b87fb39582ae10da
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20210824/e6f7e7ac/attachment-0001.htm>
More information about the debian-med-commit
mailing list