Problem
This is what my bash script looks like:
#!/bin/bash
wget LINK1 >/dev/null 2>&1
wget LINK2 >/dev/null 2>&1
wget LINK3 >/dev/null 2>&1
wget LINK4 >/dev/null 2>&1
# ..
# ..
wget LINK4000 >/dev/null 2>&1
However, processing each line until the command is finished and then moving on to the next one takes a long time; I’d like to process, say, 20 lines at a time and then process another 20 lines when they’re done.
I considered using wget LINK1 >/dev/null 2>&1 & to send the command to the background and continue, but with 4000 lines here, I’ll have speed issues, not to mention a limit on how many processes I can start at once, so this isn’t a smart idea.
One approach I’m considering right now is to check whether one of the instructions is still active; for example, after 20 lines, I could add the following loop:
while [ $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ]; do
sleep 1
done
Of course, I’ll have to append & to the end of the line in this situation! But I have a feeling this isn’t the best way to go about it.
So, how do I group each 20 lines together and wait for them to finish before moving on to the next 20 lines? This script is dynamically generated, so I can do whatever math I want while it’s being generated, but it DOES NOT HAVE TO USE wget; it was just an example, so any wget-specific solution will not help me.
Asked by AL-Kateb
Solution #1
Make use of the built-in wait timer:
process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait
In the example above, four processes (process1 through process4) would be initiated in the background, and the shell would wait for them to finish before moving on to the next set.
The following is taken from the GNU manual:
Answered by devnull
Solution #2
See the similarity. It has a similar syntax to xargs, except it executes instructions in parallel.
Answered by choroba
Solution #3
In fact, xargs may perform multiple commands at the same time for you. For this, there is a command-line option called -P max procs. See the xargs man page for more information.
Answered by Vader B
Solution #4
You can use the command to run 20 processes:
wait
When all of your background jobs are completed, your script will wait and proceed.
Answered by Binpix
Post is based on https://stackoverflow.com/questions/19543139/bash-script-processing-limited-number-of-commands-in-parallel