pgrep -f Matches Itself: Why Your "Wait for the Job" Loop Never Ends (and the PID-File Fix)

A common way to wait for a background job in a shell script is to poll for it by name:

while pgrep -f "train.py" > /dev/null; do sleep 10; done
echo "training finished"

It looks right, and in an interactive terminal it usually works. But I've had this loop never finish twice on my home server, long after the job had exited. Both times the cause was the same: pgrep -f was finding a process that wasn't the job.

Reproducing it

This script starts a fake job (a 3-second sleep renamed to look like python train.py), then waits for it with the loop above, run through bash -c, the way cron, ssh, nohup and many tools start commands:

#!/bin/bash
# Reproduce: a waiter started with `bash -c "..."` matches its own command line.
# A fake "training job" that runs for 3 seconds:
bash -c 'exec -a "python train.py" sleep 3' &

# The waiter, started the way cron / ssh / nohup / systemd often start things:
timeout 10 bash -c '
  start=$SECONDS
  echo "pgrep -af train.py sees:"; pgrep -af "train.py"
  while pgrep -f "train.py" > /dev/null; do sleep 1; done
  echo "job finished after $((SECONDS - start)) s"
' || echo "waiter gave up after 10 s; train.py was gone after 3 s"

wait

Output:

pgrep -af train.py sees:
2808915 python train.py 3
2808916 timeout 10 bash -c    start=$SECONDS   echo "pgrep -af train.py sees:"; pgrep -af "train.py"   while pgrep -f "train.py" > /dev/null; do sleep 1; done   echo "job finished after $((SECONDS - start)) s" 
2808917 bash -c    start=$SECONDS   echo "pgrep -af train.py sees:"; pgrep -af "train.py"   while pgrep -f "train.py" > /dev/null; do sleep 1; done   echo "job finished after $((SECONDS - start)) s" 
waiter gave up after 10 s; train.py was gone after 3 s

The job was gone after 3 seconds, but the loop only stopped because timeout killed it at 10. Look at what pgrep -f matched: the job, and also the waiting shell itself and the timeout wrapper. Their command lines contain the loop's source code, and the source code contains the string train.py. As long as the waiter is running, it finds itself, so it runs forever.

Why -f does this

Without -f, pgrep matches only the process name (like python or sleep). With -f, it matches against the full command line of every process. pgrep is careful to skip its own process, but it has no idea which other processes are "related" to you. Anything whose command line contains your pattern counts:

• the shell running your loop, when it was started with bash -c '…train.py…'
• wrappers like timeout, nohup, sudo, ssh host '…' or watch
• an editor or less with a file open named train.py
• someone else's process with the same name. One of my hangs was a different user account on the same machine running a script with the same name. pgrep checks all users unless you pass -u.

If you run the loop from a script file (bash wait.sh), the shell's command line is just bash wait.sh and the bug disappears. That's why it works when you test it and hangs when cron or another tool runs it.

The fix: wait on the PID, not the name

You already know which process you started. Save its PID at launch and wait on that PID:

#!/bin/bash
# Fix: record the PID when you start the job, then wait on that PID.
bash -c 'exec -a "python train.py" sleep 3' &
echo $! > /tmp/train.pid

start=$SECONDS
pid=$(cat /tmp/train.pid)
while kill -0 "$pid" 2>/dev/null; do sleep 1; done
echo "job finished after $((SECONDS - start)) s"
rm -f /tmp/train.pid

job finished after 3 s

kill -0 sends no signal; it just checks whether the PID exists. Writing the PID to a file means a completely separate script (or a later SSH session) can wait on the job too. This is what my background-job helper does now: start writes a PID file, wait polls it.

If the waiter is the same shell that started the job, it's even simpler: job & pid=$!, then wait "$pid".

If you really have to match by name

Sometimes you didn't start the process and don't have its PID. Then make the match as narrow as possible:

• Limit to your user: pgrep -u "$(id -u)" -f … rules out other accounts.
• Anchor the pattern: pgrep -f '^python train\.py' only matches command lines that start with the job's command, so a bash -c '…' wrapper won't match.
• Use the bracket trick: a pattern like '[t]rain\.py' still matches train.py, but your own command line contains [t]rain, which doesn't match. It only hides your own command line, though: while testing this post, the bracket pattern still matched my terminal's launcher process, whose command line contained the literal text python train.py.
• Look before you trust it: pgrep -af pattern prints the full command lines, which is exactly how I found both of my hangs.

Summary

• pgrep -f searches every process's full command line, including the shell running your loop.
• When the waiter is started with bash -c, timeout, ssh and similar tools, it matches itself and never exits.
• Save the PID when you start a job and wait with kill -0 or wait.
• If you must match by name: -u, an anchored pattern, and check with pgrep -af first.