systemd "start-pre operation timed out": Why Your ExecStartPre Wait Loop Dies After 90 Seconds

A common way to make a service wait for the network is a small retry loop in ExecStartPre: try to reach the server, sleep, try again. I use one in the systemd user service that keeps Claude Code Remote Control running 24/7 on my GX10, because After=network-online.target does nothing in a user unit.

It worked fine, until a reboot where the network came up slowly and unreliably. Then systemd itself started killing my wait loop: "start-pre operation timed out. Terminating." This post covers why that happens and the two fixes I made. The second one I only found while writing the first one.

The symptom

After the machine rebooted, the connection to the API kept dropping in and out for a while. The service never got past its pre-start step. The journal showed the same pair of lines over and over, about every 100 seconds:

$ journalctl --user -u claude-remote-hub.service | grep -E "timed out|result"
... 12:46:33 systemd[...]: claude-remote-hub.service: start-pre operation timed out. Terminating.
... 12:46:33 systemd[...]: claude-remote-hub.service: Failed with result 'timeout'.
... 12:48:14 systemd[...]: claude-remote-hub.service: start-pre operation timed out. Terminating.
... 12:48:14 systemd[...]: claude-remote-hub.service: Failed with result 'timeout'.
... 12:49:54 systemd[...]: claude-remote-hub.service: start-pre operation timed out. Terminating.

$ systemctl --user show -p DefaultTimeoutStartUSec
DefaultTimeoutStartUSec=1min 30s

That 100-second rhythm is the clue: 90 seconds of start timeout plus the 10 seconds of RestartSec. And DefaultTimeoutStartUSec=1min 30s confirms it.

Why: ExecStartPre counts toward TimeoutStartSec

systemd gives a service a fixed amount of time to start, set by TimeoutStartSec=. If you don't set it, it defaults to 90 seconds. The important detail is that this time includes every ExecStartPre= command, not only the main process. So a loop that is designed to wait "up to 5 minutes" gets killed after 90 seconds, every time.

You don't notice when the network is fine, because the first try succeeds in a fraction of a second. You only notice on the boots where you actually needed the loop.

Restart=always did retry after each timeout, so the service would eventually have started once the network settled. But each retry restarted the wait from zero and got cut off at 90 seconds again, so the "wait up to 5 minutes" behavior never actually happened.

Fix 1: raise TimeoutStartSec

The obvious fix: give the start phase more time than the loop needs. My loop was "up to 5 minutes", so I set TimeoutStartSec=360 (6 minutes).

Fix 2: a "60 tries" loop is not a "5 minute" loop

While checking the numbers for Fix 1, I noticed my loop was never really 5 minutes. It counts attempts, not time:

# The old wait loop: "60 tries" is not "5 minutes"
ExecStartPre=/bin/sh -c 'for i in $(seq 1 60); do curl -s -o /dev/null --max-time 5 https://api.anthropic.com && exit 0; sleep 5; done; exit 0'

# network down, fails instantly:   60 x (0 s + 5 s sleep) = about 5 minutes
# network dropping packets:        60 x (5 s timeout + 5 s sleep) = about 10 minutes

When the network is completely down, curl fails instantly and the loop takes about 5 minutes. But in the situation I actually had, a flaky network where packets get lost, each curl waits the full --max-time 5 before giving up. Now every attempt takes 10 seconds and the loop can run for 10 minutes. A 6-minute TimeoutStartSec would still kill it.

The fix is to put a wall-clock deadline on the loop, so its maximum length no longer depends on how each attempt fails:

[Service]
# Wait for the network, but stop after 300 seconds of wall-clock time, however slow each try is.
# In unit files, write $$ for a shell $ and %% for a literal % (systemd uses both itself).
ExecStartPre=/bin/sh -c 'end=$$(( $$(date +%%s) + 300 )); while [ "$$(date +%%s)" -lt "$$end" ]; do curl -s -o /dev/null --max-time 5 https://api.anthropic.com && exit 0; sleep 5; done; exit 0'
ExecStart=...
# ExecStartPre counts toward the start timeout. Give it the 300 s wait plus headroom.
TimeoutStartSec=360
Restart=always
RestartSec=10

Two details that are easy to get wrong:

  • Escape $ and %. systemd expands $NAME environment variables and % specifiers (like %h) in Exec lines before the shell ever sees them. Write $$ to pass a literal $ to the shell and %% for a literal %. Without that, $end would be replaced with an empty string by systemd, and %s in date +%s would be treated as a systemd specifier.
  • Keep exit 0 at the end. If the network never comes up, the main process still gets a chance to start, and Restart=always handles the rest. If the pre-start step itself fails, the whole start counts as failed and the main command never runs.

Testing it without waiting 5 minutes

I tested the exact loop in a throwaway user unit with a 20-second deadline and a non-routable address (10.255.255.1), which makes curl wait the full 5 seconds, just like lost packets. It gave up after 20 seconds (two attempts), printed its message and then ran the main command, so the escaping and the deadline both work:

# ~/.config/systemd/user/zz-test-prewait.service  (20 s deadline, unreachable address)
[Service]
Type=oneshot
ExecStartPre=/bin/sh -c 'end=$$(( $$(date +%%s) + 20 )); while [ "$$(date +%%s)" -lt "$$end" ]; do curl -s -o /dev/null --max-time 5 http://10.255.255.1 && exit 0; sleep 5; done; echo "gave up waiting"; exit 0'
ExecStart=/bin/echo started
TimeoutStartSec=60

$ systemctl --user daemon-reload
$ time systemctl --user start zz-test-prewait.service      # took 20 s
$ journalctl --user -u zz-test-prewait.service -o cat
gave up waiting
started

Delete the test unit and run systemctl --user daemon-reload again afterwards. On the real services I only ran daemon-reload, without restarting them. The new loop takes effect at the next start, and the sessions that were running weren't interrupted.

Summary

  • ExecStartPre runs inside the start timeout. The default is 90 seconds, so any wait loop longer than that needs TimeoutStartSec=.
  • The error to search for is "start-pre operation timed out. Terminating." followed by "Failed with result 'timeout'".
  • Bound a wait loop by time, not by attempt count. Each attempt can take as long as its own timeout.
  • In unit files, write $$ and %% for shell variables and literal percent signs.
  • Check your defaults with systemctl --user show -p DefaultTimeoutStartUSec (or without --user for system services).

Comments

Post a Comment

Popular posts from this blog

Gear Ratio Calculator: Multi-Stage Ratio, Output Speed, Torque and Direction

When you design a gear train, the questions are always the same: what's the overall ratio, how fast does the output turn, how much torque do you get, and which way does it spin? Enter the tooth counts for up to three stages, plus your motor's speed and torque. Stage 1 driver teeth Stage 1 driven teeth Stage 2 driver teeth (optional) Stage 2 driven teeth Stage 3 driver teeth (optional) Stage 3 driven teeth Input speed (rpm) Input torque (kgf·cm) Efficiency per stage (%) Calculate Enter values and press Calculate. The formulas Each stage's ratio is driven teeth ÷ driver teeth . In a compound train (two gears fixed on the same shaft), the overall ratio is the product of the stages: Ratio = (Z₂ ÷ Z₁) × (Z₄ ÷ Z₃) × … Output speed = input speed ÷ ratio Output torque = input torque × ratio × overall efficiency (overall efficiency = per-stage efficiency raised to the number of stages) With the default values, 10 teeth drive 40 (4 : 1), then a 12-tooth gear on ...
Read more

Mecanum-Wheel Pick-and-Place Robot: From Arduino UNO to Mega2560 + ESP32-CAM

🌏 繁體中文版：[DIY專案] 麥克納姆輪取物機器人：從 UNO 初版到 Mega2560＋ESP32-CAM This mechatronics course project is a mecanum-wheel car that can drive in any direction and pick things up with a small robotic arm. This post covers the mechanical and hardware design, and how the car evolved from the first Arduino UNO version to an upgraded Mega2560 + ESP32-CAM build. Upgraded version: mecanum-wheel chassis with a 4-axis robotic arm Mecanum wheels: how a car can slide sideways A mecanum wheel has a ring of small rollers around its rim, each mounted at 45°. When the wheel spins, it pushes diagonally rather than straight ahead. By combining the direction and speed of all four wheels, those diagonal forces either cancel or add up, so the chassis can drive forward and back, strafe left and right, move diagonally, or spin in place, all without turning first. Rollers come in left-handed and right-handed versions, and the four wheels must be installed in mirrored pairs. So in Inventor, both the rollers and the wh...
Read more

Voltage Divider Calculator: Output Voltage, or the Best E24 Resistor Pair for a Target Voltage

A voltage divider uses two resistors to scale a voltage down: reading a battery voltage with an ADC, dropping a 5 V signal to 3.3 V, or making a reference voltage. This calculator works both ways: known resistors → output voltage , or target voltage → the closest pairs of standard E24 resistors . Mode Known R1 and R2 → output voltage Target output voltage → pick resistors Input voltage Vin (V) R1 (top, kΩ) R2 (bottom, kΩ) Target output Vout (V) Calculate Enter values and press Calculate. The formula R1 sits between the input and the output, and R2 between the output and ground: Vout = Vin × R2 ÷ (R1 + R2) Example: Vin = 5 V, R1 = 10 kΩ, R2 = 20 kΩ gives Vout = 5 × 20 ÷ 30 ≈ 3.33 V, with about 0.17 mA flowing through the divider all the time. In "pick resistors" mode, the calculator tries every E24 value for R2, works out the R1 that would be needed, rounds it to the nearest E24 value, and lists the three pairs with the smallest output error. Pairs that give t...
Read more