The checks failed, but the AI saw exit code 0. It came from tail

Posted on Nov 5 • Originally published at zenn.dev

Disclosure (#ABotWroteThis): this post was written by an AI (Claude) for a solo developer in Japan, from the incident notes the developer and the AI kept during the work. "I" below is the developer. It is adapted from a Japanese article on Zenn.

My iPhone app has a check script, run.sh. The checks are grouped into sections. Each section prints OK or NG (the script's marker for a failed check) as it goes, and at the end the script prints a line counting how many sections ran, something like === Sections run: N ===. If any check reports NG, it exits 1.

One day in September 2026, Claude Code ran it like this:

./run.sh 2>&1 | tail -60 Enter fullscreen mode Exit fullscreen mode My notes from that day say it read the last lines, saw the line saying 69 sections had run, saw exit code 0, and reported "no NG." run.sh itself had exited 1. There were two NGs. The 0 belonged to tail. Both were known failures I had already planned to fix, so no harm was done that time, but the report was still wrong.

That afternoon it happened again. I was away from my desk, and Claude Code was working through a task list on its own, one session after another. It ran the checks with | tail -80, saw at the bottom that 80 sections had run, saw exit code 0, and took that as green. (Checks were being added all day, so the count had grown since the first run.

That it matched the 80 in tail -80 was a coincidence.) When the checks were run again with the script's own exit code printed ( EXIT=1 ), the run had failed, with an NG in section 44. That section checks the values the app sends in its requests to an AI API against a list of allowed values. Code that Claude Code itself had added earlier that day, during the same unattended work, was sending values that weren't on the list.

Unlike the two NGs from the first run, this failure was new.

The script already had a comment in it, there since August, saying (in Japanese): "If you pipe this, the exit code becomes tail's, so write it to a file first." But the comment sat above one step inside the script, a color contrast check, and was about how the script runs that step. It wasn't a rule for how to run the script itself.

false | tail -1; echo $? Enter fullscreen mode Exit fullscreen mode It prints 0. A pipeline's exit status is the status of the last command in it, and tail succeeded. The exit code that Claude Code's Bash tool reports is that last one too. The failure is still recorded; you just have to ask for it:

false | tail -1; echo " ${ PIPESTATUS } " # bash: 1 false | tail -1; echo " ${ pipestatus } " # zsh: 1 Enter fullscreen mode Exit fullscreen mode Read it right after the pipe, before you run anything else. The variable name and where the numbering starts differ between the two shells, and the wrong one prints an empty line instead of an error. Despite its name, Claude Code's Bash tool runs in your usual shell (zsh by default on a Mac). Run ps -p $$ -o comm= inside the tool to see which one.

Or turn on pipefail before the pipe, which works in both shells:

set -o pipefail; false | tail -1; echo $? # 1 Enter fullscreen mode Exit fullscreen mode Claude Code's Bash tool doesn't carry shell state from one call to the next, so put set -o pipefail in the same command as the pipe.

(This section isn't from the incident notes. The AI writing this post added it after reading the GitHub Docs page linked below.)

GitHub Actions makes this easy to miss. According to the GitHub Docs workflow syntax page, a run: step on Linux or macOS with no shell: set (not on the step, and not through defaults.run.shell ) runs as:

bash -e {0} Enter fullscreen mode Exit fullscreen mode Write shell: bash explicitly, and it runs as:

bash --noprofile --norc -eo pipefail {0} Enter fullscreen mode Exit fullscreen mode The docs point it out themselves: the unspecified default "runs a different command to when bash is specified explicitly." So./run.sh 2>&1 | tail -60 fails the step if the workflow sets shell: bash, and passes if no shell is set anywhere. Same command, same failing checks. ( -e alone doesn't catch it: run false | tail -1 under bash -e, and the shell keeps going, because the pipeline as a whole succeeded.) If your local runs and your agent's runs don't use pipefail, they behave like the no- shell: case, and the failure never shows up in the exit code.

Don't pipe the checks into anything. Write the output to a file and print the exit code right there. Tell me the result from the exit code and the NG lines in the log, not from the last few lines.

./run.sh > /tmp/run.log 2>&1; echo "EXIT= $? " grep -nw NG /tmp/run.log Enter fullscreen mode Exit fullscreen mode EXIT=0 and no output from grep means it passed. grep prints each NG line with its line number, so you also see where in the log it failed. -w matches NG only as a whole word, so it skips words like WARNING that happen to contain it. If your checks glue NG onto another word ( checkNG ), -w misses it. Drop the -w then, and expect a few extra matches like WARNING. Change NG to whatever your checks print on failure.

Read the result from the EXIT= line. If you run both lines in a single call to Claude Code's Bash tool, the exit code the tool reports is grep 's. When there are no NGs, that is 1: everything passed, but the tool shows a failure. When there are NGs, it is 0, and the tool shows success. Don't read the last command's exit code as the result here either.

pipefail doesn't solve the whole problem, though. It fixes the exit code, but tail -60 still hides any NG printed above the last 60 lines, and the count at the bottom of my script only says how many sections ran, not whether they passed.

When an AI tells me checks passed, I ask two things: