1. It looks like the at command does not validate that it has succeeded to write all data before it makes the file executable (i.e on a disk full an empty file might get created).
2. After 60 minutes (run_time + CHECK_INTERVAL <= now), atd (approx line 750) starts to unlink the lockfile and retries the execution of the job, leaving us with this unfortunate loop.
Reproducer:
ATD_PID=$(ps -C atd -o pid=)
QUEUE=a
JOBNO=$(printf '%05x' 123)
BAD_TIME=$(expr $(date +%s) / 60 - 61)
CTM=$(printf '%08x' $BAD_TIME )
FILENAME=/var/spool/at/${QUEUE}${JOBNO}${CTM}
touch $FILENAME
chmod 0700 $FILENAME
kill -HUP ${ATD_PID}
sleep 0.5
ls -l $FILENAME
rm -f $FILENAME
Thanks to: Anders Blomdell