The at command schedules one-time tasks for future execution, but when these jobs fail to run, diagnosing the issue can be tricky. Unlike cron's recurring schedule, at jobs are fire-and-forget, making failures harder to detect.

Introduction

This article covers troubleshooting steps and solutions for How to Fix Linux at Job Failed Errors. The error typically occurs in production environments and can cause service disruptions if not addressed promptly.

Symptoms

Common error messages include:

bash
Cannot open lock file /var/spool/cron/atjobs/.SEQ: Permission denied
You do not have permission to use at.
Warning: commands will be executed using /bin/sh
Can't open /var/run/atd.pid to signal atd. No atd running?

```bash # Check atd service status systemctl status atd

# Check if process exists ps aux | grep atd pgrep -a atd

# Start atd if not running systemctl start atd systemctl enable atd

# If systemd unit doesn't exist atd ```

```bash # List pending jobs for current user atq

# List all jobs (as root) sudo atq

# View job details at -c job_number

# View job with less paging at -c job_number | less

# Check job files in spool directory ls -la /var/spool/cron/atjobs/ ls -la /var/spool/at/jobs/ # Alternative location ```

Common Causes

  • Configuration misconfiguration
  • Missing or incorrect credentials
  • Network connectivity issues
  • Version compatibility problems
  • Resource exhaustion or limits
  • Permission or access denied

Step-by-Step Fix

  1. 1.Check logs for specific error messages
  2. 2.Verify configuration settings
  3. 3.Test network connectivity
  4. 4.Review recent changes
  5. 5.Apply corrective action
  6. 6.Verify the fix

Understanding the at Command

The at daemon (atd) manages scheduled one-time jobs. Jobs are stored in a queue and executed at the specified time. Unlike cron, at jobs are removed after execution.

Typical Error Messages and Symptoms

bash
Cannot open lock file /var/spool/cron/atjobs/.SEQ: Permission denied
You do not have permission to use at.
Warning: commands will be executed using /bin/sh
Can't open /var/run/atd.pid to signal atd. No atd running?

Common symptoms: - Jobs don't execute at scheduled time - Jobs disappear without running - Permission denied when scheduling - at daemon not running

Diagnosing at Job Issues

Step 1: Check if at Daemon is Running

```bash # Check atd service status systemctl status atd

# Check if process exists ps aux | grep atd pgrep -a atd

# Start atd if not running systemctl start atd systemctl enable atd

# If systemd unit doesn't exist atd ```

Step 2: Check at Job Queue

```bash # List pending jobs for current user atq

# List all jobs (as root) sudo atq

# View job details at -c job_number

# View job with less paging at -c job_number | less

# Check job files in spool directory ls -la /var/spool/cron/atjobs/ ls -la /var/spool/at/jobs/ # Alternative location ```

Step 3: Check Permissions

```bash # Check at.allow and at.deny files cat /etc/at.allow # If exists, only these users can use at cat /etc/at.deny # If exists, these users cannot use at

# If neither exists, typically all users can use at # Check directory permissions ls -la /var/spool/cron/atjobs/ ls -la /var/spool/at/

# Directory should be owned by daemon or root # Files should be owned by user who created them ```

Step 4: Check Logs

```bash # Check system logs for at daemon messages journalctl -u atd -n 50

# Check syslog for at messages grep -i "at[" /var/log/syslog grep -i "atd" /var/log/messages

# Check mail for at job output (default behavior) mail # Or check user's mail file cat /var/mail/$USER ```

Common Issues and Solutions

Issue 1: at Daemon Not Running

```bash # Check if atd is installed which atd dpkg -l | grep at # Debian/Ubuntu rpm -qa | grep at # RHEL/CentOS

# Install if missing sudo apt-get install at # Debian/Ubuntu sudo yum install at # RHEL/CentOS

# Start the service sudo systemctl start atd sudo systemctl enable atd

# Verify it's running systemctl status atd pgrep -a atd ```

Issue 2: Permission Denied

```bash # Check if user is allowed cat /etc/at.allow cat /etc/at.deny

# If at.allow exists, add your user echo "$USER" | sudo tee -a /etc/at.allow

# If at.deny exists and user is in it, remove them sudo sed -i "/^$USER$/d" /etc/at.deny

# Check spool directory permissions ls -la /var/spool/cron/atjobs/

# Fix permissions if needed sudo chmod 1770 /var/spool/cron/atjobs/ sudo chown daemon:daemon /var/spool/cron/atjobs/ # Or root:root depending on distro ```

Issue 3: Jobs Not Executing

```bash # Verify job exists in queue atq

# Check job content at -c 1 # Where 1 is the job number

# Check job execution time atq -q a # Show jobs in queue 'a'

# Ensure atd is running systemctl status atd

# Check if job already executed grep "at[" /var/log/syslog | tail -20

# Job output is mailed by default, check mail cat /var/mail/$USER

# Or redirect output when creating job echo "/path/to/script.sh > /tmp/at-output.log 2>&1" | at now + 5 minutes ```

Issue 4: Environment Variables Missing

Like cron, at jobs run with a limited environment.

```bash # When creating an at job, source your environment at now + 5 minutes << 'EOF' source /home/user/.bashrc /path/to/script.sh EOF

# Or use full paths echo "/usr/bin/python3 /home/user/script.py" | at 14:30

# Check what environment the job sees at now + 1 minute << 'EOF' env > /tmp/at-env.txt EOF # Check output after job runs cat /tmp/at-env.txt ```

Issue 5: Script Works Manually But Not in at

```bash # Use the -f flag to run a script file at -f /path/to/script.sh now + 5 minutes

# Test with a simple command first echo "date > /tmp/at-test.txt" | at now + 1 minute

# Check if it ran cat /tmp/at-test.txt

# If using shell-specific features, specify the shell # at uses /bin/sh by default echo "/bin/bash /path/to/script.sh" | at 15:00

# Or change SHELL for the job SHELL=/bin/bash at now + 5 minutes << 'EOF' your bash-specific commands here EOF ```

Issue 6: Job Queue Issues

```bash # at uses different queues (a-z, A-Z) # Lowercase = batch jobs (nice value) # Uppercase = special treatment

# Check job in specific queue atq -q a

# Submit job to specific queue echo "/path/to/script.sh" | at -q b now + 1 hour

# Batch queue runs when system load is low batch << 'EOF' /path/to/cpu-intensive-script.sh EOF

# Check batch queue atq -q = # Batch queue is '=' ```

Issue 7: Wrong Execution Time

```bash # at uses 24-hour format by default at 14:30 # 2:30 PM at 2:30 PM # Also 2:30 PM (if AM/PM is supported) at 1430 # Also 2:30 PM

# Common time specifications at now # Immediately at now + 5 minutes at now + 1 hour at now + 2 days at 14:30 tomorrow at 14:30 next week at 14:30 Feb 28 at 14:30 2/28 at teatime # 4:00 PM (some systems) at midnight # 12:00 AM at noon # 12:00 PM

# Check system timezone timedatectl date

# at uses system timezone # To verify job time atq # Shows: 1 Fri Feb 16 14:30:00 2024 a username ```

Issue 8: Output Lost

By default, at jobs mail their output to the user.

```bash # Redirect output to a file at now + 5 minutes << 'EOF' /path/to/script.sh > /tmp/job-output.log 2>&1 EOF

# Capture both stdout and stderr echo "/path/to/script.sh >> /var/log/at-job.log 2>&1" | at now + 10 minutes

# Check mail configuration which mail dpkg -l mailutils # Debian/Ubuntu

# If mail isn't configured, redirect to file # Jobs will fail silently if output can't be mailed ```

Testing and Debugging

Test at Functionality

```bash # Simple test job echo "echo 'at works at $(date)' > /tmp/at-test-\$(date +%s).txt" | at now + 1 minute

# Wait and check sleep 70 ls -la /tmp/at-test-*.txt cat /tmp/at-test-*.txt

# Test with a more complex command at now + 2 minutes << 'EOF' echo "Environment:" env > /tmp/at-environment.txt echo "Current directory: $(pwd)" >> /tmp/at-environment.txt echo "User: $(whoami)" >> /tmp/at-environment.txt echo "Path: $PATH" >> /tmp/at-environment.txt EOF

# Check output cat /tmp/at-environment.txt ```

Monitor at Jobs

```bash # Watch at queue watch -n 1 atq

# Follow at logs journalctl -u atd -f

# Create monitoring alias alias atwatch='watch -n 1 "atq && echo && sudo tail -5 /var/log/syslog | grep at"' ```

Removing and Managing Jobs

```bash # Remove a job atrm 2 # Remove job number 2

# Remove multiple jobs atrm 3 4 5

# Remove all your jobs atrm $(atq | awk '{print $1}')

# Show job content before removing at -c 2 atrm 2 ```

at vs cron: When to Use Each

Featureatcron
FrequencyOne-timeRecurring
ScheduleFuture timePeriodic pattern
Job storageRemoved after runKept indefinitely
Use caseDelayed executionRegular maintenance

Use at for: - Running a backup after hours - Sending a reminder - Executing a long job during off-peak - Any one-time delayed task

Use cron for: - Daily backups - Hourly log rotation - Weekly reports - Any recurring task

Verification

After fixing issues:

```bash # Verify atd is running systemctl status atd

# Test a simple job echo "date > /tmp/at-verify.txt" | at now + 1 minute

# Wait and check sleep 70 cat /tmp/at-verify.txt

# Check job queue is being processed atq

# Check logs journalctl -u atd -n 10 ```

Prevention

```bash # 1. Always redirect output echo "/path/to/script.sh >> /var/log/at.log 2>&1" | at now + 1 hour

# 2. Use full paths echo "/usr/bin/rsync -av /src/ /dest/" | at midnight

# 3. Use here-documents for complex commands at now + 1 hour << 'EOF' source ~/.bashrc cd /home/user/project ./run-backup.sh >> /var/log/backup.log 2>&1 EOF

# 4. Verify job was created echo "Job created at $(date)" echo "sleep 300 && /path/to/script.sh" | at now + 5 minutes atq

# 5. Document why the job was created echo "# Post-deployment cleanup" | at now + 30 minutes

# 6. Use batch for CPU-intensive jobs batch << 'EOF' /usr/bin/nice -n 19 /path/to/cpu-intensive-job.sh EOF ```

  • [Fix Ceph Cluster Not Healthy](fix-ceph-cluster-not-healthy)
  • [Fix Corosync Cluster Not Forming](fix-corosync-cluster-not-forming)
  • [Fix GlusterFS Volume Not Mounting](fix-glusterfs-volume-not-mounting)
  • [Fix Fix Linux Acpi Event Not Handled Issue in Linux](fix-linux-acpi-event-not-handled)
  • [Fix Fix Linux Anacron Missed Jobs Issue in Linux](fix-linux-anacron-missed-jobs)

<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "How to Fix Linux at Job Failed Errors", "description": "Learn how to fix Linux at job failures. Diagnose why one-time scheduled tasks fail, check at daemon status, and resolve configuration issues.", "url": "https://www.fixwikihub.com/fix-linux-at-job-failed", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2025-11-24T16:36:46.596Z", "dateModified": "2025-11-24T16:36:46.596Z" } </script>