Home / Ansible / WordPress troubleshooting: Ansible Build Output Path Changed but th

Ansible

WordPress troubleshooting: Ansible Build Output Path Changed but th

Fix Ansible Tower/AWX project builds where runtime serves stale artifacts after output path changes, caused by cached symlinks, job output references, and stale project paths.

Published: Feb 12, 20269 min readBy FixWikiHub Editorial Team

Abstract illustration for a troubleshooting knowledge base category.

Introduction

When you modify the project build output path in Ansible Tower or AWX (changing where projects are stored, migrating to a new storage backend, or updating the PROJECTS_ROOT configuration), subsequent playbook runs may continue using artifacts from the old path. This occurs because Tower maintains symbolic links from job records to project versions, and those links reference the old location even after the project is rebuilt at the new path.

The issue manifests when you update storage configuration, migrate from local filesystem to object storage, or reorganize project directories. New jobs execute with stale playbook content, inventory files, or roles that no longer match the current repository state, leading to unexpected behavior and deployment failures.

Symptoms

Tower executes playbooks from the old project path despite a successful project update:

```bash # Project update shows success JOB 12345: Project Update - successful Updating project from https://github.com/org/ansible-playbooks... Version: main (abc123def456) Successfully updated to /var/lib/awx/projects/production/playbooks_main_abc123

# But job output shows it used old path TASK [Include deployment role] ********** fatal: [web-server-01]: FAILED! => { "msg": "the file /var/lib/awx/projects/production/playbooks_main_OLDVERSION/roles/deploy/tasks/main.yml does not exist" } ```

The project directory shows both old and new versions:

bash

$ ls -la /var/lib/awx/projects/production/
total 16
drwxr-xr-x 4 awx awx 4096 Apr 11 14:51 .
drwxr-xr-x 6 awx awx 4096 Apr 11 14:00 ..
lrwxrwxrwx 1 awx awx   42 Apr 11 10:22 playbooks -> playbooks_main_oldversion
drwxr-xr-x 4 awx awx 4096 Apr 11 14:51 playbooks_main_abc123def
drwxr-xr-x 4 awx awx 4096 Apr 11 10:22 playbooks_main_oldversion

Tower API shows job referencing old project version:

```bash $ curl -s -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/jobs/12346/" | jq '.project, .scm_revision' "https://tower.example.com/api/v2/projects/15/" "oldversion123" # Should be abc123def

$ curl -s -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/projects/15/" | jq '.local_path' "playbooks_main_oldversion" # Should point to new version ```

Job logs show playbook content that doesn't match repository:

```yaml # Job output shows deprecated playbook content TASK [Deploy application] *********** fatal: [web-server-01]: FAILED! => {"msg": "Parameters 'legacy_mode' is deprecated"}

# But the repository has been updated to remove legacy_mode $ git show main:playbooks/deploy.yml | grep -A5 "Deploy application" - name: Deploy application module_name: param: value # No legacy_mode parameter ```

Common Causes

1. Symlink Not Updated on Path Change

Tower creates a symlink pointing to the current project version. When the output path changes, the symlink may not be updated:

```bash # Old configuration PROJECTS_ROOT=/var/lib/awx/projects

# New configuration PROJECTS_ROOT=/mnt/shared/awx/projects

# Symlink still points to old location ls -la /var/lib/awx/projects/production/playbooks lrwxrwxrwx 1 awx awx 42 Apr 11 10:22 playbooks -> playbooks_main_oldversion ```

2. Job Record References Old Project Version

When a job is created, it captures the current project version. If the project is updated after job creation but before execution:

```python # Tower internal (simplified) class Job: project_version = project.current_version # Captured at job creation

def run(self): # Uses project_version even if project was updated playbook_path = f"{PROJECTS_ROOT}/{self.project_version}/playbook.yml" ```

3. Cached Project Version in Tower Database

Tower caches the project version in the database and doesn't immediately update on SCM changes:

```sql SELECT id, name, local_path, scm_revision FROM main_project WHERE id = 15;

4. Kubernetes/AWX PVC Mount Stale Data

In AWX deployments on Kubernetes, Persistent Volume Claims may cache directory contents:

```yaml # AWX deployment with PVC volumes: - name: projects-volume persistentVolumeClaim: claimName: awx-projects-claim

# If the PVC backend changes, pods may still see old data # until they are restarted ```

5. Multiple Tower Nodes with Inconsistent State

In a clustered Tower deployment, different nodes may have different project versions:

```bash # Node 1 $ ls /var/lib/awx/projects/production/ playbooks_main_abc123

# Node 2 $ ls /var/lib/awx/projects/production/ playbooks_main_oldversion ```

Step-by-Step Fix

Step 1: Diagnose the Path Discrepancy

Identify where the mismatch exists:

```bash # Check current project configuration curl -s -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/projects/" | jq '.results[] | {name, local_path, scm_revision}'

# Check actual filesystem state ls -la /var/lib/awx/projects/*/

# Check for broken symlinks find /var/lib/awx/projects -type l ! -exec test -e {} \; -print

# Check Tower settings docker exec tower_task cat /etc/tower/settings.py | grep PROJECTS_ROOT

# For AWX on Kubernetes kubectl exec -n awx deployment/awx-task -- ls -la /var/lib/awx/projects/ ```

Step 2: Force Project Update to Refresh Paths

Trigger a full project update to regenerate symlinks:

```bash # Via API curl -X POST -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/projects/15/update/"

# Via CLI awx projects update 15 --wait

# Or via playbook ansible tower -m uri -a "url=https://tower.example.com/api/v2/projects/15/update/ method=POST headers='Authorization: Bearer $TOKEN'" ```

For manual cleanup and refresh:

```bash # Stop Tower services ansible-tower-service stop

# Remove old symlinks find /var/lib/awx/projects -type l -delete

# Clear project cache rm -rf /var/lib/awx/projects/__pycache__ rm -rf /var/lib/awx/projects/*/__pycache__

# Update project from SCM cd /var/lib/awx/projects/production git fetch --all git checkout main git reset --hard origin/main

# Recreate proper symlink ln -sf playbooks_main_$(git rev-parse --short HEAD) playbooks

# Restart Tower ansible-tower-service start ```

Step 3: Update Tower Database Project References

Directly update the database if symlinks are correct but Tower still uses old paths:

```sql -- Connect to Tower database psql -U awx -d awx

-- Check current project state SELECT id, name, local_path, scm_revision, scm_type FROM main_project;

-- Update project path to current version UPDATE main_project SET local_path = 'playbooks_main_abc123def' WHERE id = 15;

-- Clear any cached playbook paths DELETE FROM main_job WHERE status = 'pending';

-- Verify update SELECT id, name, local_path FROM main_project WHERE id = 15; ```

Step 4: Configure Consistent Project Paths

Update Tower configuration to use consistent, versioned paths:

```python # /etc/tower/settings.py

# Use absolute paths for projects PROJECTS_ROOT = '/var/lib/awx/projects'

# Enable project versioning AWX_PROJECT_USE_VENV_PATH = '/var/lib/awx/venv'

# Configure project update behavior PROJECT_UPDATE_VVV = True

# Ensure symlinks are updated on every project sync AWX_ALWAYS_UPDATE_PROJECT_SYMLINKS = True ```

For AWX, configure the AWX custom resource:

yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  projects_persistence: true
  projects_existing_claim: awx-projects-claim
  task_env:
    - name: AWX_ALWAYS_UPDATE_PROJECT_SYMLINKS
      value: "true"
    - name: PROJECTS_ROOT
      value: "/var/lib/awx/projects"

Step 5: Implement Pre-Job Path Validation

Create a job callback that validates project paths before execution:

```python # /var/lib/awx/custom/callback_plugins/path_validator.py

from ansible.plugins.callback import CallbackBase import os import subprocess

class CallbackModule(CallbackBase): CALLBACK_VERSION = 2.0 CALLBACK_TYPE = 'notification' CALLBACK_NAME = 'path_validator' CALLBACK_NEEDS_WHITELIST = False

def v2_playbook_on_start(self, playbook): # Get the playbook path playbook_path = playbook._file_name project_dir = os.path.dirname(os.path.dirname(playbook_path))

# Check if this matches the expected project expected_revision = os.environ.get('AWX_PROJECT_REVISION', '')

if expected_revision: current_revision = self._get_git_revision(project_dir) if current_revision != expected_revision: self._display.error( f"Project revision mismatch! Expected {expected_revision}, " f"but found {current_revision} at {project_dir}" ) raise Exception("Project path validation failed - stale artifact detected")

self._display.display(f"Project path validated: {project_dir}")

def _get_git_revision(self, path): try: result = subprocess.run( ['git', 'rev-parse', '--short', 'HEAD'], cwd=path, capture_output=True, text=True ) return result.stdout.strip() except Exception: return 'unknown' ```

Step 6: Create a Project Path Sync Playbook

Automate path synchronization across Tower nodes:

```yaml # sync_project_paths.yml - name: Synchronize Tower project paths hosts: tower_nodes become: yes vars: project_name: "{{ project | default('all') }}" force_update: false

tasks: - name: Get project information from Tower API uri: url: "https://localhost/api/v2/projects/?name={{ project_name }}" method: GET user: "{{ tower_admin_user }}" password: "{{ tower_admin_password }}" validate_certs: no register: project_info delegate_to: localhost

name: Get expected project revision
set_fact:
expected_revision: "{{ project_info.json.results[0].scm_revision }}"
expected_path: "{{ project_info.json.results[0].local_path }}"
when: project_info.json.results | length > 0

name: Check current project directory
stat:
path: "/var/lib/awx/projects/{{ expected_path }}"
register: project_stat

name: Check for symlink mismatch
stat:
path: "/var/lib/awx/projects/{{ project_name }}"
register: symlink_stat

name: Remove stale symlinks
file:
path: "/var/lib/awx/projects/{{ project_name }}"
state: absent
when:
- symlink_stat.stat.exists
- symlink_stat.stat.islnk
- symlink_stat.stat.lnk_target != expected_path

name: Create correct symlink
file:
src: "/var/lib/awx/projects/{{ expected_path }}"
dest: "/var/lib/awx/projects/{{ project_name }}"
state: link
force: yes
when: force_update | bool or not project_stat.stat.exists

name: Verify project revision
command: git rev-parse HEAD
args:
chdir: "/var/lib/awx/projects/{{ expected_path }}"
register: current_revision
changed_when: false

name: Report revision mismatch
fail:
msg: "Project revision mismatch: expected {{ expected_revision }}, got {{ current_revision.stdout }}"
when: current_revision.stdout != expected_revision

name: Report successful sync
debug:
msg: "Project {{ project_name }} synchronized to revision {{ expected_revision }}"
`

Verification

Confirm project paths are correctly updated:

```bash # Check project symlink points to correct version readlink -f /var/lib/awx/projects/production/playbooks # Should output: /var/lib/awx/projects/production/playbooks_main_abc123

# Verify Tower API reports correct version curl -s -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/projects/15/" | jq '.local_path, .scm_revision'

# Run a test job and verify it uses correct playbook curl -X POST -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/job_templates/1/launch/" \ -d '{"verbosity": 3}'

# Check job output for correct paths curl -s -H "Authorization: Bearer $TOKEN" \ "https://tower.example.com/api/v2/jobs/$(curl -s -X POST ... | jq -r '.job')/stdout/" | grep "playbook" ```

For clustered Tower, verify all nodes have consistent state:

```bash # Check each node for node in tower1 tower2 tower3; do echo "=== $node ===" ssh $node "ls -la /var/lib/awx/projects/production/" done

# All should show the same symlink target ```

[ansible-cache-serves-old-data-after-deployment](/articles/ansible-cache-serves-old-data-after-deployment) - Cached data issues after deployment
[ansible-artifact-download-uses-an-old-mirror-after-proxy-change](/articles/ansible-artifact-download-uses-an-old-mirror-after-proxy-change) - Stale artifact references
[ansible-fresh-deployment-uses-the-correct-role-but-a-cron-worker-keeps-an-old-token](/articles/ansible-fresh-deployment-uses-the-correct-role-but-a-cron-worker-keeps-an-old-token) - Worker state persistence

[WordPress troubleshooting: Ansible Artifact Download Uses an Old Mi](ansible-artifact-download-uses-an-old-mirror-after-proxy-change)
[WordPress troubleshooting: Ansible Audit Trail Misses Events Under ](ansible-audit-trail-misses-events-under-burst-load)
[WordPress troubleshooting: Ansible Background Worker Gets Stuck in ](ansible-background-worker-stuck-in-a-retry-loop)
[WordPress troubleshooting: Ansible Backup Completes but Restore Fai](ansible-backup-completes-but-restore-fails-checksum-validation)
[WordPress troubleshooting: Ansible Batch Importer Duplicates Rows A](ansible-batch-importer-duplicates-rows-after-a-retry)

Was this guide helpful?

Related search paths

People also search for

If the symptom is close but not identical, these search paths usually surface the right neighboring fixes faster than scrolling the full archive.

WordPress troubleshooting: Ansible Build Output Path Changed but th WordPress troubleshooting: Ansible Build Output Path Changed but th Ansible WordPress troubleshooting: Ansible Build Output Path Changed but th troubleshooting WordPress troubleshooting: Ansible Build Output Path Changed but th fix Fix Ansible Tower/AWX project builds where runtime serves stale artifacts after output path changes, caused by cached symlinks, job output references, and stale project paths Ansible Fix Ansible Tower/AWX project builds where runtime serves stale artifacts after output path changes, caused by cached symlinks, job output references, and stale project paths

Explore Related Topics

Browse Guides from Other Categories

Discover troubleshooting guides from related categories to expand your knowledge.

FAQ

Ansible Troubleshooting FAQs

Common questions about troubleshooting and preventing similar issues

How do I know if this ansible-errors troubleshooting guide applies to my situation?

This guide is designed for ansible-errors issues. If you're experiencing similar symptoms described in the article, follow the step-by-step instructions. Start with the most common causes and work through the diagnostic process.

Is it safe to follow these ansible-errors troubleshooting steps?

Yes, all steps are designed to be safe and non-destructive. We recommend creating backups before making significant changes and testing each step before proceeding to the next.

How long does it typically take to resolve this type of ansible-errors issue?

Most ansible-errors issues can be resolved within 30 minutes to 2 hours, depending on the complexity and root cause. Follow the troubleshooting flow to identify and fix the problem efficiently.

How can I prevent this ansible-errors issue from happening again?

Regular maintenance, monitoring, and following best practices for ansible-errors configuration can help prevent recurrence. Consider implementing automated checks and alerts for early detection.

Written by

FixWikiHub Editorial Team

Our editorial team consists of experienced DevOps engineers, systems administrators, and cloud architects with hands-on experience in production environments across AWS, Azure, GCP, and on-premises infrastructure.

Every guide undergoes technical review for accuracy and is updated when software versions, commands, or best practices change.

Last updated: Feb 12, 2026

About our team

Important Notice

Disclaimer & Safety Guidelines

The troubleshooting steps in this guide are provided for educational and informational purposes. Before applying any changes to production systems:

Test in a staging environment first — Always verify commands and configurations in a non-production environment before deploying to live systems.
Create backups — Ensure you have current backups of databases, configurations, and critical files before making changes.
Understand the impact — Review how each step may affect your specific environment, dependencies, and users.
Consult official documentation — This guide supplements, but does not replace, official vendor documentation and best practices.

FixWikiHub is not responsible for any damages arising from the use of this content. See our Terms of Use for more information.

Resources

Official Documentation & Further Reading

For authoritative information, consult the official documentation for the technologies discussed in this guide. Our troubleshooting content supplements, but does not replace, vendor documentation.

AWS Documentation — Official Amazon Web Services guides and API references
Kubernetes Documentation — Official Kubernetes documentation
Nginx Documentation — Official Nginx web server documentation
Apache Documentation — Official Apache HTTP Server documentation
Docker Documentation — Official Docker container documentation

WordPress troubleshooting: Ansible Build Output Path Changed but th

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Step 1: Diagnose the Path Discrepancy

Step 2: Force Project Update to Refresh Paths

Step 3: Update Tower Database Project References

Step 4: Configure Consistent Project Paths

Step 5: Implement Pre-Job Path Validation

Step 6: Create a Project Path Sync Playbook

Verification

People also search for

Browse Guides from Other Categories

WordPress

SSL

DNS

Ansible Troubleshooting FAQs

FixWikiHub Editorial Team

Disclaimer & Safety Guidelines

Official Documentation & Further Reading

WordPress troubleshooting: Ansible Build Output Path Changed but th

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Step 1: Diagnose the Path Discrepancy

Step 2: Force Project Update to Refresh Paths

Step 3: Update Tower Database Project References

Step 4: Configure Consistent Project Paths

Step 5: Implement Pre-Job Path Validation

Step 6: Create a Project Path Sync Playbook

Verification

Related Issues

Related Articles

People also search for

Share this guide

More Ansible Troubleshooting Guides

Browse Guides from Other Categories

Ansible Troubleshooting FAQs

FixWikiHub Editorial Team

Disclaimer & Safety Guidelines

Official Documentation & Further Reading