Home / Monitoring / Fix Prometheus Scrape Error

Monitoring

Fix Prometheus Scrape Error

Resolve Prometheus scrape errors by checking target configuration, endpoint accessibility, and scrape interval settings.

Published: Apr 11, 20266 min readBy FixWikiHub Editorial Team

Abstract illustration for a troubleshooting knowledge base category.

Introduction

Prometheus cannot scrape metrics from configured targets. Targets show as down in the UI and metrics are not being collected.

Symptoms

Prometheus UI shows target down:

bash

Status: Down
Error: Get "http://target:9090/metrics": dial tcp 10.0.0.1:9090: connection refused
Last Scrape: Never

Prometheus logs:

```bash $ journalctl -u prometheus -f

level=error ts=2024-01-01T00:00:00.000Z caller=scrape.go:123 component="scrape manager" msg="scrape failed" target=target:9090 err="connection refused" ```

Common Causes

1.Target not running - Application or exporter process stopped
2.Wrong endpoint - Incorrect URL or port configured
3.Network blocking - Firewall or network policy blocking access
4.DNS resolution - Target hostname not resolving
5.Authentication - Target requires auth but not configured
6.TLS issues - Certificate validation failing
7.Scrape timeout - Endpoint responding too slowly

Step-by-Step Fix

1.Check logs for specific error messages
2.Verify configuration settings
3.Test network connectivity
4.Review recent changes
5.Apply corrective action
6.Verify the fix

Step 1: Check Prometheus Target Status

```bash # Check Prometheus UI: # http://localhost:9090/targets

# API query: curl -s http://localhost:9090/api/v1/targets | jq

# List all targets: curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health, lastError: .lastError}'

# Count healthy vs unhealthy: curl -s http://localhost:9090/api/v1/targets | jq '[.data.activeTargets[].health] | group_by(.) | map({health: .[0], count: length})' ```

Step 2: Check Prometheus Configuration

```bash # View Prometheus config: cat /etc/prometheus/prometheus.yml

# Validate config syntax: promtool check config /etc/prometheus/prometheus.yml

# Reload config: curl -X POST http://localhost:9090/-/reload systemctl reload prometheus ```

Step 3: Test Target Endpoint Accessibility

```bash # Test direct HTTP access: curl -v http://target:9090/metrics

# Test with timeout: curl -m 10 http://target:9090/metrics

# Check if port is listening on target: ssh target 'ss -tlnp | grep 9090'

# Test network connectivity: nc -zv target 9090

# Check DNS resolution: nslookup target dig target ```

Step 4: Check Target Application Status

```bash # On target host:

# Check if exporter running: systemctl status node_exporter

# Check process: ps aux | grep exporter

# Check port: ss -tlnp | grep 9090

# Test locally on target: curl http://localhost:9090/metrics

# Start exporter if not running: systemctl start node_exporter ```

Step 5: Check Firewall and Network

```bash # On Prometheus host - check outbound: iptables -L OUTPUT -n -v

# On target host - check inbound: iptables -L INPUT -n -v | grep 9090

# Allow Prometheus to scrape: iptables -I INPUT -s prometheus_ip -p tcp --dport 9090 -j ACCEPT

# Test connectivity after firewall change: nc -zv target 9090 ```

Step 6: Check TLS Configuration

```bash # For HTTPS endpoints:

# Check certificate: curl -v https://target:9090/metrics

# TLS error example: # SSL certificate problem: unable to get local issuer certificate

# Test TLS connection: openssl s_client -connect target:9090 -showcerts

# Check certificate expiration: openssl s_client -connect target:9090 2>/dev/null | openssl x509 -noout -dates ```

Step 7: Check Authentication Configuration

```bash # For targets requiring authentication:

# Basic auth config in prometheus.yml: scrape_configs: - job_name: myapp basic_auth_configs: username: admin password: secret

# Test auth manually: curl -u admin:secret http://target:9090/metrics

# Token auth: curl -H "Authorization: Bearer mytoken" http://target:9090/metrics ```

Step 8: Check Scrape Interval and Timeout

```bash # Check global scrape interval: curl -s http://localhost:9090/api/v1/status/config | jq '.data.global.scrape_interval'

# Check job-specific: cat /etc/prometheus/prometheus.yml | grep -A 5 scrape_interval

# Set scrape timeout: scrape_configs: - job_name: slow-target scrape_interval: 30s scrape_timeout: 10s ```

Step 9: Check Prometheus Logs

```bash # View Prometheus logs: journalctl -u prometheus -f

# Check for scrape errors: journalctl -u prometheus --since "1 hour ago" | grep -i "scrape"

# Look for connection errors: journalctl -u prometheus | grep -E "connection refused|timeout|unreachable"

# Enable debug logging: # In prometheus.yml: log.level: debug systemctl restart prometheus ```

Step 10: Verify and Monitor Scrape

```bash # Check target status: curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="myapp") | .health'

# Query metrics from target: curl -s 'http://localhost:9090/api/v1/query?query=up{job="myapp"}' | jq '.data.result[].value[1]'

# Monitor targets in real-time: watch -n 5 'curl -s http://localhost:9090/api/v1/targets | jq ".data.activeTargets[].health"' ```

Prometheus Scrape Error Checklist

Check	Command	Expected
Target health	/api/v1/targets	health: up
Endpoint access	curl /metrics	Metrics returned
Network	nc -zv	Port open
DNS resolution	nslookup	IP returned
Config syntax	promtool check	SUCCESS
Firewall	iptables -L	Port allowed

Verification

```bash # 1. Check target status curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | .health' // "up"

# 2. Query metrics from target curl -s 'http://localhost:9090/api/v1/query?query=up{job="myapp"}' | jq '.data.result[].value[1]' // "1"

# 3. Test endpoint directly curl http://target:9090/metrics // Metrics returned

# 4. Check no errors in logs journalctl -u prometheus --since "5 min ago" | grep -i "scrape failed" // No output ```

[Fix Prometheus Query Timeout](/articles/fix-prometheus-query-timeout)
[Fix Grafana Dashboard Not Loading](/articles/fix-grafana-dashboard-not-loading)

[WordPress troubleshooting: Fix IAM Timeout Error - Complete Trouble](fix-iam-timeout-error)
[Technical troubleshooting: Fix Cloudwatch Alarm Not Triggering Issue in Monit](cloudwatch-alarm-not-triggering)
[Fix Datadog Agent Not Sending Metrics Issue in Monitoring](datadog-agent-not-sending-metrics)
[Fix Elasticsearch Cluster Red Yellow Status Issue in Monitoring](elasticsearch-cluster-red-yellow-status)
[Fix Alertmanager Notification Failed](fix-alertmanager-notification-failed)

Was this guide helpful?

Related search paths

People also search for

If the symptom is close but not identical, these search paths usually surface the right neighboring fixes faster than scrolling the full archive.

Prometheus Scrape Error Prometheus Scrape Error Monitoring Prometheus Scrape Error troubleshooting Prometheus Scrape Error fix Resolve Prometheus scrape errors by checking target configuration, endpoint accessibility, and scrape interval settings Monitoring Resolve Prometheus scrape errors by checking target configuration, endpoint accessibility, and scrape interval settings

Explore Related Topics

Browse Guides from Other Categories

Discover troubleshooting guides from related categories to expand your knowledge.

FAQ

Monitoring Troubleshooting FAQs

Common questions about troubleshooting and preventing similar issues

How do I know if this monitoring-errors troubleshooting guide applies to my situation?

This guide is designed for monitoring-errors issues. If you're experiencing similar symptoms described in the article, follow the step-by-step instructions. Start with the most common causes and work through the diagnostic process.

Is it safe to follow these monitoring-errors troubleshooting steps?

Yes, all steps are designed to be safe and non-destructive. We recommend creating backups before making significant changes and testing each step before proceeding to the next.

How long does it typically take to resolve this type of monitoring-errors issue?

Most monitoring-errors issues can be resolved within 30 minutes to 2 hours, depending on the complexity and root cause. Follow the troubleshooting flow to identify and fix the problem efficiently.

How can I prevent this monitoring-errors issue from happening again?

Regular maintenance, monitoring, and following best practices for monitoring-errors configuration can help prevent recurrence. Consider implementing automated checks and alerts for early detection.

Written by

FixWikiHub Editorial Team

Our editorial team consists of experienced DevOps engineers, systems administrators, and cloud architects with hands-on experience in production environments across AWS, Azure, GCP, and on-premises infrastructure.

Every guide undergoes technical review for accuracy and is updated when software versions, commands, or best practices change.

Last updated: Apr 11, 2026

About our team

Important Notice

Disclaimer & Safety Guidelines

The troubleshooting steps in this guide are provided for educational and informational purposes. Before applying any changes to production systems:

Test in a staging environment first — Always verify commands and configurations in a non-production environment before deploying to live systems.
Create backups — Ensure you have current backups of databases, configurations, and critical files before making changes.
Understand the impact — Review how each step may affect your specific environment, dependencies, and users.
Consult official documentation — This guide supplements, but does not replace, official vendor documentation and best practices.

FixWikiHub is not responsible for any damages arising from the use of this content. See our Terms of Use for more information.

Resources

Official Documentation & Further Reading

For authoritative information, consult the official documentation for the technologies discussed in this guide. Our troubleshooting content supplements, but does not replace, vendor documentation.

AWS Documentation — Official Amazon Web Services guides and API references
Kubernetes Documentation — Official Kubernetes documentation
Nginx Documentation — Official Nginx web server documentation
Apache Documentation — Official Apache HTTP Server documentation
Docker Documentation — Official Docker container documentation