Home / Monitoring / Fix Fluentd Output Connection Refused

Monitoring

Fix Fluentd Output Connection Refused

Resolve Fluentd output connection refused by checking network connectivity, plugin configuration, and service availability.

Published: Apr 5, 20267 min readBy FixWikiHub Editorial Team

Abstract illustration for a troubleshooting knowledge base category.

Introduction

Fluentd cannot connect to output destinations like Elasticsearch, Kafka, or S3. Logs fail to be delivered and may be lost or buffered indefinitely.

Symptoms

Connection refused:

bash

[error]: #0 unexpected error error_class=Errno::ECONNREFUSED error="Connection refused"

Output error:

bash

[error]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2024-01-01 00:00:00 +0000 chunk="5a8f3b..."
[error]: #0 error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Could not connect to Elasticsearch"

Timeout:

bash

[warn]: #0 timed out to send data to output plugin
[error]: #0 send data failed error_class=Timeout::Error error="execution expired"

Common Causes

1.Destination down - Elasticsearch, Kafka, or other destination not running
2.Wrong endpoint - Incorrect host or port in configuration
3.Network blocked - Firewall or security group blocking connection
4.DNS failure - Hostname not resolving correctly
5.TLS issues - Certificate problems for HTTPS connections
6.Service overload - Destination not accepting new connections

Step-by-Step Fix

Step 1: Check Fluentd Status

```bash # Check Fluentd running: systemctl status td-agent # Or: systemctl status fluentd

# Check Fluentd logs: tail -f /var/log/td-agent/td-agent.log

# Check Fluentd configuration: cat /etc/td-agent/td-agent.conf

# Check Fluentd metrics: curl http://localhost:24220/api/plugins.json

# Check buffer status: ls -la /var/log/td-agent/buffer/ ```

Step 2: Check Destination Availability

```bash # Check Elasticsearch: curl -I http://elasticsearch:9200

# Check Kafka: nc -zv kafka 9092

# Check specific port: nc -zv destination-host 9200

# Test connectivity: telnet elasticsearch 9200

# Check DNS: nslookup elasticsearch dig elasticsearch

# Check if destination is running: # For Elasticsearch: systemctl status elasticsearch curl http://elasticsearch:9200/_cluster/health

# For Kafka: systemctl status kafka kafka-topics.sh --bootstrap-server kafka:9092 --list

# Check from Fluentd host: ping elasticsearch ```

Step 3: Check Output Configuration

```xml # Check td-agent.conf: <match *.**> @type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix fluentd

# Connection settings connect_timeout 10 request_timeout 60

# Buffer settings <buffer> @type file path /var/log/td-agent/buffer timekey 60 timekey_wait 30 flush_mode interval flush_interval 10s flush_thread_count 4 retry_type exponential_backoff retry_wait 1s retry_max_interval 60s retry_timeout 24h retry_forever true chunk_limit_size 16MB queue_limit_length 512 </buffer> </match> ```

Step 4: Check Network Connectivity

```bash # Check firewall: iptables -L -n | grep 9200 iptables -L -n | grep 9092

# Allow port: iptables -I INPUT -p tcp --dport 9200 -j ACCEPT

# For firewalld: firewall-cmd --list-ports firewall-cmd --add-port=9200/tcp --permanent firewall-cmd --reload

# Check SELinux: getenforce

# If enforcing, may block outbound connections: setsebool -P httpd_can_network_connect 1

# Check if port listening on destination: ss -tlnp | grep 9200

# Test from Fluentd server: curl -v http://elasticsearch:9200 ```

Step 5: Configure Retry Settings

```xml # Robust retry configuration: <match *.**> @type elasticsearch host elasticsearch port 9200

# Reconnect settings reconnect_on_error true reload_on_failure true reload_connections false

# Timeout settings connect_timeout 30 request_timeout 120 http_backend typhoeus

<buffer> @type file path /var/log/td-agent/buffer

# Time-based chunking timekey 60 timekey_wait 30

# Flush settings flush_mode interval flush_interval 10s flush_thread_count 8

# Retry settings retry_type exponential_backoff retry_wait 1s retry_max_interval 300s retry_timeout 72h retry_forever true

# Buffer limits chunk_limit_size 16MB total_limit_size 2GB queue_limit_length 1024

# Overflow action overflow_action block </buffer>

# TLS settings ssl_verify true ca_file /etc/ssl/certs/ca.crt </match> ```

Step 6: Check TLS Configuration

```xml # For HTTPS connections: <match secure.**> @type elasticsearch host elasticsearch-secure port 9243 scheme https

# TLS verification ssl_verify true ca_file /etc/td-agent/ca.crt client_cert /etc/td-agent/client.crt client_key /etc/td-agent/client.key client_key_passphrase your_passphrase </match>

# For testing (disable verification): <match secure.**> @type elasticsearch host elasticsearch-secure port 9243 scheme https ssl_verify false ssl_version TLSv1_2 </match>

# Check certificates: openssl s_client -connect elasticsearch:9243 -showcerts ```

Step 7: Check Authentication

```xml # Elasticsearch with authentication: <match *.**> @type elasticsearch host elasticsearch port 9200 user fluentd password your_password

# Or use API key: api_key your_api_key_base64 </match>

# Kafka with SASL: <match kafka.**> @type kafka2 brokers kafka1:9092,kafka2:9092 topic logs

sasl_over_ssl true username fluentd password your_password ssl_ca_cert /etc/td-agent/kafka-ca.crt </match>

# S3 with IAM: <match s3.**> @type s3 aws_key_id your_key aws_sec_key your_secret s3_bucket your-bucket s3_region us-east-1 path logs/ </match> ```

Step 8: Test Output Manually

```bash # Test Elasticsearch output: curl -X POST "http://elasticsearch:9200/fluentd/_doc" \ -H 'Content-Type: application/json' \ -d '{"message":"test"}'

# Test Kafka output: kafka-console-producer.sh --bootstrap-server kafka:9092 --topic logs # Type messages, should succeed

# Test with Fluentd debug: td-agent -c /etc/td-agent/td-agent.conf -vv

# Or with Fluentd command: fluentd -c /etc/td-agent/td-agent.conf -vv

# Send test message: curl -X POST http://localhost:8888/test.tag -d '{"message":"test"}' ```

Step 9: Monitor Buffer Status

```bash # Check buffer directory: ls -la /var/log/td-agent/buffer/

# Buffer files growing: du -sh /var/log/td-agent/buffer/ watch -n 5 'du -sh /var/log/td-agent/buffer/'

# Check buffer stats via API: curl http://localhost:24220/api/plugins.json | jq '.plugins[] | select(.type=="elasticsearch")'

# Monitor metrics: curl http://localhost:24220/api/metrics.json | jq

# Create monitoring script: cat << 'EOF' > /usr/local/bin/monitor-fluentd.sh #!/bin/bash

echo "=== Fluentd Status ===" systemctl status td-agent --no-pager | head -5

echo "" echo "=== Buffer Size ===" du -sh /var/log/td-agent/buffer/

echo "" echo "=== Output Errors ===" grep -c "error" /var/log/td-agent/td-agent.log

echo "" echo "=== Connection Status ===" curl -s http://localhost:24220/api/plugins.json | jq '.plugins[] | {type: .type, output: .output}'

echo "" echo "=== Recent Errors ===" tail -20 /var/log/td-agent/td-agent.log | grep error EOF

chmod +x /usr/local/bin/monitor-fluentd.sh ```

Step 10: Configure Fallback Output

```xml # Add fallback for resilience: <match **> @type copy <store> @type elasticsearch host elasticsearch-primary port 9200 <buffer> @type file path /var/log/td-agent/buffer-primary </buffer> </store> <store> @type file path /var/log/td-agent/fallback.log <buffer> @type file path /var/log/td-agent/buffer-fallback </buffer> </store> </match>

# Or use secondary output: <match **> @type elasticsearch host elasticsearch-primary port 9200

<buffer> @type file path /var/log/td-agent/buffer retry_forever true </buffer>

<secondary> @type file path /var/log/td-agent/failed-logs.log </secondary> </match> ```

Fluentd Output Connection Checklist

Check	Command	Expected
Fluentd running	systemctl status	Active
Destination up	curl nc	Responding
Network	ping nc	Connected
DNS	nslookup	Resolved
Config	td-agent.conf	Correct endpoint
Buffer	du buffer	Not full
Logs	tail log	No errors

Verification

```bash # After fixing connection issue

# 1. Test connection nc -zv elasticsearch 9200 // Connected

# 2. Restart Fluentd systemctl restart td-agent

# 3. Send test log curl -X POST http://localhost:8888/test -d '{"message":"test"}' // Accepted

# 4. Check destination curl http://elasticsearch:9200/fluentd/_search?q=test // Document found

# 5. Check buffer draining du -sh /var/log/td-agent/buffer/ // Size decreasing

# 6. Monitor logs tail -f /var/log/td-agent/td-agent.log // No connection errors ```

[Fix Fluentd Buffer Overflow](/articles/fix-fluentd-buffer-overflow)
[Fix Elasticsearch Cluster Red Status](/articles/fix-elasticsearch-cluster-red-status)
[Fix Kafka Producer Retries Exceeded](/articles/fix-kafka-producer-retries-exceeded)

[WordPress troubleshooting: Fix IAM Timeout Error - Complete Trouble](fix-iam-timeout-error)
[Technical troubleshooting: Fix Cloudwatch Alarm Not Triggering Issue in Monit](cloudwatch-alarm-not-triggering)
[Fix Datadog Agent Not Sending Metrics Issue in Monitoring](datadog-agent-not-sending-metrics)
[Fix Elasticsearch Cluster Red Yellow Status Issue in Monitoring](elasticsearch-cluster-red-yellow-status)
[Fix Alertmanager Notification Failed](fix-alertmanager-notification-failed)

Was this guide helpful?

Related search paths

People also search for

If the symptom is close but not identical, these search paths usually surface the right neighboring fixes faster than scrolling the full archive.

Fluentd Output Connection Refused Fluentd Output Connection Refused Monitoring Fluentd Output Connection Refused troubleshooting Fluentd Output Connection Refused fix Resolve Fluentd output connection refused by checking network connectivity, plugin configuration, and service availability Monitoring Resolve Fluentd output connection refused by checking network connectivity, plugin configuration, and service availability

Explore Related Topics

Browse Guides from Other Categories

Discover troubleshooting guides from related categories to expand your knowledge.

FAQ

Monitoring Troubleshooting FAQs

Common questions about troubleshooting and preventing similar issues

How do I know if this monitoring-errors troubleshooting guide applies to my situation?

This guide is designed for monitoring-errors issues. If you're experiencing similar symptoms described in the article, follow the step-by-step instructions. Start with the most common causes and work through the diagnostic process.

Is it safe to follow these monitoring-errors troubleshooting steps?

Yes, all steps are designed to be safe and non-destructive. We recommend creating backups before making significant changes and testing each step before proceeding to the next.

How long does it typically take to resolve this type of monitoring-errors issue?

Most monitoring-errors issues can be resolved within 30 minutes to 2 hours, depending on the complexity and root cause. Follow the troubleshooting flow to identify and fix the problem efficiently.

How can I prevent this monitoring-errors issue from happening again?

Regular maintenance, monitoring, and following best practices for monitoring-errors configuration can help prevent recurrence. Consider implementing automated checks and alerts for early detection.

Written by

FixWikiHub Editorial Team

Our editorial team consists of experienced DevOps engineers, systems administrators, and cloud architects with hands-on experience in production environments across AWS, Azure, GCP, and on-premises infrastructure.

Every guide undergoes technical review for accuracy and is updated when software versions, commands, or best practices change.

Last updated: Apr 5, 2026

About our team

Important Notice

Disclaimer & Safety Guidelines

The troubleshooting steps in this guide are provided for educational and informational purposes. Before applying any changes to production systems:

Test in a staging environment first — Always verify commands and configurations in a non-production environment before deploying to live systems.
Create backups — Ensure you have current backups of databases, configurations, and critical files before making changes.
Understand the impact — Review how each step may affect your specific environment, dependencies, and users.
Consult official documentation — This guide supplements, but does not replace, official vendor documentation and best practices.

FixWikiHub is not responsible for any damages arising from the use of this content. See our Terms of Use for more information.

Resources

Official Documentation & Further Reading

For authoritative information, consult the official documentation for the technologies discussed in this guide. Our troubleshooting content supplements, but does not replace, vendor documentation.

AWS Documentation — Official Amazon Web Services guides and API references
Kubernetes Documentation — Official Kubernetes documentation
Nginx Documentation — Official Nginx web server documentation
Apache Documentation — Official Apache HTTP Server documentation
Docker Documentation — Official Docker container documentation

Fix Fluentd Output Connection Refused

Introduction

Symptoms

Common Causes

Step-by-Step Fix

Step 1: Check Fluentd Status

Step 2: Check Destination Availability

Step 3: Check Output Configuration

Step 4: Check Network Connectivity

Step 5: Configure Retry Settings

Step 6: Check TLS Configuration

Step 7: Check Authentication

Step 8: Test Output Manually

Step 9: Monitor Buffer Status

Step 10: Configure Fallback Output

Fluentd Output Connection Checklist

Verification

Related Issues

Related Articles

People also search for

Share this guide

More Monitoring Troubleshooting Guides

Browse Guides from Other Categories

Monitoring Troubleshooting FAQs

FixWikiHub Editorial Team

Disclaimer & Safety Guidelines

Official Documentation & Further Reading