Introduction
ActiveMQ uses JDBC master-slave replication where one broker holds a database lock. If the lock renewal fails or the lease expires, the broker loses its master status, potentially causing split-brain or failover issues.
Symptoms
Lock expired:
```bash $ tail -f /var/log/activemq/activemq.log
[ERROR] Failed to renew database lock: Lock has been held by another broker [WARN] Master broker lost lock, becoming slave [ERROR] org.apache.activemq.store.jdbc.JDBCPersistenceAdapter - Lock keepalive failed ```
Database connection error:
[ERROR] Cannot acquire database lock: Connection refused to database
[ERROR] java.sql.SQLException: Connection is closedSplit-brain warning:
[WARN] Multiple brokers claiming master status detected
[ERROR] Network partition detected: both brokers claiming lock ownershipCommon Causes
- 1.Lock keepalive timeout too short - Broker can't renew lock fast enough
- 2.Database connection issues - Connection dropped before renewal
- 3.Database overloaded - Slow response prevents lock renewal
- 4.Network latency - High latency causes renewal delays
- 5.Lock acquired by another broker - Another broker stole the lock
- 6.JDBC driver issues - Driver connection pooling problems
Step-by-Step Fix
Step 1: Check Lock Status
```bash # Check if broker is master curl -u admin:admin http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MasterBroker
# Check lock table in database # MySQL: mysql -u activemq -p -e "SELECT * FROM activemq.ACTIVEMQ_LOCK;"
# PostgreSQL: psql -U activemq -d activemq -c "SELECT * FROM activemq_lock;"
# Oracle: sqlplus activemq@activemq "SELECT * FROM ACTIVEMQ_LOCK;"
# Check lock acquisition time # In activemq.log: grep -i "acquired lock|lost lock" /var/log/activemq/activemq.log | tail -20 ```
Step 2: Configure Lock Keepalive
```xml <!-- In activemq.xml, adjust JDBC persistence adapter --> <persistenceAdapter> <jdbcPersistenceAdapter dataSource="#mysql-ds" lockKeepAlivePeriod="5000" lockAcquireSleepInterval="10000" createTablesOnStartup="false"> <locker> <databaseLocker lockAcquireSleepInterval="10000"/> </locker> </jdbcPersistenceAdapter> </persistenceAdapter>
<!-- Key parameters --> <!-- lockKeepAlivePeriod: How often to renew lock (default 30000ms) --> <!-- lockAcquireSleepInterval: How long to wait before retrying lock acquisition -->
<!-- For high-latency environments, increase keepalive period --> <jdbcPersistenceAdapter dataSource="#mysql-ds" lockKeepAlivePeriod="10000"/> ```
Step 3: Configure Database Connection Pool
```xml <!-- In activemq.xml, use connection pool --> <bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="jdbc:mysql://localhost:3306/activemq?autoReconnect=true"/> <property name="username" value="activemq"/> <property name="password" value="password"/> <property name="maxTotal" value="20"/> <property name="maxIdle" value="10"/> <property name="minIdle" value="5"/> <property name="validationQuery" value="SELECT 1"/> <property name="testOnBorrow" value="true"/> <property name="testWhileIdle" value="true"/> <property name="timeBetweenEvictionRunsMillis" value="30000"/> <property name="minEvictableIdleTimeMillis" value="60000"/> </bean>
<!-- Enable auto-reconnect --> <property name="url" value="jdbc:mysql://localhost:3306/activemq?autoReconnect=true&connectTimeout=5000&socketTimeout=30000"/> ```
Step 4: Check Database Connectivity
```bash # Test database connection mysql -h db-host -u activemq -p activemq -e "SELECT 1;"
# Check database response time mysql -h db-host -u activemq -p -e "SELECT BENCHMARK(1000000, SHA1('test'));"
# Check database status mysqladmin -h db-host status
# Check for connection limits mysql -h db-host -u activemq -p -e "SHOW VARIABLES LIKE 'max_connections';" mysql -h db-host -u activemq -p -e "SHOW STATUS LIKE 'Threads_connected';"
# Check network latency to database ping db-host traceroute db-host ```
Step 5: Handle Lock Acquisition Failures
```bash # If lock was stolen by another broker # Check which broker has the lock curl -u admin:admin http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=*/MasterBroker
# Stop broker that shouldn't be master systemctl stop activemq
# Clear lock table manually (emergency only) mysql -u activemq -p -e "DELETE FROM activemq.ACTIVEMQ_LOCK;"
# Restart broker that should be master systemctl start activemq
# Watch lock acquisition tail -f /var/log/activemq/activemq.log | grep -i "lock" ```
Step 6: Use Lease Database Locker
```xml <!-- Use lease-based locker for better timeout handling --> <persistenceAdapter> <jdbcPersistenceAdapter dataSource="#mysql-ds"> <locker> <lease-databaseLocker lockAcquireSleepInterval="10000" leaseHolderId="broker-1" leaseKeepAlivePeriod="5000" leaseExpirationTime="60000"/> </locker> </jdbcPersistenceAdapter> </persistenceAdapter>
<!-- Lease-based locker benefits --> <!-- - Explicit expiration time --> <!-- - Better timeout handling --> <!-- - Avoids split-brain scenarios --> ```
Step 7: Configure Network Timeout
```xml <!-- In JDBC URL, set timeouts --> <bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource"> <property name="url" value="jdbc:mysql://db-host:3306/activemq?connectTimeout=10000&socketTimeout=60000&autoReconnect=true"/> </bean>
<!-- For PostgreSQL --> <property name="url" value="jdbc:postgresql://db-host:5432/activemq?connectTimeout=10&socketTimeout=60&loginTimeout=10"/>
<!-- For Oracle --> <property name="url" value="jdbc:oracle:thin:@db-host:1521:activemq?connectTimeout=10000"/> ```
Step 8: Monitor Lock Health
```bash # Check lock renewal in logs grep -i "renewed lock|keepalive" /var/log/activemq/activemq.log | tail -50
# Enable lock logging # In logging.properties: log4j.logger.org.apache.activemq.store.jdbc=DEBUG log4j.logger.org.apache.activemq.store.jdbc.locker=DEBUG
# Monitor via JMX curl -u admin:admin http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/PersistenceAdapter/lockTime
# Set up alerting for lock failures grep -i "lost lock|failed to renew" /var/log/activemq/activemq.log ```
Step 9: Handle Split-Brain Scenario
```bash # If multiple brokers claiming master status # Stop all brokers systemctl stop activemq
# Check lock table mysql -u activemq -p -e "SELECT * FROM activemq.ACTIVEMQ_LOCK;"
# Clear lock table mysql -u activemq -p -e "DELETE FROM activemq.ACTIVEMQ_LOCK;"
# Start designated master first # On master broker: systemctl start activemq
# Wait for lock acquisition tail -f /var/log/activemq/activemq.log | grep -i "acquired lock"
# Start slave brokers # On slave brokers: systemctl start activemq
# Verify only one master curl -u admin:admin http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=*/MasterBroker ```
Step 10: Configure Failover URL
```java // Configure clients with failover URL ConnectionFactory factory = new ActiveMQConnectionFactory( "failover:(tcp://master:61616,tcp://slave:61616)?randomize=false&priorityBackup=true&timeout=5000" );
// Client detects master change and reconnects // Failover parameters: // randomize=false: Always try master first // priorityBackup=true: Prefer master // timeout: Max wait time for connection ```
JDBC Lock Settings
| Setting | Default | Recommended |
|---|---|---|
| lockKeepAlivePeriod | 30000ms | 5000-10000ms |
| lockAcquireSleepInterval | 10000ms | 5000-10000ms |
| connectTimeout | 0 (infinite) | 10000ms |
| socketTimeout | 0 (infinite) | 60000ms |
Verification
```bash # After configuration changes # Restart ActiveMQ systemctl restart activemq
# Check lock acquisition tail -f /var/log/activemq/activemq.log | grep -i "lock"
# Should see: "Acquired lock" and regular "Renewed lock"
# Verify master status curl -u admin:admin http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/MasterBroker
# Should return true for master broker
# Check database lock table mysql -u activemq -p -e "SELECT * FROM activemq.ACTIVEMQ_LOCK;"
# Monitor for lock renewals grep -c "renewed lock" /var/log/activemq/activemq.log ```
Prevention
To prevent ActiveMQ JDBC lock expired issues from recurring, implement these proactive measures:
1. Monitor Lock Status
groups:
- name: activemq-ha
rules:
- alert: ActiveMQJDBCLockExpired
expr: |
activemq_jdbc_lock_expired > 0
for: 1m
labels:
severity: critical
annotations:
summary: "ActiveMQ JDBC lock expired"2. Configure Proper Lock Settings
```xml <!-- activemq.xml --> <persistenceAdapter> <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="5000" lockAcquireSleepInterval="5000"/> </persistenceAdapter>
<bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource"> <property name="url" value="jdbc:mysql://mysql-host:3306/activemq"/> <property name="username" value="activemq"/> <property name="password" value="password"/> <property name="maxTotal" value="20"/> <property name="maxIdle" value="10"/> <property name="connectionProperties" value="connectTimeout=10000;socketTimeout=60000"/> </bean> ```
3. Monitor Database Connectivity
```bash # Regular connectivity test cat << 'EOF' > /usr/local/bin/check_activemq_db.sh #!/bin/bash mysql -u activemq -p$ACTIVEMQ_DB_PASS -e "SELECT 1" activemq if [ $? -ne 0 ]; then echo "Database connectivity issue" systemctl restart activemq fi EOF
chmod +x /usr/local/bin/check_activemq_db.sh echo "*/5 * * * * root /usr/local/bin/check_activemq_db.sh" > /etc/cron.d/activemq-db-check ```
Best Practices Checklist
- [ ] Monitor lock status with alerts
- [ ] Configure proper lock keep-alive
- [ ] Monitor database connectivity
- [ ] Set appropriate timeouts
- [ ] Use connection pooling
- [ ] Test failover regularly
Related Issues
- [Fix ActiveMQ Master Slave Failover](/articles/fix-activemq-master-slave-failover)
- [Fix ActiveMQ Destination Full](/articles/fix-activemq-destination-full)
- [Fix MySQL Connection Refused](/articles/fix-mysql-connection-refused)
Related Articles
- [Fix Fix Activemq Broker Down Issue in Messaging](fix-activemq-broker-down)
- [Fix ActiveMQ Destination Full](fix-activemq-destination-full)
- [Fix ActiveMQ Kaha PIndex Corrupted](fix-activemq-kaha-pindex-corrupted)
- [Fix ActiveMQ Master Slave Failover](fix-activemq-master-slave-failover)
- [Fix ActiveMQ Slow Consumer](fix-activemq-slow-consumer)
<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Fix ActiveMQ JDBC Lock Expired", "description": "Troubleshoot ActiveMQ JDBC lock expired errors. Configure lock keepalive, database connections, and failover handling.", "url": "https://www.fixwikihub.com/fix-activemq-jdbc-lock-expired", "publisher": { "@type": "Organization", "name": "FixWikiHub", "url": "https://www.fixwikihub.com" }, "author": { "@type": "Person", "name": "FixWikiHub Editorial Team" }, "datePublished": "2026-04-03T21:44:40.140Z", "dateModified": "2026-04-03T21:44:40.140Z" } </script>