Fixing peer disconnections due to slow processing & Transaction expiry

Transaction expiry wasn't happening. Use NTP.getTime to check whether
transactions have expired. Also reject expired transactions when trying
to add them to unconfirmed pool.

Sometimes producing a task took way too long, causing massive
spikes in the number of threads and peer disconnections.

This is down to a repository pool exhaustion, so
RepositoryManager.getRepository() would block (for up to 5 minutes).

The key method at fault was Network.getConnectablePeer().

Various fixes:

NetworkProcessor's executor now reaps old threads after only 10 seconds
instead of the usual 60 seconds.

Change logging in Network to help diagnose disconnection and repository
issues.

RepositoryManager now has a tryRepository() call that is non-blocking
and returns null if repository pool is exhausted.

Repository pool size increased from default (10) to 100.

Pruning peers is now opportunistic, using tryRepository(), and returns
early if repository pool is exhausted.

getConnectablePeer() is now opportunistic, using tryRepository(), and
returns null (no peer candidate for connection) if repository pool
is exhausted.

Merging peers is not opportunistic, using tryRepository().

Peer ping interval increased from 8s to 20s.

HSQLDBRepositoryFactory now logs when getConnection() takes over 1000ms.

Added more trace-level logging to ExecuteProduceConsume to
highlight slow produceTask() calls.
This commit is contained in:
catbref
2019-08-09 13:30:26 +01:00
parent 1094db288e
commit fa0b7615a6
11 changed files with 266 additions and 99 deletions

View File

@@ -0,0 +1,46 @@
package org.hsqldb.jdbc;
import java.sql.Connection;
import java.sql.SQLException;
import org.hsqldb.jdbc.JDBCPool;
import org.hsqldb.jdbc.pool.JDBCPooledConnection;
public class HSQLDBPool extends JDBCPool {
public HSQLDBPool(int poolSize) {
super(poolSize);
}
/**
* Tries to retrieve a new connection using the properties that have already been
* set.
*
* @return a connection to the data source, or null if no spare connections in pool
* @exception SQLException if a database access error occurs
*/
public Connection tryConnection() throws SQLException {
for (int i = 0; i < states.length(); i++) {
if (states.compareAndSet(i, RefState.available, RefState.allocated)) {
return connections[i].getConnection();
}
if (states.compareAndSet(i, RefState.empty, RefState.allocated)) {
try {
JDBCPooledConnection connection = (JDBCPooledConnection) source.getPooledConnection();
connection.addConnectionEventListener(this);
connection.addStatementEventListener(this);
connections[i] = connection;
return connections[i].getConnection();
} catch (SQLException e) {
states.set(i, RefState.empty);
}
}
}
return null;
}
}