Merge pull request #266 from jschulthess/reticulum

Reticulum updates, Core v5.0.2

.github/workflows/pr-testing.yml

@@ -21,7 +21,11 @@ jobs:
       with:
         java-version: '11'
         distribution: 'adopt'
-
+        
+    - name: Load custom deps
+      run: |
+        mvn install -DskipTests=true --file pom.xml
+        
     - name: Run all tests
       run: |
         mvn -B clean test -DskipTests=false --file pom.xml

.vscode/settings.json

@@ -1,3 +0,0 @@
-{
-    "java.compile.nullAnalysis.mode": "automatic"
-}

README.md

@@ -1,12 +1,56 @@
-# Qortal Project - Official Repo
+# Qortal Project - Qortal Core - Primary Repository
+The Qortal Core is the blockchain and node component of the overall project. It contains the primary API, and ability to make calls to create transactions, and interact with the Qortal Blockchain Network. 
 
-## Build / run
+In order to run the Qortal Core, a machine with java 11+ installed is required. Minimum RAM specs will vary depending on settings, but as low as 4GB of RAM should be acceptable in most scenarios. 
+
+Qortal is a complete infrastructure platform with a blockchain backend, it is capable of indefinite web and application hosting with no continual fees, replacement of DNS and centralized name systems and communications systems, and is the foundation of the next generation digital infrastructure of the world. Qortal is unique in nearly every way, and was written from scratch to address as many concerns from both the existing 'blockchain space' and the 'typical internet' as possible, while maintaining a system that is easy to use and able to run on 'any' computer. 
+
+Qortal contains extensive functionality geared toward complete decentralization of the digital world. Removal of 'middlemen' of any kind from all transactions, and ability to publish websites and applications that require no continual fees, on a name that is truly owned by the account that registered it, or purchased it from another. A single name on Qortal is capable of being both a namespace and a 'username'. That single name can have an application, website, public and private data, communications, authentication, the namespace itself and more, which can subsequently be sold to anyone else without the need to change any type of 'hosting' or DNS entries that do not exist, email that doesn't exist, etc. Maintaining the same functionality as those replaced features of web 2.0. 
+
+Over time Qortal has progressed into a fully featured environment catering to any and all types of people and organizations, and will continue to advance as time goes on. Brining more features, capability, device support, and availale replacements for web2.0. Ultimately building a new, completely user-controlled digital world without limits. 
+
+Qortal has no owner, no company on top of it, and is completely community built, run, and funded. A community-established and run group of developers known as the 'dev-group' or Qortal Development Group, make group_approval based decisions for the project's future. If you are a developer interested in assisting with the project, you meay reach out to the Qortal Development Group in any of the available Qortal community locations. Either on the Qortal network itself, or on one of the temporary centralized social media locations. 
+
+Building the future one block at a time. Welcome to Qortal. 
+
+# Building the Qortal Core from Source
+
+## Build / Run
 
 - Requires Java 11. OpenJDK 11 recommended over Java SE.
 - Install Maven
 - Use Maven to fetch dependencies and build: `mvn clean package`
+- Update Maven dependencies: `mvn install` 
 - Built JAR should be something like `target/qortal-1.0.jar`
 - Create basic *settings.json* file: `echo '{}' > settings.json`
 - Run JAR in same working directory as *settings.json*: `java -jar target/qortal-1.0.jar`
 - Wrap in shell script, add JVM flags, redirection, backgrounding, etc. as necessary.
 - Or use supplied example shell script: *start.sh*
+
+## IntelliJ IDEA Configuration
+
+- Run -> Edit Configurations
+- Add New Application
+- Name: qortal
+- SDK: java 11
+- Main Class: org.qortal.controller.Controller
+- Program arguments: settings.json -Dlog4j.configurationFile=log4j2.properties -ea
+- Environment variables: Djava.net.preferIPv4Stack=false
+
+# Using a pre-built Qortal 'jar' binary
+
+If you prefer to utilize a released version of Qortal, you may do so by downloading one of the available releases from the releases page, that are also linked on https://qortal.org and https://qortal.dev. 
+
+# Learning Q-App Development
+
+https://qortal.dev contains dev documentation for building JS/React (and other client-side languages) applications or 'Q-Apps' on Qortal. Q-Apps are published on Registered Qortal Names, and aside from a single Name Registration fee, and a fraction of QORT for a publish transaction, require zero continual costs. These applications get more redundant with each new access from a new Qortal Node, making your application faster for the next user to download, and stronger as time goes on. Q-Apps live indefinitely in the history of the blockchain-secured Qortal Data Network (QDN).
+
+# How to learn more
+
+If the project interests you, you may learn more from the various web2 and QDN based websites focused on introductory information. 
+
+https://qortal.org - primary internet presence 
+https://qortal.dev - secondary and development focused website with links to many new developments and documentation
+https://wiki.qortal.org - community built and managed wiki with detailed information regarding the project
+
+links to telegram and discord communities are at the top of https://qortal.org as well. 

WindowsInstaller/Install Files/AppData/settings.json

@@ -1,3 +1,4 @@
 {
-  "apiDocumentationEnabled": true
+  "apiDocumentationEnabled": true,
+  "apiWhitelistEnabled": false
 }

WindowsInstaller/README.md

@@ -2,7 +2,9 @@
 
 ## Prerequisites
 
-* AdvancedInstaller v19.4 or better, and enterprise licence if translations are required
+* AdvancedInstaller v19.4 or better, and enterprise licence. 
+* Qortal has an open source license, however it currently (as of December 2024) only supports up to version 19. (We may need to reach out to Advanced Installer again to obtain a new license at some point, if needed. 
+* Reach out to @crowetic for links to the installer install files, and license. 
 * Installed AdoptOpenJDK v17 64bit, full JDK *not* JRE
 
 ## General build instructions
@@ -10,6 +12,12 @@
 If this is your first time opening the `qortal.aip` file then you might need to adjust
 configured paths, or create a dummy `D:` drive with the expected layout.
 
+Opening the aip file from within a clone of the qortal repo also works, if you have a separate windows machine setup to do the build. 
+
+You May need to change the location of the 'jre64' files inside Advanced Installer, if it is set to a path that your build machine doesn't have. 
+
+The Java Memory Arguments can be set manually, but as of December 2024 they have been reset back to system defaults. This should include G1GC Garbage Collector.
+
 Typical build procedure:
 
 * Place the `qortal.jar` file in `Install-Files\`

lib/org/hsqldb/hsqldb/2.7.4/Notes.txt

@@ -0,0 +1,5 @@
+This is the production hsqldb-2.7.4 with the manifest file updated 
+
+Sealed: false
+
+Allows the addition of the custom Qortal HSQLDBPool and Monitoring Classes

log4j2.properties

@@ -6,6 +6,34 @@ rootLogger.level = info
 rootLogger.appenderRef.console.ref = stdout
 rootLogger.appenderRef.rolling.ref = FILE
 
+### additional debug options (in case of problems eg. 202411
+
+#to see more QDN details - add the stuff below
+#logger.arbitrary.name = org.qortal.arbitrary
+#logger.arbitrary.level = trace
+
+#to see more QDN networking details - add stuff below
+#logger.arbitrarycontroller.name = org.qortal.controller.arbitrary
+#logger.arbitrarycontroller.level = debug
+
+# Support optional, Network Task debugging
+#logger.networkTask.name = org.qortal.network.task
+#logger.networkTask.level = debug
+
+# Support optional, Network Task tracing
+#logger.networkTask.name = org.qortal.network.task
+#logger.networkTask.level = trace
+
+# Support optional, Block debugging
+#logger.block.name = org.qortal.block
+#logger.block.level = debug
+
+# Support optional, Block tracing
+#logger.block.name = org.qortal.block
+#logger.block.level = trace
+
+### end additional debug options
+
 # Suppress extraneous bitcoinj library output
 logger.bitcoinj.name = org.bitcoinj
 logger.bitcoinj.level = error
@@ -18,6 +46,10 @@ logger.hsqldb.level = warn
 logger.hsqldbRepository.name = org.qortal.repository.hsqldb
 logger.hsqldbRepository.level = debug
 
+## Support optional, controller repository debugging
+#logger.controllerRepository.name = org.qortal.controller.repository
+#logger.controllerRepository.level = debug
+
 # Suppress extraneous Jersey warning
 logger.jerseyInject.name = org.glassfish.jersey.internal.inject.Providers
 logger.jerseyInject.level = off

pom.xml

@@ -3,7 +3,7 @@
 	<modelVersion>4.0.0</modelVersion>
 	<groupId>org.qortal</groupId>
 	<artifactId>qortal</artifactId>
-	<version>4.6.0</version>
+	<version>5.0.2</version>
 	<packaging>jar</packaging>
 	<properties>
 		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -16,19 +16,19 @@
 		<ciyam-at.version>1.4.2</ciyam-at.version>
 		<commons-net.version>3.8.0</commons-net.version>
 		<commons-text.version>1.12.0</commons-text.version>
-		<commons-io.version>2.17.0</commons-io.version>
+		<commons-io.version>2.18.0</commons-io.version>
 		<commons-compress.version>1.27.1</commons-compress.version>
 		<commons-lang3.version>3.17.0</commons-lang3.version>
 		<dagger.version>1.2.2</dagger.version>
 		<extendedset.version>0.12.3</extendedset.version>
 		<git-commit-id-plugin.version>4.9.10</git-commit-id-plugin.version>
-		<grpc.version>1.66.0</grpc.version>
-		<guava.version>33.3.0-jre</guava.version>
+		<grpc.version>1.68.1</grpc.version>
+		<guava.version>33.3.1-jre</guava.version>
 		<hamcrest-library.version>2.2</hamcrest-library.version>
 		<homoglyph.version>1.2.1</homoglyph.version>
-		<hsqldb.version>2.5.1</hsqldb.version>
-		<icu4j.version>75.1</icu4j.version>
-		<java-diff-utils.version>4.12</java-diff-utils.version>
+		<hsqldb.version>2.7.4</hsqldb.version>
+		<icu4j.version>76.1</icu4j.version>
+		<java-diff-utils.version>4.15</java-diff-utils.version>
 		<javax.servlet-api.version>4.0.1</javax.servlet-api.version>
 		<jaxb-runtime.version>2.3.9</jaxb-runtime.version>
 		<jersey.version>2.42</jersey.version>
@@ -45,19 +45,25 @@
 		<maven-dependency-plugin.version>3.6.1</maven-dependency-plugin.version>
 		<maven-jar-plugin.version>3.4.2</maven-jar-plugin.version>
 		<maven-package-info-plugin.version>1.1.0</maven-package-info-plugin.version>
-		<maven-plugin.version>2.17.1</maven-plugin.version>
+		<maven-plugin.version>2.18.0</maven-plugin.version>
 		<maven-reproducible-build-plugin.version>0.17</maven-reproducible-build-plugin.version>
 		<maven-resources-plugin.version>3.3.1</maven-resources-plugin.version>
 		<maven-shade-plugin.version>3.6.0</maven-shade-plugin.version>
-		<maven-surefire-plugin.version>3.5.0</maven-surefire-plugin.version>
+		<maven-install-plugin.version>3.1.3</maven-install-plugin.version>
+		<maven-surefire-plugin.version>3.5.2</maven-surefire-plugin.version>
 		<protobuf.version>3.25.3</protobuf.version>
 		<replacer.version>1.5.3</replacer.version>
+		<reticulum.version>e47f1be</reticulum.version>
 		<simplemagic.version>1.17</simplemagic.version>
-		<slf4j.version>1.7.36</slf4j.version>
 		<swagger-api.version>2.0.10</swagger-api.version>
-		<swagger-ui.version>5.17.14</swagger-ui.version>
+		<swagger-ui.version>5.18.2</swagger-ui.version>
 		<upnp.version>1.2</upnp.version>
 		<xz.version>1.10</xz.version>
+   	<lombok.version>1.18.30</lombok.version>
+   	<jackson.version>2.16.1</jackson.version>
+		<slf4j.version>2.0.12</slf4j.version>
+    <nitrite.version>4.3.0</nitrite.version>
+    <junit.version>5.9.2</junit.version>
 	</properties>
 	<build>
 		<sourceDirectory>src/main/java</sourceDirectory>
@@ -303,6 +309,30 @@
 					</archive>
 				</configuration>
 			</plugin>
+			<!-- Copy modified hsqldb.jar to install / modified MANIFEST.MF-->
+			<plugin>
+				<groupId>org.apache.maven.plugins</groupId>
+				<artifactId>maven-install-plugin</artifactId>
+				<version>${maven-install-plugin.version}</version>
+				<configuration>
+					<groupId>org.hsqldb</groupId>
+					<artifactId>hsqldb</artifactId>
+					<version>${hsqldb.version}</version>
+					<packaging>jar</packaging>
+				</configuration>
+				<executions>
+					<execution>
+						<phase>install</phase>
+						<goals>
+							<goal>install-file</goal>
+						</goals>
+						<configuration>
+							<file>${project.basedir}/lib/org/hsqldb/hsqldb/${hsqldb.version}/hsqldb.jar</file>
+						</configuration>
+					</execution>
+				</executions>
+			</plugin>
+
 			<plugin>
 				<groupId>org.apache.maven.plugins</groupId>
 				<artifactId>maven-shade-plugin</artifactId>
@@ -426,13 +456,41 @@
 			<id>project.local</id>
 			<name>project</name>
 			<url>file:${project.basedir}/lib</url>
+      <snapshots>
+        <enabled>true</enabled>
+        <updatePolicy>always</updatePolicy>
+      </snapshots>
 		</repository>
 		<!-- jitpack for build-on-demand of altcoinj -->
 		<repository>
 			<id>jitpack.io</id>
 			<url>https://jitpack.io</url>
+      <snapshots>
+        <enabled>true</enabled>
+        <updatePolicy>always</updatePolicy>
+      </snapshots>
 		</repository>
 	</repositories>
+	<!--
+    <dependencyManagement>
+      <dependencies>
+        <dependency>
+          <groupId>org.dizitart</groupId>
+          <artifactId>nitrite-bom</artifactId>
+          <version>${nitrite.version}</version>
+          <scope>import</scope>
+          <type>pom</type>
+        </dependency>
+        <dependency>
+          <groupId>org.apache.logging.log4j</groupId>
+          <artifactId>log4j-bom</artifactId>
+          <version>${log4j.version}</version>
+          <scope>import</scope>
+          <type>pom</type>
+        </dependency>
+      </dependencies>
+  	</dependencyManagement>
+    -->
 	<dependencies>
 		<!-- https://mvnrepository.com/artifact/org.codehaus.mojo/build-helper-maven-plugin -->
 		<dependency>
@@ -450,7 +508,7 @@
 			<scope>provided</scope>
 			<!-- needed for build, not for runtime -->
 		</dependency>
-		<!-- HSQLDB for repository -->
+		<!-- HSQLDB for repository should use local version with Sealed: false -->
 		<dependency>
 			<groupId>org.hsqldb</groupId>
 			<artifactId>hsqldb</artifactId>
@@ -480,6 +538,13 @@
 			<artifactId>altcoinj</artifactId>
 			<version>${altcoinj.version}</version>
 		</dependency>
+		<!-- Build Reticulum from Source -->
+		<dependency>
+			<!--<groupId>com.github.sergst83</groupId>-->
+			<groupId>com.github.jschulthess</groupId>
+			<artifactId>reticulum-network-stack</artifactId>
+			<version>${reticulum.version}</version>
+		</dependency>
 		<!-- Utilities -->
 		<dependency>
 			<groupId>com.googlecode.json-simple</groupId>
@@ -558,35 +623,33 @@
 			<artifactId>guava</artifactId>
 			<version>${guava.version}</version>
 		</dependency>
-		<!-- Logging: log4j2 -->
-		<dependency>
-			<groupId>org.apache.logging.log4j</groupId>
-			<artifactId>log4j-core</artifactId>
-			<version>${log4j.version}</version>
-		</dependency>
-		<dependency>
-			<groupId>org.apache.logging.log4j</groupId>
-			<artifactId>log4j-api</artifactId>
-			<version>${log4j.version}</version>
-		</dependency>
-		<!-- redirect slf4j to log4j2 -->
-		<dependency>
-			<groupId>org.apache.logging.log4j</groupId>
-			<artifactId>log4j-slf4j-impl</artifactId>
-			<version>${log4j.version}</version>
-		</dependency>
-		<!-- redirect java.utils.logging to log4j2 -->
-		<dependency>
-			<groupId>org.apache.logging.log4j</groupId>
-			<artifactId>log4j-jul</artifactId>
-			<version>${log4j.version}</version>
-		</dependency>
-		<!-- Logging: slf4j used by Jetty/Jersey -->
+    <!-- logging: slf4j -->
 		<dependency>
 			<groupId>org.slf4j</groupId>
 			<artifactId>slf4j-api</artifactId>
 			<version>${slf4j.version}</version>
 		</dependency>
+    <dependency>
+       <groupId>org.apache.logging.log4j</groupId>
+       <artifactId>log4j-slf4j2-impl</artifactId>
+       <version>${log4j.version}</version>
+    </dependency>
+		<!-- Logging: log4j2 -->
+    <dependency>
+       <groupId>org.apache.logging.log4j</groupId>
+       <artifactId>log4j-core</artifactId>
+       <version>${log4j.version}</version>
+    </dependency>
+    <dependency>
+    <groupId>org.apache.logging.log4j</groupId>
+       <artifactId>log4j-api</artifactId>
+       <version>${log4j.version}</version>
+    </dependency>
+    <dependency>
+       <groupId>org.apache.logging.log4j</groupId>
+       <artifactId>log4j-jul</artifactId>
+       <version>${log4j.version}</version>
+     </dependency> 
 		<!-- Servlet related -->
 		<dependency>
 			<groupId>javax.servlet</groupId>
@@ -721,13 +784,18 @@
 		<!-- BouncyCastle for crypto, including TLS secure networking -->
 		<dependency>
 			<groupId>org.bouncycastle</groupId>
-			<artifactId>bcprov-jdk15on</artifactId>
+			<artifactId>bcprov-jdk15to18</artifactId>
 			<version>${bouncycastle.version}</version>
 		</dependency>
 		<dependency>
 			<groupId>org.bouncycastle</groupId>
-			<artifactId>bctls-jdk15on</artifactId>
+			<artifactId>bctls-jdk15to18</artifactId>
 			<version>${bouncycastle.version}</version>
+		</dependency><!-- https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15to18 -->
+		<dependency>
+		    <groupId>org.bouncycastle</groupId>
+		    <artifactId>bcprov-jdk15to18</artifactId>
+		    <version>${bouncycastle.version}</version>
 		</dependency>
 		<dependency>
 			<groupId>org.jsoup</groupId>
@@ -770,5 +838,128 @@
 			<artifactId>jaxb-runtime</artifactId>
 			<version>${jaxb-runtime.version}</version>
 		</dependency>
+    <dependency>
+      <groupId>com.fasterxml.jackson.core</groupId>
+      <artifactId>jackson-databind</artifactId>
+      <version>${jackson.version}</version>
+    </dependency>
+    <!-- https://mvnrepository.com/artifact/commons-codec/commons-codec -->
+    <dependency>
+      <groupId>commons-codec</groupId>
+      <artifactId>commons-codec</artifactId>
+      <version>1.16.1</version>
+    </dependency>
+    <!-- already declared earlier
+    <dependency>
+      <groupId>org.apache.commons</groupId>
+      <artifactId>commons-lang3</artifactId>
+      <version>3.14.0</version>
+    </dependency>
+    -->
+    <dependency>
+      <groupId>org.apache.commons</groupId>
+      <artifactId>commons-collections4</artifactId>
+      <version>4.4</version>
+    </dependency>
+    <!-- already declared earlier
+    <dependency>
+      <groupId>org.apache.commons</groupId>
+      <artifactId>commons-compress</artifactId>
+      <version>1.26.0</version>
+    </dependency>
+    <dependency>
+      <groupId>org.slf4j</groupId>
+      <artifactId>slf4j-api</artifactId>
+      <version>${slf4j.version}</version>
+    </dependency>
+    -->
+    <!-- note: covered ? -->
+    <dependency>
+      <groupId>com.fasterxml.jackson.dataformat</groupId>
+      <artifactId>jackson-dataformat-yaml</artifactId>
+      <version>${jackson.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>org.yaml</groupId>
+      <artifactId>snakeyaml</artifactId>
+      <version>2.2</version>
+    </dependency>
+    <dependency>
+      <groupId>org.msgpack</groupId>
+      <artifactId>jackson-dataformat-msgpack</artifactId>
+      <version>0.9.8</version>
+    </dependency>
+    <dependency>
+      <groupId>org.projectlombok</groupId>
+      <artifactId>lombok</artifactId>
+      <version>${lombok.version}</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.bouncycastle</groupId>
+      <artifactId>bcpkix-jdk15on</artifactId>
+      <version>1.70</version>
+    </dependency>
+    <dependency>
+      <groupId>com.macasaet.fernet</groupId>
+      <artifactId>fernet-java8</artifactId>
+      <version>1.5.0</version>
+    </dependency>
+    <dependency>
+      <groupId>com.igormaznitsa</groupId>
+      <artifactId>jbbp</artifactId>
+      <version>2.0.6</version>
+    </dependency>
+    <dependency>
+      <groupId>io.netty</groupId>
+      <artifactId>netty-all</artifactId>
+      <!--<version>4.1.106.Final</version>-->
+      <version>5.0.0.Alpha2</version>
+    </dependency>
+    <dependency>
+      <groupId>com.github.seancfoley</groupId>
+      <artifactId>ipaddress</artifactId>
+      <version>5.4.2</version>
+    </dependency>
+    <!-- Nitrite Modules -->
+    <dependency>
+        <groupId>org.dizitart</groupId>
+        <artifactId>nitrite</artifactId>
+        <version>${nitrite.version}</version>
+    </dependency>
+    <dependency>
+        <groupId>org.dizitart</groupId>
+        <artifactId>nitrite-mvstore-adapter</artifactId>
+        <version>${nitrite.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>com.h2database</groupId>
+      <artifactId>h2</artifactId>
+      <version>2.3.230</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>com.h2database</groupId>
+      <artifactId>h2-mvstore</artifactId>
+      <version>2.3.230</version>
+    </dependency>
+    <dependency>
+      <groupId>org.junit.jupiter</groupId>
+      <artifactId>junit-jupiter</artifactId>
+      <version>${junit.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mockito</groupId>
+      <artifactId>mockito-junit-jupiter</artifactId>
+      <version>5.10.0</version>
+      <scope>test</scope>
+    </dependency>
+
+    <dependency>
+      <groupId>org.apache.tika</groupId>
+      <artifactId>tika-core</artifactId>
+      <version>3.1.0</version> 
+      </dependency>
 	</dependencies>
 </project>

src/main/java/org/hsqldb/jdbc/HSQLDBPoolMonitored.java

@@ -0,0 +1,173 @@
+package org.hsqldb.jdbc;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.hsqldb.jdbc.pool.JDBCPooledConnection;
+import org.qortal.data.system.DbConnectionInfo;
+import org.qortal.repository.hsqldb.HSQLDBRepositoryFactory;
+
+import javax.sql.ConnectionEvent;
+import javax.sql.PooledConnection;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+
+/**
+ * Class HSQLDBPoolMonitored
+ *
+ * This class uses the same logic as HSQLDBPool. The only difference is it monitors the state of every connection
+ * to the database. This is used for debugging purposes only.
+ */
+public class HSQLDBPoolMonitored extends HSQLDBPool {
+
+	private static final Logger LOGGER = LogManager.getLogger(HSQLDBRepositoryFactory.class);
+
+	private static final String EMPTY = "Empty";
+	private static final String AVAILABLE = "Available";
+	private static final String ALLOCATED = "Allocated";
+
+	private ConcurrentHashMap<Integer, DbConnectionInfo> infoByIndex;
+
+	public HSQLDBPoolMonitored(int poolSize) {
+		super(poolSize);
+
+		this.infoByIndex = new ConcurrentHashMap<>(poolSize);
+	}
+
+	/**
+	 * Tries to retrieve a new connection using the properties that have already been
+	 * set.
+	 *
+	 * @return  a connection to the data source, or null if no spare connections in pool
+	 * @exception SQLException if a database access error occurs
+	 */
+	public Connection tryConnection() throws SQLException {
+		for (int i = 0; i < states.length(); i++) {
+			if (states.compareAndSet(i, RefState.available, RefState.allocated)) {
+				JDBCPooledConnection pooledConnection = connections[i];
+
+				if (pooledConnection == null)
+					// Probably shutdown situation
+					return null;
+
+				infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
+
+				return pooledConnection.getConnection();
+			}
+
+			if (states.compareAndSet(i, RefState.empty, RefState.allocated)) {
+				try {
+					JDBCPooledConnection pooledConnection = (JDBCPooledConnection) source.getPooledConnection();
+
+					if (pooledConnection == null)
+						// Probably shutdown situation
+						return null;
+
+					pooledConnection.addConnectionEventListener(this);
+					pooledConnection.addStatementEventListener(this);
+					connections[i] = pooledConnection;
+
+					infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
+
+					return pooledConnection.getConnection();
+				} catch (SQLException e) {
+					states.set(i, RefState.empty);
+					infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
+				}
+			}
+		}
+
+		return null;
+	}
+
+	public Connection getConnection() throws SQLException {
+		int var1 = 300;
+		if (this.source.loginTimeout != 0) {
+			var1 = this.source.loginTimeout * 10;
+		}
+
+		if (this.closed) {
+			throw new SQLException("connection pool is closed");
+		} else {
+			for(int var2 = 0; var2 < var1; ++var2) {
+				for(int var3 = 0; var3 < this.states.length(); ++var3) {
+					if (this.states.compareAndSet(var3, 1, 2)) {
+						infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
+						return this.connections[var3].getConnection();
+					}
+
+					if (this.states.compareAndSet(var3, 0, 2)) {
+						try {
+							JDBCPooledConnection var4 = (JDBCPooledConnection)this.source.getPooledConnection();
+							var4.addConnectionEventListener(this);
+							var4.addStatementEventListener(this);
+							this.connections[var3] = var4;
+
+							infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), ALLOCATED));
+
+							return this.connections[var3].getConnection();
+						} catch (SQLException var6) {
+							this.states.set(var3, 0);
+							infoByIndex.put(var3, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
+						}
+					}
+				}
+
+				try {
+					Thread.sleep(100L);
+				} catch (InterruptedException var5) {
+				}
+			}
+
+			throw JDBCUtil.invalidArgument();
+		}
+	}
+
+	public void connectionClosed(ConnectionEvent event) {
+		PooledConnection connection = (PooledConnection) event.getSource();
+
+		for (int i = 0; i < connections.length; i++) {
+			if (connections[i] == connection) {
+				states.set(i, RefState.available);
+				infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), AVAILABLE));
+				break;
+			}
+		}
+	}
+
+	public void connectionErrorOccurred(ConnectionEvent event) {
+		PooledConnection connection = (PooledConnection) event.getSource();
+
+		for (int i = 0; i < connections.length; i++) {
+			if (connections[i] == connection) {
+				states.set(i, RefState.allocated);
+				connections[i] = null;
+				states.set(i, RefState.empty);
+				infoByIndex.put(i, new DbConnectionInfo(System.currentTimeMillis(), Thread.currentThread().getName(), EMPTY));
+				break;
+			}
+		}
+	}
+
+	public List<DbConnectionInfo> getDbConnectionsStates() {
+
+		return infoByIndex.values().stream()
+			.sorted(Comparator.comparingLong(DbConnectionInfo::getUpdated))
+			.collect(Collectors.toList());
+	}
+
+	private int findConnectionIndex(ConnectionEvent connectionEvent) {
+		PooledConnection pooledConnection = (PooledConnection) connectionEvent.getSource();
+
+		for(int i = 0; i < this.connections.length; ++i) {
+			if (this.connections[i] == pooledConnection) {
+				return i;
+			}
+		}
+
+		return -1;
+	}
+}

src/main/java/org/qortal/ApplyRestart.java

@@ -1,14 +1,17 @@
 package org.qortal;
 
+import org.apache.commons.io.FileUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.bouncycastle.jce.provider.BouncyCastleProvider;
 import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider;
 import org.qortal.api.ApiKey;
 import org.qortal.api.ApiRequest;
+import org.qortal.controller.Controller;
 import org.qortal.controller.RestartNode;
 import org.qortal.settings.Settings;
 
+import java.io.File;
 import java.io.IOException;
 import java.lang.management.ManagementFactory;
 import java.nio.file.Files;
@@ -16,6 +19,8 @@ import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.security.Security;
 import java.util.*;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
 import java.util.stream.Collectors;
 
 import static org.qortal.controller.RestartNode.AGENTLIB_JVM_HOLDER_ARG;
@@ -38,7 +43,7 @@ public class ApplyRestart {
 	private static final String JAVA_TOOL_OPTIONS_NAME = "JAVA_TOOL_OPTIONS";
 	private static final String JAVA_TOOL_OPTIONS_VALUE = "";
 
-	private static final long CHECK_INTERVAL = 10 * 1000L; // ms
+	private static final long CHECK_INTERVAL = 30 * 1000L; // ms
 	private static final int MAX_ATTEMPTS = 12;
 
 	public static void main(String[] args) {
@@ -51,21 +56,38 @@ public class ApplyRestart {
 		else
 			Settings.getInstance();
 
-		LOGGER.info("Applying restart...");
+		LOGGER.info("Applying restart this can take up to 5 minutes...");
 
 		// Shutdown node using API
 		if (!shutdownNode())
 			return;
 
-		// Restart node
-		restartNode(args);
+		try {
+			// Give some time for shutdown
+			TimeUnit.SECONDS.sleep(60);
 
-		LOGGER.info("Restarting...");
+			// Remove blockchain lock if exist
+			ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
+			if (blockchainLock.isLocked())
+				blockchainLock.unlock();
+
+			// Remove blockchain lock file if still exist
+			TimeUnit.SECONDS.sleep(60);
+			deleteLock();
+
+			// Restart node
+			TimeUnit.SECONDS.sleep(15);
+			restartNode(args);
+
+			LOGGER.info("Restarting...");
+		} catch (InterruptedException e) {
+			LOGGER.error("Unable to restart", e);
+		}
 	}
 
 	private static boolean shutdownNode() {
 		String baseUri = "http://localhost:" + Settings.getInstance().getApiPort() + "/";
-		LOGGER.info(() -> String.format("Shutting down node using API via %s", baseUri));
+		LOGGER.debug(() -> String.format("Shutting down node using API via %s", baseUri));
 
 		// The /admin/stop endpoint requires an API key, which may or may not be already generated
 		boolean apiKeyNewlyGenerated = false;
@@ -95,10 +117,17 @@ public class ApplyRestart {
 			String response = ApiRequest.perform(baseUri + "admin/stop", params);
 			if (response == null) {
 				// No response - consider node shut down
+				try {
+					TimeUnit.SECONDS.sleep(30);
+				} catch (InterruptedException e) {
+					throw new RuntimeException(e);
+				}
+
 				if (apiKeyNewlyGenerated) {
 					// API key was newly generated for restarting node, so we need to remove it
 					ApplyRestart.removeGeneratedApiKey();
 				}
+
 				return true;
 			}
 
@@ -134,7 +163,22 @@ public class ApplyRestart {
 			apiKey.delete();
 
 		} catch (IOException e) {
-			LOGGER.info("Error loading or deleting API key: {}", e.getMessage());
+			LOGGER.error("Error loading or deleting API key: {}", e.getMessage());
+		}
+	}
+
+	private static void deleteLock() {
+		// Get the repository path from settings
+		String repositoryPath = Settings.getInstance().getRepositoryPath();
+		LOGGER.debug(String.format("Repository path is: %s", repositoryPath));
+
+		try {
+			Path root = Paths.get(repositoryPath);
+			File lockFile = new File(root.resolve("blockchain.lck").toUri());
+			LOGGER.debug("Lockfile is: {}", lockFile);
+			FileUtils.forceDelete(FileUtils.getFile(lockFile));
+		} catch (IOException e) {
+			LOGGER.debug("Error deleting blockchain lock file: {}", e.getMessage());
 		}
 	}
 
@@ -150,9 +194,10 @@ public class ApplyRestart {
 
 		List<String> javaCmd;
 		if (Files.exists(exeLauncher)) {
-			javaCmd = Arrays.asList(exeLauncher.toString());
+			javaCmd = List.of(exeLauncher.toString());
 		} else {
 			javaCmd = new ArrayList<>();
+
 			// Java runtime binary itself
 			javaCmd.add(javaBinary.toString());
 

src/main/java/org/qortal/account/Account.java

@@ -2,23 +2,30 @@ package org.qortal.account;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.qortal.api.resource.TransactionsResource;
 import org.qortal.block.BlockChain;
 import org.qortal.controller.LiteNode;
 import org.qortal.data.account.AccountBalanceData;
 import org.qortal.data.account.AccountData;
 import org.qortal.data.account.RewardShareData;
 import org.qortal.data.naming.NameData;
+import org.qortal.data.transaction.TransactionData;
 import org.qortal.repository.DataException;
 import org.qortal.repository.GroupRepository;
 import org.qortal.repository.NameRepository;
 import org.qortal.repository.Repository;
 import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Groups;
 
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 
+import java.util.Comparator;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
+import java.util.Optional;
 
 import static org.qortal.utils.Amounts.prettyAmount;
 
@@ -198,66 +205,85 @@ public class Account {
 
 	/** Returns whether account can be considered a "minting account".
 	 * <p>
-	 * To be considered a "minting account", the account needs to pass all of these tests:<br>
+	 * To be considered a "minting account", the account needs to pass some of these tests:<br>
 	 * <ul>
 	 * <li>account's level is at least <tt>minAccountLevelToMint</tt> from blockchain config</li>
-	 * <li>account's address have registered a name</li>
-	 * <li>account's address is member of minter group</li>
+	 * <li>account's address has registered a name</li>
+	 * <li>account's address is a member of the minter group</li>
 	 * </ul>
 	 *
+	 * @param isGroupValidated true if this account has already been validated for MINTER Group membership
 	 * @return true if account can be considered "minting account"
 	 * @throws DataException
 	 */
-	public boolean canMint() throws DataException {
+	public boolean canMint(boolean isGroupValidated) throws DataException {
 		AccountData accountData = this.repository.getAccountRepository().getAccount(this.address);
 		NameRepository nameRepository = this.repository.getNameRepository();
 		GroupRepository groupRepository = this.repository.getGroupRepository();
+		String myAddress = accountData.getAddress();
 
 		int blockchainHeight = this.repository.getBlockRepository().getBlockchainHeight();
-		int nameCheckHeight = BlockChain.getInstance().getOnlyMintWithNameHeight();
-		int levelToMint = BlockChain.getInstance().getMinAccountLevelToMint();
+
+		int levelToMint;
+
+		if( blockchainHeight >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() ) {
+			levelToMint = 0;
+		}
+		else {
+			levelToMint = BlockChain.getInstance().getMinAccountLevelToMint();
+		}
+
 		int level = accountData.getLevel();
-		int groupIdToMint = BlockChain.getInstance().getMintingGroupId();
+		List<Integer> groupIdsToMint = Groups.getGroupIdsToMint( BlockChain.getInstance(), blockchainHeight );
+		int nameCheckHeight = BlockChain.getInstance().getOnlyMintWithNameHeight();
 		int groupCheckHeight = BlockChain.getInstance().getGroupMemberCheckHeight();
+		int removeNameCheckHeight = BlockChain.getInstance().getRemoveOnlyMintWithNameHeight();
 
-		String myAddress = accountData.getAddress();
-		List<NameData> myName = nameRepository.getNamesByOwner(myAddress);
-		boolean isMember = groupRepository.memberExists(groupIdToMint, myAddress);
+		// Can only mint if:
+		// Account's level is at least minAccountLevelToMint from blockchain config
+		if (blockchainHeight < nameCheckHeight) {
+			if (Account.isFounder(accountData.getFlags())) {
+				return accountData.getBlocksMintedPenalty() == 0;
+			} else {
+				return level >= levelToMint;
+			}
+		}
 
-		if (accountData == null)
-			return false;
+		// Can only mint on onlyMintWithNameHeight from blockchain config if:
+		// Account's level is at least minAccountLevelToMint from blockchain config
+		// Account's address has registered a name
+		if (blockchainHeight >= nameCheckHeight && blockchainHeight < groupCheckHeight) {
+			List<NameData> myName = nameRepository.getNamesByOwner(myAddress);
+			if (Account.isFounder(accountData.getFlags())) {
+				return accountData.getBlocksMintedPenalty() == 0 && !myName.isEmpty();
+			} else {
+				return level >= levelToMint && !myName.isEmpty();
+			}
+		}
 
-		// Can only mint if level is at least minAccountLevelToMint< from blockchain config
-		if (blockchainHeight < nameCheckHeight && level >= levelToMint)
-			return true;
+		// Can only mint on groupMemberCheckHeight from blockchain config if:
+		// Account's level is at least minAccountLevelToMint from blockchain config
+		// Account's address has registered a name
+		// Account's address is a member of the minter group
+		if (blockchainHeight >= groupCheckHeight && blockchainHeight < removeNameCheckHeight) {
+			List<NameData> myName = nameRepository.getNamesByOwner(myAddress);
+			if (Account.isFounder(accountData.getFlags())) {
+				return accountData.getBlocksMintedPenalty() == 0 && !myName.isEmpty() && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
+			} else {
+				return level >= levelToMint && !myName.isEmpty() && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
+			}
+		}
 
-		// Can only mint if have registered a name
-		if (blockchainHeight >= nameCheckHeight && blockchainHeight < groupCheckHeight && level >= levelToMint && !myName.isEmpty())
-			return true;
-
-		// Can only mint if have registered a name and is member of minter group id
-		if (blockchainHeight >= groupCheckHeight && level >= levelToMint && !myName.isEmpty() && isMember)
-			return true;
-
-		// Founders needs to pass same tests like minters
-		if (blockchainHeight < nameCheckHeight &&
-				Account.isFounder(accountData.getFlags()) &&
-				accountData.getBlocksMintedPenalty() == 0)
-			return true;
-
-		if (blockchainHeight >= nameCheckHeight &&
-				blockchainHeight < groupCheckHeight &&
-				Account.isFounder(accountData.getFlags()) &&
-				accountData.getBlocksMintedPenalty() == 0 &&
-				!myName.isEmpty())
-			return true;
-
-		if (blockchainHeight >= groupCheckHeight &&
-				Account.isFounder(accountData.getFlags()) &&
-				accountData.getBlocksMintedPenalty() == 0 &&
-				!myName.isEmpty() &&
-				isMember)
-			return true;
+		// Can only mint on removeOnlyMintWithNameHeight from blockchain config if:
+		// Account's level is at least minAccountLevelToMint from blockchain config
+		// Account's address is a member of the minter group
+		if (blockchainHeight >= removeNameCheckHeight) {
+			if (Account.isFounder(accountData.getFlags())) {
+				return accountData.getBlocksMintedPenalty() == 0 && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
+			} else {
+				return level >= levelToMint && (isGroupValidated || Groups.memberExistsInAnyGroup(groupRepository, groupIdsToMint, myAddress));
+			}
+		}
 
 		return false;
 	}
@@ -272,7 +298,6 @@ public class Account {
 		return this.repository.getAccountRepository().getBlocksMintedPenaltyCount(this.address);
 	}
 
-
 	/** Returns whether account can build reward-shares.
 	 * <p>
 	 * To be able to create reward-shares, the account needs to pass at least one of these tests:<br>
@@ -286,6 +311,7 @@ public class Account {
 	 */
 	public boolean canRewardShare() throws DataException {
 		AccountData accountData = this.repository.getAccountRepository().getAccount(this.address);
+
 		if (accountData == null)
 			return false;
 
@@ -296,6 +322,9 @@ public class Account {
 		if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
 			return true;
 
+		if( this.repository.getBlockRepository().getBlockchainHeight() >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() )
+			return true;
+
 		return false;
 	}
 
@@ -339,10 +368,164 @@ public class Account {
 	}
 
 	/**
-	 * Returns 'effective' minting level, or zero if reward-share does not exist.
+	 * Get Primary Name
+	 *
+	 * @return the primary name for this address if present, otherwise empty
+	 *
+	 * @throws DataException
+	 */
+	public Optional<String> getPrimaryName() throws DataException {
+
+		return this.repository.getNameRepository().getPrimaryName(this.address);
+	}
+
+	/**
+	 * Remove Primary Name
+	 *
+	 * @throws DataException
+	 */
+	public void removePrimaryName() throws DataException {
+		this.repository.getNameRepository().removePrimaryName(this.address);
+	}
+
+	/**
+	 * Reset Primary Name
+	 *
+	 * Set primary name based on the names (and their history) this account owns.
+	 *
+	 * @param confirmationStatus the status of the transactions for the determining the primary name
+	 *
+	 * @return the primary name, empty if their isn't one
+	 *
+	 * @throws DataException
+	 */
+	public Optional<String> resetPrimaryName(TransactionsResource.ConfirmationStatus confirmationStatus) throws DataException {
+		Optional<String> primaryName = determinePrimaryName(confirmationStatus);
+
+		if(primaryName.isPresent()) {
+			return setPrimaryName(primaryName.get());
+		}
+		else {
+			return primaryName;
+		}
+	}
+
+	/**
+	 * Determine Primary Name
+	 *
+	 * Determine primary name based on a list of registered names.
+	 *
+	 * @param confirmationStatus the status of the transactions for this determination
+	 *
+	 * @return the primary name, empty if there is no primary name
+	 *
+	 * @throws DataException
+	 */
+	public Optional<String> determinePrimaryName(TransactionsResource.ConfirmationStatus confirmationStatus) throws DataException {
+
+		// all registered names for the owner
+		List<NameData> names = this.repository.getNameRepository().getNamesByOwner(this.address);
+
+		Optional<String> primaryName;
+
+		// if no registered names, the no primary name possible
+		if (names.isEmpty()) {
+			primaryName = Optional.empty();
+		}
+		// if names
+		else {
+			// if one name, then that is the primary name
+			if (names.size() == 1) {
+				primaryName = Optional.of( names.get(0).getName() );
+			}
+			// if more than one name, then seek the earliest name acquisition that was never released
+			else {
+				Map<String, TransactionData> txByName = new HashMap<>(names.size());
+
+				// for each name, get the latest transaction
+				for (NameData nameData : names) {
+
+					// since the name is currently registered to the owner,
+					// we assume the latest transaction involving this name was the transaction that the acquired
+					// name through registration, purchase or update
+					Optional<TransactionData> latestTransaction
+							= this.repository
+							.getTransactionRepository()
+							.getTransactionsInvolvingName(
+									nameData.getName(),
+									confirmationStatus
+							)
+							.stream()
+							.sorted(Comparator.comparing(
+									TransactionData::getTimestamp).reversed()
+							)
+							.findFirst(); // first is the last, since it was reversed
+
+					// if there is a latest transaction, expected for all registered names
+					if (latestTransaction.isPresent()) {
+						txByName.put(nameData.getName(), latestTransaction.get());
+					}
+					// if there is no latest transaction, then
+					else {
+						LOGGER.warn("No matching transaction for name: " + nameData.getName());
+					}
+				}
+
+				// get the first name aqcuistion for this address
+				Optional<Map.Entry<String, TransactionData>> firstNameEntry
+						= txByName.entrySet().stream().sorted(Comparator.comparing(entry -> entry.getValue().getTimestamp())).findFirst();
+
+				// if their is a name acquisition, then the first one is the primary name
+				if (firstNameEntry.isPresent()) {
+					primaryName = Optional.of( firstNameEntry.get().getKey() );
+				}
+				// if there is no nameacquistion, then there is no primary name
+				else {
+					primaryName =  Optional.empty();
+				}
+			}
+		}
+		return primaryName;
+	}
+
+	/**
+	 * Set Primary Name
+	 *
+	 * @param primaryName the primary to set to this address
+	 *
+	 * @return the primary name if successful, empty if unsuccessful
+	 *
+	 * @throws DataException
+	 */
+	public Optional<String> setPrimaryName( String primaryName ) throws DataException {
+		int changed = this.repository.getNameRepository().setPrimaryName(this.address, primaryName);
+
+		return changed > 0 ? Optional.of(primaryName) : Optional.empty();
+	}
+
+	/**
+	 * Returns reward-share minting address, or unknown if reward-share does not exist.
 	 * 
 	 * @param repository
 	 * @param rewardSharePublicKey
+	 * @return address or unknown
+	 * @throws DataException
+	 */
+	public static String getRewardShareMintingAddress(Repository repository, byte[] rewardSharePublicKey) throws DataException {
+		// Find actual minter address
+		RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(rewardSharePublicKey);
+
+		if (rewardShareData == null)
+			return "Unknown";
+
+		return rewardShareData.getMinter();
+	}
+
+	/**
+	 * Returns 'effective' minting level, or zero if reward-share does not exist.
+	 *
+	 * @param repository
+	 * @param rewardSharePublicKey
 	 * @return 0+
 	 * @throws DataException
 	 */
@@ -355,6 +538,7 @@ public class Account {
 		Account rewardShareMinter = new Account(repository, rewardShareData.getMinter());
 		return rewardShareMinter.getEffectiveMintingLevel();
 	}
+
 	/**
 	 * Returns 'effective' minting level, with a fix for the zero level.
 	 * <p>

src/main/java/org/qortal/api/ApiService.java

@@ -46,6 +46,7 @@ public class ApiService {
 	private ApiService() {
 		this.config = new ResourceConfig();
 		this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource");
+		this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);
@@ -194,8 +195,10 @@ public class ApiService {
 
 			context.addServlet(AdminStatusWebSocket.class, "/websockets/admin/status");
 			context.addServlet(BlocksWebSocket.class, "/websockets/blocks");
+			context.addServlet(DataMonitorSocket.class, "/websockets/datamonitor");
 			context.addServlet(ActiveChatsWebSocket.class, "/websockets/chat/active/*");
 			context.addServlet(ChatMessagesWebSocket.class, "/websockets/chat/messages");
+			context.addServlet(UnsignedFeesSocket.class, "/websockets/crosschain/unsignedfees");
 			context.addServlet(TradeOffersWebSocket.class, "/websockets/crosschain/tradeoffers");
 			context.addServlet(TradeBotWebSocket.class, "/websockets/crosschain/tradebot");
 			context.addServlet(TradePresenceWebSocket.class, "/websockets/crosschain/tradepresence");

src/main/java/org/qortal/api/DevProxyService.java

@@ -40,6 +40,7 @@ public class DevProxyService {
 	private DevProxyService() {
 		this.config = new ResourceConfig();
 		this.config.packages("org.qortal.api.proxy.resource", "org.qortal.api.resource");
+		this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);

src/main/java/org/qortal/api/DomainMapService.java

@@ -39,6 +39,7 @@ public class DomainMapService {
 	private DomainMapService() {
 		this.config = new ResourceConfig();
 		this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource");
+		this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);

src/main/java/org/qortal/api/GatewayService.java

@@ -39,6 +39,7 @@ public class GatewayService {
 	private GatewayService() {
 		this.config = new ResourceConfig();
 		this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource");
+		this.config.register(org.glassfish.jersey.media.multipart.MultiPartFeature.class);
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);

src/main/java/org/qortal/api/HTMLParser.java

@@ -1,14 +1,13 @@
 package org.qortal.api;
 
+
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.jsoup.Jsoup;
 import org.jsoup.nodes.Document;
 import org.jsoup.select.Elements;
 import org.qortal.arbitrary.misc.Service;
-
 import java.util.Objects;
-
 public class HTMLParser {
 
     private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class);
@@ -22,10 +21,11 @@ public class HTMLParser {
     private String identifier;
     private String path;
     private String theme;
+    private String lang;
     private boolean usingCustomRouting;
 
     public HTMLParser(String resourceId, String inPath, String prefix, boolean includeResourceIdInPrefix, byte[] data,
-                      String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting) {
+                      String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting,  String lang) {
         String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : String.format("/%s",inPath);
         this.qdnBase = includeResourceIdInPrefix ? String.format("%s/%s", prefix, resourceId) : prefix;
         this.qdnBaseWithPath = includeResourceIdInPrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : String.format("%s%s", prefix, inPathWithoutFilename);
@@ -36,6 +36,7 @@ public class HTMLParser {
         this.identifier = identifier;
         this.path = inPath;
         this.theme = theme;
+        this.lang = lang;
         this.usingCustomRouting = usingCustomRouting;
     }
 
@@ -61,9 +62,13 @@ public class HTMLParser {
             String identifier = this.identifier != null ? this.identifier.replace("\\", "").replace("\"","\\\"") : "";
             String path = this.path != null ? this.path.replace("\\", "").replace("\"","\\\"") : "";
             String theme = this.theme != null ? this.theme.replace("\\", "").replace("\"","\\\"") : "";
+            String lang = this.lang != null ? this.lang.replace("\\", "").replace("\"", "\\\"") : "";
             String qdnBase = this.qdnBase != null ? this.qdnBase.replace("\\", "").replace("\"","\\\"") : "";
             String qdnBaseWithPath = this.qdnBaseWithPath != null ? this.qdnBaseWithPath.replace("\\", "").replace("\"","\\\"") : "";
-            String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>", qdnContext, theme, service, name, identifier, path, qdnBase, qdnBaseWithPath);
+            String qdnContextVar = String.format(
+                "<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnLang=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>",
+                qdnContext, theme, lang, service, name, identifier, path, qdnBase, qdnBaseWithPath
+              );
             head.get(0).prepend(qdnContextVar);
 
             // Add base href tag

src/main/java/org/qortal/api/model/ApiOnlineAccount.java

@@ -1,7 +1,13 @@
 package org.qortal.api.model;
 
+import org.qortal.account.Account;
+import org.qortal.repository.DataException;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.repository.Repository;
+
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlElement;
 
 // All properties to be converted to JSON via JAXB
 @XmlAccessorType(XmlAccessType.FIELD)
@@ -47,4 +53,31 @@ public class ApiOnlineAccount {
 		return this.recipientAddress;
 	}
 
+	public int getMinterLevelFromPublicKey() {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			return Account.getRewardShareEffectiveMintingLevel(repository, this.rewardSharePublicKey);
+		} catch (DataException e) {
+			return 0;
+		}
+	}
+
+	public boolean getIsMember() {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			return repository.getGroupRepository().memberExists(694, getMinterAddress());
+		} catch (DataException e) {
+			return false;
+		}
+	}
+
+	// JAXB special
+	
+	@XmlElement(name = "minterLevel")
+	protected int getMinterLevel() {
+		return getMinterLevelFromPublicKey();
+	}
+
+	@XmlElement(name = "isMinterMember")
+	protected boolean getMinterMember() {
+		return getIsMember();
+	}
 }

src/main/java/org/qortal/api/model/BlockMintingInfo.java

@@ -9,6 +9,7 @@ import java.math.BigInteger;
 public class BlockMintingInfo {
 
 	public byte[] minterPublicKey;
+	public String minterAddress;
 	public int minterLevel;
 	public int onlineAccountsCount;
 	public BigDecimal maxDistance;
@@ -19,5 +20,4 @@ public class BlockMintingInfo {
 
 	public BlockMintingInfo() {
 	}
-
 }

src/main/java/org/qortal/api/model/CrossChainTradeLedgerEntry.java

@@ -0,0 +1,72 @@
+package org.qortal.api.model;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import org.qortal.data.crosschain.CrossChainTradeData;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
+
+// All properties to be converted to JSON via JAXB
+@XmlAccessorType(XmlAccessType.FIELD)
+public class CrossChainTradeLedgerEntry {
+
+	private String market;
+
+	private String currency;
+
+	@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
+	private long quantity;
+
+	@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
+	private long feeAmount;
+
+	private String feeCurrency;
+
+	@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
+	private long totalPrice;
+
+	private long tradeTimestamp;
+
+	protected CrossChainTradeLedgerEntry() {
+		/* For JAXB */
+	}
+
+	public CrossChainTradeLedgerEntry(String market, String currency, long quantity, long feeAmount, String feeCurrency, long totalPrice, long tradeTimestamp) {
+		this.market = market;
+		this.currency = currency;
+		this.quantity = quantity;
+		this.feeAmount = feeAmount;
+		this.feeCurrency = feeCurrency;
+		this.totalPrice = totalPrice;
+		this.tradeTimestamp = tradeTimestamp;
+	}
+
+	public String getMarket() {
+		return market;
+	}
+
+	public String getCurrency() {
+		return currency;
+	}
+
+	public long getQuantity() {
+		return quantity;
+	}
+
+	public long getFeeAmount() {
+		return feeAmount;
+	}
+
+	public String getFeeCurrency() {
+		return feeCurrency;
+	}
+
+	public long getTotalPrice() {
+		return totalPrice;
+	}
+
+	public long getTradeTimestamp() {
+		return tradeTimestamp;
+	}
+}

src/main/java/org/qortal/api/model/DatasetStatus.java

@@ -0,0 +1,50 @@
+package org.qortal.api.model;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import java.util.Objects;
+
+// All properties to be converted to JSON via JAXB
+@XmlAccessorType(XmlAccessType.FIELD)
+public class DatasetStatus {
+
+    private String name;
+
+    private long count;
+
+    public DatasetStatus() {}
+
+    public DatasetStatus(String name, long count) {
+        this.name = name;
+        this.count = count;
+    }
+
+    public String getName() {
+        return name;
+    }
+
+    public long getCount() {
+        return count;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        DatasetStatus that = (DatasetStatus) o;
+        return count == that.count && Objects.equals(name, that.name);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(name, count);
+    }
+
+    @Override
+    public String toString() {
+        return "DatasetStatus{" +
+                "name='" + name + '\'' +
+                ", count=" + count +
+                '}';
+    }
+}

src/main/java/org/qortal/api/model/crosschain/BitcoinyTBDRequest.java

@@ -304,11 +304,11 @@ public class BitcoinyTBDRequest {
     private String networkName;
 
     /**
-     * Fee Ceiling
+     * Fee Required
      *
-     *  web search, LTC fee ceiling = 1000L
+     *  web search, LTC fee required = 1000L
      */
-    private long feeCeiling;
+    private long feeRequired;
 
     /**
      * Extended Public Key
@@ -570,8 +570,8 @@ public class BitcoinyTBDRequest {
         return this.networkName;
     }
 
-    public long getFeeCeiling() {
-        return this.feeCeiling;
+    public long getFeeRequired() {
+        return this.feeRequired;
     }
 
     public String getExtendedPublicKey() {
@@ -671,7 +671,7 @@ public class BitcoinyTBDRequest {
                 ", minimumOrderAmount=" + minimumOrderAmount +
                 ", feePerKb=" + feePerKb +
                 ", networkName='" + networkName + '\'' +
-                ", feeCeiling=" + feeCeiling +
+                ", feeRequired=" + feeRequired +
                 ", extendedPublicKey='" + extendedPublicKey + '\'' +
                 ", sendAmount=" + sendAmount +
                 ", sendingFeePerByte=" + sendingFeePerByte +

src/main/java/org/qortal/api/proxy/resource/DevProxyServerResource.java

@@ -142,10 +142,20 @@ public class DevProxyServerResource {
             }
         }
 
+        String lang = request.getParameter("lang");
+        if (lang == null || lang.isBlank()) {
+            lang = "en"; // fallback
+        }
+
+        String theme = request.getParameter("theme");
+        if (theme == null || theme.isBlank()) {
+            theme = "light";
+        }
+
         // Parse and modify output if needed
         if (HTMLParser.isHtmlFile(filename)) {
             // HTML file - needs to be parsed
-            HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, "light", true);
+            HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, theme , true, lang);
             htmlParser.addAdditionalHeaderTags();
             response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' ws:; font-src 'self' data:;");
             response.setContentType(con.getContentType());

src/main/java/org/qortal/api/resource/ArbitraryResource.java

@@ -3,6 +3,7 @@ package org.qortal.api.resource;
 import com.google.common.primitives.Bytes;
 import com.j256.simplemagic.ContentInfo;
 import com.j256.simplemagic.ContentInfoUtil;
+
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.Parameter;
 import io.swagger.v3.oas.annotations.media.ArraySchema;
@@ -12,6 +13,7 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.security.SecurityRequirement;
 import io.swagger.v3.oas.annotations.tags.Tag;
+
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.lang3.ArrayUtils;
 import org.apache.logging.log4j.LogManager;
@@ -33,9 +35,13 @@ import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
 import org.qortal.controller.arbitrary.ArbitraryMetadataManager;
 import org.qortal.data.account.AccountData;
 import org.qortal.data.arbitrary.ArbitraryCategoryInfo;
+import org.qortal.data.arbitrary.ArbitraryDataIndexDetail;
+import org.qortal.data.arbitrary.ArbitraryDataIndexScoreKey;
+import org.qortal.data.arbitrary.ArbitraryDataIndexScorecard;
 import org.qortal.data.arbitrary.ArbitraryResourceData;
 import org.qortal.data.arbitrary.ArbitraryResourceMetadata;
 import org.qortal.data.arbitrary.ArbitraryResourceStatus;
+import org.qortal.data.arbitrary.IndexCache;
 import org.qortal.data.naming.NameData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
@@ -59,18 +65,36 @@ import javax.servlet.http.HttpServletResponse;
 import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
+
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileWriter;
 import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
 import java.net.FileNameMap;
 import java.net.URLConnection;
 import java.nio.file.Files;
 import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Comparator;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.zip.GZIPOutputStream;
+
+import org.apache.tika.Tika;
+import org.apache.tika.mime.MimeTypeException;
+import org.apache.tika.mime.MimeTypes;
+
+import javax.ws.rs.core.Response;
+
+import org.glassfish.jersey.media.multipart.FormDataParam;
+import static org.qortal.api.ApiError.REPOSITORY_ISSUE;
 
 @Path("/arbitrary")
 @Tag(name = "Arbitrary")
@@ -172,6 +196,7 @@ public class ArbitraryResource {
 			@Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names,
 			@Parameter(description = "Title (searches title metadata field only)") @QueryParam("title") String title,
 			@Parameter(description = "Description (searches description metadata field only)") @QueryParam("description") String description,
+			@Parameter(description = "Keyword (searches description metadata field by keywords)") @QueryParam("keywords") List<String> keywords,
 			@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
 			@Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly,
 			@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
@@ -212,7 +237,7 @@ public class ArbitraryResource {
 			}
 
 			List<ArbitraryResourceData> resources = repository.getArbitraryRepository()
-					.searchArbitraryResources(service, query, identifier, names, title, description, usePrefixOnly,
+					.searchArbitraryResources(service, query, identifier, names, title, description, keywords, usePrefixOnly,
 							exactMatchNames, defaultRes, mode, minLevel, followedOnly, excludeBlocked, includeMetadata, includeStatus,
 							before, after, limit, offset, reverse);
 
@@ -678,20 +703,20 @@ public class ArbitraryResource {
 					)
 			}
 	)
-	public HttpServletResponse get(@PathParam("service") Service service,
+	public void get(@PathParam("service") Service service,
 								   @PathParam("name") String name,
 								   @QueryParam("filepath") String filepath,
 								   @QueryParam("encoding") String encoding,
 								   @QueryParam("rebuild") boolean rebuild,
 								   @QueryParam("async") boolean async,
-								   @QueryParam("attempts") Integer attempts) {
+								   @QueryParam("attempts") Integer attempts, @QueryParam("attachment") boolean attachment, @QueryParam("attachmentFilename") String attachmentFilename) {
 
 		// Authentication can be bypassed in the settings, for those running public QDN nodes
 		if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
 			Security.checkApiCallAllowed(request);
 		}
 
-		return this.download(service, name, null, filepath, encoding, rebuild, async, attempts);
+		 this.download(service, name, null, filepath, encoding, rebuild, async, attempts, attachment, attachmentFilename);
 	}
 
 	@GET
@@ -711,21 +736,21 @@ public class ArbitraryResource {
 					)
 			}
 	)
-	public HttpServletResponse get(@PathParam("service") Service service,
+	public void get(@PathParam("service") Service service,
 								   @PathParam("name") String name,
 								   @PathParam("identifier") String identifier,
 								   @QueryParam("filepath") String filepath,
 								   @QueryParam("encoding") String encoding,
 								   @QueryParam("rebuild") boolean rebuild,
 								   @QueryParam("async") boolean async,
-								   @QueryParam("attempts") Integer attempts) {
+								   @QueryParam("attempts") Integer attempts, @QueryParam("attachment") boolean attachment, @QueryParam("attachmentFilename") String attachmentFilename) {
 
 		// Authentication can be bypassed in the settings, for those running public QDN nodes
 		if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
 			Security.checkApiCallAllowed(request, null);
 		}
 
-		return this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts);
+		this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts, attachment, attachmentFilename);
 	}
 
 
@@ -870,6 +895,464 @@ public class ArbitraryResource {
 	}
 
 
+	@GET
+	@Path("/check/tmp")
+	@Produces(MediaType.TEXT_PLAIN)
+	@Operation(
+		summary = "Check if the disk has enough disk space for an upcoming upload",
+		responses = {
+			@ApiResponse(description = "OK if sufficient space", responseCode = "200"),
+			@ApiResponse(description = "Insufficient space", responseCode = "507") // 507 = Insufficient Storage
+		}
+	)
+	@SecurityRequirement(name = "apiKey")
+	public Response checkUploadSpace(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
+									 @QueryParam("totalSize") Long totalSize) {
+		Security.checkApiCallAllowed(request);
+	
+		if (totalSize == null || totalSize <= 0) {
+			return Response.status(Response.Status.BAD_REQUEST)
+					.entity("Missing or invalid totalSize parameter").build();
+		}
+	
+		File uploadDir = new File("uploads-temp");
+		if (!uploadDir.exists()) {
+			uploadDir.mkdirs(); // ensure the folder exists
+		}
+
+		long usableSpace = uploadDir.getUsableSpace();
+		long requiredSpace = (long)(((double)totalSize) * 2.2); // estimate for chunks + merge
+
+		if (usableSpace < requiredSpace) {
+			return Response.status(507).entity("Insufficient disk space").build();
+		}
+
+		return Response.ok("Sufficient disk space").build();
+	}
+
+	@POST
+@Path("/{service}/{name}/chunk")
+@Consumes(MediaType.MULTIPART_FORM_DATA)
+@Operation(
+    summary = "Upload a single file chunk to be later assembled into a complete arbitrary resource (no identifier)",
+    requestBody = @RequestBody(
+        required = true,
+        content = @Content(
+            mediaType = MediaType.MULTIPART_FORM_DATA,
+            schema = @Schema(
+                implementation = Object.class
+            )
+        )
+    ),
+    responses = {
+        @ApiResponse(
+            description = "Chunk uploaded successfully",
+            responseCode = "200"
+        ),
+        @ApiResponse(
+            description = "Error writing chunk",
+            responseCode = "500"
+        )
+    }
+)
+@SecurityRequirement(name = "apiKey")
+public Response uploadChunkNoIdentifier(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
+                                        @PathParam("service") String serviceString,
+                                        @PathParam("name") String name,
+                                        @FormDataParam("chunk") InputStream chunkStream,
+                                        @FormDataParam("index") int index) {
+    Security.checkApiCallAllowed(request);
+
+    try {
+		String safeService = Paths.get(serviceString).getFileName().toString();
+        String safeName = Paths.get(name).getFileName().toString();
+
+		java.nio.file.Path tempDir = Paths.get("uploads-temp", safeService, safeName);
+		Files.createDirectories(tempDir);
+		
+        
+
+        java.nio.file.Path chunkFile = tempDir.resolve("chunk_" + index);
+        Files.copy(chunkStream, chunkFile, StandardCopyOption.REPLACE_EXISTING);
+
+        return Response.ok("Chunk " + index + " received").build();
+    } catch (IOException e) {
+		LOGGER.error("Failed to write chunk {} for service '{}' and name '{}'", index, serviceString, name, e);
+        return Response.serverError().entity("Failed to write chunk: " + e.getMessage()).build();
+    }
+}
+
+@POST
+@Path("/{service}/{name}/finalize")
+@Produces(MediaType.TEXT_PLAIN)
+@Operation(
+    summary = "Finalize a chunked upload (no identifier) and build a raw, unsigned, ARBITRARY transaction",
+    responses = {
+        @ApiResponse(
+            description = "raw, unsigned, ARBITRARY transaction encoded in Base58",
+            content = @Content(mediaType = MediaType.TEXT_PLAIN)
+        )
+    }
+)
+@SecurityRequirement(name = "apiKey")
+public String finalizeUploadNoIdentifier(
+    @HeaderParam(Security.API_KEY_HEADER) String apiKey,
+    @PathParam("service") String serviceString,
+    @PathParam("name") String name,
+    @QueryParam("title") String title,
+    @QueryParam("description") String description,
+    @QueryParam("tags") List<String> tags,
+    @QueryParam("category") Category category,
+    @QueryParam("filename") String filename,
+    @QueryParam("fee") Long fee,
+    @QueryParam("preview") Boolean preview,
+    @QueryParam("isZip") Boolean isZip
+) {
+    Security.checkApiCallAllowed(request);
+    java.nio.file.Path tempFile = null;
+    java.nio.file.Path tempDir = null;
+	java.nio.file.Path chunkDir = null;
+    String safeService = Paths.get(serviceString).getFileName().toString();
+	String safeName = Paths.get(name).getFileName().toString();
+
+
+
+    try {
+		chunkDir = Paths.get("uploads-temp", safeService, safeName);
+
+        if (!Files.exists(chunkDir) || !Files.isDirectory(chunkDir)) {
+            throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No chunks found for upload");
+        }
+
+        String safeFilename = (filename == null || filename.isBlank()) ? "qortal-" + NTP.getTime() : filename;
+        tempDir = Files.createTempDirectory("qortal-");
+        String sanitizedFilename = Paths.get(safeFilename).getFileName().toString();
+		tempFile = tempDir.resolve(sanitizedFilename);
+
+        try (OutputStream out = Files.newOutputStream(tempFile, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING)) {
+            byte[] buffer = new byte[65536];
+            for (java.nio.file.Path chunk : Files.list(chunkDir)
+                    .filter(path -> path.getFileName().toString().startsWith("chunk_"))
+                    .sorted(Comparator.comparingInt(path -> {
+                        String name2 = path.getFileName().toString();
+                        String numberPart = name2.substring("chunk_".length());
+                        return Integer.parseInt(numberPart);
+                    })).collect(Collectors.toList())) {
+                try (InputStream in = Files.newInputStream(chunk)) {
+                    int bytesRead;
+                    while ((bytesRead = in.read(buffer)) != -1) {
+                        out.write(buffer, 0, bytesRead);
+                    }
+                }
+            }
+        }
+
+        String detectedExtension = "";
+        String uploadFilename = null;
+        boolean extensionIsValid = false;
+
+        if (filename != null && !filename.isBlank()) {
+            int lastDot = filename.lastIndexOf('.');
+            if (lastDot > 0 && lastDot < filename.length() - 1) {
+                extensionIsValid = true;
+                uploadFilename = filename;
+            }
+        }
+
+        if (!extensionIsValid) {
+            Tika tika = new Tika();
+            String mimeType = tika.detect(tempFile.toFile());
+            try {
+                MimeTypes allTypes = MimeTypes.getDefaultMimeTypes();
+                org.apache.tika.mime.MimeType mime = allTypes.forName(mimeType);
+                detectedExtension = mime.getExtension();
+            } catch (MimeTypeException e) {
+                LOGGER.warn("Could not determine file extension for MIME type: {}", mimeType, e);
+            }
+
+            if (filename != null && !filename.isBlank()) {
+                int lastDot = filename.lastIndexOf('.');
+                String baseName = (lastDot > 0) ? filename.substring(0, lastDot) : filename;
+                uploadFilename = baseName + (detectedExtension != null ? detectedExtension : "");
+            } else {
+                uploadFilename = "qortal-" + NTP.getTime() + (detectedExtension != null ? detectedExtension : "");
+            }
+        }
+
+		Boolean isZipBoolean = false;
+
+		if (isZip != null && isZip) {
+			isZipBoolean = true;
+		}
+        
+
+        // ✅ Call upload with `null` as identifier
+        return this.upload(
+            Service.valueOf(serviceString),
+            name,
+            null, // no identifier
+            tempFile.toString(),
+            null,
+            null,
+            isZipBoolean,
+            fee,
+            uploadFilename,
+            title,
+            description,
+            tags,
+            category,
+            preview
+        );
+
+    } catch (IOException e) {
+		LOGGER.error("Failed to merge chunks for service='{}', name='{}'", serviceString, name, e);
+
+        throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, "Failed to merge chunks: " + e.getMessage());
+    } finally {
+        if (tempDir != null) {
+            try {
+                Files.walk(tempDir)
+                    .sorted(Comparator.reverseOrder())
+                    .map(java.nio.file.Path::toFile)
+                    .forEach(File::delete);
+            } catch (IOException e) {
+                LOGGER.warn("Failed to delete temp directory: {}", tempDir, e);
+            }
+        }
+
+        try {
+            Files.walk(chunkDir)
+                .sorted(Comparator.reverseOrder())
+                .map(java.nio.file.Path::toFile)
+                .forEach(File::delete);
+        } catch (IOException e) {
+            LOGGER.warn("Failed to delete chunk directory: {}", chunkDir, e);
+        }
+    }
+}
+
+
+
+	@POST
+@Path("/{service}/{name}/{identifier}/chunk")
+@Consumes(MediaType.MULTIPART_FORM_DATA)
+@Operation(
+    summary = "Upload a single file chunk to be later assembled into a complete arbitrary resource",
+    requestBody = @RequestBody(
+        required = true,
+        content = @Content(
+            mediaType = MediaType.MULTIPART_FORM_DATA,
+            schema = @Schema(
+                implementation = Object.class
+            )
+        )
+    ),
+    responses = {
+        @ApiResponse(
+            description = "Chunk uploaded successfully",
+            responseCode = "200"
+        ),
+        @ApiResponse(
+            description = "Error writing chunk",
+            responseCode = "500"
+        )
+    }
+)
+@SecurityRequirement(name = "apiKey")
+public Response uploadChunk(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
+                            @PathParam("service") String serviceString,
+                            @PathParam("name") String name,
+                            @PathParam("identifier") String identifier,
+                            @FormDataParam("chunk") InputStream chunkStream,
+                            @FormDataParam("index") int index) {
+    Security.checkApiCallAllowed(request);
+
+    try {
+        String safeService = Paths.get(serviceString).getFileName().toString();
+        String safeName = Paths.get(name).getFileName().toString();
+        String safeIdentifier = Paths.get(identifier).getFileName().toString();
+
+        java.nio.file.Path tempDir = Paths.get("uploads-temp", safeService, safeName, safeIdentifier);
+
+        Files.createDirectories(tempDir);
+
+        java.nio.file.Path chunkFile = tempDir.resolve("chunk_" + index);
+        Files.copy(chunkStream, chunkFile, StandardCopyOption.REPLACE_EXISTING);
+
+        return Response.ok("Chunk " + index + " received").build();
+    } catch (IOException e) {
+		LOGGER.error("Failed to write chunk {} for service='{}', name='{}', identifier='{}'", index, serviceString, name, identifier, e);
+        return Response.serverError().entity("Failed to write chunk: " + e.getMessage()).build();
+    }
+}
+
+@POST
+@Path("/{service}/{name}/{identifier}/finalize")
+@Produces(MediaType.TEXT_PLAIN)
+@Operation(
+    summary = "Finalize a chunked upload and build a raw, unsigned, ARBITRARY transaction",
+    responses = {
+        @ApiResponse(
+            description = "raw, unsigned, ARBITRARY transaction encoded in Base58",
+            content = @Content(mediaType = MediaType.TEXT_PLAIN)
+        )
+    }
+)
+@SecurityRequirement(name = "apiKey")
+public String finalizeUpload(
+    @HeaderParam(Security.API_KEY_HEADER) String apiKey,
+    @PathParam("service") String serviceString,
+    @PathParam("name") String name,
+    @PathParam("identifier") String identifier,
+    @QueryParam("title") String title,
+    @QueryParam("description") String description,
+    @QueryParam("tags") List<String> tags,
+    @QueryParam("category") Category category,
+    @QueryParam("filename") String filename,
+    @QueryParam("fee") Long fee,
+    @QueryParam("preview") Boolean preview,
+	@QueryParam("isZip") Boolean isZip
+) {
+    Security.checkApiCallAllowed(request);
+    java.nio.file.Path tempFile = null;
+    java.nio.file.Path tempDir = null;
+	java.nio.file.Path chunkDir = null;
+
+	
+	
+	
+
+    try {
+		String safeService = Paths.get(serviceString).getFileName().toString();
+	String safeName = Paths.get(name).getFileName().toString();
+	String safeIdentifier = Paths.get(identifier).getFileName().toString();
+	java.nio.file.Path baseUploadsDir = Paths.get("uploads-temp"); // relative to Qortal working dir
+	chunkDir = baseUploadsDir.resolve(safeService).resolve(safeName).resolve(safeIdentifier);
+		
+        if (!Files.exists(chunkDir) || !Files.isDirectory(chunkDir)) {
+            throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No chunks found for upload");
+        }
+
+        // Step 1: Determine a safe filename for disk temp file (regardless of extension correctness)
+        String safeFilename = filename;
+        if (filename == null || filename.isBlank()) {
+			safeFilename = "qortal-" + NTP.getTime();
+		} 
+
+        tempDir = Files.createTempDirectory("qortal-");
+        String sanitizedFilename = Paths.get(safeFilename).getFileName().toString();
+		tempFile = tempDir.resolve(sanitizedFilename);
+
+
+        // Step 2: Merge chunks
+  
+        try (OutputStream out = Files.newOutputStream(tempFile, StandardOpenOption.CREATE, StandardOpenOption.TRUNCATE_EXISTING)) {
+            byte[] buffer = new byte[65536];
+            for (java.nio.file.Path chunk : Files.list(chunkDir)
+                    .filter(path -> path.getFileName().toString().startsWith("chunk_"))
+                    .sorted(Comparator.comparingInt(path -> {
+                        String name2 = path.getFileName().toString();
+                        String numberPart = name2.substring("chunk_".length());
+                        return Integer.parseInt(numberPart);
+                    })).collect(Collectors.toList())) {
+                try (InputStream in = Files.newInputStream(chunk)) {
+                    int bytesRead;
+                    while ((bytesRead = in.read(buffer)) != -1) {
+                        out.write(buffer, 0, bytesRead);
+                    }
+                }
+            }
+        }
+       
+
+        // Step 3: Determine correct extension
+        String detectedExtension = "";
+        String uploadFilename  = null;
+        boolean extensionIsValid = false;
+
+        if (filename != null && !filename.isBlank()) {
+            int lastDot = filename.lastIndexOf('.');
+            if (lastDot > 0 && lastDot < filename.length() - 1) {
+                extensionIsValid = true;
+                uploadFilename = filename;
+            }
+        }
+
+        if (!extensionIsValid) {
+            Tika tika = new Tika();
+            String mimeType = tika.detect(tempFile.toFile());
+            try {
+                MimeTypes allTypes = MimeTypes.getDefaultMimeTypes();
+                org.apache.tika.mime.MimeType mime = allTypes.forName(mimeType);
+                detectedExtension = mime.getExtension();
+            } catch (MimeTypeException e) {
+                LOGGER.warn("Could not determine file extension for MIME type: {}", mimeType, e);
+            }
+
+            if (filename != null && !filename.isBlank()) {
+                int lastDot = filename.lastIndexOf('.');
+                String baseName = (lastDot > 0) ? filename.substring(0, lastDot) : filename;
+                uploadFilename = baseName + (detectedExtension != null ? detectedExtension : "");
+            } else {
+                uploadFilename = "qortal-" + NTP.getTime() + (detectedExtension != null ? detectedExtension : "");
+            }
+        }
+
+
+		Boolean isZipBoolean = false;
+
+		if (isZip != null && isZip) {
+			isZipBoolean = true;
+		}
+        
+
+        return this.upload(
+            Service.valueOf(serviceString),
+            name,
+            identifier,
+            tempFile.toString(),
+            null,
+            null,
+            isZipBoolean,
+            fee,
+            uploadFilename,
+            title,
+            description,
+            tags,
+            category,
+            preview
+        );
+
+    } catch (IOException e) {
+		LOGGER.error("Unexpected error in finalizeUpload for service='{}', name='{}', name='{}'", serviceString, name, identifier, e);
+
+        throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, "Failed to merge chunks: " + e.getMessage());
+    } finally {
+        if (tempDir != null) {
+            try {
+                Files.walk(tempDir)
+                    .sorted(Comparator.reverseOrder())
+                    .map(java.nio.file.Path::toFile)
+                    .forEach(File::delete);
+            } catch (IOException e) {
+                LOGGER.warn("Failed to delete temp directory: {}", tempDir, e);
+            }
+        }
+
+        try {
+            Files.walk(chunkDir)
+                .sorted(Comparator.reverseOrder())
+                .map(java.nio.file.Path::toFile)
+                .forEach(File::delete);
+        } catch (IOException e) {
+            LOGGER.warn("Failed to delete chunk directory: {}", chunkDir, e);
+        }
+    }
+}
+
+
+
+
 
 	// Upload base64-encoded data
 
@@ -1185,6 +1668,90 @@ public class ArbitraryResource {
 		}
 	}
 
+	@GET
+	@Path("/indices")
+	@Operation(
+			summary = "Find matching arbitrary resource indices",
+			description = "",
+			responses = {
+					@ApiResponse(
+							description = "indices",
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = ArbitraryDataIndexScorecard.class
+											)
+									)
+							)
+					)
+			}
+	)
+	public List<ArbitraryDataIndexScorecard> searchIndices(@QueryParam("terms") String[] terms) {
+
+		List<ArbitraryDataIndexDetail> indices = new ArrayList<>();
+
+		// get index details for each term
+		for( String term : terms ) {
+			List<ArbitraryDataIndexDetail> details = IndexCache.getInstance().getIndicesByTerm().get(term);
+
+			if( details != null ) {
+				indices.addAll(details);
+			}
+		}
+
+		// sum up the scores for each index with identical attributes
+		Map<ArbitraryDataIndexScoreKey, Double> scoreForKey
+			= indices.stream()
+				.collect(
+					Collectors.groupingBy(
+						index -> new ArbitraryDataIndexScoreKey(index.name, index.category, index.link),
+						Collectors.summingDouble(detail -> 1.0 / detail.rank)
+					)
+				);
+
+		// create scorecards for each index group and put them in descending order by score
+		List<ArbitraryDataIndexScorecard> scorecards
+			= scoreForKey.entrySet().stream().map(
+				entry
+				->
+				new ArbitraryDataIndexScorecard(
+					entry.getValue(),
+					entry.getKey().name,
+					entry.getKey().category,
+					entry.getKey().link)
+				)
+				.sorted(Comparator.comparingDouble(ArbitraryDataIndexScorecard::getScore).reversed())
+				.collect(Collectors.toList());
+
+		return scorecards;
+	}
+
+	@GET
+	@Path("/indices/{name}/{idPrefix}")
+	@Operation(
+			summary = "Find matching arbitrary resource indices for a registered name and identifier prefix",
+			description = "",
+			responses = {
+					@ApiResponse(
+							description = "indices",
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = ArbitraryDataIndexDetail.class
+											)
+									)
+							)
+					)
+			}
+	)
+	public List<ArbitraryDataIndexDetail> searchIndicesByName(@PathParam("name") String name, @PathParam("idPrefix") String idPrefix) {
+
+		return
+			IndexCache.getInstance().getIndicesByIssuer()
+				.getOrDefault(name, new ArrayList<>(0)).stream()
+					.filter( indexDetail -> indexDetail.indexIdentifer.startsWith(idPrefix))
+					.collect(Collectors.toList());
+	}
 
 	// Shared methods
 
@@ -1251,7 +1818,7 @@ public class ArbitraryResource {
 			if (path == null) {
 				// See if we have a string instead
 				if (string != null) {
-					if (filename == null) {
+					if (filename == null || filename.isBlank()) {
 						// Use current time as filename
 						filename = String.format("qortal-%d", NTP.getTime());
 					}
@@ -1266,7 +1833,7 @@ public class ArbitraryResource {
 				}
 				// ... or base64 encoded raw data
 				else if (base64 != null) {
-					if (filename == null) {
+					if (filename == null || filename.isBlank()) {
 						// Use current time as filename
 						filename = String.format("qortal-%d", NTP.getTime());
 					}
@@ -1317,6 +1884,7 @@ public class ArbitraryResource {
 				);
 
 				transactionBuilder.build();
+
 				// Don't compute nonce - this is done by the client (or via POST /arbitrary/compute)
 				ArbitraryTransactionData transactionData = transactionBuilder.getArbitraryTransactionData();
 				return Base58.encode(ArbitraryTransactionTransformer.toBytes(transactionData));
@@ -1332,22 +1900,20 @@ public class ArbitraryResource {
 		}
 	}
 
-	private HttpServletResponse download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts) {
-
+	private void download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts, boolean attachment, String attachmentFilename) {
 		try {
 			ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
-
+	
 			int attempts = 0;
 			if (maxAttempts == null) {
 				maxAttempts = 5;
 			}
-
+	
 			// Loop until we have data
 			if (async) {
 				// Asynchronous
 				arbitraryDataReader.loadAsynchronously(false, 1);
-			}
-			else {
+			} else {
 				// Synchronous
 				while (!Controller.isStopping()) {
 					attempts++;
@@ -1357,88 +1923,189 @@ public class ArbitraryResource {
 							break;
 						} catch (MissingDataException e) {
 							if (attempts > maxAttempts) {
-								// Give up after 5 attempts
 								throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Data unavailable. Please try again later.");
 							}
 						}
 					}
-					Thread.sleep(3000L);
 				}
 			}
-
+	
 			java.nio.file.Path outputPath = arbitraryDataReader.getFilePath();
 			if (outputPath == null) {
 				// Assume the resource doesn't exist
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, "File not found");
 			}
-
+	
 			if (filepath == null || filepath.isEmpty()) {
 				// No file path supplied - so check if this is a single file resource
 				String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal");
 				if (files != null && files.length == 1) {
 					// This is a single file resource
 					filepath = files[0];
-				}
-				else {
-					throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA,
-							"filepath is required for resources containing more than one file");
+				} else {
+					throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "filepath is required for resources containing more than one file");
 				}
 			}
-
+	
 			java.nio.file.Path path = Paths.get(outputPath.toString(), filepath);
 			if (!Files.exists(path)) {
-				String message = String.format("No file exists at filepath: %s", filepath);
-				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message);
+				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "No file exists at filepath: " + filepath);
 			}
-
-			byte[] data;
-			int fileSize = (int)path.toFile().length();
-			int length = fileSize;
-
-			// Parse "Range" header
-			Integer rangeStart = null;
-			Integer rangeEnd = null;
+	
+			if (attachment) {
+				String rawFilename;
+	
+				if (attachmentFilename != null && !attachmentFilename.isEmpty()) {
+					// 1. Sanitize first
+					String safeAttachmentFilename = attachmentFilename.replaceAll("[\\\\/:*?\"<>|]", "_");
+	
+					// 2. Check for a valid extension (3–5 alphanumeric chars)
+					if (!safeAttachmentFilename.matches(".*\\.[a-zA-Z0-9]{2,5}$")) {
+						safeAttachmentFilename += ".bin";
+					}
+	
+					rawFilename = safeAttachmentFilename;
+				} else {
+					// Fallback if no filename is provided
+					String baseFilename = (identifier != null && !identifier.isEmpty())
+						? name + "-" + identifier
+						: name;
+					rawFilename = baseFilename.replaceAll("[\\\\/:*?\"<>|]", "_") + ".bin";
+				}
+	
+				// Optional: trim length
+				rawFilename = rawFilename.length() > 100 ? rawFilename.substring(0, 100) : rawFilename;
+	
+				// 3. Set Content-Disposition header
+				response.setHeader("Content-Disposition", "attachment; filename=\"" + rawFilename + "\"");
+			}
+	
+			// Determine the total size of the requested file
+			long fileSize = Files.size(path);
+			String mimeType = context.getMimeType(path.toString());
+	
+			// Attempt to read the "Range" header from the request to support partial content delivery (e.g., for video streaming or resumable downloads)
 			String range = request.getHeader("Range");
-			if (range != null) {
-				range = range.replace("bytes=", "");
-				String[] parts = range.split("-");
-				rangeStart = (parts != null && parts.length > 0) ? Integer.parseInt(parts[0]) : null;
-				rangeEnd = (parts != null && parts.length > 1) ? Integer.parseInt(parts[1]) : fileSize;
+	
+			long rangeStart = 0;
+			long rangeEnd = fileSize - 1;
+			boolean isPartial = false;
+	
+			// If a Range header is present and no base64 encoding is requested, parse the range values
+			if (range != null && encoding == null) {
+				range = range.replace("bytes=", ""); // Remove the "bytes=" prefix
+				String[] parts = range.split("-"); // Split the range into start and end
+	
+				// Parse range start
+				if (parts.length > 0 && !parts[0].isEmpty()) {
+					rangeStart = Long.parseLong(parts[0]);
+				}
+	
+				// Parse range end, if present
+				if (parts.length > 1 && !parts[1].isEmpty()) {
+					rangeEnd = Long.parseLong(parts[1]);
+				}
+	
+				isPartial = true; // Indicate that this is a partial content request
 			}
-
-			if (rangeStart != null && rangeEnd != null) {
-				// We have a range, so update the requested length
-				length = rangeEnd - rangeStart;
+	
+			// Calculate how many bytes should be sent in the response
+			long contentLength = rangeEnd - rangeStart + 1;
+	
+			// Inform the client that byte ranges are supported
+			response.setHeader("Accept-Ranges", "bytes");
+	
+			if (isPartial) {
+				// If partial content was requested, return 206 Partial Content with appropriate headers
+				response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
+				response.setHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd, fileSize));
+			} else {
+				// Otherwise, return the entire file with status 200 OK
+				response.setStatus(HttpServletResponse.SC_OK);
 			}
-
-			if (length < fileSize && encoding == null) {
-				// Partial content requested, and not encoding the data
-				response.setStatus(206);
-				response.addHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd-1, fileSize));
-				data = FilesystemUtils.readFromFile(path.toString(), rangeStart, length);
+	
+			// Initialize output streams for writing the file to the response
+			OutputStream rawOut = null;
+			OutputStream base64Out = null;
+			OutputStream gzipOut = null;
+	
+			try {
+				rawOut = response.getOutputStream();
+	
+				if (encoding != null && "base64".equalsIgnoreCase(encoding)) {
+					// If base64 encoding is requested, override content type
+					response.setContentType("text/plain");
+	
+					// Check if the client accepts gzip encoding
+					String acceptEncoding = request.getHeader("Accept-Encoding");
+					boolean wantsGzip = acceptEncoding != null && acceptEncoding.contains("gzip");
+	
+					if (wantsGzip) {
+						// Wrap output in GZIP and Base64 streams if gzip is accepted
+						response.setHeader("Content-Encoding", "gzip");
+						gzipOut = new GZIPOutputStream(rawOut);
+						base64Out = java.util.Base64.getEncoder().wrap(gzipOut);
+					} else {
+						// Wrap output in Base64 only
+						base64Out = java.util.Base64.getEncoder().wrap(rawOut);
+					}
+	
+					rawOut = base64Out; // Use the wrapped stream for writing
+				} else {
+					// For raw binary output, set the content type and length
+					response.setContentType(mimeType != null ? mimeType : "application/octet-stream");
+					response.setContentLength((int) contentLength);
+				}
+	
+				// Stream file content
+				try (InputStream inputStream = Files.newInputStream(path)) {
+					if (rangeStart > 0) {
+						inputStream.skip(rangeStart);
+					}
+	
+					byte[] buffer = new byte[65536];
+					long bytesRemaining = contentLength;
+					int bytesRead;
+	
+					while (bytesRemaining > 0 && (bytesRead = inputStream.read(buffer, 0, (int) Math.min(buffer.length, bytesRemaining))) != -1) {
+						rawOut.write(buffer, 0, bytesRead);
+						bytesRemaining -= bytesRead;
+					}
+				}
+	
+				// Stream finished
+				if (base64Out != null) {
+					base64Out.close(); // Also flushes and closes the wrapped gzipOut
+				} else if (gzipOut != null) {
+					gzipOut.close(); // Only close gzipOut if it wasn't wrapped by base64Out
+				} else {
+					rawOut.flush(); // Flush only the base output stream if nothing was wrapped
+				}
+	
+				if (!response.isCommitted()) {
+					response.setStatus(HttpServletResponse.SC_OK);
+					response.getWriter().write(" ");
+				}
+	
+			} catch (IOException e) {
+				// Streaming errors should not rethrow — just log
+				LOGGER.warn(String.format("Streaming error for %s %s: %s", service, name, e.getMessage()));
 			}
-			else {
-				// Full content requested (or encoded data)
-				response.setStatus(200);
-				data = Files.readAllBytes(path); // TODO: limit file size that can be read into memory
+	
+		} catch (IOException | ApiException | DataException e) {
+			LOGGER.warn(String.format("Unable to load %s %s: %s", service, name, e.getMessage()));
+			if (!response.isCommitted()) {
+				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
 			}
-
-			// Encode the data if requested
-			if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) {
-				data = Base64.encode(data);
+		} catch (NumberFormatException e) {
+			LOGGER.warn(String.format("Invalid range for %s %s: %s", service, name, e.getMessage()));
+			if (!response.isCommitted()) {
+				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
 			}
-
-			response.addHeader("Accept-Ranges", "bytes");
-			response.setContentType(context.getMimeType(path.toString()));
-			response.setContentLength(data.length);
-			response.getOutputStream().write(data);
-
-			return response;
-		} catch (Exception e) {
-			LOGGER.debug(String.format("Unable to load %s %s: %s", service, name, e.getMessage()));
-			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
 		}
 	}
+	
+	
 
 	private FileProperties getFileProperties(Service service, String name, String identifier) {
 		try {

src/main/java/org/qortal/api/resource/AssetsResource.java

@@ -16,9 +16,13 @@ import org.qortal.api.model.AggregatedOrder;
 import org.qortal.api.model.TradeWithOrderInfo;
 import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
 import org.qortal.asset.Asset;
+import org.qortal.controller.hsqldb.HSQLDBBalanceRecorder;
 import org.qortal.crypto.Crypto;
 import org.qortal.data.account.AccountBalanceData;
 import org.qortal.data.account.AccountData;
+import org.qortal.data.account.AddressAmountData;
+import org.qortal.data.account.BlockHeightRange;
+import org.qortal.data.account.BlockHeightRangeAddressAmounts;
 import org.qortal.data.asset.AssetData;
 import org.qortal.data.asset.OrderData;
 import org.qortal.data.asset.RecentTradeData;
@@ -33,6 +37,7 @@ import org.qortal.transaction.Transaction;
 import org.qortal.transaction.Transaction.ValidationResult;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.transaction.*;
+import org.qortal.utils.BalanceRecorderUtils;
 import org.qortal.utils.Base58;
 
 import javax.servlet.http.HttpServletRequest;
@@ -42,6 +47,7 @@ import javax.ws.rs.core.MediaType;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
+import java.util.Optional;
 import java.util.stream.Collectors;
 
 @Path("/assets")
@@ -179,6 +185,122 @@ public class AssetsResource {
 		}
 	}
 
+	@GET
+	@Path("/balancedynamicranges")
+	@Operation(
+			summary = "Get balance dynamic ranges listed.",
+			description = ".",
+			responses = {
+					@ApiResponse(
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = BlockHeightRange.class
+											)
+									)
+							)
+					)
+			}
+	)
+	public List<BlockHeightRange> getBalanceDynamicRanges(
+			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
+			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
+			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
+
+		Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
+
+		if( recorder.isPresent()) {
+			return recorder.get().getRanges(offset, limit, reverse);
+		}
+		else {
+			return new ArrayList<>(0);
+		}
+	}
+
+	@GET
+	@Path("/balancedynamicrange/{height}")
+	@Operation(
+			summary = "Get balance dynamic range for a given height.",
+			description = ".",
+			responses = {
+					@ApiResponse(
+							content = @Content(
+										schema = @Schema(
+												implementation = BlockHeightRange.class
+										)
+							)
+					)
+			}
+	)
+	@ApiErrors({
+			ApiError.INVALID_CRITERIA, ApiError.INVALID_DATA
+	})
+	public BlockHeightRange getBalanceDynamicRange(@PathParam("height") int height) {
+
+		Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
+
+		if( recorder.isPresent()) {
+			Optional<BlockHeightRange> range = recorder.get().getRange(height);
+
+			if( range.isPresent() ) {
+				return range.get();
+			}
+			else {
+				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+			}
+		}
+		else {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
+		}
+	}
+
+	@GET
+	@Path("/balancedynamicamounts/{begin}/{end}")
+	@Operation(
+			summary = "Get balance dynamic ranges address amounts listed.",
+			description = ".",
+			responses = {
+					@ApiResponse(
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = AddressAmountData.class
+											)
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({
+			ApiError.INVALID_CRITERIA, ApiError.INVALID_DATA
+	})
+	public List<AddressAmountData> getBalanceDynamicAddressAmounts(
+			@PathParam("begin") int begin,
+			@PathParam("end") int end,
+			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
+			@Parameter(ref = "limit") @QueryParam("limit") Integer limit) {
+
+		Optional<HSQLDBBalanceRecorder> recorder = HSQLDBBalanceRecorder.getInstance();
+
+		if( recorder.isPresent()) {
+			Optional<BlockHeightRangeAddressAmounts> addressAmounts = recorder.get().getAddressAmounts(new BlockHeightRange(begin, end, false));
+
+			if( addressAmounts.isPresent() ) {
+				return addressAmounts.get().getAmounts().stream()
+					.sorted(BalanceRecorderUtils.ADDRESS_AMOUNT_DATA_COMPARATOR.reversed())
+					.skip(offset)
+					.limit(limit)
+					.collect(Collectors.toList());
+			}
+			else {
+				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+			}
+		}
+		else {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
+		}
+	}
+
 	@GET
 	@Path("/openorders/{assetid}/{otherassetid}")
 	@Operation(

src/main/java/org/qortal/api/resource/BlocksResource.java

@@ -19,6 +19,8 @@ import org.qortal.crypto.Crypto;
 import org.qortal.data.account.AccountData;
 import org.qortal.data.block.BlockData;
 import org.qortal.data.block.BlockSummaryData;
+import org.qortal.data.block.DecodedOnlineAccountData;
+import org.qortal.data.network.OnlineAccountData;
 import org.qortal.data.transaction.TransactionData;
 import org.qortal.repository.BlockArchiveReader;
 import org.qortal.repository.DataException;
@@ -27,6 +29,7 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.block.BlockTransformer;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Blocks;
 import org.qortal.utils.Triple;
 
 import javax.servlet.http.HttpServletRequest;
@@ -45,6 +48,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Comparator;
 import java.util.List;
+import java.util.Set;
 
 @Path("/blocks")
 @Tag(name = "Blocks")
@@ -542,6 +546,7 @@ public class BlocksResource {
 				}
 			}
 
+			String minterAddress = Account.getRewardShareMintingAddress(repository, blockData.getMinterPublicKey());
 			int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
 			if (minterLevel == 0)
 				// This may be unavailable when requesting a trimmed block
@@ -554,6 +559,7 @@ public class BlocksResource {
 
 			BlockMintingInfo blockMintingInfo = new BlockMintingInfo();
 			blockMintingInfo.minterPublicKey = blockData.getMinterPublicKey();
+			blockMintingInfo.minterAddress = minterAddress;
 			blockMintingInfo.minterLevel = minterLevel;
 			blockMintingInfo.onlineAccountsCount = blockData.getOnlineAccountsCount();
 			blockMintingInfo.maxDistance = new BigDecimal(block.MAX_DISTANCE);
@@ -888,4 +894,49 @@ public class BlocksResource {
 		}
 	}
 
-}
+	@GET
+	@Path("/onlineaccounts/{height}")
+	@Operation(
+			summary = "Get online accounts for block",
+			description = "Returns the online accounts who submitted signatures for this block",
+			responses = {
+					@ApiResponse(
+							description = "online accounts",
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = DecodedOnlineAccountData.class
+											)
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({
+			ApiError.BLOCK_UNKNOWN, ApiError.REPOSITORY_ISSUE
+	})
+	public Set<DecodedOnlineAccountData> getOnlineAccounts(@PathParam("height") int height) {
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// get block from database
+			BlockData blockData = repository.getBlockRepository().fromHeight(height);
+
+			// if block data is not in the database, then try the archive
+			if (blockData == null) {
+				blockData = repository.getBlockArchiveRepository().fromHeight(height);
+
+				// if the block is not in the database or the archive, then the block is unknown
+				if( blockData == null ) {
+					throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
+				}
+			}
+
+			Set<DecodedOnlineAccountData> onlineAccounts = Blocks.getDecodedOnlineAccountsForBlock(repository, blockData);
+
+			return onlineAccounts;
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE);
+		}
+	}
+}

src/main/java/org/qortal/api/resource/ChatResource.java

@@ -234,17 +234,21 @@ public class ChatResource {
 		}
 	)
 	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
-	public ActiveChats getActiveChats(@PathParam("address") String address, @QueryParam("encoding") Encoding encoding) {
+	public ActiveChats getActiveChats(
+		@PathParam("address") String address,
+		@QueryParam("encoding") Encoding encoding,
+		@QueryParam("haschatreference") Boolean hasChatReference
+	) {
 		if (address == null || !Crypto.isValidAddress(address))
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
-
+	
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			return repository.getChatRepository().getActiveChats(address, encoding);
+			return repository.getChatRepository().getActiveChats(address, encoding, hasChatReference);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		}
 	}
-
+	
 	@POST
 	@Operation(
 		summary = "Build raw, unsigned, CHAT transaction",

src/main/java/org/qortal/api/resource/CrossChainBitcoinResource.java

@@ -502,10 +502,10 @@ public class CrossChainBitcoinResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns Bitcoin fee per Kb.",
-			description = "Returns Bitcoin fee per Kb.",
+			summary = "The total fee required for unlocking BTC to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -516,17 +516,17 @@ public class CrossChainBitcoinResource {
 					)
 			}
 	)
-	public String getBitcoinFeeCeiling() {
+	public String getBitcoinFeeRequired() {
 		Bitcoin bitcoin = Bitcoin.getInstance();
 
-		return String.valueOf(bitcoin.getFeeCeiling());
+		return String.valueOf(bitcoin.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets Bitcoin fee ceiling.",
-			description = "Sets Bitcoin fee ceiling.",
+			summary = "The total fee required for unlocking BTC to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -545,13 +545,13 @@ public class CrossChainBitcoinResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setBitcoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setBitcoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		Bitcoin bitcoin = Bitcoin.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(bitcoin, fee);
+			return CrossChainUtils.setFeeRequired(bitcoin, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainDigibyteResource.java

@@ -502,10 +502,10 @@ public class CrossChainDigibyteResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns Digibyte fee per Kb.",
-			description = "Returns Digibyte fee per Kb.",
+			summary = "The total fee required for unlocking DGB to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -516,17 +516,17 @@ public class CrossChainDigibyteResource {
 					)
 			}
 	)
-	public String getDigibyteFeeCeiling() {
+	public String getDigibyteFeeRequired() {
 		Digibyte digibyte = Digibyte.getInstance();
 
-		return String.valueOf(digibyte.getFeeCeiling());
+		return String.valueOf(digibyte.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets Digibyte fee ceiling.",
-			description = "Sets Digibyte fee ceiling.",
+			summary = "The total fee required for unlocking DGB to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -545,13 +545,13 @@ public class CrossChainDigibyteResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setDigibyteFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setDigibyteFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		Digibyte digibyte = Digibyte.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(digibyte, fee);
+			return CrossChainUtils.setFeeRequired(digibyte, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainDogecoinResource.java

@@ -502,10 +502,10 @@ public class CrossChainDogecoinResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns Dogecoin fee per Kb.",
-			description = "Returns Dogecoin fee per Kb.",
+			summary = "The total fee required for unlocking DOGE to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -516,17 +516,17 @@ public class CrossChainDogecoinResource {
 					)
 			}
 	)
-	public String getDogecoinFeeCeiling() {
+	public String getDogecoinFeeRequired() {
 		Dogecoin dogecoin = Dogecoin.getInstance();
 
-		return String.valueOf(dogecoin.getFeeCeiling());
+		return String.valueOf(dogecoin.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets Dogecoin fee ceiling.",
-			description = "Sets Dogecoin fee ceiling.",
+			summary = "The total fee required for unlocking DOGE to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -545,13 +545,13 @@ public class CrossChainDogecoinResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setDogecoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setDogecoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		Dogecoin dogecoin = Dogecoin.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(dogecoin, fee);
+			return CrossChainUtils.setFeeRequired(dogecoin, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainLitecoinResource.java

@@ -540,10 +540,10 @@ public class CrossChainLitecoinResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns Litecoin fee per Kb.",
-			description = "Returns Litecoin fee per Kb.",
+			summary = "The total fee required for unlocking LTC to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -554,17 +554,17 @@ public class CrossChainLitecoinResource {
 					)
 			}
 	)
-	public String getLitecoinFeeCeiling() {
+	public String getLitecoinFeeRequired() {
 		Litecoin litecoin = Litecoin.getInstance();
 
-		return String.valueOf(litecoin.getFeeCeiling());
+		return String.valueOf(litecoin.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets Litecoin fee ceiling.",
-			description = "Sets Litecoin fee ceiling.",
+			summary = "The total fee required for unlocking LTC to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -583,13 +583,13 @@ public class CrossChainLitecoinResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setLitecoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setLitecoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		Litecoin litecoin = Litecoin.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(litecoin, fee);
+			return CrossChainUtils.setFeeRequired(litecoin, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainPirateChainResource.java

@@ -587,10 +587,10 @@ public class CrossChainPirateChainResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns PirateChain fee per Kb.",
-			description = "Returns PirateChain fee per Kb.",
+			summary = "The total fee required for unlocking ARRR to the trade offer creator.",
+			description = "The total fee required for unlocking ARRR to the trade offer creator.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -601,17 +601,17 @@ public class CrossChainPirateChainResource {
 					)
 			}
 	)
-	public String getPirateChainFeeCeiling() {
+	public String getPirateChainFeeRequired() {
 		PirateChain pirateChain = PirateChain.getInstance();
 
-		return String.valueOf(pirateChain.getFeeCeiling());
+		return String.valueOf(pirateChain.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets PirateChain fee ceiling.",
-			description = "Sets PirateChain fee ceiling.",
+			summary = "The total fee required for unlocking ARRR to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -630,13 +630,13 @@ public class CrossChainPirateChainResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setPirateChainFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setPirateChainFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		PirateChain pirateChain = PirateChain.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(pirateChain, fee);
+			return CrossChainUtils.setFeeRequired(pirateChain, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainRavencoinResource.java

@@ -502,10 +502,10 @@ public class CrossChainRavencoinResource {
 	}
 
 	@GET
-	@Path("/feeceiling")
+	@Path("/feerequired")
 	@Operation(
-			summary = "Returns Ravencoin fee per Kb.",
-			description = "Returns Ravencoin fee per Kb.",
+			summary = "The total fee required for unlocking RVN to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			responses = {
 					@ApiResponse(
 							content = @Content(
@@ -516,17 +516,17 @@ public class CrossChainRavencoinResource {
 					)
 			}
 	)
-	public String getRavencoinFeeCeiling() {
+	public String getRavencoinFeeRequired() {
 		Ravencoin ravencoin = Ravencoin.getInstance();
 
-		return String.valueOf(ravencoin.getFeeCeiling());
+		return String.valueOf(ravencoin.getFeeRequired());
 	}
 
 	@POST
-	@Path("/updatefeeceiling")
+	@Path("/updatefeerequired")
 	@Operation(
-			summary = "Sets Ravencoin fee ceiling.",
-			description = "Sets Ravencoin fee ceiling.",
+			summary = "The total fee required for unlocking RVN to the trade offer creator.",
+			description = "This is in sats for a transaction that is approximately 300 kB in size.",
 			requestBody = @RequestBody(
 					required = true,
 					content = @Content(
@@ -545,13 +545,13 @@ public class CrossChainRavencoinResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.INVALID_CRITERIA})
-	public String setRavencoinFeeCeiling(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
+	public String setRavencoinFeeRequired(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String fee) {
 		Security.checkApiCallAllowed(request);
 
 		Ravencoin ravencoin = Ravencoin.getInstance();
 
 		try {
-			return CrossChainUtils.setFeeCeiling(ravencoin, fee);
+			return CrossChainUtils.setFeeRequired(ravencoin, fee);
 		}
 		catch (IllegalArgumentException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);

src/main/java/org/qortal/api/resource/CrossChainResource.java

@@ -10,12 +10,17 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.security.SecurityRequirement;
 import io.swagger.v3.oas.annotations.tags.Tag;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.glassfish.jersey.media.multipart.ContentDisposition;
 import org.qortal.api.ApiError;
 import org.qortal.api.ApiErrors;
 import org.qortal.api.ApiExceptionFactory;
 import org.qortal.api.Security;
 import org.qortal.api.model.CrossChainCancelRequest;
+import org.qortal.api.model.CrossChainTradeLedgerEntry;
 import org.qortal.api.model.CrossChainTradeSummary;
+import org.qortal.controller.ForeignFeesManager;
 import org.qortal.controller.tradebot.TradeBot;
 import org.qortal.crosschain.ACCT;
 import org.qortal.crosschain.AcctMode;
@@ -27,6 +32,8 @@ import org.qortal.data.at.ATData;
 import org.qortal.data.at.ATStateData;
 import org.qortal.data.crosschain.CrossChainTradeData;
 import org.qortal.data.crosschain.TransactionSummary;
+import org.qortal.data.crosschain.ForeignFeeDecodedData;
+import org.qortal.data.crosschain.ForeignFeeEncodedData;
 import org.qortal.data.transaction.BaseTransactionData;
 import org.qortal.data.transaction.MessageTransactionData;
 import org.qortal.data.transaction.TransactionData;
@@ -44,21 +51,36 @@ import org.qortal.utils.Base58;
 import org.qortal.utils.ByteArray;
 import org.qortal.utils.NTP;
 
+import javax.servlet.ServletContext;
 import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
 import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
+import javax.ws.rs.core.HttpHeaders;
 import javax.ws.rs.core.MediaType;
+import java.io.IOException;
 import java.util.*;
 import java.util.function.Supplier;
 import java.util.stream.Collectors;
 
+
+
 @Path("/crosschain")
 @Tag(name = "Cross-Chain")
 public class CrossChainResource {
 
+	private static final Logger LOGGER = LogManager.getLogger(CrossChainResource.class);
+
 	@Context
 	HttpServletRequest request;
 
+	@Context
+	HttpServletResponse response;
+
+	@Context
+	ServletContext context;
+
+
 	@GET
 	@Path("/tradeoffers")
 	@Operation(
@@ -255,6 +277,12 @@ public class CrossChainResource {
 				description = "Only return trades that completed on/after this timestamp (milliseconds since epoch)",
 				example = "1597310000000"
 			) @QueryParam("minimumTimestamp") Long minimumTimestamp,
+			@Parameter(
+				description = "Optionally filter by buyer Qortal public key"
+			) @QueryParam("buyerPublicKey") String buyerPublicKey58,
+			@Parameter(
+				description = "Optionally filter by seller Qortal public key"
+			) @QueryParam("sellerPublicKey") String sellerPublicKey58,
 			@Parameter( ref = "limit") @QueryParam("limit") Integer limit,
 			@Parameter( ref = "offset" ) @QueryParam("offset") Integer offset,
 			@Parameter( ref = "reverse" ) @QueryParam("reverse") Boolean reverse) {
@@ -266,6 +294,10 @@ public class CrossChainResource {
 		if (minimumTimestamp != null && minimumTimestamp <= 0)
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
 
+		// Decode public keys
+		byte[] buyerPublicKey = decodePublicKey(buyerPublicKey58);
+		byte[] sellerPublicKey = decodePublicKey(sellerPublicKey58);
+
 		final Boolean isFinished = Boolean.TRUE;
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
@@ -296,7 +328,7 @@ public class CrossChainResource {
 				byte[] codeHash = acctInfo.getKey().value;
 				ACCT acct = acctInfo.getValue().get();
 
-				List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
+				List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash, buyerPublicKey, sellerPublicKey,
 						isFinished, acct.getModeByteOffset(), (long) AcctMode.REDEEMED.value, minimumFinalHeight,
 						limit, offset, reverse);
 
@@ -335,6 +367,215 @@ public class CrossChainResource {
 		}
 	}
 
+	@POST
+	@Path("/signedfees")
+	@Operation(
+		summary = "",
+		description = "",
+		requestBody = @RequestBody(
+			required = true,
+				content = @Content(
+					mediaType = MediaType.APPLICATION_JSON,
+					array = @ArraySchema(
+						schema = @Schema(
+								implementation = ForeignFeeEncodedData.class
+						)
+					)
+				)
+		),
+		responses = {
+			@ApiResponse(
+				description = "true on success",
+				content = @Content(
+					mediaType = MediaType.TEXT_PLAIN,
+					schema = @Schema(
+							type = "boolean"
+					)
+				)
+			)
+		}
+	)
+	public String postSignedForeignFees(List<ForeignFeeEncodedData> signedFees) {
+
+		LOGGER.info("signedFees = " + signedFees);
+
+		try {
+			ForeignFeesManager.getInstance().addSignedFees(signedFees);
+
+			return "true";
+		}
+		catch( Exception e ) {
+
+			LOGGER.error(e.getMessage(), e);
+
+			return "false";
+		}
+	}
+
+	@GET
+	@Path("/unsignedfees/{address}")
+	@Operation(
+		summary = "",
+		description = "",
+		responses = {
+			@ApiResponse(
+				content = @Content(
+					array = @ArraySchema(
+						schema = @Schema(
+							implementation = ForeignFeeEncodedData.class
+						)
+					)
+				)
+			)
+		}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
+	public List<ForeignFeeEncodedData> getUnsignedFees(@PathParam("address") String address) {
+
+		List<ForeignFeeEncodedData> unsignedFeesForAddress = ForeignFeesManager.getInstance().getUnsignedFeesForAddress(address);
+
+		LOGGER.info("address = " + address);
+		LOGGER.info("returning unsigned = " + unsignedFeesForAddress);
+		return unsignedFeesForAddress;
+	}
+
+	@GET
+	@Path("/signedfees")
+	@Operation(
+		summary = "",
+		description = "",
+		responses = {
+			@ApiResponse(
+				content = @Content(
+					array = @ArraySchema(
+						schema = @Schema(
+							implementation = ForeignFeeDecodedData.class
+						)
+					)
+				)
+			)
+		}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
+	public List<ForeignFeeDecodedData> getSignedFees() {
+
+		return ForeignFeesManager.getInstance().getSignedFees();
+	}
+
+	/**
+	 * Decode Public Key
+	 *
+	 * @param publicKey58 the public key in a string
+	 *
+	 * @return the public key in bytes
+	 */
+	private byte[] decodePublicKey(String publicKey58) {
+
+		if( publicKey58 == null ) return null;
+		if( publicKey58.isEmpty() ) return new byte[0];
+
+		byte[] publicKey;
+		try {
+			publicKey = Base58.decode(publicKey58);
+		} catch (NumberFormatException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PUBLIC_KEY, e);
+		}
+
+		// Correct size for public key?
+		if (publicKey.length != Transformer.PUBLIC_KEY_LENGTH)
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PUBLIC_KEY);
+
+		return publicKey;
+	}
+
+	@GET
+	@Path("/ledger/{publicKey}")
+	@Operation(
+			summary = "Accounting entries for all trades.",
+			description = "Returns accounting entries for all completed cross-chain trades",
+			responses = {
+					@ApiResponse(
+							content = @Content(
+									schema = @Schema(
+											type = "string",
+											format = "byte"
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
+	public HttpServletResponse getLedgerEntries(
+			@PathParam("publicKey") String publicKey58,
+			@Parameter(
+					description = "Only return trades that completed on/after this timestamp (milliseconds since epoch)",
+					example = "1597310000000"
+			) @QueryParam("minimumTimestamp") Long minimumTimestamp) {
+
+		byte[] publicKey = decodePublicKey(publicKey58);
+
+		// minimumTimestamp (if given) needs to be positive
+		if (minimumTimestamp != null && minimumTimestamp <= 0)
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			Integer minimumFinalHeight = null;
+
+			if (minimumTimestamp != null) {
+				minimumFinalHeight = repository.getBlockRepository().getHeightFromTimestamp(minimumTimestamp);
+				// If not found in the block repository it will return either 0 or 1
+				if (minimumFinalHeight == 0 || minimumFinalHeight == 1) {
+					// Try the archive
+					minimumFinalHeight = repository.getBlockArchiveRepository().getHeightFromTimestamp(minimumTimestamp);
+				}
+
+				if (minimumFinalHeight == 0)
+					// We don't have any blocks since minimumTimestamp, let alone trades, so nothing to return
+					return response;
+
+				// height returned from repository is for block BEFORE timestamp
+				// but we want trades AFTER timestamp so bump height accordingly
+				minimumFinalHeight++;
+			}
+
+			List<CrossChainTradeLedgerEntry> crossChainTradeLedgerEntries = new ArrayList<>();
+
+			Map<ByteArray, Supplier<ACCT>> acctsByCodeHash = SupportedBlockchain.getAcctMap();
+
+			// collect ledger entries for each ACCT
+			for (Map.Entry<ByteArray, Supplier<ACCT>> acctInfo : acctsByCodeHash.entrySet()) {
+				byte[] codeHash = acctInfo.getKey().value;
+				ACCT acct = acctInfo.getValue().get();
+
+				// collect buys and sells
+				CrossChainUtils.collectLedgerEntries(publicKey, repository, minimumFinalHeight, crossChainTradeLedgerEntries, codeHash, acct, true);
+				CrossChainUtils.collectLedgerEntries(publicKey, repository, minimumFinalHeight, crossChainTradeLedgerEntries, codeHash, acct, false);
+			}
+
+			crossChainTradeLedgerEntries.sort((a, b) -> Longs.compare(a.getTradeTimestamp(), b.getTradeTimestamp()));
+
+			response.setStatus(HttpServletResponse.SC_OK);
+			response.setContentType("text/csv");
+			response.setHeader(
+				HttpHeaders.CONTENT_DISPOSITION,
+				ContentDisposition
+					.type("attachment")
+					.fileName(CrossChainUtils.createLedgerFileName(Crypto.toAddress(publicKey)))
+					.build()
+					.toString()
+			);
+
+			CrossChainUtils.writeToLedger( response.getWriter(), crossChainTradeLedgerEntries);
+
+			return response;
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		} catch (IOException e) {
+			response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+			return response;
+		}
+	}
+
 	@GET
 	@Path("/price/{blockchain}")
 	@Operation(

src/main/java/org/qortal/api/resource/CrossChainUtils.java

@@ -10,21 +10,37 @@ import org.bitcoinj.script.ScriptBuilder;
 
 import org.bouncycastle.util.Strings;
 import org.json.simple.JSONObject;
+import org.qortal.api.model.CrossChainTradeLedgerEntry;
 import org.qortal.api.model.crosschain.BitcoinyTBDRequest;
+import org.qortal.asset.Asset;
 import org.qortal.crosschain.*;
+import org.qortal.data.account.AccountBalanceData;
 import org.qortal.data.at.ATData;
+import org.qortal.data.at.ATStateData;
 import org.qortal.data.crosschain.*;
+import org.qortal.event.EventBus;
+import org.qortal.event.LockingFeeUpdateEvent;
+import org.qortal.event.RequiredFeeUpdateEvent;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
+import org.qortal.utils.Amounts;
 import org.qortal.utils.BitTwiddling;
 
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.Writer;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
 import java.util.*;
+import java.util.function.Function;
 import java.util.stream.Collectors;
 
 
 public class CrossChainUtils {
+    public static final String QORT_CURRENCY_CODE = "QORT";
     private static final Logger LOGGER = LogManager.getLogger(CrossChainUtils.class);
     public static final String CORE_API_CALL = "Core API Call";
+    public static final String QORTAL_EXCHANGE_LABEL = "Qortal";
 
     public static ServerConfigurationInfo buildServerConfigurationInfo(Bitcoiny blockchain) {
 
@@ -88,11 +104,13 @@ public class CrossChainUtils {
 
         bitcoiny.setFeePerKb(Coin.valueOf(satoshis) );
 
+        EventBus.INSTANCE.notify(new LockingFeeUpdateEvent());
+
         return String.valueOf(bitcoiny.getFeePerKb().value);
     }
 
     /**
-     * Set Fee Ceiling
+     * Set Fee Required
      *
      * @param bitcoiny the blockchain support
      * @param fee the fee in satoshis
@@ -101,14 +119,16 @@ public class CrossChainUtils {
      *
      * @throws IllegalArgumentException if invalid
      */
-    public static String setFeeCeiling(Bitcoiny bitcoiny, String fee)  throws IllegalArgumentException{
+    public static String setFeeRequired(Bitcoiny bitcoiny, String fee)  throws IllegalArgumentException{
 
         long satoshis = Long.parseLong(fee);
         if( satoshis < 0 ) throw new IllegalArgumentException("can't set fee to negative number");
 
-        bitcoiny.setFeeCeiling( Long.parseLong(fee));
+        bitcoiny.setFeeRequired( Long.parseLong(fee));
 
-        return String.valueOf(bitcoiny.getFeeCeiling());
+        EventBus.INSTANCE.notify(new RequiredFeeUpdateEvent(bitcoiny));
+
+        return String.valueOf(bitcoiny.getFeeRequired());
     }
 
     /**
@@ -217,6 +237,9 @@ public class CrossChainUtils {
         return bitcoiny.getBlockchainProvider().removeServer(server);
     }
 
+    public static ChainableServer getCurrentServer( Bitcoiny bitcoiny ) {
+        return bitcoiny.getBlockchainProvider().getCurrentServer();
+    }
     /**
      * Set Current Server
      *
@@ -632,4 +655,170 @@ public class CrossChainUtils {
         byte[] lockTimeABytes = BitTwiddling.toBEByteArray((long) lockTimeA);
         return Bytes.concat(partnerBitcoinPKH, hashOfSecretA, lockTimeABytes);
     }
+
+    /**
+     * Write To Ledger
+     *
+     * @param writer the writer to the ledger
+     * @param entries the entries to write to the ledger
+     *
+     * @throws IOException
+     */
+    public static void writeToLedger(Writer writer, List<CrossChainTradeLedgerEntry> entries) throws IOException {
+
+        BufferedWriter bufferedWriter = new BufferedWriter(writer);
+
+        StringJoiner header = new StringJoiner(",");
+        header.add("Market");
+        header.add("Currency");
+        header.add("Quantity");
+        header.add("Commission Paid");
+        header.add("Commission Currency");
+        header.add("Total Price");
+        header.add("Date Time");
+        header.add("Exchange");
+
+        bufferedWriter.append(header.toString());
+
+        DateFormat dateFormatter = new SimpleDateFormat("yyyyMMdd HH:mm");
+        dateFormatter.setTimeZone(TimeZone.getTimeZone("UTC"));
+
+        for( CrossChainTradeLedgerEntry entry : entries ) {
+            StringJoiner joiner = new StringJoiner(",");
+
+            joiner.add(entry.getMarket());
+            joiner.add(entry.getCurrency());
+            joiner.add(String.valueOf(Amounts.prettyAmount(entry.getQuantity())));
+            joiner.add(String.valueOf(Amounts.prettyAmount(entry.getFeeAmount())));
+            joiner.add(entry.getFeeCurrency());
+            joiner.add(String.valueOf(Amounts.prettyAmount(entry.getTotalPrice())));
+            joiner.add(dateFormatter.format(new Date(entry.getTradeTimestamp())));
+            joiner.add(QORTAL_EXCHANGE_LABEL);
+
+            bufferedWriter.newLine();
+            bufferedWriter.append(joiner.toString());
+        }
+
+        bufferedWriter.newLine();
+        bufferedWriter.flush();
+    }
+
+    /**
+     * Create Ledger File Name
+     *
+     * Create a file name the includes timestamp and address.
+     *
+     * @param address the address
+     *
+     * @return the file name created
+     */
+    public static String createLedgerFileName(String address) {
+        DateFormat dateFormatter = new SimpleDateFormat("yyyyMMddHHmmss");
+        String fileName = "ledger-" + address + "-" + dateFormatter.format(new Date());
+        return fileName;
+    }
+
+    /**
+     * Collect Ledger Entries
+     *
+     * @param publicKey the public key for the ledger entries, buy and sell
+     * @param repository the data repository
+     * @param minimumFinalHeight the minimum block height for entries to be collected
+     * @param entries the ledger entries to add to
+     * @param codeHash code hash for the entry blockchain
+     * @param acct the ACCT for the entry blockchain
+     * @param isBuy true collecting entries for a buy, otherwise false
+     *
+     * @throws DataException
+     */
+    public static void collectLedgerEntries(
+            byte[] publicKey,
+            Repository repository,
+            Integer minimumFinalHeight,
+            List<CrossChainTradeLedgerEntry> entries,
+            byte[] codeHash,
+            ACCT acct,
+            boolean isBuy) throws DataException {
+
+        // get all the final AT states for the code hash (foreign coin)
+        List<ATStateData> atStates
+            = repository.getATRepository().getMatchingFinalATStates(
+                codeHash,
+                isBuy ? publicKey : null,
+                !isBuy ? publicKey : null,
+                Boolean.TRUE, acct.getModeByteOffset(),
+                (long) AcctMode.REDEEMED.value,
+                minimumFinalHeight,
+                null, null, false
+        );
+
+        String foreignBlockchainCurrencyCode = acct.getBlockchain().getCurrencyCode();
+
+        // for each trade, build ledger entry, collect ledger entry
+        for (ATStateData atState : atStates) {
+            CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
+
+            // We also need block timestamp for use as trade timestamp
+            long localTimestamp = repository.getBlockRepository().getTimestampFromHeight(atState.getHeight());
+
+            if (localTimestamp == 0) {
+                // Try the archive
+                localTimestamp = repository.getBlockArchiveRepository().getTimestampFromHeight(atState.getHeight());
+            }
+
+            CrossChainTradeLedgerEntry ledgerEntry
+                = new CrossChainTradeLedgerEntry(
+                    isBuy ? QORT_CURRENCY_CODE : foreignBlockchainCurrencyCode,
+                    isBuy ? foreignBlockchainCurrencyCode : QORT_CURRENCY_CODE,
+                    isBuy ? crossChainTradeData.qortAmount : crossChainTradeData.expectedForeignAmount,
+                    0,
+                    foreignBlockchainCurrencyCode,
+                    isBuy ? crossChainTradeData.expectedForeignAmount : crossChainTradeData.qortAmount,
+                    localTimestamp);
+
+            entries.add(ledgerEntry);
+        }
+    }
+
+    public static List<CrossChainTradeData> populateTradeDataList(Repository repository, ACCT acct, List<ATData> atDataList) throws DataException {
+
+        if(atDataList.isEmpty()) return new ArrayList<>(0);
+
+        List<ATStateData> latestATStates
+            = repository.getATRepository()
+                .getLatestATStates(
+                    atDataList.stream()
+                        .map(ATData::getATAddress)
+                        .collect(Collectors.toList())
+                );
+
+        Map<String, ATStateData> atStateDataByAtAddress
+            = latestATStates.stream().collect(Collectors.toMap(ATStateData::getATAddress, Function.identity()));
+
+        Map<String, ATData> atDataByAtAddress
+                = atDataList.stream().collect(Collectors.toMap(ATData::getATAddress, Function.identity()));
+
+        Map<String, Long> balanceByAtAddress
+            = repository
+                .getAccountRepository()
+                .getBalances(new ArrayList<>(atDataByAtAddress.keySet()), Asset.QORT)
+                .stream().collect(Collectors.toMap(AccountBalanceData::getAddress, AccountBalanceData::getBalance));
+
+        List<CrossChainTradeData> crossChainTradeDataList = new ArrayList<>(latestATStates.size());
+
+        for( ATStateData atStateData : latestATStates ) {
+            ATData atData = atDataByAtAddress.get(atStateData.getATAddress());
+            crossChainTradeDataList.add(
+                acct.populateTradeData(
+                    repository,
+                    atData.getCreatorPublicKey(),
+                    atData.getCreation(),
+                    atStateData,
+                    OptionalLong.of(balanceByAtAddress.get(atStateData.getATAddress()))
+                )
+            );
+        }
+
+        return crossChainTradeDataList;
+    }
 }

src/main/java/org/qortal/api/resource/NamesResource.java

@@ -33,6 +33,7 @@ import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
 import java.util.List;
+import java.util.Optional;
 import java.util.stream.Collectors;
 
 @Path("/names")
@@ -104,6 +105,45 @@ public class NamesResource {
 		}
 	}
 
+	@GET
+	@Path("/primary/{address}")
+	@Operation(
+			summary = "primary name owned by address",
+			responses = {
+					@ApiResponse(
+							description = "registered primary name info",
+							content = @Content(
+									mediaType = MediaType.APPLICATION_JSON,
+									schema = @Schema(implementation = NameSummary.class)
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE, ApiError.UNAUTHORIZED})
+	public NameSummary getPrimaryNameByAddress(@PathParam("address") String address) {
+		if (!Crypto.isValidAddress(address))
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			if (Settings.getInstance().isLite()) {
+				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.UNAUTHORIZED);
+			}
+			else {
+				Optional<String> primaryName = repository.getNameRepository().getPrimaryName(address);
+
+				if(primaryName.isPresent()) {
+					return new NameSummary(new NameData(primaryName.get(), address));
+				}
+				else {
+					return new NameSummary((new NameData(null, address)));
+				}
+			}
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
 	@GET
 	@Path("/{name}")
 	@Operation(

src/main/java/org/qortal/api/restricted/resource/AdminResource.java

@@ -32,6 +32,7 @@ import org.qortal.controller.Synchronizer.SynchronizationResult;
 import org.qortal.controller.repository.BlockArchiveRebuilder;
 import org.qortal.data.account.MintingAccountData;
 import org.qortal.data.account.RewardShareData;
+import org.qortal.data.system.DbConnectionInfo;
 import org.qortal.network.Network;
 import org.qortal.network.Peer;
 import org.qortal.network.PeerAddress;
@@ -40,6 +41,7 @@ import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
+import org.qortal.data.system.SystemInfo;
 import org.qortal.utils.Base58;
 import org.qortal.utils.NTP;
 
@@ -52,6 +54,7 @@ import java.net.InetSocketAddress;
 import java.net.UnknownHostException;
 import java.nio.file.Files;
 import java.nio.file.Paths;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
@@ -459,7 +462,7 @@ public class AdminResource {
 
 			// Qortal: check reward-share's minting account is still allowed to mint
 			Account rewardShareMintingAccount = new Account(repository, rewardShareData.getMinter());
-			if (!rewardShareMintingAccount.canMint())
+			if (!rewardShareMintingAccount.canMint(false))
 				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.CANNOT_MINT);
 
 			MintingAccountData mintingAccountData = new MintingAccountData(mintingAccount.getPrivateKey(), mintingAccount.getPublicKey());
@@ -1064,4 +1067,29 @@ public class AdminResource {
 		return "true";
 	}
 
-}
+	@GET
+	@Path("/systeminfo")
+	@Operation(
+			summary = "System Information",
+			description = "System memory usage and available processors.",
+			responses = {
+					@ApiResponse(
+							description = "memory usage and available processors",
+							content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = SystemInfo.class))
+					)
+			}
+	)
+	@ApiErrors({ApiError.REPOSITORY_ISSUE})
+	public SystemInfo getSystemInformation() {
+
+		SystemInfo info
+			= new SystemInfo(
+				Runtime.getRuntime().freeMemory(),
+				Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(),
+				Runtime.getRuntime().totalMemory(),
+				Runtime.getRuntime().maxMemory(),
+				Runtime.getRuntime().availableProcessors());
+
+		return info;
+	}
+}

src/main/java/org/qortal/api/restricted/resource/RenderResource.java

@@ -71,33 +71,33 @@ public class RenderResource {
     @Path("/signature/{signature}")
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature,
-                                                   @QueryParam("theme") String theme) {
+                                                   @QueryParam("theme") String theme, @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
 
-        return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme);
+        return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme, lang);
     }
 
     @GET
     @Path("/signature/{signature}/{path:.*}")
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath,
-                                                  @QueryParam("theme") String theme) {
+                                                  @QueryParam("theme") String theme, @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
 
-        return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme);
+        return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme, lang);
     }
 
     @GET
     @Path("/hash/{hash}")
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58,
-                                              @QueryParam("theme") String theme) {
+                                              @QueryParam("theme") String theme,  @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
 
-        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme);
+        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme, lang);
     }
 
     @GET
@@ -105,11 +105,11 @@ public class RenderResource {
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath,
                                              @QueryParam("secret") String secret58,
-                                             @QueryParam("theme") String theme) {
+                                             @QueryParam("theme") String theme, @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
 
-        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme);
+        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme, lang);
     }
 
     @GET
@@ -119,12 +119,12 @@ public class RenderResource {
                                              @PathParam("name") String name,
                                              @PathParam("path") String inPath,
                                              @QueryParam("identifier") String identifier,
-                                             @QueryParam("theme") String theme) {
+                                             @QueryParam("theme") String theme, @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, name, service, null);
 
         String prefix = String.format("/render/%s", service);
-        return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme);
+        return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme, lang);
     }
 
     @GET
@@ -133,18 +133,18 @@ public class RenderResource {
     public HttpServletResponse getIndexByName(@PathParam("service") Service service,
                                               @PathParam("name") String name,
                                               @QueryParam("identifier") String identifier,
-                                              @QueryParam("theme") String theme) {
+                                              @QueryParam("theme") String theme, @QueryParam("lang") String lang) {
         if (!Settings.getInstance().isQDNAuthBypassEnabled())
             Security.requirePriorAuthorization(request, name, service, null);
 
         String prefix = String.format("/render/%s", service);
-        return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme);
+        return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme, lang);
     }
 
 
 
     private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
-                                    String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme) {
+                                    String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme, String lang) {
 
         ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
                 secret58, prefix, includeResourceIdInPrefix, async, "render", request, response, context);
@@ -152,6 +152,9 @@ public class RenderResource {
         if (theme != null) {
             renderer.setTheme(theme);
         }
+        if (lang != null) {
+            renderer.setLang(lang);
+        }
         return renderer.render();
     }
 

src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java

@@ -77,7 +77,9 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
 		}
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session));
+			Boolean hasChatReference = getHasChatReference(session);
+
+			ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session), hasChatReference);
 
 			StringWriter stringWriter = new StringWriter();
 
@@ -103,4 +105,20 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
 		return Encoding.valueOf(encoding);
 	}
 
+	private Boolean getHasChatReference(Session session) {
+		Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
+		List<String> hasChatReferenceList = queryParams.get("haschatreference");
+	
+		// Return null if not specified
+		if (hasChatReferenceList != null && hasChatReferenceList.size() == 1) {
+			String value = hasChatReferenceList.get(0).toLowerCase();
+			if (value.equals("true")) {
+				return true;
+			} else if (value.equals("false")) {
+				return false;
+			}
+		}
+		return null; // Ignored if not present
+	}
+
 }

src/main/java/org/qortal/api/websocket/DataMonitorSocket.java

@@ -0,0 +1,102 @@
+package org.qortal.api.websocket;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.eclipse.jetty.websocket.api.Session;
+import org.eclipse.jetty.websocket.api.WebSocketException;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
+import org.eclipse.jetty.websocket.api.annotations.WebSocket;
+import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
+import org.qortal.api.ApiError;
+import org.qortal.controller.Controller;
+import org.qortal.data.arbitrary.DataMonitorInfo;
+import org.qortal.event.DataMonitorEvent;
+import org.qortal.event.Event;
+import org.qortal.event.EventBus;
+import org.qortal.event.Listener;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.utils.Base58;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.List;
+
+@WebSocket
+@SuppressWarnings("serial")
+public class DataMonitorSocket extends ApiWebSocket implements Listener {
+
+	private static final Logger LOGGER = LogManager.getLogger(DataMonitorSocket.class);
+
+	@Override
+	public void configure(WebSocketServletFactory factory) {
+		LOGGER.info("configure");
+
+		factory.register(DataMonitorSocket.class);
+
+		EventBus.INSTANCE.addListener(this);
+	}
+
+	@Override
+	public void listen(Event event) {
+		if (!(event instanceof DataMonitorEvent))
+			return;
+
+		DataMonitorEvent dataMonitorEvent = (DataMonitorEvent) event;
+
+		for (Session session : getSessions())
+			sendDataEventSummary(session, buildInfo(dataMonitorEvent));
+	}
+
+	private DataMonitorInfo buildInfo(DataMonitorEvent dataMonitorEvent) {
+
+		return new DataMonitorInfo(
+			dataMonitorEvent.getTimestamp(),
+			dataMonitorEvent.getIdentifier(),
+			dataMonitorEvent.getName(),
+			dataMonitorEvent.getService(),
+			dataMonitorEvent.getDescription(),
+			dataMonitorEvent.getTransactionTimestamp(),
+			dataMonitorEvent.getLatestPutTimestamp()
+		);
+	}
+
+	@OnWebSocketConnect
+	@Override
+	public void onWebSocketConnect(Session session) {
+		super.onWebSocketConnect(session);
+	}
+
+	@OnWebSocketClose
+	@Override
+	public void onWebSocketClose(Session session, int statusCode, String reason) {
+		super.onWebSocketClose(session, statusCode, reason);
+	}
+
+	@OnWebSocketError
+	public void onWebSocketError(Session session, Throwable throwable) {
+		/* We ignore errors for now, but method here to silence log spam */
+	}
+
+	@OnWebSocketMessage
+	public void onWebSocketMessage(Session session, String message) {
+		LOGGER.info("onWebSocketMessage: message = " + message);
+	}
+
+	private void sendDataEventSummary(Session session, DataMonitorInfo dataMonitorInfo) {
+		StringWriter stringWriter = new StringWriter();
+
+		try {
+			marshall(stringWriter, dataMonitorInfo);
+
+			session.getRemote().sendStringByFuture(stringWriter.toString());
+		} catch (IOException | WebSocketException e) {
+			// No output this time
+		}
+	}
+
+}

src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java

@@ -98,7 +98,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 						byte[] codeHash = acctInfo.getKey().value;
 						ACCT acct = acctInfo.getValue().get();
 
-						List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
+						List<ATStateData> atStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null,
 								isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
 								null, null, null);
 
@@ -259,7 +259,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 				ACCT acct = acctInfo.getValue().get();
 
 				Integer dataByteOffset = acct.getModeByteOffset();
-				List<ATStateData> initialAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
+				List<ATStateData> initialAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null,
 					isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
 					null, null, null);
 
@@ -298,7 +298,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 				byte[] codeHash = acctInfo.getKey().value;
 				ACCT acct = acctInfo.getValue().get();
 
-				List<ATStateData> historicAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash,
+				List<ATStateData> historicAtStates = repository.getATRepository().getMatchingFinalATStates(codeHash, null, null,
 					isFinished, dataByteOffset, expectedValue, minimumFinalHeight,
 					null, null, null);
 

src/main/java/org/qortal/api/websocket/UnsignedFeesSocket.java

@@ -0,0 +1,83 @@
+package org.qortal.api.websocket;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.eclipse.jetty.websocket.api.Session;
+import org.eclipse.jetty.websocket.api.WebSocketException;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError;
+import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage;
+import org.eclipse.jetty.websocket.api.annotations.WebSocket;
+import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
+import org.qortal.data.crosschain.UnsignedFeeEvent;
+import org.qortal.event.Event;
+import org.qortal.event.EventBus;
+import org.qortal.event.FeeWaitingEvent;
+import org.qortal.event.Listener;
+
+import java.io.IOException;
+import java.io.StringWriter;
+
+@WebSocket
+@SuppressWarnings("serial")
+public class UnsignedFeesSocket extends ApiWebSocket implements Listener {
+
+	private static final Logger LOGGER = LogManager.getLogger(UnsignedFeesSocket.class);
+
+	@Override
+	public void configure(WebSocketServletFactory factory) {
+		LOGGER.info("configure");
+
+		factory.register(UnsignedFeesSocket.class);
+
+		EventBus.INSTANCE.addListener(this);
+	}
+
+	@Override
+	public void listen(Event event) {
+		if (!(event instanceof FeeWaitingEvent))
+			return;
+
+		for (Session session : getSessions()) {
+			FeeWaitingEvent feeWaitingEvent = (FeeWaitingEvent) event;
+			sendUnsignedFeeEvent(session, new UnsignedFeeEvent(feeWaitingEvent.isPositive(), feeWaitingEvent.getAddress()));
+		}
+	}
+
+
+	@OnWebSocketConnect
+	@Override
+	public void onWebSocketConnect(Session session) {
+		super.onWebSocketConnect(session);
+	}
+
+	@OnWebSocketClose
+	@Override
+	public void onWebSocketClose(Session session, int statusCode, String reason) {
+		super.onWebSocketClose(session, statusCode, reason);
+	}
+
+	@OnWebSocketError
+	public void onWebSocketError(Session session, Throwable throwable) {
+		/* We ignore errors for now, but method here to silence log spam */
+	}
+
+	@OnWebSocketMessage
+	public void onWebSocketMessage(Session session, String message) {
+		LOGGER.info("onWebSocketMessage: message = " + message);
+	}
+
+	private void sendUnsignedFeeEvent(Session session, UnsignedFeeEvent unsignedFeeEvent) {
+		StringWriter stringWriter = new StringWriter();
+
+		try {
+			marshall(stringWriter, unsignedFeeEvent);
+
+			session.getRemote().sendStringByFuture(stringWriter.toString());
+		} catch (IOException | WebSocketException e) {
+			// No output this time
+		}
+	}
+
+}

src/main/java/org/qortal/arbitrary/ArbitraryDataDigest.java

@@ -4,9 +4,12 @@ import org.qortal.repository.DataException;
 import org.qortal.utils.Base58;
 
 import java.io.IOException;
+import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.FileSystem;
 import java.nio.file.Files;
 import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
 import java.util.Arrays;
@@ -23,37 +26,53 @@ public class ArbitraryDataDigest {
     }
 
     public void compute() throws IOException, DataException {
-        List<Path> allPaths = Files.walk(path).filter(Files::isRegularFile).sorted().collect(Collectors.toList());
+        List<Path> allPaths = Files.walk(path)
+            .filter(Files::isRegularFile)
+            .sorted()
+            .collect(Collectors.toList());
+    
         Path basePathAbsolute = this.path.toAbsolutePath();
-
+    
         MessageDigest sha256;
         try {
             sha256 = MessageDigest.getInstance("SHA-256");
         } catch (NoSuchAlgorithmException e) {
             throw new DataException("SHA-256 hashing algorithm unavailable");
         }
-
+    
         for (Path path : allPaths) {
             // We need to work with paths relative to the base path, to ensure the same hash
             // is generated on different systems
             Path relativePath = basePathAbsolute.relativize(path.toAbsolutePath());
-
+    
             // Exclude Qortal folder since it can be different each time
             // We only care about hashing the actual user data
             if (relativePath.startsWith(".qortal/")) {
                 continue;
             }
-
+    
+            // Account for \ VS / : Linux VS Windows
+            String pathString = relativePath.toString();
+            if (relativePath.getFileSystem().toString().contains("Windows")) {
+                pathString = pathString.replace("\\", "/");
+            }
+    
             // Hash path
-            byte[] filePathBytes = relativePath.toString().getBytes(StandardCharsets.UTF_8);
+            byte[] filePathBytes = pathString.getBytes(StandardCharsets.UTF_8);
             sha256.update(filePathBytes);
-
-            // Hash contents
-            byte[] fileContent = Files.readAllBytes(path);
-            sha256.update(fileContent);
+    
+            try (InputStream in = Files.newInputStream(path)) {
+                byte[] buffer = new byte[65536]; // 64 KB
+                int bytesRead;
+                while ((bytesRead = in.read(buffer)) != -1) {
+                    sha256.update(buffer, 0, bytesRead);
+                }
+            }
         }
+    
         this.hash = sha256.digest();
     }
+    
 
     public boolean isHashValid(byte[] hash) {
         return Arrays.equals(hash, this.hash);

src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java

@@ -52,7 +52,7 @@ public class ArbitraryDataFile {
 
     private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFile.class);
 
-    public static final long MAX_FILE_SIZE = 500 * 1024 * 1024; // 500MiB
+    public static final long MAX_FILE_SIZE = 2L * 1024 * 1024 * 1024; // 2 GiB
     protected static final int MAX_CHUNK_SIZE = 1 * 1024 * 1024; // 1MiB
     public static final int CHUNK_SIZE = 512 * 1024; // 0.5MiB
     public static int SHORT_DIGEST_LENGTH = 8;

src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java

@@ -439,7 +439,15 @@ public class ArbitraryDataReader {
             // Ensure the complete hash matches the joined chunks
             if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) {
                 // Delete the invalid file
-                arbitraryDataFile.delete();
+                LOGGER.info("Deleting invalid file: path = " + arbitraryDataFile.getFilePath());
+
+                if( arbitraryDataFile.delete() ) {
+                    LOGGER.info("Deleted invalid file successfully: path = " + arbitraryDataFile.getFilePath());
+                }
+                else {
+                    LOGGER.warn("Could not delete invalid file: path = " + arbitraryDataFile.getFilePath());
+                }
+
                 throw new DataException("Unable to validate complete file hash");
             }
         }

src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java

@@ -1,6 +1,7 @@
 package org.qortal.arbitrary;
 
 import com.google.common.io.Resources;
+
 import org.apache.commons.io.FileUtils;
 import org.apache.commons.lang3.ArrayUtils;
 import org.apache.logging.log4j.LogManager;
@@ -15,11 +16,13 @@ import org.qortal.settings.Settings;
 import javax.servlet.ServletContext;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
+
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.net.URL;
+
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.NoSuchFileException;
@@ -37,6 +40,7 @@ public class ArbitraryDataRenderer {
     private final Service service;
     private final String identifier;
     private String theme = "light";
+    private String lang = "en"; 
     private String inPath;
     private final String secret58;
     private final String prefix;
@@ -166,9 +170,16 @@ public class ArbitraryDataRenderer {
             if (HTMLParser.isHtmlFile(filename)) {
                 // HTML file - needs to be parsed
                 byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory
-                HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting);
+                String encodedResourceId;
+
+                if (resourceIdType == ResourceIdType.NAME) {
+                    encodedResourceId = resourceId.replace(" ", "%20");
+                } else {
+                    encodedResourceId = resourceId;
+                }
+                HTMLParser htmlParser = new HTMLParser(encodedResourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting, lang);
                 htmlParser.addAdditionalHeaderTags();
-                response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;");
+               response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' wss: blob:;");
                 response.setContentType(context.getMimeType(filename));
                 response.setContentLength(htmlParser.getData().length);
                 response.getOutputStream().write(htmlParser.getData());
@@ -256,5 +267,8 @@ public class ArbitraryDataRenderer {
     public void setTheme(String theme) {
         this.theme = theme;
     }
+    public void setLang(String lang) {
+        this.lang = lang;
+    }    
 
 }

src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java

@@ -29,6 +29,7 @@ import org.qortal.utils.FilesystemUtils;
 import org.qortal.utils.NTP;
 
 import java.io.IOException;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.List;
@@ -197,7 +198,7 @@ public class ArbitraryDataTransactionBuilder {
 
             // We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain
             final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false);
-            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
+            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(Files.size(path)) <= ArbitraryTransaction.MAX_DATA_SIZE);
             if (shouldUseOnChainData) {
                 LOGGER.info("Data size is small enough to go on chain - using PUT");
                 return Method.PUT;
@@ -245,7 +246,7 @@ public class ArbitraryDataTransactionBuilder {
 
             // Single file resources are handled differently, especially for very small data payloads, as these go on chain
             final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false);
-            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
+            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(Files.size(path)) <= ArbitraryTransaction.MAX_DATA_SIZE);
 
             // Use zip compression if data isn't going on chain
             Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP;

src/main/java/org/qortal/arbitrary/misc/Service.java

@@ -37,7 +37,7 @@ public enum Service {
             if (files != null && files[0] != null) {
                 final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase();
                 // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string
-                final List<String> allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
+                final List<String> allowedExtensions = Arrays.asList("qortal", "zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
                 if (extension == null || !allowedExtensions.contains(extension)) {
                     return ValidationResult.INVALID_FILE_EXTENSION;
                 }
@@ -62,7 +62,17 @@ public enum Service {
 
             // Custom validation function to require an index HTML file in the root directory
             List<String> fileNames = ArbitraryDataRenderer.indexFiles();
-            String[] files = path.toFile().list();
+            List<String> files;
+
+            // single files are paackaged differently
+            if( path.toFile().isFile() ) {
+                files = new ArrayList<>(1);
+                files.add(path.getFileName().toString());
+            }
+            else {
+                files = new ArrayList<>(Arrays.asList(path.toFile().list()));
+            }
+
             if (files != null) {
                 for (String file : files) {
                     Path fileName = Paths.get(file).getFileName();

src/main/java/org/qortal/block/Block.java

@@ -23,12 +23,11 @@ import org.qortal.data.at.ATStateData;
 import org.qortal.data.block.BlockData;
 import org.qortal.data.block.BlockSummaryData;
 import org.qortal.data.block.BlockTransactionData;
+import org.qortal.data.group.GroupAdminData;
 import org.qortal.data.network.OnlineAccountData;
 import org.qortal.data.transaction.TransactionData;
-import org.qortal.repository.ATRepository;
-import org.qortal.repository.DataException;
-import org.qortal.repository.Repository;
-import org.qortal.repository.TransactionRepository;
+import org.qortal.group.Group;
+import org.qortal.repository.*;
 import org.qortal.settings.Settings;
 import org.qortal.transaction.AtTransaction;
 import org.qortal.transaction.Transaction;
@@ -40,6 +39,7 @@ import org.qortal.transform.block.BlockTransformer;
 import org.qortal.transform.transaction.TransactionTransformer;
 import org.qortal.utils.Amounts;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Groups;
 import org.qortal.utils.NTP;
 
 import java.io.ByteArrayOutputStream;
@@ -144,11 +144,14 @@ public class Block {
 		private final Account mintingAccount;
 		private final AccountData mintingAccountData;
 		private final boolean isMinterFounder;
+		private final boolean isMinterMember;
 
 		private final Account recipientAccount;
 		private final AccountData recipientAccountData;
 
-		ExpandedAccount(Repository repository, RewardShareData rewardShareData) throws DataException {
+		final BlockChain blockChain = BlockChain.getInstance();
+
+		ExpandedAccount(Repository repository, RewardShareData rewardShareData, int blockHeight) throws DataException {
 			this.rewardShareData = rewardShareData;
 			this.sharePercent = this.rewardShareData.getSharePercent();
 
@@ -157,6 +160,12 @@ public class Block {
 			this.isMinterFounder = Account.isFounder(mintingAccountData.getFlags());
 
 			this.isRecipientAlsoMinter = this.rewardShareData.getRecipient().equals(this.mintingAccount.getAddress());
+			this.isMinterMember
+				= Groups.memberExistsInAnyGroup(
+					repository.getGroupRepository(),
+					Groups.getGroupIdsToMint(BlockChain.getInstance(), blockHeight),
+					this.mintingAccount.getAddress()
+			);
 
 			if (this.isRecipientAlsoMinter) {
 				// Self-share: minter is also recipient
@@ -169,6 +178,19 @@ public class Block {
 			}
 		}
 
+		/**
+		 * Get Effective Minting Level
+		 *
+		 * @return the effective minting level, if a data exception is thrown, it catches the exception and returns a zero
+		 */
+		public int getEffectiveMintingLevel() {
+			try {
+				return this.mintingAccount.getEffectiveMintingLevel();
+			} catch (DataException e) {
+				return 0;
+			}
+		}
+
 		public Account getMintingAccount() {
 			return this.mintingAccount;
 		}
@@ -181,19 +203,23 @@ public class Block {
 		 * <p>
 		 * This is a method, not a final variable, because account's level can change between construction and call,
 		 * e.g. during Block.process() where account levels are bumped right before Block.distributeBlockReward().
-		 * 
+		 *
 		 *  @return account-level share "bin" from blockchain config, or null if founder / none found
 		 */
 		public AccountLevelShareBin getShareBin(int blockHeight) {
-			if (this.isMinterFounder)
+			if (this.isMinterFounder && blockHeight < BlockChain.getInstance().getAdminsReplaceFoundersHeight())
 				return null;
 
 			final int accountLevel = this.mintingAccountData.getLevel();
 			if (accountLevel <= 0)
 				return null; // level 0 isn't included in any share bins
 
+			if (blockHeight >= blockChain.getFixBatchRewardHeight()) {
+				if (!this.isMinterMember)
+					return null; // not member of minter group isn't included in any share bins
+			}
+
 			// Select the correct set of share bins based on block height
-			final BlockChain blockChain = BlockChain.getInstance();
 			final AccountLevelShareBin[] shareBinsByLevel = (blockHeight >= blockChain.getSharesByLevelV2Height()) ?
 					blockChain.getShareBinsByAccountLevelV2() : blockChain.getShareBinsByAccountLevelV1();
 
@@ -262,7 +288,7 @@ public class Block {
 	 * Constructs new Block without loading transactions and AT states.
 	 * <p>
 	 * Transactions and AT states are loaded on first call to getTransactions() or getATStates() respectively.
-	 * 
+	 *
 	 * @param repository
 	 * @param blockData
 	 */
@@ -333,7 +359,7 @@ public class Block {
 
 	/**
 	 * Constructs new Block with empty transaction list, using passed minter account.
-	 * 
+	 *
 	 * @param repository
 	 * @param blockData
 	 * @param minter
@@ -351,7 +377,7 @@ public class Block {
 	 * This constructor typically used when minting a new block.
 	 * <p>
 	 * Note that CIYAM ATs will be executed and AT-Transactions prepended to this block, along with AT state data and fees.
-	 * 
+	 *
 	 * @param repository
 	 * @param parentBlockData
 	 * @param minter
@@ -377,7 +403,7 @@ public class Block {
 		byte[] encodedOnlineAccounts = new byte[0];
 		int onlineAccountsCount = 0;
 		byte[] onlineAccountsSignatures = null;
-		
+
 		if (isBatchRewardDistributionBlock(height)) {
 			// Batch reward distribution block - copy online accounts from recent block with highest online accounts count
 
@@ -398,7 +424,9 @@ public class Block {
 			onlineAccounts.removeIf(a -> a.getNonce() == null || a.getNonce() < 0);
 
 			// After feature trigger, remove any online accounts that are level 0
-			if (height >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
+			// but only if they are before the ignore level feature trigger
+			if (height < BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() &&
+					height >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
 				onlineAccounts.removeIf(a -> {
 					try {
 						return Account.getRewardShareEffectiveMintingLevel(repository, a.getPublicKey()) == 0;
@@ -409,6 +437,21 @@ public class Block {
 				});
 			}
 
+			// After feature trigger, remove any online accounts that are not minter group member
+			if (height >= BlockChain.getInstance().getGroupMemberCheckHeight()) {
+				onlineAccounts.removeIf(a -> {
+					try {
+						List<Integer> groupIdsToMint = Groups.getGroupIdsToMint(BlockChain.getInstance(), height);
+						String address = Account.getRewardShareMintingAddress(repository, a.getPublicKey());
+						boolean isMinterGroupMember = Groups.memberExistsInAnyGroup(repository.getGroupRepository(), groupIdsToMint, address);
+						return !isMinterGroupMember;
+					} catch (DataException e) {
+						// Something went wrong, so remove the account
+						return true;
+					}
+				});
+			}
+
 			if (onlineAccounts.isEmpty()) {
 				LOGGER.debug("No online accounts - not even our own?");
 				return null;
@@ -512,7 +555,7 @@ public class Block {
 	 * Mints new block using this block as template, but with different minting account.
 	 * <p>
 	 * NOTE: uses the same transactions list, AT states, etc.
-	 * 
+	 *
 	 * @param minter
 	 * @return
 	 * @throws DataException
@@ -598,7 +641,7 @@ public class Block {
 
 	/**
 	 * Return composite block signature (minterSignature + transactionsSignature).
-	 * 
+	 *
 	 * @return byte[], or null if either component signature is null.
 	 */
 	public byte[] getSignature() {
@@ -613,7 +656,7 @@ public class Block {
 	 * <p>
 	 * We're starting with version 4 as a nod to being newer than successor Qora,
 	 * whose latest block version was 3.
-	 * 
+	 *
 	 * @return 1, 2, 3 or 4
 	 */
 	public int getNextBlockVersion() {
@@ -627,7 +670,7 @@ public class Block {
 	 * Return block's transactions.
 	 * <p>
 	 * If the block was loaded from repository then it's possible this method will call the repository to fetch the transactions if not done already.
-	 * 
+	 *
 	 * @return
 	 * @throws DataException
 	 */
@@ -661,7 +704,7 @@ public class Block {
 	 * If the block was loaded from repository then it's possible this method will call the repository to fetch the AT states if not done already.
 	 * <p>
 	 * <b>Note:</b> AT states fetched from repository only contain summary info, not actual data like serialized state data or AT creation timestamps!
-	 * 
+	 *
 	 * @return
 	 * @throws DataException
 	 */
@@ -697,7 +740,7 @@ public class Block {
 	 * <p>
 	 * Typically called as part of Block.process() or Block.orphan()
 	 * so ideally after any calls to Block.isValid().
-	 * 
+	 *
 	 * @throws DataException
 	 */
 	public List<ExpandedAccount> getExpandedAccounts() throws DataException {
@@ -715,10 +758,12 @@ public class Block {
 
 		List<ExpandedAccount> expandedAccounts = new ArrayList<>();
 
-		for (RewardShareData rewardShare : this.cachedOnlineRewardShares)
-			expandedAccounts.add(new ExpandedAccount(repository, rewardShare));
+		for (RewardShareData rewardShare : this.cachedOnlineRewardShares) {
+			expandedAccounts.add(new ExpandedAccount(repository, rewardShare, this.blockData.getHeight()));
+		}
 
 		this.cachedExpandedAccounts = expandedAccounts;
+		LOGGER.trace(() -> String.format("Online reward-shares after expanded accounts %s", this.cachedOnlineRewardShares));
 
 		return this.cachedExpandedAccounts;
 	}
@@ -727,7 +772,7 @@ public class Block {
 
 	/**
 	 * Load parent block's data from repository via this block's reference.
-	 * 
+	 *
 	 * @return parent's BlockData, or null if no parent found
 	 * @throws DataException
 	 */
@@ -741,7 +786,7 @@ public class Block {
 
 	/**
 	 * Load child block's data from repository via this block's signature.
-	 * 
+	 *
 	 * @return child's BlockData, or null if no parent found
 	 * @throws DataException
 	 */
@@ -761,7 +806,7 @@ public class Block {
 	 * Used when constructing a new block during minting.
 	 * <p>
 	 * Requires block's {@code minter} being a {@code PrivateKeyAccount} so block's transactions signature can be recalculated.
-	 * 
+	 *
 	 * @param transactionData
 	 * @return true if transaction successfully added to block, false otherwise
 	 * @throws IllegalStateException
@@ -814,7 +859,7 @@ public class Block {
 	 * Used when constructing a new block during minting.
 	 * <p>
 	 * Requires block's {@code minter} being a {@code PrivateKeyAccount} so block's transactions signature can be recalculated.
-	 * 
+	 *
 	 * @param transactionData
 	 * @throws IllegalStateException
 	 *             if block's {@code minter} is not a {@code PrivateKeyAccount}.
@@ -859,7 +904,7 @@ public class Block {
 	 * previous block's minter signature + minter's public key + (encoded) online-accounts data
 	 * <p>
 	 * (Previous block's minter signature is extracted from this block's reference).
-	 * 
+	 *
 	 * @throws IllegalStateException
 	 *             if block's {@code minter} is not a {@code PrivateKeyAccount}.
 	 * @throws RuntimeException
@@ -876,7 +921,7 @@ public class Block {
 	 * Recalculate block's transactions signature.
 	 * <p>
 	 * Requires block's {@code minter} being a {@code PrivateKeyAccount}.
-	 * 
+	 *
 	 * @throws IllegalStateException
 	 *             if block's {@code minter} is not a {@code PrivateKeyAccount}.
 	 * @throws RuntimeException
@@ -998,7 +1043,7 @@ public class Block {
 	 * Recalculate block's minter and transactions signatures, thus giving block full signature.
 	 * <p>
 	 * Note: Block instance must have been constructed with a <tt>PrivateKeyAccount</tt> minter or this call will throw an <tt>IllegalStateException</tt>.
-	 * 
+	 *
 	 * @throws IllegalStateException
 	 *             if block's {@code minter} is not a {@code PrivateKeyAccount}.
 	 */
@@ -1011,7 +1056,7 @@ public class Block {
 
 	/**
 	 * Returns whether this block's signatures are valid.
-	 * 
+	 *
 	 * @return true if both minter and transaction signatures are valid, false otherwise
 	 */
 	public boolean isSignatureValid() {
@@ -1035,7 +1080,7 @@ public class Block {
 	 * <p>
 	 * Used by BlockMinter to check whether it's time to mint a new block,
 	 * and also used by Block.isValid for checks (if not a testchain).
-	 * 
+	 *
 	 * @return ValidationResult.OK if timestamp valid, or some other ValidationResult otherwise.
 	 * @throws DataException
 	 */
@@ -1124,14 +1169,32 @@ public class Block {
 		if (onlineRewardShares == null)
 			return ValidationResult.ONLINE_ACCOUNT_UNKNOWN;
 
-		// After feature trigger, require all online account minters to be greater than level 0
-		if (this.getBlockData().getHeight() >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
-			List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
+		// After feature trigger, require all online account minters to be greater than level 0,
+		// but only if it is before the feature trigger where we ignore level again
+		if (this.blockData.getHeight() < BlockChain.getInstance().getIgnoreLevelForRewardShareHeight() &&
+			this.getBlockData().getHeight() >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
+			List<ExpandedAccount> expandedAccounts
+					= this.getExpandedAccounts().stream()
+					.filter(expandedAccount -> expandedAccount.isMinterMember)
+					.collect(Collectors.toList());
+
 			for (ExpandedAccount account : expandedAccounts) {
 				if (account.getMintingAccount().getEffectiveMintingLevel() == 0)
 					return ValidationResult.ONLINE_ACCOUNTS_INVALID;
+
+				if (this.getBlockData().getHeight() >= BlockChain.getInstance().getFixBatchRewardHeight()) {
+					if (!account.isMinterMember)
+						return ValidationResult.ONLINE_ACCOUNTS_INVALID;
+				}
 			}
 		}
+		else if (this.blockData.getHeight() >= BlockChain.getInstance().getIgnoreLevelForRewardShareHeight()){
+			Optional<ExpandedAccount> anyInvalidAccount
+					= this.getExpandedAccounts().stream()
+					.filter(account -> !account.isMinterMember)
+					.findAny();
+			if( anyInvalidAccount.isPresent() ) return ValidationResult.ONLINE_ACCOUNTS_INVALID;
+		}
 
 		// If block is past a certain age then we simply assume the signatures were correct
 		long signatureRequirementThreshold = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMinLifetime();
@@ -1215,7 +1278,7 @@ public class Block {
 	 * <p>
 	 * Checks block's transactions by testing their validity then processing them.<br>
 	 * Hence uses a repository savepoint during execution.
-	 * 
+	 *
 	 * @return ValidationResult.OK if block is valid, or some other ValidationResult otherwise.
 	 * @throws DataException
 	 */
@@ -1258,6 +1321,7 @@ public class Block {
 
 		// Online Accounts
 		ValidationResult onlineAccountsResult = this.areOnlineAccountsValid();
+		LOGGER.trace("Accounts valid = {}", onlineAccountsResult);
 		if (onlineAccountsResult != ValidationResult.OK)
 			return onlineAccountsResult;
 
@@ -1346,7 +1410,7 @@ public class Block {
 				// Check transaction can even be processed
 				validationResult = transaction.isProcessable();
 				if (validationResult != Transaction.ValidationResult.OK) {
-					LOGGER.info(String.format("Error during transaction validation, tx %s: %s", Base58.encode(transactionData.getSignature()), validationResult.name()));
+					LOGGER.debug(String.format("Error during transaction validation, tx %s: %s", Base58.encode(transactionData.getSignature()), validationResult.name()));
 					return ValidationResult.TRANSACTION_INVALID;
 				}
 
@@ -1386,7 +1450,7 @@ public class Block {
 	 * <p>
 	 * NOTE: will execute ATs locally if not already done.<br>
 	 * This is so we have locally-generated AT states for comparison.
-	 * 
+	 *
 	 * @return OK, or some AT-related validation result
 	 * @throws DataException
 	 */
@@ -1462,11 +1526,11 @@ public class Block {
 	 * Note: this method does not store new AT state data into repository - that is handled by <tt>process()</tt>.
 	 * <p>
 	 * This method is not needed if fetching an existing block from the repository as AT state data will be loaded from repository as well.
-	 * 
+	 *
 	 * @see #isValid()
-	 * 
+	 *
 	 * @throws DataException
-	 * 
+	 *
 	 */
 	private void executeATs() throws DataException {
 		// We're expecting a lack of AT state data at this point.
@@ -1518,7 +1582,7 @@ public class Block {
 			return false;
 
 		Account mintingAccount = new PublicKeyAccount(this.repository, rewardShareData.getMinterPublicKey());
-		return mintingAccount.canMint();
+		return mintingAccount.canMint(false);
 	}
 
 	/**
@@ -1538,7 +1602,7 @@ public class Block {
 
 	/**
 	 * Process block, and its transactions, adding them to the blockchain.
-	 * 
+	 *
 	 * @throws DataException
 	 */
 	public void process() throws DataException {
@@ -1547,6 +1611,7 @@ public class Block {
 		this.blockData.setHeight(blockchainHeight + 1);
 
 		LOGGER.trace(() -> String.format("Processing block %d", this.blockData.getHeight()));
+		LOGGER.trace(() -> String.format("Online Reward Shares in process %s", this.cachedOnlineRewardShares));
 
 		if (this.blockData.getHeight() > 1) {
 
@@ -1575,6 +1640,8 @@ public class Block {
 					SelfSponsorshipAlgoV2Block.processAccountPenalties(this);
 				} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) {
 					SelfSponsorshipAlgoV3Block.processAccountPenalties(this);
+				} else if (this.blockData.getHeight() == BlockChain.getInstance().getMultipleNamesPerAccountHeight()) {
+					PrimaryNamesBlock.processNames(this.repository);
 				}
 			}
 		}
@@ -1618,7 +1685,17 @@ public class Block {
 		final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel();
 		final int maximumLevel = cumulativeBlocksByLevel.size() - 1;
 
-		final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
+		final List<ExpandedAccount> expandedAccounts;
+
+		if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
+			expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
+		}
+		else {
+			expandedAccounts
+				= this.getExpandedAccounts().stream()
+					.filter(expandedAccount -> expandedAccount.isMinterMember)
+					.collect(Collectors.toList());
+		}
 
 		Set<AccountData> allUniqueExpandedAccounts = new HashSet<>();
 		for (ExpandedAccount expandedAccount : expandedAccounts) {
@@ -1646,11 +1723,19 @@ public class Block {
 			accountData.setBlocksMinted(accountData.getBlocksMinted() + 1);
 			LOGGER.trace(() -> String.format("Block minter %s up to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
 
-			final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
+			int blocksMintedAdjustment
+				=
+				(this.blockData.getHeight() > BlockChain.getInstance().getMintedBlocksAdjustmentRemovalHeight())
+				?
+				0
+				:
+				accountData.getBlocksMintedAdjustment();
+
+			final int effectiveBlocksMinted = accountData.getBlocksMinted() + blocksMintedAdjustment + accountData.getBlocksMintedPenalty();
 
 			for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
 				if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
-					if (newLevel > accountData.getLevel()) {
+					if (newLevel != accountData.getLevel()) {
 						// Account has increased in level!
 						accountData.setLevel(newLevel);
 						bumpedAccounts.put(accountData.getAddress(), newLevel);
@@ -1839,7 +1924,7 @@ public class Block {
 
 	/**
 	 * Removes block from blockchain undoing transactions and adding them to unconfirmed pile.
-	 * 
+	 *
 	 * @throws DataException
 	 */
 	public void orphan() throws DataException {
@@ -1877,9 +1962,11 @@ public class Block {
 					SelfSponsorshipAlgoV2Block.orphanAccountPenalties(this);
 				} else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV3Height()) {
 					SelfSponsorshipAlgoV3Block.orphanAccountPenalties(this);
+				} else if (this.blockData.getHeight() == BlockChain.getInstance().getMultipleNamesPerAccountHeight()) {
+					PrimaryNamesBlock.orphanNames( this.repository );
 				}
 			}
-			
+
 			// Account levels and block rewards are only processed/orphaned on block reward distribution blocks
 			if (this.isRewardDistributionBlock()) {
 				// Block rewards, including transaction fees, removed after transactions undone
@@ -2018,7 +2105,17 @@ public class Block {
 		final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel();
 		final int maximumLevel = cumulativeBlocksByLevel.size() - 1;
 
-		final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
+		final List<ExpandedAccount> expandedAccounts;
+
+		if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
+			expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
+		}
+		else {
+			expandedAccounts
+				= this.getExpandedAccounts().stream()
+					.filter(expandedAccount -> expandedAccount.isMinterMember)
+					.collect(Collectors.toList());
+		}
 
 		Set<AccountData> allUniqueExpandedAccounts = new HashSet<>();
 		for (ExpandedAccount expandedAccount : expandedAccounts) {
@@ -2042,11 +2139,19 @@ public class Block {
 			accountData.setBlocksMinted(accountData.getBlocksMinted() - 1);
 			LOGGER.trace(() -> String.format("Block minter %s down to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
 
-			final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
+			int blocksMintedAdjustment
+				=
+				(this.blockData.getHeight() -1 > BlockChain.getInstance().getMintedBlocksAdjustmentRemovalHeight())
+				?
+				0
+				:
+				accountData.getBlocksMintedAdjustment();
+
+			final int effectiveBlocksMinted = accountData.getBlocksMinted() + blocksMintedAdjustment + accountData.getBlocksMintedPenalty();
 
 			for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
 				if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
-					if (newLevel < accountData.getLevel()) {
+					if (newLevel != accountData.getLevel()) {
 						// Account has decreased in level!
 						accountData.setLevel(newLevel);
 						repository.getAccountRepository().setLevel(accountData);
@@ -2213,6 +2318,7 @@ public class Block {
 		List<AccountBalanceData> accountBalanceDeltas = balanceChanges.entrySet().stream()
 				.map(entry -> new AccountBalanceData(entry.getKey(), Asset.QORT, entry.getValue()))
 				.collect(Collectors.toList());
+		LOGGER.trace("Account Balance Deltas: {}", accountBalanceDeltas);
 		this.repository.getAccountRepository().modifyAssetBalances(accountBalanceDeltas);
 	}
 
@@ -2221,34 +2327,44 @@ public class Block {
 		List<BlockRewardCandidate> rewardCandidates = new ArrayList<>();
 
 		// All online accounts
-		final List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
+		final List<ExpandedAccount> expandedAccounts;
+
+		if (this.getBlockData().getHeight() < BlockChain.getInstance().getFixBatchRewardHeight()) {
+			expandedAccounts = this.getExpandedAccounts().stream().collect(Collectors.toList());
+		}
+		else {
+			expandedAccounts
+				= this.getExpandedAccounts().stream()
+					.filter(expandedAccount -> expandedAccount.isMinterMember)
+					.collect(Collectors.toList());
+		}
 
 		/*
 		 * Distribution rules:
-		 * 
+		 *
 		 * Distribution is based on the minting account of 'online' reward-shares.
-		 * 
+		 *
 		 * If ANY founders are online, then they receive the leftover non-distributed reward.
 		 * If NO founders are online, then account-level-based rewards are scaled up so 100% of reward is allocated.
-		 * 
+		 *
 		 * If ANY non-maxxed legacy QORA holders exist then they are always allocated their fixed share (e.g. 20%).
-		 * 
+		 *
 		 * There has to be either at least one 'online' account for blocks to be minted
 		 * so there is always either one account-level-based or founder reward candidate.
-		 * 
+		 *
 		 * Examples:
-		 * 
+		 *
 		 * With at least one founder online:
 		 * Level 1/2 accounts: 5%
 		 * Legacy QORA holders: 20%
 		 * Founders: ~75%
-		 * 
+		 *
 		 * No online founders:
 		 * Level 1/2 accounts: 5%
 		 * Level 5/6 accounts: 15%
 		 * Legacy QORA holders: 20%
 		 * Total: 40%
-		 * 
+		 *
 		 * After scaling account-level-based shares to fill 100%:
 		 * Level 1/2 accounts: 20%
 		 * Level 5/6 accounts: 60%
@@ -2264,7 +2380,6 @@ public class Block {
 		// Select the correct set of share bins based on block height
 		List<AccountLevelShareBin> accountLevelShareBinsForBlock = (this.blockData.getHeight() >= BlockChain.getInstance().getSharesByLevelV2Height()) ?
 				BlockChain.getInstance().getAccountLevelShareBinsV2() : BlockChain.getInstance().getAccountLevelShareBinsV1();
-
 		// Determine reward candidates based on account level
 		// This needs a deep copy, so the shares can be modified when tiers aren't activated yet
 		List<AccountLevelShareBin> accountLevelShareBins = new ArrayList<>();
@@ -2347,7 +2462,7 @@ public class Block {
 		final long qoraHoldersShare = BlockChain.getInstance().getQoraHoldersShareAtHeight(this.blockData.getHeight());
 
 		// Perform account-level-based reward scaling if appropriate
-		if (!haveFounders) {
+		if (!haveFounders && this.blockData.getHeight() < BlockChain.getInstance().getAdminsReplaceFoundersHeight() ) {
 			// Recalculate distribution ratios based on candidates
 
 			// Nothing shared? This shouldn't happen
@@ -2383,18 +2498,103 @@ public class Block {
 		}
 
 		// Add founders as reward candidate if appropriate
-		if (haveFounders) {
+		if (haveFounders && this.blockData.getHeight() < BlockChain.getInstance().getAdminsReplaceFoundersHeight()) {
 			// Yes: add to reward candidates list
 			BlockRewardDistributor founderDistributor = (distributionAmount, balanceChanges) -> distributeBlockRewardShare(distributionAmount, onlineFounderAccounts, balanceChanges);
 
 			final long foundersShare = 1_00000000 - totalShares;
 			BlockRewardCandidate rewardCandidate = new BlockRewardCandidate("Founders", foundersShare, founderDistributor);
 			rewardCandidates.add(rewardCandidate);
+			LOGGER.info("logging foundersShare prior to reward modifications {}",foundersShare);
+		}
+		else if (this.blockData.getHeight() >= BlockChain.getInstance().getAdminsReplaceFoundersHeight()) {
+			try (final Repository repository = RepositoryManager.getRepository()) {
+				GroupRepository groupRepository = repository.getGroupRepository();
+
+				List<Integer> mintingGroupIds = Groups.getGroupIdsToMint(BlockChain.getInstance(), this.blockData.getHeight());
+
+				// all minter admins
+				List<String> minterAdmins = Groups.getAllAdmins(groupRepository, mintingGroupIds);
+
+				// all minter admins that are online
+				List<ExpandedAccount> onlineMinterAdminAccounts
+					= expandedAccounts.stream()
+						.filter(expandedAccount ->  minterAdmins.contains(expandedAccount.getMintingAccount().getAddress()))
+						.collect(Collectors.toList());
+
+				long minterAdminShare;
+
+				if( onlineMinterAdminAccounts.isEmpty() ) {
+					minterAdminShare = 0;
+				}
+				else {
+					BlockRewardDistributor minterAdminDistributor
+							= (distributionAmount, balanceChanges)
+							->
+							distributeBlockRewardShare(distributionAmount, onlineMinterAdminAccounts, balanceChanges);
+
+					long adminShare = 1_00000000 - totalShares;
+					LOGGER.info("initial total Shares: {}", totalShares);
+					LOGGER.info("logging adminShare after hardfork, this is the primary reward that will be split {}", adminShare);
+
+					minterAdminShare = adminShare / 2;
+					BlockRewardCandidate minterAdminRewardCandidate
+							= new BlockRewardCandidate("Minter Admins", minterAdminShare, minterAdminDistributor);
+					rewardCandidates.add(minterAdminRewardCandidate);
+
+					totalShares += minterAdminShare;
+				}
+
+				LOGGER.info("MINTER ADMIN SHARE: {}",minterAdminShare);
+
+				// all dev admins
+				List<String> devAdminAddresses
+						= groupRepository.getGroupAdmins(1).stream()
+						.map(GroupAdminData::getAdmin)
+						.collect(Collectors.toList());
+
+				LOGGER.info("Removing NULL Account Address, Dev Admin Count = {}", devAdminAddresses.size());
+				devAdminAddresses.removeIf( address -> Group.NULL_OWNER_ADDRESS.equals(address) );
+				LOGGER.info("Removed NULL Account Address, Dev Admin Count = {}", devAdminAddresses.size());
+
+				BlockRewardDistributor devAdminDistributor
+					= (distributionAmount, balanceChanges) -> distributeToAccounts(distributionAmount, devAdminAddresses, balanceChanges);
+
+				long devAdminShare = 1_00000000 - totalShares;
+				LOGGER.info("DEV ADMIN SHARE: {}",devAdminShare);
+				BlockRewardCandidate devAdminRewardCandidate
+					= new BlockRewardCandidate("Dev Admins", devAdminShare,devAdminDistributor);
+				rewardCandidates.add(devAdminRewardCandidate);
+			}
 		}
 
 		return rewardCandidates;
 	}
 
+	/**
+	 * Distribute To Accounts
+	 *
+	 * Merges distribute shares to a map of distribution shares.
+	 *
+	 * @param distributionAmount the amount to distribute
+	 * @param accountAddressess the addresses to distribute to
+	 * @param balanceChanges the map of distribution shares, this gets appended to
+	 *
+	 * @return the total amount mapped to addresses for distribution
+	 */
+	public static long distributeToAccounts(long distributionAmount, List<String> accountAddressess, Map<String, Long> balanceChanges) {
+
+		if( accountAddressess.isEmpty() ) return 0;
+
+		long distibutionShare = distributionAmount / accountAddressess.size();
+
+		for(String accountAddress : accountAddressess ) {
+			balanceChanges.merge(accountAddress, distibutionShare, Long::sum);
+		}
+
+		return distibutionShare * accountAddressess.size();
+	}
+
 	private static long distributeBlockRewardShare(long distributionAmount, List<ExpandedAccount> accounts, Map<String, Long> balanceChanges) {
 		// Collate all expanded accounts by minting account
 		Map<String, List<ExpandedAccount>> accountsByMinter = new HashMap<>();
@@ -2554,9 +2754,11 @@ public class Block {
 				return;
 
 			int minterLevel = Account.getRewardShareEffectiveMintingLevel(this.repository, this.getMinter().getPublicKey());
+			String minterAddress = Account.getRewardShareMintingAddress(this.repository, this.getMinter().getPublicKey());
 
 			LOGGER.debug(String.format("======= BLOCK %d (%.8s) =======", this.getBlockData().getHeight(), Base58.encode(this.getSignature())));
 			LOGGER.debug(String.format("Timestamp: %d", this.getBlockData().getTimestamp()));
+			LOGGER.debug(String.format("Minter address: %s", minterAddress));
 			LOGGER.debug(String.format("Minter level: %d", minterLevel));
 			LOGGER.debug(String.format("Online accounts: %d", this.getBlockData().getOnlineAccountsCount()));
 			LOGGER.debug(String.format("AT count: %d", this.getBlockData().getATCount()));

src/main/java/org/qortal/block/BlockChain.java

@@ -71,6 +71,7 @@ public class BlockChain {
 		transactionV6Timestamp,
 		disableReferenceTimestamp,
 		increaseOnlineAccountsDifficultyTimestamp,
+		decreaseOnlineAccountsDifficultyTimestamp,
 		onlineAccountMinterLevelValidationHeight,
 		selfSponsorshipAlgoV1Height,
 		selfSponsorshipAlgoV2Height,
@@ -85,7 +86,15 @@ public class BlockChain {
 		disableRewardshareHeight,
 		enableRewardshareHeight,
 		onlyMintWithNameHeight,
-		groupMemberCheckHeight
+		removeOnlyMintWithNameHeight,
+		groupMemberCheckHeight,
+		fixBatchRewardHeight,
+		adminsReplaceFoundersHeight,
+		nullGroupMembershipHeight,
+		ignoreLevelForRewardShareHeight,
+		adminQueryFixHeight,
+		multipleNamesPerAccountHeight,
+		mintedBlocksAdjustmentRemovalHeight
 	}
 
 	// Custom transaction fees
@@ -105,7 +114,8 @@ public class BlockChain {
 	/** Whether to use legacy, broken RIPEMD160 implementation when converting public keys to addresses. */
 	private boolean useBrokenMD160ForAddresses = false;
 
-	/** Whether only one registered name is allowed per account. */
+	/** This should get ignored and overwritten in the oneNamePerAccount(int blockchainHeight) method,
+	 * because it is based on block height, not based on the genesis block.*/
 	private boolean oneNamePerAccount = false;
 
 	/** Checkpoints */
@@ -205,7 +215,13 @@ public class BlockChain {
 	private int minAccountLevelToRewardShare;
 	private int maxRewardSharesPerFounderMintingAccount;
 	private int founderEffectiveMintingLevel;
-	private int mintingGroupId;
+
+	public static class IdsForHeight {
+		public int height;
+		public List<Integer> ids;
+	}
+
+	private List<IdsForHeight> mintingGroupIds;
 
 	/** Minimum time to retain online account signatures (ms) for block validity checks. */
 	private long onlineAccountSignaturesMinLifetime;
@@ -217,6 +233,10 @@ public class BlockChain {
 	 * featureTriggers because unit tests need to set this value via Reflection. */
 	private long onlineAccountsModulusV2Timestamp;
 
+	/** Feature trigger timestamp for ONLINE_ACCOUNTS_MODULUS time interval decrease. Can't use
+	 * featureTriggers because unit tests need to set this value via Reflection. */
+	private long onlineAccountsModulusV3Timestamp;
+
 	/** Snapshot timestamp for self sponsorship algo V1 */
 	private long selfSponsorshipAlgoV1SnapshotTimestamp;
 
@@ -403,6 +423,10 @@ public class BlockChain {
 		return this.onlineAccountsModulusV2Timestamp;
 	}
 
+	public long getOnlineAccountsModulusV3Timestamp() {
+		return this.onlineAccountsModulusV3Timestamp;
+	}
+
 	/* Block reward batching */
 	public long getBlockRewardBatchStartHeight() {
 		return this.blockRewardBatchStartHeight;
@@ -453,8 +477,9 @@ public class BlockChain {
 		return this.useBrokenMD160ForAddresses;
 	}
 
-	public boolean oneNamePerAccount() {
-		return this.oneNamePerAccount;
+	public boolean oneNamePerAccount(int blockchainHeight) {
+		// this is not set on a simple blockchain setting, it is based on a feature trigger height
+		return blockchainHeight < this.getMultipleNamesPerAccountHeight();
 	}
 
 	public List<Checkpoint> getCheckpoints() {
@@ -529,8 +554,8 @@ public class BlockChain {
 		return this.onlineAccountSignaturesMaxLifetime;
 	}
 
-	public int getMintingGroupId() {
-		return this.mintingGroupId;
+	public List<IdsForHeight> getMintingGroupIds() {
+		return mintingGroupIds;
 	}
 
 	public CiyamAtSettings getCiyamAtSettings() {
@@ -579,6 +604,10 @@ public class BlockChain {
 		return this.featureTriggers.get(FeatureTrigger.increaseOnlineAccountsDifficultyTimestamp.name()).longValue();
 	}
 
+	public long getDecreaseOnlineAccountsDifficultyTimestamp() {
+		return this.featureTriggers.get(FeatureTrigger.decreaseOnlineAccountsDifficultyTimestamp.name()).longValue();
+	}
+
 	public int getSelfSponsorshipAlgoV1Height() {
 		return this.featureTriggers.get(FeatureTrigger.selfSponsorshipAlgoV1Height.name()).intValue();
 	}
@@ -635,10 +664,42 @@ public class BlockChain {
 		return this.featureTriggers.get(FeatureTrigger.onlyMintWithNameHeight.name()).intValue();
 	}
 
+	public int getRemoveOnlyMintWithNameHeight() {
+		return this.featureTriggers.get(FeatureTrigger.removeOnlyMintWithNameHeight.name()).intValue();
+	}
+
 	public int getGroupMemberCheckHeight() {
 		return this.featureTriggers.get(FeatureTrigger.groupMemberCheckHeight.name()).intValue();
 	}
 
+	public int getFixBatchRewardHeight() {
+		return this.featureTriggers.get(FeatureTrigger.fixBatchRewardHeight.name()).intValue();
+	}
+
+	public int getAdminsReplaceFoundersHeight() {
+		return this.featureTriggers.get(FeatureTrigger.adminsReplaceFoundersHeight.name()).intValue();
+	}
+
+	public int getNullGroupMembershipHeight() {
+		return this.featureTriggers.get(FeatureTrigger.nullGroupMembershipHeight.name()).intValue();
+	}
+
+	public int getIgnoreLevelForRewardShareHeight() {
+		return this.featureTriggers.get(FeatureTrigger.ignoreLevelForRewardShareHeight.name()).intValue();
+	}
+
+	public int getAdminQueryFixHeight() {
+		return this.featureTriggers.get(FeatureTrigger.adminQueryFixHeight.name()).intValue();
+	}
+
+	public int getMultipleNamesPerAccountHeight() {
+		return this.featureTriggers.get(FeatureTrigger.multipleNamesPerAccountHeight.name()).intValue();
+	}
+
+	public int getMintedBlocksAdjustmentRemovalHeight() {
+		return this.featureTriggers.get(FeatureTrigger.mintedBlocksAdjustmentRemovalHeight.name()).intValue();
+	}
+
 	// More complex getters for aspects that change by height or timestamp
 
 	public long getRewardAtHeight(int ourHeight) {

src/main/java/org/qortal/block/PrimaryNamesBlock.java

@@ -0,0 +1,47 @@
+package org.qortal.block;
+
+import org.qortal.account.Account;
+import org.qortal.api.resource.TransactionsResource;
+import org.qortal.data.naming.NameData;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Class PrimaryNamesBlock
+ */
+public class PrimaryNamesBlock {
+
+    /**
+     * Process Primary Names
+     *
+     * @param repository
+     * @throws DataException
+     */
+    public static void processNames(Repository repository) throws DataException {
+
+        Set<String> addressesWithNames
+            = repository.getNameRepository().getAllNames().stream()
+                .map(NameData::getOwner).collect(Collectors.toSet());
+
+        // for each address with a name, set primary name to the address
+        for( String address : addressesWithNames ) {
+
+            Account account = new Account(repository, address);
+            account.resetPrimaryName(TransactionsResource.ConfirmationStatus.CONFIRMED);
+        }
+    }
+
+    /**
+     * Orphan the Primary Names Block
+     *
+     * @param repository
+     * @throws DataException
+     */
+    public static void orphanNames(Repository repository) throws DataException {
+
+        repository.getNameRepository().clearPrimaryNames();
+    }
+}

src/main/java/org/qortal/controller/BlockMinter.java

@@ -64,6 +64,7 @@ public class BlockMinter extends Thread {
 	@Override
 	public void run() {
 		Thread.currentThread().setName("BlockMinter");
+		Thread.currentThread().setPriority(MAX_PRIORITY);
 
 		if (Settings.getInstance().isTopOnly() || Settings.getInstance().isLite()) {
 			// Top only and lite nodes do not sign blocks
@@ -96,364 +97,375 @@ public class BlockMinter extends Thread {
 
 		final boolean isSingleNodeTestnet = Settings.getInstance().isSingleNodeTestnet();
 
-		try (final Repository repository = RepositoryManager.getRepository()) {
-			// Going to need this a lot...
-			BlockRepository blockRepository = repository.getBlockRepository();
-
-			// Flags for tracking change in whether minting is possible,
-			// so we can notify Controller, and further update SysTray, etc.
-			boolean isMintingPossible = false;
-			boolean wasMintingPossible = isMintingPossible;
+		// Flags for tracking change in whether minting is possible,
+		// so we can notify Controller, and further update SysTray, etc.
+		boolean isMintingPossible = false;
+		boolean wasMintingPossible = isMintingPossible;
+		try {
 			while (running) {
-				if (isMintingPossible != wasMintingPossible)
-					Controller.getInstance().onMintingPossibleChange(isMintingPossible);
+				// recreate repository for new loop iteration
+				try (final Repository repository = RepositoryManager.getRepository()) {
 
-				wasMintingPossible = isMintingPossible;
+					// Going to need this a lot...
+					BlockRepository blockRepository = repository.getBlockRepository();
 
-				try {
-					// Free up any repository locks
-					repository.discardChanges();
+					if (isMintingPossible != wasMintingPossible)
+						Controller.getInstance().onMintingPossibleChange(isMintingPossible);
 
-					// Sleep for a while.
-					// It's faster on single node testnets, to allow lots of blocks to be minted quickly.
-					Thread.sleep(isSingleNodeTestnet ? 50 : 1000);
-
-					isMintingPossible = false;
-
-					final Long now = NTP.getTime();
-					if (now == null)
-						continue;
-
-					final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
-					if (minLatestBlockTimestamp == null)
-						continue;
-
-					List<MintingAccountData> mintingAccountsData = repository.getAccountRepository().getMintingAccounts();
-					// No minting accounts?
-					if (mintingAccountsData.isEmpty())
-						continue;
-
-					// Disregard minting accounts that are no longer valid, e.g. by transfer/loss of founder flag or account level
-					// Note that minting accounts are actually reward-shares in Qortal
-					Iterator<MintingAccountData> madi = mintingAccountsData.iterator();
-					while (madi.hasNext()) {
-						MintingAccountData mintingAccountData = madi.next();
-
-						RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
-						if (rewardShareData == null) {
-							// Reward-share doesn't exist - probably cancelled but not yet removed from node's list of minting accounts
-							madi.remove();
-							continue;
-						}
-
-						Account mintingAccount = new Account(repository, rewardShareData.getMinter());
-						if (!mintingAccount.canMint()) {
-							// Minting-account component of reward-share can no longer mint - disregard
-							madi.remove();
-							continue;
-						}
-
-						// Optional (non-validated) prevention of block submissions below a defined level.
-						// This is an unvalidated version of Blockchain.minAccountLevelToMint
-						// and exists only to reduce block candidates by default.
-						int level = mintingAccount.getEffectiveMintingLevel();
-						if (level < BlockChain.getInstance().getMinAccountLevelForBlockSubmissions()) {
-							madi.remove();
-                        }
-					}
-
-					// Needs a mutable copy of the unmodifiableList
-					List<Peer> peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
-					BlockData lastBlockData = blockRepository.getLastBlock();
-
-					// Disregard peers that have "misbehaved" recently
-					peers.removeIf(Controller.hasMisbehaved);
-
-					// Disregard peers that don't have a recent block, but only if we're not in recovery mode.
-					// In that mode, we want to allow minting on top of older blocks, to recover stalled networks.
-					if (!Synchronizer.getInstance().getRecoveryMode())
-						peers.removeIf(Controller.hasNoRecentBlock);
-
-					// Don't mint if we don't have enough up-to-date peers as where would the transactions/consensus come from?
-					if (peers.size() < Settings.getInstance().getMinBlockchainPeers())
-						continue;
-
-					// If we are stuck on an invalid block, we should allow an alternative to be minted
-					boolean recoverInvalidBlock = false;
-					if (Synchronizer.getInstance().timeInvalidBlockLastReceived != null) {
-						// We've had at least one invalid block
-						long timeSinceLastValidBlock = NTP.getTime() - Synchronizer.getInstance().timeValidBlockLastReceived;
-						long timeSinceLastInvalidBlock = NTP.getTime() - Synchronizer.getInstance().timeInvalidBlockLastReceived;
-						if (timeSinceLastValidBlock > INVALID_BLOCK_RECOVERY_TIMEOUT) {
-							if (timeSinceLastInvalidBlock < INVALID_BLOCK_RECOVERY_TIMEOUT) {
-								// Last valid block was more than 10 mins ago, but we've had an invalid block since then
-								// Assume that the chain has stalled because there is no alternative valid candidate
-								// Enter recovery mode to allow alternative, valid candidates to be minted
-								recoverInvalidBlock = true;
-							}
-						}
-					}
-
-					// If our latest block isn't recent then we need to synchronize instead of minting, unless we're in recovery mode.
-					if (!peers.isEmpty() && lastBlockData.getTimestamp() < minLatestBlockTimestamp)
-						if (!Synchronizer.getInstance().getRecoveryMode() && !recoverInvalidBlock)
-							continue;
-
-					// There are enough peers with a recent block and our latest block is recent
-					// so go ahead and mint a block if possible.
-					isMintingPossible = true;
-
-					// Check blockchain hasn't changed
-					if (previousBlockData == null || !Arrays.equals(previousBlockData.getSignature(), lastBlockData.getSignature())) {
-						previousBlockData = lastBlockData;
-						newBlocks.clear();
-
-						// Reduce log timeout
-						logTimeout = 10 * 1000L;
-
-						// Last low weight block is no longer valid
-						parentSignatureForLastLowWeightBlock = null;
-					}
-
-					// Discard accounts we have already built blocks with
-					mintingAccountsData.removeIf(mintingAccountData -> newBlocks.stream().anyMatch(newBlock -> Arrays.equals(newBlock.getBlockData().getMinterPublicKey(), mintingAccountData.getPublicKey())));
-
-					// Do we need to build any potential new blocks?
-					List<PrivateKeyAccount> newBlocksMintingAccounts = mintingAccountsData.stream().map(accountData -> new PrivateKeyAccount(repository, accountData.getPrivateKey())).collect(Collectors.toList());
-
-					// We might need to sit the next block out, if one of our minting accounts signed the previous one
-					// Skip this check for single node testnets, since they definitely need to mint every block
-					byte[] previousBlockMinter = previousBlockData.getMinterPublicKey();
-					boolean mintedLastBlock = mintingAccountsData.stream().anyMatch(mintingAccount -> Arrays.equals(mintingAccount.getPublicKey(), previousBlockMinter));
-					if (mintedLastBlock && !isSingleNodeTestnet) {
-						LOGGER.trace(String.format("One of our keys signed the last block, so we won't sign the next one"));
-						continue;
-					}
-
-					if (parentSignatureForLastLowWeightBlock != null) {
-						// The last iteration found a higher weight block in the network, so sleep for a while
-						// to allow is to sync the higher weight chain. We are sleeping here rather than when
-						// detected as we don't want to hold the blockchain lock open.
-						LOGGER.info("Sleeping for 10 seconds...");
-						Thread.sleep(10 * 1000L);
-					}
-
-					for (PrivateKeyAccount mintingAccount : newBlocksMintingAccounts) {
-						// First block does the AT heavy-lifting
-						if (newBlocks.isEmpty()) {
-							Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
-							if (newBlock == null) {
-								// For some reason we can't mint right now
-								moderatedLog(() -> LOGGER.info("Couldn't build a to-be-minted block"));
-								continue;
-							}
-
-							newBlocks.add(newBlock);
-						} else {
-							// The blocks for other minters require less effort...
-							Block newBlock = newBlocks.get(0).remint(mintingAccount);
-							if (newBlock == null) {
-								// For some reason we can't mint right now
-								moderatedLog(() -> LOGGER.error("Couldn't rebuild a to-be-minted block"));
-								continue;
-							}
-
-							newBlocks.add(newBlock);
-						}
-					}
-
-					// No potential block candidates?
-					if (newBlocks.isEmpty())
-						continue;
-
-					// Make sure we're the only thread modifying the blockchain
-					ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
-					if (!blockchainLock.tryLock(30, TimeUnit.SECONDS)) {
-						LOGGER.debug("Couldn't acquire blockchain lock even after waiting 30 seconds");
-						continue;
-					}
-
-					boolean newBlockMinted = false;
-					Block newBlock = null;
+					wasMintingPossible = isMintingPossible;
 
 					try {
-						// Clear repository session state so we have latest view of data
+						// reset the repository, to the repository recreated for this loop iteration
+						for( Block newBlock : newBlocks ) newBlock.setRepository(repository);
+
+						// Free up any repository locks
 						repository.discardChanges();
 
-						// Now that we have blockchain lock, do final check that chain hasn't changed
-						BlockData latestBlockData = blockRepository.getLastBlock();
-						if (!Arrays.equals(lastBlockData.getSignature(), latestBlockData.getSignature()))
+						// Sleep for a while.
+						// It's faster on single node testnets, to allow lots of blocks to be minted quickly.
+						Thread.sleep(isSingleNodeTestnet ? 50 : 1000);
+
+						isMintingPossible = false;
+
+						final Long now = NTP.getTime();
+						if (now == null)
 							continue;
 
-						List<Block> goodBlocks = new ArrayList<>();
-						boolean wasInvalidBlockDiscarded = false;
-						Iterator<Block> newBlocksIterator = newBlocks.iterator();
+						final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
+						if (minLatestBlockTimestamp == null)
+							continue;
 
-						while (newBlocksIterator.hasNext()) {
-							Block testBlock = newBlocksIterator.next();
+						List<MintingAccountData> mintingAccountsData = repository.getAccountRepository().getMintingAccounts();
+						// No minting accounts?
+						if (mintingAccountsData.isEmpty())
+							continue;
 
-							// Is new block's timestamp valid yet?
-							// We do a separate check as some timestamp checks are skipped for testchains
-							if (testBlock.isTimestampValid() != ValidationResult.OK)
+						// Disregard minting accounts that are no longer valid, e.g. by transfer/loss of founder flag or account level
+						// Note that minting accounts are actually reward-shares in Qortal
+						Iterator<MintingAccountData> madi = mintingAccountsData.iterator();
+						while (madi.hasNext()) {
+							MintingAccountData mintingAccountData = madi.next();
+
+							RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
+							if (rewardShareData == null) {
+								// Reward-share doesn't exist - probably cancelled but not yet removed from node's list of minting accounts
+								madi.remove();
+								continue;
+							}
+
+							Account mintingAccount = new Account(repository, rewardShareData.getMinter());
+							if (!mintingAccount.canMint(true)) {
+								// Minting-account component of reward-share can no longer mint - disregard
+								madi.remove();
+								continue;
+							}
+
+							// Optional (non-validated) prevention of block submissions below a defined level.
+							// This is an unvalidated version of Blockchain.minAccountLevelToMint
+							// and exists only to reduce block candidates by default.
+							int level = mintingAccount.getEffectiveMintingLevel();
+							if (level < BlockChain.getInstance().getMinAccountLevelForBlockSubmissions()) {
+								madi.remove();
+							}
+						}
+
+						// Needs a mutable copy of the unmodifiableList
+						List<Peer> peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
+						BlockData lastBlockData = blockRepository.getLastBlock();
+
+						// Disregard peers that have "misbehaved" recently
+						peers.removeIf(Controller.hasMisbehaved);
+
+						// Disregard peers that don't have a recent block, but only if we're not in recovery mode.
+						// In that mode, we want to allow minting on top of older blocks, to recover stalled networks.
+						if (!Synchronizer.getInstance().getRecoveryMode())
+							peers.removeIf(Controller.hasNoRecentBlock);
+
+						// Don't mint if we don't have enough up-to-date peers as where would the transactions/consensus come from?
+						if (peers.size() < Settings.getInstance().getMinBlockchainPeers())
+							continue;
+
+						// If we are stuck on an invalid block, we should allow an alternative to be minted
+						boolean recoverInvalidBlock = false;
+						if (Synchronizer.getInstance().timeInvalidBlockLastReceived != null) {
+							// We've had at least one invalid block
+							long timeSinceLastValidBlock = NTP.getTime() - Synchronizer.getInstance().timeValidBlockLastReceived;
+							long timeSinceLastInvalidBlock = NTP.getTime() - Synchronizer.getInstance().timeInvalidBlockLastReceived;
+							if (timeSinceLastValidBlock > INVALID_BLOCK_RECOVERY_TIMEOUT) {
+								if (timeSinceLastInvalidBlock < INVALID_BLOCK_RECOVERY_TIMEOUT) {
+									// Last valid block was more than 10 mins ago, but we've had an invalid block since then
+									// Assume that the chain has stalled because there is no alternative valid candidate
+									// Enter recovery mode to allow alternative, valid candidates to be minted
+									recoverInvalidBlock = true;
+								}
+							}
+						}
+
+						// If our latest block isn't recent then we need to synchronize instead of minting, unless we're in recovery mode.
+						if (!peers.isEmpty() && lastBlockData.getTimestamp() < minLatestBlockTimestamp)
+							if (!Synchronizer.getInstance().getRecoveryMode() && !recoverInvalidBlock)
 								continue;
 
-							testBlock.preProcess();
+						// There are enough peers with a recent block and our latest block is recent
+						// so go ahead and mint a block if possible.
+						isMintingPossible = true;
 
-							// Is new block valid yet? (Before adding unconfirmed transactions)
-							ValidationResult result = testBlock.isValid();
-							if (result != ValidationResult.OK) {
-								moderatedLog(() -> LOGGER.error(String.format("To-be-minted block invalid '%s' before adding transactions?", result.name())));
+						// Check blockchain hasn't changed
+						if (previousBlockData == null || !Arrays.equals(previousBlockData.getSignature(), lastBlockData.getSignature())) {
+							previousBlockData = lastBlockData;
+							newBlocks.clear();
 
-								newBlocksIterator.remove();
-								wasInvalidBlockDiscarded = true;
-								/*
-								 * Bail out fast so that we loop around from the top again.
-								 * This gives BlockMinter the possibility to remint this candidate block using another block from newBlocks,
-								 * via the Blocks.remint() method, which avoids having to re-process Block ATs all over again.
-								 * Particularly useful if some aspect of Blocks changes due a timestamp-based feature-trigger (see BlockChain class).
-								 */
-								break;
-							}
+							// Reduce log timeout
+							logTimeout = 10 * 1000L;
 
-							goodBlocks.add(testBlock);
+							// Last low weight block is no longer valid
+							parentSignatureForLastLowWeightBlock = null;
 						}
 
-						if (wasInvalidBlockDiscarded || goodBlocks.isEmpty())
+						// Discard accounts we have already built blocks with
+						mintingAccountsData.removeIf(mintingAccountData -> newBlocks.stream().anyMatch(newBlock -> Arrays.equals(newBlock.getBlockData().getMinterPublicKey(), mintingAccountData.getPublicKey())));
+
+						// Do we need to build any potential new blocks?
+						List<PrivateKeyAccount> newBlocksMintingAccounts = mintingAccountsData.stream().map(accountData -> new PrivateKeyAccount(repository, accountData.getPrivateKey())).collect(Collectors.toList());
+
+						// We might need to sit the next block out, if one of our minting accounts signed the previous one
+						// Skip this check for single node testnets, since they definitely need to mint every block
+						byte[] previousBlockMinter = previousBlockData.getMinterPublicKey();
+						boolean mintedLastBlock = mintingAccountsData.stream().anyMatch(mintingAccount -> Arrays.equals(mintingAccount.getPublicKey(), previousBlockMinter));
+						if (mintedLastBlock && !isSingleNodeTestnet) {
+							LOGGER.trace(String.format("One of our keys signed the last block, so we won't sign the next one"));
 							continue;
-
-						// Pick best block
-						final int parentHeight = previousBlockData.getHeight();
-						final byte[] parentBlockSignature = previousBlockData.getSignature();
-
-						BigInteger bestWeight = null;
-
-						for (int bi = 0; bi < goodBlocks.size(); ++bi) {
-							BlockData blockData = goodBlocks.get(bi).getBlockData();
-
-							BlockSummaryData blockSummaryData = new BlockSummaryData(blockData);
-							int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
-							blockSummaryData.setMinterLevel(minterLevel);
-
-							BigInteger blockWeight = Block.calcBlockWeight(parentHeight, parentBlockSignature, blockSummaryData);
-
-							if (bestWeight == null || blockWeight.compareTo(bestWeight) < 0) {
-								newBlock = goodBlocks.get(bi);
-								bestWeight = blockWeight;
-							}
 						}
 
-						try {
-							if (this.higherWeightChainExists(repository, bestWeight)) {
+						if (parentSignatureForLastLowWeightBlock != null) {
+							// The last iteration found a higher weight block in the network, so sleep for a while
+							// to allow is to sync the higher weight chain. We are sleeping here rather than when
+							// detected as we don't want to hold the blockchain lock open.
+							LOGGER.info("Sleeping for 10 seconds...");
+							Thread.sleep(10 * 1000L);
+						}
 
-								// Check if the base block has updated since the last time we were here
-								if (parentSignatureForLastLowWeightBlock == null || timeOfLastLowWeightBlock == null ||
-										!Arrays.equals(parentSignatureForLastLowWeightBlock, previousBlockData.getSignature())) {
-									// We've switched to a different chain, so reset the timer
-									timeOfLastLowWeightBlock = NTP.getTime();
-								}
-								parentSignatureForLastLowWeightBlock = previousBlockData.getSignature();
-
-								// If less than 30 seconds has passed since first detection the higher weight chain,
-								// we should skip our block submission to give us the opportunity to sync to the better chain
-								if (NTP.getTime() - timeOfLastLowWeightBlock < 30 * 1000L) {
-									LOGGER.info("Higher weight chain found in peers, so not signing a block this round");
-									LOGGER.info("Time since detected: {}", NTP.getTime() - timeOfLastLowWeightBlock);
+						for (PrivateKeyAccount mintingAccount : newBlocksMintingAccounts) {
+							// First block does the AT heavy-lifting
+							if (newBlocks.isEmpty()) {
+								Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
+								if (newBlock == null) {
+									// For some reason we can't mint right now
+									moderatedLog(() -> LOGGER.info("Couldn't build a to-be-minted block"));
 									continue;
-								} else {
-									// More than 30 seconds have passed, so we should submit our block candidate anyway.
-									LOGGER.info("More than 30 seconds passed, so proceeding to submit block candidate...");
 								}
+
+								newBlocks.add(newBlock);
 							} else {
-								LOGGER.debug("No higher weight chain found in peers");
+								// The blocks for other minters require less effort...
+								Block newBlock = newBlocks.get(0).remint(mintingAccount);
+								if (newBlock == null) {
+									// For some reason we can't mint right now
+									moderatedLog(() -> LOGGER.error("Couldn't rebuild a to-be-minted block"));
+									continue;
+								}
+
+								newBlocks.add(newBlock);
 							}
-						} catch (DataException e) {
-							LOGGER.debug("Unable to check for a higher weight chain. Proceeding anyway...");
 						}
 
-						// Discard any uncommitted changes as a result of the higher weight chain detection
-						repository.discardChanges();
+						// No potential block candidates?
+						if (newBlocks.isEmpty())
+							continue;
 
-						// Clear variables that track low weight blocks
-						parentSignatureForLastLowWeightBlock = null;
-						timeOfLastLowWeightBlock = null;
-
-						Long unconfirmedStartTime = NTP.getTime();
-
-						// Add unconfirmed transactions
-						addUnconfirmedTransactions(repository, newBlock);
-
-						LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime()-unconfirmedStartTime)));
-
-						// Sign to create block's signature
-						newBlock.sign();
-
-						// Is newBlock still valid?
-						ValidationResult validationResult = newBlock.isValid();
-						if (validationResult != ValidationResult.OK) {
-							// No longer valid? Report and discard
-							LOGGER.error(String.format("To-be-minted block now invalid '%s' after adding unconfirmed transactions?", validationResult.name()));
-
-							// Rebuild block candidates, just to be sure
-							newBlocks.clear();
+						// Make sure we're the only thread modifying the blockchain
+						ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
+						if (!blockchainLock.tryLock(30, TimeUnit.SECONDS)) {
+							LOGGER.debug("Couldn't acquire blockchain lock even after waiting 30 seconds");
 							continue;
 						}
 
-						// Add to blockchain - something else will notice and broadcast new block to network
+						boolean newBlockMinted = false;
+						Block newBlock = null;
+
 						try {
-							newBlock.process();
+							// Clear repository session state so we have latest view of data
+							repository.discardChanges();
 
-							repository.saveChanges();
+							// Now that we have blockchain lock, do final check that chain hasn't changed
+							BlockData latestBlockData = blockRepository.getLastBlock();
+							if (!Arrays.equals(lastBlockData.getSignature(), latestBlockData.getSignature()))
+								continue;
 
-							LOGGER.info(String.format("Minted new block: %d", newBlock.getBlockData().getHeight()));
+							List<Block> goodBlocks = new ArrayList<>();
+							boolean wasInvalidBlockDiscarded = false;
+							Iterator<Block> newBlocksIterator = newBlocks.iterator();
 
-							RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(newBlock.getBlockData().getMinterPublicKey());
+							while (newBlocksIterator.hasNext()) {
+								Block testBlock = newBlocksIterator.next();
 
-							if (rewardShareData != null) {
-								LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s on behalf of %s",
-										newBlock.getBlockData().getHeight(),
-										Base58.encode(newBlock.getBlockData().getSignature()),
-										Base58.encode(newBlock.getParent().getSignature()),
-										rewardShareData.getMinter(),
-										rewardShareData.getRecipient()));
-							} else {
-								LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s",
-										newBlock.getBlockData().getHeight(),
-										Base58.encode(newBlock.getBlockData().getSignature()),
-										Base58.encode(newBlock.getParent().getSignature()),
-										newBlock.getMinter().getAddress()));
+								// Is new block's timestamp valid yet?
+								// We do a separate check as some timestamp checks are skipped for testchains
+								if (testBlock.isTimestampValid() != ValidationResult.OK)
+									continue;
+
+								testBlock.preProcess();
+
+								// Is new block valid yet? (Before adding unconfirmed transactions)
+								ValidationResult result = testBlock.isValid();
+								if (result != ValidationResult.OK) {
+									moderatedLog(() -> LOGGER.error(String.format("To-be-minted block invalid '%s' before adding transactions?", result.name())));
+
+									newBlocksIterator.remove();
+									wasInvalidBlockDiscarded = true;
+									/*
+									 * Bail out fast so that we loop around from the top again.
+									 * This gives BlockMinter the possibility to remint this candidate block using another block from newBlocks,
+									 * via the Blocks.remint() method, which avoids having to re-process Block ATs all over again.
+									 * Particularly useful if some aspect of Blocks changes due a timestamp-based feature-trigger (see BlockChain class).
+									 */
+									break;
+								}
+
+								goodBlocks.add(testBlock);
 							}
 
-							// Notify network after we're released blockchain lock
-							newBlockMinted = true;
+							if (wasInvalidBlockDiscarded || goodBlocks.isEmpty())
+								continue;
 
-							// Notify Controller
-							repository.discardChanges(); // clear transaction status to prevent deadlocks
-							Controller.getInstance().onNewBlock(newBlock.getBlockData());
-						} catch (DataException e) {
-							// Unable to process block - report and discard
-							LOGGER.error("Unable to process newly minted block?", e);
-							newBlocks.clear();
-						} catch (ArithmeticException e) {
-							// Unable to process block - report and discard
-							LOGGER.error("Unable to process newly minted block?", e);
-							newBlocks.clear();
+							// Pick best block
+							final int parentHeight = previousBlockData.getHeight();
+							final byte[] parentBlockSignature = previousBlockData.getSignature();
+
+							BigInteger bestWeight = null;
+
+							for (int bi = 0; bi < goodBlocks.size(); ++bi) {
+								BlockData blockData = goodBlocks.get(bi).getBlockData();
+
+								BlockSummaryData blockSummaryData = new BlockSummaryData(blockData);
+								int minterLevel = Account.getRewardShareEffectiveMintingLevel(repository, blockData.getMinterPublicKey());
+								blockSummaryData.setMinterLevel(minterLevel);
+
+								BigInteger blockWeight = Block.calcBlockWeight(parentHeight, parentBlockSignature, blockSummaryData);
+
+								if (bestWeight == null || blockWeight.compareTo(bestWeight) < 0) {
+									newBlock = goodBlocks.get(bi);
+									bestWeight = blockWeight;
+								}
+							}
+
+							try {
+								if (this.higherWeightChainExists(repository, bestWeight)) {
+
+									// Check if the base block has updated since the last time we were here
+									if (parentSignatureForLastLowWeightBlock == null || timeOfLastLowWeightBlock == null ||
+											!Arrays.equals(parentSignatureForLastLowWeightBlock, previousBlockData.getSignature())) {
+										// We've switched to a different chain, so reset the timer
+										timeOfLastLowWeightBlock = NTP.getTime();
+									}
+									parentSignatureForLastLowWeightBlock = previousBlockData.getSignature();
+
+									// If less than 30 seconds has passed since first detection the higher weight chain,
+									// we should skip our block submission to give us the opportunity to sync to the better chain
+									if (NTP.getTime() - timeOfLastLowWeightBlock < 30 * 1000L) {
+										LOGGER.info("Higher weight chain found in peers, so not signing a block this round");
+										LOGGER.info("Time since detected: {}", NTP.getTime() - timeOfLastLowWeightBlock);
+										continue;
+									} else {
+										// More than 30 seconds have passed, so we should submit our block candidate anyway.
+										LOGGER.info("More than 30 seconds passed, so proceeding to submit block candidate...");
+									}
+								} else {
+									LOGGER.debug("No higher weight chain found in peers");
+								}
+							} catch (DataException e) {
+								LOGGER.debug("Unable to check for a higher weight chain. Proceeding anyway...");
+							}
+
+							// Discard any uncommitted changes as a result of the higher weight chain detection
+							repository.discardChanges();
+
+							// Clear variables that track low weight blocks
+							parentSignatureForLastLowWeightBlock = null;
+							timeOfLastLowWeightBlock = null;
+
+							Long unconfirmedStartTime = NTP.getTime();
+
+							// Add unconfirmed transactions
+							addUnconfirmedTransactions(repository, newBlock);
+
+							LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime() - unconfirmedStartTime)));
+
+							// Sign to create block's signature
+							newBlock.sign();
+
+							// Is newBlock still valid?
+							ValidationResult validationResult = newBlock.isValid();
+							if (validationResult != ValidationResult.OK) {
+								// No longer valid? Report and discard
+								LOGGER.error(String.format("To-be-minted block now invalid '%s' after adding unconfirmed transactions?", validationResult.name()));
+
+								// Rebuild block candidates, just to be sure
+								newBlocks.clear();
+								continue;
+							}
+
+							// Add to blockchain - something else will notice and broadcast new block to network
+							try {
+								newBlock.process();
+
+								repository.saveChanges();
+
+								LOGGER.info(String.format("Minted new block: %d", newBlock.getBlockData().getHeight()));
+
+								RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(newBlock.getBlockData().getMinterPublicKey());
+
+								if (rewardShareData != null) {
+									LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s on behalf of %s",
+											newBlock.getBlockData().getHeight(),
+											Base58.encode(newBlock.getBlockData().getSignature()),
+											Base58.encode(newBlock.getParent().getSignature()),
+											rewardShareData.getMinter(),
+											rewardShareData.getRecipient()));
+								} else {
+									LOGGER.info(String.format("Minted block %d, sig %.8s, parent sig: %.8s by %s",
+											newBlock.getBlockData().getHeight(),
+											Base58.encode(newBlock.getBlockData().getSignature()),
+											Base58.encode(newBlock.getParent().getSignature()),
+											newBlock.getMinter().getAddress()));
+								}
+
+								// Notify network after we're released blockchain lock
+								newBlockMinted = true;
+
+								// Notify Controller
+								repository.discardChanges(); // clear transaction status to prevent deadlocks
+								Controller.getInstance().onNewBlock(newBlock.getBlockData());
+							} catch (DataException e) {
+								// Unable to process block - report and discard
+								LOGGER.error("Unable to process newly minted block?", e);
+								newBlocks.clear();
+							} catch (ArithmeticException e) {
+								// Unable to process block - report and discard
+								LOGGER.error("Unable to process newly minted block?", e);
+								newBlocks.clear();
+							}
+						} finally {
+							blockchainLock.unlock();
 						}
-					} finally {
-						blockchainLock.unlock();
-					}
 
-					if (newBlockMinted) {
-						// Broadcast our new chain to network
-						Network.getInstance().broadcastOurChain();
-					}
+						if (newBlockMinted) {
+							// Broadcast our new chain to network
+							Network.getInstance().broadcastOurChain();
+						}
 
-				} catch (InterruptedException e) {
-					// We've been interrupted - time to exit
-					return;
+					} catch (InterruptedException e) {
+						// We've been interrupted - time to exit
+						return;
+					}
+				} catch (DataException e) {
+					LOGGER.warn("Repository issue while running block minter - NO LONGER MINTING", e);
+				} catch (Exception e) {
+					LOGGER.error(e.getMessage(), e);
 				}
 			}
-		} catch (DataException e) {
-			LOGGER.warn("Repository issue while running block minter - NO LONGER MINTING", e);
+		} catch (Exception e) {
+			LOGGER.error(e.getMessage(), e);
 		}
 	}
 

src/main/java/org/qortal/controller/OnlineAccountsManager.java

@@ -13,6 +13,7 @@ import org.qortal.crypto.MemoryPoW;
 import org.qortal.crypto.Qortal25519Extras;
 import org.qortal.data.account.MintingAccountData;
 import org.qortal.data.account.RewardShareData;
+import org.qortal.data.group.GroupMemberData;
 import org.qortal.data.network.OnlineAccountData;
 import org.qortal.network.Network;
 import org.qortal.network.Peer;
@@ -24,6 +25,7 @@ import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Groups;
 import org.qortal.utils.NTP;
 import org.qortal.utils.NamedThreadFactory;
 
@@ -44,6 +46,7 @@ public class OnlineAccountsManager {
      */
     private static final long ONLINE_TIMESTAMP_MODULUS_V1 = 5 * 60 * 1000L;
     private static final long ONLINE_TIMESTAMP_MODULUS_V2 = 30 * 60 * 1000L;
+    private static final long ONLINE_TIMESTAMP_MODULUS_V3 = 10 * 60 * 1000L;
 
     /**
      * How many 'current' timestamp-sets of online accounts we cache.
@@ -67,12 +70,13 @@ public class OnlineAccountsManager {
     private static final long ONLINE_ACCOUNTS_COMPUTE_INITIAL_SLEEP_INTERVAL = 30 * 1000L; // ms
 
     // MemoryPoW - mainnet
-    public static final int POW_BUFFER_SIZE = 1 * 1024 * 1024; // bytes
+    public static final int POW_BUFFER_SIZE = 1024 * 1024; // bytes
     public static final int POW_DIFFICULTY_V1 = 18; // leading zero bits
     public static final int POW_DIFFICULTY_V2 = 19; // leading zero bits
+    public static final int POW_DIFFICULTY_V3 = 6; // leading zero bits
 
     // MemoryPoW - testnet
-    public static final int POW_BUFFER_SIZE_TESTNET = 1 * 1024 * 1024; // bytes
+    public static final int POW_BUFFER_SIZE_TESTNET = 1024 * 1024; // bytes
     public static final int POW_DIFFICULTY_TESTNET = 5; // leading zero bits
 
     // IMPORTANT: if we ever need to dynamically modify the buffer size using a feature trigger, the
@@ -80,7 +84,7 @@ public class OnlineAccountsManager {
     // one for the transition period.
     private static long[] POW_VERIFY_WORK_BUFFER = new long[getPoWBufferSize() / 8];
 
-    private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("OnlineAccounts"));
+    private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("OnlineAccounts", Thread.NORM_PRIORITY));
     private volatile boolean isStopping = false;
 
     private final Set<OnlineAccountData> onlineAccountsImportQueue = ConcurrentHashMap.newKeySet();
@@ -106,11 +110,15 @@ public class OnlineAccountsManager {
 
     public static long getOnlineTimestampModulus() {
         Long now = NTP.getTime();
-        if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV2Timestamp()) {
+        if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV2Timestamp() && now < BlockChain.getInstance().getOnlineAccountsModulusV3Timestamp()) {
             return ONLINE_TIMESTAMP_MODULUS_V2;
         }
+        if (now != null && now >= BlockChain.getInstance().getOnlineAccountsModulusV3Timestamp()) {
+            return ONLINE_TIMESTAMP_MODULUS_V3;
+        }
         return ONLINE_TIMESTAMP_MODULUS_V1;
     }
+
     public static Long getCurrentOnlineAccountTimestamp() {
         Long now = NTP.getTime();
         if (now == null)
@@ -135,9 +143,12 @@ public class OnlineAccountsManager {
         if (Settings.getInstance().isTestNet())
             return POW_DIFFICULTY_TESTNET;
 
-        if (timestamp >= BlockChain.getInstance().getIncreaseOnlineAccountsDifficultyTimestamp())
+        if (timestamp >= BlockChain.getInstance().getIncreaseOnlineAccountsDifficultyTimestamp() && timestamp < BlockChain.getInstance().getDecreaseOnlineAccountsDifficultyTimestamp())
             return POW_DIFFICULTY_V2;
 
+        if (timestamp >= BlockChain.getInstance().getDecreaseOnlineAccountsDifficultyTimestamp())
+            return POW_DIFFICULTY_V3;
+
         return POW_DIFFICULTY_V1;
     }
 
@@ -215,6 +226,15 @@ public class OnlineAccountsManager {
         Set<OnlineAccountData> onlineAccountsToAdd = new HashSet<>();
         Set<OnlineAccountData> onlineAccountsToRemove = new HashSet<>();
         try (final Repository repository = RepositoryManager.getRepository()) {
+
+            int blockHeight = repository.getBlockRepository().getBlockchainHeight();
+
+            List<String> mintingGroupMemberAddresses
+                = Groups.getAllMembers(
+                    repository.getGroupRepository(),
+                    Groups.getGroupIdsToMint(BlockChain.getInstance(), blockHeight)
+            );
+
             for (OnlineAccountData onlineAccountData : this.onlineAccountsImportQueue) {
                 if (isStopping)
                     return;
@@ -227,7 +247,7 @@ public class OnlineAccountsManager {
                     continue;
                 }
 
-                boolean isValid = this.isValidCurrentAccount(repository, onlineAccountData);
+                boolean isValid = this.isValidCurrentAccount(repository, mintingGroupMemberAddresses, onlineAccountData);
                 if (isValid)
                     onlineAccountsToAdd.add(onlineAccountData);
 
@@ -306,7 +326,7 @@ public class OnlineAccountsManager {
         return inplaceArray;
     }
 
-    private static boolean isValidCurrentAccount(Repository repository, OnlineAccountData onlineAccountData) throws DataException {
+    private static boolean isValidCurrentAccount(Repository repository, List<String> mintingGroupMemberAddresses, OnlineAccountData onlineAccountData) throws DataException {
         final Long now = NTP.getTime();
         if (now == null)
             return false;
@@ -341,9 +361,14 @@ public class OnlineAccountsManager {
             LOGGER.trace(() -> String.format("Rejecting unknown online reward-share public key %s", Base58.encode(rewardSharePublicKey)));
             return false;
         }
+        // reject account address that are not in the MINTER Group
+        else if( !mintingGroupMemberAddresses.contains(rewardShareData.getMinter())) {
+            LOGGER.trace(() -> String.format("Rejecting online reward-share that is not in MINTER Group, account %s", rewardShareData.getMinter()));
+            return false;
+        }
 
         Account mintingAccount = new Account(repository, rewardShareData.getMinter());
-        if (!mintingAccount.canMint()) {
+        if (!mintingAccount.canMint(true)) {  // group validation is a few lines above
             // Minting-account component of reward-share can no longer mint - disregard
             LOGGER.trace(() -> String.format("Rejecting online reward-share with non-minting account %s", mintingAccount.getAddress()));
             return false;
@@ -530,7 +555,7 @@ public class OnlineAccountsManager {
                     }
 
                     Account mintingAccount = new Account(repository, rewardShareData.getMinter());
-                    if (!mintingAccount.canMint()) {
+                    if (!mintingAccount.canMint(true)) {
                         // Minting-account component of reward-share can no longer mint - disregard
                         iterator.remove();
                         continue;

src/main/java/org/qortal/controller/PirateChainWalletController.java

@@ -65,6 +65,7 @@ public class PirateChainWalletController extends Thread {
     @Override
     public void run() {
         Thread.currentThread().setName("Pirate Chain Wallet Controller");
+        Thread.currentThread().setPriority(MIN_PRIORITY);
 
         try {
             while (running && !Controller.isStopping()) {

src/main/java/org/qortal/controller/RNSTransactionImporter.java

@@ -0,0 +1,460 @@
+package org.qortal.controller;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.data.block.BlockData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.network.RNSNetwork;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.GetTransactionMessage;
+import org.qortal.network.message.Message;
+import org.qortal.network.message.TransactionMessage;
+import org.qortal.network.message.TransactionSignaturesMessage;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.TransformationException;
+import org.qortal.utils.Base58;
+import org.qortal.utils.NTP;
+
+import java.util.*;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+public class RNSTransactionImporter extends Thread {
+
+    private static final Logger LOGGER = LogManager.getLogger(RNSTransactionImporter.class);
+
+    private static RNSTransactionImporter instance;
+    private volatile boolean isStopping = false;
+
+    private static final int MAX_INCOMING_TRANSACTIONS = 5000;
+
+    /** Minimum time before considering an invalid unconfirmed transaction as "stale" */
+    public static final long INVALID_TRANSACTION_STALE_TIMEOUT = 30 * 60 * 1000L; // ms
+    /** Minimum frequency to re-request stale unconfirmed transactions from peers, to recheck validity */
+    public static final long INVALID_TRANSACTION_RECHECK_INTERVAL = 60 * 60 * 1000L; // ms\
+    /** Minimum frequency to re-request expired unconfirmed transactions from peers, to recheck validity
+     * This mainly exists to stop expired transactions from bloating the list */
+    public static final long EXPIRED_TRANSACTION_RECHECK_INTERVAL = 10 * 60 * 1000L; // ms
+
+
+    /** Map of incoming transaction that are in the import queue. Key is transaction data, value is whether signature has been validated. */
+    private final Map<TransactionData, Boolean> incomingTransactions = Collections.synchronizedMap(new HashMap<>());
+
+    /** Map of recent invalid unconfirmed transactions. Key is base58 transaction signature, value is do-not-request expiry timestamp. */
+    private final Map<String, Long> invalidUnconfirmedTransactions = Collections.synchronizedMap(new HashMap<>());
+
+    /** Cached list of unconfirmed transactions, used when counting per creator. This is replaced regularly */
+    public static List<TransactionData> unconfirmedTransactionsCache = null;
+
+
+    public static synchronized RNSTransactionImporter getInstance() {
+        if (instance == null) {
+            instance = new RNSTransactionImporter();
+        }
+
+        return instance;
+    }
+
+    @Override
+    public void run() {
+        Thread.currentThread().setName("Transaction Importer");
+
+        try {
+            while (!Controller.isStopping()) {
+                Thread.sleep(500L);
+
+                // Process incoming transactions queue
+                validateTransactionsInQueue();
+                importTransactionsInQueue();
+
+                // Clean up invalid incoming transactions list
+                cleanupInvalidTransactionsList(NTP.getTime());
+            }
+        } catch (InterruptedException e) {
+            // Fall through to exit thread
+        }
+    }
+
+    public void shutdown() {
+        isStopping = true;
+        this.interrupt();
+    }
+
+
+    // Incoming transactions queue
+
+    private boolean incomingTransactionQueueContains(byte[] signature) {
+        synchronized (incomingTransactions) {
+            return incomingTransactions.keySet().stream().anyMatch(t -> Arrays.equals(t.getSignature(), signature));
+        }
+    }
+
+    private void removeIncomingTransaction(byte[] signature) {
+        incomingTransactions.keySet().removeIf(t -> Arrays.equals(t.getSignature(), signature));
+    }
+
+    /**
+     * Retrieve all pending unconfirmed transactions that have had their signatures validated.
+     * @return a list of TransactionData objects, with valid signatures.
+     */
+    private List<TransactionData> getCachedSigValidTransactions() {
+        synchronized (this.incomingTransactions) {
+            return this.incomingTransactions.entrySet().stream()
+                    .filter(t -> Boolean.TRUE.equals(t.getValue()))
+                    .map(Map.Entry::getKey)
+                    .collect(Collectors.toList());
+        }
+    }
+
+    /**
+     * Validate the signatures of any transactions pending import, then update their
+     * entries in the queue to mark them as valid/invalid.
+     *
+     * No database lock is required.
+     */
+    private void validateTransactionsInQueue() {
+        if (this.incomingTransactions.isEmpty()) {
+            // Nothing to do?
+            return;
+        }
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Take a snapshot of incomingTransactions, so we don't need to lock it while processing
+            Map<TransactionData, Boolean> incomingTransactionsCopy = Map.copyOf(this.incomingTransactions);
+
+            int unvalidatedCount = Collections.frequency(incomingTransactionsCopy.values(), Boolean.FALSE);
+            int validatedCount = 0;
+
+            if (unvalidatedCount > 0) {
+                LOGGER.debug("Validating signatures in incoming transactions queue (size {})...", unvalidatedCount);
+            }
+
+            // A list of all currently pending transactions that have valid signatures
+            List<Transaction> sigValidTransactions = new ArrayList<>();
+
+            // A list of signatures that became valid in this round
+            List<byte[]> newlyValidSignatures = new ArrayList<>();
+
+            boolean isLiteNode = Settings.getInstance().isLite();
+
+            // We need the latest block in order to check for expired transactions
+            BlockData latestBlock = Controller.getInstance().getChainTip();
+
+            // Signature validation round - does not require blockchain lock
+            for (Map.Entry<TransactionData, Boolean> transactionEntry : incomingTransactionsCopy.entrySet()) {
+                // Quick exit?
+                if (isStopping) {
+                    return;
+                }
+
+                TransactionData transactionData = transactionEntry.getKey();
+                Transaction transaction = Transaction.fromData(repository, transactionData);
+                String signature58 = Base58.encode(transactionData.getSignature());
+
+                Long now = NTP.getTime();
+                if (now == null) {
+                    return;
+                }
+
+                // Drop expired transactions before they are considered "sig valid"
+                if (latestBlock != null && transaction.getDeadline() <= latestBlock.getTimestamp()) {
+                    LOGGER.debug("Removing expired {} transaction {} from import queue", transactionData.getType().name(), signature58);
+                    removeIncomingTransaction(transactionData.getSignature());
+                    invalidUnconfirmedTransactions.put(signature58, (now + EXPIRED_TRANSACTION_RECHECK_INTERVAL));
+                    continue;
+                }
+
+                // Only validate signature if we haven't already done so
+                Boolean isSigValid = transactionEntry.getValue();
+                if (!Boolean.TRUE.equals(isSigValid)) {
+                    if (isLiteNode) {
+                        // Lite nodes can't easily validate transactions, so for now we will have to assume that everything is valid
+                        sigValidTransactions.add(transaction);
+                        newlyValidSignatures.add(transactionData.getSignature());
+                        // Add mark signature as valid if transaction still exists in import queue
+                        incomingTransactions.computeIfPresent(transactionData, (k, v) -> Boolean.TRUE);
+                        continue;
+                    }
+
+                    if (!transaction.isSignatureValid()) {
+                        LOGGER.debug("Ignoring {} transaction {} with invalid signature", transactionData.getType().name(), signature58);
+                        removeIncomingTransaction(transactionData.getSignature());
+
+                        // Also add to invalidIncomingTransactions map
+                        now = NTP.getTime();
+                        if (now != null) {
+                            Long expiry = now + INVALID_TRANSACTION_RECHECK_INTERVAL;
+                            LOGGER.trace("Adding invalid transaction {} to invalidUnconfirmedTransactions...", signature58);
+                            // Add to invalidUnconfirmedTransactions so that we don't keep requesting it
+                            invalidUnconfirmedTransactions.put(signature58, expiry);
+                        }
+
+                        // We're done with this transaction
+                        continue;
+                    }
+
+                    // Count the number that were validated in this round, for logging purposes
+                    validatedCount++;
+
+                    // Add mark signature as valid if transaction still exists in import queue
+                    incomingTransactions.computeIfPresent(transactionData, (k, v) -> Boolean.TRUE);
+
+                    // Signature validated in this round
+                    newlyValidSignatures.add(transactionData.getSignature());
+
+                } else {
+                    LOGGER.trace(() -> String.format("Transaction %s known to have valid signature", Base58.encode(transactionData.getSignature())));
+                }
+
+                // Signature valid - add to shortlist
+                sigValidTransactions.add(transaction);
+            }
+
+            if (unvalidatedCount > 0) {
+                LOGGER.debug("Finished validating signatures in incoming transactions queue (valid this round: {}, total pending import: {})...", validatedCount, sigValidTransactions.size());
+            }
+
+        } catch (DataException e) {
+            LOGGER.error("Repository issue while processing incoming transactions", e);
+        }
+    }
+
+    /**
+     * Import any transactions in the queue that have valid signatures.
+     *
+     * A database lock is required.
+     */
+    private void importTransactionsInQueue() {
+        List<TransactionData> sigValidTransactions = this.getCachedSigValidTransactions();
+        if (sigValidTransactions.isEmpty()) {
+            // Don't bother locking if there are no new transactions to process
+            return;
+        }
+
+        if (Synchronizer.getInstance().isSyncRequested() || Synchronizer.getInstance().isSynchronizing()) {
+            // Prioritize syncing, and don't attempt to lock
+            return;
+        }
+
+        ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
+        if (!blockchainLock.tryLock()) {
+            LOGGER.debug("Too busy to import incoming transactions queue");
+            return;
+        }
+
+        LOGGER.debug("Importing incoming transactions queue (size {})...", sigValidTransactions.size());
+
+        int processedCount = 0;
+        try (final Repository repository = RepositoryManager.getRepository()) {
+
+            // Use a single copy of the unconfirmed transactions list for each cycle, to speed up constant lookups
+            // when counting unconfirmed transactions by creator.
+            List<TransactionData> unconfirmedTransactions = repository.getTransactionRepository().getUnconfirmedTransactions();
+            unconfirmedTransactions.removeIf(t -> t.getType() == Transaction.TransactionType.CHAT);
+            unconfirmedTransactionsCache = unconfirmedTransactions;
+
+            // A list of signatures were imported in this round
+            List<byte[]> newlyImportedSignatures = new ArrayList<>();
+
+            // Import transactions with valid signatures
+            try {
+                for (int i = 0; i < sigValidTransactions.size(); ++i) {
+                    if (isStopping) {
+                        return;
+                    }
+
+                    if (Synchronizer.getInstance().isSyncRequestPending()) {
+                        LOGGER.debug("Breaking out of transaction importing with {} remaining, because a sync request is pending", sigValidTransactions.size() - i);
+                        return;
+                    }
+
+                    TransactionData transactionData = sigValidTransactions.get(i);
+                    Transaction transaction = Transaction.fromData(repository, transactionData);
+
+                    Transaction.ValidationResult validationResult = transaction.importAsUnconfirmed();
+                    processedCount++;
+
+                    switch (validationResult) {
+                        case TRANSACTION_ALREADY_EXISTS: {
+                            LOGGER.trace(() -> String.format("Ignoring existing transaction %s", Base58.encode(transactionData.getSignature())));
+                            break;
+                        }
+
+                        case NO_BLOCKCHAIN_LOCK: {
+                            // Is this even possible considering we acquired blockchain lock above?
+                            LOGGER.trace(() -> String.format("Couldn't lock blockchain to import unconfirmed transaction %s", Base58.encode(transactionData.getSignature())));
+                            break;
+                        }
+
+                        case OK: {
+                            LOGGER.debug(() -> String.format("Imported %s transaction %s", transactionData.getType().name(), Base58.encode(transactionData.getSignature())));
+
+                            // Add to the unconfirmed transactions cache
+                            if (transactionData.getType() != Transaction.TransactionType.CHAT && unconfirmedTransactionsCache != null) {
+                                unconfirmedTransactionsCache.add(transactionData);
+                            }
+
+                            // Signature imported in this round
+                            newlyImportedSignatures.add(transactionData.getSignature());
+
+                            break;
+                        }
+
+                        // All other invalid cases:
+                        default: {
+                            final String signature58 = Base58.encode(transactionData.getSignature());
+                            LOGGER.debug(() -> String.format("Ignoring invalid (%s) %s transaction %s", validationResult.name(), transactionData.getType().name(), signature58));
+
+                            Long now = NTP.getTime();
+                            if (now != null && now - transactionData.getTimestamp() > INVALID_TRANSACTION_STALE_TIMEOUT) {
+                                Long expiryLength = INVALID_TRANSACTION_RECHECK_INTERVAL;
+
+                                if (validationResult == Transaction.ValidationResult.TIMESTAMP_TOO_OLD) {
+                                    // Use shorter recheck interval for expired transactions
+                                    expiryLength = EXPIRED_TRANSACTION_RECHECK_INTERVAL;
+                                }
+
+                                Long expiry = now + expiryLength;
+                                LOGGER.trace("Adding stale invalid transaction {} to invalidUnconfirmedTransactions...", signature58);
+                                // Invalid, unconfirmed transaction has become stale - add to invalidUnconfirmedTransactions so that we don't keep requesting it
+                                invalidUnconfirmedTransactions.put(signature58, expiry);
+                            }
+                        }
+                    }
+
+                    // Transaction has been processed, even if only to reject it
+                    removeIncomingTransaction(transactionData.getSignature());
+                }
+
+                if (!newlyImportedSignatures.isEmpty()) {
+                    LOGGER.debug("Broadcasting {} newly imported signatures", newlyImportedSignatures.size());
+                    Message newTransactionSignatureMessage = new TransactionSignaturesMessage(newlyImportedSignatures);
+                    RNSNetwork.getInstance().broadcast(broadcastPeer -> newTransactionSignatureMessage);
+                }
+            } finally {
+                LOGGER.debug("Finished importing {} incoming transaction{}", processedCount, (processedCount == 1 ? "" : "s"));
+                blockchainLock.unlock();
+
+                // Clear the unconfirmed transaction cache so new data can be populated in the next cycle
+                unconfirmedTransactionsCache = null;
+            }
+        } catch (DataException e) {
+            LOGGER.error("Repository issue while importing incoming transactions", e);
+        }
+    }
+
+    private void cleanupInvalidTransactionsList(Long now) {
+        if (now == null) {
+            return;
+        }
+        // Periodically remove invalid unconfirmed transactions from the list, so that they can be fetched again
+        invalidUnconfirmedTransactions.entrySet().removeIf(entry -> entry.getValue() == null || entry.getValue() < now);
+    }
+
+
+    // Network handlers
+
+    public void onNetworkTransactionMessage(RNSPeer peer, Message message) {
+        TransactionMessage transactionMessage = (TransactionMessage) message;
+        TransactionData transactionData = transactionMessage.getTransactionData();
+
+        if (this.incomingTransactions.size() < MAX_INCOMING_TRANSACTIONS) {
+            synchronized (this.incomingTransactions) {
+                if (!incomingTransactionQueueContains(transactionData.getSignature())) {
+                    this.incomingTransactions.put(transactionData, Boolean.FALSE);
+                }
+            }
+        }
+    }
+
+    public void onNetworkGetTransactionMessage(RNSPeer peer, Message message) {
+        GetTransactionMessage getTransactionMessage = (GetTransactionMessage) message;
+        byte[] signature = getTransactionMessage.getSignature();
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Firstly check the sig-valid transactions that are currently queued for import
+            TransactionData transactionData = this.getCachedSigValidTransactions().stream()
+                    .filter(t -> Arrays.equals(signature, t.getSignature()))
+                    .findFirst().orElse(null);
+
+            if (transactionData == null) {
+                // Not found in import queue, so try the database
+                transactionData = repository.getTransactionRepository().fromSignature(signature);
+            }
+
+            if (transactionData == null) {
+                // Still not found - so we don't have this transaction
+                LOGGER.debug(() -> String.format("Ignoring GET_TRANSACTION request from peer %s for unknown transaction %s", peer, Base58.encode(signature)));
+                // Send no response at all???
+                return;
+            }
+
+            Message transactionMessage = new TransactionMessage(transactionData);
+            transactionMessage.setId(message.getId());
+            peer.sendMessage(transactionMessage);
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
+        } catch (TransformationException e) {
+            LOGGER.error(String.format("Serialization issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
+        }
+    }
+
+    public void onNetworkGetUnconfirmedTransactionsMessage(RNSPeer peer, Message message) {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            List<byte[]> signatures = Collections.emptyList();
+
+            // If we're NOT up-to-date then don't send out unconfirmed transactions
+            // as it's possible they are already included in a later block that we don't have.
+            if (Controller.getInstance().isUpToDate())
+                signatures = repository.getTransactionRepository().getUnconfirmedTransactionSignatures();
+
+            Message transactionSignaturesMessage = new TransactionSignaturesMessage(signatures);
+            peer.sendMessage(transactionSignaturesMessage);
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while sending unconfirmed transaction signatures to peer %s", peer), e);
+        }
+    }
+
+    public void onNetworkTransactionSignaturesMessage(RNSPeer peer, Message message) {
+        TransactionSignaturesMessage transactionSignaturesMessage = (TransactionSignaturesMessage) message;
+        List<byte[]> signatures = transactionSignaturesMessage.getSignatures();
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            for (byte[] signature : signatures) {
+                String signature58 = Base58.encode(signature);
+                if (invalidUnconfirmedTransactions.containsKey(signature58)) {
+                    // Previously invalid transaction - don't keep requesting it
+                    // It will be periodically removed from invalidUnconfirmedTransactions to allow for rechecks
+                    continue;
+                }
+
+                // Ignore if this transaction is in the queue
+                if (incomingTransactionQueueContains(signature)) {
+                    LOGGER.trace(() -> String.format("Ignoring existing queued transaction %s from peer %s", Base58.encode(signature), peer));
+                    continue;
+                }
+
+                // Do we have it already? (Before requesting transaction data itself)
+                if (repository.getTransactionRepository().exists(signature)) {
+                    LOGGER.trace(() -> String.format("Ignoring existing transaction %s from peer %s", Base58.encode(signature), peer));
+                    continue;
+                }
+
+                // Check isInterrupted() here and exit fast
+                if (Thread.currentThread().isInterrupted())
+                    return;
+
+                // Fetch actual transaction data from peer
+                Message getTransactionMessage = new GetTransactionMessage(signature);
+                peer.sendMessage(getTransactionMessage);
+            }
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while processing unconfirmed transactions from peer %s", peer), e);
+        }
+    }
+
+}

src/main/java/org/qortal/controller/Synchronizer.java

@@ -118,8 +118,12 @@ public class Synchronizer extends Thread {
 	}
 
 	public static Synchronizer getInstance() {
-		if (instance == null)
+		if (instance == null) {
 			instance = new Synchronizer();
+			instance.setPriority(Settings.getInstance().getSynchronizerThreadPriority());
+
+			LOGGER.info("thread priority = " + instance.getPriority());
+		}
 
 		return instance;
 	}

src/main/java/org/qortal/controller/TransactionImporter.java

@@ -2,6 +2,7 @@ package org.qortal.controller;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.qortal.controller.arbitrary.PeerMessage;
 import org.qortal.data.block.BlockData;
 import org.qortal.data.transaction.TransactionData;
 import org.qortal.network.Network;
@@ -20,7 +21,11 @@ import org.qortal.utils.Base58;
 import org.qortal.utils.NTP;
 
 import java.util.*;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Function;
 import java.util.stream.Collectors;
 
 public class TransactionImporter extends Thread {
@@ -50,6 +55,10 @@ public class TransactionImporter extends Thread {
     /** Cached list of unconfirmed transactions, used when counting per creator. This is replaced regularly */
     public static List<TransactionData> unconfirmedTransactionsCache = null;
 
+    public TransactionImporter() {
+        signatureMessageScheduler.scheduleAtFixedRate(this::processNetworkTransactionSignaturesMessage, 60, 1, TimeUnit.SECONDS);
+        getTransactionMessageScheduler.scheduleAtFixedRate(this::processNetworkGetTransactionMessages, 60, 1, TimeUnit.SECONDS);
+    }
 
     public static synchronized TransactionImporter getInstance() {
         if (instance == null) {
@@ -371,36 +380,104 @@ public class TransactionImporter extends Thread {
         }
     }
 
+    // List to collect messages
+    private final List<PeerMessage> getTransactionMessageList = new ArrayList<>();
+    // Lock to synchronize access to the list
+    private final Object getTransactionMessageLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService getTransactionMessageScheduler = Executors.newScheduledThreadPool(1);
+
     public void onNetworkGetTransactionMessage(Peer peer, Message message) {
-        GetTransactionMessage getTransactionMessage = (GetTransactionMessage) message;
-        byte[] signature = getTransactionMessage.getSignature();
 
-        try (final Repository repository = RepositoryManager.getRepository()) {
+        synchronized (getTransactionMessageLock) {
+            getTransactionMessageList.add(new PeerMessage(peer, message));
+        }
+    }
+
+    private void processNetworkGetTransactionMessages() {
+
+        try {
+            List<PeerMessage> messagesToProcess;
+            synchronized (getTransactionMessageLock) {
+                messagesToProcess = new ArrayList<>(getTransactionMessageList);
+                getTransactionMessageList.clear();
+            }
+
+            if( messagesToProcess.isEmpty() ) return;
+
+            Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
+
+            for( PeerMessage peerMessage : messagesToProcess ) {
+                GetTransactionMessage getTransactionMessage = (GetTransactionMessage) peerMessage.getMessage();
+                byte[] signature = getTransactionMessage.getSignature();
+
+                peerMessageBySignature58.put(Base58.encode(signature), peerMessage);
+            }
+
             // Firstly check the sig-valid transactions that are currently queued for import
-            TransactionData transactionData = this.getCachedSigValidTransactions().stream()
-                    .filter(t -> Arrays.equals(signature, t.getSignature()))
-                    .findFirst().orElse(null);
+            Map<String, TransactionData> transactionsCachedBySignature58
+                = this.getCachedSigValidTransactions().stream()
+                    .collect(Collectors.toMap(t -> Base58.encode(t.getSignature()), Function.identity()));
 
-            if (transactionData == null) {
+            Map<Boolean, List<Map.Entry<String, PeerMessage>>> transactionsCachedBySignature58Partition
+                = peerMessageBySignature58.entrySet().stream()
+                    .collect(Collectors.partitioningBy(entry -> transactionsCachedBySignature58.containsKey(entry.getKey())));
+
+            List<byte[]> signaturesNeeded
+                = transactionsCachedBySignature58Partition.get(false).stream()
+                    .map(Map.Entry::getValue)
+                    .map(PeerMessage::getMessage)
+                    .map(message -> (GetTransactionMessage) message)
+                    .map(GetTransactionMessage::getSignature)
+                    .collect(Collectors.toList());
+
+            // transaction found in the import queue
+            Map<String, TransactionData> transactionsToSendBySignature58 = new HashMap<>(messagesToProcess.size());
+            for( Map.Entry<String, PeerMessage> entry : transactionsCachedBySignature58Partition.get(true)) {
+                transactionsToSendBySignature58.put(entry.getKey(), transactionsCachedBySignature58.get(entry.getKey()));
+            }
+
+            if( !signaturesNeeded.isEmpty() ) {
                 // Not found in import queue, so try the database
-                transactionData = repository.getTransactionRepository().fromSignature(signature);
+                try (final Repository repository = RepositoryManager.getRepository()) {
+                    transactionsToSendBySignature58.putAll(
+                            repository.getTransactionRepository().fromSignatures(signaturesNeeded).stream()
+                                    .collect(Collectors.toMap(transactionData -> Base58.encode(transactionData.getSignature()), Function.identity()))
+                    );
+                } catch (DataException e) {
+                    LOGGER.error(e.getMessage(), e);
+                }
             }
 
-            if (transactionData == null) {
-                // Still not found - so we don't have this transaction
-                LOGGER.debug(() -> String.format("Ignoring GET_TRANSACTION request from peer %s for unknown transaction %s", peer, Base58.encode(signature)));
-                // Send no response at all???
-                return;
-            }
+            for( final Map.Entry<String, TransactionData> entry : transactionsToSendBySignature58.entrySet() ) {
 
-            Message transactionMessage = new TransactionMessage(transactionData);
+                PeerMessage peerMessage = peerMessageBySignature58.get(entry.getKey());
+                final Message message = peerMessage.getMessage();
+                final Peer peer = peerMessage.getPeer();
+
+                Runnable sendTransactionMessageRunner = () -> sendTransactionMessage(entry.getKey(), entry.getValue(), message, peer);
+                Thread sendTransactionMessageThread = new Thread(sendTransactionMessageRunner);
+                sendTransactionMessageThread.start();
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(),e);
+        }
+    }
+
+    private static void sendTransactionMessage(String signature58, TransactionData data, Message message, Peer peer) {
+        try {
+            Message transactionMessage = new TransactionMessage(data);
             transactionMessage.setId(message.getId());
+
             if (!peer.sendMessage(transactionMessage))
                 peer.disconnect("failed to send transaction");
-        } catch (DataException e) {
-            LOGGER.error(String.format("Repository issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
-        } catch (TransformationException e) {
-            LOGGER.error(String.format("Serialization issue while sending transaction %s to peer %s", Base58.encode(signature), peer), e);
+        }
+        catch (TransformationException e) {
+            LOGGER.error(String.format("Serialization issue while sending transaction %s to peer %s", signature58, peer), e);
+        }
+        catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 
@@ -421,44 +498,86 @@ public class TransactionImporter extends Thread {
         }
     }
 
+    // List to collect messages
+    private final List<PeerMessage> signatureMessageList = new ArrayList<>();
+    // Lock to synchronize access to the list
+    private final Object signatureMessageLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService signatureMessageScheduler = Executors.newScheduledThreadPool(1);
+
     public void onNetworkTransactionSignaturesMessage(Peer peer, Message message) {
-        TransactionSignaturesMessage transactionSignaturesMessage = (TransactionSignaturesMessage) message;
-        List<byte[]> signatures = transactionSignaturesMessage.getSignatures();
+        synchronized (signatureMessageLock) {
+            signatureMessageList.add(new PeerMessage(peer, message));
+        }
+    }
 
-        try (final Repository repository = RepositoryManager.getRepository()) {
-            for (byte[] signature : signatures) {
-                String signature58 = Base58.encode(signature);
-                if (invalidUnconfirmedTransactions.containsKey(signature58)) {
-                    // Previously invalid transaction - don't keep requesting it
-                    // It will be periodically removed from invalidUnconfirmedTransactions to allow for rechecks
-                    continue;
-                }
+    public void processNetworkTransactionSignaturesMessage() {
 
-                // Ignore if this transaction is in the queue
-                if (incomingTransactionQueueContains(signature)) {
-                    LOGGER.trace(() -> String.format("Ignoring existing queued transaction %s from peer %s", Base58.encode(signature), peer));
-                    continue;
-                }
+        try {
+            List<PeerMessage> messagesToProcess;
+            synchronized (signatureMessageLock) {
+                messagesToProcess = new ArrayList<>(signatureMessageList);
+                signatureMessageList.clear();
+            }
 
-                // Do we have it already? (Before requesting transaction data itself)
-                if (repository.getTransactionRepository().exists(signature)) {
-                    LOGGER.trace(() -> String.format("Ignoring existing transaction %s from peer %s", Base58.encode(signature), peer));
-                    continue;
-                }
+            Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size() * 10);
+            Map<String, Peer> peerBySignature58 = new HashMap<>( messagesToProcess.size() * 10 );
 
-                // Check isInterrupted() here and exit fast
-                if (Thread.currentThread().isInterrupted())
-                    return;
+            for( PeerMessage peerMessage : messagesToProcess ) {
 
-                // Fetch actual transaction data from peer
-                Message getTransactionMessage = new GetTransactionMessage(signature);
-                if (!peer.sendMessage(getTransactionMessage)) {
-                    peer.disconnect("failed to request transaction");
-                    return;
+                TransactionSignaturesMessage transactionSignaturesMessage = (TransactionSignaturesMessage) peerMessage.getMessage();
+                List<byte[]> signatures = transactionSignaturesMessage.getSignatures();
+
+                for (byte[] signature : signatures) {
+                    String signature58 = Base58.encode(signature);
+                    if (invalidUnconfirmedTransactions.containsKey(signature58)) {
+                        // Previously invalid transaction - don't keep requesting it
+                        // It will be periodically removed from invalidUnconfirmedTransactions to allow for rechecks
+                        continue;
+                    }
+
+                    // Ignore if this transaction is in the queue
+                    if (incomingTransactionQueueContains(signature)) {
+                        LOGGER.trace(() -> String.format("Ignoring existing queued transaction %s from peer %s", Base58.encode(signature), peerMessage.getPeer()));
+                        continue;
+                    }
+
+                    signatureBySignature58.put(signature58, signature);
+                    peerBySignature58.put(signature58, peerMessage.getPeer());
                 }
             }
-        } catch (DataException e) {
-            LOGGER.error(String.format("Repository issue while processing unconfirmed transactions from peer %s", peer), e);
+
+            if( !signatureBySignature58.isEmpty() ) {
+                try (final Repository repository = RepositoryManager.getRepository()) {
+
+                    // remove signatures in db already
+                    repository.getTransactionRepository()
+                            .fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
+                            .map(TransactionData::getSignature)
+                            .map(signature -> Base58.encode(signature))
+                            .forEach(signature58 -> signatureBySignature58.remove(signature58));
+                } catch (DataException e) {
+                    LOGGER.error(String.format("Repository issue while processing unconfirmed transactions from peer"), e);
+                }
+            }
+
+            // Check isInterrupted() here and exit fast
+            if (Thread.currentThread().isInterrupted())
+                return;
+
+            for (Map.Entry<String, byte[]> entry : signatureBySignature58.entrySet()) {
+
+                Peer peer = peerBySignature58.get(entry.getKey());
+
+                // Fetch actual transaction data from peer
+                Message getTransactionMessage = new GetTransactionMessage(entry.getValue());
+                if (peer != null && !peer.sendMessage(getTransactionMessage)) {
+                    peer.disconnect("failed to request transaction");
+                }
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataBuilderThread.java

@@ -14,6 +14,7 @@ import java.io.IOException;
 import java.util.Comparator;
 import java.util.Map;
 
+import static java.lang.Thread.NORM_PRIORITY;
 import static org.qortal.data.arbitrary.ArbitraryResourceStatus.Status.NOT_PUBLISHED;
 
 
@@ -28,6 +29,7 @@ public class ArbitraryDataBuilderThread implements Runnable {
     @Override
     public void run() {
         Thread.currentThread().setName("Arbitrary Data Builder Thread");
+        Thread.currentThread().setPriority(NORM_PRIORITY);
         ArbitraryDataBuildManager buildManager = ArbitraryDataBuildManager.getInstance();
 
         while (!Controller.isStopping()) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCacheManager.java

@@ -2,22 +2,30 @@ package org.qortal.controller.arbitrary;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.qortal.api.resource.TransactionsResource;
 import org.qortal.controller.Controller;
 import org.qortal.data.arbitrary.ArbitraryResourceData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.event.DataMonitorEvent;
+import org.qortal.event.EventBus;
 import org.qortal.gui.SplashFrame;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.transaction.ArbitraryTransaction;
-import org.qortal.transaction.Transaction;
 import org.qortal.utils.Base58;
 
+import java.text.NumberFormat;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 
 public class ArbitraryDataCacheManager extends Thread {
 
@@ -29,6 +37,11 @@ public class ArbitraryDataCacheManager extends Thread {
     /** Queue of arbitrary transactions that require cache updates */
     private final List<ArbitraryTransactionData> updateQueue = Collections.synchronizedList(new ArrayList<>());
 
+    private static final NumberFormat FORMATTER = NumberFormat.getNumberInstance();
+
+    static {
+        FORMATTER.setGroupingUsed(true);
+    }
 
     public static synchronized ArbitraryDataCacheManager getInstance() {
         if (instance == null) {
@@ -41,20 +54,26 @@ public class ArbitraryDataCacheManager extends Thread {
     @Override
     public void run() {
         Thread.currentThread().setName("Arbitrary Data Cache Manager");
+        Thread.currentThread().setPriority(NORM_PRIORITY);
 
         try {
             while (!Controller.isStopping()) {
-                Thread.sleep(500L);
+                try {
+                    Thread.sleep(500L);
 
-                // Process queue
-                processResourceQueue();
+                    // Process queue
+                    processResourceQueue();
+                } catch (Exception e) {
+                    LOGGER.error(e.getMessage(), e);
+                    Thread.sleep(600_000L); // wait 10 minutes to continue
+                }
             }
-        } catch (InterruptedException e) {
-            // Fall through to exit thread
-        }
 
-        // Clear queue before terminating thread
-        processResourceQueue();
+            // Clear queue before terminating thread
+            processResourceQueue();
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
     }
 
     public void shutdown() {
@@ -84,14 +103,25 @@ public class ArbitraryDataCacheManager extends Thread {
                 // Update arbitrary resource caches
                 try {
                     ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
-                    arbitraryTransaction.updateArbitraryResourceCache(repository);
-                    arbitraryTransaction.updateArbitraryMetadataCache(repository);
+                    arbitraryTransaction.updateArbitraryResourceCacheIncludingMetadata(repository, new HashSet<>(0), new HashMap<>(0));
                     repository.saveChanges();
 
                     // Update status as separate commit, as this is more prone to failure
                     arbitraryTransaction.updateArbitraryResourceStatus(repository);
                     repository.saveChanges();
 
+                    EventBus.INSTANCE.notify(
+                        new DataMonitorEvent(
+                            System.currentTimeMillis(),
+                            transactionData.getIdentifier(),
+                            transactionData.getName(),
+                            transactionData.getService().name(),
+                            "updated resource cache and status, queue",
+                            transactionData.getTimestamp(),
+                            transactionData.getTimestamp()
+                        )
+                    );
+
                     LOGGER.debug(() -> String.format("Finished processing transaction %.8s in arbitrary resource queue...", Base58.encode(transactionData.getSignature())));
 
                 } catch (DataException e) {
@@ -102,6 +132,9 @@ public class ArbitraryDataCacheManager extends Thread {
         } catch (DataException e) {
             LOGGER.error("Repository issue while processing arbitrary resource cache updates", e);
         }
+        catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
     }
 
     public void addToUpdateQueue(ArbitraryTransactionData transactionData) {
@@ -147,34 +180,66 @@ public class ArbitraryDataCacheManager extends Thread {
             LOGGER.info("Building arbitrary resources cache...");
             SplashFrame.getInstance().updateStatus("Building QDN cache - please wait...");
 
-            final int batchSize = 100;
+            final int batchSize = Settings.getInstance().getBuildArbitraryResourcesBatchSize();
             int offset = 0;
 
+            List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder
+                    = repository.getArbitraryRepository().getLatestArbitraryTransactions();
+
+            LOGGER.info("arbitrary transactions: count = " + allArbitraryTransactionsInDescendingOrder.size());
+
+            List<ArbitraryResourceData> resources = repository.getArbitraryRepository().getArbitraryResources(null, null, true);
+
+            Map<ArbitraryTransactionDataHashWrapper, ArbitraryResourceData> resourceByWrapper = new HashMap<>(resources.size());
+            for( ArbitraryResourceData resource : resources ) {
+                resourceByWrapper.put(
+                    new ArbitraryTransactionDataHashWrapper(resource.service.value, resource.name, resource.identifier),
+                    resource
+                );
+            }
+
+            LOGGER.info("arbitrary resources: count = " + resourceByWrapper.size());
+
+            Set<ArbitraryTransactionDataHashWrapper> latestTransactionsWrapped = new HashSet<>(allArbitraryTransactionsInDescendingOrder.size());
+
             // Loop through all ARBITRARY transactions, and determine latest state
             while (!Controller.isStopping()) {
-                LOGGER.info("Fetching arbitrary transactions {} - {}", offset, offset+batchSize-1);
+                LOGGER.info(
+                    "Fetching arbitrary transactions {} - {} / {} Total",
+                    FORMATTER.format(offset),
+                    FORMATTER.format(offset+batchSize-1),
+                    FORMATTER.format(allArbitraryTransactionsInDescendingOrder.size())
+                );
 
-                List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, List.of(Transaction.TransactionType.ARBITRARY), null, null, null, TransactionsResource.ConfirmationStatus.BOTH, batchSize, offset, false);
-                if (signatures.isEmpty()) {
+                List<ArbitraryTransactionData> transactionsToProcess
+                    = allArbitraryTransactionsInDescendingOrder.stream()
+                        .skip(offset)
+                        .limit(batchSize)
+                        .collect(Collectors.toList());
+
+                if (transactionsToProcess.isEmpty()) {
                     // Complete
                     break;
                 }
 
-                // Expand signatures to transactions
-                for (byte[] signature : signatures) {
-                    ArbitraryTransactionData transactionData = (ArbitraryTransactionData) repository
-                            .getTransactionRepository().fromSignature(signature);
+                try {
+                    for( ArbitraryTransactionData transactionData : transactionsToProcess) {
+                        if (transactionData.getService() == null) {
+                            // Unsupported service - ignore this resource
+                            continue;
+                        }
 
-                    if (transactionData.getService() == null) {
-                        // Unsupported service - ignore this resource
-                        continue;
+                        latestTransactionsWrapped.add(new ArbitraryTransactionDataHashWrapper(transactionData));
+
+                        // Update arbitrary resource caches
+                        ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
+                        arbitraryTransaction.updateArbitraryResourceCacheIncludingMetadata(repository, latestTransactionsWrapped, resourceByWrapper);
                     }
-
-                    // Update arbitrary resource caches
-                    ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
-                    arbitraryTransaction.updateArbitraryResourceCache(repository);
-                    arbitraryTransaction.updateArbitraryMetadataCache(repository);
                     repository.saveChanges();
+                } catch (DataException e) {
+                    repository.discardChanges();
+
+                    LOGGER.error(e.getMessage(), e);
                 }
                 offset += batchSize;
             }
@@ -192,6 +257,11 @@ public class ArbitraryDataCacheManager extends Thread {
             repository.discardChanges();
             throw new DataException("Build of arbitrary resources cache failed.");
         }
+        catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+
+            return false;
+        }
     }
 
     private boolean refreshArbitraryStatuses(Repository repository) throws DataException {
@@ -199,27 +269,48 @@ public class ArbitraryDataCacheManager extends Thread {
             LOGGER.info("Refreshing arbitrary resource statuses for locally hosted transactions...");
             SplashFrame.getInstance().updateStatus("Refreshing statuses - please wait...");
 
-            final int batchSize = 100;
+            final int batchSize = Settings.getInstance().getBuildArbitraryResourcesBatchSize();
             int offset = 0;
 
+            List<ArbitraryTransactionData> allHostedTransactions
+                = ArbitraryDataStorageManager.getInstance()
+                    .listAllHostedTransactions(repository, null, null);
+
             // Loop through all ARBITRARY transactions, and determine latest state
             while (!Controller.isStopping()) {
-                LOGGER.info("Fetching hosted transactions {} - {}", offset, offset+batchSize-1);
+                LOGGER.info(
+                    "Fetching hosted transactions {} - {} / {} Total",
+                    FORMATTER.format(offset),
+                    FORMATTER.format(offset+batchSize-1),
+                    FORMATTER.format(allHostedTransactions.size())
+                );
+
+                List<ArbitraryTransactionData> hostedTransactions
+                    = allHostedTransactions.stream()
+                        .skip(offset)
+                        .limit(batchSize)
+                        .collect(Collectors.toList());
 
-                List<ArbitraryTransactionData> hostedTransactions = ArbitraryDataStorageManager.getInstance().listAllHostedTransactions(repository, batchSize, offset);
                 if (hostedTransactions.isEmpty()) {
                     // Complete
                     break;
                 }
 
-                // Loop through hosted transactions
-                for (ArbitraryTransactionData transactionData : hostedTransactions) {
+                try {
+                    // Loop through hosted transactions
+                    for (ArbitraryTransactionData transactionData : hostedTransactions) {
 
-                    // Determine status and update cache
-                    ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
-                    arbitraryTransaction.updateArbitraryResourceStatus(repository);
+                        // Determine status and update cache
+                        ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
+                        arbitraryTransaction.updateArbitraryResourceStatus(repository);
+                    }
                     repository.saveChanges();
+                } catch (DataException e) {
+                    repository.discardChanges();
+
+                    LOGGER.error(e.getMessage(), e);
                 }
+
                 offset += batchSize;
             }
 
@@ -233,6 +324,11 @@ public class ArbitraryDataCacheManager extends Thread {
             repository.discardChanges();
             throw new DataException("Refresh of arbitrary resource statuses failed.");
         }
+        catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+
+            return false;
+        }
     }
 
 }

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java

@@ -2,9 +2,10 @@ package org.qortal.controller.arbitrary;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
+import org.qortal.event.DataMonitorEvent;
+import org.qortal.event.EventBus;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
@@ -21,8 +22,12 @@ import java.nio.file.Paths;
 import java.security.SecureRandom;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
 
 import static org.qortal.controller.arbitrary.ArbitraryDataStorageManager.DELETION_THRESHOLD;
 
@@ -71,11 +76,25 @@ public class ArbitraryDataCleanupManager extends Thread {
 	@Override
 	public void run() {
 		Thread.currentThread().setName("Arbitrary Data Cleanup Manager");
+		Thread.currentThread().setPriority(NORM_PRIORITY);
 
 		// Paginate queries when fetching arbitrary transactions
 		final int limit = 100;
 		int offset = 0;
 
+		List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			allArbitraryTransactionsInDescendingOrder
+					= repository.getArbitraryRepository()
+					.getLatestArbitraryTransactions();
+		} catch( Exception e) {
+			LOGGER.error(e.getMessage(), e);
+			allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
+		}
+
+		Set<ArbitraryTransactionData> processedTransactions = new HashSet<>();
+
 		try {
 			while (!isStopping) {
 				Thread.sleep(30000);
@@ -106,27 +125,31 @@ public class ArbitraryDataCleanupManager extends Thread {
 
 				// Any arbitrary transactions we want to fetch data for?
 				try (final Repository repository = RepositoryManager.getRepository()) {
-					List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, null, null, ConfirmationStatus.BOTH, limit, offset, true);
-					// LOGGER.info("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
+					List<ArbitraryTransactionData> transactions = allArbitraryTransactionsInDescendingOrder.stream().skip(offset).limit(limit).collect(Collectors.toList());
 					if (isStopping) {
 						return;
 					}
 
-					if (signatures == null || signatures.isEmpty()) {
+					if (transactions == null || transactions.isEmpty()) {
 						offset = 0;
-						continue;
+						allArbitraryTransactionsInDescendingOrder
+								= repository.getArbitraryRepository()
+								.getLatestArbitraryTransactions();
+						transactions = allArbitraryTransactionsInDescendingOrder.stream().limit(limit).collect(Collectors.toList());
+						processedTransactions.clear();
 					}
+
 					offset += limit;
 					now = NTP.getTime();
 
 					// Loop through the signatures in this batch
-					for (int i=0; i<signatures.size(); i++) {
+					for (int i=0; i<transactions.size(); i++) {
 						if (isStopping) {
 							return;
 						}
 
-						byte[] signature = signatures.get(i);
-						if (signature == null) {
+						ArbitraryTransactionData arbitraryTransactionData = transactions.get(i);
+						if (arbitraryTransactionData == null) {
 							continue;
 						}
 
@@ -135,9 +158,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 							Thread.sleep(5000);
 						}
 
-						// Fetch the transaction data
-						ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-						if (arbitraryTransactionData == null || arbitraryTransactionData.getService() == null) {
+						if (arbitraryTransactionData.getService() == null) {
 							continue;
 						}
 
@@ -146,6 +167,8 @@ public class ArbitraryDataCleanupManager extends Thread {
 							continue;
 						}
 
+						boolean mostRecentTransaction = processedTransactions.add(arbitraryTransactionData);
+
 						// Check if we have the complete file
 						boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData);
 
@@ -166,20 +189,54 @@ public class ArbitraryDataCleanupManager extends Thread {
 							LOGGER.info("Deleting transaction {} because we can't host its data",
 									Base58.encode(arbitraryTransactionData.getSignature()));
 							ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData);
+
+							EventBus.INSTANCE.notify(
+								new DataMonitorEvent(
+									System.currentTimeMillis(),
+									arbitraryTransactionData.getIdentifier(),
+									arbitraryTransactionData.getName(),
+									arbitraryTransactionData.getService().name(),
+									"can't store data, deleting",
+									arbitraryTransactionData.getTimestamp(),
+									arbitraryTransactionData.getTimestamp()
+								)
+							);
 							continue;
 						}
 
 						// Check to see if we have had a more recent PUT
-						boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
-						if (hasMoreRecentPutTransaction) {
+						if (!mostRecentTransaction) {
 							// There is a more recent PUT transaction than the one we are currently processing.
 							// When a PUT is issued, it replaces any layers that would have been there before.
 							// Therefore any data relating to this older transaction is no longer needed.
 							LOGGER.info(String.format("Newer PUT found for %s %s since transaction %s. " +
 											"Deleting all files associated with the earlier transaction.", arbitraryTransactionData.getService(),
-									arbitraryTransactionData.getName(), Base58.encode(signature)));
+									arbitraryTransactionData.getName(), Base58.encode(arbitraryTransactionData.getSignature())));
 
 							ArbitraryTransactionUtils.deleteCompleteFileAndChunks(arbitraryTransactionData);
+
+							Optional<ArbitraryTransactionData> moreRecentPutTransaction
+								= processedTransactions.stream()
+									.filter(data -> data.equals(arbitraryTransactionData))
+									.findAny();
+
+							if( moreRecentPutTransaction.isPresent() ) {
+								EventBus.INSTANCE.notify(
+									new DataMonitorEvent(
+										System.currentTimeMillis(),
+										arbitraryTransactionData.getIdentifier(),
+										arbitraryTransactionData.getName(),
+										arbitraryTransactionData.getService().name(),
+										"deleting data due to replacement",
+										arbitraryTransactionData.getTimestamp(),
+										moreRecentPutTransaction.get().getTimestamp()
+									)
+								);
+							}
+							else {
+								LOGGER.warn("Something went wrong with the most recent put transaction determination!");
+							}
+
 							continue;
 						}
 
@@ -198,7 +255,21 @@ public class ArbitraryDataCleanupManager extends Thread {
 							LOGGER.debug(String.format("Transaction %s has complete file and all chunks",
 									Base58.encode(arbitraryTransactionData.getSignature())));
 
-							ArbitraryTransactionUtils.deleteCompleteFile(arbitraryTransactionData, now, STALE_FILE_TIMEOUT);
+							boolean wasDeleted = ArbitraryTransactionUtils.deleteCompleteFile(arbitraryTransactionData, now, STALE_FILE_TIMEOUT);
+
+							if( wasDeleted ) {
+								EventBus.INSTANCE.notify(
+									new DataMonitorEvent(
+										System.currentTimeMillis(),
+										arbitraryTransactionData.getIdentifier(),
+										arbitraryTransactionData.getName(),
+										arbitraryTransactionData.getService().name(),
+										"deleting file, retaining chunks",
+										arbitraryTransactionData.getTimestamp(),
+										arbitraryTransactionData.getTimestamp()
+									)
+								);
+							}
 							continue;
 						}
 
@@ -236,17 +307,6 @@ public class ArbitraryDataCleanupManager extends Thread {
 						this.storageLimitReached(repository);
 					}
 
-					// Delete random data associated with name if we're over our storage limit for this name
-					// Use the DELETION_THRESHOLD, for the same reasons as above
-					for (String followedName : ListUtils.followedNames()) {
-						if (isStopping) {
-							return;
-						}
-						if (!storageManager.isStorageSpaceAvailableForName(repository, followedName, DELETION_THRESHOLD)) {
-							this.storageLimitReachedForName(repository, followedName);
-						}
-					}
-
 				} catch (DataException e) {
 					LOGGER.error("Repository issue when cleaning up arbitrary transaction data", e);
 				}
@@ -325,25 +385,6 @@ public class ArbitraryDataCleanupManager extends Thread {
 		// FUTURE: consider reducing the expiry time of the reader cache
 	}
 
-	public void storageLimitReachedForName(Repository repository, String name) throws InterruptedException {
-		// We think that the storage limit has been reached for supplied name - but we should double check
-		if (ArbitraryDataStorageManager.getInstance().isStorageSpaceAvailableForName(repository, name, DELETION_THRESHOLD)) {
-			// We have space available for this name, so don't delete anything
-			return;
-		}
-
-		// Delete a batch of random chunks associated with this name
-		// This reduces the chance of too many nodes deleting the same chunk
-		// when they reach their storage limit
-		Path dataPath = Paths.get(Settings.getInstance().getDataPath());
-		for (int i=0; i<CHUNK_DELETION_BATCH_SIZE; i++) {
-			if (isStopping) {
-				return;
-			}
-			this.deleteRandomFile(repository, dataPath.toFile(), name);
-		}
-	}
-
 	/**
 	 * Iteratively walk through given directory and delete a single random file
 	 *
@@ -422,6 +463,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 				}
 
 				LOGGER.info("Deleting random file {} because we have reached max storage capacity...", randomItem.toString());
+				fireRandomItemDeletionNotification(randomItem, repository, "Deleting random file, because we have reached max storage capacity");
 				boolean success = randomItem.delete();
 				if (success) {
 					try {
@@ -436,6 +478,35 @@ public class ArbitraryDataCleanupManager extends Thread {
 		return false;
 	}
 
+	private void fireRandomItemDeletionNotification(File randomItem, Repository repository, String reason) {
+		try {
+			Path parentFileNamePath = randomItem.toPath().toAbsolutePath().getParent().getFileName();
+			if (parentFileNamePath != null) {
+				String signature58 = parentFileNamePath.toString();
+				byte[] signature = Base58.decode(signature58);
+				TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+				if (transactionData != null && transactionData.getType() == Transaction.TransactionType.ARBITRARY) {
+					ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
+
+					EventBus.INSTANCE.notify(
+						new DataMonitorEvent(
+							System.currentTimeMillis(),
+							arbitraryTransactionData.getIdentifier(),
+							arbitraryTransactionData.getName(),
+							arbitraryTransactionData.getService().name(),
+							reason,
+							arbitraryTransactionData.getTimestamp(),
+							arbitraryTransactionData.getTimestamp()
+						)
+					);
+				}
+			}
+
+		} catch (Exception e) {
+			LOGGER.error(e.getMessage(), e);
+		}
+	}
+
 	private void cleanupTempDirectory(String folder, long now, long minAge) {
 		String baseDir = Settings.getInstance().getTempDataPath();
 		Path tempDir = Paths.get(baseDir, folder);

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataExamination.java

@@ -0,0 +1,21 @@
+package org.qortal.controller.arbitrary;
+
+public class ArbitraryDataExamination {
+
+    private boolean pass;
+
+    private String notes;
+
+    public ArbitraryDataExamination(boolean pass, String notes) {
+        this.pass = pass;
+        this.notes = notes;
+    }
+
+    public boolean isPass() {
+        return pass;
+    }
+
+    public String getNotes() {
+        return notes;
+    }
+}

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java

@@ -25,6 +25,10 @@ import org.qortal.utils.NTP;
 import org.qortal.utils.Triple;
 
 import java.util.*;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 
 import static org.qortal.controller.arbitrary.ArbitraryDataFileManager.MAX_FILE_HASH_RESPONSES;
 
@@ -73,6 +77,8 @@ public class ArbitraryDataFileListManager {
 
 
     private ArbitraryDataFileListManager() {
+        getArbitraryDataFileListMessageScheduler.scheduleAtFixedRate(this::processNetworkGetArbitraryDataFileListMessage, 60, 1, TimeUnit.SECONDS);
+        arbitraryDataFileListMessageScheduler.scheduleAtFixedRate(this::processNetworkArbitraryDataFileListMessage, 60, 1, TimeUnit.SECONDS);
     }
 
     public static ArbitraryDataFileListManager getInstance() {
@@ -118,8 +124,8 @@ public class ArbitraryDataFileListManager {
         if (timeSinceLastAttempt > 15 * 1000L) {
             // We haven't tried for at least 15 seconds
 
-            if (networkBroadcastCount < 3) {
-                // We've made less than 3 total attempts
+            if (networkBroadcastCount < 12) {
+                // We've made less than 12 total attempts
                 return true;
             }
         }
@@ -128,8 +134,8 @@ public class ArbitraryDataFileListManager {
         if (timeSinceLastAttempt > 60 * 1000L) {
             // We haven't tried for at least 1 minute
 
-            if (networkBroadcastCount < 8) {
-                // We've made less than 8 total attempts
+            if (networkBroadcastCount < 40) {
+                // We've made less than 40 total attempts
                 return true;
             }
         }
@@ -396,11 +402,11 @@ public class ArbitraryDataFileListManager {
         return true;
     }
 
-    public void deleteFileListRequestsForSignature(byte[] signature) {
-        String signature58 = Base58.encode(signature);
+    public void deleteFileListRequestsForSignature(String signature58) {
+
         for (Iterator<Map.Entry<Integer, Triple<String, Peer, Long>>> it = arbitraryDataFileListRequests.entrySet().iterator(); it.hasNext();) {
             Map.Entry<Integer, Triple<String, Peer, Long>> entry = it.next();
-            if (entry == null || entry.getKey() == null || entry.getValue() != null) {
+            if (entry == null || entry.getKey() == null || entry.getValue() == null) {
                 continue;
             }
             if (Objects.equals(entry.getValue().getA(), signature58)) {
@@ -413,70 +419,116 @@ public class ArbitraryDataFileListManager {
 
     // Network handlers
 
+    // List to collect messages
+    private final List<PeerMessage> arbitraryDataFileListMessageList = new ArrayList<>();
+    // Lock to synchronize access to the list
+    private final Object arbitraryDataFileListMessageLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService arbitraryDataFileListMessageScheduler = Executors.newScheduledThreadPool(1);
+
     public void onNetworkArbitraryDataFileListMessage(Peer peer, Message message) {
         // Don't process if QDN is disabled
         if (!Settings.getInstance().isQdnEnabled()) {
             return;
         }
 
-        ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
-        LOGGER.debug("Received hash list from peer {} with {} hashes", peer, arbitraryDataFileListMessage.getHashes().size());
-
-        if (LOGGER.isDebugEnabled() && arbitraryDataFileListMessage.getRequestTime() != null) {
-            long totalRequestTime = NTP.getTime() - arbitraryDataFileListMessage.getRequestTime();
-            LOGGER.debug("totalRequestTime: {}, requestHops: {}, peerAddress: {}, isRelayPossible: {}",
-                    totalRequestTime, arbitraryDataFileListMessage.getRequestHops(),
-                    arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+        synchronized (arbitraryDataFileListMessageLock) {
+            arbitraryDataFileListMessageList.add(new PeerMessage(peer, message));
         }
+    }
 
-        // Do we have a pending request for this data?
-        Triple<String, Peer, Long> request = arbitraryDataFileListRequests.get(message.getId());
-        if (request == null || request.getA() == null) {
-            return;
-        }
-        boolean isRelayRequest = (request.getB() != null);
+    private void processNetworkArbitraryDataFileListMessage() {
 
-        // Does this message's signature match what we're expecting?
-        byte[] signature = arbitraryDataFileListMessage.getSignature();
-        String signature58 = Base58.encode(signature);
-        if (!request.getA().equals(signature58)) {
-            return;
-        }
+        try {
+            List<PeerMessage> messagesToProcess;
+            synchronized (arbitraryDataFileListMessageLock) {
+                messagesToProcess = new ArrayList<>(arbitraryDataFileListMessageList);
+                arbitraryDataFileListMessageList.clear();
+            }
 
-        List<byte[]> hashes = arbitraryDataFileListMessage.getHashes();
-        if (hashes == null || hashes.isEmpty()) {
-            return;
-        }
+            if (messagesToProcess.isEmpty()) return;
 
-        ArbitraryTransactionData arbitraryTransactionData = null;
+            Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, Boolean> isRelayRequestBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, List<byte[]>> hashesBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, Triple<String, Peer, Long>> requestBySignature58 = new HashMap<>(messagesToProcess.size());
 
-        // Check transaction exists and hashes are correct
-        try (final Repository repository = RepositoryManager.getRepository()) {
-            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
-            if (!(transactionData instanceof ArbitraryTransactionData))
+            for (PeerMessage peerMessage : messagesToProcess) {
+                Peer peer = peerMessage.getPeer();
+                Message message = peerMessage.getMessage();
+
+                ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
+                LOGGER.debug("Received hash list from peer {} with {} hashes", peer, arbitraryDataFileListMessage.getHashes().size());
+
+                if (LOGGER.isDebugEnabled() && arbitraryDataFileListMessage.getRequestTime() != null) {
+                    long totalRequestTime = NTP.getTime() - arbitraryDataFileListMessage.getRequestTime();
+                    LOGGER.debug("totalRequestTime: {}, requestHops: {}, peerAddress: {}, isRelayPossible: {}",
+                            totalRequestTime, arbitraryDataFileListMessage.getRequestHops(),
+                            arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+                }
+
+                // Do we have a pending request for this data?
+                Triple<String, Peer, Long> request = arbitraryDataFileListRequests.get(message.getId());
+                if (request == null || request.getA() == null) {
+                    continue;
+                }
+                boolean isRelayRequest = (request.getB() != null);
+
+                // Does this message's signature match what we're expecting?
+                byte[] signature = arbitraryDataFileListMessage.getSignature();
+                String signature58 = Base58.encode(signature);
+                if (!request.getA().equals(signature58)) {
+                    continue;
+                }
+
+                List<byte[]> hashes = arbitraryDataFileListMessage.getHashes();
+                if (hashes == null || hashes.isEmpty()) {
+                    continue;
+                }
+
+                peerMessageBySignature58.put(signature58, peerMessage);
+                signatureBySignature58.put(signature58, signature);
+                isRelayRequestBySignature58.put(signature58, isRelayRequest);
+                hashesBySignature58.put(signature58, hashes);
+                requestBySignature58.put(signature58, request);
+            }
+
+            if (signatureBySignature58.isEmpty()) return;
+
+            List<ArbitraryTransactionData> arbitraryTransactionDataList;
+
+            // Check transaction exists and hashes are correct
+            try (final Repository repository = RepositoryManager.getRepository()) {
+                arbitraryTransactionDataList
+                        = repository.getTransactionRepository()
+                        .fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
+                        .filter(data -> data instanceof ArbitraryTransactionData)
+                        .map(data -> (ArbitraryTransactionData) data)
+                        .collect(Collectors.toList());
+            } catch (DataException e) {
+                LOGGER.error(String.format("Repository issue while finding arbitrary transaction data list"), e);
                 return;
+            }
 
-            arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
+            for (ArbitraryTransactionData arbitraryTransactionData : arbitraryTransactionDataList) {
 
-//          // Load data file(s)
-//          ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
-//
-//			// Check all hashes exist
-//			for (byte[] hash : hashes) {
-//				//LOGGER.debug("Received hash {}", Base58.encode(hash));
-//				if (!arbitraryDataFile.containsChunk(hash)) {
-//					// Check the hash against the complete file
-//					if (!Arrays.equals(arbitraryDataFile.getHash(), hash)) {
-//						LOGGER.info("Received non-matching chunk hash {} for signature {}. This could happen if we haven't obtained the metadata file yet.", Base58.encode(hash), signature58);
-//						return;
-//					}
-//				}
-//			}
+                byte[] signature = arbitraryTransactionData.getSignature();
+                String signature58 = Base58.encode(signature);
 
-            if (!isRelayRequest || !Settings.getInstance().isRelayModeEnabled()) {
-                Long now = NTP.getTime();
+                List<byte[]> hashes = hashesBySignature58.get(signature58);
+
+                PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
+                Peer peer = peerMessage.getPeer();
+                Message message = peerMessage.getMessage();
+
+                ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
+
+                Boolean isRelayRequest = isRelayRequestBySignature58.get(signature58);
+                if (!isRelayRequest || !Settings.getInstance().isRelayModeEnabled()) {
+                    Long now = NTP.getTime();
 
-                if (ArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.size() < MAX_FILE_HASH_RESPONSES) {
                     // Keep track of the hashes this peer reports to have access to
                     for (byte[] hash : hashes) {
                         String hash58 = Base58.encode(hash);
@@ -487,233 +539,300 @@ public class ArbitraryDataFileListManager {
                         ArbitraryFileListResponseInfo responseInfo = new ArbitraryFileListResponseInfo(hash58, signature58,
                                 peer, now, arbitraryDataFileListMessage.getRequestTime(), requestHops);
 
-                        ArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.add(responseInfo);
+                        ArbitraryDataFileManager.getInstance().addResponse(responseInfo);
+                    }
+
+                    // Keep track of the source peer, for direct connections
+                    if (arbitraryDataFileListMessage.getPeerAddress() != null) {
+                        ArbitraryDataFileManager.getInstance().addDirectConnectionInfoIfUnique(
+                                new ArbitraryDirectConnectionInfo(signature, arbitraryDataFileListMessage.getPeerAddress(), hashes, now));
                     }
                 }
 
-                // Keep track of the source peer, for direct connections
-                if (arbitraryDataFileListMessage.getPeerAddress() != null) {
-                    ArbitraryDataFileManager.getInstance().addDirectConnectionInfoIfUnique(
-                            new ArbitraryDirectConnectionInfo(signature, arbitraryDataFileListMessage.getPeerAddress(), hashes, now));
-                }
-            }
+                // Forwarding
+                if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
 
-        } catch (DataException e) {
-            LOGGER.error(String.format("Repository issue while finding arbitrary transaction data list for peer %s", peer), e);
-        }
+                    boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
+                    if (!isBlocked) {
+                        Triple<String, Peer, Long> request = requestBySignature58.get(signature58);
+                        Peer requestingPeer = request.getB();
+                        if (requestingPeer != null) {
+                            Long requestTime = arbitraryDataFileListMessage.getRequestTime();
+                            Integer requestHops = arbitraryDataFileListMessage.getRequestHops();
 
-        // Forwarding
-        if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
-            boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
-            if (!isBlocked) {
-                Peer requestingPeer = request.getB();
-                if (requestingPeer != null) {
-                    Long requestTime = arbitraryDataFileListMessage.getRequestTime();
-                    Integer requestHops = arbitraryDataFileListMessage.getRequestHops();
+                            // Add each hash to our local mapping so we know who to ask later
+                            Long now = NTP.getTime();
+                            for (byte[] hash : hashes) {
+                                String hash58 = Base58.encode(hash);
+                                ArbitraryRelayInfo relayInfo = new ArbitraryRelayInfo(hash58, signature58, peer, now, requestTime, requestHops);
+                                ArbitraryDataFileManager.getInstance().addToRelayMap(relayInfo);
+                            }
 
-                    // Add each hash to our local mapping so we know who to ask later
-                    Long now = NTP.getTime();
-                    for (byte[] hash : hashes) {
-                        String hash58 = Base58.encode(hash);
-                        ArbitraryRelayInfo relayInfo = new ArbitraryRelayInfo(hash58, signature58, peer, now, requestTime, requestHops);
-                        ArbitraryDataFileManager.getInstance().addToRelayMap(relayInfo);
-                    }
+                            // Bump requestHops if it exists
+                            if (requestHops != null) {
+                                requestHops++;
+                            }
 
-                    // Bump requestHops if it exists
-                    if (requestHops != null) {
-                        requestHops++;
-                    }
+                            ArbitraryDataFileListMessage forwardArbitraryDataFileListMessage;
 
-                    ArbitraryDataFileListMessage forwardArbitraryDataFileListMessage;
+                            // Remove optional parameters if the requesting peer doesn't support it yet
+                            // A message with less statistical data is better than no message at all
+                            if (!requestingPeer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
+                                forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
+                            } else {
+                                forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
+                                        arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+                            }
+                            forwardArbitraryDataFileListMessage.setId(message.getId());
 
-                    // Remove optional parameters if the requesting peer doesn't support it yet
-                    // A message with less statistical data is better than no message at all
-                    if (!requestingPeer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
-                        forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
-                    } else {
-                        forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
-                                arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
-                    }
-                    forwardArbitraryDataFileListMessage.setId(message.getId());
-
-                    // Forward to requesting peer
-                    LOGGER.debug("Forwarding file list with {} hashes to requesting peer: {}", hashes.size(), requestingPeer);
-                    if (!requestingPeer.sendMessage(forwardArbitraryDataFileListMessage)) {
-                        requestingPeer.disconnect("failed to forward arbitrary data file list");
+                            // Forward to requesting peer
+                            LOGGER.debug("Forwarding file list with {} hashes to requesting peer: {}", hashes.size(), requestingPeer);
+                            requestingPeer.sendMessage(forwardArbitraryDataFileListMessage);
+                        }
                     }
                 }
             }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 
+    // List to collect messages
+    private final List<PeerMessage> getArbitraryDataFileListMessageList = new ArrayList<>();
+    // Lock to synchronize access to the list
+    private final Object getArbitraryDataFileListMessageLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService getArbitraryDataFileListMessageScheduler = Executors.newScheduledThreadPool(1);
+
     public void onNetworkGetArbitraryDataFileListMessage(Peer peer, Message message) {
         // Don't respond if QDN is disabled
         if (!Settings.getInstance().isQdnEnabled()) {
             return;
         }
 
-        Controller.getInstance().stats.getArbitraryDataFileListMessageStats.requests.incrementAndGet();
-
-        GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
-        byte[] signature = getArbitraryDataFileListMessage.getSignature();
-        String signature58 = Base58.encode(signature);
-        Long now = NTP.getTime();
-        Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
-
-        // If we've seen this request recently, then ignore
-        if (arbitraryDataFileListRequests.putIfAbsent(message.getId(), newEntry) != null) {
-            LOGGER.trace("Ignoring hash list request from peer {} for signature {}", peer, signature58);
-            return;
+        synchronized (getArbitraryDataFileListMessageLock) {
+            getArbitraryDataFileListMessageList.add(new PeerMessage(peer, message));
         }
+    }
 
-        List<byte[]> requestedHashes = getArbitraryDataFileListMessage.getHashes();
-        int hashCount = requestedHashes != null ? requestedHashes.size() : 0;
-        String requestingPeer = getArbitraryDataFileListMessage.getRequestingPeer();
+    private void processNetworkGetArbitraryDataFileListMessage() {
 
-        if (requestingPeer != null) {
-            LOGGER.debug("Received hash list request with {} hashes from peer {} (requesting peer {}) for signature {}", hashCount, peer, requestingPeer, signature58);
-        }
-        else {
-            LOGGER.debug("Received hash list request with {} hashes from peer {} for signature {}", hashCount, peer, signature58);
-        }
+        try {
+            List<PeerMessage> messagesToProcess;
+            synchronized (getArbitraryDataFileListMessageLock) {
+                messagesToProcess = new ArrayList<>(getArbitraryDataFileListMessageList);
+                getArbitraryDataFileListMessageList.clear();
+            }
 
-        List<byte[]> hashes = new ArrayList<>();
-        ArbitraryTransactionData transactionData = null;
-        boolean allChunksExist = false;
-        boolean hasMetadata = false;
+            if (messagesToProcess.isEmpty()) return;
 
-        try (final Repository repository = RepositoryManager.getRepository()) {
+            Map<String, byte[]> signatureBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, List<byte[]>> requestedHashesBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, String> requestingPeerBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String, Long> nowBySignature58 = new HashMap<>((messagesToProcess.size()));
+            Map<String, PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
 
-            // Firstly we need to lookup this file on chain to get a list of its hashes
-            transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
-            if (transactionData instanceof ArbitraryTransactionData) {
+            for (PeerMessage messagePeer : messagesToProcess) {
+                Controller.getInstance().stats.getArbitraryDataFileListMessageStats.requests.incrementAndGet();
+
+                Message message = messagePeer.message;
+                Peer peer = messagePeer.peer;
+
+                GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
+                byte[] signature = getArbitraryDataFileListMessage.getSignature();
+                String signature58 = Base58.encode(signature);
+                Long now = NTP.getTime();
+                Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
+
+                // If we've seen this request recently, then ignore
+                if (arbitraryDataFileListRequests.putIfAbsent(message.getId(), newEntry) != null) {
+                    LOGGER.trace("Ignoring hash list request from peer {} for signature {}", peer, signature58);
+                    continue;
+                }
+
+                List<byte[]> requestedHashes = getArbitraryDataFileListMessage.getHashes();
+                int hashCount = requestedHashes != null ? requestedHashes.size() : 0;
+                String requestingPeer = getArbitraryDataFileListMessage.getRequestingPeer();
+
+                if (requestingPeer != null) {
+                    LOGGER.debug("Received hash list request with {} hashes from peer {} (requesting peer {}) for signature {}", hashCount, peer, requestingPeer, signature58);
+                } else {
+                    LOGGER.debug("Received hash list request with {} hashes from peer {} for signature {}", hashCount, peer, signature58);
+                }
+
+                signatureBySignature58.put(signature58, signature);
+                requestedHashesBySignature58.put(signature58, requestedHashes);
+                requestingPeerBySignature58.put(signature58, requestingPeer);
+                nowBySignature58.put(signature58, now);
+                peerMessageBySignature58.put(signature58, messagePeer);
+            }
+
+            if (signatureBySignature58.isEmpty()) {
+                return;
+            }
+
+            List<byte[]> hashes = new ArrayList<>();
+            boolean allChunksExist = false;
+            boolean hasMetadata = false;
+
+            List<ArbitraryTransactionData> transactionDataList;
+            try (final Repository repository = RepositoryManager.getRepository()) {
+
+                // Firstly we need to lookup this file on chain to get a list of its hashes
+                transactionDataList
+                        = repository.getTransactionRepository()
+                        .fromSignatures(new ArrayList<>(signatureBySignature58.values())).stream()
+                        .filter(data -> data instanceof ArbitraryTransactionData)
+                        .map(data -> (ArbitraryTransactionData) data)
+                        .collect(Collectors.toList());
+
+            } catch (DataException e) {
+                LOGGER.error(String.format("Repository issue while fetching arbitrary file list for peer"), e);
+                return;
+            }
+
+            for (ArbitraryTransactionData transactionData : transactionDataList) {
+                byte[] signature = transactionData.getSignature();
+                String signature58 = Base58.encode(signature);
+                List<byte[]> requestedHashes = requestedHashesBySignature58.get(signature58);
 
                 // Check if we're even allowed to serve data for this transaction
                 if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
 
-                    // Load file(s) and add any that exist to the list of hashes
-                    ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
+                    try {
+                        // Load file(s) and add any that exist to the list of hashes
+                        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
-                    // If the peer didn't supply a hash list, we need to return all hashes for this transaction
-                    if (requestedHashes == null || requestedHashes.isEmpty()) {
-                        requestedHashes = new ArrayList<>();
+                        // If the peer didn't supply a hash list, we need to return all hashes for this transaction
+                        if (requestedHashes == null || requestedHashes.isEmpty()) {
+                            requestedHashes = new ArrayList<>();
 
-                        // Add the metadata file
-                        if (arbitraryDataFile.getMetadataHash() != null) {
-                            requestedHashes.add(arbitraryDataFile.getMetadataHash());
-                            hasMetadata = true;
+                            // Add the metadata file
+                            if (arbitraryDataFile.getMetadataHash() != null) {
+                                requestedHashes.add(arbitraryDataFile.getMetadataHash());
+                                hasMetadata = true;
+                            }
+
+                            // Add the chunk hashes
+                            if (!arbitraryDataFile.getChunkHashes().isEmpty()) {
+                                requestedHashes.addAll(arbitraryDataFile.getChunkHashes());
+                            }
+                            // Add complete file if there are no hashes
+                            else {
+                                requestedHashes.add(arbitraryDataFile.getHash());
+                            }
                         }
 
-                        // Add the chunk hashes
-                        if (!arbitraryDataFile.getChunkHashes().isEmpty()) {
-                            requestedHashes.addAll(arbitraryDataFile.getChunkHashes());
-                        }
-                        // Add complete file if there are no hashes
-                        else {
-                            requestedHashes.add(arbitraryDataFile.getHash());
+
+                        // Assume all chunks exists, unless one can't be found below
+                        allChunksExist = true;
+
+                        for (byte[] requestedHash : requestedHashes) {
+                            ArbitraryDataFileChunk chunk = ArbitraryDataFileChunk.fromHash(requestedHash, signature);
+                            if (chunk.exists()) {
+                                hashes.add(chunk.getHash());
+                                //LOGGER.trace("Added hash {}", chunk.getHash58());
+                            } else {
+                                LOGGER.trace("Couldn't add hash {} because it doesn't exist", chunk.getHash58());
+                                allChunksExist = false;
+                            }
                         }
+                    } catch (DataException e) {
+                        LOGGER.error(e.getMessage(), e);
+                    }
+                }
+
+                // If the only file we have is the metadata then we shouldn't respond. Most nodes will already have that,
+                // or can use the separate metadata protocol to fetch it. This should greatly reduce network spam.
+                if (hasMetadata && hashes.size() == 1) {
+                    hashes.clear();
+                }
+
+                PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
+                Peer peer = peerMessage.getPeer();
+                Message message = peerMessage.getMessage();
+
+                Long now = nowBySignature58.get(signature58);
+
+                // We should only respond if we have at least one hash
+                String requestingPeer = requestingPeerBySignature58.get(signature58);
+                if (!hashes.isEmpty()) {
+
+                    // Firstly we should keep track of the requesting peer, to allow for potential direct connections later
+                    ArbitraryDataFileManager.getInstance().addRecentDataRequest(requestingPeer);
+
+                    // We have all the chunks, so update requests map to reflect that we've sent it
+                    // There is no need to keep track of the request, as we can serve all the chunks
+                    if (allChunksExist) {
+                        Triple<String, Peer, Long> newEntry = new Triple<>(null, null, now);
+                        arbitraryDataFileListRequests.put(message.getId(), newEntry);
                     }
 
-                    // Assume all chunks exists, unless one can't be found below
-                    allChunksExist = true;
+                    String ourAddress = Network.getInstance().getOurExternalIpAddressAndPort();
+                    ArbitraryDataFileListMessage arbitraryDataFileListMessage;
+
+                    // Remove optional parameters if the requesting peer doesn't support it yet
+                    // A message with less statistical data is better than no message at all
+                    if (!peer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
+                        arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
+                    } else {
+                        arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature,
+                                hashes, NTP.getTime(), 0, ourAddress, true);
+                    }
+
+                    arbitraryDataFileListMessage.setId(message.getId());
+
+                    if (!peer.sendMessage(arbitraryDataFileListMessage)) {
+                        LOGGER.debug("Couldn't send list of hashes");
+                        continue;
+                    }
+
+                    if (allChunksExist) {
+                        // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
+                        LOGGER.debug("No need for any forwarding because file list request is fully served");
+                        continue;
+                    }
+
+                }
+
+                // We may need to forward this request on
+                boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
+                if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
+                    // In relay mode - so ask our other peers if they have it
+
+
+                    GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
+
+                    long requestTime = getArbitraryDataFileListMessage.getRequestTime();
+                    int requestHops = getArbitraryDataFileListMessage.getRequestHops() + 1;
+                    long totalRequestTime = now - requestTime;
+
+                    if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
+                        // Relay request hasn't timed out yet, so can potentially be rebroadcast
+                        if (requestHops < RELAY_REQUEST_MAX_HOPS) {
+                            // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
+
+                            Message relayGetArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops, requestingPeer);
+                            relayGetArbitraryDataFileListMessage.setId(message.getId());
+
+                            LOGGER.debug("Rebroadcasting hash list request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
+                            Network.getInstance().broadcast(
+                                    broadcastPeer ->
+                                            !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
+                                                    broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryDataFileListMessage
+                            );
 
-                    for (byte[] requestedHash : requestedHashes) {
-                        ArbitraryDataFileChunk chunk = ArbitraryDataFileChunk.fromHash(requestedHash, signature);
-                        if (chunk.exists()) {
-                            hashes.add(chunk.getHash());
-                            //LOGGER.trace("Added hash {}", chunk.getHash58());
                         } else {
-                            LOGGER.trace("Couldn't add hash {} because it doesn't exist", chunk.getHash58());
-                            allChunksExist = false;
+                            // This relay request has reached the maximum number of allowed hops
                         }
+                    } else {
+                        // This relay request has timed out
                     }
                 }
             }
-
-        } catch (DataException e) {
-            LOGGER.error(String.format("Repository issue while fetching arbitrary file list for peer %s", peer), e);
-        }
-
-        // If the only file we have is the metadata then we shouldn't respond. Most nodes will already have that,
-        // or can use the separate metadata protocol to fetch it. This should greatly reduce network spam.
-        if (hasMetadata && hashes.size() == 1) {
-            hashes.clear();
-        }
-
-        // We should only respond if we have at least one hash
-        if (!hashes.isEmpty()) {
-
-            // Firstly we should keep track of the requesting peer, to allow for potential direct connections later
-            ArbitraryDataFileManager.getInstance().addRecentDataRequest(requestingPeer);
-
-            // We have all the chunks, so update requests map to reflect that we've sent it
-            // There is no need to keep track of the request, as we can serve all the chunks
-            if (allChunksExist) {
-                newEntry = new Triple<>(null, null, now);
-                arbitraryDataFileListRequests.put(message.getId(), newEntry);
-            }
-
-            String ourAddress = Network.getInstance().getOurExternalIpAddressAndPort();
-            ArbitraryDataFileListMessage arbitraryDataFileListMessage;
-
-            // Remove optional parameters if the requesting peer doesn't support it yet
-            // A message with less statistical data is better than no message at all
-            if (!peer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
-                arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
-            } else {
-                arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature,
-                        hashes, NTP.getTime(), 0, ourAddress, true);
-            }
-
-            arbitraryDataFileListMessage.setId(message.getId());
-
-            if (!peer.sendMessage(arbitraryDataFileListMessage)) {
-                LOGGER.debug("Couldn't send list of hashes");
-                peer.disconnect("failed to send list of hashes");
-                return;
-            }
-            LOGGER.debug("Sent list of hashes (count: {})", hashes.size());
-
-            if (allChunksExist) {
-                // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
-                LOGGER.debug("No need for any forwarding because file list request is fully served");
-                return;
-            }
-
-        }
-
-        // We may need to forward this request on
-        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
-        if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
-            // In relay mode - so ask our other peers if they have it
-
-            long requestTime = getArbitraryDataFileListMessage.getRequestTime();
-            int requestHops = getArbitraryDataFileListMessage.getRequestHops() + 1;
-            long totalRequestTime = now - requestTime;
-
-            if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
-                // Relay request hasn't timed out yet, so can potentially be rebroadcast
-                if (requestHops < RELAY_REQUEST_MAX_HOPS) {
-                    // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
-
-                    Message relayGetArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops, requestingPeer);
-                    relayGetArbitraryDataFileListMessage.setId(message.getId());
-
-                    LOGGER.debug("Rebroadcasting hash list request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
-                    Network.getInstance().broadcast(
-                            broadcastPeer ->
-                                    !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
-                                    broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryDataFileListMessage
-                    );
-
-                }
-                else {
-                    // This relay request has reached the maximum number of allowed hops
-                }
-            }
-            else {
-                // This relay request has timed out
-            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java

@@ -1,6 +1,7 @@
 package org.qortal.controller.arbitrary;
 
 import com.google.common.net.InetAddresses;
+
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.qortal.arbitrary.ArbitraryDataFile;
@@ -12,6 +13,7 @@ import org.qortal.data.network.PeerData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.network.Network;
 import org.qortal.network.Peer;
+import org.qortal.network.PeerSendManagement;
 import org.qortal.network.message.*;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
@@ -23,12 +25,16 @@ import org.qortal.utils.NTP;
 
 import java.security.SecureRandom;
 import java.util.*;
-import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 
 public class ArbitraryDataFileManager extends Thread {
 
+    public static final int SEND_TIMEOUT_MS = 500;
     private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileManager.class);
 
     private static ArbitraryDataFileManager instance;
@@ -48,7 +54,7 @@ public class ArbitraryDataFileManager extends Thread {
     /**
      * List to keep track of any arbitrary data file hash responses
      */
-    public final List<ArbitraryFileListResponseInfo> arbitraryDataFileHashResponses = Collections.synchronizedList(new ArrayList<>());
+    private final List<ArbitraryFileListResponseInfo> arbitraryDataFileHashResponses = Collections.synchronizedList(new ArrayList<>());
 
     /**
      * List to keep track of peers potentially available for direct connections, based on recent requests
@@ -65,8 +71,9 @@ public class ArbitraryDataFileManager extends Thread {
 
     public static int MAX_FILE_HASH_RESPONSES = 1000;
 
-
     private ArbitraryDataFileManager() {
+        this.arbitraryDataFileHashResponseScheduler.scheduleAtFixedRate( this::processResponses, 60, 1, TimeUnit.SECONDS);
+        this.arbitraryDataFileHashResponseScheduler.scheduleAtFixedRate(this::handleFileListRequestProcess, 60, 1, TimeUnit.SECONDS);
     }
 
     public static ArbitraryDataFileManager getInstance() {
@@ -76,18 +83,13 @@ public class ArbitraryDataFileManager extends Thread {
         return instance;
     }
 
+
+
     @Override
     public void run() {
         Thread.currentThread().setName("Arbitrary Data File Manager");
 
         try {
-            // Use a fixed thread pool to execute the arbitrary data file requests
-            int threadCount = 5;
-            ExecutorService arbitraryDataFileRequestExecutor = Executors.newFixedThreadPool(threadCount);
-            for (int i = 0; i < threadCount; i++) {
-                arbitraryDataFileRequestExecutor.execute(new ArbitraryDataFileRequestThread());
-            }
-
             while (!isStopping) {
                 // Nothing to do yet
                 Thread.sleep(1000);
@@ -112,7 +114,6 @@ public class ArbitraryDataFileManager extends Thread {
 
         final long relayMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_RELAY_TIMEOUT;
         arbitraryRelayMap.removeIf(entry -> entry == null || entry.getTimestamp() == null || entry.getTimestamp() < relayMinimumTimestamp);
-        arbitraryDataFileHashResponses.removeIf(entry -> entry.getTimestamp() < relayMinimumTimestamp);
 
         final long directConnectionInfoMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT;
         directConnectionInfo.removeIf(entry -> entry.getTimestamp() < directConnectionInfoMinimumTimestamp);
@@ -125,8 +126,7 @@ public class ArbitraryDataFileManager extends Thread {
 
     // Fetch data files by hash
 
-    public boolean fetchArbitraryDataFiles(Repository repository,
-                                           Peer peer,
+    public boolean fetchArbitraryDataFiles(Peer peer,
                                            byte[] signature,
                                            ArbitraryTransactionData arbitraryTransactionData,
                                            List<byte[]> hashes) throws DataException {
@@ -146,21 +146,15 @@ public class ArbitraryDataFileManager extends Thread {
                 if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) {
                     LOGGER.debug("Requesting data file {} from peer {}", hash58, peer);
                     Long startTime = NTP.getTime();
-                    ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, null, arbitraryTransactionData, signature, hash, null);
+                    ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, arbitraryTransactionData, signature, hash);
                     Long endTime = NTP.getTime();
                     if (receivedArbitraryDataFile != null) {
                         LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime));
                         receivedAtLeastOneFile = true;
-
-                        // Remove this hash from arbitraryDataFileHashResponses now that we have received it
-                        arbitraryDataFileHashResponses.remove(hash58);
                     }
                     else {
                         LOGGER.debug("Peer {} didn't respond with data file {} for signature {}. Time taken: {} ms", peer, Base58.encode(hash), Base58.encode(signature), (endTime-startTime));
 
-                        // Remove this hash from arbitraryDataFileHashResponses now that we have failed to receive it
-                        arbitraryDataFileHashResponses.remove(hash58);
-
                         // Stop asking for files from this peer
                         break;
                     }
@@ -169,10 +163,6 @@ public class ArbitraryDataFileManager extends Thread {
                     LOGGER.trace("Already requesting data file {} for signature {} from peer {}", arbitraryDataFile, Base58.encode(signature), peer);
                 }
             }
-            else {
-                // Remove this hash from arbitraryDataFileHashResponses because we have a local copy
-                arbitraryDataFileHashResponses.remove(hash58);
-            }
         }
 
         if (receivedAtLeastOneFile) {
@@ -191,14 +181,103 @@ public class ArbitraryDataFileManager extends Thread {
         return receivedAtLeastOneFile;
     }
 
-    private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, Peer requestingPeer, ArbitraryTransactionData arbitraryTransactionData, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
-        ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
-        boolean fileAlreadyExists = existingFile.exists();
-        String hash58 = Base58.encode(hash);
+    // Lock to synchronize access to the list
+    private final Object arbitraryDataFileHashResponseLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService arbitraryDataFileHashResponseScheduler = Executors.newScheduledThreadPool(1);
+
+
+    public void addResponse( ArbitraryFileListResponseInfo responseInfo ) {
+
+        synchronized (arbitraryDataFileHashResponseLock) {
+            this.arbitraryDataFileHashResponses.add(responseInfo);
+        }
+    }
+
+    private void processResponses() {
+        try {
+            List<ArbitraryFileListResponseInfo> responsesToProcess;
+            synchronized (arbitraryDataFileHashResponseLock) {
+                responsesToProcess = new ArrayList<>(arbitraryDataFileHashResponses);
+                arbitraryDataFileHashResponses.clear();
+            }
+
+            if (responsesToProcess.isEmpty()) return;
+
+            Long now = NTP.getTime();
+
+            ArbitraryDataFileRequestThread.getInstance().processFileHashes(now, responsesToProcess, this);
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, ArbitraryTransactionData arbitraryTransactionData, byte[] signature, byte[] hash) throws DataException {
         ArbitraryDataFile arbitraryDataFile;
 
-        // Fetch the file if it doesn't exist locally
-        if (!fileAlreadyExists) {
+        try {
+            ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
+            boolean fileAlreadyExists = existingFile.exists();
+            String hash58 = Base58.encode(hash);
+
+            // Fetch the file if it doesn't exist locally
+            if (!fileAlreadyExists) {
+                LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
+                arbitraryDataFileRequests.put(hash58, NTP.getTime());
+                Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
+
+                Message response = null;
+                try {
+                    response = peer.getResponseWithTimeout(getArbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
+                } catch (InterruptedException e) {
+                    // Will return below due to null response
+                }
+                arbitraryDataFileRequests.remove(hash58);
+                LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
+
+                if (response == null) {
+                    LOGGER.debug("Received null response from peer {}", peer);
+                    return null;
+                }
+                if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
+                    LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
+                    return null;
+                }
+
+                ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
+                arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
+            } else {
+                LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
+                arbitraryDataFile = existingFile;
+            }
+
+            if (arbitraryDataFile != null) {
+
+                arbitraryDataFile.save();
+
+                // If this is a metadata file then we need to update the cache
+                if (arbitraryTransactionData != null && arbitraryTransactionData.getMetadataHash() != null) {
+                    if (Arrays.equals(arbitraryTransactionData.getMetadataHash(), hash)) {
+                        ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
+                    }
+                }
+
+                // We may need to remove the file list request, if we have all the files for this transaction
+                this.handleFileListRequests(signature);
+            }
+        } catch (DataException e) {
+            LOGGER.error(e.getMessage(), e);
+            arbitraryDataFile = null;
+        }
+
+        return arbitraryDataFile;
+    }
+
+    private void fetchFileForRelay(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
+        try {
+            String hash58 = Base58.encode(hash);
+
             LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
             arbitraryDataFileRequests.put(hash58, NTP.getTime());
             Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
@@ -212,73 +291,73 @@ public class ArbitraryDataFileManager extends Thread {
             arbitraryDataFileRequests.remove(hash58);
             LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
 
-            // We may need to remove the file list request, if we have all the files for this transaction
-            this.handleFileListRequests(signature);
-
             if (response == null) {
                 LOGGER.debug("Received null response from peer {}", peer);
-                return null;
+                return;
             }
             if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
                 LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
-                return null;
-            }
-
-            ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
-            arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
-        } else {
-            LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
-            arbitraryDataFile = existingFile;
-        }
-
-        if (arbitraryDataFile == null) {
-            // We don't have a file, so give up here
-            return null;
-        }
-
-        // We might want to forward the request to the peer that originally requested it
-        this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
-
-        boolean isRelayRequest = (requestingPeer != null);
-        if (isRelayRequest) {
-            if (!fileAlreadyExists) {
-                // File didn't exist locally before the request, and it's a forwarding request, so delete it if it exists.
-                // It shouldn't exist on the filesystem yet, but leaving this here just in case.
-                arbitraryDataFile.delete(10);
-            }
-        }
-        else {
-            arbitraryDataFile.save();
-        }
-
-        // If this is a metadata file then we need to update the cache
-        if (arbitraryTransactionData != null && arbitraryTransactionData.getMetadataHash() != null) {
-            if (Arrays.equals(arbitraryTransactionData.getMetadataHash(), hash)) {
-                ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
-            }
-        }
-
-        return arbitraryDataFile;
-    }
-
-    private void handleFileListRequests(byte[] signature) {
-        try (final Repository repository = RepositoryManager.getRepository()) {
-
-            // Fetch the transaction data
-            ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-            if (arbitraryTransactionData == null) {
                 return;
             }
 
-            boolean allChunksExist = ArbitraryTransactionUtils.allChunksExist(arbitraryTransactionData);
+            ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
+            ArbitraryDataFile arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
 
-            if (allChunksExist) {
-                // Update requests map to reflect that we've received all chunks
-                ArbitraryDataFileListManager.getInstance().deleteFileListRequestsForSignature(signature);
+            if (arbitraryDataFile != null) {
+
+                // We might want to forward the request to the peer that originally requested it
+                this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    Map<String, byte[]> signatureBySignature58 = new HashMap<>();
+
+    // Lock to synchronize access to the list
+    private final Object handleFileListRequestsLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService handleFileListRequestsScheduler = Executors.newScheduledThreadPool(1);
+
+    private void handleFileListRequests(byte[] signature) {
+
+        synchronized (handleFileListRequestsLock) {
+            signatureBySignature58.put(Base58.encode(signature), signature);
+        }
+    }
+
+    private void handleFileListRequestProcess() {
+
+        Map<String, byte[]> signaturesToProcess;
+
+        synchronized (handleFileListRequestsLock) {
+            signaturesToProcess = new HashMap<>(signatureBySignature58);
+            signatureBySignature58.clear();
+        }
+
+        if( signaturesToProcess.isEmpty() ) return;
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+
+            // Fetch the transaction data
+            List<ArbitraryTransactionData> arbitraryTransactionDataList
+                = ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signaturesToProcess.values()));
+
+            for( ArbitraryTransactionData arbitraryTransactionData : arbitraryTransactionDataList ) {
+                boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData);
+
+                if (completeFileExists) {
+                    String signature58 = Base58.encode(arbitraryTransactionData.getSignature());
+                    LOGGER.debug("All chunks or complete file exist for transaction {}", signature58);
+
+                    ArbitraryDataFileListManager.getInstance().deleteFileListRequestsForSignature(signature58);
+                }
             }
 
-        } catch (DataException e) {
-            LOGGER.debug("Unable to handle file list requests: {}", e.getMessage());
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 
@@ -295,15 +374,14 @@ public class ArbitraryDataFileManager extends Thread {
 
         LOGGER.debug("Received arbitrary data file - forwarding is needed");
 
-        // The ID needs to match that of the original request
-        message.setId(originalMessage.getId());
+        try {
+            // The ID needs to match that of the original request
+            message.setId(originalMessage.getId());
 
-        if (!requestingPeer.sendMessageWithTimeout(message, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
-            LOGGER.debug("Failed to forward arbitrary data file to peer {}", requestingPeer);
-            requestingPeer.disconnect("failed to forward arbitrary data file");
-        }
-        else {
-            LOGGER.debug("Forwarded arbitrary data file to peer {}", requestingPeer);
+            PeerSendManagement.getInstance().getOrCreateSendManager(requestingPeer).queueMessage(message, SEND_TIMEOUT_MS);
+
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 
@@ -577,13 +655,9 @@ public class ArbitraryDataFileManager extends Thread {
                 LOGGER.debug("Sending file {}...", arbitraryDataFile);
                 ArbitraryDataFileMessage arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, arbitraryDataFile);
                 arbitraryDataFileMessage.setId(message.getId());
-                if (!peer.sendMessageWithTimeout(arbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
-                    LOGGER.debug("Couldn't send file {}", arbitraryDataFile);
-                    peer.disconnect("failed to send file");
-                }
-                else {
-                    LOGGER.debug("Sent file {}", arbitraryDataFile);
-                }
+
+                PeerSendManagement.getInstance().getOrCreateSendManager(peer).queueMessage(arbitraryDataFileMessage, SEND_TIMEOUT_MS);
+
             }
             else if (relayInfo != null) {
                 LOGGER.debug("We have relay info for hash {}", Base58.encode(hash));
@@ -595,7 +669,7 @@ public class ArbitraryDataFileManager extends Thread {
                     LOGGER.debug("Asking peer {} for hash {}", peerToAsk, hash58);
                     // No need to pass arbitraryTransactionData below because this is only used for metadata caching,
                     // and metadata isn't retained when relaying.
-                    this.fetchArbitraryDataFile(peerToAsk, peer, null, signature, hash, message);
+                    this.fetchFileForRelay(peerToAsk, peer, signature, hash, message);
                 }
                 else {
                     LOGGER.debug("Peer {} not found in relay info", peer);
@@ -617,7 +691,6 @@ public class ArbitraryDataFileManager extends Thread {
                 fileUnknownMessage.setId(message.getId());
                 if (!peer.sendMessage(fileUnknownMessage)) {
                     LOGGER.debug("Couldn't sent file-unknown response");
-                    peer.disconnect("failed to send file-unknown response");
                 }
                 else {
                     LOGGER.debug("Sent file-unknown response for file {}", arbitraryDataFile);

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java

@@ -4,122 +4,186 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.qortal.controller.Controller;
 import org.qortal.data.arbitrary.ArbitraryFileListResponseInfo;
+import org.qortal.data.arbitrary.ArbitraryResourceData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.network.Peer;
+import org.qortal.network.message.MessageType;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
 import org.qortal.utils.ArbitraryTransactionUtils;
 import org.qortal.utils.Base58;
 import org.qortal.utils.NTP;
+import org.qortal.utils.NamedThreadFactory;
 
+import java.net.http.HttpResponse;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Comparator;
+import java.util.HashMap;
 import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
 
-public class ArbitraryDataFileRequestThread implements Runnable {
+import static java.lang.Thread.NORM_PRIORITY;
+
+public class ArbitraryDataFileRequestThread {
 
     private static final Logger LOGGER = LogManager.getLogger(ArbitraryDataFileRequestThread.class);
 
-    public ArbitraryDataFileRequestThread() {
+    private static final Integer FETCHER_LIMIT_PER_PEER = Settings.getInstance().getMaxThreadsForMessageType(MessageType.GET_ARBITRARY_DATA_FILE);
+    private static final String FETCHER_THREAD_PREFIX = "Arbitrary Data Fetcher ";
 
+    private ConcurrentHashMap<String, ExecutorService> executorByPeer = new ConcurrentHashMap<>();
+
+    private ArbitraryDataFileRequestThread() {
+        cleanupExecutorByPeerScheduler.scheduleAtFixedRate(this::cleanupExecutorsByPeer, 1, 1, TimeUnit.MINUTES);
     }
 
-    @Override
-    public void run() {
-        Thread.currentThread().setName("Arbitrary Data File Request Thread");
+    private static ArbitraryDataFileRequestThread instance = null;
+
+    public static ArbitraryDataFileRequestThread getInstance() {
+
+        if( instance == null ) {
+            instance = new ArbitraryDataFileRequestThread();
+        }
+
+        return instance;
+    }
+
+    private final ScheduledExecutorService cleanupExecutorByPeerScheduler = Executors.newScheduledThreadPool(1);
+
+    private void cleanupExecutorsByPeer() {
 
         try {
-            while (!Controller.isStopping()) {
-                Long now = NTP.getTime();
-                this.processFileHashes(now);
-            }
-        } catch (InterruptedException e) {
-            // Fall-through to exit thread...
+            this.executorByPeer.forEach((key, value) -> {
+                if (value instanceof ThreadPoolExecutor) {
+                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) value;
+                    if (threadPoolExecutor.getActiveCount() == 0) {
+                        threadPoolExecutor.shutdown();
+                        if (this.executorByPeer.computeIfPresent(key, (k, v) -> null) == null) {
+                            LOGGER.trace("removed executor: peer = " + key);
+                        }
+                    }
+                } else {
+                    LOGGER.warn("casting issue in cleanup");
+                }
+            });
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
         }
     }
 
-    private void processFileHashes(Long now) throws InterruptedException {
+    public void processFileHashes(Long now, List<ArbitraryFileListResponseInfo> responseInfos, ArbitraryDataFileManager arbitraryDataFileManager) {
 		if (Controller.isStopping()) {
             return;
         }
 
-        ArbitraryDataFileManager arbitraryDataFileManager = ArbitraryDataFileManager.getInstance();
-        String signature58 = null;
-        String hash58 = null;
-        Peer peer = null;
-        boolean shouldProcess = false;
+        Map<String, byte[]> signatureBySignature58 = new HashMap<>(responseInfos.size());
+        Map<String, List<ArbitraryFileListResponseInfo>> responseInfoBySignature58 = new HashMap<>();
 
-        synchronized (arbitraryDataFileManager.arbitraryDataFileHashResponses) {
-            if (!arbitraryDataFileManager.arbitraryDataFileHashResponses.isEmpty()) {
+        for( ArbitraryFileListResponseInfo responseInfo : responseInfos) {
 
-                // Sort by lowest number of node hops first
-                Comparator<ArbitraryFileListResponseInfo> lowestHopsFirstComparator =
-                        Comparator.comparingInt(ArbitraryFileListResponseInfo::getRequestHops);
-                arbitraryDataFileManager.arbitraryDataFileHashResponses.sort(lowestHopsFirstComparator);
+            if( responseInfo == null ) continue;
 
-                Iterator iterator = arbitraryDataFileManager.arbitraryDataFileHashResponses.iterator();
-                while (iterator.hasNext()) {
-                    if (Controller.isStopping()) {
-                        return;
-                    }
-
-                    ArbitraryFileListResponseInfo responseInfo = (ArbitraryFileListResponseInfo) iterator.next();
-                    if (responseInfo == null) {
-                        iterator.remove();
-                        continue;
-                    }
-
-                    hash58 = responseInfo.getHash58();
-                    peer = responseInfo.getPeer();
-                    signature58 = responseInfo.getSignature58();
-                    Long timestamp = responseInfo.getTimestamp();
-
-                    if (now - timestamp >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT || signature58 == null || peer == null) {
-                        // Ignore - to be deleted
-                        iterator.remove();
-                        continue;
-                    }
-
-                    // Skip if already requesting, but don't remove, as we might want to retry later
-                    if (arbitraryDataFileManager.arbitraryDataFileRequests.containsKey(hash58)) {
-                        // Already requesting - leave this attempt for later
-                        continue;
-                    }
-
-                    // We want to process this file
-                    shouldProcess = true;
-                    iterator.remove();
-                    break;
-                }
+            if (Controller.isStopping()) {
+                return;
             }
+
+            Peer peer = responseInfo.getPeer();
+
+            // if relay timeout, then move on
+            if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT || responseInfo.getSignature58() == null || peer == null) {
+                continue;
+            }
+
+            // Skip if already requesting, but don't remove, as we might want to retry later
+            if (arbitraryDataFileManager.arbitraryDataFileRequests.containsKey(responseInfo.getHash58())) {
+                // Already requesting - leave this attempt for later
+                arbitraryDataFileManager.addResponse(responseInfo); // don't remove -> adding back, beacause it was removed already above
+                continue;
+            }
+
+
+            byte[] hash = Base58.decode(responseInfo.getHash58());
+            byte[] signature = Base58.decode(responseInfo.getSignature58());
+
+            // check for null
+            if (signature == null || hash == null || peer == null) {
+                continue;
+            }
+
+            // We want to process this file, store and map data to process later
+            signatureBySignature58.put(responseInfo.getSignature58(), signature);
+            responseInfoBySignature58
+                    .computeIfAbsent(responseInfo.getSignature58(), signature58 -> new ArrayList<>())
+                    .add(responseInfo);
         }
 
-        if (!shouldProcess) {
-            // Nothing to do
-            Thread.sleep(1000L);
-            return;
-        }
+        // if there are no signatures, then there is nothing to process and nothing query the database
+        if( signatureBySignature58.isEmpty() ) return;
 
-        byte[] hash = Base58.decode(hash58);
-        byte[] signature = Base58.decode(signature58);
+        List<ArbitraryTransactionData> arbitraryTransactionDataList = new ArrayList<>();
 
         // Fetch the transaction data
         try (final Repository repository = RepositoryManager.getRepository()) {
-            ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-            if (arbitraryTransactionData == null) {
-                return;
-            }
-
-            if (signature == null || hash == null || peer == null || arbitraryTransactionData == null) {
-                return;
-            }
-
-            LOGGER.trace("Fetching file {} from peer {} via request thread...", hash58, peer);
-            arbitraryDataFileManager.fetchArbitraryDataFiles(repository, peer, signature, arbitraryTransactionData, Arrays.asList(hash));
-
+            arbitraryTransactionDataList.addAll(
+                ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signatureBySignature58.values())));
         } catch (DataException e) {
-            LOGGER.debug("Unable to process file hashes: {}", e.getMessage());
+            LOGGER.warn("Unable to fetch transaction data: {}", e.getMessage());
+        }
+
+        if( !arbitraryTransactionDataList.isEmpty() ) {
+            long start = System.currentTimeMillis();
+
+            for(ArbitraryTransactionData data : arbitraryTransactionDataList ) {
+                String signature58 = Base58.encode(data.getSignature());
+                for( ArbitraryFileListResponseInfo responseInfo : responseInfoBySignature58.get(signature58)) {
+                    Runnable fetcher = () -> arbitraryDataFileFetcher(arbitraryDataFileManager, responseInfo, data);
+                    this.executorByPeer
+                            .computeIfAbsent(
+                                responseInfo.getPeer().toString(),
+                                peer -> Executors.newFixedThreadPool(
+                                    FETCHER_LIMIT_PER_PEER,
+                                    new NamedThreadFactory(FETCHER_THREAD_PREFIX + responseInfo.getPeer().toString(), NORM_PRIORITY)
+                                )
+                            )
+                            .execute(fetcher);
+                }
+            }
+            long timeLapse = System.currentTimeMillis() - start;
         }
     }
-}
+
+    private void arbitraryDataFileFetcher(ArbitraryDataFileManager arbitraryDataFileManager, ArbitraryFileListResponseInfo responseInfo, ArbitraryTransactionData arbitraryTransactionData)  {
+        try {
+            Long now = NTP.getTime();
+
+            if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT ) {
+
+                Peer peer = responseInfo.getPeer();
+                String hash58 = responseInfo.getHash58();
+                String signature58 = responseInfo.getSignature58();
+                LOGGER.debug("Peer {} version {} didn't fetch data file {} for signature {} due to relay timeout.", peer, peer.getPeersVersionString(), hash58, signature58);
+                return;
+            }
+
+            arbitraryDataFileManager.fetchArbitraryDataFiles(
+                responseInfo.getPeer(),
+                arbitraryTransactionData.getSignature(),
+                arbitraryTransactionData,
+                Arrays.asList(Base58.decode(responseInfo.getHash58()))
+            );
+        } catch (DataException e) {
+            LOGGER.warn("Unable to process file hashes: {}", e.getMessage());
+        }
+    }
+}

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java

@@ -10,6 +10,8 @@ import org.qortal.arbitrary.misc.Service;
 import org.qortal.controller.Controller;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
+import org.qortal.event.DataMonitorEvent;
+import org.qortal.event.EventBus;
 import org.qortal.network.Network;
 import org.qortal.network.Peer;
 import org.qortal.repository.DataException;
@@ -28,6 +30,7 @@ import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.*;
+import java.util.stream.Collectors;
 
 public class ArbitraryDataManager extends Thread {
 
@@ -39,10 +42,10 @@ public class ArbitraryDataManager extends Thread {
 	private int powDifficulty = 14; // Must not be final, as unit tests need to reduce this value
 
 	/** Request timeout when transferring arbitrary data */
-	public static final long ARBITRARY_REQUEST_TIMEOUT = 12 * 1000L; // ms
+	public static final long ARBITRARY_REQUEST_TIMEOUT = 24 * 1000L; // ms
 
 	/** Maximum time to hold information about an in-progress relay */
-	public static final long ARBITRARY_RELAY_TIMEOUT = 60 * 1000L; // ms
+	public static final long ARBITRARY_RELAY_TIMEOUT = 120 * 1000L; // ms
 
 	/** Maximum time to hold direct peer connection information */
 	public static final long ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT = 2 * 60 * 1000L; // ms
@@ -91,6 +94,7 @@ public class ArbitraryDataManager extends Thread {
 	@Override
 	public void run() {
 		Thread.currentThread().setName("Arbitrary Data Manager");
+		Thread.currentThread().setPriority(NORM_PRIORITY);
 
 		// Create data directory in case it doesn't exist yet
 		this.createDataDirectory();
@@ -194,13 +198,35 @@ public class ArbitraryDataManager extends Thread {
 		final int limit = 100;
 		int offset = 0;
 
+		List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			if( name == null ) {
+				allArbitraryTransactionsInDescendingOrder
+						= repository.getArbitraryRepository()
+						.getLatestArbitraryTransactions();
+			}
+			else {
+				allArbitraryTransactionsInDescendingOrder
+						= repository.getArbitraryRepository()
+						.getLatestArbitraryTransactionsByName(name);
+			}
+		} catch( Exception e) {
+			LOGGER.error(e.getMessage(), e);
+			allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
+		}
+
+		// collect processed transactions in a set to ensure outdated data transactions do not get fetched
+		Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
+
 		while (!isStopping) {
 			Thread.sleep(1000L);
 
 			// Any arbitrary transactions we want to fetch data for?
 			try (final Repository repository = RepositoryManager.getRepository()) {
-				List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, name, null, ConfirmationStatus.BOTH, limit, offset, true);
-				// LOGGER.trace("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
+				List<byte[]> signatures = processTransactionsForSignatures(limit, offset, allArbitraryTransactionsInDescendingOrder, processedTransactions);
+
 				if (signatures == null || signatures.isEmpty()) {
 					offset = 0;
 					break;
@@ -222,14 +248,38 @@ public class ArbitraryDataManager extends Thread {
 					ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) arbitraryTransaction.getTransactionData();
 
 					// Skip transactions that we don't need to proactively store data for
-					if (!storageManager.shouldPreFetchData(repository, arbitraryTransactionData)) {
+					ArbitraryDataExamination arbitraryDataExamination = storageManager.shouldPreFetchData(repository, arbitraryTransactionData);
+					if (!arbitraryDataExamination.isPass()) {
 						iterator.remove();
+
+						EventBus.INSTANCE.notify(
+							new DataMonitorEvent(
+								System.currentTimeMillis(),
+								arbitraryTransactionData.getIdentifier(),
+								arbitraryTransactionData.getName(),
+								arbitraryTransactionData.getService().name(),
+								arbitraryDataExamination.getNotes(),
+								arbitraryTransactionData.getTimestamp(),
+								arbitraryTransactionData.getTimestamp()
+							)
+						);
 						continue;
 					}
 
 					// Remove transactions that we already have local data for
 					if (hasLocalData(arbitraryTransaction)) {
 						iterator.remove();
+						EventBus.INSTANCE.notify(
+							new DataMonitorEvent(
+								System.currentTimeMillis(),
+								arbitraryTransactionData.getIdentifier(),
+								arbitraryTransactionData.getName(),
+								arbitraryTransactionData.getService().name(),
+								"already have local data, skipping",
+								arbitraryTransactionData.getTimestamp(),
+								arbitraryTransactionData.getTimestamp()
+							)
+						);
                     }
 				}
 
@@ -247,8 +297,21 @@ public class ArbitraryDataManager extends Thread {
 
 				// Check to see if we have had a more recent PUT
 				ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-				boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
-				if (hasMoreRecentPutTransaction) {
+
+				Optional<ArbitraryTransactionData> moreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
+
+				if (moreRecentPutTransaction.isPresent()) {
+					EventBus.INSTANCE.notify(
+						new DataMonitorEvent(
+							System.currentTimeMillis(),
+							arbitraryTransactionData.getIdentifier(),
+							arbitraryTransactionData.getName(),
+							arbitraryTransactionData.getService().name(),
+							"not fetching old data",
+							arbitraryTransactionData.getTimestamp(),
+							moreRecentPutTransaction.get().getTimestamp()
+						)
+					);
 					// There is a more recent PUT transaction than the one we are currently processing.
 					// When a PUT is issued, it replaces any layers that would have been there before.
 					// Therefore any data relating to this older transaction is no longer needed and we
@@ -256,10 +319,34 @@ public class ArbitraryDataManager extends Thread {
 					continue;
 				}
 
+				EventBus.INSTANCE.notify(
+					new DataMonitorEvent(
+						System.currentTimeMillis(),
+						arbitraryTransactionData.getIdentifier(),
+						arbitraryTransactionData.getName(),
+						arbitraryTransactionData.getService().name(),
+						"fetching data",
+						arbitraryTransactionData.getTimestamp(),
+						arbitraryTransactionData.getTimestamp()
+					)
+				);
+
 				// Ask our connected peers if they have files for this signature
 				// This process automatically then fetches the files themselves if a peer is found
 				fetchData(arbitraryTransactionData);
 
+				EventBus.INSTANCE.notify(
+					new DataMonitorEvent(
+						System.currentTimeMillis(),
+						arbitraryTransactionData.getIdentifier(),
+						arbitraryTransactionData.getName(),
+						arbitraryTransactionData.getService().name(),
+						"fetched data",
+						arbitraryTransactionData.getTimestamp(),
+						arbitraryTransactionData.getTimestamp()
+					)
+				);
+
 			} catch (DataException e) {
 				LOGGER.error("Repository issue when fetching arbitrary transaction data", e);
 			}
@@ -273,6 +360,20 @@ public class ArbitraryDataManager extends Thread {
 		final int limit = 100;
 		int offset = 0;
 
+		List<ArbitraryTransactionData> allArbitraryTransactionsInDescendingOrder;
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			allArbitraryTransactionsInDescendingOrder
+					= repository.getArbitraryRepository()
+						.getLatestArbitraryTransactions();
+		} catch( Exception e) {
+			LOGGER.error(e.getMessage(), e);
+			allArbitraryTransactionsInDescendingOrder = new ArrayList<>(0);
+		}
+
+		// collect processed transactions in a set to ensure outdated data transactions do not get fetched
+		Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
+
 		while (!isStopping) {
 			final int minSeconds = 3;
 			final int maxSeconds = 10;
@@ -281,8 +382,8 @@ public class ArbitraryDataManager extends Thread {
 
 			// Any arbitrary transactions we want to fetch data for?
 			try (final Repository repository = RepositoryManager.getRepository()) {
-				List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, ARBITRARY_TX_TYPE, null, null, null, ConfirmationStatus.BOTH, limit, offset, true);
-				// LOGGER.trace("Found {} arbitrary transactions at offset: {}, limit: {}", signatures.size(), offset, limit);
+				List<byte[]> signatures = processTransactionsForSignatures(limit, offset, allArbitraryTransactionsInDescendingOrder, processedTransactions);
+
 				if (signatures == null || signatures.isEmpty()) {
 					offset = 0;
 					break;
@@ -327,26 +428,74 @@ public class ArbitraryDataManager extends Thread {
 					continue;
 				}
 
-				// Check to see if we have had a more recent PUT
+				// No longer need to see if we have had a more recent PUT since we compared the transactions to process
+				// to the transactions previously processed, so we can fetch the transactiondata, notify the event bus,
+				// fetch the metadata and notify the event bus again
 				ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-				boolean hasMoreRecentPutTransaction = ArbitraryTransactionUtils.hasMoreRecentPutTransaction(repository, arbitraryTransactionData);
-				if (hasMoreRecentPutTransaction) {
-					// There is a more recent PUT transaction than the one we are currently processing.
-					// When a PUT is issued, it replaces any layers that would have been there before.
-					// Therefore any data relating to this older transaction is no longer needed and we
-					// shouldn't fetch it from the network.
-					continue;
-				}
 
 				// Ask our connected peers if they have metadata for this signature
 				fetchMetadata(arbitraryTransactionData);
 
+				EventBus.INSTANCE.notify(
+					new DataMonitorEvent(
+						System.currentTimeMillis(),
+						arbitraryTransactionData.getIdentifier(),
+						arbitraryTransactionData.getName(),
+						arbitraryTransactionData.getService().name(),
+						"fetched metadata",
+						arbitraryTransactionData.getTimestamp(),
+						arbitraryTransactionData.getTimestamp()
+					)
+				);
 			} catch (DataException e) {
 				LOGGER.error("Repository issue when fetching arbitrary transaction data", e);
+			} catch (Exception e) {
+				LOGGER.error(e.getMessage(), e);
 			}
 		}
 	}
 
+	private static List<byte[]> processTransactionsForSignatures(
+			int limit,
+			int offset,
+			List<ArbitraryTransactionData> transactionsInDescendingOrder,
+			Set<ArbitraryTransactionDataHashWrapper> processedTransactions) {
+		// these transactions are in descending order, latest transactions come first
+		List<ArbitraryTransactionData> transactions
+				= transactionsInDescendingOrder.stream()
+					.skip(offset)
+					.limit(limit)
+					.collect(Collectors.toList());
+
+		// wrap the transactions, so they can be used for hashing and comparing
+		// Class ArbitraryTransactionDataHashWrapper supports hashCode() and equals(...) for this purpose
+		List<ArbitraryTransactionDataHashWrapper> wrappedTransactions
+				= transactions.stream()
+					.map(transaction -> new ArbitraryTransactionDataHashWrapper(transaction))
+					.collect(Collectors.toList());
+
+		// create a set of wrappers and populate it first to last, so that all outdated transactions get rejected
+		Set<ArbitraryTransactionDataHashWrapper> transactionsToProcess = new HashSet<>(wrappedTransactions.size());
+		for(ArbitraryTransactionDataHashWrapper wrappedTransaction : wrappedTransactions) {
+			transactionsToProcess.add(wrappedTransaction);
+		}
+
+		// remove the matches for previously processed transactions,
+		// because these transactions have had updates that have already been processed
+		transactionsToProcess.removeAll(processedTransactions);
+
+		// add to processed transactions to compare and remove matches from future processing iterations
+		processedTransactions.addAll(transactionsToProcess);
+
+		List<byte[]> signatures
+				= transactionsToProcess.stream()
+					.map(transactionToProcess -> transactionToProcess.getData()
+					.getSignature())
+					.collect(Collectors.toList());
+
+		return signatures;
+	}
+
 	private ArbitraryTransaction fetchTransaction(final Repository repository, byte[] signature) {
 		try {
 			TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataRenderManager.java

@@ -36,6 +36,7 @@ public class ArbitraryDataRenderManager extends Thread {
     @Override
     public void run() {
         Thread.currentThread().setName("Arbitrary Data Render Manager");
+        Thread.currentThread().setPriority(NORM_PRIORITY);
 
         try {
             while (!isStopping) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java

@@ -47,15 +47,15 @@ public class ArbitraryDataStorageManager extends Thread {
 
     private static final long DIRECTORY_SIZE_CHECK_INTERVAL = 10 * 60 * 1000L; // 10 minutes
 
-    /** Treat storage as full at 90% usage, to reduce risk of going over the limit.
+    /** Treat storage as full at 80% usage, to reduce risk of going over the limit.
      * This is necessary because we don't calculate total storage values before every write.
      * It also helps avoid a fetch/delete loop, as we will stop fetching before the hard limit.
      * This must be lower than DELETION_THRESHOLD. */
-    private static final double STORAGE_FULL_THRESHOLD = 0.90f; // 90%
+    private static final double STORAGE_FULL_THRESHOLD = 0.8f; // 80%
 
-    /** Start deleting files once we reach 98% usage.
+    /** Start deleting files once we reach 90% usage.
      * This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */
-    public static final double DELETION_THRESHOLD = 0.98f; // 98%
+    public static final double DELETION_THRESHOLD = 0.9f; // 90%
 
     private static final long PER_NAME_STORAGE_MULTIPLIER = 4L;
 
@@ -72,6 +72,8 @@ public class ArbitraryDataStorageManager extends Thread {
     @Override
     public void run() {
         Thread.currentThread().setName("Arbitrary Data Storage Manager");
+        Thread.currentThread().setPriority(NORM_PRIORITY);
+
         try {
             while (!isStopping) {
                 Thread.sleep(1000);
@@ -153,31 +155,24 @@ public class ArbitraryDataStorageManager extends Thread {
      * @param arbitraryTransactionData - the transaction
      * @return boolean - whether to prefetch or not
      */
-    public boolean shouldPreFetchData(Repository repository, ArbitraryTransactionData arbitraryTransactionData) {
+    public ArbitraryDataExamination shouldPreFetchData(Repository repository, ArbitraryTransactionData arbitraryTransactionData) {
         String name = arbitraryTransactionData.getName();
 
         // Only fetch data associated with hashes, as we already have RAW_DATA
         if (arbitraryTransactionData.getDataType() != ArbitraryTransactionData.DataType.DATA_HASH) {
-            return false;
+            return new ArbitraryDataExamination(false, "Only fetch data associated with hashes");
         }
 
         // Don't fetch anything more if we're (nearly) out of space
         // Make sure to keep STORAGE_FULL_THRESHOLD considerably less than 1, to
         // avoid a fetch/delete loop
         if (!this.isStorageSpaceAvailable(STORAGE_FULL_THRESHOLD)) {
-            return false;
-        }
-
-        // Don't fetch anything if we're (nearly) out of space for this name
-        // Again, make sure to keep STORAGE_FULL_THRESHOLD considerably less than 1, to
-        // avoid a fetch/delete loop
-        if (!this.isStorageSpaceAvailableForName(repository, arbitraryTransactionData.getName(), STORAGE_FULL_THRESHOLD)) {
-            return false;
+            return new ArbitraryDataExamination(false,"Don't fetch anything more if we're (nearly) out of space");
         }
 
         // Don't store data unless it's an allowed type (public/private)
         if (!this.isDataTypeAllowed(arbitraryTransactionData)) {
-            return false;
+            return new ArbitraryDataExamination(false, "Don't store data unless it's an allowed type (public/private)");
         }
 
         // Handle transactions without names differently
@@ -187,21 +182,21 @@ public class ArbitraryDataStorageManager extends Thread {
 
         // Never fetch data from blocked names, even if they are followed
         if (ListUtils.isNameBlocked(name)) {
-            return false;
+            return new ArbitraryDataExamination(false, "blocked name");
         }
 
         switch (Settings.getInstance().getStoragePolicy()) {
             case FOLLOWED:
             case FOLLOWED_OR_VIEWED:
-                return ListUtils.isFollowingName(name);
+                return new ArbitraryDataExamination(ListUtils.isFollowingName(name), Settings.getInstance().getStoragePolicy().name());
                 
             case ALL:
-                return true;
+                return new ArbitraryDataExamination(true, Settings.getInstance().getStoragePolicy().name());
 
             case NONE:
             case VIEWED:
             default:
-                return false;
+                return new ArbitraryDataExamination(false, Settings.getInstance().getStoragePolicy().name());
         }
     }
 
@@ -212,17 +207,17 @@ public class ArbitraryDataStorageManager extends Thread {
      *
      * @return boolean - whether the storage policy allows for unnamed data
      */
-    private boolean shouldPreFetchDataWithoutName() {
+    private ArbitraryDataExamination shouldPreFetchDataWithoutName() {
         switch (Settings.getInstance().getStoragePolicy()) {
             case ALL:
-                return true;
+                return new ArbitraryDataExamination(true, "Fetching all data");
 
             case NONE:
             case VIEWED:
             case FOLLOWED:
             case FOLLOWED_OR_VIEWED:
             default:
-                return false;
+                return new ArbitraryDataExamination(false, Settings.getInstance().getStoragePolicy().name());
         }
     }
 
@@ -482,51 +477,6 @@ public class ArbitraryDataStorageManager extends Thread {
         return true;
     }
 
-    public boolean isStorageSpaceAvailableForName(Repository repository, String name, double threshold) {
-        if (!this.isStorageSpaceAvailable(threshold)) {
-            // No storage space available at all, so no need to check this name
-            return false;
-        }
-
-        if (Settings.getInstance().getStoragePolicy() == StoragePolicy.ALL) {
-            // Using storage policy ALL, so don't limit anything per name
-            return true;
-        }
-
-        if (name == null) {
-            // This transaction doesn't have a name, so fall back to total space limitations
-            return true;
-        }
-
-        int followedNamesCount = ListUtils.followedNamesCount();
-        if (followedNamesCount == 0) {
-            // Not following any names, so we have space
-            return true;
-        }
-
-        long totalSizeForName = 0;
-        long maxStoragePerName = this.storageCapacityPerName(threshold);
-
-        // Fetch all hosted transactions
-        List<ArbitraryTransactionData> hostedTransactions = this.listAllHostedTransactions(repository, null, null);
-        for (ArbitraryTransactionData transactionData : hostedTransactions) {
-            String transactionName = transactionData.getName();
-            if (!Objects.equals(name, transactionName)) {
-                // Transaction relates to a different name
-                continue;
-            }
-
-            totalSizeForName += transactionData.getSize();
-        }
-
-        // Have we reached the limit for this name?
-        if (totalSizeForName > maxStoragePerName) {
-            return false;
-        }
-
-        return true;
-    }
-
     public long storageCapacityPerName(double threshold) {
         int followedNamesCount = ListUtils.followedNamesCount();
         if (followedNamesCount == 0) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java

@@ -24,6 +24,11 @@ import org.qortal.utils.Triple;
 
 import java.io.IOException;
 import java.util.*;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 
 import static org.qortal.controller.arbitrary.ArbitraryDataFileListManager.*;
 
@@ -61,6 +66,7 @@ public class ArbitraryMetadataManager {
 
 
     private ArbitraryMetadataManager() {
+        scheduler.scheduleAtFixedRate(this::processNetworkGetArbitraryMetadataMessage, 60, 1, TimeUnit.SECONDS);
     }
 
     public static ArbitraryMetadataManager getInstance() {
@@ -354,9 +360,8 @@ public class ArbitraryMetadataManager {
 
                         // Forward to requesting peer
                         LOGGER.debug("Forwarding metadata to requesting peer: {}", requestingPeer);
-                        if (!requestingPeer.sendMessage(forwardArbitraryMetadataMessage)) {
-                            requestingPeer.disconnect("failed to forward arbitrary metadata");
-                        }
+                        requestingPeer.sendMessage(forwardArbitraryMetadataMessage);
+                       
                     }
                 }
             }
@@ -371,107 +376,159 @@ public class ArbitraryMetadataManager {
         }
     }
 
+    // List to collect messages
+    private final List<PeerMessage> messageList = new ArrayList<>();
+    // Lock to synchronize access to the list
+    private final Object lock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
+
     public void onNetworkGetArbitraryMetadataMessage(Peer peer, Message message) {
+
         // Don't respond if QDN is disabled
         if (!Settings.getInstance().isQdnEnabled()) {
             return;
         }
 
-        Controller.getInstance().stats.getArbitraryMetadataMessageStats.requests.incrementAndGet();
-
-        GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) message;
-        byte[] signature = getArbitraryMetadataMessage.getSignature();
-        String signature58 = Base58.encode(signature);
-        Long now = NTP.getTime();
-        Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peer, now);
-
-        // If we've seen this request recently, then ignore
-        if (arbitraryMetadataRequests.putIfAbsent(message.getId(), newEntry) != null) {
-            LOGGER.debug("Ignoring metadata request from peer {} for signature {}", peer, signature58);
-            return;
-        }
-
-        LOGGER.debug("Received metadata request from peer {} for signature {}", peer, signature58);
-
-        ArbitraryTransactionData transactionData = null;
-        ArbitraryDataFile metadataFile = null;
-
-        try (final Repository repository = RepositoryManager.getRepository()) {
-
-            // Firstly we need to lookup this file on chain to get its metadata hash
-            transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
-            if (transactionData instanceof ArbitraryTransactionData) {
-
-                // Check if we're even allowed to serve metadata for this transaction
-                if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
-
-                    byte[] metadataHash = transactionData.getMetadataHash();
-                    if (metadataHash != null) {
-
-                        // Load metadata file
-                        metadataFile = ArbitraryDataFile.fromHash(metadataHash, signature);
-                    }
-                }
-            }
-
-        } catch (DataException e) {
-            LOGGER.error(String.format("Repository issue while fetching arbitrary metadata for peer %s", peer), e);
-        }
-
-        // We should only respond if we have the metadata file
-        if (metadataFile != null && metadataFile.exists()) {
-
-            // We have the metadata file, so update requests map to reflect that we've sent it
-            newEntry = new Triple<>(null, null, now);
-            arbitraryMetadataRequests.put(message.getId(), newEntry);
-
-            ArbitraryMetadataMessage arbitraryMetadataMessage = new ArbitraryMetadataMessage(signature, metadataFile);
-            arbitraryMetadataMessage.setId(message.getId());
-            if (!peer.sendMessage(arbitraryMetadataMessage)) {
-                LOGGER.debug("Couldn't send metadata");
-                peer.disconnect("failed to send metadata");
-                return;
-            }
-            LOGGER.debug("Sent metadata");
-
-            // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
-            LOGGER.debug("No need for any forwarding because metadata request is fully served");
-            return;
-
-        }
-
-        // We may need to forward this request on
-        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
-        if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
-            // In relay mode - so ask our other peers if they have it
-
-            long requestTime = getArbitraryMetadataMessage.getRequestTime();
-            int requestHops = getArbitraryMetadataMessage.getRequestHops() + 1;
-            long totalRequestTime = now - requestTime;
-
-            if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
-                // Relay request hasn't timed out yet, so can potentially be rebroadcast
-                if (requestHops < RELAY_REQUEST_MAX_HOPS) {
-                    // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
-
-                    Message relayGetArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, requestTime, requestHops);
-                    relayGetArbitraryMetadataMessage.setId(message.getId());
-
-                    LOGGER.debug("Rebroadcasting metadata request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
-                    Network.getInstance().broadcast(
-                            broadcastPeer ->
-                                    !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
-                                    broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryMetadataMessage);
-
-                }
-                else {
-                    // This relay request has reached the maximum number of allowed hops
-                }
-            }
-            else {
-                // This relay request has timed out
-            }
+        synchronized (lock) {
+            messageList.add(new PeerMessage(peer, message));
         }
     }
 
+    private void processNetworkGetArbitraryMetadataMessage() {
+
+        try {
+            List<PeerMessage> messagesToProcess;
+            synchronized (lock) {
+                messagesToProcess = new ArrayList<>(messageList);
+                messageList.clear();
+            }
+
+            Map<String, byte[]> signatureBySignature58 = new HashMap<>((messagesToProcess.size()));
+            Map<String, Long> nowBySignature58 = new HashMap<>(messagesToProcess.size());
+            Map<String,PeerMessage> peerMessageBySignature58 = new HashMap<>(messagesToProcess.size());
+
+            for( PeerMessage peerMessage : messagesToProcess) {
+                Controller.getInstance().stats.getArbitraryMetadataMessageStats.requests.incrementAndGet();
+
+                GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) peerMessage.message;
+                byte[] signature = getArbitraryMetadataMessage.getSignature();
+                String signature58 = Base58.encode(signature);
+                Long now = NTP.getTime();
+                Triple<String, Peer, Long> newEntry = new Triple<>(signature58, peerMessage.peer, now);
+
+                // If we've seen this request recently, then ignore
+                if (arbitraryMetadataRequests.putIfAbsent(peerMessage.message.getId(), newEntry) != null) {
+                    LOGGER.debug("Ignoring metadata request from peer {} for signature {}", peerMessage.peer, signature58);
+                    continue;
+                }
+
+                LOGGER.debug("Received metadata request from peer {} for signature {}", peerMessage.peer, signature58);
+
+                signatureBySignature58.put(signature58, signature);
+                nowBySignature58.put(signature58, now);
+                peerMessageBySignature58.put(signature58, peerMessage);
+            }
+
+            if( signatureBySignature58.isEmpty() ) return;
+
+            List<TransactionData> transactionDataList;
+            try (final Repository repository = RepositoryManager.getRepository()) {
+
+                // Firstly we need to lookup this file on chain to get its metadata hash
+                transactionDataList = repository.getTransactionRepository().fromSignatures(new ArrayList(signatureBySignature58.values()));
+            } catch (DataException e) {
+                LOGGER.error(String.format("Repository issue while fetching arbitrary transactions"), e);
+                return;
+            }
+
+            Map<String, ArbitraryTransactionData> dataBySignature58
+                = transactionDataList.stream()
+                    .filter(data -> data instanceof ArbitraryTransactionData)
+                    .map(ArbitraryTransactionData.class::cast)
+                    .collect(Collectors.toMap(data -> Base58.encode(data.getSignature()), Function.identity()));
+
+            for(Map.Entry<String, ArbitraryTransactionData> entry : dataBySignature58.entrySet()) {
+                String signature58 = entry.getKey();
+                ArbitraryTransactionData transactionData = entry.getValue();
+
+                try {
+
+                    // Check if we're even allowed to serve metadata for this transaction
+                    if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
+
+                        byte[] metadataHash = transactionData.getMetadataHash();
+                        if (metadataHash != null) {
+
+                            // Load metadata file
+                            ArbitraryDataFile metadataFile = ArbitraryDataFile.fromHash(metadataHash, transactionData.getSignature());
+                            // We should only respond if we have the metadata file
+                            if (metadataFile != null && metadataFile.exists()) {
+
+                                PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
+                                Message message = peerMessage.message;
+                                Peer peer = peerMessage.peer;
+
+                                // We have the metadata file, so update requests map to reflect that we've sent it
+                                Triple newEntry = new Triple<>(null, null, nowBySignature58.get(signature58));
+                                arbitraryMetadataRequests.put(message.getId(), newEntry);
+
+                                ArbitraryMetadataMessage arbitraryMetadataMessage = new ArbitraryMetadataMessage(entry.getValue().getSignature(), metadataFile);
+                                arbitraryMetadataMessage.setId(message.getId());
+                                if (!peer.sendMessage(arbitraryMetadataMessage)) {
+                                    LOGGER.debug("Couldn't send metadata");
+                                    continue;
+                                }
+                                LOGGER.debug("Sent metadata");
+
+                                // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
+                                LOGGER.debug("No need for any forwarding because metadata request is fully served");
+                            }
+
+                        }
+                    }
+                } catch (DataException e) {
+                    LOGGER.error(String.format("Repository issue while fetching arbitrary metadata"), e);
+                }
+
+                // We may need to forward this request on
+                boolean isBlocked = (transactionDataList == null || ListUtils.isNameBlocked(transactionData.getName()));
+                if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
+                    // In relay mode - so ask our other peers if they have it
+
+                    PeerMessage peerMessage = peerMessageBySignature58.get(signature58);
+                    GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) peerMessage.message;
+                    long requestTime = getArbitraryMetadataMessage.getRequestTime();
+                    int requestHops = getArbitraryMetadataMessage.getRequestHops() + 1;
+                    long totalRequestTime = nowBySignature58.get(signature58) - requestTime;
+
+                    if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
+                        // Relay request hasn't timed out yet, so can potentially be rebroadcast
+                        if (requestHops < RELAY_REQUEST_MAX_HOPS) {
+                            // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
+
+                            byte[] signature = signatureBySignature58.get(signature58);
+                            Message relayGetArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, requestTime, requestHops);
+                            relayGetArbitraryMetadataMessage.setId(getArbitraryMetadataMessage.getId());
+
+                            Peer peer = peerMessage.peer;
+                            LOGGER.debug("Rebroadcasting metadata request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
+                            Network.getInstance().broadcast(
+                                    broadcastPeer ->
+                                            !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
+                                                    broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryMetadataMessage);
+
+                        } else {
+                            // This relay request has reached the maximum number of allowed hops
+                        }
+                    } else {
+                        // This relay request has timed out
+                    }
+                }
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
 }

src/main/java/org/qortal/controller/arbitrary/ArbitraryTransactionDataHashWrapper.java

@@ -0,0 +1,48 @@
+package org.qortal.controller.arbitrary;
+
+import org.qortal.arbitrary.misc.Service;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+
+import java.util.Objects;
+
+public class ArbitraryTransactionDataHashWrapper {
+
+    private ArbitraryTransactionData data;
+
+    private int service;
+
+    private String name;
+
+    private String identifier;
+
+    public ArbitraryTransactionDataHashWrapper(ArbitraryTransactionData data) {
+        this.data = data;
+
+        this.service = data.getService().value;
+        this.name = data.getName();
+        this.identifier = data.getIdentifier();
+    }
+
+    public ArbitraryTransactionDataHashWrapper(int service, String name, String identifier) {
+        this.service = service;
+        this.name = name;
+        this.identifier = identifier;
+    }
+
+    public ArbitraryTransactionData getData() {
+        return data;
+    }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        ArbitraryTransactionDataHashWrapper that = (ArbitraryTransactionDataHashWrapper) o;
+        return service == that.service && name.equals(that.name) && Objects.equals(identifier, that.identifier);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(service, name, identifier);
+    }
+}

src/main/java/org/qortal/controller/arbitrary/Follower.java

@@ -0,0 +1,130 @@
+package org.qortal.controller.arbitrary;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.utils.ListUtils;
+import org.qortal.utils.NamedThreadFactory;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.OptionalInt;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+public class Follower {
+
+    private static final Logger LOGGER = LogManager.getLogger(Follower.class);
+
+    private ScheduledExecutorService service
+            = Executors.newScheduledThreadPool(2, new NamedThreadFactory("Follower", Thread.NORM_PRIORITY));
+
+    private Follower() {
+
+    }
+
+    private static Follower instance;
+
+    public static Follower getInstance() {
+
+        if( instance == null ) {
+            instance = new Follower();
+        }
+
+        return instance;
+    }
+
+    public void start() {
+
+        // fetch arbitrary transactions from followed names from the last 100 blocks every 2 minutes
+        service.scheduleWithFixedDelay(() -> fetch(OptionalInt.of(100)), 10, 2, TimeUnit.MINUTES);
+
+        // fetch arbitrary transaction from followed names from any block every 24 hours
+        service.scheduleWithFixedDelay(() -> fetch(OptionalInt.empty()), 4, 24, TimeUnit.HOURS);
+    }
+
+    private void fetch(OptionalInt limit) {
+
+        try {
+            // for each followed name, get arbitraty transactions, then examine those transactions before fetching
+            for (String name : ListUtils.followedNames()) {
+
+                List<ArbitraryTransactionData> transactionsInReverseOrder;
+
+                // open database to get the transactions in reverse order for the followed name
+                try (final Repository repository = RepositoryManager.getRepository()) {
+
+                    List<ArbitraryTransactionData> latestArbitraryTransactionsByName
+                            = repository.getArbitraryRepository().getLatestArbitraryTransactionsByName(name);
+
+                    if (limit.isPresent()) {
+                        final int blockHeightThreshold = repository.getBlockRepository().getBlockchainHeight() - limit.getAsInt();
+
+                        transactionsInReverseOrder
+                                = latestArbitraryTransactionsByName.stream().filter(tx -> tx.getBlockHeight() > blockHeightThreshold)
+                                .collect(Collectors.toList());
+                    } else {
+                        transactionsInReverseOrder = latestArbitraryTransactionsByName;
+                    }
+
+                } catch (Exception e) {
+                    LOGGER.error(e.getMessage(), e);
+                    transactionsInReverseOrder = new ArrayList<>(0);
+                }
+
+                // collect process transaction hashes, so we don't fetch outdated transactions
+                Set<ArbitraryTransactionDataHashWrapper> processedTransactions = new HashSet<>();
+
+                ArbitraryDataStorageManager storageManager = ArbitraryDataStorageManager.getInstance();
+
+                // for each arbitrary transaction for the followed name process, evaluate, fetch
+                for (ArbitraryTransactionData arbitraryTransaction : transactionsInReverseOrder) {
+
+                    boolean examined = false;
+
+                    try (final Repository repository = RepositoryManager.getRepository()) {
+
+                        // if not processed
+                        if (!processedTransactions.contains(new ArbitraryTransactionDataHashWrapper(arbitraryTransaction))) {
+                            boolean isLocal = repository.getArbitraryRepository().isDataLocal(arbitraryTransaction.getSignature());
+
+                            // if not local, then continue to evaluate
+                            if (!isLocal) {
+
+                                // evaluate fetching status for this transaction on this node
+                                ArbitraryDataExamination examination = storageManager.shouldPreFetchData(repository, arbitraryTransaction);
+
+                                // if the evaluation passed, then fetch
+                                examined = examination.isPass();
+                            }
+                            // if locally stored, then nothing needs to be done
+
+                            // add to processed transactions
+                            processedTransactions.add(new ArbitraryTransactionDataHashWrapper(arbitraryTransaction));
+                        }
+                    }
+
+                    // if passed examination for fetching, then fetch
+                    if (examined) {
+                        LOGGER.info("for {} on {}, fetching {}", name, arbitraryTransaction.getService(), arbitraryTransaction.getIdentifier());
+                        boolean fetched = ArbitraryDataFileListManager.getInstance().fetchArbitraryDataFileList(arbitraryTransaction);
+
+                        LOGGER.info("fetched = " + fetched);
+                    }
+
+                    // pause a second before moving on to another transaction
+                    Thread.sleep(1000);
+                }
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+}

src/main/java/org/qortal/controller/arbitrary/PeerMessage.java

@@ -0,0 +1,22 @@
+package org.qortal.controller.arbitrary;
+
+import org.qortal.network.Peer;
+import org.qortal.network.message.Message;
+
+public class PeerMessage {
+    Peer peer;
+    Message message;
+
+    public PeerMessage(Peer peer, Message message) {
+        this.peer = peer;
+        this.message = message;
+    }
+
+    public Peer getPeer() {
+        return peer;
+    }
+
+    public Message getMessage() {
+        return message;
+    }
+}

src/main/java/org/qortal/controller/arbitrary/RNSArbitraryDataFileListManager.java

@@ -0,0 +1,731 @@
+package org.qortal.controller.arbitrary;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.arbitrary.ArbitraryDataFile;
+import org.qortal.arbitrary.ArbitraryDataFileChunk;
+import org.qortal.controller.Controller;
+import org.qortal.data.arbitrary.RNSArbitraryDirectConnectionInfo;
+import org.qortal.data.arbitrary.RNSArbitraryFileListResponseInfo;
+import org.qortal.data.arbitrary.RNSArbitraryRelayInfo;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.network.RNSNetwork;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.ArbitraryDataFileListMessage;
+import org.qortal.network.message.GetArbitraryDataFileListMessage;
+import org.qortal.network.message.Message;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
+import org.qortal.utils.NTP;
+import org.qortal.utils.Triple;
+
+import java.util.*;
+
+import static org.qortal.controller.arbitrary.RNSArbitraryDataFileManager.MAX_FILE_HASH_RESPONSES;
+
+public class RNSArbitraryDataFileListManager {
+
+    private static final Logger LOGGER = LogManager.getLogger(RNSArbitraryDataFileListManager.class);
+
+    private static RNSArbitraryDataFileListManager instance;
+
+    private static String MIN_PEER_VERSION_FOR_FILE_LIST_STATS = "3.2.0";
+
+    /**
+     * Map of recent incoming requests for ARBITRARY transaction data file lists.
+     * <p>
+     * Key is original request's message ID<br>
+     * Value is Triple&lt;transaction signature in base58, first requesting peer, first request's timestamp&gt;
+     * <p>
+     * If peer is null then either:<br>
+     * <ul>
+     * <li>we are the original requesting peer</li>
+     * <li>we have already sent data payload to original requesting peer.</li>
+     * </ul>
+     * If signature is null then we have already received the file list and either:<br>
+     * <ul>
+     * <li>we are the original requesting peer and have processed it</li>
+     * <li>we have forwarded the file list</li>
+     * </ul>
+     */
+    public Map<Integer, Triple<String, RNSPeer, Long>> arbitraryDataFileListRequests = Collections.synchronizedMap(new HashMap<>());
+
+    /**
+     * Map to keep track of in progress arbitrary data signature requests
+     * Key: string - the signature encoded in base58
+     * Value: Triple<networkBroadcastCount, directPeerRequestCount, lastAttemptTimestamp>
+     */
+    private Map<String, Triple<Integer, Integer, Long>> arbitraryDataSignatureRequests = Collections.synchronizedMap(new HashMap<>());
+
+
+    /** Maximum number of seconds that a file list relay request is able to exist on the network */
+    public static long RELAY_REQUEST_MAX_DURATION = 5000L;
+    /** Maximum number of hops that a file list relay request is allowed to make */
+    public static int RELAY_REQUEST_MAX_HOPS = 4;
+
+    /** Minimum peer version to use relay */
+    public static String RELAY_MIN_PEER_VERSION = "3.4.0";
+
+
+    private RNSArbitraryDataFileListManager() {
+    }
+
+    public static RNSArbitraryDataFileListManager getInstance() {
+        if (instance == null)
+            instance = new RNSArbitraryDataFileListManager();
+
+        return instance;
+    }
+
+
+    public void cleanupRequestCache(Long now) {
+        if (now == null) {
+            return;
+        }
+        final long requestMinimumTimestamp = now - ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT;
+        arbitraryDataFileListRequests.entrySet().removeIf(entry -> entry.getValue().getC() == null || entry.getValue().getC() < requestMinimumTimestamp);
+    }
+
+
+    // Track file list lookups by signature
+
+    private boolean shouldMakeFileListRequestForSignature(String signature58) {
+        Triple<Integer, Integer, Long> request = arbitraryDataSignatureRequests.get(signature58);
+
+        if (request == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        // Extract the components
+        Integer networkBroadcastCount = request.getA();
+        // Integer directPeerRequestCount = request.getB();
+        Long lastAttemptTimestamp = request.getC();
+
+        if (lastAttemptTimestamp == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        long timeSinceLastAttempt = NTP.getTime() - lastAttemptTimestamp;
+
+        // Allow a second attempt after 15 seconds, and another after 30 seconds
+        if (timeSinceLastAttempt > 15 * 1000L) {
+            // We haven't tried for at least 15 seconds
+
+            if (networkBroadcastCount < 3) {
+                // We've made less than 3 total attempts
+                return true;
+            }
+        }
+
+        // Then allow another 5 attempts, each 1 minute apart
+        if (timeSinceLastAttempt > 60 * 1000L) {
+            // We haven't tried for at least 1 minute
+
+            if (networkBroadcastCount < 8) {
+                // We've made less than 8 total attempts
+                return true;
+            }
+        }
+
+        // Then allow another 8 attempts, each 15 minutes apart
+        if (timeSinceLastAttempt > 15 * 60 * 1000L) {
+            // We haven't tried for at least 15 minutes
+
+            if (networkBroadcastCount < 16) {
+                // We've made less than 16 total attempts
+                return true;
+            }
+        }
+
+        // From then on, only try once every 6 hours, to reduce network spam
+        if (timeSinceLastAttempt > 6 * 60 * 60 * 1000L) {
+            // We haven't tried for at least 6 hours
+            return true;
+        }
+
+        return false;
+    }
+
+    private boolean shouldMakeDirectFileRequestsForSignature(String signature58) {
+        if (!Settings.getInstance().isDirectDataRetrievalEnabled()) {
+            // Direct connections are disabled in the settings
+            return false;
+        }
+
+        Triple<Integer, Integer, Long> request = arbitraryDataSignatureRequests.get(signature58);
+
+        if (request == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        // Extract the components
+        //Integer networkBroadcastCount = request.getA();
+        Integer directPeerRequestCount = request.getB();
+        Long lastAttemptTimestamp = request.getC();
+
+        if (lastAttemptTimestamp == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        if (directPeerRequestCount == 0) {
+            // We haven't tried asking peers directly yet, so we should
+            return true;
+        }
+
+        long timeSinceLastAttempt = NTP.getTime() - lastAttemptTimestamp;
+        if (timeSinceLastAttempt > 10 * 1000L) {
+            // We haven't tried for at least 10 seconds
+            if (directPeerRequestCount < 5) {
+                // We've made less than 5 total attempts
+                return true;
+            }
+        }
+
+        if (timeSinceLastAttempt > 5 * 60 * 1000L) {
+            // We haven't tried for at least 5 minutes
+            if (directPeerRequestCount < 10) {
+                // We've made less than 10 total attempts
+                return true;
+            }
+        }
+
+        if (timeSinceLastAttempt > 60 * 60 * 1000L) {
+            // We haven't tried for at least 1 hour
+            return true;
+        }
+
+        return false;
+    }
+
+    public boolean isSignatureRateLimited(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+        return !this.shouldMakeFileListRequestForSignature(signature58)
+                && !this.shouldMakeDirectFileRequestsForSignature(signature58);
+    }
+
+    public long lastRequestForSignature(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+        Triple<Integer, Integer, Long> request = arbitraryDataSignatureRequests.get(signature58);
+
+        if (request == null) {
+            // Not attempted yet
+            return 0;
+        }
+
+        // Extract the components
+        Long lastAttemptTimestamp = request.getC();
+        if (lastAttemptTimestamp != null) {
+            return  lastAttemptTimestamp;
+        }
+        return 0;
+    }
+
+    public void addToSignatureRequests(String signature58, boolean incrementNetworkRequests, boolean incrementPeerRequests) {
+        Triple<Integer, Integer, Long> request  = arbitraryDataSignatureRequests.get(signature58);
+        Long now = NTP.getTime();
+
+        if (request == null) {
+            // No entry yet
+            Triple<Integer, Integer, Long> newRequest = new Triple<>(0, 0, now);
+            arbitraryDataSignatureRequests.put(signature58, newRequest);
+        }
+        else {
+            // There is an existing entry
+            if (incrementNetworkRequests) {
+                request.setA(request.getA() + 1);
+            }
+            if (incrementPeerRequests) {
+                request.setB(request.getB() + 1);
+            }
+            request.setC(now);
+            arbitraryDataSignatureRequests.put(signature58, request);
+        }
+    }
+
+    public void removeFromSignatureRequests(String signature58) {
+        arbitraryDataSignatureRequests.remove(signature58);
+    }
+
+
+    // Lookup file lists by signature (and optionally hashes)
+
+    public boolean fetchArbitraryDataFileList(ArbitraryTransactionData arbitraryTransactionData) {
+        byte[] signature = arbitraryTransactionData.getSignature();
+        String signature58 = Base58.encode(signature);
+
+        // Require an NTP sync
+        Long now = NTP.getTime();
+        if (now == null) {
+            return false;
+        }
+
+        // If we've already tried too many times in a short space of time, make sure to give up
+        if (!this.shouldMakeFileListRequestForSignature(signature58)) {
+            // Check if we should make direct connections to peers
+            if (this.shouldMakeDirectFileRequestsForSignature(signature58)) {
+                return RNSArbitraryDataFileManager.getInstance().fetchDataFilesFromPeersForSignature(signature);
+            }
+
+            LOGGER.trace("Skipping file list request for signature {} due to rate limit", signature58);
+            return false;
+        }
+        this.addToSignatureRequests(signature58, true, false);
+
+        //List<Peer> handshakedPeers = Network.getInstance().getImmutableHandshakedPeers();
+        List<RNSPeer> handshakedPeers = RNSNetwork.getInstance().getLinkedPeers();
+        List<byte[]> missingHashes = null;
+
+        // Find hashes that we are missing
+        try {
+            ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
+            missingHashes = arbitraryDataFile.missingHashes();
+        } catch (DataException e) {
+            // Leave missingHashes as null, so that all hashes are requested
+        }
+        int hashCount = missingHashes != null ? missingHashes.size() : 0;
+
+        LOGGER.debug(String.format("Sending data file list request for signature %s with %d hashes to %d peers...", signature58, hashCount, handshakedPeers.size()));
+
+        //// Send our address as requestingPeer, to allow for potential direct connections with seeds/peers
+        //String requestingPeer = Network.getInstance().getOurExternalIpAddressAndPort();
+        String requestingPeer = null;
+
+        // Build request
+        Message getArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, missingHashes, now, 0, requestingPeer);
+
+        // Save our request into requests map
+        Triple<String, RNSPeer, Long> requestEntry = new Triple<>(signature58, null, NTP.getTime());
+
+        // Assign random ID to this message
+        int id;
+        do {
+            id = new Random().nextInt(Integer.MAX_VALUE - 1) + 1;
+
+            // Put queue into map (keyed by message ID) so we can poll for a response
+            // If putIfAbsent() doesn't return null, then this ID is already taken
+        } while (arbitraryDataFileListRequests.put(id, requestEntry) != null);
+        getArbitraryDataFileListMessage.setId(id);
+
+        // Broadcast request
+        RNSNetwork.getInstance().broadcast(peer -> getArbitraryDataFileListMessage);
+
+        // Poll to see if data has arrived
+        final long singleWait = 100;
+        long totalWait = 0;
+        while (totalWait < ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT) {
+            try {
+                Thread.sleep(singleWait);
+            } catch (InterruptedException e) {
+                break;
+            }
+
+            requestEntry = arbitraryDataFileListRequests.get(id);
+            if (requestEntry == null)
+                return false;
+
+            if (requestEntry.getA() == null)
+                break;
+
+            totalWait += singleWait;
+        }
+        return true;
+    }
+
+    public boolean fetchArbitraryDataFileList(RNSPeer peer, byte[] signature) {
+        String signature58 = Base58.encode(signature);
+
+        // Require an NTP sync
+        Long now = NTP.getTime();
+        if (now == null) {
+            return false;
+        }
+
+        int hashCount = 0;
+        LOGGER.debug(String.format("Sending data file list request for signature %s with %d hashes to peer %s...", signature58, hashCount, peer));
+
+        // Build request
+        // Use a time in the past, so that the recipient peer doesn't try and relay it
+        // Also, set hashes to null since it's easier to request all hashes than it is to determine which ones we need
+        // This could be optimized in the future
+        long timestamp = now - 60000L;
+        List<byte[]> hashes = null;
+        Message getArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, timestamp, 0, null);
+
+        // Save our request into requests map
+        Triple<String, RNSPeer, Long> requestEntry = new Triple<>(signature58, null, NTP.getTime());
+
+        // Assign random ID to this message
+        int id;
+        do {
+            id = new Random().nextInt(Integer.MAX_VALUE - 1) + 1;
+
+            // Put queue into map (keyed by message ID) so we can poll for a response
+            // If putIfAbsent() doesn't return null, then this ID is already taken
+        } while (arbitraryDataFileListRequests.put(id, requestEntry) != null);
+        getArbitraryDataFileListMessage.setId(id);
+
+        // Send the request
+        peer.sendMessage(getArbitraryDataFileListMessage);
+
+        // Poll to see if data has arrived
+        final long singleWait = 100;
+        long totalWait = 0;
+        while (totalWait < ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT) {
+            try {
+                Thread.sleep(singleWait);
+            } catch (InterruptedException e) {
+                break;
+            }
+
+            requestEntry = arbitraryDataFileListRequests.get(id);
+            if (requestEntry == null)
+                return false;
+
+            if (requestEntry.getA() == null)
+                break;
+
+            totalWait += singleWait;
+        }
+        return true;
+    }
+
+    public void deleteFileListRequestsForSignature(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+        for (Iterator<Map.Entry<Integer, Triple<String, RNSPeer, Long>>> it = arbitraryDataFileListRequests.entrySet().iterator(); it.hasNext();) {
+            Map.Entry<Integer, Triple<String, RNSPeer, Long>> entry = it.next();
+            if (entry == null || entry.getKey() == null || entry.getValue() != null) {
+                continue;
+            }
+            if (Objects.equals(entry.getValue().getA(), signature58)) {
+                // Update requests map to reflect that we've received all chunks
+                Triple<String, RNSPeer, Long> newEntry = new Triple<>(null, null, entry.getValue().getC());
+                arbitraryDataFileListRequests.put(entry.getKey(), newEntry);
+            }
+        }
+    }
+
+    // Network handlers
+
+    public void onNetworkArbitraryDataFileListMessage(RNSPeer peer, Message message) {
+        // Don't process if QDN is disabled
+        if (!Settings.getInstance().isQdnEnabled()) {
+            return;
+        }
+
+        ArbitraryDataFileListMessage arbitraryDataFileListMessage = (ArbitraryDataFileListMessage) message;
+        LOGGER.debug("Received hash list from peer {} with {} hashes", peer, arbitraryDataFileListMessage.getHashes().size());
+
+        if (LOGGER.isDebugEnabled() && arbitraryDataFileListMessage.getRequestTime() != null) {
+            long totalRequestTime = NTP.getTime() - arbitraryDataFileListMessage.getRequestTime();
+            LOGGER.debug("totalRequestTime: {}, requestHops: {}, peerAddress: {}, isRelayPossible: {}",
+                    totalRequestTime, arbitraryDataFileListMessage.getRequestHops(),
+                    arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+        }
+
+        // Do we have a pending request for this data?
+        Triple<String, RNSPeer, Long> request = arbitraryDataFileListRequests.get(message.getId());
+        if (request == null || request.getA() == null) {
+            return;
+        }
+        boolean isRelayRequest = (request.getB() != null);
+
+        // Does this message's signature match what we're expecting?
+        byte[] signature = arbitraryDataFileListMessage.getSignature();
+        String signature58 = Base58.encode(signature);
+        if (!request.getA().equals(signature58)) {
+            return;
+        }
+
+        List<byte[]> hashes = arbitraryDataFileListMessage.getHashes();
+        if (hashes == null || hashes.isEmpty()) {
+            return;
+        }
+
+        ArbitraryTransactionData arbitraryTransactionData = null;
+
+        // Check transaction exists and hashes are correct
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            if (!(transactionData instanceof ArbitraryTransactionData))
+                return;
+
+            arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
+
+//          // Load data file(s)
+//          ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
+//
+//			// Check all hashes exist
+//			for (byte[] hash : hashes) {
+//				//LOGGER.debug("Received hash {}", Base58.encode(hash));
+//				if (!arbitraryDataFile.containsChunk(hash)) {
+//					// Check the hash against the complete file
+//					if (!Arrays.equals(arbitraryDataFile.getHash(), hash)) {
+//						LOGGER.info("Received non-matching chunk hash {} for signature {}. This could happen if we haven't obtained the metadata file yet.", Base58.encode(hash), signature58);
+//						return;
+//					}
+//				}
+//			}
+
+            if (!isRelayRequest || !Settings.getInstance().isRelayModeEnabled()) {
+                Long now = NTP.getTime();
+
+                if (RNSArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.size() < MAX_FILE_HASH_RESPONSES) {
+                    // Keep track of the hashes this peer reports to have access to
+                    for (byte[] hash : hashes) {
+                        String hash58 = Base58.encode(hash);
+
+                        // Treat null request hops as 100, so that they are able to be sorted (and put to the end of the list)
+                        int requestHops = arbitraryDataFileListMessage.getRequestHops() != null ? arbitraryDataFileListMessage.getRequestHops() : 100;
+
+                        RNSArbitraryFileListResponseInfo responseInfo = new RNSArbitraryFileListResponseInfo(hash58, signature58,
+                                peer, now, arbitraryDataFileListMessage.getRequestTime(), requestHops);
+
+                        RNSArbitraryDataFileManager.getInstance().arbitraryDataFileHashResponses.add(responseInfo);
+                    }
+                }
+
+                // Keep track of the source peer, for direct connections
+                if (arbitraryDataFileListMessage.getPeerAddress() != null) {
+                    RNSArbitraryDataFileManager.getInstance().addDirectConnectionInfoIfUnique(
+                            new RNSArbitraryDirectConnectionInfo(signature, arbitraryDataFileListMessage.getPeerAddress(), hashes, now));
+                }
+            }
+
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while finding arbitrary transaction data list for peer %s", peer), e);
+        }
+
+        // Forwarding
+        if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
+            boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
+            if (!isBlocked) {
+                RNSPeer requestingPeer = request.getB();
+                if (requestingPeer != null) {
+                    Long requestTime = arbitraryDataFileListMessage.getRequestTime();
+                    Integer requestHops = arbitraryDataFileListMessage.getRequestHops();
+
+                    // Add each hash to our local mapping so we know who to ask later
+                    Long now = NTP.getTime();
+                    for (byte[] hash : hashes) {
+                        String hash58 = Base58.encode(hash);
+                        RNSArbitraryRelayInfo relayInfo = new RNSArbitraryRelayInfo(hash58, signature58, peer, now, requestTime, requestHops);
+                        RNSArbitraryDataFileManager.getInstance().addToRelayMap(relayInfo);
+                    }
+
+                    // Bump requestHops if it exists
+                    if (requestHops != null) {
+                        requestHops++;
+                    }
+
+                    ArbitraryDataFileListMessage forwardArbitraryDataFileListMessage;
+
+                    //// TODO - rework for Reticulum
+                    //// Remove optional parameters if the requesting peer doesn't support it yet
+                    //// A message with less statistical data is better than no message at all
+                    //if (!requestingPeer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
+                    //    forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
+                    //} else {
+                    //    forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
+                    //            arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+                    //}
+                    forwardArbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops,
+                            arbitraryDataFileListMessage.getPeerAddress(), arbitraryDataFileListMessage.isRelayPossible());
+                    forwardArbitraryDataFileListMessage.setId(message.getId());
+
+                    // Forward to requesting peer
+                    LOGGER.debug("Forwarding file list with {} hashes to requesting peer: {}", hashes.size(), requestingPeer);
+                    //if (!requestingPeer.sendMessage(forwardArbitraryDataFileListMessage)) {
+                    //    requestingPeer.disconnect("failed to forward arbitrary data file list");
+                    //}
+                    requestingPeer.sendMessage(forwardArbitraryDataFileListMessage);
+                }
+            }
+        }
+    }
+
+    public void onNetworkGetArbitraryDataFileListMessage(RNSPeer peer, Message message) {
+        // Don't respond if QDN is disabled
+        if (!Settings.getInstance().isQdnEnabled()) {
+            return;
+        }
+
+        Controller.getInstance().stats.getArbitraryDataFileListMessageStats.requests.incrementAndGet();
+
+        GetArbitraryDataFileListMessage getArbitraryDataFileListMessage = (GetArbitraryDataFileListMessage) message;
+        byte[] signature = getArbitraryDataFileListMessage.getSignature();
+        String signature58 = Base58.encode(signature);
+        Long now = NTP.getTime();
+        Triple<String, RNSPeer, Long> newEntry = new Triple<>(signature58, peer, now);
+
+        // If we've seen this request recently, then ignore
+        if (arbitraryDataFileListRequests.putIfAbsent(message.getId(), newEntry) != null) {
+            LOGGER.trace("Ignoring hash list request from peer {} for signature {}", peer, signature58);
+            return;
+        }
+
+        List<byte[]> requestedHashes = getArbitraryDataFileListMessage.getHashes();
+        int hashCount = requestedHashes != null ? requestedHashes.size() : 0;
+        String requestingPeer = getArbitraryDataFileListMessage.getRequestingPeer();
+
+        if (requestingPeer != null) {
+            LOGGER.debug("Received hash list request with {} hashes from peer {} (requesting peer {}) for signature {}", hashCount, peer, requestingPeer, signature58);
+        }
+        else {
+            LOGGER.debug("Received hash list request with {} hashes from peer {} for signature {}", hashCount, peer, signature58);
+        }
+
+        List<byte[]> hashes = new ArrayList<>();
+        ArbitraryTransactionData transactionData = null;
+        boolean allChunksExist = false;
+        boolean hasMetadata = false;
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+
+            // Firstly we need to lookup this file on chain to get a list of its hashes
+            transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
+            if (transactionData instanceof ArbitraryTransactionData) {
+
+                // Check if we're even allowed to serve data for this transaction
+                if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
+
+                    // Load file(s) and add any that exist to the list of hashes
+                    ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
+
+                    // If the peer didn't supply a hash list, we need to return all hashes for this transaction
+                    if (requestedHashes == null || requestedHashes.isEmpty()) {
+                        requestedHashes = new ArrayList<>();
+
+                        // Add the metadata file
+                        if (arbitraryDataFile.getMetadataHash() != null) {
+                            requestedHashes.add(arbitraryDataFile.getMetadataHash());
+                            hasMetadata = true;
+                        }
+
+                        // Add the chunk hashes
+                        if (!arbitraryDataFile.getChunkHashes().isEmpty()) {
+                            requestedHashes.addAll(arbitraryDataFile.getChunkHashes());
+                        }
+                        // Add complete file if there are no hashes
+                        else {
+                            requestedHashes.add(arbitraryDataFile.getHash());
+                        }
+                    }
+
+                    // Assume all chunks exists, unless one can't be found below
+                    allChunksExist = true;
+
+                    for (byte[] requestedHash : requestedHashes) {
+                        ArbitraryDataFileChunk chunk = ArbitraryDataFileChunk.fromHash(requestedHash, signature);
+                        if (chunk.exists()) {
+                            hashes.add(chunk.getHash());
+                            //LOGGER.trace("Added hash {}", chunk.getHash58());
+                        } else {
+                            LOGGER.trace("Couldn't add hash {} because it doesn't exist", chunk.getHash58());
+                            allChunksExist = false;
+                        }
+                    }
+                }
+            }
+
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while fetching arbitrary file list for peer %s", peer), e);
+        }
+
+        // If the only file we have is the metadata then we shouldn't respond. Most nodes will already have that,
+        // or can use the separate metadata protocol to fetch it. This should greatly reduce network spam.
+        if (hasMetadata && hashes.size() == 1) {
+            hashes.clear();
+        }
+
+        // We should only respond if we have at least one hash
+        if (!hashes.isEmpty()) {
+
+            // Firstly we should keep track of the requesting peer, to allow for potential direct connections later
+            RNSArbitraryDataFileManager.getInstance().addRecentDataRequest(requestingPeer);
+
+            // We have all the chunks, so update requests map to reflect that we've sent it
+            // There is no need to keep track of the request, as we can serve all the chunks
+            if (allChunksExist) {
+                newEntry = new Triple<>(null, null, now);
+                arbitraryDataFileListRequests.put(message.getId(), newEntry);
+            }
+
+            //String ourAddress = RNSNetwork.getInstance().getOurExternalIpAddressAndPort();
+            String ourAddress = RNSNetwork.getInstance().getBaseDestination().getHexHash();
+            ArbitraryDataFileListMessage arbitraryDataFileListMessage;
+
+            // TODO: rework for Reticulum
+            // Remove optional parameters if the requesting peer doesn't support it yet
+            // A message with less statistical data is better than no message at all
+            //if (!peer.isAtLeastVersion(MIN_PEER_VERSION_FOR_FILE_LIST_STATS)) {
+            //    arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
+            //} else {
+            //    arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature,
+            //            hashes, NTP.getTime(), 0, ourAddress, true);
+            //}
+            arbitraryDataFileListMessage = new ArbitraryDataFileListMessage(signature, hashes);
+
+            arbitraryDataFileListMessage.setId(message.getId());
+
+            //if (!peer.sendMessage(arbitraryDataFileListMessage)) {
+            //    LOGGER.debug("Couldn't send list of hashes");
+            //    peer.disconnect("failed to send list of hashes");
+            //    return;
+            //}
+            peer.sendMessage(arbitraryDataFileListMessage);
+            LOGGER.debug("Sent list of hashes (count: {})", hashes.size());
+
+            if (allChunksExist) {
+                // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
+                LOGGER.debug("No need for any forwarding because file list request is fully served");
+                return;
+            }
+
+        }
+
+        // We may need to forward this request on
+        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
+        if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
+            // In relay mode - so ask our other peers if they have it
+
+            long requestTime = getArbitraryDataFileListMessage.getRequestTime();
+            int requestHops = getArbitraryDataFileListMessage.getRequestHops() + 1;
+            long totalRequestTime = now - requestTime;
+
+            if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
+                // Relay request hasn't timed out yet, so can potentially be rebroadcast
+                if (requestHops < RELAY_REQUEST_MAX_HOPS) {
+                    // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
+
+                    Message relayGetArbitraryDataFileListMessage = new GetArbitraryDataFileListMessage(signature, hashes, requestTime, requestHops, requestingPeer);
+                    relayGetArbitraryDataFileListMessage.setId(message.getId());
+
+                    LOGGER.debug("Rebroadcasting hash list request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
+                    //Network.getInstance().broadcast(
+                    //        broadcastPeer ->
+                    //                !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
+                    //                broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryDataFileListMessage
+                    //);
+                    RNSNetwork.getInstance().broadcast(broadcastPeer -> relayGetArbitraryDataFileListMessage);
+
+                }
+                else {
+                    // This relay request has reached the maximum number of allowed hops
+                }
+            }
+            else {
+                // This relay request has timed out
+            }
+        }
+    }
+
+}

src/main/java/org/qortal/controller/arbitrary/RNSArbitraryDataFileManager.java

@@ -0,0 +1,718 @@
+package org.qortal.controller.arbitrary;
+
+import com.google.common.net.InetAddresses;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.arbitrary.ArbitraryDataFile;
+import org.qortal.controller.Controller;
+import org.qortal.data.arbitrary.RNSArbitraryDirectConnectionInfo;
+import org.qortal.data.arbitrary.RNSArbitraryFileListResponseInfo;
+import org.qortal.data.arbitrary.RNSArbitraryRelayInfo;
+import org.qortal.data.network.RNSPeerData;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.network.RNSNetwork;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.*;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.utils.ArbitraryTransactionUtils;
+import org.qortal.utils.Base58;
+import org.qortal.utils.NTP;
+
+//import java.security.SecureRandom;
+//import java.util.*;
+//import java.util.concurrent.ExecutorService;
+//import java.util.concurrent.Executors;
+//import java.util.stream.Collectors;
+
+import java.security.SecureRandom;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+public class RNSArbitraryDataFileManager extends Thread {
+
+    private static final Logger LOGGER = LogManager.getLogger(RNSArbitraryDataFileManager.class);
+
+    private static RNSArbitraryDataFileManager instance;
+    private volatile boolean isStopping = false;
+
+
+    /**
+     * Map to keep track of our in progress (outgoing) arbitrary data file requests
+     */
+    public Map<String, Long> arbitraryDataFileRequests = Collections.synchronizedMap(new HashMap<>());
+
+    /**
+     * Map to keep track of hashes that we might need to relay
+     */
+    public final List<RNSArbitraryRelayInfo> arbitraryRelayMap = Collections.synchronizedList(new ArrayList<>());
+
+    /**
+     * List to keep track of any arbitrary data file hash responses
+     */
+    public final List<RNSArbitraryFileListResponseInfo> arbitraryDataFileHashResponses = Collections.synchronizedList(new ArrayList<>());
+
+    /**
+     * List to keep track of peers potentially available for direct connections, based on recent requests
+     */
+    private final List<RNSArbitraryDirectConnectionInfo> directConnectionInfo = Collections.synchronizedList(new ArrayList<>());
+
+    /**
+     * Map to keep track of peers requesting QDN data that we hold.
+     * Key = peer address string, value = time of last request.
+     * This allows for additional "burst" connections beyond existing limits.
+     */
+    private Map<String, Long> recentDataRequests = Collections.synchronizedMap(new HashMap<>());
+
+
+    public static int MAX_FILE_HASH_RESPONSES = 1000;
+
+
+    private RNSArbitraryDataFileManager() {
+    }
+
+    public static RNSArbitraryDataFileManager getInstance() {
+        if (instance == null)
+            instance = new RNSArbitraryDataFileManager();
+
+        return instance;
+    }
+
+    @Override
+    public void run() {
+        Thread.currentThread().setName("Arbitrary Data File Manager");
+
+        try {
+            while (!isStopping) {
+                // Nothing to do yet
+                Thread.sleep(1000);
+            }
+        } catch (InterruptedException e) {
+            // Fall-through to exit thread...
+        }
+    }
+
+    public void shutdown() {
+        isStopping = true;
+        this.interrupt();
+    }
+
+
+    public void cleanupRequestCache(Long now) {
+        if (now == null) {
+            return;
+        }
+        final long requestMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_REQUEST_TIMEOUT;
+        arbitraryDataFileRequests.entrySet().removeIf(entry -> entry.getValue() == null || entry.getValue() < requestMinimumTimestamp);
+
+        final long relayMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_RELAY_TIMEOUT;
+        arbitraryRelayMap.removeIf(entry -> entry == null || entry.getTimestamp() == null || entry.getTimestamp() < relayMinimumTimestamp);
+
+        final long directConnectionInfoMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_DIRECT_CONNECTION_INFO_TIMEOUT;
+        directConnectionInfo.removeIf(entry -> entry.getTimestamp() < directConnectionInfoMinimumTimestamp);
+
+        final long recentDataRequestMinimumTimestamp = now - ArbitraryDataManager.getInstance().ARBITRARY_RECENT_DATA_REQUESTS_TIMEOUT;
+        recentDataRequests.entrySet().removeIf(entry -> entry.getValue() < recentDataRequestMinimumTimestamp);
+    }
+
+
+
+    // Fetch data files by hash
+
+    public boolean fetchArbitraryDataFiles(RNSPeer peer,
+                                           byte[] signature,
+                                           ArbitraryTransactionData arbitraryTransactionData,
+                                           List<byte[]> hashes) throws DataException {
+
+        // Load data file(s)
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
+        boolean receivedAtLeastOneFile = false;
+
+        // Now fetch actual data from this peer
+        for (byte[] hash : hashes) {
+            if (isStopping) {
+                return false;
+            }
+            String hash58 = Base58.encode(hash);
+            if (!arbitraryDataFile.chunkExists(hash)) {
+                // Only request the file if we aren't already requesting it from someone else
+                if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) {
+                    LOGGER.debug("Requesting data file {} from peer {}", hash58, peer);
+                    Long startTime = NTP.getTime();
+                    ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, arbitraryTransactionData, signature, hash);
+                    Long endTime = NTP.getTime();
+                    if (receivedArbitraryDataFile != null) {
+                        LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime));
+                        receivedAtLeastOneFile = true;
+                    }
+                    else {
+                        LOGGER.debug("Peer {} didn't respond with data file {} for signature {}. Time taken: {} ms", peer, Base58.encode(hash), Base58.encode(signature), (endTime-startTime));
+
+                        // Stop asking for files from this peer
+                        break;
+                    }
+                }
+                else {
+                    LOGGER.trace("Already requesting data file {} for signature {} from peer {}", arbitraryDataFile, Base58.encode(signature), peer);
+                }
+            }
+        }
+
+        if (receivedAtLeastOneFile) {
+            // Invalidate the hosted transactions cache as we are now hosting something new
+            ArbitraryDataStorageManager.getInstance().invalidateHostedTransactionsCache();
+
+            // Check if we have all the files we need for this transaction
+            if (arbitraryDataFile.allFilesExist()) {
+
+                // We have all the chunks for this transaction, so we should invalidate the transaction's name's
+                // data cache so that it is rebuilt the next time we serve it
+                ArbitraryDataManager.getInstance().invalidateCache(arbitraryTransactionData);
+            }
+        }
+
+        return receivedAtLeastOneFile;
+    }
+
+    // Lock to synchronize access to the list
+    private final Object arbitraryDataFileHashResponseLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService arbitraryDataFileHashResponseScheduler = Executors.newScheduledThreadPool(1);
+
+
+    public void addResponse( RNSArbitraryFileListResponseInfo responseInfo ) {
+
+        synchronized (arbitraryDataFileHashResponseLock) {
+            this.arbitraryDataFileHashResponses.add(responseInfo);
+        }
+    }
+
+    private void processResponses() {
+        try {
+            List<RNSArbitraryFileListResponseInfo> responsesToProcess;
+            synchronized (arbitraryDataFileHashResponseLock) {
+                responsesToProcess = new ArrayList<>(arbitraryDataFileHashResponses);
+                arbitraryDataFileHashResponses.clear();
+            }
+
+            if (responsesToProcess.isEmpty()) return;
+
+            Long now = NTP.getTime();
+
+            RNSArbitraryDataFileRequestThread.getInstance().processFileHashes(now, responsesToProcess, this);
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    private ArbitraryDataFile fetchArbitraryDataFile(RNSPeer peer, ArbitraryTransactionData arbitraryTransactionData, byte[] signature, byte[] hash) throws DataException {
+        ArbitraryDataFile arbitraryDataFile;
+
+        try {
+            ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
+            boolean fileAlreadyExists = existingFile.exists();
+            String hash58 = Base58.encode(hash);
+
+            // Fetch the file if it doesn't exist locally
+            if (!fileAlreadyExists) {
+                LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
+                arbitraryDataFileRequests.put(hash58, NTP.getTime());
+                Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
+
+                Message response = null;
+                try {
+                    response = peer.getResponseWithTimeout(getArbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
+                } catch (InterruptedException e) {
+                    // Will return below due to null response
+                }
+                arbitraryDataFileRequests.remove(hash58);
+                LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
+
+                if (response == null) {
+                    LOGGER.debug("Received null response from peer {}", peer);
+                    return null;
+                }
+                if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
+                    LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
+                    return null;
+                }
+
+                ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
+                arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
+            } else {
+                LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
+                arbitraryDataFile = existingFile;
+            }
+
+            if (arbitraryDataFile != null) {
+
+                arbitraryDataFile.save();
+
+                // If this is a metadata file then we need to update the cache
+                if (arbitraryTransactionData != null && arbitraryTransactionData.getMetadataHash() != null) {
+                    if (Arrays.equals(arbitraryTransactionData.getMetadataHash(), hash)) {
+                        ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
+                    }
+                }
+
+                // We may need to remove the file list request, if we have all the files for this transaction
+                this.handleFileListRequests(signature);
+            }
+        } catch (DataException e) {
+            LOGGER.error(e.getMessage(), e);
+            arbitraryDataFile = null;
+        }
+
+        return arbitraryDataFile;
+    }
+
+    private void fetchFileForRelay(RNSPeer peer, RNSPeer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
+        try {
+            String hash58 = Base58.encode(hash);
+
+            LOGGER.debug(String.format("Fetching data file %.8s from peer %s", hash58, peer));
+            arbitraryDataFileRequests.put(hash58, NTP.getTime());
+            Message getArbitraryDataFileMessage = new GetArbitraryDataFileMessage(signature, hash);
+
+            Message response = null;
+            // TODO - revisit with RNS
+            try {
+                response = peer.getResponseWithTimeout(getArbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
+            } catch (InterruptedException e) {
+                // Will return below due to null response
+            }
+            arbitraryDataFileRequests.remove(hash58);
+            LOGGER.trace(String.format("Removed hash %.8s from arbitraryDataFileRequests", hash58));
+
+            if (response == null) {
+                LOGGER.debug("Received null response from peer {}", peer);
+                return;
+            }
+            if (response.getType() != MessageType.ARBITRARY_DATA_FILE) {
+                LOGGER.debug("Received response with invalid type: {} from peer {}", response.getType(), peer);
+                return;
+            }
+
+            ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
+            ArbitraryDataFile arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
+
+            if (arbitraryDataFile != null) {
+
+                // We might want to forward the request to the peer that originally requested it
+                this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
+            }
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    Map<String, byte[]> signatureBySignature58 = new HashMap<>();
+
+    // Lock to synchronize access to the list
+    private final Object handleFileListRequestsLock = new Object();
+
+    // Scheduled executor service to process messages every second
+    private final ScheduledExecutorService handleFileListRequestsScheduler = Executors.newScheduledThreadPool(1);
+
+    private void handleFileListRequests(byte[] signature) {
+
+        synchronized (handleFileListRequestsLock) {
+            signatureBySignature58.put(Base58.encode(signature), signature);
+        }
+    }
+
+    private void handleFileListRequestProcess() {
+
+        Map<String, byte[]> signaturesToProcess;
+
+        synchronized (handleFileListRequestsLock) {
+            signaturesToProcess = new HashMap<>(signatureBySignature58);
+            signatureBySignature58.clear();
+        }
+
+        if( signaturesToProcess.isEmpty() ) return;
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+
+            // Fetch the transaction data
+            List<ArbitraryTransactionData> arbitraryTransactionDataList
+                = ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signaturesToProcess.values()));
+
+            for( ArbitraryTransactionData arbitraryTransactionData : arbitraryTransactionDataList ) {
+                boolean completeFileExists = ArbitraryTransactionUtils.completeFileExists(arbitraryTransactionData);
+
+                if (completeFileExists) {
+                    String signature58 = Base58.encode(arbitraryTransactionData.getSignature());
+                    LOGGER.debug("All chunks or complete file exist for transaction {}", signature58);
+
+                    ArbitraryDataFileListManager.getInstance().deleteFileListRequestsForSignature(signature58);
+                }
+            }
+
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    public void handleArbitraryDataFileForwarding(RNSPeer requestingPeer, Message message, Message originalMessage) {
+        // Return if there is no originally requesting peer to forward to
+        if (requestingPeer == null) {
+            return;
+        }
+
+        // Return if we're not in relay mode or if this request doesn't need forwarding
+        if (!Settings.getInstance().isRelayModeEnabled()) {
+            return;
+        }
+
+        LOGGER.debug("Received arbitrary data file - forwarding is needed");
+
+        // The ID needs to match that of the original request
+        message.setId(originalMessage.getId());
+
+        //if (!requestingPeer.sendMessageWithTimeout(message, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
+        //    LOGGER.debug("Failed to forward arbitrary data file to peer {}", requestingPeer);
+        //    requestingPeer.disconnect("failed to forward arbitrary data file");
+        //}
+        //else {
+        //    LOGGER.debug("Forwarded arbitrary data file to peer {}", requestingPeer);
+        //}
+        requestingPeer.sendMessageWithTimeout(message, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
+    }
+
+
+    // Fetch data directly from peers
+
+    private List<RNSArbitraryDirectConnectionInfo> getDirectConnectionInfoForSignature(byte[] signature) {
+        synchronized (directConnectionInfo) {
+            return directConnectionInfo.stream().filter(i -> Arrays.equals(i.getSignature(), signature)).collect(Collectors.toList());
+        }
+    }
+
+    /**
+     * Add an ArbitraryDirectConnectionInfo item, but only if one with this peer-signature combination
+     * doesn't already exist.
+     * @param connectionInfo - the direct connection info to add
+     */
+    public void addDirectConnectionInfoIfUnique(RNSArbitraryDirectConnectionInfo connectionInfo) {
+        boolean peerAlreadyExists;
+        synchronized (directConnectionInfo) {
+            peerAlreadyExists = directConnectionInfo.stream()
+                    .anyMatch(i -> Arrays.equals(i.getSignature(), connectionInfo.getSignature())
+                            && Objects.equals(i.getPeerAddress(), connectionInfo.getPeerAddress()));
+        }
+        if (!peerAlreadyExists) {
+            directConnectionInfo.add(connectionInfo);
+        }
+    }
+
+    private void removeDirectConnectionInfo(RNSArbitraryDirectConnectionInfo connectionInfo) {
+        this.directConnectionInfo.remove(connectionInfo);
+    }
+
+    public boolean fetchDataFilesFromPeersForSignature(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+
+        boolean success = false;
+
+        try {
+            while (!success) {
+                if (isStopping) {
+                    return false;
+                }
+                Thread.sleep(500L);
+
+                // Firstly fetch peers that claim to be hosting files for this signature
+                List<RNSArbitraryDirectConnectionInfo> connectionInfoList = getDirectConnectionInfoForSignature(signature);
+                if (connectionInfoList == null || connectionInfoList.isEmpty()) {
+                    LOGGER.debug("No remaining direct connection peers found for signature {}", signature58);
+                    return false;
+                }
+
+                LOGGER.debug("Attempting a direct peer connection for signature {}...", signature58);
+
+                // Peers found, so pick one with the highest number of chunks
+                Comparator<RNSArbitraryDirectConnectionInfo> highestChunkCountFirstComparator =
+                        Comparator.comparingInt(RNSArbitraryDirectConnectionInfo::getHashCount).reversed();
+                RNSArbitraryDirectConnectionInfo directConnectionInfo = connectionInfoList.stream()
+                        .sorted(highestChunkCountFirstComparator).findFirst().orElse(null);
+
+                if (directConnectionInfo == null) {
+                    return false;
+                }
+
+                // Remove from the list so that a different peer is tried next time
+                removeDirectConnectionInfo(directConnectionInfo);
+
+                //// TODO - rework this section (RNS network address?)
+                //String peerAddressString = directConnectionInfo.getPeerAddress();
+                //
+                //// Parse the peer address to find the host and port
+                //String host = null;
+                //int port = -1;
+                //String[] parts = peerAddressString.split(":");
+                //if (parts.length > 1) {
+                //    host = parts[0];
+                //    port = Integer.parseInt(parts[1]);
+                //} else {
+                //    // Assume no port included
+                //    host = peerAddressString;
+                //    // Use default listen port
+                //    port = Settings.getInstance().getDefaultListenPort();
+                //}
+                //
+                //String peerAddressStringWithPort = String.format("%s:%d", host, port);
+                //success = Network.getInstance().requestDataFromPeer(peerAddressStringWithPort, signature);
+                //
+                //int defaultPort = Settings.getInstance().getDefaultListenPort();
+                //
+                //// If unsuccessful, and using a non-standard port, try a second connection with the default listen port,
+                //// since almost all nodes use that. This is a workaround to account for any ephemeral ports that may
+                //// have made it into the dataset.
+                //if (!success) {
+                //    if (host != null && port > 0) {
+                //        if (port != defaultPort) {
+                //            String newPeerAddressString = String.format("%s:%d", host, defaultPort);
+                //            success = Network.getInstance().requestDataFromPeer(newPeerAddressString, signature);
+                //        }
+                //    }
+                //}
+                //
+                //// If _still_ unsuccessful, try matching the peer's IP address with some known peers, and then connect
+                //// to each of those in turn until one succeeds.
+                //if (!success) {
+                //    if (host != null) {
+                //        final String finalHost = host;
+                //        List<PeerData> knownPeers = Network.getInstance().getAllKnownPeers().stream()
+                //                .filter(knownPeerData -> knownPeerData.getAddress().getHost().equals(finalHost))
+                //                .collect(Collectors.toList());
+                //        // Loop through each match and attempt a connection
+                //        for (PeerData matchingPeer : knownPeers) {
+                //            String matchingPeerAddress = matchingPeer.getAddress().toString();
+                //            int matchingPeerPort = matchingPeer.getAddress().getPort();
+                //            // Make sure that it's not a port we've already tried
+                //            if (matchingPeerPort != port && matchingPeerPort != defaultPort) {
+                //                success = Network.getInstance().requestDataFromPeer(matchingPeerAddress, signature);
+                //                if (success) {
+                //                    // Successfully connected, so stop making connections
+                //                    break;
+                //                }
+                //            }
+                //        }
+                //    }
+                //}
+
+                if (success) {
+                    // We were able to connect with a peer, so track the request
+                    RNSArbitraryDataFileListManager.getInstance().addToSignatureRequests(signature58, false, true);
+                }
+
+            }
+        } catch (InterruptedException e) {
+            // Do nothing
+        }
+
+        return success;
+    }
+
+
+    // Relays
+
+    private List<RNSArbitraryRelayInfo> getRelayInfoListForHash(String hash58) {
+        synchronized (arbitraryRelayMap) {
+            return arbitraryRelayMap.stream()
+                    .filter(relayInfo -> Objects.equals(relayInfo.getHash58(), hash58))
+                    .collect(Collectors.toList());
+        }
+    }
+
+    private RNSArbitraryRelayInfo getOptimalRelayInfoEntryForHash(String hash58) {
+        LOGGER.trace("Fetching relay info for hash: {}", hash58);
+        List<RNSArbitraryRelayInfo> relayInfoList = this.getRelayInfoListForHash(hash58);
+        if (relayInfoList != null && !relayInfoList.isEmpty()) {
+
+            // Remove any with null requestHops
+            relayInfoList.removeIf(r -> r.getRequestHops() == null);
+
+            // If list is now empty, then just return one at random
+            if (relayInfoList.isEmpty()) {
+                return this.getRandomRelayInfoEntryForHash(hash58);
+            }
+
+            // Sort by number of hops (lowest first)
+            relayInfoList.sort(Comparator.comparingInt(RNSArbitraryRelayInfo::getRequestHops));
+
+            // FUTURE: secondary sort by requestTime?
+
+            RNSArbitraryRelayInfo relayInfo = relayInfoList.get(0);
+
+            LOGGER.trace("Returning optimal relay info for hash: {} (requestHops {})", hash58, relayInfo.getRequestHops());
+            return relayInfo;
+        }
+        LOGGER.trace("No relay info exists for hash: {}", hash58);
+        return null;
+    }
+
+    private RNSArbitraryRelayInfo getRandomRelayInfoEntryForHash(String hash58) {
+        LOGGER.trace("Fetching random relay info for hash: {}", hash58);
+        List<RNSArbitraryRelayInfo> relayInfoList = this.getRelayInfoListForHash(hash58);
+        if (relayInfoList != null && !relayInfoList.isEmpty()) {
+
+            // Pick random item
+            int index = new SecureRandom().nextInt(relayInfoList.size());
+            LOGGER.trace("Returning random relay info for hash: {} (index {})", hash58, index);
+            return relayInfoList.get(index);
+        }
+        LOGGER.trace("No relay info exists for hash: {}", hash58);
+        return null;
+    }
+
+    public void addToRelayMap(RNSArbitraryRelayInfo newEntry) {
+        if (newEntry == null || !newEntry.isValid()) {
+            return;
+        }
+
+        // Remove existing entry for this peer if it exists, to renew the timestamp
+        this.removeFromRelayMap(newEntry);
+
+        // Re-add
+        arbitraryRelayMap.add(newEntry);
+        LOGGER.debug("Added entry to relay map: {}", newEntry);
+    }
+
+    private void removeFromRelayMap(RNSArbitraryRelayInfo entry) {
+        arbitraryRelayMap.removeIf(relayInfo -> relayInfo.equals(entry));
+    }
+
+
+    // Peers requesting QDN data from us
+
+    /**
+     * Add an address string of a peer that is trying to request data from us.
+     * @param peerAddress
+     */
+    public void addRecentDataRequest(String peerAddress) {
+        if (peerAddress == null) {
+            return;
+        }
+
+        Long now = NTP.getTime();
+        if (now == null) {
+            return;
+        }
+
+        // Make sure to remove the port, since it isn't guaranteed to match next time
+        String[] parts = peerAddress.split(":");
+        if (parts.length == 0) {
+            return;
+        }
+        String host = parts[0];
+        if (!InetAddresses.isInetAddress(host)) {
+            // Invalid host
+            return;
+        }
+
+        this.recentDataRequests.put(host, now);
+    }
+
+    public boolean isPeerRequestingData(String peerAddressWithoutPort) {
+        return this.recentDataRequests.containsKey(peerAddressWithoutPort);
+    }
+
+    public boolean hasPendingDataRequest() {
+        return !this.recentDataRequests.isEmpty();
+    }
+
+
+    // Network handlers
+
+    public void onNetworkGetArbitraryDataFileMessage(RNSPeer peer, Message message) {
+        // Don't respond if QDN is disabled
+        if (!Settings.getInstance().isQdnEnabled()) {
+            return;
+        }
+
+        GetArbitraryDataFileMessage getArbitraryDataFileMessage = (GetArbitraryDataFileMessage) message;
+        byte[] hash = getArbitraryDataFileMessage.getHash();
+        String hash58 = Base58.encode(hash);
+        byte[] signature = getArbitraryDataFileMessage.getSignature();
+        Controller.getInstance().stats.getArbitraryDataFileMessageStats.requests.incrementAndGet();
+
+        LOGGER.debug("Received GetArbitraryDataFileMessage from peer {} for hash {}", peer, Base58.encode(hash));
+
+        try {
+            ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
+            RNSArbitraryRelayInfo relayInfo = this.getOptimalRelayInfoEntryForHash(hash58);
+
+            if (arbitraryDataFile.exists()) {
+                LOGGER.trace("Hash {} exists", hash58);
+
+                // We can serve the file directly as we already have it
+                LOGGER.debug("Sending file {}...", arbitraryDataFile);
+                ArbitraryDataFileMessage arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, arbitraryDataFile);
+                arbitraryDataFileMessage.setId(message.getId());
+                //if (!peer.sendMessageWithTimeout(arbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
+                //    LOGGER.debug("Couldn't send file {}", arbitraryDataFile);
+                //    peer.disconnect("failed to send file");
+                //}
+                //else {
+                //    LOGGER.debug("Sent file {}", arbitraryDataFile);
+                //}
+                peer.sendMessageWithTimeout(arbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT);
+            }
+            //// TODO: rework (doesn't work with Reticulum)
+            //else if (relayInfo != null) {
+            //    LOGGER.debug("We have relay info for hash {}", Base58.encode(hash));
+            //    // We need to ask this peer for the file
+            //    Peer peerToAsk = relayInfo.getPeer();
+            //    if (peerToAsk != null) {
+            //
+            //        // Forward the message to this peer
+            //        LOGGER.debug("Asking peer {} for hash {}", peerToAsk, hash58);
+            //        // No need to pass arbitraryTransactionData below because this is only used for metadata caching,
+            //        // and metadata isn't retained when relaying.
+            //        this.fetchArbitraryDataFile(peerToAsk, peer, null, signature, hash, message);
+            //    }
+            //    else {
+            //        LOGGER.debug("Peer {} not found in relay info", peer);
+            //    }
+            //}
+            else {
+                LOGGER.debug("Hash {} doesn't exist and we don't have relay info", hash58);
+
+                // We don't have this file
+                Controller.getInstance().stats.getArbitraryDataFileMessageStats.unknownFiles.getAndIncrement();
+
+                // Send valid, yet unexpected message type in response, so peer's synchronizer doesn't have to wait for timeout
+                LOGGER.debug(String.format("Sending 'file unknown' response to peer %s for GET_FILE request for unknown file %s", peer, arbitraryDataFile));
+
+                //// Send generic 'unknown' message as it's very short
+                //Message fileUnknownMessage = peer.getPeersVersion() >= GenericUnknownMessage.MINIMUM_PEER_VERSION
+                //        ? new GenericUnknownMessage()
+                //        : new BlockSummariesMessage(Collections.emptyList());
+                //fileUnknownMessage.setId(message.getId());
+                //if (!peer.sendMessage(fileUnknownMessage)) {
+                //    LOGGER.debug("Couldn't sent file-unknown response");
+                //    peer.disconnect("failed to send file-unknown response");
+                //}
+                //else {
+                //    LOGGER.debug("Sent file-unknown response for file {}", arbitraryDataFile);
+                //}
+                Message fileUnknownMessage = new GenericUnknownMessage();
+                peer.sendMessage(fileUnknownMessage);
+            }
+        }
+        catch (DataException e) {
+            LOGGER.debug("Unable to handle request for arbitrary data file: {}", hash58);
+        }
+    }
+
+}

src/main/java/org/qortal/controller/arbitrary/RNSArbitraryDataFileRequestThread.java

@@ -0,0 +1,189 @@
+package org.qortal.controller.arbitrary;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.controller.Controller;
+import org.qortal.data.arbitrary.RNSArbitraryFileListResponseInfo;
+import org.qortal.data.arbitrary.ArbitraryResourceData;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.MessageType;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.utils.ArbitraryTransactionUtils;
+import org.qortal.utils.Base58;
+import org.qortal.utils.NTP;
+import org.qortal.utils.NamedThreadFactory;
+
+import java.net.http.HttpResponse;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import static java.lang.Thread.NORM_PRIORITY;
+
+public class RNSArbitraryDataFileRequestThread {
+
+    private static final Logger LOGGER = LogManager.getLogger(RNSArbitraryDataFileRequestThread.class);
+
+    private static final Integer FETCHER_LIMIT_PER_PEER = Settings.getInstance().getMaxThreadsForMessageType(MessageType.GET_ARBITRARY_DATA_FILE);
+    private static final String FETCHER_THREAD_PREFIX = "Arbitrary Data Fetcher ";
+
+    private ConcurrentHashMap<String, ExecutorService> executorByPeer = new ConcurrentHashMap<>();
+
+    private RNSArbitraryDataFileRequestThread() {
+        cleanupExecutorByPeerScheduler.scheduleAtFixedRate(this::cleanupExecutorsByPeer, 1, 1, TimeUnit.MINUTES);
+    }
+
+    private static RNSArbitraryDataFileRequestThread instance = null;
+
+    public static RNSArbitraryDataFileRequestThread getInstance() {
+
+        if( instance == null ) {
+            instance = new RNSArbitraryDataFileRequestThread();
+        }
+
+        return instance;
+    }
+
+    private final ScheduledExecutorService cleanupExecutorByPeerScheduler = Executors.newScheduledThreadPool(1);
+
+    private void cleanupExecutorsByPeer() {
+
+        try {
+            this.executorByPeer.forEach((key, value) -> {
+                if (value instanceof ThreadPoolExecutor) {
+                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) value;
+                    if (threadPoolExecutor.getActiveCount() == 0) {
+                        threadPoolExecutor.shutdown();
+                        if (this.executorByPeer.computeIfPresent(key, (k, v) -> null) == null) {
+                            LOGGER.trace("removed executor: peer = " + key);
+                        }
+                    }
+                } else {
+                    LOGGER.warn("casting issue in cleanup");
+                }
+            });
+        } catch (Exception e) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+
+    public void processFileHashes(Long now, List<RNSArbitraryFileListResponseInfo> responseInfos, RNSArbitraryDataFileManager arbitraryDataFileManager) {
+		if (Controller.isStopping()) {
+            return;
+        }
+
+        Map<String, byte[]> signatureBySignature58 = new HashMap<>(responseInfos.size());
+        Map<String, List<RNSArbitraryFileListResponseInfo>> responseInfoBySignature58 = new HashMap<>();
+
+        for( RNSArbitraryFileListResponseInfo responseInfo : responseInfos) {
+
+            if( responseInfo == null ) continue;
+
+            if (Controller.isStopping()) {
+                return;
+            }
+
+            RNSPeer peer = responseInfo.getPeer();
+
+            // if relay timeout, then move on
+            if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT || responseInfo.getSignature58() == null || peer == null) {
+                continue;
+            }
+
+            // Skip if already requesting, but don't remove, as we might want to retry later
+            if (arbitraryDataFileManager.arbitraryDataFileRequests.containsKey(responseInfo.getHash58())) {
+                // Already requesting - leave this attempt for later
+                arbitraryDataFileManager.addResponse(responseInfo); // don't remove -> adding back, beacause it was removed already above
+                continue;
+            }
+
+
+            byte[] hash = Base58.decode(responseInfo.getHash58());
+            byte[] signature = Base58.decode(responseInfo.getSignature58());
+
+            // check for null
+            if (signature == null || hash == null || peer == null) {
+                continue;
+            }
+
+            // We want to process this file, store and map data to process later
+            signatureBySignature58.put(responseInfo.getSignature58(), signature);
+            responseInfoBySignature58
+                    .computeIfAbsent(responseInfo.getSignature58(), signature58 -> new ArrayList<>())
+                    .add(responseInfo);
+        }
+
+        // if there are no signatures, then there is nothing to process and nothing query the database
+        if( signatureBySignature58.isEmpty() ) return;
+
+        List<ArbitraryTransactionData> arbitraryTransactionDataList = new ArrayList<>();
+
+        // Fetch the transaction data
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            arbitraryTransactionDataList.addAll(
+                ArbitraryTransactionUtils.fetchTransactionDataList(repository, new ArrayList<>(signatureBySignature58.values())));
+        } catch (DataException e) {
+            LOGGER.warn("Unable to fetch transaction data: {}", e.getMessage());
+        }
+
+        if( !arbitraryTransactionDataList.isEmpty() ) {
+            long start = System.currentTimeMillis();
+
+            for(ArbitraryTransactionData data : arbitraryTransactionDataList ) {
+                String signature58 = Base58.encode(data.getSignature());
+                for( RNSArbitraryFileListResponseInfo responseInfo : responseInfoBySignature58.get(signature58)) {
+                    Runnable fetcher = () -> arbitraryDataFileFetcher(arbitraryDataFileManager, responseInfo, data);
+                    this.executorByPeer
+                            .computeIfAbsent(
+                                responseInfo.getPeer().toString(),
+                                peer -> Executors.newFixedThreadPool(
+                                    FETCHER_LIMIT_PER_PEER,
+                                    new NamedThreadFactory(FETCHER_THREAD_PREFIX + responseInfo.getPeer().toString(), NORM_PRIORITY)
+                                )
+                            )
+                            .execute(fetcher);
+                }
+            }
+            long timeLapse = System.currentTimeMillis() - start;
+        }
+    }
+
+    private void arbitraryDataFileFetcher(RNSArbitraryDataFileManager arbitraryDataFileManager, RNSArbitraryFileListResponseInfo responseInfo, ArbitraryTransactionData arbitraryTransactionData)  {
+        try {
+            Long now = NTP.getTime();
+
+            if (now - responseInfo.getTimestamp() >= ArbitraryDataManager.ARBITRARY_RELAY_TIMEOUT ) {
+
+                RNSPeer peer = responseInfo.getPeer();
+                String hash58 = responseInfo.getHash58();
+                String signature58 = responseInfo.getSignature58();
+                LOGGER.debug("Peer {} version {} didn't fetch data file {} for signature {} due to relay timeout.", peer, hash58, signature58);
+                return;
+            }
+
+            arbitraryDataFileManager.fetchArbitraryDataFiles(
+                responseInfo.getPeer(),
+                arbitraryTransactionData.getSignature(),
+                arbitraryTransactionData,
+                Arrays.asList(Base58.decode(responseInfo.getHash58()))
+            );
+        } catch (DataException e) {
+            LOGGER.warn("Unable to process file hashes: {}", e.getMessage());
+        }
+    }
+}

src/main/java/org/qortal/controller/arbitrary/RNSArbitraryMetadataManager.java

@@ -0,0 +1,481 @@
+package org.qortal.controller.arbitrary;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.arbitrary.ArbitraryDataFile;
+import org.qortal.arbitrary.ArbitraryDataResource;
+import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
+import org.qortal.controller.Controller;
+import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.network.RNSNetwork;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.ArbitraryMetadataMessage;
+import org.qortal.network.message.GetArbitraryMetadataMessage;
+import org.qortal.network.message.Message;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
+import org.qortal.utils.NTP;
+import org.qortal.utils.Triple;
+
+import java.io.IOException;
+import java.util.*;
+
+import static org.qortal.controller.arbitrary.ArbitraryDataFileListManager.*;
+
+public class RNSArbitraryMetadataManager {
+
+    private static final Logger LOGGER = LogManager.getLogger(ArbitraryMetadataManager.class);
+
+    private static RNSArbitraryMetadataManager instance;
+
+    /**
+     * Map of recent incoming requests for ARBITRARY transaction metadata.
+     * <p>
+     * Key is original request's message ID<br>
+     * Value is Triple&lt;transaction signature in base58, first requesting peer, first request's timestamp&gt;
+     * <p>
+     * If peer is null then either:<br>
+     * <ul>
+     * <li>we are the original requesting peer</li>
+     * <li>we have already sent data payload to original requesting peer.</li>
+     * </ul>
+     * If signature is null then we have already received the file list and either:<br>
+     * <ul>
+     * <li>we are the original requesting peer and have processed it</li>
+     * <li>we have forwarded the metadata</li>
+     * </ul>
+     */
+    public Map<Integer, Triple<String, RNSPeer, Long>> arbitraryMetadataRequests = Collections.synchronizedMap(new HashMap<>());
+
+    /**
+     * Map to keep track of in progress arbitrary metadata requests
+     * Key: string - the signature encoded in base58
+     * Value: Triple<networkBroadcastCount, directPeerRequestCount, lastAttemptTimestamp>
+     */
+    private Map<String, Triple<Integer, Integer, Long>> arbitraryMetadataSignatureRequests = Collections.synchronizedMap(new HashMap<>());
+
+
+    private RNSArbitraryMetadataManager() {
+    }
+
+    public static RNSArbitraryMetadataManager getInstance() {
+        if (instance == null)
+            instance = new RNSArbitraryMetadataManager();
+
+        return instance;
+    }
+
+    public void cleanupRequestCache(Long now) {
+        if (now == null) {
+            return;
+        }
+        final long requestMinimumTimestamp = now - ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT;
+        arbitraryMetadataRequests.entrySet().removeIf(entry -> entry.getValue().getC() == null || entry.getValue().getC() < requestMinimumTimestamp);
+    }
+
+
+    public ArbitraryDataTransactionMetadata fetchMetadata(ArbitraryDataResource arbitraryDataResource, boolean useRateLimiter) {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Find latest transaction
+            ArbitraryTransactionData latestTransaction = repository.getArbitraryRepository()
+                    .getLatestTransaction(arbitraryDataResource.getResourceId(), arbitraryDataResource.getService(),
+                            null, arbitraryDataResource.getIdentifier());
+
+            if (latestTransaction != null) {
+                byte[] signature = latestTransaction.getSignature();
+                byte[] metadataHash = latestTransaction.getMetadataHash();
+                if (metadataHash == null) {
+                    // This resource doesn't have metadata
+                    throw new IllegalArgumentException("This resource doesn't have metadata");
+                }
+
+                ArbitraryDataFile metadataFile = ArbitraryDataFile.fromHash(metadataHash, signature);
+                if (!metadataFile.exists()) {
+                    // Request from network
+                    this.fetchArbitraryMetadata(latestTransaction, useRateLimiter);
+                }
+
+                // Now check again as it may have been downloaded above
+                if (metadataFile.exists()) {
+                    // Use local copy
+                    ArbitraryDataTransactionMetadata transactionMetadata = new ArbitraryDataTransactionMetadata(metadataFile.getFilePath());
+                    try {
+                        transactionMetadata.read();
+                    } catch (DataException e) {
+                        // Invalid file, so delete it
+                        LOGGER.info("Deleting invalid metadata file due to exception: {}", e.getMessage());
+                        transactionMetadata.delete();
+                        return null;
+                    }
+                    return transactionMetadata;
+                }
+            }
+
+        } catch (DataException | IOException e) {
+            LOGGER.error("Repository issue when fetching arbitrary transaction metadata", e);
+        }
+
+        return null;
+    }
+
+
+    // Request metadata from network
+
+    public byte[] fetchArbitraryMetadata(ArbitraryTransactionData arbitraryTransactionData, boolean useRateLimiter) {
+        byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
+        if (metadataHash == null) {
+            return null;
+        }
+
+        byte[] signature = arbitraryTransactionData.getSignature();
+        String signature58 = Base58.encode(signature);
+
+        // Require an NTP sync
+        Long now = NTP.getTime();
+        if (now == null) {
+            return null;
+        }
+
+        // If we've already tried too many times in a short space of time, make sure to give up
+        if (useRateLimiter && !this.shouldMakeMetadataRequestForSignature(signature58)) {
+            LOGGER.trace("Skipping metadata request for signature {} due to rate limit", signature58);
+            return null;
+        }
+        this.addToSignatureRequests(signature58, true, false);
+
+        //List<Peer> handshakedPeers = Network.getInstance().getImmutableHandshakedPeers();
+        List<RNSPeer> handshakedPeers = RNSNetwork.getInstance().getLinkedPeers();
+        LOGGER.debug(String.format("Sending metadata request for signature %s to %d peers...", signature58, handshakedPeers.size()));
+
+        // Build request
+        Message getArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, now, 0);
+
+        // Save our request into requests map
+        Triple<String, RNSPeer, Long> requestEntry = new Triple<>(signature58, null, NTP.getTime());
+
+        // Assign random ID to this message
+        int id;
+        do {
+            id = new Random().nextInt(Integer.MAX_VALUE - 1) + 1;
+
+            // Put queue into map (keyed by message ID) so we can poll for a response
+            // If putIfAbsent() doesn't return null, then this ID is already taken
+        } while (arbitraryMetadataRequests.put(id, requestEntry) != null);
+        getArbitraryMetadataMessage.setId(id);
+
+        // Broadcast request
+        RNSNetwork.getInstance().broadcast(peer -> getArbitraryMetadataMessage);
+
+        // Poll to see if data has arrived
+        final long singleWait = 100;
+        long totalWait = 0;
+        while (totalWait < ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT) {
+            try {
+                Thread.sleep(singleWait);
+            } catch (InterruptedException e) {
+                break;
+            }
+
+            requestEntry = arbitraryMetadataRequests.get(id);
+            if (requestEntry == null)
+                return null;
+
+            if (requestEntry.getA() == null)
+                break;
+
+            totalWait += singleWait;
+        }
+
+        try {
+            ArbitraryDataFile metadataFile = ArbitraryDataFile.fromHash(metadataHash, signature);
+            if (metadataFile.exists()) {
+                return metadataFile.getBytes();
+            }
+        } catch (DataException e) {
+            // Do nothing
+        }
+
+        return null;
+    }
+
+
+    // Track metadata lookups by signature
+
+    private boolean shouldMakeMetadataRequestForSignature(String signature58) {
+        Triple<Integer, Integer, Long> request = arbitraryMetadataSignatureRequests.get(signature58);
+
+        if (request == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        // Extract the components
+        Integer networkBroadcastCount = request.getA();
+        // Integer directPeerRequestCount = request.getB();
+        Long lastAttemptTimestamp = request.getC();
+
+        if (lastAttemptTimestamp == null) {
+            // Not attempted yet
+            return true;
+        }
+
+        long timeSinceLastAttempt = NTP.getTime() - lastAttemptTimestamp;
+
+        // Allow a second attempt after 60 seconds
+        if (timeSinceLastAttempt > 60 * 1000L) {
+            // We haven't tried for at least 60 seconds
+
+            if (networkBroadcastCount < 2) {
+                // We've made less than 2 total attempts
+                return true;
+            }
+        }
+
+        // Then allow another attempt after 60 minutes
+        if (timeSinceLastAttempt > 60 * 60 * 1000L) {
+            // We haven't tried for at least 60 minutes
+
+            if (networkBroadcastCount < 3) {
+                // We've made less than 3 total attempts
+                return true;
+            }
+        }
+
+        return false;
+    }
+
+    public boolean isSignatureRateLimited(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+        return !this.shouldMakeMetadataRequestForSignature(signature58);
+    }
+
+    public long lastRequestForSignature(byte[] signature) {
+        String signature58 = Base58.encode(signature);
+        Triple<Integer, Integer, Long> request = arbitraryMetadataSignatureRequests.get(signature58);
+
+        if (request == null) {
+            // Not attempted yet
+            return 0;
+        }
+
+        // Extract the components
+        Long lastAttemptTimestamp = request.getC();
+        if (lastAttemptTimestamp != null) {
+            return  lastAttemptTimestamp;
+        }
+        return 0;
+    }
+
+    public void addToSignatureRequests(String signature58, boolean incrementNetworkRequests, boolean incrementPeerRequests) {
+        Triple<Integer, Integer, Long> request  = arbitraryMetadataSignatureRequests.get(signature58);
+        Long now = NTP.getTime();
+
+        if (request == null) {
+            // No entry yet
+            Triple<Integer, Integer, Long> newRequest = new Triple<>(0, 0, now);
+            arbitraryMetadataSignatureRequests.put(signature58, newRequest);
+        }
+        else {
+            // There is an existing entry
+            if (incrementNetworkRequests) {
+                request.setA(request.getA() + 1);
+            }
+            if (incrementPeerRequests) {
+                request.setB(request.getB() + 1);
+            }
+            request.setC(now);
+            arbitraryMetadataSignatureRequests.put(signature58, request);
+        }
+    }
+
+    public void removeFromSignatureRequests(String signature58) {
+        arbitraryMetadataSignatureRequests.remove(signature58);
+    }
+
+
+    // Network handlers
+
+    public void onNetworkArbitraryMetadataMessage(RNSPeer peer, Message message) {
+        // Don't process if QDN is disabled
+        if (!Settings.getInstance().isQdnEnabled()) {
+            return;
+        }
+
+        ArbitraryMetadataMessage arbitraryMetadataMessage = (ArbitraryMetadataMessage) message;
+        LOGGER.debug("Received metadata from peer {}", peer);
+
+        // Do we have a pending request for this data?
+        Triple<String, RNSPeer, Long> request = arbitraryMetadataRequests.get(message.getId());
+        if (request == null || request.getA() == null) {
+            return;
+        }
+        boolean isRelayRequest = (request.getB() != null);
+
+        // Does this message's signature match what we're expecting?
+        byte[] signature = arbitraryMetadataMessage.getSignature();
+        String signature58 = Base58.encode(signature);
+        if (!request.getA().equals(signature58)) {
+            return;
+        }
+
+        // Update requests map to reflect that we've received this metadata
+        Triple<String, RNSPeer, Long> newEntry = new Triple<>(null, null, request.getC());
+        arbitraryMetadataRequests.put(message.getId(), newEntry);
+
+        // Get transaction info
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            if (!(transactionData instanceof ArbitraryTransactionData)) {
+                return;
+            }
+            ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
+
+            // Check if the name is blocked
+            boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
+
+            // Save if not blocked
+            ArbitraryDataFile arbitraryMetadataFile = arbitraryMetadataMessage.getArbitraryMetadataFile();
+            if (!isBlocked && arbitraryMetadataFile != null) {
+                arbitraryMetadataFile.save();
+            }
+
+            // Forwarding
+            if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
+                if (!isBlocked) {
+                    RNSPeer requestingPeer = request.getB();
+                    if (requestingPeer != null) {
+
+                        ArbitraryMetadataMessage forwardArbitraryMetadataMessage = new ArbitraryMetadataMessage(signature, arbitraryMetadataMessage.getArbitraryMetadataFile());
+                        forwardArbitraryMetadataMessage.setId(arbitraryMetadataMessage.getId());
+
+                        // Forward to requesting peer
+                        LOGGER.debug("Forwarding metadata to requesting peer: {}", requestingPeer);
+                        //if (!requestingPeer.sendMessage(forwardArbitraryMetadataMessage)) {
+                        //    requestingPeer.disconnect("failed to forward arbitrary metadata");
+                        //}
+                        requestingPeer.sendMessage(forwardArbitraryMetadataMessage);
+                    }
+                }
+            }
+
+            // Add to resource queue to update arbitrary resource caches
+            if (arbitraryTransactionData != null) {
+                ArbitraryDataCacheManager.getInstance().addToUpdateQueue(arbitraryTransactionData);
+            }
+
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while saving arbitrary transaction metadata from peer %s", peer), e);
+        }
+    }
+
+    public void onNetworkGetArbitraryMetadataMessage(RNSPeer peer, Message message) {
+        // Don't respond if QDN is disabled
+        if (!Settings.getInstance().isQdnEnabled()) {
+            return;
+        }
+
+        Controller.getInstance().stats.getArbitraryMetadataMessageStats.requests.incrementAndGet();
+
+        GetArbitraryMetadataMessage getArbitraryMetadataMessage = (GetArbitraryMetadataMessage) message;
+        byte[] signature = getArbitraryMetadataMessage.getSignature();
+        String signature58 = Base58.encode(signature);
+        Long now = NTP.getTime();
+        Triple<String, RNSPeer, Long> newEntry = new Triple<>(signature58, peer, now);
+
+        // If we've seen this request recently, then ignore
+        if (arbitraryMetadataRequests.putIfAbsent(message.getId(), newEntry) != null) {
+            LOGGER.debug("Ignoring metadata request from peer {} for signature {}", peer, signature58);
+            return;
+        }
+
+        LOGGER.debug("Received metadata request from peer {} for signature {}", peer, signature58);
+
+        ArbitraryTransactionData transactionData = null;
+        ArbitraryDataFile metadataFile = null;
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+
+            // Firstly we need to lookup this file on chain to get its metadata hash
+            transactionData = (ArbitraryTransactionData)repository.getTransactionRepository().fromSignature(signature);
+            if (transactionData instanceof ArbitraryTransactionData) {
+
+                // Check if we're even allowed to serve metadata for this transaction
+                if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
+
+                    byte[] metadataHash = transactionData.getMetadataHash();
+                    if (metadataHash != null) {
+
+                        // Load metadata file
+                        metadataFile = ArbitraryDataFile.fromHash(metadataHash, signature);
+                    }
+                }
+            }
+
+        } catch (DataException e) {
+            LOGGER.error(String.format("Repository issue while fetching arbitrary metadata for peer %s", peer), e);
+        }
+
+        // We should only respond if we have the metadata file
+        if (metadataFile != null && metadataFile.exists()) {
+
+            // We have the metadata file, so update requests map to reflect that we've sent it
+            newEntry = new Triple<>(null, null, now);
+            arbitraryMetadataRequests.put(message.getId(), newEntry);
+
+            ArbitraryMetadataMessage arbitraryMetadataMessage = new ArbitraryMetadataMessage(signature, metadataFile);
+            arbitraryMetadataMessage.setId(message.getId());
+            //if (!peer.sendMessage(arbitraryMetadataMessage)) {
+            //    LOGGER.debug("Couldn't send metadata");
+            //    peer.disconnect("failed to send metadata");
+            //    return;
+            //}
+            peer.sendMessage(arbitraryMetadataMessage);
+            LOGGER.debug("Sent metadata");
+
+            // Nothing left to do, so return to prevent any unnecessary forwarding from occurring
+            LOGGER.debug("No need for any forwarding because metadata request is fully served");
+            return;
+
+        }
+
+        // We may need to forward this request on
+        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
+        if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
+            // In relay mode - so ask our other peers if they have it
+
+            long requestTime = getArbitraryMetadataMessage.getRequestTime();
+            int requestHops = getArbitraryMetadataMessage.getRequestHops() + 1;
+            long totalRequestTime = now - requestTime;
+
+            if (totalRequestTime < RELAY_REQUEST_MAX_DURATION) {
+                // Relay request hasn't timed out yet, so can potentially be rebroadcast
+                if (requestHops < RELAY_REQUEST_MAX_HOPS) {
+                    // Relay request hasn't reached the maximum number of hops yet, so can be rebroadcast
+
+                    Message relayGetArbitraryMetadataMessage = new GetArbitraryMetadataMessage(signature, requestTime, requestHops);
+                    relayGetArbitraryMetadataMessage.setId(message.getId());
+
+                    LOGGER.debug("Rebroadcasting metadata request from peer {} for signature {} to our other peers... totalRequestTime: {}, requestHops: {}", peer, Base58.encode(signature), totalRequestTime, requestHops);
+                    //Network.getInstance().broadcast(
+                    //        broadcastPeer ->
+                    //                !broadcastPeer.isAtLeastVersion(RELAY_MIN_PEER_VERSION) ? null :
+                    //                broadcastPeer == peer || Objects.equals(broadcastPeer.getPeerData().getAddress().getHost(), peer.getPeerData().getAddress().getHost()) ? null : relayGetArbitraryMetadataMessage);
+                    RNSNetwork.getInstance().broadcast(broadcastPeer -> relayGetArbitraryMetadataMessage);
+
+                }
+                else {
+                    // This relay request has reached the maximum number of allowed hops
+                }
+            }
+            else {
+                // This relay request has timed out
+            }
+        }
+    }
+
+}

src/main/java/org/qortal/controller/arbitrary/RebuildArbitraryResourceCacheTask.java

@@ -0,0 +1,33 @@
+package org.qortal.controller.arbitrary;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+
+import java.util.TimerTask;
+
+public class RebuildArbitraryResourceCacheTask extends TimerTask {
+
+    private static final Logger LOGGER = LogManager.getLogger(RebuildArbitraryResourceCacheTask.class);
+
+    public static final long MILLIS_IN_HOUR = 60 * 60 * 1000;
+
+    public static final long MILLIS_IN_MINUTE = 60 * 1000;
+
+    private static final String REBUILD_ARBITRARY_RESOURCE_CACHE_TASK = "Rebuild Arbitrary Resource Cache Task";
+
+    @Override
+    public void run() {
+
+        Thread.currentThread().setName(REBUILD_ARBITRARY_RESOURCE_CACHE_TASK);
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            ArbitraryDataCacheManager.getInstance().buildArbitraryResourcesCache(repository, true);
+        }
+        catch( DataException e ) {
+            LOGGER.error(e.getMessage(), e);
+        }
+    }
+}

src/main/java/org/qortal/controller/hsqldb/HSQLDBBalanceRecorder.java

@@ -0,0 +1,139 @@
+package org.qortal.controller.hsqldb;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.util.PropertySource;
+import org.qortal.data.account.AccountBalanceData;
+import org.qortal.data.account.BlockHeightRange;
+import org.qortal.data.account.BlockHeightRangeAddressAmounts;
+import org.qortal.repository.hsqldb.HSQLDBCacheUtils;
+import org.qortal.settings.Settings;
+import org.qortal.utils.BalanceRecorderUtils;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+public class HSQLDBBalanceRecorder extends Thread{
+
+    private static final Logger LOGGER = LogManager.getLogger(HSQLDBBalanceRecorder.class);
+
+    private static HSQLDBBalanceRecorder SINGLETON = null;
+
+    private ConcurrentHashMap<Integer, List<AccountBalanceData>> balancesByHeight = new ConcurrentHashMap<>();
+
+    private ConcurrentHashMap<String, List<AccountBalanceData>> balancesByAddress = new ConcurrentHashMap<>();
+
+    private CopyOnWriteArrayList<BlockHeightRangeAddressAmounts> balanceDynamics = new CopyOnWriteArrayList<>();
+
+    private int priorityRequested;
+    private int frequency;
+    private int capacity;
+
+    private HSQLDBBalanceRecorder( int priorityRequested, int frequency, int capacity) {
+
+        super("Balance Recorder");
+
+        this.priorityRequested = priorityRequested;
+        this.frequency = frequency;
+        this.capacity = capacity;
+    }
+
+    public static Optional<HSQLDBBalanceRecorder> getInstance() {
+
+        if( SINGLETON == null ) {
+
+            SINGLETON
+                = new HSQLDBBalanceRecorder(
+                    Settings.getInstance().getBalanceRecorderPriority(),
+                    Settings.getInstance().getBalanceRecorderFrequency(),
+                    Settings.getInstance().getBalanceRecorderCapacity()
+            );
+
+        }
+        else if( SINGLETON == null ) {
+
+            return Optional.empty();
+        }
+
+        return Optional.of(SINGLETON);
+    }
+
+    @Override
+    public void run() {
+
+        Thread.currentThread().setName("Balance Recorder");
+
+        HSQLDBCacheUtils.startRecordingBalances(this.balancesByHeight, this.balanceDynamics, this.priorityRequested, this.frequency, this.capacity);
+    }
+
+    public List<BlockHeightRangeAddressAmounts> getLatestDynamics(int limit, long offset) {
+
+        List<BlockHeightRangeAddressAmounts> latest = this.balanceDynamics.stream()
+                .sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_ADDRESS_AMOUNTS_COMPARATOR.reversed())
+                .skip(offset)
+                .limit(limit)
+                .collect(Collectors.toList());
+
+        return latest;
+    }
+
+    public List<BlockHeightRange> getRanges(Integer offset, Integer limit, Boolean reverse) {
+
+        if( reverse ) {
+            return this.balanceDynamics.stream()
+                    .map(BlockHeightRangeAddressAmounts::getRange)
+                    .sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_COMPARATOR.reversed())
+                    .skip(offset)
+                    .limit(limit)
+                    .collect(Collectors.toList());
+        }
+        else {
+            return this.balanceDynamics.stream()
+                    .map(BlockHeightRangeAddressAmounts::getRange)
+                    .sorted(BalanceRecorderUtils.BLOCK_HEIGHT_RANGE_COMPARATOR)
+                    .skip(offset)
+                    .limit(limit)
+                    .collect(Collectors.toList());
+        }
+    }
+
+    public Optional<BlockHeightRangeAddressAmounts> getAddressAmounts(BlockHeightRange range) {
+
+        return this.balanceDynamics.stream()
+            .filter( dynamic -> dynamic.getRange().equals(range))
+            .findAny();
+    }
+
+    public Optional<BlockHeightRange> getRange( int height ) {
+        return this.balanceDynamics.stream()
+            .map(BlockHeightRangeAddressAmounts::getRange)
+            .filter( range -> range.getBegin() < height && range.getEnd() >= height )
+            .findAny();
+    }
+
+    private Optional<Integer> getLastHeight() {
+        return this.balancesByHeight.keySet().stream().sorted(Comparator.reverseOrder()).findFirst();
+    }
+
+    public List<Integer> getBlocksRecorded() {
+
+        return this.balancesByHeight.keySet().stream().collect(Collectors.toList());
+    }
+
+    public List<AccountBalanceData> getAccountBalanceRecordings(String address) {
+        return this.balancesByAddress.get(address);
+    }
+
+    @Override
+    public String toString() {
+        return "HSQLDBBalanceRecorder{" +
+                "priorityRequested=" + priorityRequested +
+                ", frequency=" + frequency +
+                ", capacity=" + capacity +
+                '}';
+    }
+}

src/main/java/org/qortal/controller/hsqldb/HSQLDBDataCacheManager.java

@@ -0,0 +1,22 @@
+package org.qortal.controller.hsqldb;
+
+import org.qortal.data.arbitrary.ArbitraryResourceCache;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.repository.hsqldb.HSQLDBCacheUtils;
+import org.qortal.repository.hsqldb.HSQLDBRepository;
+import org.qortal.settings.Settings;
+
+public class HSQLDBDataCacheManager extends Thread{
+
+    public HSQLDBDataCacheManager() {}
+
+    @Override
+    public void run() {
+        Thread.currentThread().setName("HSQLDB Data Cache Manager");
+
+        HSQLDBCacheUtils.startCaching(
+            Settings.getInstance().getDbCacheThreadPriority(),
+            Settings.getInstance().getDbCacheFrequency()
+        );
+    }
+}

src/main/java/org/qortal/controller/repository/AtStatesPruner.java

@@ -11,6 +11,8 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.NTP;
 
+import static java.lang.Thread.MIN_PRIORITY;
+
 public class AtStatesPruner implements Runnable {
 
 	private static final Logger LOGGER = LogManager.getLogger(AtStatesPruner.class);
@@ -37,82 +39,97 @@ public class AtStatesPruner implements Runnable {
 			}
 		}
 
+		int pruneStartHeight;
+		int maxLatestAtStatesHeight;
+
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			int pruneStartHeight = repository.getATRepository().getAtPruneHeight();
-			int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
+			pruneStartHeight = repository.getATRepository().getAtPruneHeight();
+			maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
 
 			repository.discardChanges();
 			repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
 			repository.saveChanges();
+		} catch (Exception e) {
+			LOGGER.error("AT States Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			return;
+		}
 
-			while (!Controller.isStopping()) {
-				repository.discardChanges();
+		while (!Controller.isStopping()) {
+			try (final Repository repository = RepositoryManager.getRepository()) {
 
-				Thread.sleep(Settings.getInstance().getAtStatesPruneInterval());
+				try {
+					repository.discardChanges();
 
-				BlockData chainTip = Controller.getInstance().getChainTip();
-				if (chainTip == null || NTP.getTime() == null)
-					continue;
+					Thread.sleep(Settings.getInstance().getAtStatesPruneInterval());
 
-				// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
-				if (Synchronizer.getInstance().isSynchronizing())
-					continue;
+					BlockData chainTip = Controller.getInstance().getChainTip();
+					if (chainTip == null || NTP.getTime() == null)
+						continue;
 
-				// Prune AT states for all blocks up until our latest minus pruneBlockLimit
-				final int ourLatestHeight = chainTip.getHeight();
-				int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
+					// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+					if (Synchronizer.getInstance().isSynchronizing())
+						continue;
 
-				// In archive mode we are only allowed to trim blocks that have already been archived
-				if (archiveMode) {
-					upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
+					// Prune AT states for all blocks up until our latest minus pruneBlockLimit
+					final int ourLatestHeight = chainTip.getHeight();
+					int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
 
-					// TODO: validate that the actual archived data exists before pruning it?
-				}
+					// In archive mode we are only allowed to trim blocks that have already been archived
+					if (archiveMode) {
+						upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
 
-				int upperBatchHeight = pruneStartHeight + Settings.getInstance().getAtStatesPruneBatchSize();
-				int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
+						// TODO: validate that the actual archived data exists before pruning it?
+					}
 
-				if (pruneStartHeight >= upperPruneHeight)
-					continue;
+					int upperBatchHeight = pruneStartHeight + Settings.getInstance().getAtStatesPruneBatchSize();
+					int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
 
-				LOGGER.debug(String.format("Pruning AT states between blocks %d and %d...", pruneStartHeight, upperPruneHeight));
+					if (pruneStartHeight >= upperPruneHeight)
+						continue;
 
-				int numAtStatesPruned = repository.getATRepository().pruneAtStates(pruneStartHeight, upperPruneHeight);
-				repository.saveChanges();
-				int numAtStateDataRowsTrimmed = repository.getATRepository().trimAtStates(
-						pruneStartHeight, upperPruneHeight, Settings.getInstance().getAtStatesTrimLimit());
-				repository.saveChanges();
+					LOGGER.info(String.format("Pruning AT states between blocks %d and %d...", pruneStartHeight, upperPruneHeight));
 
-				if (numAtStatesPruned > 0 || numAtStateDataRowsTrimmed > 0) {
-					final int finalPruneStartHeight = pruneStartHeight;
-					LOGGER.debug(() -> String.format("Pruned %d AT state%s between blocks %d and %d",
-							numAtStatesPruned, (numAtStatesPruned != 1 ? "s" : ""),
-							finalPruneStartHeight, upperPruneHeight));
-				} else {
-					// Can we move onto next batch?
-					if (upperPrunableHeight > upperBatchHeight) {
-						pruneStartHeight = upperBatchHeight;
-						repository.getATRepository().setAtPruneHeight(pruneStartHeight);
-						maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
-						repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
-						repository.saveChanges();
+					int numAtStatesPruned = repository.getATRepository().pruneAtStates(pruneStartHeight, upperPruneHeight);
+					repository.saveChanges();
+					int numAtStateDataRowsTrimmed = repository.getATRepository().trimAtStates(
+							pruneStartHeight, upperPruneHeight, Settings.getInstance().getAtStatesTrimLimit());
+					repository.saveChanges();
 
+					if (numAtStatesPruned > 0 || numAtStateDataRowsTrimmed > 0) {
 						final int finalPruneStartHeight = pruneStartHeight;
-						LOGGER.debug(() -> String.format("Bumping AT state base prune height to %d", finalPruneStartHeight));
+						LOGGER.info(() -> String.format("Pruned %d AT state%s between blocks %d and %d",
+								numAtStatesPruned, (numAtStatesPruned != 1 ? "s" : ""),
+								finalPruneStartHeight, upperPruneHeight));
+					} else {
+						// Can we move onto next batch?
+						if (upperPrunableHeight > upperBatchHeight) {
+							pruneStartHeight = upperBatchHeight;
+							repository.getATRepository().setAtPruneHeight(pruneStartHeight);
+							maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
+							repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
+							repository.saveChanges();
+
+							final int finalPruneStartHeight = pruneStartHeight;
+							LOGGER.info(() -> String.format("Bumping AT state base prune height to %d", finalPruneStartHeight));
+						} else {
+							// We've pruned up to the upper prunable height
+							// Back off for a while to save CPU for syncing
+							repository.discardChanges();
+							Thread.sleep(5 * 60 * 1000L);
+						}
 					}
-					else {
-						// We've pruned up to the upper prunable height
-						// Back off for a while to save CPU for syncing
-						repository.discardChanges();
-						Thread.sleep(5*60*1000L);
+				} catch (InterruptedException e) {
+					if (Controller.isStopping()) {
+						LOGGER.info("AT States Pruning Shutting Down");
+					} else {
+						LOGGER.warn("AT States Pruning interrupted. Trying again. Report this error immediately to the developers.", e);
 					}
+				} catch (Exception e) {
+					LOGGER.warn("AT States Pruning stopped working. Trying again. Report this error immediately to the developers.", e);
 				}
+			} catch(Exception e){
+				LOGGER.error("AT States Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
 			}
-		} catch (DataException e) {
-			LOGGER.warn(String.format("Repository issue trying to prune AT states: %s", e.getMessage()));
-		} catch (InterruptedException e) {
-			// Time to exit
 		}
 	}
-
 }

src/main/java/org/qortal/controller/repository/AtStatesTrimmer.java

@@ -11,6 +11,8 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.NTP;
 
+import static java.lang.Thread.MIN_PRIORITY;
+
 public class AtStatesTrimmer implements Runnable {
 
 	private static final Logger LOGGER = LogManager.getLogger(AtStatesTrimmer.class);
@@ -24,66 +26,83 @@ public class AtStatesTrimmer implements Runnable {
 			return;
 		}
 
+		int trimStartHeight;
+		int maxLatestAtStatesHeight;
+
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			int trimStartHeight = repository.getATRepository().getAtTrimHeight();
-			int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
+			trimStartHeight = repository.getATRepository().getAtTrimHeight();
+			maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
 
 			repository.discardChanges();
 			repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
 			repository.saveChanges();
+		} catch (Exception e) {
+			LOGGER.error("AT States Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			return;
+		}
 
-			while (!Controller.isStopping()) {
-				repository.discardChanges();
+		while (!Controller.isStopping()) {
+			try (final Repository repository = RepositoryManager.getRepository()) {
+				try {
+					repository.discardChanges();
 
-				Thread.sleep(Settings.getInstance().getAtStatesTrimInterval());
+					Thread.sleep(Settings.getInstance().getAtStatesTrimInterval());
 
-				BlockData chainTip = Controller.getInstance().getChainTip();
-				if (chainTip == null || NTP.getTime() == null)
-					continue;
+					BlockData chainTip = Controller.getInstance().getChainTip();
+					if (chainTip == null || NTP.getTime() == null)
+						continue;
 
-				// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
-				if (Synchronizer.getInstance().isSynchronizing())
-					continue;
+					// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+					if (Synchronizer.getInstance().isSynchronizing())
+						continue;
 
-				long currentTrimmableTimestamp = NTP.getTime() - Settings.getInstance().getAtStatesMaxLifetime();
-				// We want to keep AT states near the tip of our copy of blockchain so we can process/orphan nearby blocks
-				long chainTrimmableTimestamp = chainTip.getTimestamp() - Settings.getInstance().getAtStatesMaxLifetime();
+					long currentTrimmableTimestamp = NTP.getTime() - Settings.getInstance().getAtStatesMaxLifetime();
+					// We want to keep AT states near the tip of our copy of blockchain so we can process/orphan nearby blocks
+					long chainTrimmableTimestamp = chainTip.getTimestamp() - Settings.getInstance().getAtStatesMaxLifetime();
 
-				long upperTrimmableTimestamp = Math.min(currentTrimmableTimestamp, chainTrimmableTimestamp);
-				int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
+					long upperTrimmableTimestamp = Math.min(currentTrimmableTimestamp, chainTrimmableTimestamp);
+					int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
 
-				int upperBatchHeight = trimStartHeight + Settings.getInstance().getAtStatesTrimBatchSize();
-				int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
+					int upperBatchHeight = trimStartHeight + Settings.getInstance().getAtStatesTrimBatchSize();
+					int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
 
-				if (trimStartHeight >= upperTrimHeight)
-					continue;
+					if (trimStartHeight >= upperTrimHeight)
+						continue;
 
-				int numAtStatesTrimmed = repository.getATRepository().trimAtStates(trimStartHeight, upperTrimHeight, Settings.getInstance().getAtStatesTrimLimit());
-				repository.saveChanges();
-
-				if (numAtStatesTrimmed > 0) {
-					final int finalTrimStartHeight = trimStartHeight;
-					LOGGER.debug(() -> String.format("Trimmed %d AT state%s between blocks %d and %d",
-							numAtStatesTrimmed, (numAtStatesTrimmed != 1 ? "s" : ""),
-							finalTrimStartHeight, upperTrimHeight));
-				} else {
-					// Can we move onto next batch?
-					if (upperTrimmableHeight > upperBatchHeight) {
-						trimStartHeight = upperBatchHeight;
-						repository.getATRepository().setAtTrimHeight(trimStartHeight);
-						maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
-						repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
-						repository.saveChanges();
+					int numAtStatesTrimmed = repository.getATRepository().trimAtStates(trimStartHeight, upperTrimHeight, Settings.getInstance().getAtStatesTrimLimit());
+					repository.saveChanges();
 
+					if (numAtStatesTrimmed > 0) {
 						final int finalTrimStartHeight = trimStartHeight;
-						LOGGER.debug(() -> String.format("Bumping AT state base trim height to %d", finalTrimStartHeight));
+						LOGGER.info(() -> String.format("Trimmed %d AT state%s between blocks %d and %d",
+								numAtStatesTrimmed, (numAtStatesTrimmed != 1 ? "s" : ""),
+								finalTrimStartHeight, upperTrimHeight));
+					} else {
+						// Can we move onto next batch?
+						if (upperTrimmableHeight > upperBatchHeight) {
+							trimStartHeight = upperBatchHeight;
+							repository.getATRepository().setAtTrimHeight(trimStartHeight);
+							maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
+							repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
+							repository.saveChanges();
+
+							final int finalTrimStartHeight = trimStartHeight;
+							LOGGER.info(() -> String.format("Bumping AT state base trim height to %d", finalTrimStartHeight));
+						}
 					}
+				} catch (InterruptedException e) {
+					if(Controller.isStopping()) {
+						LOGGER.info("AT States Trimming Shutting Down");
+					}
+					else {
+						LOGGER.warn("AT States Trimming interrupted. Trying again. Report this error immediately to the developers.", e);
+					}
+				} catch (Exception e) {
+					LOGGER.warn("AT States Trimming stopped working. Trying again. Report this error immediately to the developers.", e);
 				}
+			} catch (Exception e) {
+				LOGGER.error("AT States Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
 			}
-		} catch (DataException e) {
-			LOGGER.warn(String.format("Repository issue trying to trim AT states: %s", e.getMessage()));
-		} catch (InterruptedException e) {
-			// Time to exit
 		}
 	}
 

src/main/java/org/qortal/controller/repository/BlockArchiver.java

@@ -15,11 +15,13 @@ import org.qortal.utils.NTP;
 
 import java.io.IOException;
 
+import static java.lang.Thread.NORM_PRIORITY;
+
 public class BlockArchiver implements Runnable {
 
 	private static final Logger LOGGER = LogManager.getLogger(BlockArchiver.class);
 
-	private static final long INITIAL_SLEEP_PERIOD = 5 * 60 * 1000L + 1234L; // ms
+	private static final long INITIAL_SLEEP_PERIOD = 15 * 60 * 1000L; // ms
 
 	public void run() {
 		Thread.currentThread().setName("Block archiver");
@@ -28,11 +30,13 @@ public class BlockArchiver implements Runnable {
 			return;
 		}
 
+		int startHeight;
+
 		try (final Repository repository = RepositoryManager.getRepository()) {
 			// Don't even start building until initial rush has ended
 			Thread.sleep(INITIAL_SLEEP_PERIOD);
 
-			int startHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight();
+			startHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight();
 
 			// Don't attempt to archive if we have no ATStatesHeightIndex, as it will be too slow
 			boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex();
@@ -41,77 +45,87 @@ public class BlockArchiver implements Runnable {
 				repository.discardChanges();
 				return;
 			}
-
-			LOGGER.info("Starting block archiver from height {}...", startHeight);
-
-			while (!Controller.isStopping()) {
-				repository.discardChanges();
-
-				Thread.sleep(Settings.getInstance().getArchiveInterval());
-
-				BlockData chainTip = Controller.getInstance().getChainTip();
-				if (chainTip == null || NTP.getTime() == null) {
-					continue;
-				}
-
-				// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
-				if (Synchronizer.getInstance().isSynchronizing()) {
-					continue;
-				}
-
-				// Don't attempt to archive if we're not synced yet
-				final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
-				if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
-					continue;
-				}
-
-
-				// Build cache of blocks
-				try {
-					final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
-					BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, repository);
-					BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
-					switch (result) {
-						case OK:
-							// Increment block archive height
-							startHeight += writer.getWrittenCount();
-							repository.getBlockArchiveRepository().setBlockArchiveHeight(startHeight);
-							repository.saveChanges();
-							break;
-
-						case STOPPING:
-							return;
-
-							// We've reached the limit of the blocks we can archive
-							// Sleep for a while to allow more to become available
-						case NOT_ENOUGH_BLOCKS:
-							// We didn't reach our file size target, so that must mean that we don't have enough blocks
-							// yet or something went wrong. Sleep for a while and then try again.
-							repository.discardChanges();
-							Thread.sleep(60 * 60 * 1000L); // 1 hour
-							break;
-
-						case BLOCK_NOT_FOUND:
-							// We tried to archive a block that didn't exist. This is a major failure and likely means
-							// that a bootstrap or re-sync is needed. Try again every minute until then.
-							LOGGER.info("Error: block not found when building archive. If this error persists, " +
-									"a bootstrap or re-sync may be needed.");
-							repository.discardChanges();
-							Thread.sleep( 60 * 1000L); // 1 minute
-							break;
-					}
-
-				} catch (IOException | TransformationException e) {
-					LOGGER.info("Caught exception when creating block cache", e);
-				}
-
-			}
-		} catch (DataException e) {
-			LOGGER.info("Caught exception when creating block cache", e);
-		} catch (InterruptedException e) {
-			// Do nothing
+		} catch (Exception e) {
+			LOGGER.error("Block Archiving is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			return;
 		}
 
-	}
+		LOGGER.info("Starting block archiver from height {}...", startHeight);
 
+		while (!Controller.isStopping()) {
+			try (final Repository repository = RepositoryManager.getRepository()) {
+
+				try {
+					repository.discardChanges();
+
+					Thread.sleep(Settings.getInstance().getArchiveInterval());
+
+					BlockData chainTip = Controller.getInstance().getChainTip();
+					if (chainTip == null || NTP.getTime() == null) {
+						continue;
+					}
+
+					// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+					if (Synchronizer.getInstance().isSynchronizing()) {
+						continue;
+					}
+
+					// Don't attempt to archive if we're not synced yet
+					final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
+					if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
+						continue;
+					}
+
+					// Build cache of blocks
+					try {
+						final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+						BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, repository);
+						BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+						switch (result) {
+							case OK:
+								// Increment block archive height
+								startHeight += writer.getWrittenCount();
+								repository.getBlockArchiveRepository().setBlockArchiveHeight(startHeight);
+								repository.saveChanges();
+								break;
+
+							case STOPPING:
+								return;
+
+								// We've reached the limit of the blocks we can archive
+								// Sleep for a while to allow more to become available
+							case NOT_ENOUGH_BLOCKS:
+								// We didn't reach our file size target, so that must mean that we don't have enough blocks
+								// yet or something went wrong. Sleep for a while and then try again.
+								repository.discardChanges();
+								Thread.sleep(2 * 60 * 60 * 1000L); // 1 hour
+								break;
+
+							case BLOCK_NOT_FOUND:
+								// We tried to archive a block that didn't exist. This is a major failure and likely means
+								// that a bootstrap or re-sync is needed. Try again every minute until then.
+								LOGGER.info("Error: block not found when building archive. If this error persists, " +
+										"a bootstrap or re-sync may be needed.");
+								repository.discardChanges();
+								Thread.sleep(60 * 1000L); // 1 minute
+								break;
+						}
+
+					} catch (IOException | TransformationException e) {
+						LOGGER.info("Caught exception when creating block cache", e);
+					}
+				} catch (InterruptedException e) {
+					if (Controller.isStopping()) {
+						LOGGER.info("Block Archiving Shutting Down");
+					} else {
+						LOGGER.warn("Block Archiving interrupted. Trying again. Report this error immediately to the developers.", e);
+					}
+				} catch (Exception e) {
+					LOGGER.warn("Block Archiving stopped working. Trying again. Report this error immediately to the developers.", e);
+				}
+			} catch(Exception e){
+				LOGGER.error("Block Archiving is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			}
+		}
+	}
 }

src/main/java/org/qortal/controller/repository/BlockPruner.java

@@ -11,6 +11,8 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.NTP;
 
+import static java.lang.Thread.NORM_PRIORITY;
+
 public class BlockPruner implements Runnable {
 
 	private static final Logger LOGGER = LogManager.getLogger(BlockPruner.class);
@@ -37,8 +39,10 @@ public class BlockPruner implements Runnable {
 			}
 		}
 
+		int pruneStartHeight;
+
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			int pruneStartHeight = repository.getBlockRepository().getBlockPruneHeight();
+			pruneStartHeight = repository.getBlockRepository().getBlockPruneHeight();
 
 			// Don't attempt to prune if we have no ATStatesHeightIndex, as it will be too slow
 			boolean hasAtStatesHeightIndex = repository.getATRepository().hasAtStatesHeightIndex();
@@ -46,75 +50,90 @@ public class BlockPruner implements Runnable {
 				LOGGER.info("Unable to start block pruner due to missing ATStatesHeightIndex. Bootstrapping is recommended.");
 				return;
 			}
+		} catch (Exception e) {
+			LOGGER.error("Block Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			return;
+		}
 
-			while (!Controller.isStopping()) {
-				repository.discardChanges();
+		while (!Controller.isStopping()) {
 
-				Thread.sleep(Settings.getInstance().getBlockPruneInterval());
+			try (final Repository repository = RepositoryManager.getRepository()) {
 
-				BlockData chainTip = Controller.getInstance().getChainTip();
-				if (chainTip == null || NTP.getTime() == null)
-					continue;
+				try {
+					repository.discardChanges();
 
-				// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
-				if (Synchronizer.getInstance().isSynchronizing()) {
-					continue;
-				}
+					Thread.sleep(Settings.getInstance().getBlockPruneInterval());
 
-				// Don't attempt to prune if we're not synced yet
-				final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
-				if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
-					continue;
-				}
+					BlockData chainTip = Controller.getInstance().getChainTip();
+					if (chainTip == null || NTP.getTime() == null)
+						continue;
 
-				// Prune all blocks up until our latest minus pruneBlockLimit
-				final int ourLatestHeight = chainTip.getHeight();
-				int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
+					// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+					if (Synchronizer.getInstance().isSynchronizing()) {
+						continue;
+					}
 
-				// In archive mode we are only allowed to trim blocks that have already been archived
-				if (archiveMode) {
-					upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
-				}
+					// Don't attempt to prune if we're not synced yet
+					final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
+					if (minLatestBlockTimestamp == null || chainTip.getTimestamp() < minLatestBlockTimestamp) {
+						continue;
+					}
 
-				int upperBatchHeight = pruneStartHeight + Settings.getInstance().getBlockPruneBatchSize();
-				int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
+					// Prune all blocks up until our latest minus pruneBlockLimit
+					final int ourLatestHeight = chainTip.getHeight();
+					int upperPrunableHeight = ourLatestHeight - Settings.getInstance().getPruneBlockLimit();
 
-				if (pruneStartHeight >= upperPruneHeight) {
-					continue;
-				}
+					// In archive mode we are only allowed to trim blocks that have already been archived
+					if (archiveMode) {
+						upperPrunableHeight = repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1;
+					}
 
-				LOGGER.debug(String.format("Pruning blocks between %d and %d...", pruneStartHeight, upperPruneHeight));
+					int upperBatchHeight = pruneStartHeight + Settings.getInstance().getBlockPruneBatchSize();
+					int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
 
-				int numBlocksPruned = repository.getBlockRepository().pruneBlocks(pruneStartHeight, upperPruneHeight);
-				repository.saveChanges();
+					if (pruneStartHeight >= upperPruneHeight) {
+						continue;
+					}
 
-				if (numBlocksPruned > 0) {
-					LOGGER.debug(String.format("Pruned %d block%s between %d and %d",
-							numBlocksPruned, (numBlocksPruned != 1 ? "s" : ""),
-							pruneStartHeight, upperPruneHeight));
-				} else {
-					final int nextPruneHeight = upperPruneHeight + 1;
-					repository.getBlockRepository().setBlockPruneHeight(nextPruneHeight);
+					LOGGER.info(String.format("Pruning blocks between %d and %d...", pruneStartHeight, upperPruneHeight));
+
+					int numBlocksPruned = repository.getBlockRepository().pruneBlocks(pruneStartHeight, upperPruneHeight);
 					repository.saveChanges();
-					LOGGER.debug(String.format("Bumping block base prune height to %d", pruneStartHeight));
 
-					// Can we move onto next batch?
-					if (upperPrunableHeight > nextPruneHeight) {
-						pruneStartHeight = nextPruneHeight;
+					if (numBlocksPruned > 0) {
+						LOGGER.info(String.format("Pruned %d block%s between %d and %d",
+								numBlocksPruned, (numBlocksPruned != 1 ? "s" : ""),
+								pruneStartHeight, upperPruneHeight));
+					} else {
+						final int nextPruneHeight = upperPruneHeight + 1;
+						repository.getBlockRepository().setBlockPruneHeight(nextPruneHeight);
+						repository.saveChanges();
+						LOGGER.info(String.format("Bumping block base prune height to %d", pruneStartHeight));
+
+						// Can we move onto next batch?
+						if (upperPrunableHeight > nextPruneHeight) {
+							pruneStartHeight = nextPruneHeight;
+						}
+						else {
+							// We've pruned up to the upper prunable height
+							// Back off for a while to save CPU for syncing
+							repository.discardChanges();
+							Thread.sleep(10*60*1000L);
+						}
+					}
+				} catch (InterruptedException e) {
+					if(Controller.isStopping()) {
+						LOGGER.info("Block Pruning Shutting Down");
 					}
 					else {
-						// We've pruned up to the upper prunable height
-						// Back off for a while to save CPU for syncing
-						repository.discardChanges();
-						Thread.sleep(10*60*1000L);
+						LOGGER.warn("Block Pruning interrupted. Trying again. Report this error immediately to the developers.", e);
 					}
+				} catch (Exception e) {
+					LOGGER.warn("Block Pruning stopped working. Trying again. Report this error immediately to the developers.", e);
 				}
+			} catch(Exception e){
+				LOGGER.error("Block Pruning is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
 			}
-		} catch (DataException e) {
-			LOGGER.warn(String.format("Repository issue trying to prune blocks: %s", e.getMessage()));
-		} catch (InterruptedException e) {
-			// Time to exit
 		}
 	}
-
 }

src/main/java/org/qortal/controller/repository/OnlineAccountsSignaturesTrimmer.java

@@ -12,6 +12,8 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.NTP;
 
+import static java.lang.Thread.NORM_PRIORITY;
+
 public class OnlineAccountsSignaturesTrimmer implements Runnable {
 
 	private static final Logger LOGGER = LogManager.getLogger(OnlineAccountsSignaturesTrimmer.class);
@@ -26,61 +28,77 @@ public class OnlineAccountsSignaturesTrimmer implements Runnable {
 			return;
 		}
 
+		int trimStartHeight;
+
 		try (final Repository repository = RepositoryManager.getRepository()) {
 			// Don't even start trimming until initial rush has ended
 			Thread.sleep(INITIAL_SLEEP_PERIOD);
 
-			int trimStartHeight = repository.getBlockRepository().getOnlineAccountsSignaturesTrimHeight();
+			trimStartHeight = repository.getBlockRepository().getOnlineAccountsSignaturesTrimHeight();
+		} catch (Exception e) {
+			LOGGER.error("Online Accounts Signatures Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
+			return;
+		}
 
-			while (!Controller.isStopping()) {
-				repository.discardChanges();
+		while (!Controller.isStopping()) {
+			try (final Repository repository = RepositoryManager.getRepository()) {
 
-				Thread.sleep(Settings.getInstance().getOnlineSignaturesTrimInterval());
+				try {
+					repository.discardChanges();
 
-				BlockData chainTip = Controller.getInstance().getChainTip();
-				if (chainTip == null || NTP.getTime() == null)
-					continue;
+					Thread.sleep(Settings.getInstance().getOnlineSignaturesTrimInterval());
 
-				// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
-				if (Synchronizer.getInstance().isSynchronizing())
-					continue;
+					BlockData chainTip = Controller.getInstance().getChainTip();
+					if (chainTip == null || NTP.getTime() == null)
+						continue;
 
-				// Trim blockchain by removing 'old' online accounts signatures
-				long upperTrimmableTimestamp = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMaxLifetime();
-				int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
+					// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+					if (Synchronizer.getInstance().isSynchronizing())
+						continue;
 
-				int upperBatchHeight = trimStartHeight + Settings.getInstance().getOnlineSignaturesTrimBatchSize();
-				int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
+					// Trim blockchain by removing 'old' online accounts signatures
+					long upperTrimmableTimestamp = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMaxLifetime();
+					int upperTrimmableHeight = repository.getBlockRepository().getHeightFromTimestamp(upperTrimmableTimestamp);
 
-				if (trimStartHeight >= upperTrimHeight)
-					continue;
+					int upperBatchHeight = trimStartHeight + Settings.getInstance().getOnlineSignaturesTrimBatchSize();
+					int upperTrimHeight = Math.min(upperBatchHeight, upperTrimmableHeight);
 
-				int numSigsTrimmed = repository.getBlockRepository().trimOldOnlineAccountsSignatures(trimStartHeight, upperTrimHeight);
-				repository.saveChanges();
+					if (trimStartHeight >= upperTrimHeight)
+						continue;
 
-				if (numSigsTrimmed > 0) {
-					final int finalTrimStartHeight = trimStartHeight;
-					LOGGER.debug(() -> String.format("Trimmed %d online accounts signature%s between blocks %d and %d",
-							numSigsTrimmed, (numSigsTrimmed != 1 ? "s" : ""),
-							finalTrimStartHeight, upperTrimHeight));
-				} else {
-					// Can we move onto next batch?
-					if (upperTrimmableHeight > upperBatchHeight) {
-						trimStartHeight = upperBatchHeight;
-
-						repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(trimStartHeight);
-						repository.saveChanges();
+					int numSigsTrimmed = repository.getBlockRepository().trimOldOnlineAccountsSignatures(trimStartHeight, upperTrimHeight);
+					repository.saveChanges();
 
+					if (numSigsTrimmed > 0) {
 						final int finalTrimStartHeight = trimStartHeight;
-						LOGGER.debug(() -> String.format("Bumping online accounts signatures base trim height to %d", finalTrimStartHeight));
+						LOGGER.info(() -> String.format("Trimmed %d online accounts signature%s between blocks %d and %d",
+								numSigsTrimmed, (numSigsTrimmed != 1 ? "s" : ""),
+								finalTrimStartHeight, upperTrimHeight));
+					} else {
+						// Can we move onto next batch?
+						if (upperTrimmableHeight > upperBatchHeight) {
+							trimStartHeight = upperBatchHeight;
+
+							repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(trimStartHeight);
+							repository.saveChanges();
+
+							final int finalTrimStartHeight = trimStartHeight;
+							LOGGER.info(() -> String.format("Bumping online accounts signatures base trim height to %d", finalTrimStartHeight));
+						}
 					}
+				} catch (InterruptedException e) {
+					if(Controller.isStopping()) {
+						LOGGER.info("Online Accounts Signatures Trimming Shutting Down");
+					}
+					else {
+						LOGGER.warn("Online Accounts Signatures Trimming interrupted. Trying again. Report this error immediately to the developers.", e);
+					}
+				} catch (Exception e) {
+					LOGGER.warn("Online Accounts Signatures Trimming stopped working. Trying again. Report this error immediately to the developers.", e);
 				}
+			} catch (Exception e) {
+				LOGGER.error("Online Accounts Signatures Trimming is not working! Not trying again. Restart ASAP. Report this error immediately to the developers.", e);
 			}
-		} catch (DataException e) {
-			LOGGER.warn(String.format("Repository issue trying to trim online accounts signatures: %s", e.getMessage()));
-		} catch (InterruptedException e) {
-			// Time to exit
 		}
 	}
-
 }

src/main/java/org/qortal/controller/repository/PruneManager.java

@@ -40,7 +40,7 @@ public class PruneManager {
     }
 
     public void start() {
-        this.executorService = Executors.newCachedThreadPool(new DaemonThreadFactory());
+        this.executorService = Executors.newCachedThreadPool(new DaemonThreadFactory(Settings.getInstance().getPruningThreadPriority()));
 
         if (Settings.getInstance().isTopOnly()) {
             // Top-only-sync

src/main/java/org/qortal/controller/tradebot/RNSTradeBot.java

@@ -0,0 +1,778 @@
+package org.qortal.controller.tradebot;
+
+import com.google.common.primitives.Longs;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.bitcoinj.core.ECKey;
+import org.qortal.account.PrivateKeyAccount;
+import org.qortal.api.model.crosschain.TradeBotCreateRequest;
+import org.qortal.controller.Controller;
+import org.qortal.controller.Synchronizer;
+import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult;
+import org.qortal.crosschain.*;
+import org.qortal.crypto.Crypto;
+import org.qortal.data.at.ATData;
+import org.qortal.data.crosschain.CrossChainTradeData;
+import org.qortal.data.crosschain.TradeBotData;
+import org.qortal.data.network.TradePresenceData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.event.Event;
+import org.qortal.event.EventBus;
+import org.qortal.event.Listener;
+import org.qortal.gui.SysTray;
+import org.qortal.network.RNSNetwork;
+import org.qortal.network.RNSPeer;
+import org.qortal.network.message.GetTradePresencesMessage;
+import org.qortal.network.message.Message;
+import org.qortal.network.message.TradePresencesMessage;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.repository.hsqldb.HSQLDBImportExport;
+import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
+import org.qortal.utils.ByteArray;
+import org.qortal.utils.NTP;
+
+import java.awt.TrayIcon.MessageType;
+import java.security.SecureRandom;
+import java.util.*;
+import java.util.function.Supplier;
+
+/**
+ * Performing cross-chain trading steps on behalf of user.
+ * <p>
+ * We deal with three different independent state-spaces here:
+ * <ul>
+ * 	<li>Qortal blockchain</li>
+ * 	<li>Foreign blockchain</li>
+ * 	<li>Trade-bot entries</li>
+ * </ul>
+ */
+public class RNSTradeBot implements Listener {
+
+	private static final Logger LOGGER = LogManager.getLogger(TradeBot.class);
+	private static final Random RANDOM = new SecureRandom();
+
+	/** Maximum lifetime of trade presence timestamp. 30 mins in ms. */
+	private static final long PRESENCE_LIFETIME = 30 * 60 * 1000L;
+	/** How soon before expiry of our own trade presence timestamp that we want to trigger renewal. 5 mins in ms. */
+	private static final long EARLY_RENEWAL_PERIOD = 5 * 60 * 1000L;
+	/** Trade presence timestamps are rounded up to this nearest interval. Bigger values improve grouping of entries in [GET_]TRADE_PRESENCES network messages. 15 mins in ms. */
+	private static final long EXPIRY_ROUNDING = 15 * 60 * 1000L;
+	/** How often we want to broadcast our list of all known trade presences to peers. 5 mins in ms. */
+	private static final long PRESENCE_BROADCAST_INTERVAL = 5 * 60 * 1000L;
+
+	public interface StateNameAndValueSupplier {
+		public String getState();
+		public int getStateValue();
+	}
+
+	public static class StateChangeEvent implements Event {
+		private final TradeBotData tradeBotData;
+
+		public StateChangeEvent(TradeBotData tradeBotData) {
+			this.tradeBotData = tradeBotData;
+		}
+
+		public TradeBotData getTradeBotData() {
+			return this.tradeBotData;
+		}
+	}
+
+	public static class TradePresenceEvent implements Event {
+		private final TradePresenceData tradePresenceData;
+
+		public TradePresenceEvent(TradePresenceData tradePresenceData) {
+			this.tradePresenceData = tradePresenceData;
+		}
+
+		public TradePresenceData getTradePresenceData() {
+			return this.tradePresenceData;
+		}
+	}
+
+	private static final Map<Class<? extends ACCT>, Supplier<AcctTradeBot>> acctTradeBotSuppliers = new HashMap<>();
+	static {
+		acctTradeBotSuppliers.put(BitcoinACCTv1.class, BitcoinACCTv1TradeBot::getInstance);
+		acctTradeBotSuppliers.put(BitcoinACCTv3.class, BitcoinACCTv3TradeBot::getInstance);
+		acctTradeBotSuppliers.put(LitecoinACCTv1.class, LitecoinACCTv1TradeBot::getInstance);
+		acctTradeBotSuppliers.put(LitecoinACCTv3.class, LitecoinACCTv3TradeBot::getInstance);
+		acctTradeBotSuppliers.put(DogecoinACCTv1.class, DogecoinACCTv1TradeBot::getInstance);
+		acctTradeBotSuppliers.put(DogecoinACCTv3.class, DogecoinACCTv3TradeBot::getInstance);
+		acctTradeBotSuppliers.put(DigibyteACCTv3.class, DigibyteACCTv3TradeBot::getInstance);
+		acctTradeBotSuppliers.put(RavencoinACCTv3.class, RavencoinACCTv3TradeBot::getInstance);
+		acctTradeBotSuppliers.put(PirateChainACCTv3.class, PirateChainACCTv3TradeBot::getInstance);
+	}
+
+	private static RNSTradeBot instance;
+
+	private final Map<ByteArray, Long> ourTradePresenceTimestampsByPubkey = Collections.synchronizedMap(new HashMap<>());
+	private final List<TradePresenceData> pendingTradePresences = Collections.synchronizedList(new ArrayList<>());
+
+	private final Map<ByteArray, TradePresenceData> allTradePresencesByPubkey = Collections.synchronizedMap(new HashMap<>());
+	private Map<ByteArray, TradePresenceData> safeAllTradePresencesByPubkey = Collections.emptyMap();
+	private long nextTradePresenceBroadcastTimestamp = 0L;
+
+	private Map<String, Long> failedTrades = new HashMap<>();
+	private Map<String, Long> validTrades = new HashMap<>();
+
+	private RNSTradeBot() {
+		EventBus.INSTANCE.addListener(event -> RNSTradeBot.getInstance().listen(event));
+	}
+
+	public static synchronized RNSTradeBot getInstance() {
+		if (instance == null)
+			instance = new RNSTradeBot();
+
+		return instance;
+	}
+
+	public ACCT getAcctUsingAtData(ATData atData) {
+		byte[] codeHash = atData.getCodeHash();
+		if (codeHash == null)
+			return null;
+
+		return SupportedBlockchain.getAcctByCodeHash(codeHash);
+	}
+
+	public CrossChainTradeData populateTradeData(Repository repository, ATData atData) throws DataException {
+		ACCT acct = this.getAcctUsingAtData(atData);
+		if (acct == null)
+			return null;
+
+		return acct.populateTradeData(repository, atData);
+	}
+
+	/**
+	 * Creates a new trade-bot entry from the "Bob" viewpoint,
+	 * i.e. OFFERing QORT in exchange for foreign blockchain currency.
+	 * <p>
+	 * Generates:
+	 * <ul>
+	 * 	<li>new 'trade' private key</li>
+	 * 	<li>secret(s)</li>
+	 * </ul>
+	 * Derives:
+	 * <ul>
+	 * 	<li>'native' (as in Qortal) public key, public key hash, address (starting with Q)</li>
+	 * 	<li>'foreign' public key, public key hash</li>
+	 *	<li>hash(es) of secret(s)</li>
+	 * </ul>
+	 * A Qortal AT is then constructed including the following as constants in the 'data segment':
+	 * <ul>
+	 * 	<li>'native' (Qortal) 'trade' address - used to MESSAGE AT</li>
+	 * 	<li>'foreign' public key hash - used by Alice's to allow redeem of currency on foreign blockchain</li>
+	 * 	<li>hash(es) of secret(s) - used by AT (optional) and foreign blockchain as needed</li>
+	 * 	<li>QORT amount on offer by Bob</li>
+	 * 	<li>foreign currency amount expected in return by Bob (from Alice)</li>
+	 * 	<li>trading timeout, in case things go wrong and everyone needs to refund</li>
+	 * </ul>
+	 * Returns a DEPLOY_AT transaction that needs to be signed and broadcast to the Qortal network.
+	 * <p>
+	 * Trade-bot will wait for Bob's AT to be deployed before taking next step.
+	 * <p>
+	 * @param repository
+	 * @param tradeBotCreateRequest
+	 * @return raw, unsigned DEPLOY_AT transaction
+	 * @throws DataException
+	 */
+	public byte[] createTrade(Repository repository, TradeBotCreateRequest tradeBotCreateRequest) throws DataException {
+		// Fetch latest ACCT version for requested foreign blockchain
+		ACCT acct = tradeBotCreateRequest.foreignBlockchain.getLatestAcct();
+
+		AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
+		if (acctTradeBot == null)
+			return null;
+
+		return acctTradeBot.createTrade(repository, tradeBotCreateRequest);
+	}
+
+	/**
+	 * Creates a trade-bot entry from the 'Alice' viewpoint,
+	 * i.e. matching foreign blockchain currency to an existing QORT offer.
+	 * <p>
+	 * Requires a chosen trade offer from Bob, passed by <tt>crossChainTradeData</tt>
+	 * and access to a foreign blockchain wallet via <tt>foreignKey</tt>.
+	 * <p>
+	 * @param repository
+	 * @param crossChainTradeData chosen trade OFFER that Alice wants to match
+	 * @param foreignKey foreign blockchain wallet key
+	 * @throws DataException
+	 */
+	public ResponseResult startResponse(Repository repository, ATData atData, ACCT acct,
+			CrossChainTradeData crossChainTradeData, String foreignKey, String receivingAddress) throws DataException {
+		AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
+		if (acctTradeBot == null) {
+			LOGGER.debug(() -> String.format("Couldn't find ACCT trade-bot for AT %s", atData.getATAddress()));
+			return ResponseResult.NETWORK_ISSUE;
+		}
+
+		// Check Alice doesn't already have an existing, on-going trade-bot entry for this AT.
+		if (repository.getCrossChainRepository().existsTradeWithAtExcludingStates(atData.getATAddress(), acctTradeBot.getEndStates()))
+			return ResponseResult.TRADE_ALREADY_EXISTS;
+
+		return acctTradeBot.startResponse(repository, atData, acct, crossChainTradeData, foreignKey, receivingAddress);
+	}
+
+	/**
+	 * Creates a trade-bot entries from the 'Alice' viewpoint,
+	 * i.e. matching foreign blockchain currency to existing QORT offers.
+	 * <p>
+	 * Requires chosen trade offers from Bob, passed by <tt>crossChainTradeData</tt>
+	 * and access to a foreign blockchain wallet via <tt>foreignKey</tt>.
+	 * <p>
+	 * @param repository
+	 * @param crossChainTradeDataList chosen trade OFFERs that Alice wants to match
+	 * @param receiveAddress Alice's Qortal address to receive her QORT
+	 * @param foreignKey foreign blockchain wallet key
+	 * @param bitcoiny
+	 * @throws DataException
+	 */
+	public ResponseResult startResponseMultiple(
+			Repository repository,
+			ACCT acct,
+			List<CrossChainTradeData> crossChainTradeDataList,
+			String receiveAddress,
+			String foreignKey,
+			Bitcoiny bitcoiny) throws DataException {
+		AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
+		if (acctTradeBot == null) {
+			LOGGER.debug(() -> String.format("Couldn't find ACCT trade-bot for %s", acct.getBlockchain()));
+			return ResponseResult.NETWORK_ISSUE;
+		}
+
+		for( CrossChainTradeData tradeData : crossChainTradeDataList) {
+			// Check Alice doesn't already have an existing, on-going trade-bot entry for this AT.
+			if (repository.getCrossChainRepository().existsTradeWithAtExcludingStates(tradeData.qortalAtAddress, acctTradeBot.getEndStates()))
+				return ResponseResult.TRADE_ALREADY_EXISTS;
+		}
+		return TradeBotUtils.startResponseMultiple(repository, acct, crossChainTradeDataList, receiveAddress, foreignKey, bitcoiny);
+	}
+
+	public boolean deleteEntry(Repository repository, byte[] tradePrivateKey) throws DataException {
+		TradeBotData tradeBotData = repository.getCrossChainRepository().getTradeBotData(tradePrivateKey);
+		if (tradeBotData == null)
+			// Can't delete what we don't have!
+			return false;
+
+		boolean canDelete = false;
+
+		ACCT acct = SupportedBlockchain.getAcctByName(tradeBotData.getAcctName());
+		if (acct == null)
+			// We can't/no longer support this ACCT
+			canDelete = true;
+		else {
+			AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
+			canDelete = acctTradeBot == null || acctTradeBot.canDelete(repository, tradeBotData);
+		}
+
+		if (canDelete) {
+			repository.getCrossChainRepository().delete(tradePrivateKey);
+			repository.saveChanges();
+		}
+
+		return canDelete;
+	}
+
+	@Override
+	public void listen(Event event) {
+		if (!(event instanceof Synchronizer.NewChainTipEvent))
+			return;
+
+		// Don't process trade bots or broadcast presence timestamps if our chain is more than 60 minutes old
+		final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
+		if (!Controller.getInstance().isUpToDate(minLatestBlockTimestamp))
+			return;
+
+		synchronized (this) {
+			expireOldPresenceTimestamps();
+
+			List<TradeBotData> allTradeBotData;
+
+			try (final Repository repository = RepositoryManager.getRepository()) {
+				allTradeBotData = repository.getCrossChainRepository().getAllTradeBotData();
+			} catch (DataException e) {
+				LOGGER.error("Couldn't run trade bot due to repository issue", e);
+				return;
+			}
+
+			for (TradeBotData tradeBotData : allTradeBotData)
+				try (final Repository repository = RepositoryManager.getRepository()) {
+					// Find ACCT-specific trade-bot for this entry
+					ACCT acct = SupportedBlockchain.getAcctByName(tradeBotData.getAcctName());
+					if (acct == null) {
+						LOGGER.debug(() -> String.format("Couldn't find ACCT matching name %s", tradeBotData.getAcctName()));
+						continue;
+					}
+
+					AcctTradeBot acctTradeBot = findTradeBotForAcct(acct);
+					if (acctTradeBot == null) {
+						LOGGER.debug(() -> String.format("Couldn't find ACCT trade-bot matching name %s", tradeBotData.getAcctName()));
+						continue;
+					}
+
+					acctTradeBot.progress(repository, tradeBotData);
+				} catch (DataException e) {
+					LOGGER.error("Couldn't run trade bot due to repository issue", e);
+				} catch (ForeignBlockchainException e) {
+					LOGGER.warn(() -> String.format("Foreign blockchain issue processing trade-bot entry for AT %s: %s", tradeBotData.getAtAddress(), e.getMessage()));
+				}
+
+			broadcastPresenceTimestamps();
+		}
+	}
+
+	public static byte[] generateTradePrivateKey() {
+		// The private key is used for both Curve25519 and secp256k1 so needs to be valid for both.
+		// Curve25519 accepts any seed, so generate a valid secp256k1 key and use that.
+		return new ECKey().getPrivKeyBytes();
+	}
+
+	public static byte[] deriveTradeNativePublicKey(byte[] privateKey) {
+		return Crypto.toPublicKey(privateKey);
+	}
+
+	public static byte[] deriveTradeForeignPublicKey(byte[] privateKey) {
+		return ECKey.fromPrivate(privateKey).getPubKey();
+	}
+
+	/*package*/ public static byte[] generateSecret() {
+		byte[] secret = new byte[32];
+		RANDOM.nextBytes(secret);
+		return secret;
+	}
+
+	/*package*/ static void backupTradeBotData(Repository repository, List<TradeBotData> additional) {
+		// Attempt to backup the trade bot data. This an optional step and doesn't impact trading, so don't throw an exception on failure
+		try {
+			LOGGER.info("About to backup trade bot data...");
+			HSQLDBImportExport.backupTradeBotStates(repository, additional);
+		} catch (DataException e) {
+			LOGGER.info(String.format("Repository issue when exporting trade bot data: %s", e.getMessage()));
+		}
+	}
+
+	/** Updates trade-bot entry to new state, with current timestamp, logs message and notifies state-change listeners. */
+	/*package*/ static void updateTradeBotState(Repository repository, TradeBotData tradeBotData,
+			String newState, int newStateValue, Supplier<String> logMessageSupplier) throws DataException {
+		tradeBotData.setState(newState);
+		tradeBotData.setStateValue(newStateValue);
+		tradeBotData.setTimestamp(NTP.getTime());
+		repository.getCrossChainRepository().save(tradeBotData);
+		repository.saveChanges();
+
+		if (Settings.getInstance().isTradebotSystrayEnabled())
+			SysTray.getInstance().showMessage("Trade-Bot", String.format("%s: %s", tradeBotData.getAtAddress(), newState), MessageType.INFO);
+
+		if (logMessageSupplier != null)
+			LOGGER.info(logMessageSupplier.get());
+
+		LOGGER.debug(() -> String.format("new state for trade-bot entry based on AT %s: %s", tradeBotData.getAtAddress(), newState));
+
+		notifyStateChange(tradeBotData);
+	}
+
+	/** Updates trade-bot entry to new state, with current timestamp, logs message and notifies state-change listeners. */
+	/*package*/ static void updateTradeBotState(Repository repository, TradeBotData tradeBotData, StateNameAndValueSupplier newStateSupplier, Supplier<String> logMessageSupplier) throws DataException {
+		updateTradeBotState(repository, tradeBotData, newStateSupplier.getState(), newStateSupplier.getStateValue(), logMessageSupplier);
+	}
+
+	/** Updates trade-bot entry to new state, with current timestamp, logs message and notifies state-change listeners. */
+	/*package*/ static void updateTradeBotState(Repository repository, TradeBotData tradeBotData, Supplier<String> logMessageSupplier) throws DataException {
+		updateTradeBotState(repository, tradeBotData, tradeBotData.getState(), tradeBotData.getStateValue(), logMessageSupplier);
+	}
+
+	/*package*/ static void notifyStateChange(TradeBotData tradeBotData) {
+		StateChangeEvent stateChangeEvent = new StateChangeEvent(tradeBotData);
+		EventBus.INSTANCE.notify(stateChangeEvent);
+	}
+
+	/*package*/ static AcctTradeBot findTradeBotForAcct(ACCT acct) {
+		Supplier<AcctTradeBot> acctTradeBotSupplier = acctTradeBotSuppliers.get(acct.getClass());
+		if (acctTradeBotSupplier == null)
+			return null;
+
+		return acctTradeBotSupplier.get();
+	}
+
+	// PRESENCE-related
+
+	public Collection<TradePresenceData> getAllTradePresences() {
+		return this.safeAllTradePresencesByPubkey.values();
+	}
+
+	/** Trade presence timestamps expire in the 'future' so any that reach 'now' have expired and are removed. */
+	private void expireOldPresenceTimestamps() {
+		long now = NTP.getTime();
+
+		int allRemovedCount = 0;
+		synchronized (this.allTradePresencesByPubkey) {
+			int preRemoveCount = this.allTradePresencesByPubkey.size();
+			this.allTradePresencesByPubkey.values().removeIf(tradePresenceData -> tradePresenceData.getTimestamp() <= now);
+			allRemovedCount = this.allTradePresencesByPubkey.size() - preRemoveCount;
+		}
+
+		int ourRemovedCount = 0;
+		synchronized (this.ourTradePresenceTimestampsByPubkey) {
+			int preRemoveCount = this.ourTradePresenceTimestampsByPubkey.size();
+			this.ourTradePresenceTimestampsByPubkey.values().removeIf(timestamp -> timestamp < now);
+			ourRemovedCount = this.ourTradePresenceTimestampsByPubkey.size() - preRemoveCount;
+		}
+
+		if (allRemovedCount > 0)
+			LOGGER.debug("Removed {} expired trade presences, of which {} ours", allRemovedCount, ourRemovedCount);
+	}
+
+	/*package*/ void updatePresence(Repository repository, TradeBotData tradeBotData, CrossChainTradeData tradeData)
+			throws DataException {
+		String atAddress = tradeBotData.getAtAddress();
+
+		PrivateKeyAccount tradeNativeAccount = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
+		String signerAddress = tradeNativeAccount.getAddress();
+
+		/*
+		* There's no point in Alice trying to broadcast presence for an AT that isn't locked to her,
+		* as other peers won't be able to verify as signing public key isn't yet in the AT's data segment.
+		*/
+		if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
+			// Signer is neither Bob, nor trade locked to Alice
+			LOGGER.trace("Can't provide trade presence for our AT {} as it's not yet locked to Alice", atAddress);
+			return;
+		}
+
+		long now = NTP.getTime();
+		long newExpiry = generateExpiry(now);
+		ByteArray pubkeyByteArray = ByteArray.wrap(tradeNativeAccount.getPublicKey());
+
+		// If map entry's timestamp is missing, or within early renewal period, use the new expiry - otherwise use existing timestamp.
+		synchronized (this.ourTradePresenceTimestampsByPubkey) {
+			Long currentTimestamp = this.ourTradePresenceTimestampsByPubkey.get(pubkeyByteArray);
+
+			if (currentTimestamp != null && currentTimestamp - now > EARLY_RENEWAL_PERIOD) {
+				// timestamp still good
+				LOGGER.trace("Current trade presence timestamp {} still good for our trade {}", currentTimestamp, atAddress);
+				return;
+			}
+
+			this.ourTradePresenceTimestampsByPubkey.put(pubkeyByteArray, newExpiry);
+		}
+
+		// Create signature
+		byte[] signature = tradeNativeAccount.sign(Longs.toByteArray(newExpiry));
+
+		// Add new trade presence to queue to be broadcast around network
+		TradePresenceData tradePresenceData = new TradePresenceData(newExpiry, tradeNativeAccount.getPublicKey(), signature, atAddress);
+		this.pendingTradePresences.add(tradePresenceData);
+
+		this.allTradePresencesByPubkey.put(pubkeyByteArray, tradePresenceData);
+		rebuildSafeAllTradePresences();
+
+		LOGGER.trace("New trade presence timestamp {} for our trade {}", newExpiry, atAddress);
+
+		EventBus.INSTANCE.notify(new TradePresenceEvent(tradePresenceData));
+	}
+
+	private void rebuildSafeAllTradePresences() {
+		synchronized (this.allTradePresencesByPubkey) {
+			// Collect into a *new* unmodifiable map.
+			this.safeAllTradePresencesByPubkey = Map.copyOf(this.allTradePresencesByPubkey);
+		}
+	}
+
+	private void broadcastPresenceTimestamps() {
+		// If we have new trade presences that are pending broadcast, send those as a priority
+		if (!this.pendingTradePresences.isEmpty()) {
+			// Create a copy for Network to safely use in another thread
+			List<TradePresenceData> safeTradePresences;
+			synchronized (this.pendingTradePresences) {
+				safeTradePresences = List.copyOf(this.pendingTradePresences);
+				this.pendingTradePresences.clear();
+			}
+
+			LOGGER.debug("Broadcasting {} new trade presences", safeTradePresences.size());
+
+			TradePresencesMessage tradePresencesMessage = new TradePresencesMessage(safeTradePresences);
+			RNSNetwork.getInstance().broadcast(peer -> tradePresencesMessage);
+
+			return;
+		}
+
+		// As we have no new trade presences, check whether it's time to do a general broadcast
+		Long now = NTP.getTime();
+		if (now == null || now < nextTradePresenceBroadcastTimestamp)
+			return;
+
+		nextTradePresenceBroadcastTimestamp = now + PRESENCE_BROADCAST_INTERVAL;
+
+		List<TradePresenceData> safeTradePresences = List.copyOf(this.safeAllTradePresencesByPubkey.values());
+
+		LOGGER.debug("Broadcasting all {} known trade presences. Next broadcast timestamp: {}",
+				safeTradePresences.size(), nextTradePresenceBroadcastTimestamp
+		);
+
+		GetTradePresencesMessage getTradePresencesMessage = new GetTradePresencesMessage(safeTradePresences);
+		RNSNetwork.getInstance().broadcast(peer -> getTradePresencesMessage);
+	}
+
+	// Network message processing
+
+	public void onGetTradePresencesMessage(RNSPeer peer, Message message) {
+		GetTradePresencesMessage getTradePresencesMessage = (GetTradePresencesMessage) message;
+
+		List<TradePresenceData> peersTradePresences = getTradePresencesMessage.getTradePresences();
+
+		// Create mutable copy from safe snapshot
+		Map<ByteArray, TradePresenceData> entriesUnknownToPeer = new HashMap<>(this.safeAllTradePresencesByPubkey);
+		int knownCount = entriesUnknownToPeer.size();
+
+		for (TradePresenceData peersTradePresence : peersTradePresences) {
+			ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
+
+			TradePresenceData ourEntry = entriesUnknownToPeer.get(pubkeyByteArray);
+
+			if (ourEntry != null && ourEntry.getTimestamp() == peersTradePresence.getTimestamp())
+				entriesUnknownToPeer.remove(pubkeyByteArray);
+		}
+
+		if (entriesUnknownToPeer.isEmpty())
+			return;
+
+		LOGGER.debug("Sending {} trade presences to peer {} after excluding their {} from known {}",
+				entriesUnknownToPeer.size(), peer, peersTradePresences.size(), knownCount
+		);
+
+		// Send complement to peer
+		List<TradePresenceData> safeTradePresences = List.copyOf(entriesUnknownToPeer.values());
+		Message responseMessage = new TradePresencesMessage(safeTradePresences);
+		//if (!peer.sendMessage(responseMessage)) {
+		//	peer.disconnect("failed to send TRADE_PRESENCES response");
+		//	return;
+		//}
+		peer.sendMessage(responseMessage);
+	}
+
+	public void onTradePresencesMessage(RNSPeer peer, Message message) {
+		TradePresencesMessage tradePresencesMessage = (TradePresencesMessage) message;
+
+		List<TradePresenceData> peersTradePresences = tradePresencesMessage.getTradePresences();
+
+		long now = NTP.getTime();
+		// Timestamps before this are too far into the past
+		long pastThreshold = now;
+		// Timestamps after this are too far into the future
+		long futureThreshold = now + PRESENCE_LIFETIME;
+
+		Map<ByteArray, Supplier<ACCT>> acctSuppliersByCodeHash = SupportedBlockchain.getAcctMap();
+
+		int newCount = 0;
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			for (TradePresenceData peersTradePresence : peersTradePresences) {
+				long timestamp = peersTradePresence.getTimestamp();
+
+				// Ignore if timestamp is out of bounds
+				if (timestamp < pastThreshold || timestamp > futureThreshold) {
+					if (timestamp < pastThreshold)
+						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too old vs {}",
+								peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
+								);
+					else
+						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too new vs {}",
+								peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
+						);
+
+					continue;
+				}
+
+				ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
+
+				// Ignore if we've previously verified this timestamp+publickey combo or sent timestamp is older
+				TradePresenceData existingTradeData = this.safeAllTradePresencesByPubkey.get(pubkeyByteArray);
+				if (existingTradeData != null && timestamp <= existingTradeData.getTimestamp()) {
+					if (timestamp == existingTradeData.getTimestamp())
+						LOGGER.trace("Ignoring trade presence {} from peer {} as we have verified timestamp {} before",
+								peersTradePresence.getAtAddress(), peer, timestamp
+						);
+					else
+						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is older than latest {}",
+								peersTradePresence.getAtAddress(), peer, timestamp, existingTradeData.getTimestamp()
+						);
+
+					continue;
+				}
+
+				// Check timestamp signature
+				byte[] timestampSignature = peersTradePresence.getSignature();
+				byte[] timestampBytes = Longs.toByteArray(timestamp);
+				byte[] publicKey = peersTradePresence.getPublicKey();
+				if (!Crypto.verify(publicKey, timestampSignature, timestampBytes)) {
+					LOGGER.trace("Ignoring trade presence {} from peer {} as signature failed to verify",
+							peersTradePresence.getAtAddress(), peer
+					);
+
+					continue;
+				}
+
+				ATData atData = repository.getATRepository().fromATAddress(peersTradePresence.getAtAddress());
+				if (atData == null || atData.getIsFrozen() || atData.getIsFinished()) {
+					if (atData == null)
+						LOGGER.trace("Ignoring trade presence {} from peer {} as AT doesn't exist",
+								peersTradePresence.getAtAddress(), peer
+						);
+					else
+						LOGGER.trace("Ignoring trade presence {} from peer {} as AT is frozen or finished",
+								peersTradePresence.getAtAddress(), peer
+						);
+
+					continue;
+				}
+
+				ByteArray atCodeHash = ByteArray.wrap(atData.getCodeHash());
+				Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(atCodeHash);
+				if (acctSupplier == null) {
+					LOGGER.trace("Ignoring trade presence {} from peer {} as AT isn't a known ACCT?",
+							peersTradePresence.getAtAddress(), peer
+					);
+
+					continue;
+				}
+
+				CrossChainTradeData tradeData = acctSupplier.get().populateTradeData(repository, atData);
+				if (tradeData == null) {
+					LOGGER.trace("Ignoring trade presence {} from peer {} as trade data not found?",
+							peersTradePresence.getAtAddress(), peer
+					);
+
+					continue;
+				}
+
+				// Convert signer's public key to address form
+				String signerAddress = peersTradePresence.getTradeAddress();
+
+				// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
+				if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
+					LOGGER.trace("Ignoring trade presence {} from peer {} as signer isn't Alice or Bob?",
+							peersTradePresence.getAtAddress(), peer
+					);
+
+					continue;
+				}
+
+				// This is new to us
+				this.allTradePresencesByPubkey.put(pubkeyByteArray, peersTradePresence);
+				++newCount;
+
+				LOGGER.trace("Added trade presence {} from peer {} with timestamp {}",
+						peersTradePresence.getAtAddress(), peer, timestamp
+				);
+
+				EventBus.INSTANCE.notify(new TradePresenceEvent(peersTradePresence));
+			}
+		} catch (DataException e) {
+			LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
+		}
+
+		if (newCount > 0) {
+			LOGGER.debug("New trade presences: {}, all trade presences: {}", newCount, allTradePresencesByPubkey.size());
+			rebuildSafeAllTradePresences();
+		}
+	}
+
+	public void bridgePresence(long timestamp, byte[] publicKey, byte[] signature, String atAddress) {
+		long expiry = generateExpiry(timestamp);
+		ByteArray pubkeyByteArray = ByteArray.wrap(publicKey);
+
+		TradePresenceData fakeTradePresenceData = new TradePresenceData(expiry, publicKey, signature, atAddress);
+
+		// Only bridge if trade presence expiry timestamp is newer
+		TradePresenceData computedTradePresenceData = this.allTradePresencesByPubkey.compute(pubkeyByteArray, (k, v) ->
+				v == null || v.getTimestamp() < expiry ? fakeTradePresenceData : v
+		);
+
+		if (computedTradePresenceData == fakeTradePresenceData) {
+			LOGGER.trace("Bridged PRESENCE transaction for trade {} with timestamp {}", atAddress, expiry);
+			rebuildSafeAllTradePresences();
+
+			EventBus.INSTANCE.notify(new TradePresenceEvent(fakeTradePresenceData));
+		}
+	}
+
+	/** Decorates a CrossChainTradeData object with Alice / Bob trade-bot presence timestamp, if available. */
+	public void decorateTradeDataWithPresence(CrossChainTradeData crossChainTradeData) {
+		// Match by AT address, then check for Bob vs Alice
+		this.safeAllTradePresencesByPubkey.values().stream()
+				.filter(tradePresenceData -> tradePresenceData.getAtAddress().equals(crossChainTradeData.qortalAtAddress))
+				.forEach(tradePresenceData -> {
+					String signerAddress = tradePresenceData.getTradeAddress();
+
+					// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
+					if (signerAddress.equals(crossChainTradeData.qortalCreatorTradeAddress))
+						crossChainTradeData.creatorPresenceExpiry = tradePresenceData.getTimestamp();
+					else if (signerAddress.equals(crossChainTradeData.qortalPartnerAddress))
+						crossChainTradeData.partnerPresenceExpiry = tradePresenceData.getTimestamp();
+				});
+	}
+
+	/** Removes any trades that have had multiple failures */
+	public List<CrossChainTradeData> removeFailedTrades(Repository repository, List<CrossChainTradeData> crossChainTrades) {
+		Long now = NTP.getTime();
+		if (now == null) {
+			return crossChainTrades;
+		}
+
+		List<CrossChainTradeData> updatedCrossChainTrades = new ArrayList<>(crossChainTrades);
+		int getMaxTradeOfferAttempts = Settings.getInstance().getMaxTradeOfferAttempts();
+
+		for (CrossChainTradeData crossChainTradeData : crossChainTrades) {
+			// We only care about trades in the OFFERING state
+			if (crossChainTradeData.mode != AcctMode.OFFERING) {
+				failedTrades.remove(crossChainTradeData.qortalAtAddress);
+				validTrades.remove(crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+
+			// Return recently cached values if they exist
+			Long failedTimestamp = failedTrades.get(crossChainTradeData.qortalAtAddress);
+			if (failedTimestamp != null && now - failedTimestamp < 60 * 60 * 1000L) {
+				updatedCrossChainTrades.remove(crossChainTradeData);
+				//LOGGER.info("Removing cached failed trade AT {}", crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+			Long validTimestamp = validTrades.get(crossChainTradeData.qortalAtAddress);
+			if (validTimestamp != null && now - validTimestamp < 60 * 60 * 1000L) {
+				//LOGGER.info("NOT removing cached valid trade AT {}", crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+
+			try {
+				List<TransactionData> transactions = repository.getTransactionRepository().getUnconfirmedTransactions(Arrays.asList(Transaction.TransactionType.MESSAGE), null, null, null, null);
+
+				for (TransactionData transactionData : transactions) {
+					// Treat as failed if buy attempt was more than 60 mins ago (as it's still in the OFFERING state)
+					if (transactionData.getRecipient().equals(crossChainTradeData.qortalCreatorTradeAddress) && now - transactionData.getTimestamp() > 60*60*1000L) {
+						failedTrades.put(crossChainTradeData.qortalAtAddress, now);
+						updatedCrossChainTrades.remove(crossChainTradeData);
+					} else {
+						validTrades.put(crossChainTradeData.qortalAtAddress, now);
+					}
+				}
+
+			} catch (DataException e) {
+				LOGGER.info("Unable to determine failed state of AT {}", crossChainTradeData.qortalAtAddress);
+            }
+		}
+
+		return updatedCrossChainTrades;
+	}
+
+	public boolean isFailedTrade(Repository repository, CrossChainTradeData crossChainTradeData) {
+		List<CrossChainTradeData> results = removeFailedTrades(repository, Arrays.asList(crossChainTradeData));
+		return results.isEmpty();
+	}
+
+	private long generateExpiry(long timestamp) {
+		return ((timestamp - 1) / EXPIRY_ROUNDING) * EXPIRY_ROUNDING + PRESENCE_LIFETIME;
+	}
+
+}

src/main/java/org/qortal/controller/tradebot/TradeBot.java

@@ -8,6 +8,7 @@ import org.qortal.account.PrivateKeyAccount;
 import org.qortal.api.model.crosschain.TradeBotCreateRequest;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
+import org.qortal.controller.arbitrary.PeerMessage;
 import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult;
 import org.qortal.crosschain.*;
 import org.qortal.crypto.Crypto;
@@ -37,7 +38,12 @@ import org.qortal.utils.NTP;
 import java.awt.TrayIcon.MessageType;
 import java.security.SecureRandom;
 import java.util.*;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
 import java.util.function.Supplier;
+import java.util.stream.Collectors;
 
 /**
  * Performing cross-chain trading steps on behalf of user.
@@ -118,6 +124,9 @@ public class TradeBot implements Listener {
 	private Map<String, Long> validTrades = new HashMap<>();
 
 	private TradeBot() {
+
+		tradePresenceMessageScheduler.scheduleAtFixedRate( this::processTradePresencesMessages, 60, 1, TimeUnit.SECONDS);
+
 		EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event));
 	}
 
@@ -551,77 +560,139 @@ public class TradeBot implements Listener {
 		}
 	}
 
+	// List to collect messages
+	private final List<PeerMessage> tradePresenceMessageList = new ArrayList<>();
+	// Lock to synchronize access to the list
+	private final Object tradePresenceMessageLock = new Object();
+
+	// Scheduled executor service to process messages every second
+	private final ScheduledExecutorService tradePresenceMessageScheduler = Executors.newScheduledThreadPool(1);
+
 	public void onTradePresencesMessage(Peer peer, Message message) {
-		TradePresencesMessage tradePresencesMessage = (TradePresencesMessage) message;
 
-		List<TradePresenceData> peersTradePresences = tradePresencesMessage.getTradePresences();
+		synchronized (tradePresenceMessageLock) {
+			tradePresenceMessageList.add(new PeerMessage(peer, message));
+		}
+	}
 
-		long now = NTP.getTime();
-		// Timestamps before this are too far into the past
-		long pastThreshold = now;
-		// Timestamps after this are too far into the future
-		long futureThreshold = now + PRESENCE_LIFETIME;
+	public void processTradePresencesMessages() {
 
-		Map<ByteArray, Supplier<ACCT>> acctSuppliersByCodeHash = SupportedBlockchain.getAcctMap();
+		try {
+			List<PeerMessage> messagesToProcess;
+			synchronized (tradePresenceMessageLock) {
+				messagesToProcess = new ArrayList<>(tradePresenceMessageList);
+				tradePresenceMessageList.clear();
+			}
 
-		int newCount = 0;
+			if( messagesToProcess.isEmpty() ) return;
 
-		try (final Repository repository = RepositoryManager.getRepository()) {
-			for (TradePresenceData peersTradePresence : peersTradePresences) {
-				long timestamp = peersTradePresence.getTimestamp();
+			Map<Peer, List<TradePresenceData>> tradePresencesByPeer = new HashMap<>(messagesToProcess.size());
 
-				// Ignore if timestamp is out of bounds
-				if (timestamp < pastThreshold || timestamp > futureThreshold) {
-					if (timestamp < pastThreshold)
-						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too old vs {}",
-								peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
-								);
-					else
-						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too new vs {}",
-								peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
+			// map all trade presences from the messages to their peer
+			for( PeerMessage peerMessage : messagesToProcess ) {
+				TradePresencesMessage tradePresencesMessage = (TradePresencesMessage) peerMessage.getMessage();
+
+				List<TradePresenceData> peersTradePresences = tradePresencesMessage.getTradePresences();
+
+				tradePresencesByPeer.put(peerMessage.getPeer(), peersTradePresences);
+			}
+
+			long now = NTP.getTime();
+			// Timestamps before this are too far into the past
+			long pastThreshold = now;
+			// Timestamps after this are too far into the future
+			long futureThreshold = now + PRESENCE_LIFETIME;
+
+			Map<ByteArray, Supplier<ACCT>> acctSuppliersByCodeHash = SupportedBlockchain.getAcctMap();
+
+			int newCount = 0;
+
+			Map<String, List<Peer>> peersByAtAddress = new HashMap<>(tradePresencesByPeer.size());
+			Map<String, TradePresenceData> tradePresenceByAtAddress = new HashMap<>(tradePresencesByPeer.size());
+
+			// for each batch of trade presence data from a peer, validate and populate the maps declared above
+			for ( Map.Entry<Peer, List<TradePresenceData>> entry: tradePresencesByPeer.entrySet()) {
+
+				Peer peer = entry.getKey();
+
+				for( TradePresenceData peersTradePresence : entry.getValue() ) {
+					// TradePresenceData peersTradePresence
+					long timestamp = peersTradePresence.getTimestamp();
+
+					// Ignore if timestamp is out of bounds
+					if (timestamp < pastThreshold || timestamp > futureThreshold) {
+						if (timestamp < pastThreshold)
+							LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too old vs {}",
+									peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
+							);
+						else
+							LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is too new vs {}",
+									peersTradePresence.getAtAddress(), peer, timestamp, pastThreshold
+							);
+
+						continue;
+					}
+
+					ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
+
+					// Ignore if we've previously verified this timestamp+publickey combo or sent timestamp is older
+					TradePresenceData existingTradeData = this.safeAllTradePresencesByPubkey.get(pubkeyByteArray);
+					if (existingTradeData != null && timestamp <= existingTradeData.getTimestamp()) {
+						if (timestamp == existingTradeData.getTimestamp())
+							LOGGER.trace("Ignoring trade presence {} from peer {} as we have verified timestamp {} before",
+									peersTradePresence.getAtAddress(), peer, timestamp
+							);
+						else
+							LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is older than latest {}",
+									peersTradePresence.getAtAddress(), peer, timestamp, existingTradeData.getTimestamp()
+							);
+
+						continue;
+					}
+
+					// Check timestamp signature
+					byte[] timestampSignature = peersTradePresence.getSignature();
+					byte[] timestampBytes = Longs.toByteArray(timestamp);
+					byte[] publicKey = peersTradePresence.getPublicKey();
+					if (!Crypto.verify(publicKey, timestampSignature, timestampBytes)) {
+						LOGGER.trace("Ignoring trade presence {} from peer {} as signature failed to verify",
+								peersTradePresence.getAtAddress(), peer
 						);
 
-					continue;
+						continue;
+					}
+
+					peersByAtAddress.computeIfAbsent(peersTradePresence.getAtAddress(), address -> new ArrayList<>()).add(peer);
+					tradePresenceByAtAddress.put(peersTradePresence.getAtAddress(), peersTradePresence);
 				}
+			}
 
-				ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
+			if( tradePresenceByAtAddress.isEmpty() ) return;
 
-				// Ignore if we've previously verified this timestamp+publickey combo or sent timestamp is older
-				TradePresenceData existingTradeData = this.safeAllTradePresencesByPubkey.get(pubkeyByteArray);
-				if (existingTradeData != null && timestamp <= existingTradeData.getTimestamp()) {
-					if (timestamp == existingTradeData.getTimestamp())
-						LOGGER.trace("Ignoring trade presence {} from peer {} as we have verified timestamp {} before",
-								peersTradePresence.getAtAddress(), peer, timestamp
-						);
-					else
-						LOGGER.trace("Ignoring trade presence {} from peer {} as timestamp {} is older than latest {}",
-								peersTradePresence.getAtAddress(), peer, timestamp, existingTradeData.getTimestamp()
-						);
+			List<ATData> atDataList;
+			try (final Repository repository = RepositoryManager.getRepository()) {
+				atDataList = repository.getATRepository().fromATAddresses( new ArrayList<>(tradePresenceByAtAddress.keySet()) );
+			} catch (DataException e) {
+				LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
+				return;
+			}
 
-					continue;
-				}
+			Map<String, Supplier<ACCT>> supplierByAtAddress = new HashMap<>(atDataList.size());
 
-				// Check timestamp signature
-				byte[] timestampSignature = peersTradePresence.getSignature();
-				byte[] timestampBytes = Longs.toByteArray(timestamp);
-				byte[] publicKey = peersTradePresence.getPublicKey();
-				if (!Crypto.verify(publicKey, timestampSignature, timestampBytes)) {
-					LOGGER.trace("Ignoring trade presence {} from peer {} as signature failed to verify",
-							peersTradePresence.getAtAddress(), peer
-					);
+			List<ATData> validatedAtDataList = new ArrayList<>(atDataList.size());
 
-					continue;
-				}
+			// for each trade
+			for( ATData atData : atDataList ) {
 
-				ATData atData = repository.getATRepository().fromATAddress(peersTradePresence.getAtAddress());
+				TradePresenceData peersTradePresence = tradePresenceByAtAddress.get(atData.getATAddress());
 				if (atData == null || atData.getIsFrozen() || atData.getIsFinished()) {
 					if (atData == null)
-						LOGGER.trace("Ignoring trade presence {} from peer {} as AT doesn't exist",
-								peersTradePresence.getAtAddress(), peer
+						LOGGER.trace("Ignoring trade presence {} from peer as AT doesn't exist",
+								peersTradePresence.getAtAddress()
 						);
 					else
-						LOGGER.trace("Ignoring trade presence {} from peer {} as AT is frozen or finished",
-								peersTradePresence.getAtAddress(), peer
+						LOGGER.trace("Ignoring trade presence {} from peer as AT is frozen or finished",
+								peersTradePresence.getAtAddress()
 						);
 
 					continue;
@@ -630,51 +701,87 @@ public class TradeBot implements Listener {
 				ByteArray atCodeHash = ByteArray.wrap(atData.getCodeHash());
 				Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(atCodeHash);
 				if (acctSupplier == null) {
-					LOGGER.trace("Ignoring trade presence {} from peer {} as AT isn't a known ACCT?",
-							peersTradePresence.getAtAddress(), peer
+					LOGGER.trace("Ignoring trade presence {} from peer as AT isn't a known ACCT?",
+							peersTradePresence.getAtAddress()
 					);
 
 					continue;
 				}
-
-				CrossChainTradeData tradeData = acctSupplier.get().populateTradeData(repository, atData);
-				if (tradeData == null) {
-					LOGGER.trace("Ignoring trade presence {} from peer {} as trade data not found?",
-							peersTradePresence.getAtAddress(), peer
-					);
-
-					continue;
-				}
-
-				// Convert signer's public key to address form
-				String signerAddress = peersTradePresence.getTradeAddress();
-
-				// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
-				if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
-					LOGGER.trace("Ignoring trade presence {} from peer {} as signer isn't Alice or Bob?",
-							peersTradePresence.getAtAddress(), peer
-					);
-
-					continue;
-				}
-
-				// This is new to us
-				this.allTradePresencesByPubkey.put(pubkeyByteArray, peersTradePresence);
-				++newCount;
-
-				LOGGER.trace("Added trade presence {} from peer {} with timestamp {}",
-						peersTradePresence.getAtAddress(), peer, timestamp
-				);
-
-				EventBus.INSTANCE.notify(new TradePresenceEvent(peersTradePresence));
+				validatedAtDataList.add(atData);
 			}
-		} catch (DataException e) {
-			LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
-		}
 
-		if (newCount > 0) {
-			LOGGER.debug("New trade presences: {}, all trade presences: {}", newCount, allTradePresencesByPubkey.size());
-			rebuildSafeAllTradePresences();
+			// populated data for each trade
+			List<CrossChainTradeData> crossChainTradeDataList;
+
+			// validated trade data grouped by code (cross chain coin)
+			Map<ByteArray, List<ATData>> atDataByCodeHash
+				= validatedAtDataList.stream().collect(
+					Collectors.groupingBy(data -> ByteArray.wrap(data.getCodeHash())));
+
+			try (final Repository repository = RepositoryManager.getRepository()) {
+
+				crossChainTradeDataList = new ArrayList<>();
+
+				// for each code (cross chain coin), get each trade, then populate trade data
+				for( Map.Entry<ByteArray, List<ATData>> entry : atDataByCodeHash.entrySet() ) {
+
+					Supplier<ACCT> acctSupplier = acctSuppliersByCodeHash.get(entry.getKey());
+
+					crossChainTradeDataList.addAll(
+						acctSupplier.get().populateTradeDataList(
+								repository,
+								entry.getValue()
+						)
+						.stream().filter( data -> data != null )
+						.collect(Collectors.toList())
+					);
+				}
+			} catch (DataException e) {
+				LOGGER.error("Couldn't process TRADE_PRESENCES message due to repository issue", e);
+				return;
+			}
+
+			// for each populated trade data, validate and fire event
+			for( CrossChainTradeData tradeData : crossChainTradeDataList ) {
+
+				List<Peer> peers = peersByAtAddress.get(tradeData.qortalAtAddress);
+
+				for( Peer peer : peers ) {
+
+					TradePresenceData peersTradePresence = tradePresenceByAtAddress.get(tradeData.qortalAtAddress);
+
+					// Convert signer's public key to address form
+					String signerAddress = peersTradePresence.getTradeAddress();
+
+					// Signer's public key (in address form) must match Bob's / Alice's trade public key (in address form)
+					if (!signerAddress.equals(tradeData.qortalCreatorTradeAddress) && !signerAddress.equals(tradeData.qortalPartnerAddress)) {
+						LOGGER.trace("Ignoring trade presence {} from peer {} as signer isn't Alice or Bob?",
+								peersTradePresence.getAtAddress(), peer
+						);
+
+						continue;
+					}
+
+					ByteArray pubkeyByteArray = ByteArray.wrap(peersTradePresence.getPublicKey());
+
+					// This is new to us
+					this.allTradePresencesByPubkey.put(pubkeyByteArray, peersTradePresence);
+					++newCount;
+
+					LOGGER.trace("Added trade presence {} from peer {} with timestamp {}",
+							peersTradePresence.getAtAddress(), peer, tradeData.creationTimestamp
+					);
+
+					EventBus.INSTANCE.notify(new TradePresenceEvent(peersTradePresence));
+				}
+			}
+
+			if (newCount > 0) {
+				LOGGER.info("New trade presences: {}, all trade presences: {}", newCount, allTradePresencesByPubkey.size());
+				rebuildSafeAllTradePresences();
+			}
+		} catch (Exception e) {
+			LOGGER.error(e.getMessage(), e);
 		}
 	}
 

src/main/java/org/qortal/crosschain/ACCT.java

@@ -6,6 +6,9 @@ import org.qortal.data.crosschain.CrossChainTradeData;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 
+import java.util.List;
+import java.util.OptionalLong;
+
 public interface ACCT {
 
 	public byte[] getCodeBytesHash();
@@ -16,8 +19,12 @@ public interface ACCT {
 
 	public CrossChainTradeData populateTradeData(Repository repository, ATData atData) throws DataException;
 
+	public List<CrossChainTradeData> populateTradeDataList(Repository respository, List<ATData> atDataList) throws DataException;
+
 	public CrossChainTradeData populateTradeData(Repository repository, ATStateData atStateData) throws DataException;
 
+	CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData, OptionalLong optionalBalance) throws DataException;
+
 	public byte[] buildCancelMessage(String creatorQortalAddress);
 
 	public byte[] findSecretA(Repository repository, CrossChainTradeData crossChainTradeData) throws DataException;

src/main/java/org/qortal/crosschain/Bitcoin.java

@@ -1,5 +1,6 @@
 package org.qortal.crosschain;
 
+import org.bitcoinj.core.Coin;
 import org.bitcoinj.core.Context;
 import org.bitcoinj.core.NetworkParameters;
 import org.bitcoinj.core.Transaction;
@@ -14,15 +15,21 @@ import java.util.Arrays;
 import java.util.Collection;
 import java.util.EnumMap;
 import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
 
 public class Bitcoin extends Bitcoiny {
 
 	public static final String CURRENCY_CODE = "BTC";
 
-	private static final long MINIMUM_ORDER_AMOUNT = 100000; // 0.001 BTC minimum order, due to high fees
+	// Locking fee to lock in a QORT for BTC. This is the default value that the user should reset to
+	// a value inline with the BTC fee market. This is 5 sats per kB.
+	private static final Coin DEFAULT_FEE_PER_KB = Coin.valueOf(5_000); // 0.00005 BTC per 1000 bytes
 
-	// Temporary values until a dynamic fee system is written.
-	private static final long NEW_FEE_AMOUNT = 6_000L;
+	private static final long MINIMUM_ORDER_AMOUNT = 100_000; // 0.001 BTC minimum order, due to high fees
+
+	// Default value until user resets fee to compete with the current market. This is a total value for a
+	// p2sh transaction, size 300 kB, 5 sats per kB
+	private static final long NEW_FEE_AMOUNT = 1_500L;
 
 	private static final long NON_MAINNET_FEE = 1000L; // enough for TESTNET3 and should be OK for REGTEST
 
@@ -111,7 +118,7 @@ public class Bitcoin extends Bitcoiny {
 
 			@Override
 			public long getP2shFee(Long timestamp) {
-				return this.getFeeCeiling();
+				return this.getFeeRequired();
 			}
 		},
 		TEST3 {
@@ -173,14 +180,14 @@ public class Bitcoin extends Bitcoiny {
 			}
 		};
 
-		private long feeCeiling = NEW_FEE_AMOUNT;
+		private AtomicLong feeRequired = new AtomicLong(NEW_FEE_AMOUNT);
 
-		public long getFeeCeiling() {
-			return feeCeiling;
+		public long getFeeRequired() {
+			return feeRequired.get();
 		}
 
-		public void setFeeCeiling(long feeCeiling) {
-			this.feeCeiling = feeCeiling;
+		public void setFeeRequired(long feeRequired) {
+			this.feeRequired.set(feeRequired);
 		}
 
 		public abstract NetworkParameters getParams();
@@ -196,7 +203,7 @@ public class Bitcoin extends Bitcoiny {
 	// Constructors and instance
 
 	private Bitcoin(BitcoinNet bitcoinNet, BitcoinyBlockchainProvider blockchain, Context bitcoinjContext, String currencyCode) {
-		super(blockchain, bitcoinjContext, currencyCode, bitcoinjContext.getFeePerKb());
+		super(blockchain, bitcoinjContext, currencyCode, DEFAULT_FEE_PER_KB);
 		this.bitcoinNet = bitcoinNet;
 
 		LOGGER.info(() -> String.format("Starting Bitcoin support using %s", this.bitcoinNet.name()));
@@ -242,14 +249,14 @@ public class Bitcoin extends Bitcoiny {
 	}
 
 	@Override
-	public long getFeeCeiling() {
-		return this.bitcoinNet.getFeeCeiling();
+	public long getFeeRequired() {
+		return this.bitcoinNet.getFeeRequired();
 	}
 
 	@Override
-	public void setFeeCeiling(long fee) {
+	public void setFeeRequired(long fee) {
 
-		this.bitcoinNet.setFeeCeiling( fee );
+		this.bitcoinNet.setFeeRequired( fee );
 	}
 	/**
  	* Returns bitcoinj transaction sending <tt>amount</tt> to <tt>recipient</tt> using 20 sat/byte fee.

src/main/java/org/qortal/crosschain/BitcoinACCTv1.java

@@ -4,6 +4,7 @@ import com.google.common.hash.HashCode;
 import com.google.common.primitives.Bytes;
 import org.ciyam.at.*;
 import org.qortal.account.Account;
+import org.qortal.api.resource.CrossChainUtils;
 import org.qortal.asset.Asset;
 import org.qortal.at.QortalFunctionCode;
 import org.qortal.crypto.Crypto;
@@ -19,6 +20,7 @@ import org.qortal.utils.BitTwiddling;
 import java.nio.ByteBuffer;
 import java.util.Arrays;
 import java.util.List;
+import java.util.OptionalLong;
 
 import static org.ciyam.at.OpCode.calcOffset;
 
@@ -608,7 +610,14 @@ public class BitcoinACCTv1 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATData atData) throws DataException {
 		ATStateData atStateData = repository.getATRepository().getLatestATState(atData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
+	}
+
+	@Override
+	public List<CrossChainTradeData> populateTradeDataList(Repository repository, List<ATData> atDataList) throws DataException {
+		List<CrossChainTradeData> crossChainTradeDataList = CrossChainUtils.populateTradeDataList(repository, this, atDataList);
+
+		return crossChainTradeDataList;
 	}
 
 	/**
@@ -617,13 +626,14 @@ public class BitcoinACCTv1 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATStateData atStateData) throws DataException {
 		ATData atData = repository.getATRepository().fromATAddress(atStateData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
 	}
 
 	/**
 	 * Returns CrossChainTradeData with useful info extracted from AT.
 	 */
-	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData) throws DataException {
+	@Override
+	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData, OptionalLong optionalBalance) throws DataException {
 		byte[] addressBytes = new byte[25]; // for general use
 		String atAddress = atStateData.getATAddress();
 
@@ -636,8 +646,13 @@ public class BitcoinACCTv1 implements ACCT {
 		tradeData.qortalCreator = Crypto.toAddress(creatorPublicKey);
 		tradeData.creationTimestamp = creationTimestamp;
 
-		Account atAccount = new Account(repository, atAddress);
-		tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		if(optionalBalance.isPresent()) {
+			tradeData.qortBalance = optionalBalance.getAsLong();
+		}
+		else {
+			Account atAccount = new Account(repository, atAddress);
+			tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		}
 
 		byte[] stateData = atStateData.getStateData();
 		ByteBuffer dataByteBuffer = ByteBuffer.wrap(stateData);

src/main/java/org/qortal/crosschain/BitcoinACCTv3.java

@@ -6,6 +6,7 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.ciyam.at.*;
 import org.qortal.account.Account;
+import org.qortal.api.resource.CrossChainUtils;
 import org.qortal.asset.Asset;
 import org.qortal.at.QortalFunctionCode;
 import org.qortal.crypto.Crypto;
@@ -21,6 +22,7 @@ import org.qortal.utils.BitTwiddling;
 import java.nio.ByteBuffer;
 import java.util.Arrays;
 import java.util.List;
+import java.util.OptionalLong;
 
 import static org.ciyam.at.OpCode.calcOffset;
 
@@ -569,7 +571,14 @@ public class BitcoinACCTv3 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATData atData) throws DataException {
 		ATStateData atStateData = repository.getATRepository().getLatestATState(atData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
+	}
+
+	@Override
+	public List<CrossChainTradeData> populateTradeDataList(Repository repository, List<ATData> atDataList) throws DataException {
+		List<CrossChainTradeData> crossChainTradeDataList = CrossChainUtils.populateTradeDataList(repository, this, atDataList);
+
+		return crossChainTradeDataList;
 	}
 
 	/**
@@ -578,13 +587,14 @@ public class BitcoinACCTv3 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATStateData atStateData) throws DataException {
 		ATData atData = repository.getATRepository().fromATAddress(atStateData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
 	}
 
 	/**
 	 * Returns CrossChainTradeData with useful info extracted from AT.
 	 */
-	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData) throws DataException {
+	@Override
+	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData, OptionalLong optionalBalance) throws DataException {
 		byte[] addressBytes = new byte[25]; // for general use
 		String atAddress = atStateData.getATAddress();
 
@@ -597,8 +607,13 @@ public class BitcoinACCTv3 implements ACCT {
 		tradeData.qortalCreator = Crypto.toAddress(creatorPublicKey);
 		tradeData.creationTimestamp = creationTimestamp;
 
-		Account atAccount = new Account(repository, atAddress);
-		tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		if(optionalBalance.isPresent()) {
+			tradeData.qortBalance = optionalBalance.getAsLong();
+		}
+		else {
+			Account atAccount = new Account(repository, atAddress);
+			tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		}
 
 		byte[] stateData = atStateData.getStateData();
 		ByteBuffer dataByteBuffer = ByteBuffer.wrap(stateData);

src/main/java/org/qortal/crosschain/Bitcoiny.java

@@ -83,6 +83,7 @@ public abstract class Bitcoiny implements ForeignBlockchain {
 		return this.bitcoinjContext;
 	}
 
+	@Override
 	public String getCurrencyCode() {
 		return this.currencyCode;
 	}
@@ -839,9 +840,9 @@ public abstract class Bitcoiny implements ForeignBlockchain {
 		} while (true);
 	}
 
-	public abstract long getFeeCeiling();
+	public abstract long getFeeRequired();
 
-	public abstract void setFeeCeiling(long fee);
+	public abstract void setFeeRequired(long fee);
 
 	// UTXOProvider support
 

src/main/java/org/qortal/crosschain/BitcoinyTBD.java

@@ -89,7 +89,7 @@ public class BitcoinyTBD extends Bitcoiny {
 		NetTBD netTBD
 				= new NetTBD(
 				bitcoinyTBDRequest.getNetworkName(),
-				bitcoinyTBDRequest.getFeeCeiling(),
+				bitcoinyTBDRequest.getFeeRequired(),
 				networkParams,
 				Collections.emptyList(),
 				bitcoinyTBDRequest.getExpectedGenesisHash()
@@ -134,18 +134,18 @@ public class BitcoinyTBD extends Bitcoiny {
 	@Override
 	public long getP2shFee(Long timestamp) throws ForeignBlockchainException {
 
-		return this.netTBD.getFeeCeiling();
+		return this.netTBD.getFeeRequired();
 	}
 
 	@Override
-	public long getFeeCeiling() {
+	public long getFeeRequired() {
 
-		return this.netTBD.getFeeCeiling();
+		return this.netTBD.getFeeRequired();
 	}
 
 	@Override
-	public void setFeeCeiling(long fee) {
+	public void setFeeRequired(long fee) {
 
-		this.netTBD.setFeeCeiling( fee );
+		this.netTBD.setFeeRequired( fee );
 	}
 }

src/main/java/org/qortal/crosschain/Digibyte.java

@@ -14,6 +14,7 @@ import java.util.Arrays;
 import java.util.Collection;
 import java.util.EnumMap;
 import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
 
 public class Digibyte extends Bitcoiny {
 
@@ -59,7 +60,7 @@ public class Digibyte extends Bitcoiny {
 
 			@Override
 			public long getP2shFee(Long timestamp) {
-				return this.getFeeCeiling();
+				return this.getFeeRequired();
 			}
 		},
 		TEST3 {
@@ -109,14 +110,14 @@ public class Digibyte extends Bitcoiny {
 			}
 		};
 
-		private long feeCeiling = MAINNET_FEE;
+		private AtomicLong feeRequired = new AtomicLong(MAINNET_FEE);
 
-		public long getFeeCeiling() {
-			return feeCeiling;
+		public long getFeeRequired() {
+			return feeRequired.get();
 		}
 
-		public void setFeeCeiling(long feeCeiling) {
-			this.feeCeiling = feeCeiling;
+		public void setFeeRequired(long feeRequired) {
+			this.feeRequired.set(feeRequired);
 		}
 
 		public abstract NetworkParameters getParams();
@@ -178,13 +179,13 @@ public class Digibyte extends Bitcoiny {
 	}
 
 	@Override
-	public long getFeeCeiling() {
-		return this.digibyteNet.getFeeCeiling();
+	public long getFeeRequired() {
+		return this.digibyteNet.getFeeRequired();
 	}
 
 	@Override
-	public void setFeeCeiling(long fee) {
+	public void setFeeRequired(long fee) {
 
-		this.digibyteNet.setFeeCeiling( fee );
+		this.digibyteNet.setFeeRequired( fee );
 	}
 }

src/main/java/org/qortal/crosschain/DigibyteACCTv3.java

@@ -6,6 +6,7 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.ciyam.at.*;
 import org.qortal.account.Account;
+import org.qortal.api.resource.CrossChainUtils;
 import org.qortal.asset.Asset;
 import org.qortal.at.QortalFunctionCode;
 import org.qortal.crypto.Crypto;
@@ -21,6 +22,7 @@ import org.qortal.utils.BitTwiddling;
 import java.nio.ByteBuffer;
 import java.util.Arrays;
 import java.util.List;
+import java.util.OptionalLong;
 
 import static org.ciyam.at.OpCode.calcOffset;
 
@@ -569,7 +571,14 @@ public class DigibyteACCTv3 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATData atData) throws DataException {
 		ATStateData atStateData = repository.getATRepository().getLatestATState(atData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
+	}
+
+	@Override
+	public List<CrossChainTradeData> populateTradeDataList(Repository repository, List<ATData> atDataList) throws DataException {
+		List<CrossChainTradeData> crossChainTradeDataList = CrossChainUtils.populateTradeDataList(repository, this, atDataList);
+
+		return crossChainTradeDataList;
 	}
 
 	/**
@@ -578,13 +587,14 @@ public class DigibyteACCTv3 implements ACCT {
 	@Override
 	public CrossChainTradeData populateTradeData(Repository repository, ATStateData atStateData) throws DataException {
 		ATData atData = repository.getATRepository().fromATAddress(atStateData.getATAddress());
-		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData);
+		return populateTradeData(repository, atData.getCreatorPublicKey(), atData.getCreation(), atStateData, OptionalLong.empty());
 	}
 
 	/**
 	 * Returns CrossChainTradeData with useful info extracted from AT.
 	 */
-	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData) throws DataException {
+	@Override
+	public CrossChainTradeData populateTradeData(Repository repository, byte[] creatorPublicKey, long creationTimestamp, ATStateData atStateData, OptionalLong optionalBalance) throws DataException {
 		byte[] addressBytes = new byte[25]; // for general use
 		String atAddress = atStateData.getATAddress();
 
@@ -597,8 +607,13 @@ public class DigibyteACCTv3 implements ACCT {
 		tradeData.qortalCreator = Crypto.toAddress(creatorPublicKey);
 		tradeData.creationTimestamp = creationTimestamp;
 
-		Account atAccount = new Account(repository, atAddress);
-		tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		if(optionalBalance.isPresent()) {
+			tradeData.qortBalance = optionalBalance.getAsLong();
+		}
+		else {
+			Account atAccount = new Account(repository, atAddress);
+			tradeData.qortBalance = atAccount.getConfirmedBalance(Asset.QORT);
+		}
 
 		byte[] stateData = atStateData.getStateData();
 		ByteBuffer dataByteBuffer = ByteBuffer.wrap(stateData);