Finalized the reindex feature.

These were caused by a bug in the name rebuilding code, which has since been fixed.

Affected transactions:

CancelSellNameTransaction:74 - Error during transaction validation, tx 3FLFa9LuYS3tJ3bHB6mTDnLcAHBbcNHhQhvPT8wvcW14w59TGiJ9NabGe7HzG7XVZRpuhRQoaFDfDfJPcdrU44ry: NAME_NOT_FOR_SALE - 1177021 - 1676828446023
CancelSellNameTransaction:74 - Error during transaction validation, tx 4ZVREx4ZnBn5nfFCXvpjCXLFDV9aSqYcCqhCuYJQ2bf4h4mH6wkuAKGGgF9d2xWZWYY5ujFR2E2PBkg2zTzRhf6m: NAME_NOT_FOR_SALE - 1178222 - 1676841112553
CancelSellNameTransaction:74 - Error during transaction validation, tx 3caxAKM291kUVLmsAfpbsnrgwk9VZdTRLyt86iVjsFzhJs22gGdKf26fJqpzBt6czqhoTosPH9z4o14nQ56cZpjM: NAME_NOT_FOR_SALE - 1179201 - 1676986362069
CancelSellNameTransaction:74 - Error during transaction validation, tx 3FLFa9LuYS3tJ3bHB6mTDnLcAHBbcNHhQhvPT8wvcW14w59TGiJ9NabGe7HzG7XVZRpuhRQoaFDfDfJPcdrU44ry: NAME_NOT_FOR_SALE - 1177021 - 1676828446023
CancelSellNameTransaction:74 - Error during transaction validation, tx 4ZVREx4ZnBn5nfFCXvpjCXLFDV9aSqYcCqhCuYJQ2bf4h4mH6wkuAKGGgF9d2xWZWYY5ujFR2E2PBkg2zTzRhf6m: NAME_NOT_FOR_SALE - 1178222 - 1676841112553
CancelSellNameTransaction:74 - Error during transaction validation, tx 3caxAKM291kUVLmsAfpbsnrgwk9VZdTRLyt86iVjsFzhJs22gGdKf26fJqpzBt6czqhoTosPH9z4o14nQ56cZpjM: NAME_NOT_FOR_SALE - 1179201 - 1676986362069

Q-Apps.md

@@ -42,10 +42,15 @@ A "default" resource refers to one without an identifier. For example, when a we
 
 Here is a list of currently available services that can be used in Q-Apps:
 
+### Public services ###
+The services below are intended to be used for publicly accessible data.
+
 IMAGE,
 THUMBNAIL,
 VIDEO,
 AUDIO,
+PODCAST,
+VOICE,
 ARBITRARY_DATA,
 JSON,
 DOCUMENT,
@@ -55,7 +60,25 @@ METADATA,
 BLOG,
 BLOG_POST,
 BLOG_COMMENT,
-GIF_REPOSITORY
+GIF_REPOSITORY,
+ATTACHMENT,
+FILE,
+FILES,
+CHAIN_DATA,
+STORE,
+PRODUCT,
+OFFER,
+COUPON,
+CODE,
+PLUGIN,
+EXTENSION,
+GAME,
+ITEM,
+NFT,
+DATABASE,
+SNAPSHOT,
+COMMENT,
+CHAIN_COMMENT,
 WEBSITE,
 APP,
 QCHAT_ATTACHMENT,
@@ -63,6 +86,20 @@ QCHAT_IMAGE,
 QCHAT_AUDIO,
 QCHAT_VOICE
 
+### Private services ###
+For the services below, data is encrypted for a single recipient, and can only be decrypted using the private key of the recipient's wallet.
+
+QCHAT_ATTACHMENT_PRIVATE,
+ATTACHMENT_PRIVATE,
+FILE_PRIVATE,
+IMAGE_PRIVATE,
+VIDEO_PRIVATE,
+AUDIO_PRIVATE,
+VOICE_PRIVATE,
+DOCUMENT_PRIVATE,
+MAIL_PRIVATE,
+MESSAGE_PRIVATE
+
 
 ## Single vs multi-file resources
 
@@ -173,6 +210,7 @@ To take things a step further, the qortalRequest() function can be used to inter
 - Join groups
 - Deploy ATs (smart contracts)
 - Send QORT or any supported foreign coin
+- Add/remove items from lists
 
 In addition to the above, qortalRequest() also supports many read-only functions that are also available via direct core API calls. Using qortalRequest() helps with futureproofing, as the core APIs can be modified without breaking functionality of existing Q-Apps.
 
@@ -214,14 +252,20 @@ Here is a list of currently supported actions:
 - GET_USER_ACCOUNT
 - GET_ACCOUNT_DATA
 - GET_ACCOUNT_NAMES
+- SEARCH_NAMES
 - GET_NAME_DATA
 - LIST_QDN_RESOURCES
 - SEARCH_QDN_RESOURCES
 - GET_QDN_RESOURCE_STATUS
 - GET_QDN_RESOURCE_PROPERTIES
+- GET_QDN_RESOURCE_METADATA
+- GET_QDN_RESOURCE_URL
+- LINK_TO_QDN_RESOURCE
 - FETCH_QDN_RESOURCE
 - PUBLISH_QDN_RESOURCE
 - PUBLISH_MULTIPLE_QDN_RESOURCES
+- DECRYPT_DATA
+- SAVE_FILE
 - GET_WALLET_BALANCE
 - GET_BALANCE
 - SEND_COIN
@@ -237,8 +281,9 @@ Here is a list of currently supported actions:
 - FETCH_BLOCK_RANGE
 - SEARCH_TRANSACTIONS
 - GET_PRICE
-- GET_QDN_RESOURCE_URL
-- LINK_TO_QDN_RESOURCE
+- GET_LIST_ITEMS
+- ADD_LIST_ITEMS
+- DELETE_LIST_ITEM
 
 More functionality will be added in the future.
 
@@ -280,6 +325,18 @@ let res = await qortalRequest({
 });
 ```
 
+### Search names
+```
+let res = await qortalRequest({
+    action: "SEARCH_NAMES",
+    query: "search query goes here",
+    prefix: false, // Optional - if true, only the beginning of the name is matched
+    limit: 100,
+    offset: 0,
+    reverse: false
+});
+```
+
 ### Get name data
 ```
 let res = await qortalRequest({
@@ -299,6 +356,8 @@ let res = await qortalRequest({
     default: true, // Optional
     includeStatus: false, // Optional - will take time to respond, so only request if necessary
     includeMetadata: false, // Optional - will take time to respond, so only request if necessary
+    followedOnly: false, // Optional - include followed names only
+    excludeBlocked: false, // Optional - exclude blocked content
     limit: 100,
     offset: 0,
     reverse: true
@@ -314,9 +373,34 @@ let res = await qortalRequest({
     identifier: "search query goes here", // Optional - searches only the "identifier" field
     name: "search query goes here", // Optional - searches only the "name" field
     prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters
+    exactMatchNames: true, // Optional - if true, partial name matches are excluded
     default: false, // Optional - if true, only resources without identifiers are returned
     includeStatus: false, // Optional - will take time to respond, so only request if necessary
     includeMetadata: false, // Optional - will take time to respond, so only request if necessary
+    nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list
+    followedOnly: false, // Optional - include followed names only
+    excludeBlocked: false, // Optional - exclude blocked content
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Search QDN resources (multiple names)
+```
+let res = await qortalRequest({
+    action: "SEARCH_QDN_RESOURCES",
+    service: "THUMBNAIL",
+    query: "search query goes here", // Optional - searches both "identifier" and "name" fields
+    identifier: "search query goes here", // Optional - searches only the "identifier" field
+    names: ["QortalDemo", "crowetic", "AlphaX"], // Optional - searches only the "name" field for any of the supplied names
+    prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters
+    default: false, // Optional - if true, only resources without identifiers are returned
+    includeStatus: false, // Optional - will take time to respond, so only request if necessary
+    includeMetadata: false, // Optional - will take time to respond, so only request if necessary
+    nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list
+    followedOnly: false, // Optional - include followed names only
+    excludeBlocked: false, // Optional - exclude blocked content
     limit: 100,
     offset: 0,
     reverse: true
@@ -354,7 +438,8 @@ let res = await qortalRequest({
     action: "GET_QDN_RESOURCE_STATUS",
     name: "QortalDemo",
     service: "THUMBNAIL",
-    identifier: "qortal_avatar" // Optional
+    identifier: "qortal_avatar", // Optional
+    build: true // Optional - request that the resource is fetched & built in the background
 });
 ```
 
@@ -369,11 +454,21 @@ let res = await qortalRequest({
 // Returns: filename, size, mimeType (where available)
 ```
 
+### Get QDN resource metadata
+```
+let res = await qortalRequest({
+    action: "GET_QDN_RESOURCE_METADATA",
+    name: "QortalDemo",
+    service: "THUMBNAIL",
+    identifier: "qortal_avatar" // Optional
+});
+```
+
 ### Publish a single file to QDN
 _Requires user approval_.<br />
 Note: this publishes a single, base64-encoded file. Multi-file resource publishing (such as a WEBSITE or GIF_REPOSITORY) is not yet supported via a Q-App. It will be added in a future update.
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "PUBLISH_QDN_RESOURCE",
     name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
     service: "IMAGE",
@@ -387,7 +482,9 @@ await qortalRequest({
     // tag2: "strings", // Optional
     // tag3: "can", // Optional
     // tag4: "go", // Optional
-    // tag5: "here" // Optional
+    // tag5: "here", // Optional
+    // encrypt: true, // Optional - to be used with a private service
+    // recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true
 });
 ```
 
@@ -395,7 +492,7 @@ await qortalRequest({
 _Requires user approval_.<br />
 Note: each resource being published consists of a single, base64-encoded file, each in its own transaction. Useful for publishing two or more related things, such as a video and a video thumbnail.
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "PUBLISH_MULTIPLE_QDN_RESOURCES",
     resources: [
         name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
@@ -410,7 +507,9 @@ await qortalRequest({
         // tag2: "strings", // Optional
         // tag3: "can", // Optional
         // tag4: "go", // Optional
-        // tag5: "here" // Optional
+        // tag5: "here", // Optional
+        // encrypt: true, // Optional - to be used with a private service
+        // recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true
     ],
     [
         ... more resources here if needed ...
@@ -418,10 +517,32 @@ await qortalRequest({
 });
 ```
 
+### Decrypt encrypted/private data
+```
+let res = await qortalRequest({
+    action: "DECRYPT_DATA",
+    encryptedData: 'qortalEncryptedDatabMx4fELNTV+ifJxmv4+GcuOIJOTo+3qAvbWKNY2L1r',
+    publicKey: 'publickeygoeshere'
+});
+// Returns base64 encoded string of plaintext data
+```
+
+### Prompt user to save a file to disk
+Note: mimeType not required but recommended. If not specified, saving will fail if the mimeType is unable to be derived from the Blob.
+```
+let res = await qortalRequest({
+    action: "SAVE_FILE",
+    blob: dataBlob,
+    filename: "myfile.pdf",
+    mimeType: "application/pdf" // Optional but recommended
+});
+```
+
+
 ### Get wallet balance (QORT)
 _Requires user approval_
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "GET_WALLET_BALANCE",
     coin: "QORT"
 });
@@ -446,7 +567,7 @@ let res = await qortalRequest({
 ### Send QORT to address
 _Requires user approval_
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "SEND_COIN",
     coin: "QORT",
     destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
@@ -454,6 +575,18 @@ await qortalRequest({
 });
 ```
 
+### Send foreign coin to address
+_Requires user approval_<br />
+Note: default fees can be found [here](https://github.com/Qortal/qortal-ui/blob/master/plugins/plugins/core/qdn/browser/browser.src.js#L205-L209).
+```
+let res = await qortalRequest({
+    action: "SEND_COIN",
+    coin: "LTC",
+    destinationAddress: "LSdTvMHRm8sScqwCi6x9wzYQae8JeZhx6y",
+    amount: 1.00000000, // 1 LTC
+    fee: 0.00000020 // Optional fee per byte (default fee used if omitted, recommended) - not used for QORT or ARRR
+});
+```
 
 ### Search or list chat messages
 ```
@@ -466,6 +599,7 @@ let res = await qortalRequest({
     // reference: "reference", // Optional
     // chatReference: "chatreference", // Optional
     // hasChatReference: true, // Optional
+    encoding: "BASE64", // Optional (defaults to BASE58 if omitted)
     limit: 100,
     offset: 0,
     reverse: true
@@ -475,7 +609,7 @@ let res = await qortalRequest({
 ### Send a group chat message
 _Requires user approval_
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "SEND_CHAT_MESSAGE",
     groupId: 0,
     message: "Test"
@@ -485,7 +619,7 @@ await qortalRequest({
 ### Send a private chat message
 _Requires user approval_
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "SEND_CHAT_MESSAGE",
     destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
     message: "Test"
@@ -505,7 +639,7 @@ let res = await qortalRequest({
 ### Join a group
 _Requires user approval_
 ```
-await qortalRequest({
+let res = await qortalRequest({
     action: "JOIN_GROUP",
     groupId: 100
 });
@@ -611,6 +745,7 @@ let res = await qortalRequest({
 ```
 
 ### Get URL to load a QDN resource
+Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
 ```
 let url = await qortalRequest({
     action: "GET_QDN_RESOURCE_URL",
@@ -622,6 +757,7 @@ let url = await qortalRequest({
 ```
 
 ### Get URL to load a QDN website
+Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
 ```
 let url = await qortalRequest({
     action: "GET_QDN_RESOURCE_URL",
@@ -631,6 +767,7 @@ let url = await qortalRequest({
 ```
 
 ### Get URL to load a specific file from a QDN website
+Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
 ```
 let url = await qortalRequest({
     action: "GET_QDN_RESOURCE_URL",
@@ -661,9 +798,42 @@ let res = await qortalRequest({
 });
 ```
 
+### Get the contents of a list
+_Requires user approval_
+```
+let res = await qortalRequest({
+    action: "GET_LIST_ITEMS",
+    list_name: "followedNames"
+});
+```
+
+### Add one or more items to a list
+_Requires user approval_
+```
+let res = await qortalRequest({
+    action: "ADD_LIST_ITEMS",
+    list_name: "blockedNames",
+    items: ["QortalDemo"]
+});
+```
+
+### Delete a single item from a list
+_Requires user approval_.
+Items must be deleted one at a time.
+```
+let res = await qortalRequest({
+    action: "DELETE_LIST_ITEM",
+    list_name: "blockedNames",
+    item: "QortalDemo"
+});
+```
+
 
 # Section 4: Examples
 
+Some example projects can be found [here](https://github.com/Qortal/Q-Apps). These can be cloned and modified, or used as a reference when creating a new app.
+
+
 ## Sample App
 
 Here is a sample application to display the logged-in user's avatar:

pom.xml

@@ -3,7 +3,7 @@
 	<modelVersion>4.0.0</modelVersion>
 	<groupId>org.qortal</groupId>
 	<artifactId>qortal</artifactId>
-	<version>3.9.0</version>
+	<version>4.2.2</version>
 	<packaging>jar</packaging>
 	<properties>
 		<skipTests>true</skipTests>
@@ -148,6 +148,7 @@
 										tagsSorter: "alpha",
 										operationsSorter:
 										"alpha",
+										validatorUrl: false,
 									</value>
 								</replacement>
 							</replacements>

src/main/java/org/qortal/api/ApiError.java

@@ -79,7 +79,7 @@ public enum ApiError {
 	// BUYER_ALREADY_OWNER(411, 422),
 
 	// POLLS
-	// POLL_NO_EXISTS(501, 404),
+	POLL_NO_EXISTS(501, 404),
 	// POLL_ALREADY_EXISTS(502, 422),
 	// DUPLICATE_OPTION(503, 422),
 	// POLL_OPTION_NO_EXISTS(504, 404),

src/main/java/org/qortal/api/ApiService.java

@@ -41,6 +41,7 @@ import org.glassfish.jersey.servlet.ServletContainer;
 import org.qortal.api.resource.AnnotationPostProcessor;
 import org.qortal.api.resource.ApiDefinition;
 import org.qortal.api.websocket.*;
+import org.qortal.network.Network;
 import org.qortal.settings.Settings;
 
 public class ApiService {
@@ -95,7 +96,7 @@ public class ApiService {
 					throw new RuntimeException("Failed to start SSL API due to broken keystore");
 
 				// BouncyCastle-specific SSLContext build
-				SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
+				SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
 				KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
 
 				KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@@ -125,13 +126,13 @@ public class ApiService {
 				ServerConnector portUnifiedConnector = new ServerConnector(this.server,
 						new DetectorConnectionFactory(sslConnectionFactory),
 						httpConnectionFactory);
-				portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
+				portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
 				portUnifiedConnector.setPort(Settings.getInstance().getApiPort());
 
 				this.server.addConnector(portUnifiedConnector);
 			} else {
 				// Non-SSL
-				InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
+				InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
 				InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getApiPort());
 				this.server = new Server(endpoint);
 			}

src/main/java/org/qortal/api/DevProxyService.java

@@ -0,0 +1,173 @@
+package org.qortal.api;
+
+import io.swagger.v3.jaxrs2.integration.resources.OpenApiResource;
+import org.eclipse.jetty.http.HttpVersion;
+import org.eclipse.jetty.rewrite.handler.RewriteHandler;
+import org.eclipse.jetty.server.*;
+import org.eclipse.jetty.server.handler.ErrorHandler;
+import org.eclipse.jetty.server.handler.InetAccessHandler;
+import org.eclipse.jetty.servlet.FilterHolder;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.eclipse.jetty.servlet.ServletHolder;
+import org.eclipse.jetty.servlets.CrossOriginFilter;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+import org.glassfish.jersey.server.ResourceConfig;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.qortal.api.resource.AnnotationPostProcessor;
+import org.qortal.api.resource.ApiDefinition;
+import org.qortal.network.Network;
+import org.qortal.repository.DataException;
+import org.qortal.settings.Settings;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import java.io.InputStream;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+
+public class DevProxyService {
+
+	private static DevProxyService instance;
+
+	private final ResourceConfig config;
+	private Server server;
+
+	private DevProxyService() {
+		this.config = new ResourceConfig();
+		this.config.packages("org.qortal.api.proxy.resource", "org.qortal.api.resource");
+		this.config.register(OpenApiResource.class);
+		this.config.register(ApiDefinition.class);
+		this.config.register(AnnotationPostProcessor.class);
+	}
+
+	public static DevProxyService getInstance() {
+		if (instance == null)
+			instance = new DevProxyService();
+
+		return instance;
+	}
+
+	public Iterable<Class<?>> getResources() {
+		return this.config.getClasses();
+	}
+
+	public void start() throws DataException {
+		try {
+			// Create API server
+
+			// SSL support if requested
+			String keystorePathname = Settings.getInstance().getSslKeystorePathname();
+			String keystorePassword = Settings.getInstance().getSslKeystorePassword();
+
+			if (keystorePathname != null && keystorePassword != null) {
+				// SSL version
+				if (!Files.isReadable(Path.of(keystorePathname)))
+					throw new RuntimeException("Failed to start SSL API due to broken keystore");
+
+				// BouncyCastle-specific SSLContext build
+				SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
+				KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
+
+				KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
+
+				try (InputStream keystoreStream = Files.newInputStream(Paths.get(keystorePathname))) {
+					keyStore.load(keystoreStream, keystorePassword.toCharArray());
+				}
+
+				keyManagerFactory.init(keyStore, keystorePassword.toCharArray());
+				sslContext.init(keyManagerFactory.getKeyManagers(), null, new SecureRandom());
+
+				SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
+				sslContextFactory.setSslContext(sslContext);
+
+				this.server = new Server();
+
+				HttpConfiguration httpConfig = new HttpConfiguration();
+				httpConfig.setSecureScheme("https");
+				httpConfig.setSecurePort(Settings.getInstance().getDevProxyPort());
+
+				SecureRequestCustomizer src = new SecureRequestCustomizer();
+				httpConfig.addCustomizer(src);
+
+				HttpConnectionFactory httpConnectionFactory = new HttpConnectionFactory(httpConfig);
+				SslConnectionFactory sslConnectionFactory = new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString());
+
+				ServerConnector portUnifiedConnector = new ServerConnector(this.server,
+						new DetectorConnectionFactory(sslConnectionFactory),
+						httpConnectionFactory);
+				portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
+				portUnifiedConnector.setPort(Settings.getInstance().getDevProxyPort());
+
+				this.server.addConnector(portUnifiedConnector);
+			} else {
+				// Non-SSL
+				InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
+				InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getDevProxyPort());
+				this.server = new Server(endpoint);
+			}
+
+			// Error handler
+			ErrorHandler errorHandler = new ApiErrorHandler();
+			this.server.setErrorHandler(errorHandler);
+
+			// Request logging
+			if (Settings.getInstance().isDevProxyLoggingEnabled()) {
+				RequestLogWriter logWriter = new RequestLogWriter("devproxy-requests.log");
+				logWriter.setAppend(true);
+				logWriter.setTimeZone("UTC");
+				RequestLog requestLog = new CustomRequestLog(logWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT);
+				this.server.setRequestLog(requestLog);
+			}
+
+			// Access handler (currently no whitelist is used)
+			InetAccessHandler accessHandler = new InetAccessHandler();
+			this.server.setHandler(accessHandler);
+
+			// URL rewriting
+			RewriteHandler rewriteHandler = new RewriteHandler();
+			accessHandler.setHandler(rewriteHandler);
+
+			// Context
+			ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
+			context.setContextPath("/");
+			rewriteHandler.setHandler(context);
+
+			// Cross-origin resource sharing
+			FilterHolder corsFilterHolder = new FilterHolder(CrossOriginFilter.class);
+			corsFilterHolder.setInitParameter(CrossOriginFilter.ALLOWED_ORIGINS_PARAM, "*");
+			corsFilterHolder.setInitParameter(CrossOriginFilter.ALLOWED_METHODS_PARAM, "GET, POST, DELETE");
+			corsFilterHolder.setInitParameter(CrossOriginFilter.CHAIN_PREFLIGHT_PARAM, "false");
+			context.addFilter(corsFilterHolder, "/*", null);
+
+			// API servlet
+			ServletContainer container = new ServletContainer(this.config);
+			ServletHolder apiServlet = new ServletHolder(container);
+			apiServlet.setInitOrder(1);
+			context.addServlet(apiServlet, "/*");
+
+			// Start server
+			this.server.start();
+		} catch (Exception e) {
+			// Failed to start
+			throw new DataException("Failed to start developer proxy", e);
+		}
+	}
+
+	public void stop() {
+		try {
+			// Stop server
+			this.server.stop();
+		} catch (Exception e) {
+			// Failed to stop
+		}
+
+		this.server = null;
+		instance = null;
+	}
+
+}

src/main/java/org/qortal/api/DomainMapService.java

@@ -15,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig;
 import org.glassfish.jersey.servlet.ServletContainer;
 import org.qortal.api.resource.AnnotationPostProcessor;
 import org.qortal.api.resource.ApiDefinition;
+import org.qortal.network.Network;
 import org.qortal.settings.Settings;
 
 import javax.net.ssl.KeyManagerFactory;
@@ -68,7 +69,7 @@ public class DomainMapService {
 					throw new RuntimeException("Failed to start SSL API due to broken keystore");
 
 				// BouncyCastle-specific SSLContext build
-				SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
+				SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
 				KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
 
 				KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@@ -98,13 +99,13 @@ public class DomainMapService {
 				ServerConnector portUnifiedConnector = new ServerConnector(this.server,
 						new DetectorConnectionFactory(sslConnectionFactory),
 						httpConnectionFactory);
-				portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
+				portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
 				portUnifiedConnector.setPort(Settings.getInstance().getDomainMapPort());
 
 				this.server.addConnector(portUnifiedConnector);
 			} else {
 				// Non-SSL
-				InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
+				InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
 				InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getDomainMapPort());
 				this.server = new Server(endpoint);
 			}

src/main/java/org/qortal/api/GatewayService.java

@@ -15,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig;
 import org.glassfish.jersey.servlet.ServletContainer;
 import org.qortal.api.resource.AnnotationPostProcessor;
 import org.qortal.api.resource.ApiDefinition;
+import org.qortal.network.Network;
 import org.qortal.settings.Settings;
 
 import javax.net.ssl.KeyManagerFactory;
@@ -68,7 +69,7 @@ public class GatewayService {
 					throw new RuntimeException("Failed to start SSL API due to broken keystore");
 
 				// BouncyCastle-specific SSLContext build
-				SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
+				SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
 				KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
 
 				KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@@ -98,13 +99,13 @@ public class GatewayService {
 				ServerConnector portUnifiedConnector = new ServerConnector(this.server,
 						new DetectorConnectionFactory(sslConnectionFactory),
 						httpConnectionFactory);
-				portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
+				portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
 				portUnifiedConnector.setPort(Settings.getInstance().getGatewayPort());
 
 				this.server.addConnector(portUnifiedConnector);
 			} else {
 				// Non-SSL
-				InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
+				InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
 				InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getGatewayPort());
 				this.server = new Server(endpoint);
 			}

src/main/java/org/qortal/api/HTMLParser.java

@@ -13,7 +13,8 @@ public class HTMLParser {
 
     private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class);
 
-    private String linkPrefix;
+    private String qdnBase;
+    private String qdnBaseWithPath;
     private byte[] data;
     private String qdnContext;
     private String resourceId;
@@ -21,11 +22,13 @@ public class HTMLParser {
     private String identifier;
     private String path;
     private String theme;
+    private boolean usingCustomRouting;
 
-    public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data,
-                      String qdnContext, Service service, String identifier, String theme) {
-        String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : "";
-        this.linkPrefix = usePrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : "";
+    public HTMLParser(String resourceId, String inPath, String prefix, boolean includeResourceIdInPrefix, byte[] data,
+                      String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting) {
+        String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : String.format("/%s",inPath);
+        this.qdnBase = includeResourceIdInPrefix ? String.format("%s/%s", prefix, resourceId) : prefix;
+        this.qdnBaseWithPath = includeResourceIdInPrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : String.format("%s%s", prefix, inPathWithoutFilename);
         this.data = data;
         this.qdnContext = qdnContext;
         this.resourceId = resourceId;
@@ -33,12 +36,12 @@ public class HTMLParser {
         this.identifier = identifier;
         this.path = inPath;
         this.theme = theme;
+        this.usingCustomRouting = usingCustomRouting;
     }
 
     public void addAdditionalHeaderTags() {
         String fileContents = new String(data);
         Document document = Jsoup.parse(fileContents);
-        String baseUrl = this.linkPrefix + "/";
         Elements head = document.getElementsByTag("head");
         if (!head.isEmpty()) {
             // Add q-apps script tag
@@ -52,16 +55,21 @@ public class HTMLParser {
             }
 
             // Escape and add vars
-            String service = this.service.toString().replace("\"","\\\"");
-            String name = this.resourceId != null ? this.resourceId.replace("\"","\\\"") : "";
-            String identifier = this.identifier != null ? this.identifier.replace("\"","\\\"") : "";
-            String path = this.path != null ? this.path.replace("\"","\\\"") : "";
-            String theme = this.theme != null ? this.theme.replace("\"","\\\"") : "";
-            String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\";</script>", this.qdnContext, theme, service, name, identifier, path);
+            String qdnContext = this.qdnContext != null ? this.qdnContext.replace("\\", "").replace("\"","\\\"") : "";
+            String service = this.service.toString().replace("\\", "").replace("\"","\\\"");
+            String name = this.resourceId != null ? this.resourceId.replace("\\", "").replace("\"","\\\"") : "";
+            String identifier = this.identifier != null ? this.identifier.replace("\\", "").replace("\"","\\\"") : "";
+            String path = this.path != null ? this.path.replace("\\", "").replace("\"","\\\"") : "";
+            String theme = this.theme != null ? this.theme.replace("\\", "").replace("\"","\\\"") : "";
+            String qdnBase = this.qdnBase != null ? this.qdnBase.replace("\\", "").replace("\"","\\\"") : "";
+            String qdnBaseWithPath = this.qdnBaseWithPath != null ? this.qdnBaseWithPath.replace("\\", "").replace("\"","\\\"") : "";
+            String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>", qdnContext, theme, service, name, identifier, path, qdnBase, qdnBaseWithPath);
             head.get(0).prepend(qdnContextVar);
 
             // Add base href tag
-            String baseElement = String.format("<base href=\"%s\">", baseUrl);
+            // Exclude the path if this request was routed back to the index automatically
+            String baseHref = this.usingCustomRouting ? this.qdnBase : this.qdnBaseWithPath;
+            String baseElement = String.format("<base href=\"%s/\">", baseHref);
             head.get(0).prepend(baseElement);
 
             // Add meta charset tag
@@ -74,7 +82,7 @@ public class HTMLParser {
     }
 
     public static boolean isHtmlFile(String path) {
-        if (path.endsWith(".html") || path.endsWith(".htm")) {
+        if (path.endsWith(".html") || path.endsWith(".htm") || path.equals("")) {
             return true;
         }
         return false;

src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java

@@ -48,10 +48,10 @@ public class DomainMapResource {
     }
 
     private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
-                                    String inPath, String secret58, String prefix, boolean usePrefix, boolean async) {
+                                    String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async) {
 
         ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
-                secret58, prefix, usePrefix, async, "domainMap", request, response, context);
+                secret58, prefix, includeResourceIdInPrefix, async, "domainMap", request, response, context);
         return renderer.render();
     }
 

src/main/java/org/qortal/api/gateway/resource/GatewayResource.java

@@ -90,7 +90,7 @@ public class GatewayResource {
     }
 
     
-    private HttpServletResponse parsePath(String inPath, String qdnContext, String secret58, boolean usePrefix, boolean async) {
+    private HttpServletResponse parsePath(String inPath, String qdnContext, String secret58, boolean includeResourceIdInPrefix, boolean async) {
 
         if (inPath == null || inPath.equals("")) {
             // Assume not a real file
@@ -157,7 +157,7 @@ public class GatewayResource {
         }
 
         ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(name, ResourceIdType.NAME, service, identifier, outPath,
-                secret58, prefix, usePrefix, async, qdnContext, request, response, context);
+                secret58, prefix, includeResourceIdInPrefix, async, qdnContext, request, response, context);
         return renderer.render();
     }
 

src/main/java/org/qortal/api/model/PollVotes.java

@@ -0,0 +1,56 @@
+package org.qortal.api.model;
+
+import java.util.List;
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlElement;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+
+import org.qortal.data.voting.VoteOnPollData;
+
+@Schema(description = "Poll vote info, including voters")
+// All properties to be converted to JSON via JAX-RS
+@XmlAccessorType(XmlAccessType.FIELD)
+public class PollVotes {
+
+    @Schema(description = "List of individual votes")
+    @XmlElement(name = "votes")
+    public List<VoteOnPollData> votes;
+
+    @Schema(description = "Total number of votes")
+    public Integer totalVotes;
+
+    @Schema(description = "List of vote counts for each option")
+    public List<OptionCount> voteCounts;
+
+    // For JAX-RS
+    protected PollVotes() {
+    }
+
+    public PollVotes(List<VoteOnPollData> votes, Integer totalVotes, List<OptionCount> voteCounts) {
+        this.votes = votes;
+        this.totalVotes = totalVotes;
+        this.voteCounts = voteCounts;
+    }
+
+    @Schema(description = "Vote info")
+    // All properties to be converted to JSON via JAX-RS
+    @XmlAccessorType(XmlAccessType.FIELD)
+    public static class OptionCount {
+        @Schema(description = "Option name")
+        public String optionName;
+
+        @Schema(description = "Vote count")
+        public Integer voteCount;
+
+        // For JAX-RS
+        protected OptionCount() {
+        }
+
+        public OptionCount(String optionName, Integer voteCount) {
+            this.optionName = optionName;
+            this.voteCount = voteCount;
+        }
+    }
+}

src/main/java/org/qortal/api/proxy/resource/DevProxyServerResource.java

@@ -0,0 +1,164 @@
+package org.qortal.api.proxy.resource;
+
+import org.qortal.api.ApiError;
+import org.qortal.api.ApiExceptionFactory;
+import org.qortal.api.HTMLParser;
+import org.qortal.arbitrary.misc.Service;
+import org.qortal.controller.DevProxyManager;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Context;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.ConnectException;
+import java.net.HttpURLConnection;
+import java.net.ProtocolException;
+import java.net.URL;
+import java.util.Enumeration;
+
+
+@Path("/")
+public class DevProxyServerResource {
+
+    @Context HttpServletRequest request;
+    @Context HttpServletResponse response;
+    @Context ServletContext context;
+
+
+    @GET
+    public HttpServletResponse getProxyIndex() {
+        return this.proxy("/");
+    }
+
+    @GET
+    @Path("{path:.*}")
+    public HttpServletResponse getProxyPath(@PathParam("path") String inPath) {
+        return this.proxy(inPath);
+    }
+
+    private HttpServletResponse proxy(String inPath) {
+        try {
+            String source = DevProxyManager.getInstance().getSourceHostAndPort();
+
+            if (!inPath.startsWith("/")) {
+                inPath = "/" + inPath;
+            }
+
+            String queryString = request.getQueryString() != null ? "?" + request.getQueryString() : "";
+
+            // Open URL
+            URL url = new URL(String.format("http://%s%s%s", source, inPath, queryString));
+            HttpURLConnection con = (HttpURLConnection) url.openConnection();
+
+            // Proxy the request data
+            this.proxyRequestToConnection(request, con);
+
+            try {
+                // Make the request and proxy the response code
+                response.setStatus(con.getResponseCode());
+            }
+            catch (ConnectException e) {
+
+                // Tey converting localhost / 127.0.0.1 to IPv6 [::1]
+                if (source.startsWith("localhost") || source.startsWith("127.0.0.1")) {
+                    int port = 80;
+                    String[] parts = source.split(":");
+                    if (parts.length > 1) {
+                        port = Integer.parseInt(parts[1]);
+                    }
+                    source = String.format("[::1]:%d", port);
+                }
+
+                // Retry connection
+                url = new URL(String.format("http://%s%s%s", source, inPath, queryString));
+                con = (HttpURLConnection) url.openConnection();
+                this.proxyRequestToConnection(request, con);
+                response.setStatus(con.getResponseCode());
+            }
+
+            // Proxy the response data back to the caller
+            this.proxyConnectionToResponse(con, response, inPath);
+
+        } catch (IOException e) {
+            throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, e.getMessage());
+        }
+
+        return response;
+    }
+
+    private void proxyRequestToConnection(HttpServletRequest request, HttpURLConnection con) throws ProtocolException {
+        // Proxy the request method
+        con.setRequestMethod(request.getMethod());
+
+        // Proxy the request headers
+        Enumeration<String> headerNames = request.getHeaderNames();
+        while (headerNames.hasMoreElements()) {
+            String headerName = headerNames.nextElement();
+            String headerValue = request.getHeader(headerName);
+            con.setRequestProperty(headerName, headerValue);
+        }
+
+        // TODO: proxy any POST parameters from "request" to "con"
+    }
+
+    private void proxyConnectionToResponse(HttpURLConnection con, HttpServletResponse response, String inPath) throws IOException {
+        // Proxy the response headers
+        for (int i = 0; ; i++) {
+            String headerKey = con.getHeaderFieldKey(i);
+            String headerValue = con.getHeaderField(i);
+            if (headerKey != null && headerValue != null) {
+                response.addHeader(headerKey, headerValue);
+                continue;
+            }
+            break;
+        }
+
+        // Read the response body
+        InputStream inputStream = con.getInputStream();
+        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
+        byte[] buffer = new byte[4096];
+        int bytesRead;
+        while ((bytesRead = inputStream.read(buffer)) != -1) {
+            outputStream.write(buffer, 0, bytesRead);
+        }
+        byte[] data = outputStream.toByteArray(); // TODO: limit file size that can be read into memory
+
+        // Close the streams
+        outputStream.close();
+        inputStream.close();
+
+        // Extract filename
+        String filename = "";
+        if (inPath.contains("/")) {
+            String[] parts = inPath.split("/");
+            if (parts.length > 0) {
+                filename = parts[parts.length - 1];
+            }
+        }
+
+        // Parse and modify output if needed
+        if (HTMLParser.isHtmlFile(filename)) {
+            // HTML file - needs to be parsed
+            HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, "light", true);
+            htmlParser.addAdditionalHeaderTags();
+            response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' ws:; font-src 'self' data:;");
+            response.setContentType(con.getContentType());
+            response.setContentLength(htmlParser.getData().length);
+            response.getOutputStream().write(htmlParser.getData());
+        }
+        else {
+            // Regular file - can be streamed directly
+            response.addHeader("Content-Security-Policy", "default-src 'self'");
+            response.setContentType(con.getContentType());
+            response.setContentLength(data.length);
+            response.getOutputStream().write(data);
+        }
+    }
+
+}

src/main/java/org/qortal/api/resource/ArbitraryResource.java

@@ -65,10 +65,7 @@ import org.qortal.transaction.Transaction.ValidationResult;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.transaction.ArbitraryTransactionTransformer;
 import org.qortal.transform.transaction.TransactionTransformer;
-import org.qortal.utils.ArbitraryTransactionUtils;
-import org.qortal.utils.Base58;
-import org.qortal.utils.NTP;
-import org.qortal.utils.ZipUtils;
+import org.qortal.utils.*;
 
 @Path("/arbitrary")
 @Tag(name = "Arbitrary")
@@ -102,7 +99,9 @@ public class ArbitraryResource {
 			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
 			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
 			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
-			@Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameFilter,
+			@Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly,
+			@Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked,
+			@Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameListFilter,
 			@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
 			@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
 
@@ -125,17 +124,17 @@ public class ArbitraryResource {
 				// Filter using single name
 				names = Arrays.asList(name);
 			}
-			else if (nameFilter != null) {
+			else if (nameListFilter != null) {
 				// Filter using supplied list of names
-				names = ResourceListManager.getInstance().getStringsInList(nameFilter);
+				names = ResourceListManager.getInstance().getStringsInList(nameListFilter);
 				if (names.isEmpty()) {
-					// List doesn't exist or is empty - so there will be no matches
+					// If list is empty (or doesn't exist) we can shortcut with empty response
 					return new ArrayList<>();
 				}
 			}
 
 			List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
-					.getArbitraryResources(service, identifier, names, defaultRes, limit, offset, reverse);
+					.getArbitraryResources(service, identifier, names, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse);
 
 			if (resources == null) {
 				return new ArrayList<>();
@@ -171,22 +170,44 @@ public class ArbitraryResource {
 			@QueryParam("service") Service service,
 			@Parameter(description = "Query (searches both name and identifier fields)") @QueryParam("query") String query,
 			@Parameter(description = "Identifier (searches identifier field only)") @QueryParam("identifier") String identifier,
-			@Parameter(description = "Name (searches name field only)") @QueryParam("name") String name,
+			@Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names,
 			@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
+			@Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly,
 			@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
+			@Parameter(description = "Filter names by list (exact matches only)") @QueryParam("namefilter") String nameListFilter,
+			@Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly,
+			@Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked,
+			@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
+			@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata,
 			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
 			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
-			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
-			@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
-			@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
+			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
 
 			boolean defaultRes = Boolean.TRUE.equals(defaultResource);
 			boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
 
+			List<String> exactMatchNames = new ArrayList<>();
+
+			if (nameListFilter != null) {
+				// Load names from supplied list of names
+				exactMatchNames.addAll(ResourceListManager.getInstance().getStringsInList(nameListFilter));
+
+				// If list is empty (or doesn't exist) we can shortcut with empty response
+				if (exactMatchNames.isEmpty()) {
+					return new ArrayList<>();
+				}
+			}
+
+			// Move names to exact match list, if requested
+			if (exactMatchNamesOnly != null && exactMatchNamesOnly && names != null) {
+				exactMatchNames.addAll(names);
+				names = null;
+			}
+
 			List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
-					.searchArbitraryResources(service, query, identifier, name, usePrefixOnly, defaultRes, limit, offset, reverse);
+					.searchArbitraryResources(service, query, identifier, names, usePrefixOnly, exactMatchNames, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse);
 
 			if (resources == null) {
 				return new ArrayList<>();
@@ -206,67 +227,6 @@ public class ArbitraryResource {
 		}
 	}
 
-	@GET
-	@Path("/resources/names")
-	@Operation(
-			summary = "List arbitrary resources available on chain, grouped by creator's name",
-			responses = {
-					@ApiResponse(
-							content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = ArbitraryResourceInfo.class))
-					)
-			}
-	)
-	@ApiErrors({ApiError.REPOSITORY_ISSUE})
-	public List<ArbitraryResourceNameInfo> getResourcesGroupedByName(
-			@QueryParam("service") Service service,
-			@QueryParam("identifier") String identifier,
-			@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
-			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
-			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
-			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
-			@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
-			@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
-
-		try (final Repository repository = RepositoryManager.getRepository()) {
-
-			// Treat empty identifier as null
-			if (identifier != null && identifier.isEmpty()) {
-				identifier = null;
-			}
-
-			// Ensure that "default" and "identifier" parameters cannot coexist
-			boolean defaultRes = Boolean.TRUE.equals(defaultResource);
-			if (defaultRes == true && identifier != null) {
-				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource");
-			}
-
-			List<ArbitraryResourceNameInfo> creatorNames = repository.getArbitraryRepository()
-					.getArbitraryResourceCreatorNames(service, identifier, defaultRes, limit, offset, reverse);
-
-			for (ArbitraryResourceNameInfo creatorName : creatorNames) {
-				String name = creatorName.name;
-				if (name != null) {
-					List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
-							.getArbitraryResources(service, identifier, Arrays.asList(name), defaultRes, null, null, reverse);
-
-					if (includeStatus != null && includeStatus) {
-						resources = ArbitraryTransactionUtils.addStatusToResources(resources);
-					}
-					if (includeMetadata != null && includeMetadata) {
-						resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
-					}
-
-					creatorName.resources = resources;
-				}
-			}
-
-			return creatorNames;
-
-		} catch (DataException e) {
-			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
-		}
-	}
-
 	@GET
 	@Path("/resource/status/{service}/{name}")
 	@Operation(
@@ -758,12 +718,9 @@ public class ArbitraryResource {
 			}
 	)
 	@SecurityRequirement(name = "apiKey")
-	public ArbitraryResourceMetadata getMetadata(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
-							  @PathParam("service") Service service,
-							  @PathParam("name") String name,
-							  @PathParam("identifier") String identifier) {
-		Security.checkApiCallAllowed(request);
-
+	public ArbitraryResourceMetadata getMetadata(@PathParam("service") Service service,
+							  					 @PathParam("name") String name,
+							  					 @PathParam("identifier") String identifier) {
 		ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier);
 
 		try {
@@ -1216,7 +1173,11 @@ public class ArbitraryResource {
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, error);
 			}
 
-			final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
+			final Long now = NTP.getTime();
+			if (now == null) {
+				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NO_TIME_SYNC);
+			}
+			final Long minLatestBlockTimestamp = now - (60 * 60 * 1000L);
 			if (!Controller.getInstance().isUpToDate(minLatestBlockTimestamp)) {
 				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCKCHAIN_NEEDS_SYNC);
 			}
@@ -1274,7 +1235,7 @@ public class ArbitraryResource {
 					// The actual data will be in a randomly-named subfolder of tempDirectory
 					// Remove hidden folders, i.e. starting with "_", as some systems can add them, e.g. "__MACOSX"
 					String[] files = tempDirectory.toFile().list((parent, child) -> !child.startsWith("_"));
-					if (files.length == 1) { // Single directory or file only
+					if (files != null && files.length == 1) { // Single directory or file only
 						path = Paths.get(tempDirectory.toString(), files[0]).toString();
 					}
 				}
@@ -1306,7 +1267,8 @@ public class ArbitraryResource {
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
 			}
 
-		} catch (DataException | IOException e) {
+		} catch (Exception e) {
+			LOGGER.info("Exception when publishing data: ", e);
 			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
 		}
 	}
@@ -1354,7 +1316,7 @@ public class ArbitraryResource {
 			if (filepath == null || filepath.isEmpty()) {
 				// No file path supplied - so check if this is a single file resource
 				String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal");
-				if (files.length == 1) {
+				if (files != null && files.length == 1) {
 					// This is a single file resource
 					filepath = files[0];
 				}
@@ -1364,20 +1326,50 @@ public class ArbitraryResource {
 				}
 			}
 
-			// TODO: limit file size that can be read into memory
 			java.nio.file.Path path = Paths.get(outputPath.toString(), filepath);
 			if (!Files.exists(path)) {
 				String message = String.format("No file exists at filepath: %s", filepath);
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message);
 			}
 
-			byte[] data = Files.readAllBytes(path);
+			byte[] data;
+			int fileSize = (int)path.toFile().length();
+			int length = fileSize;
+
+			// Parse "Range" header
+			Integer rangeStart = null;
+			Integer rangeEnd = null;
+			String range = request.getHeader("Range");
+			if (range != null) {
+				range = range.replace("bytes=", "");
+				String[] parts = range.split("-");
+				rangeStart = (parts != null && parts.length > 0) ? Integer.parseInt(parts[0]) : null;
+				rangeEnd = (parts != null && parts.length > 1) ? Integer.parseInt(parts[1]) : fileSize;
+			}
+
+			if (rangeStart != null && rangeEnd != null) {
+				// We have a range, so update the requested length
+				length = rangeEnd - rangeStart;
+			}
+
+			if (length < fileSize && encoding == null) {
+				// Partial content requested, and not encoding the data
+				response.setStatus(206);
+				response.addHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd-1, fileSize));
+				data = FilesystemUtils.readFromFile(path.toString(), rangeStart, length);
+			}
+			else {
+				// Full content requested (or encoded data)
+				response.setStatus(200);
+				data = Files.readAllBytes(path); // TODO: limit file size that can be read into memory
+			}
 
 			// Encode the data if requested
 			if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) {
 				data = Base64.encode(data);
 			}
 
+			response.addHeader("Accept-Ranges", "bytes");
 			response.setContentType(context.getMimeType(path.toString()));
 			response.setContentLength(data.length);
 			response.getOutputStream().write(data);

src/main/java/org/qortal/api/resource/BlocksResource.java

@@ -222,14 +222,25 @@ public class BlocksResource {
 		}
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
-		    // Check if the block exists in either the database or archive
-			if (repository.getBlockRepository().getHeightFromSignature(signature) == 0 &&
-					repository.getBlockArchiveRepository().getHeightFromSignature(signature) == 0) {
-				// Not found in either the database or archive
-				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
-            }
+			// Check if the block exists in either the database or archive
+			int height = repository.getBlockRepository().getHeightFromSignature(signature);
+			if (height == 0) {
+				height = repository.getBlockArchiveRepository().getHeightFromSignature(signature);
+				if (height == 0) {
+					// Not found in either the database or archive
+					throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
+				}
+			}
 
-			return repository.getBlockRepository().getTransactionsFromSignature(signature, limit, offset, reverse);
+			List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, height, height);
+
+			// Expand signatures to transactions
+			List<TransactionData> transactions = new ArrayList<>(signatures.size());
+			for (byte[] s : signatures) {
+				transactions.add(repository.getTransactionRepository().fromSignature(s));
+			}
+
+			return transactions;
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		}

src/main/java/org/qortal/api/resource/ChatResource.java

@@ -119,6 +119,75 @@ public class ChatResource {
 		}
 	}
 
+	@GET
+	@Path("/messages/count")
+	@Operation(
+			summary = "Count chat messages",
+			description = "Returns count of CHAT messages that match criteria. Must provide EITHER 'txGroupId' OR two 'involving' addresses.",
+			responses = {
+					@ApiResponse(
+							description = "count of messages",
+							content = @Content(
+									mediaType = MediaType.TEXT_PLAIN,
+									schema = @Schema(
+											type = "integer"
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
+	public int countChatMessages(@QueryParam("before") Long before, @QueryParam("after") Long after,
+										@QueryParam("txGroupId") Integer txGroupId,
+										@QueryParam("involving") List<String> involvingAddresses,
+										@QueryParam("reference") String reference,
+										@QueryParam("chatreference") String chatReference,
+										@QueryParam("haschatreference") Boolean hasChatReference,
+										@QueryParam("sender") String sender,
+										@QueryParam("encoding") Encoding encoding,
+										@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
+										@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
+										@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
+		// Check args meet expectations
+		if ((txGroupId == null && involvingAddresses.size() != 2)
+				|| (txGroupId != null && !involvingAddresses.isEmpty()))
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+
+		// Check any provided addresses are valid
+		if (involvingAddresses.stream().anyMatch(address -> !Crypto.isValidAddress(address)))
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
+
+		if (before != null && before < 1500000000000L)
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+
+		if (after != null && after < 1500000000000L)
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
+
+		byte[] referenceBytes = null;
+		if (reference != null)
+			referenceBytes = Base58.decode(reference);
+
+		byte[] chatReferenceBytes = null;
+		if (chatReference != null)
+			chatReferenceBytes = Base58.decode(chatReference);
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			return repository.getChatRepository().getMessagesMatchingCriteria(
+					before,
+					after,
+					txGroupId,
+					referenceBytes,
+					chatReferenceBytes,
+					hasChatReference,
+					involvingAddresses,
+					sender,
+					encoding,
+					limit, offset, reverse).size();
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
 	@GET
 	@Path("/message/{signature}")
 	@Operation(

src/main/java/org/qortal/api/resource/CrossChainResource.java

@@ -115,6 +115,9 @@ public class CrossChainResource {
 				crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp));
 			}
 
+			// Remove any trades that have had too many failures
+			crossChainTrades = TradeBot.getInstance().removeFailedTrades(repository, crossChainTrades);
+
 			if (limit != null && limit > 0) {
 				// Make sure to not return more than the limit
 				int upperLimit = Math.min(limit, crossChainTrades.size());
@@ -129,6 +132,64 @@ public class CrossChainResource {
 		}
 	}
 
+	@GET
+	@Path("/tradeoffers/hidden")
+	@Operation(
+			summary = "Find cross-chain trade offers that have been hidden due to too many failures",
+			responses = {
+					@ApiResponse(
+							content = @Content(
+									array = @ArraySchema(
+											schema = @Schema(
+													implementation = CrossChainTradeData.class
+											)
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
+	public List<CrossChainTradeData> getHiddenTradeOffers(
+			@Parameter(
+					description = "Limit to specific blockchain",
+					example = "LITECOIN",
+					schema = @Schema(implementation = SupportedBlockchain.class)
+			) @QueryParam("foreignBlockchain") SupportedBlockchain foreignBlockchain) {
+
+		final boolean isExecutable = true;
+		List<CrossChainTradeData> crossChainTrades = new ArrayList<>();
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			Map<ByteArray, Supplier<ACCT>> acctsByCodeHash = SupportedBlockchain.getFilteredAcctMap(foreignBlockchain);
+
+			for (Map.Entry<ByteArray, Supplier<ACCT>> acctInfo : acctsByCodeHash.entrySet()) {
+				byte[] codeHash = acctInfo.getKey().value;
+				ACCT acct = acctInfo.getValue().get();
+
+				List<ATData> atsData = repository.getATRepository().getATsByFunctionality(codeHash, isExecutable, null, null, null);
+
+				for (ATData atData : atsData) {
+					CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atData);
+					if (crossChainTradeData.mode == AcctMode.OFFERING) {
+						crossChainTrades.add(crossChainTradeData);
+					}
+				}
+			}
+
+			// Sort the trades by timestamp
+			crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp));
+
+			// Remove trades that haven't failed
+			crossChainTrades.removeIf(t -> !TradeBot.getInstance().isFailedTrade(repository, t));
+
+			crossChainTrades.stream().forEach(CrossChainResource::decorateTradeDataWithPresence);
+
+			return crossChainTrades;
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
 	@GET
 	@Path("/trade/{ataddress}")
 	@Operation(

src/main/java/org/qortal/api/resource/DeveloperResource.java

@@ -0,0 +1,96 @@
+package org.qortal.api.resource;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.parameters.RequestBody;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import org.qortal.api.ApiError;
+import org.qortal.api.ApiErrors;
+import org.qortal.api.ApiExceptionFactory;
+import org.qortal.controller.DevProxyManager;
+import org.qortal.repository.DataException;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+
+
+@Path("/developer")
+@Tag(name = "Developer Tools")
+public class DeveloperResource {
+
+	@Context HttpServletRequest request;
+	@Context HttpServletResponse response;
+	@Context ServletContext context;
+
+
+	@POST
+	@Path("/proxy/start")
+	@Operation(
+			summary = "Start proxy server, for real time QDN app/website development",
+			requestBody = @RequestBody(
+					description = "Host and port of source webserver to be proxied",
+					required = true,
+					content = @Content(
+							mediaType = MediaType.TEXT_PLAIN,
+							schema = @Schema(
+									type = "string",
+									example = "127.0.0.1:5173"
+							)
+					)
+			),
+			responses = {
+					@ApiResponse(
+							description = "Port number of running server",
+							content = @Content(
+									mediaType = MediaType.TEXT_PLAIN,
+									schema = @Schema(
+											type = "number"
+									)
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.INVALID_CRITERIA})
+	public Integer startProxy(String sourceHostAndPort) {
+		// TODO: API key
+		DevProxyManager devProxyManager = DevProxyManager.getInstance();
+		try {
+			devProxyManager.setSourceHostAndPort(sourceHostAndPort);
+			devProxyManager.start();
+			return devProxyManager.getPort();
+
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, e.getMessage());
+		}
+	}
+
+	@POST
+	@Path("/proxy/stop")
+	@Operation(
+			summary = "Stop proxy server",
+			responses = {
+					@ApiResponse(
+							description = "true if stopped",
+							content = @Content(
+									mediaType = MediaType.TEXT_PLAIN,
+									schema = @Schema(
+											type = "boolean"
+									)
+							)
+					)
+			}
+	)
+	public boolean stopProxy() {
+		DevProxyManager devProxyManager = DevProxyManager.getInstance();
+		devProxyManager.stop();
+		return !devProxyManager.isRunning();
+	}
+
+}

src/main/java/org/qortal/api/resource/NamesResource.java

@@ -47,6 +47,7 @@ import org.qortal.transform.transaction.RegisterNameTransactionTransformer;
 import org.qortal.transform.transaction.SellNameTransactionTransformer;
 import org.qortal.transform.transaction.UpdateNameTransactionTransformer;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Unicode;
 
 @Path("/names")
 @Tag(name = "Names")
@@ -63,19 +64,19 @@ public class NamesResource {
 				description = "registered name info",
 				content = @Content(
 					mediaType = MediaType.APPLICATION_JSON,
-					array = @ArraySchema(schema = @Schema(implementation = NameSummary.class))
+					array = @ArraySchema(schema = @Schema(implementation = NameData.class))
 				)
 			)
 		}
 	)
 	@ApiErrors({ApiError.REPOSITORY_ISSUE})
-	public List<NameSummary> getAllNames(@Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset,
-			@Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
+	public List<NameData> getAllNames(@Parameter(description = "Return only names registered or updated after timestamp") @QueryParam("after") Long after,
+										 @Parameter(ref = "limit") @QueryParam("limit") Integer limit,
+										 @Parameter(ref = "offset") @QueryParam("offset") Integer offset,
+										 @Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			List<NameData> names = repository.getNameRepository().getAllNames(limit, offset, reverse);
 
-			// Convert to summary
-			return names.stream().map(NameSummary::new).collect(Collectors.toList());
+			return repository.getNameRepository().getAllNames(after, limit, offset, reverse);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		}
@@ -135,12 +136,13 @@ public class NamesResource {
 	public NameData getName(@PathParam("name") String name) {
 		try (final Repository repository = RepositoryManager.getRepository()) {
 			NameData nameData;
+			String reducedName = Unicode.sanitize(name);
 
 			if (Settings.getInstance().isLite()) {
 				nameData = LiteNode.getInstance().fetchNameData(name);
 			}
 			else {
-				nameData = repository.getNameRepository().fromName(name);
+				nameData = repository.getNameRepository().fromReducedName(reducedName);
 			}
 
 			if (nameData == null) {
@@ -155,6 +157,41 @@ public class NamesResource {
 		}
 	}
 
+	@GET
+	@Path("/search")
+	@Operation(
+			summary = "Search registered names",
+			responses = {
+					@ApiResponse(
+							description = "registered name info",
+							content = @Content(
+									mediaType = MediaType.APPLICATION_JSON,
+									array = @ArraySchema(schema = @Schema(implementation = NameData.class))
+							)
+					)
+			}
+	)
+	@ApiErrors({ApiError.NAME_UNKNOWN, ApiError.REPOSITORY_ISSUE})
+	public List<NameData> searchNames(@QueryParam("query") String query,
+									  @Parameter(description = "Prefix only (if true, only the beginning of the name is matched)") @QueryParam("prefix") Boolean prefixOnly,
+									  @Parameter(ref = "limit") @QueryParam("limit") Integer limit,
+									  @Parameter(ref = "offset") @QueryParam("offset") Integer offset,
+									  @Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			if (query == null) {
+				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Missing query");
+			}
+
+			boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
+
+			return repository.getNameRepository().searchNames(query, usePrefixOnly, limit, offset, reverse);
+		} catch (ApiException e) {
+			throw e;
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
 
 	@POST
 	@Path("/register")
@@ -410,4 +447,4 @@ public class NamesResource {
 		}
 	}
 
-}
+}

src/main/java/org/qortal/api/resource/PollsResource.java

@@ -0,0 +1,258 @@
+package org.qortal.api.resource;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.parameters.RequestBody;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import org.qortal.api.ApiError;
+import org.qortal.api.ApiErrors;
+import org.qortal.api.ApiExceptionFactory;
+import org.qortal.data.transaction.CreatePollTransactionData;
+import org.qortal.data.transaction.PaymentTransactionData;
+import org.qortal.data.transaction.VoteOnPollTransactionData;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.TransformationException;
+import org.qortal.transform.transaction.CreatePollTransactionTransformer;
+import org.qortal.transform.transaction.PaymentTransactionTransformer;
+import org.qortal.transform.transaction.VoteOnPollTransactionTransformer;
+import org.qortal.utils.Base58;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.media.ArraySchema;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import javax.ws.rs.GET;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.QueryParam;
+import org.qortal.api.ApiException;
+import org.qortal.api.model.PollVotes;
+import org.qortal.data.voting.PollData;
+import org.qortal.data.voting.PollOptionData;
+import org.qortal.data.voting.VoteOnPollData;
+
+@Path("/polls")
+@Tag(name = "Polls")
+public class PollsResource {
+    @Context
+    HttpServletRequest request;
+
+    @GET
+    @Operation(
+            summary = "List all polls",
+            responses = {
+                    @ApiResponse(
+                            description = "poll info",
+                            content = @Content(
+                                    mediaType = MediaType.APPLICATION_JSON,
+                                    array = @ArraySchema(schema = @Schema(implementation = PollData.class))
+                            )
+                    )
+            }
+    )
+    @ApiErrors({ApiError.REPOSITORY_ISSUE})
+    public List<PollData> getAllPolls(@Parameter(
+            ref = "limit"
+    ) @QueryParam("limit") Integer limit, @Parameter(
+            ref = "offset"
+    ) @QueryParam("offset") Integer offset, @Parameter(
+            ref = "reverse"
+    ) @QueryParam("reverse") Boolean reverse) {
+            try (final Repository repository = RepositoryManager.getRepository()) {
+		    List<PollData> allPollData = repository.getVotingRepository().getAllPolls(limit, offset, reverse);
+		    return allPollData;
+            } catch (DataException e) {
+		    throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+            }
+    }
+
+    @GET
+    @Path("/{pollName}")
+    @Operation(
+            summary = "Info on poll",
+            responses = {
+                    @ApiResponse(
+                            description = "poll info",
+                            content = @Content(
+                                    mediaType = MediaType.APPLICATION_JSON,
+                                    schema = @Schema(implementation = PollData.class)
+                            )
+                    )
+            }
+    )
+    @ApiErrors({ApiError.REPOSITORY_ISSUE})
+    public PollData getPollData(@PathParam("pollName") String pollName) {
+            try (final Repository repository = RepositoryManager.getRepository()) {
+                    PollData pollData = repository.getVotingRepository().fromPollName(pollName);
+                    if (pollData == null)
+                            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS);
+
+                    return pollData;
+            } catch (ApiException e) {
+                    throw e;
+            } catch (DataException e) {
+                    throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+            }
+    }
+
+    @GET
+    @Path("/votes/{pollName}")
+    @Operation(
+            summary = "Votes on poll",
+            responses = {
+                    @ApiResponse(
+                            description = "poll votes",
+                            content = @Content(
+                                    mediaType = MediaType.APPLICATION_JSON,
+                                    schema = @Schema(implementation = PollVotes.class)
+                            )
+                    )
+            }
+    )
+    @ApiErrors({ApiError.REPOSITORY_ISSUE})
+    public PollVotes getPollVotes(@PathParam("pollName") String pollName, @QueryParam("onlyCounts") Boolean onlyCounts) {
+            try (final Repository repository = RepositoryManager.getRepository()) {
+                    PollData pollData = repository.getVotingRepository().fromPollName(pollName);
+                    if (pollData == null)
+                            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS);
+
+                    List<VoteOnPollData> votes = repository.getVotingRepository().getVotes(pollName);
+
+                    // Initialize map for counting votes
+                    Map<String, Integer> voteCountMap = new HashMap<>();
+                    for (PollOptionData optionData : pollData.getPollOptions()) {
+                            voteCountMap.put(optionData.getOptionName(), 0);
+                    }
+
+                    int totalVotes = 0;
+                    for (VoteOnPollData vote : votes) {
+                            String selectedOption = pollData.getPollOptions().get(vote.getOptionIndex()).getOptionName();
+                            if (voteCountMap.containsKey(selectedOption)) {
+                                    voteCountMap.put(selectedOption, voteCountMap.get(selectedOption) + 1);
+                                    totalVotes++;
+                            }
+                    }
+
+                    // Convert map to list of VoteInfo
+                    List<PollVotes.OptionCount> voteCounts = voteCountMap.entrySet().stream()
+                        .map(entry -> new PollVotes.OptionCount(entry.getKey(), entry.getValue()))
+                        .collect(Collectors.toList());
+    
+                    if (onlyCounts != null && onlyCounts) {
+                        return new PollVotes(null, totalVotes, voteCounts);
+                    } else {
+                        return new PollVotes(votes, totalVotes, voteCounts);
+                    }
+            } catch (ApiException e) {
+                    throw e;
+            } catch (DataException e) {
+                    throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+            }
+    }
+
+    @POST
+    @Path("/create")
+    @Operation(
+            summary = "Build raw, unsigned, CREATE_POLL transaction",
+            requestBody = @RequestBody(
+                    required = true,
+                    content = @Content(
+                            mediaType = MediaType.APPLICATION_JSON,
+                            schema = @Schema(
+                                    implementation = CreatePollTransactionData.class
+                            )
+                    )
+            ),
+            responses = {
+                    @ApiResponse(
+                            description = "raw, unsigned, CREATE_POLL transaction encoded in Base58",
+                            content = @Content(
+                                    mediaType = MediaType.TEXT_PLAIN,
+                                    schema = @Schema(
+                                            type = "string"
+                                    )
+                            )
+                    )
+            }
+    )
+    @ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE})
+    public String CreatePoll(CreatePollTransactionData transactionData) {
+        if (Settings.getInstance().isApiRestricted())
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION);
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            Transaction transaction = Transaction.fromData(repository, transactionData);
+
+            Transaction.ValidationResult result = transaction.isValidUnconfirmed();
+            if (result != Transaction.ValidationResult.OK)
+                throw TransactionsResource.createTransactionInvalidException(request, result);
+
+            byte[] bytes = CreatePollTransactionTransformer.toBytes(transactionData);
+            return Base58.encode(bytes);
+        } catch (TransformationException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
+        } catch (DataException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+        }
+    }
+
+    @POST
+    @Path("/vote")
+    @Operation(
+            summary = "Build raw, unsigned, VOTE_ON_POLL transaction",
+            requestBody = @RequestBody(
+                    required = true,
+                    content = @Content(
+                            mediaType = MediaType.APPLICATION_JSON,
+                            schema = @Schema(
+                                    implementation = VoteOnPollTransactionData.class
+                            )
+                    )
+            ),
+            responses = {
+                    @ApiResponse(
+                            description = "raw, unsigned, VOTE_ON_POLL transaction encoded in Base58",
+                            content = @Content(
+                                    mediaType = MediaType.TEXT_PLAIN,
+                                    schema = @Schema(
+                                            type = "string"
+                                    )
+                            )
+                    )
+            }
+    )
+    @ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE})
+    public String VoteOnPoll(VoteOnPollTransactionData transactionData) {
+        if (Settings.getInstance().isApiRestricted())
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION);
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            Transaction transaction = Transaction.fromData(repository, transactionData);
+
+            Transaction.ValidationResult result = transaction.isValidUnconfirmed();
+            if (result != Transaction.ValidationResult.OK)
+                throw TransactionsResource.createTransactionInvalidException(request, result);
+
+            byte[] bytes = VoteOnPollTransactionTransformer.toBytes(transactionData);
+            return Base58.encode(bytes);
+        } catch (TransformationException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
+        } catch (DataException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+        }
+    }
+    
+}

src/main/java/org/qortal/api/resource/StatsResource.java

@@ -0,0 +1,70 @@
+package org.qortal.api.resource;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.api.*;
+import org.qortal.block.BlockChain;
+import org.qortal.repository.DataException;
+import org.qortal.repository.Repository;
+import org.qortal.repository.RepositoryManager;
+import org.qortal.utils.Amounts;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.*;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import java.math.BigDecimal;
+import java.util.List;
+
+@Path("/stats")
+@Tag(name = "Stats")
+public class StatsResource {
+
+	private static final Logger LOGGER = LogManager.getLogger(StatsResource.class);
+
+
+	@Context
+	HttpServletRequest request;
+
+	@GET
+	@Path("/supply/circulating")
+	@Operation(
+		summary = "Fetch circulating QORT supply",
+		responses = {
+			@ApiResponse(
+					description = "circulating supply of QORT",
+					content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string", format = "number"))
+			)
+		}
+	)
+	public BigDecimal circulatingSupply() {
+		long total = 0L;
+
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			int currentHeight = repository.getBlockRepository().getBlockchainHeight();
+
+			List<BlockChain.RewardByHeight> rewardsByHeight = BlockChain.getInstance().getBlockRewardsByHeight();
+			int rewardIndex = rewardsByHeight.size() - 1;
+			BlockChain.RewardByHeight rewardInfo = rewardsByHeight.get(rewardIndex);
+
+			for (int height = currentHeight; height > 1; --height) {
+				if (height < rewardInfo.height) {
+					--rewardIndex;
+					rewardInfo = rewardsByHeight.get(rewardIndex);
+				}
+
+				total += rewardInfo.reward;
+			}
+
+			return Amounts.toBigDecimal(total);
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
+}

src/main/java/org/qortal/api/resource/TransactionsResource.java

@@ -215,10 +215,25 @@ public class TransactionsResource {
 		}
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			if (repository.getBlockRepository().getHeightFromSignature(signature) == 0)
-				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
+			// Check if the block exists in either the database or archive
+			int height = repository.getBlockRepository().getHeightFromSignature(signature);
+			if (height == 0) {
+				height = repository.getBlockArchiveRepository().getHeightFromSignature(signature);
+				if (height == 0) {
+					// Not found in either the database or archive
+					throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
+				}
+			}
 
-			return repository.getBlockRepository().getTransactionsFromSignature(signature, limit, offset, reverse);
+			List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, height, height);
+
+			// Expand signatures to transactions
+			List<TransactionData> transactions = new ArrayList<>(signatures.size());
+			for (byte[] s : signatures) {
+				transactions.add(repository.getTransactionRepository().fromSignature(s));
+			}
+
+			return transactions;
 		} catch (ApiException e) {
 			throw e;
 		} catch (DataException e) {

src/main/java/org/qortal/api/restricted/resource/AdminResource.java

@@ -31,10 +31,12 @@ import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
 
+import org.apache.commons.lang3.reflect.FieldUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.apache.logging.log4j.core.LoggerContext;
 import org.apache.logging.log4j.core.appender.RollingFileAppender;
+import org.json.JSONArray;
 import org.qortal.account.Account;
 import org.qortal.account.PrivateKeyAccount;
 import org.qortal.api.*;
@@ -42,6 +44,7 @@ import org.qortal.api.model.ActivitySummary;
 import org.qortal.api.model.NodeInfo;
 import org.qortal.api.model.NodeStatus;
 import org.qortal.block.BlockChain;
+import org.qortal.controller.AutoUpdate;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
 import org.qortal.controller.Synchronizer.SynchronizationResult;
@@ -52,6 +55,7 @@ import org.qortal.network.Network;
 import org.qortal.network.Peer;
 import org.qortal.network.PeerAddress;
 import org.qortal.repository.DataException;
+import org.qortal.repository.ReindexManager;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
@@ -153,6 +157,53 @@ public class AdminResource {
 		return nodeStatus;
 	}
 
+	@GET
+	@Path("/settings")
+	@Operation(
+		summary = "Fetch node settings",
+		responses = {
+			@ApiResponse(
+				content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = Settings.class))
+			)
+		}
+	)
+	public Settings settings() {
+		Settings nodeSettings = Settings.getInstance();
+
+		return nodeSettings;
+	}
+
+	@GET
+	@Path("/settings/{setting}")
+	@Operation(
+			summary = "Fetch a single node setting",
+			responses = {
+					@ApiResponse(
+							content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
+					)
+			}
+	)
+	public String setting(@PathParam("setting") String setting) {
+		try {
+			Object settingValue = FieldUtils.readField(Settings.getInstance(), setting, true);
+			if (settingValue == null) {
+				return "null";
+			}
+			else if (settingValue instanceof String[]) {
+				JSONArray array = new JSONArray(settingValue);
+				return array.toString(4);
+			}
+			else if (settingValue instanceof List) {
+				JSONArray array = new JSONArray((List<Object>) settingValue);
+				return array.toString(4);
+			}
+
+			return settingValue.toString();
+		} catch (IllegalAccessException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA, e);
+		}
+	}
+
 	@GET
 	@Path("/stop")
 	@Operation(
@@ -183,6 +234,37 @@ public class AdminResource {
 		return "true";
 	}
 
+	@GET
+	@Path("/restart")
+	@Operation(
+		summary = "Restart",
+		description = "Restart",
+		responses = {
+			@ApiResponse(
+				description = "\"true\"",
+				content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
+			)
+		}
+	)
+	@SecurityRequirement(name = "apiKey")
+	public String restart(@HeaderParam(Security.API_KEY_HEADER) String apiKey) {
+		Security.checkApiCallAllowed(request);
+
+		new Thread(() -> {
+			// Short sleep to allow HTTP response body to be emitted
+			try {
+				Thread.sleep(1000);
+			} catch (InterruptedException e) {
+				// Not important
+			}
+
+			AutoUpdate.attemptRestart();
+
+		}).start();
+
+		return "true";
+	}
+
 	@GET
 	@Path("/summary")
 	@Operation(
@@ -793,6 +875,48 @@ public class AdminResource {
 		}
 	}
 
+
+	@POST
+	@Path("/repository/reindex")
+	@Operation(
+			summary = "Reindex repository",
+			description = "Rebuilds all transactions and balances from archived blocks. Warning: takes around 1 week, and the core will not function normally during this time. If 'false' is returned, the database may be left in an inconsistent state, requiring another reindex or a bootstrap to correct it.",
+			responses = {
+					@ApiResponse(
+							description = "\"true\"",
+							content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
+					)
+			}
+	)
+	@ApiErrors({ApiError.REPOSITORY_ISSUE})
+	@SecurityRequirement(name = "apiKey")
+	public String reindex(@HeaderParam(Security.API_KEY_HEADER) String apiKey) {
+		Security.checkApiCallAllowed(request);
+
+		try {
+			ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
+
+			blockchainLock.lockInterruptibly();
+
+			try {
+				ReindexManager reindexManager = new ReindexManager();
+				reindexManager.reindex();
+				return "true";
+
+			} catch (DataException e) {
+				LOGGER.info("DataException when reindexing: {}", e.getMessage());
+
+			} finally {
+				blockchainLock.unlock();
+			}
+		} catch (InterruptedException e) {
+			// We couldn't lock blockchain to perform reindex
+			return "false";
+		}
+
+		return "false";
+	}
+
 	@DELETE
 	@Path("/repository")
 	@Operation(

src/main/java/org/qortal/api/restricted/resource/RenderResource.java

@@ -157,10 +157,10 @@ public class RenderResource {
 
 
     private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
-                                    String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String theme) {
+                                    String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme) {
 
         ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
-                secret58, prefix, usePrefix, async, "render", request, response, context);
+                secret58, prefix, includeResourceIdInPrefix, async, "render", request, response, context);
 
         if (theme != null) {
             renderer.setTheme(theme);

src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java

@@ -24,6 +24,7 @@ import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
 import org.qortal.api.model.CrossChainOfferSummary;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
+import org.qortal.controller.tradebot.TradeBot;
 import org.qortal.crosschain.SupportedBlockchain;
 import org.qortal.crosschain.ACCT;
 import org.qortal.crosschain.AcctMode;
@@ -315,7 +316,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 					throw new DataException("Couldn't fetch historic trades from repository");
 
 				for (ATStateData historicAtState : historicAtStates) {
-					CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null);
+					CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null, null);
 
 					if (!isHistoric.test(historicOfferSummary))
 						continue;
@@ -330,8 +331,10 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 		}
 	}
 
-	private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, Long timestamp) throws DataException {
-		CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
+	private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, CrossChainTradeData crossChainTradeData, Long timestamp) throws DataException {
+		if (crossChainTradeData == null) {
+			crossChainTradeData = acct.populateTradeData(repository, atState);
+		}
 
 		long atStateTimestamp;
 
@@ -346,9 +349,16 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
 
 	private static List<CrossChainOfferSummary> produceSummaries(Repository repository, ACCT acct, List<ATStateData> atStates, Long timestamp) throws DataException {
 		List<CrossChainOfferSummary> offerSummaries = new ArrayList<>();
+		for (ATStateData atState : atStates) {
+			CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
 
-		for (ATStateData atState : atStates)
-			offerSummaries.add(produceSummary(repository, acct, atState, timestamp));
+			// Ignore trade if it has failed
+			if (TradeBot.getInstance().isFailedTrade(repository, crossChainTradeData)) {
+				continue;
+			}
+
+			offerSummaries.add(produceSummary(repository, acct, atState, crossChainTradeData, timestamp));
+		}
 
 		return offerSummaries;
 	}

src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java

@@ -54,10 +54,6 @@ public class ArbitraryDataBuilder {
     /**
      * Process transactions, but do not build anything
      * This is useful for checking the status of a given resource
-     *
-     * @throws DataException
-     * @throws IOException
-     * @throws MissingDataException
      */
     public void process() throws DataException, IOException, MissingDataException {
         this.fetchTransactions();
@@ -69,10 +65,6 @@ public class ArbitraryDataBuilder {
 
     /**
      * Build the latest state of a given resource
-     *
-     * @throws DataException
-     * @throws IOException
-     * @throws MissingDataException
      */
     public void build() throws DataException, IOException, MissingDataException {
         this.process();

src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java

@@ -79,7 +79,7 @@ public class ArbitraryDataFile {
         this.signature = signature;
     }
 
-    public ArbitraryDataFile(byte[] fileContent, byte[] signature) throws DataException {
+    public ArbitraryDataFile(byte[] fileContent, byte[] signature, boolean useTemporaryFile) throws DataException {
         if (fileContent == null) {
             LOGGER.error("fileContent is null");
             return;
@@ -90,7 +90,20 @@ public class ArbitraryDataFile {
         this.signature = signature;
         LOGGER.trace(String.format("File digest: %s, size: %d bytes", this.hash58, fileContent.length));
 
-        Path outputFilePath = getOutputFilePath(this.hash58, signature, true);
+        Path outputFilePath;
+        if (useTemporaryFile) {
+            try {
+                outputFilePath = Files.createTempFile("qortalRawData", null);
+                outputFilePath.toFile().deleteOnExit();
+            }
+            catch (IOException e) {
+                throw new DataException(String.format("Unable to write data with hash %s to temporary file: %s", this.hash58, e.getMessage()));
+            }
+        }
+        else {
+            outputFilePath = getOutputFilePath(this.hash58, signature, true);
+        }
+
         File outputFile = outputFilePath.toFile();
         try (FileOutputStream outputStream = new FileOutputStream(outputFile)) {
             outputStream.write(fileContent);
@@ -116,7 +129,7 @@ public class ArbitraryDataFile {
         if (data == null) {
             return null;
         }
-        return new ArbitraryDataFile(data, signature);
+        return new ArbitraryDataFile(data, signature, true);
     }
 
     public static ArbitraryDataFile fromTransactionData(ArbitraryTransactionData transactionData) throws DataException {

src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java

@@ -18,7 +18,7 @@ public class ArbitraryDataFileChunk extends ArbitraryDataFile {
     }
 
     public ArbitraryDataFileChunk(byte[] fileContent, byte[] signature) throws DataException {
-        super(fileContent, signature);
+        super(fileContent, signature, false);
     }
 
     public static ArbitraryDataFileChunk fromHash58(String hash58, byte[] signature) throws DataException {

src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java

@@ -9,7 +9,6 @@ import org.qortal.arbitrary.exception.MissingDataException;
 import org.qortal.arbitrary.misc.Service;
 import org.qortal.controller.arbitrary.ArbitraryDataBuildManager;
 import org.qortal.controller.arbitrary.ArbitraryDataManager;
-import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
 import org.qortal.crypto.AES;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.ArbitraryTransactionData.*;
@@ -19,10 +18,7 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.arbitrary.ArbitraryDataFile.*;
 import org.qortal.settings.Settings;
 import org.qortal.transform.Transformer;
-import org.qortal.utils.ArbitraryTransactionUtils;
-import org.qortal.utils.Base58;
-import org.qortal.utils.FilesystemUtils;
-import org.qortal.utils.ZipUtils;
+import org.qortal.utils.*;
 
 import javax.crypto.BadPaddingException;
 import javax.crypto.IllegalBlockSizeException;
@@ -38,6 +34,9 @@ import java.security.InvalidAlgorithmParameterException;
 import java.security.InvalidKeyException;
 import java.security.NoSuchAlgorithmException;
 import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
 
 public class ArbitraryDataReader {
 
@@ -63,6 +62,10 @@ public class ArbitraryDataReader {
     // The resource being read
     ArbitraryDataResource arbitraryDataResource = null;
 
+    // Track resources that are currently being loaded, to avoid duplicate concurrent builds
+    // TODO: all builds could be handled by the build queue (even synchronous ones), to avoid the need for this
+    private static Map<String, Long> inProgress = Collections.synchronizedMap(new HashMap<>());
+
     public ArbitraryDataReader(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) {
         // Ensure names are always lowercase
         if (resourceIdType == ResourceIdType.NAME) {
@@ -157,9 +160,6 @@ public class ArbitraryDataReader {
      * If no exception is thrown, you can then use getFilePath() to access the data immediately after returning
      *
      * @param overwrite - set to true to force rebuild an existing cache
-     * @throws IOException
-     * @throws DataException
-     * @throws MissingDataException
      */
     public void loadSynchronously(boolean overwrite) throws DataException, IOException, MissingDataException {
         try {
@@ -173,6 +173,12 @@ public class ArbitraryDataReader {
 
             this.arbitraryDataResource = this.createArbitraryDataResource();
 
+            // Don't allow duplicate loads
+            if (!this.canStartLoading()) {
+                LOGGER.debug("Skipping duplicate load of {}", this.arbitraryDataResource);
+                return;
+            }
+
             this.preExecute();
             this.deleteExistingFiles();
             this.fetch();
@@ -200,6 +206,7 @@ public class ArbitraryDataReader {
 
     private void preExecute() throws DataException {
         ArbitraryDataBuildManager.getInstance().setBuildInProgress(true);
+
         this.checkEnabled();
         this.createWorkingDirectory();
         this.createUncompressedDirectory();
@@ -207,6 +214,9 @@ public class ArbitraryDataReader {
 
     private void postExecute() {
         ArbitraryDataBuildManager.getInstance().setBuildInProgress(false);
+
+        this.arbitraryDataResource = this.createArbitraryDataResource();
+        ArbitraryDataReader.inProgress.remove(this.arbitraryDataResource.getUniqueKey());
     }
 
     private void checkEnabled() throws DataException {
@@ -215,6 +225,17 @@ public class ArbitraryDataReader {
         }
     }
 
+    private boolean canStartLoading() {
+        // Avoid duplicate builds if we're already loading this resource
+        String uniqueKey = this.arbitraryDataResource.getUniqueKey();
+        if (ArbitraryDataReader.inProgress.containsKey(uniqueKey)) {
+            return false;
+        }
+        ArbitraryDataReader.inProgress.put(uniqueKey, NTP.getTime());
+
+        return true;
+    }
+
     private void createWorkingDirectory() throws DataException {
         try {
             Files.createDirectories(this.workingPath);
@@ -226,7 +247,6 @@ public class ArbitraryDataReader {
     /**
      * Working directory should only be deleted on failure, since it is currently used to
      * serve a cached version of the resource for subsequent requests.
-     * @throws IOException
      */
     private void deleteWorkingDirectory() {
         try {
@@ -306,7 +326,7 @@ public class ArbitraryDataReader {
                 break;
 
             default:
-                throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType.toString()));
+                throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType));
         }
     }
 
@@ -371,9 +391,12 @@ public class ArbitraryDataReader {
         // Load data file(s)
         ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
         ArbitraryTransactionUtils.checkAndRelocateMiscFiles(transactionData);
+        if (arbitraryDataFile == null) {
+            throw new DataException(String.format("arbitraryDataFile is null"));
+        }
 
         if (!arbitraryDataFile.allFilesExist()) {
-            if (ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())) {
+            if (ListUtils.isNameBlocked(transactionData.getName())) {
                 throw new DataException(
                         String.format("Unable to request missing data for file %s because the name is blocked", arbitraryDataFile));
             } else {
@@ -446,6 +469,7 @@ public class ArbitraryDataReader {
                 Path unencryptedPath = Paths.get(this.workingPath.toString(), "zipped.zip");
                 SecretKey aesKey = new SecretKeySpec(secret, 0, secret.length, "AES");
                 AES.decryptFile(algorithm, aesKey, this.filePath.toString(), unencryptedPath.toString());
+                LOGGER.debug("Finished decrypting {} using algorithm {}", this.arbitraryDataResource, algorithm);
 
                 // Replace filePath pointer with the encrypted file path
                 // Don't delete the original ArbitraryDataFile, as this is handled in the cleanup phase
@@ -480,7 +504,9 @@ public class ArbitraryDataReader {
 
             // Handle each type of compression
             if (compression == Compression.ZIP) {
+                LOGGER.debug("Unzipping {}...", this.arbitraryDataResource);
                 ZipUtils.unzip(this.filePath.toString(), this.uncompressedPath.getParent().toString());
+                LOGGER.debug("Finished unzipping {}", this.arbitraryDataResource);
             }
             else if (compression == Compression.NONE) {
                 Files.createDirectories(this.uncompressedPath);
@@ -516,10 +542,12 @@ public class ArbitraryDataReader {
 
     private void validate() throws IOException, DataException {
         if (this.service.isValidationRequired()) {
+            LOGGER.debug("Validating {}...", this.arbitraryDataResource);
             Service.ValidationResult result = this.service.validate(this.filePath);
             if (result != Service.ValidationResult.OK) {
                 throw new DataException(String.format("Validation of %s failed: %s", this.service, result.toString()));
             }
+            LOGGER.debug("Finished validating {}", this.arbitraryDataResource);
         }
     }
 

src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java

@@ -40,7 +40,7 @@ public class ArbitraryDataRenderer {
     private String inPath;
     private final String secret58;
     private final String prefix;
-    private final boolean usePrefix;
+    private final boolean includeResourceIdInPrefix;
     private final boolean async;
     private final String qdnContext;
     private final HttpServletRequest request;
@@ -48,7 +48,7 @@ public class ArbitraryDataRenderer {
     private final ServletContext context;
 
     public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
-                                 String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String qdnContext,
+                                 String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String qdnContext,
                                  HttpServletRequest request, HttpServletResponse response, ServletContext context) {
 
         this.resourceId = resourceId;
@@ -58,7 +58,7 @@ public class ArbitraryDataRenderer {
         this.inPath = inPath;
         this.secret58 = secret58;
         this.prefix = prefix;
-        this.usePrefix = usePrefix;
+        this.includeResourceIdInPrefix = includeResourceIdInPrefix;
         this.async = async;
         this.qdnContext = qdnContext;
         this.request = request;
@@ -67,8 +67,8 @@ public class ArbitraryDataRenderer {
     }
 
     public HttpServletResponse render() {
-        if (!inPath.startsWith(File.separator)) {
-            inPath = File.separator + inPath;
+        if (!inPath.startsWith("/")) {
+            inPath = "/" + inPath;
         }
 
         // Don't render data if QDN is disabled
@@ -126,7 +126,8 @@ public class ArbitraryDataRenderer {
         try {
             String filename = this.getFilename(unzippedPath, inPath);
             Path filePath = Paths.get(unzippedPath, filename);
-
+            boolean usingCustomRouting = false;
+            
             // If the file doesn't exist, we may need to route the request elsewhere, or cleanup
             if (!Files.exists(filePath)) {
                 if (inPath.equals("/")) {
@@ -148,6 +149,7 @@ public class ArbitraryDataRenderer {
                             // Forward request to index file
                             filePath = indexPath;
                             filename = indexFile;
+                            usingCustomRouting = true;
                             break;
                         }
                     }
@@ -157,7 +159,7 @@ public class ArbitraryDataRenderer {
             if (HTMLParser.isHtmlFile(filename)) {
                 // HTML file - needs to be parsed
                 byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory
-                HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data, qdnContext, service, identifier, theme);
+                HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting);
                 htmlParser.addAdditionalHeaderTags();
                 response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;");
                 response.setContentType(context.getMimeType(filename));

src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java

@@ -11,13 +11,13 @@ import org.qortal.controller.arbitrary.ArbitraryDataManager;
 import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
 import org.qortal.data.arbitrary.ArbitraryResourceStatus;
 import org.qortal.data.transaction.ArbitraryTransactionData;
-import org.qortal.list.ResourceListManager;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.ArbitraryTransactionUtils;
 import org.qortal.utils.FilesystemUtils;
+import org.qortal.utils.ListUtils;
 import org.qortal.utils.NTP;
 
 import java.io.IOException;
@@ -74,8 +74,7 @@ public class ArbitraryDataResource {
         }
 
         // Check if the name is blocked
-        if (ResourceListManager.getInstance()
-                .listContains("blockedNames", this.resourceId, false)) {
+        if (ListUtils.isNameBlocked(this.resourceId)) {
             return new ArbitraryResourceStatus(Status.BLOCKED, this.localChunkCount, this.totalChunkCount);
         }
 
@@ -151,6 +150,9 @@ public class ArbitraryDataResource {
 
             for (ArbitraryTransactionData transactionData : transactionDataList) {
                 ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
+                if (arbitraryDataFile == null) {
+                    continue;
+                }
 
                 // Delete any chunks or complete files from each transaction
                 arbitraryDataFile.deleteAll(deleteMetadata);

src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java

@@ -191,6 +191,14 @@ public class ArbitraryDataTransactionBuilder {
                 return Method.PUT;
             }
 
+            // We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain
+            final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false);
+            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
+            if (shouldUseOnChainData) {
+                LOGGER.info("Data size is small enough to go on chain - using PUT");
+                return Method.PUT;
+            }
+
             // State is appropriate for a PATCH transaction
             return Method.PATCH;
         }
@@ -297,6 +305,9 @@ public class ArbitraryDataTransactionBuilder {
     }
 
     private boolean isMetadataEqual(ArbitraryDataTransactionMetadata existingMetadata) {
+        if (existingMetadata == null) {
+            return !this.hasMetadata();
+        }
         if (!Objects.equals(existingMetadata.getTitle(), this.title)) {
             return false;
         }
@@ -312,6 +323,10 @@ public class ArbitraryDataTransactionBuilder {
         return true;
     }
 
+    private boolean hasMetadata() {
+        return (this.title != null || this.description != null || this.category != null || this.tags != null);
+    }
+
     public void computeNonce() throws DataException {
         if (this.arbitraryTransactionData == null) {
             throw new DataException("Arbitrary transaction data is required to compute nonce");

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java

@@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.json.JSONException;
 import org.qortal.repository.DataException;
 
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileWriter;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
 
@@ -34,7 +36,7 @@ public class ArbitraryDataMetadata {
         this.filePath = filePath;
     }
 
-    protected void readJson() throws DataException {
+    protected void readJson() throws DataException, JSONException {
         // To be overridden
     }
 
@@ -44,8 +46,13 @@ public class ArbitraryDataMetadata {
 
 
     public void read() throws IOException, DataException {
-        this.loadJson();
-        this.readJson();
+        try {
+            this.loadJson();
+            this.readJson();
+
+        } catch (JSONException e) {
+            throw new DataException(String.format("Unable to read JSON at path %s: %s", this.filePath, e.getMessage()));
+        }
     }
 
     public void write() throws IOException, DataException {
@@ -58,6 +65,10 @@ public class ArbitraryDataMetadata {
         writer.close();
     }
 
+    public void delete() throws IOException {
+        Files.delete(this.filePath);
+    }
+
 
     protected void loadJson() throws IOException {
         File metadataFile = new File(this.filePath.toString());
@@ -65,7 +76,7 @@ public class ArbitraryDataMetadata {
             throw new IOException(String.format("Metadata file doesn't exist: %s", this.filePath.toString()));
         }
 
-        this.jsonString = new String(Files.readAllBytes(this.filePath));
+        this.jsonString = new String(Files.readAllBytes(this.filePath), StandardCharsets.UTF_8);
     }
 
 

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java

@@ -1,5 +1,6 @@
 package org.qortal.arbitrary.metadata;
 
+import org.json.JSONException;
 import org.json.JSONObject;
 import org.qortal.repository.DataException;
 import org.qortal.utils.Base58;
@@ -22,7 +23,7 @@ public class ArbitraryDataMetadataCache extends ArbitraryDataQortalMetadata {
     }
 
     @Override
-    protected void readJson() throws DataException {
+    protected void readJson() throws DataException, JSONException {
         if (this.jsonString == null) {
             throw new DataException("Patch JSON string is null");
         }

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java

@@ -3,6 +3,7 @@ package org.qortal.arbitrary.metadata;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.json.JSONArray;
+import org.json.JSONException;
 import org.json.JSONObject;
 import org.qortal.arbitrary.ArbitraryDataDiff.*;
 import org.qortal.repository.DataException;
@@ -40,7 +41,7 @@ public class ArbitraryDataMetadataPatch extends ArbitraryDataQortalMetadata {
     }
 
     @Override
-    protected void readJson() throws DataException {
+    protected void readJson() throws DataException, JSONException {
         if (this.jsonString == null) {
             throw new DataException("Patch JSON string is null");
         }

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java

@@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.json.JSONException;
 import org.qortal.repository.DataException;
 
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileWriter;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -46,20 +48,6 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
         return null;
     }
 
-    protected void readJson() throws DataException {
-        // To be overridden
-    }
-
-    protected void buildJson() {
-        // To be overridden
-    }
-
-
-    @Override
-    public void read() throws IOException, DataException {
-        this.loadJson();
-        this.readJson();
-    }
 
     @Override
     public void write() throws IOException, DataException {
@@ -82,7 +70,7 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
             throw new IOException(String.format("Patch file doesn't exist: %s", path.toString()));
         }
 
-        this.jsonString = new String(Files.readAllBytes(path));
+        this.jsonString = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
     }
 
 
@@ -94,9 +82,4 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
         }
     }
 
-
-    public String getJsonString() {
-        return this.jsonString;
-    }
-
 }

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java

@@ -1,11 +1,13 @@
 package org.qortal.arbitrary.metadata;
 
 import org.json.JSONArray;
+import org.json.JSONException;
 import org.json.JSONObject;
 import org.qortal.arbitrary.misc.Category;
 import org.qortal.repository.DataException;
 import org.qortal.utils.Base58;
 
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -33,7 +35,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
     }
 
     @Override
-    protected void readJson() throws DataException {
+    protected void readJson() throws DataException, JSONException {
         if (this.jsonString == null) {
             throw new DataException("Transaction metadata JSON string is null");
         }
@@ -216,6 +218,25 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
 
     // Static helper methods
 
+    public static String trimUTF8String(String string, int maxLength) {
+        byte[] inputBytes = string.getBytes(StandardCharsets.UTF_8);
+        int length = Math.min(inputBytes.length, maxLength);
+        byte[] outputBytes = new byte[length];
+
+        System.arraycopy(inputBytes, 0, outputBytes, 0, length);
+        String result = new String(outputBytes, StandardCharsets.UTF_8);
+
+        // check if last character is truncated
+        int lastIndex = result.length() - 1;
+
+        if (lastIndex > 0 && result.charAt(lastIndex) != string.charAt(lastIndex)) {
+            // last character is truncated so remove the last character
+            return result.substring(0, lastIndex);
+        }
+
+        return result;
+    }
+
     public static String limitTitle(String title) {
         if (title == null) {
             return null;
@@ -224,7 +245,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
             return null;
         }
 
-        return title.substring(0, Math.min(title.length(), MAX_TITLE_LENGTH));
+        return trimUTF8String(title, MAX_TITLE_LENGTH);
     }
 
     public static String limitDescription(String description) {
@@ -235,7 +256,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
             return null;
         }
 
-        return description.substring(0, Math.min(description.length(), MAX_DESCRIPTION_LENGTH));
+        return trimUTF8String(description, MAX_DESCRIPTION_LENGTH);
     }
 
     public static List<String> limitTags(List<String> tags) {

src/main/java/org/qortal/arbitrary/misc/Service.java

@@ -9,7 +9,6 @@ import org.qortal.utils.FilesystemUtils;
 import java.io.File;
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.*;
@@ -20,9 +19,9 @@ import static java.util.Arrays.stream;
 import static java.util.stream.Collectors.toMap;
 
 public enum Service {
-    AUTO_UPDATE(1, false, null, false, null),
-    ARBITRARY_DATA(100, false, null, false, null),
-    QCHAT_ATTACHMENT(120, true, 1024*1024L, true, null) {
+    AUTO_UPDATE(1, false, null, false, false, null),
+    ARBITRARY_DATA(100, false, null, false, false, null),
+    QCHAT_ATTACHMENT(120, true, 1024*1024L, true, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -47,7 +46,14 @@ public enum Service {
             return ValidationResult.OK;
         }
     },
-    WEBSITE(200, true, null, false, null) {
+    QCHAT_ATTACHMENT_PRIVATE(121, true, 1024*1024L, true, true, null),
+    ATTACHMENT(130, false, 50*1024*1024L, true, false, null),
+    ATTACHMENT_PRIVATE(131, true, 50*1024*1024L, true, true, null),
+    FILE(140, false, null, true, false, null),
+    FILE_PRIVATE(141, true, null, true, true, null),
+    FILES(150, false, null, false, false, null),
+    CHAIN_DATA(160, true, 239L, true, false, null),
+    WEBSITE(200, true, null, false, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -69,23 +75,30 @@ public enum Service {
             return ValidationResult.MISSING_INDEX_FILE;
         }
     },
-    GIT_REPOSITORY(300, false, null, false, null),
-    IMAGE(400, true, 10*1024*1024L, true, null),
-    THUMBNAIL(410, true, 500*1024L, true, null),
-    QCHAT_IMAGE(420, true, 500*1024L, true, null),
-    VIDEO(500, false, null, true, null),
-    AUDIO(600, false, null, true, null),
-    QCHAT_AUDIO(610, true, 10*1024*1024L, true, null),
-    QCHAT_VOICE(620, true, 10*1024*1024L, true, null),
-    BLOG(700, false, null, false, null),
-    BLOG_POST(777, false, null, true, null),
-    BLOG_COMMENT(778, false, null, true, null),
-    DOCUMENT(800, false, null, true, null),
-    LIST(900, true, null, true, null),
-    PLAYLIST(910, true, null, true, null),
-    APP(1000, true, 50*1024*1024L, false, null),
-    METADATA(1100, false, null, true, null),
-    JSON(1110, true, 25*1024L, true, null) {
+    GIT_REPOSITORY(300, false, null, false, false, null),
+    IMAGE(400, true, 10*1024*1024L, true, false, null),
+    IMAGE_PRIVATE(401, true, 10*1024*1024L, true, true, null),
+    THUMBNAIL(410, true, 500*1024L, true, false, null),
+    QCHAT_IMAGE(420, true, 500*1024L, true, false, null),
+    VIDEO(500, false, null, true, false, null),
+    VIDEO_PRIVATE(501, true, null, true, true, null),
+    AUDIO(600, false, null, true, false, null),
+    AUDIO_PRIVATE(601, true, null, true, true, null),
+    QCHAT_AUDIO(610, true, 10*1024*1024L, true, false, null),
+    QCHAT_VOICE(620, true, 10*1024*1024L, true, false, null),
+    VOICE(630, true, 10*1024*1024L, true, false, null),
+    VOICE_PRIVATE(631, true, 10*1024*1024L, true, true, null),
+    PODCAST(640, false, null, true, false, null),
+    BLOG(700, false, null, false, false, null),
+    BLOG_POST(777, false, null, true, false, null),
+    BLOG_COMMENT(778, true, 500*1024L, true, false, null),
+    DOCUMENT(800, false, null, true, false, null),
+    DOCUMENT_PRIVATE(801, true, null, true, true, null),
+    LIST(900, true, null, true, false, null),
+    PLAYLIST(910, true, null, true, false, null),
+    APP(1000, true, 50*1024*1024L, false, false, null),
+    METADATA(1100, false, null, true, false, null),
+    JSON(1110, true, 25*1024L, true, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -94,7 +107,7 @@ public enum Service {
             }
 
             // Require valid JSON
-            byte[] data = FilesystemUtils.getSingleFileContents(path);
+            byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024);
             String json = new String(data, StandardCharsets.UTF_8);
             try {
                 objectMapper.readTree(json);
@@ -104,7 +117,7 @@ public enum Service {
             }
         }
     },
-    GIF_REPOSITORY(1200, true, 25*1024*1024L, false, null) {
+    GIF_REPOSITORY(1200, true, 25*1024*1024L, false, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -139,12 +152,31 @@ public enum Service {
             }
             return ValidationResult.OK;
         }
-    };
+    },
+    STORE(1300, false, null, true, false, null),
+    PRODUCT(1310, false, null, true, false, null),
+    OFFER(1330, false, null, true, false, null),
+    COUPON(1340, false, null, true, false, null),
+    CODE(1400, false, null, true, false, null),
+    PLUGIN(1410, false, null, true, false, null),
+    EXTENSION(1420, false, null, true, false, null),
+    GAME(1500, false, null, false, false, null),
+    ITEM(1510, false, null, true, false, null),
+    NFT(1600, false, null, true, false, null),
+    DATABASE(1700, false, null, false, false, null),
+    SNAPSHOT(1710, false, null, false, false, null),
+    COMMENT(1800, true, 500*1024L, true, false, null),
+    CHAIN_COMMENT(1810, true, 239L, true, false, null),
+    MAIL(1900, true, 1024*1024L, true, false, null),
+    MAIL_PRIVATE(1901, true, 1024*1024L, true, true, null),
+    MESSAGE(1910, true, 1024*1024L, true, false, null),
+    MESSAGE_PRIVATE(1911, true, 1024*1024L, true, true, null);
 
     public final int value;
     private final boolean requiresValidation;
     private final Long maxSize;
     private final boolean single;
+    private final boolean isPrivate;
     private final List<String> requiredKeys;
 
     private static final Map<Integer, Service> map = stream(Service.values())
@@ -153,11 +185,15 @@ public enum Service {
     // For JSON validation
     private static final ObjectMapper objectMapper = new ObjectMapper();
 
-    Service(int value, boolean requiresValidation, Long maxSize, boolean single, List<String> requiredKeys) {
+    private static final String encryptedDataPrefix = "qortalEncryptedData";
+    private static final String encryptedGroupDataPrefix = "qortalGroupEncryptedData";
+
+    Service(int value, boolean requiresValidation, Long maxSize, boolean single, boolean isPrivate, List<String> requiredKeys) {
         this.value = value;
         this.requiresValidation = requiresValidation;
         this.maxSize = maxSize;
         this.single = single;
+        this.isPrivate = isPrivate;
         this.requiredKeys = requiredKeys;
     }
 
@@ -166,7 +202,9 @@ public enum Service {
             return ValidationResult.OK;
         }
 
-        byte[] data = FilesystemUtils.getSingleFileContents(path);
+        // Load the first 25KB of data. This only needs to be long enough to check the prefix
+        // and also to allow for possible additional future validation of smaller files.
+        byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024);
         long size = FilesystemUtils.getDirectorySize(path);
 
         // Validate max size if needed
@@ -181,6 +219,17 @@ public enum Service {
             return ValidationResult.INVALID_FILE_COUNT;
         }
 
+        // Validate private data for single file resources
+        if (this.single) {
+            String dataString = new String(data, StandardCharsets.UTF_8);
+            if (this.isPrivate && !dataString.startsWith(encryptedDataPrefix) && !dataString.startsWith(encryptedGroupDataPrefix)) {
+                return ValidationResult.DATA_NOT_ENCRYPTED;
+            }
+            if (!this.isPrivate && (dataString.startsWith(encryptedDataPrefix) || dataString.startsWith(encryptedGroupDataPrefix))) {
+                return ValidationResult.DATA_ENCRYPTED;
+            }
+        }
+
         // Validate required keys if needed
         if (this.requiredKeys != null) {
             if (data == null) {
@@ -199,7 +248,12 @@ public enum Service {
     }
 
     public boolean isValidationRequired() {
-        return this.requiresValidation;
+        // We must always validate single file resources, to ensure they are actually a single file
+        return this.requiresValidation || this.single;
+    }
+
+    public boolean isPrivate() {
+        return this.isPrivate;
     }
 
     public static Service valueOf(int value) {
@@ -207,10 +261,41 @@ public enum Service {
     }
 
     public static JSONObject toJsonObject(byte[] data) {
-        String dataString = new String(data);
+        String dataString = new String(data, StandardCharsets.UTF_8);
         return new JSONObject(dataString);
     }
 
+    public static List<Service> publicServices() {
+        List<Service> privateServices = new ArrayList<>();
+        for (Service service : Service.values()) {
+            if (!service.isPrivate) {
+                privateServices.add(service);
+            }
+        }
+        return privateServices;
+    }
+
+    /**
+     * Fetch a list of Service objects that require encrypted data.
+     *
+     * These can ultimately be used to help inform the cleanup manager
+     * on the best order to delete files when the node runs out of space.
+     * Public data should be given priority over private data (unless
+     * this node is part of a data market contract for that data - this
+     * isn't developed yet).
+     *
+     * @return a list of Service objects that require encrypted data.
+     */
+    public static List<Service> privateServices() {
+        List<Service> privateServices = new ArrayList<>();
+        for (Service service : Service.values()) {
+            if (service.isPrivate) {
+                privateServices.add(service);
+            }
+        }
+        return privateServices;
+    }
+
     public enum ValidationResult {
         OK(1),
         MISSING_KEYS(2),
@@ -220,7 +305,9 @@ public enum Service {
         INVALID_FILE_EXTENSION(6),
         MISSING_DATA(7),
         INVALID_FILE_COUNT(8),
-        INVALID_CONTENT(9);
+        INVALID_CONTENT(9),
+        DATA_NOT_ENCRYPTED(10),
+        DATA_ENCRYPTED(10);
 
         public final int value;
 

src/main/java/org/qortal/block/Block.java

@@ -1213,10 +1213,18 @@ public class Block {
 				// Apply fix for block 212937 but fix will be rolled back before we exit method
 				Block212937.processFix(this);
 			}
+			else if (this.blockData.getHeight() == 1333492) {
+				// Apply fix for block 1333492 but fix will be rolled back before we exit method
+				Block1333492.processFix(this);
+			}
 			else if (InvalidNameRegistrationBlocks.isAffectedBlock(this.blockData.getHeight())) {
 				// Apply fix for affected name registration blocks, but fix will be rolled back before we exit method
 				InvalidNameRegistrationBlocks.processFix(this);
 			}
+			else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) {
+				// Apply fix for affected balance blocks, but fix will be rolled back before we exit method
+				InvalidBalanceBlocks.processFix(this);
+			}
 
 			for (Transaction transaction : this.getTransactions()) {
 				TransactionData transactionData = transaction.getTransactionData();
@@ -1464,12 +1472,21 @@ public class Block {
 			// Distribute block rewards, including transaction fees, before transactions processed
 			processBlockRewards();
 
-			if (this.blockData.getHeight() == 212937)
+			if (this.blockData.getHeight() == 212937) {
 				// Apply fix for block 212937
 				Block212937.processFix(this);
-
-			else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height())
+			}
+			else if (this.blockData.getHeight() == 1333492) {
+				// Apply fix for block 1333492
+				Block1333492.processFix(this);
+			}
+			else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) {
+				// Apply fix for affected balance blocks
+				InvalidBalanceBlocks.processFix(this);
+			}
+			else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) {
 				SelfSponsorshipAlgoV1Block.processAccountPenalties(this);
+			}
 		}
 
 		// We're about to (test-)process a batch of transactions,
@@ -1686,12 +1703,14 @@ public class Block {
 					transactionData.getSignature());
 			this.repository.getBlockRepository().save(blockTransactionData);
 
-			// Update transaction's height in repository
+			// Update transaction's height in repository and local transactionData
 			transactionRepository.updateBlockHeight(transactionData.getSignature(), this.blockData.getHeight());
-
-			// Update local transactionData's height too
 			transaction.getTransactionData().setBlockHeight(this.blockData.getHeight());
 
+			// Update transaction's sequence in repository and local transactionData
+			transactionRepository.updateBlockSequence(transactionData.getSignature(), sequence);
+			transaction.getTransactionData().setBlockSequence(sequence);
+
 			// No longer unconfirmed
 			transactionRepository.confirmTransaction(transactionData.getSignature());
 
@@ -1724,12 +1743,21 @@ public class Block {
 			// Invalidate expandedAccounts as they may have changed due to orphaning TRANSFER_PRIVS transactions, etc.
 			this.cachedExpandedAccounts = null;
 
-			if (this.blockData.getHeight() == 212937)
+			if (this.blockData.getHeight() == 212937) {
 				// Revert fix for block 212937
 				Block212937.orphanFix(this);
-
-			else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height())
+			}
+			else if (this.blockData.getHeight() == 1333492) {
+				// Revert fix for block 1333492
+				Block1333492.orphanFix(this);
+			}
+			else if (InvalidBalanceBlocks.isAffectedBlock(this.blockData.getHeight())) {
+				// Revert fix for affected balance blocks
+				InvalidBalanceBlocks.orphanFix(this);
+			}
+			else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height()) {
 				SelfSponsorshipAlgoV1Block.orphanAccountPenalties(this);
+			}
 
 			// Block rewards, including transaction fees, removed after transactions undone
 			orphanBlockRewards();
@@ -1778,6 +1806,9 @@ public class Block {
 
 				// Unset height
 				transactionRepository.updateBlockHeight(transactionData.getSignature(), null);
+
+				// Unset sequence
+				transactionRepository.updateBlockSequence(transactionData.getSignature(), null);
 			}
 
 			transactionRepository.deleteParticipants(transactionData);

src/main/java/org/qortal/block/Block1333492.java

@@ -0,0 +1,101 @@
+package org.qortal.block;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.eclipse.persistence.jaxb.JAXBContextFactory;
+import org.eclipse.persistence.jaxb.UnmarshallerProperties;
+import org.qortal.data.account.AccountBalanceData;
+import org.qortal.repository.DataException;
+
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.UnmarshalException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import java.io.InputStream;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Block 1333492
+ * <p>
+ * As described in InvalidBalanceBlocks.java, legacy bugs caused a small drift in account balances.
+ * This block adjusts any remaining differences between a clean reindex/resync and a recent bootstrap.
+ * <p>
+ * The block height 1333492 isn't significant - it's simply the height of a recent bootstrap at the
+ * time of development, so that the account balances could be accessed and compared against the same
+ * block in a reindexed db.
+ * <p>
+ * As with InvalidBalanceBlocks, the discrepancies are insignificant, except for a single
+ * account which has a 3.03 QORT discrepancy. This was due to the account being the first recipient
+ * of a name sale and encountering an early bug in this area.
+ * <p>
+ * The total offset for this block is 3.02816514 QORT.
+ */
+public final class Block1333492 {
+
+	private static final Logger LOGGER = LogManager.getLogger(Block1333492.class);
+	private static final String ACCOUNT_DELTAS_SOURCE = "block-1333492-deltas.json";
+
+	private static final List<AccountBalanceData> accountDeltas = readAccountDeltas();
+
+	private Block1333492() {
+		/* Do not instantiate */
+	}
+
+	@SuppressWarnings("unchecked")
+	private static List<AccountBalanceData> readAccountDeltas() {
+		Unmarshaller unmarshaller;
+
+		try {
+			// Create JAXB context aware of classes we need to unmarshal
+			JAXBContext jc = JAXBContextFactory.createContext(new Class[] {
+				AccountBalanceData.class
+			}, null);
+
+			// Create unmarshaller
+			unmarshaller = jc.createUnmarshaller();
+
+			// Set the unmarshaller media type to JSON
+			unmarshaller.setProperty(UnmarshallerProperties.MEDIA_TYPE, "application/json");
+
+			// Tell unmarshaller that there's no JSON root element in the JSON input
+			unmarshaller.setProperty(UnmarshallerProperties.JSON_INCLUDE_ROOT, false);
+		} catch (JAXBException e) {
+			String message = "Failed to setup unmarshaller to read block 1333492 deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		}
+
+		ClassLoader classLoader = BlockChain.class.getClassLoader();
+		InputStream in = classLoader.getResourceAsStream(ACCOUNT_DELTAS_SOURCE);
+		StreamSource jsonSource = new StreamSource(in);
+
+		try  {
+			// Attempt to unmarshal JSON stream to BlockChain config
+			return (List<AccountBalanceData>) unmarshaller.unmarshal(jsonSource, AccountBalanceData.class).getValue();
+		} catch (UnmarshalException e) {
+			String message = "Failed to parse block 1333492 deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		} catch (JAXBException e) {
+			String message = "Unexpected JAXB issue while processing block 1333492 deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		}
+	}
+
+	public static void processFix(Block block) throws DataException {
+		block.repository.getAccountRepository().modifyAssetBalances(accountDeltas);
+	}
+
+	public static void orphanFix(Block block) throws DataException {
+		// Create inverse deltas
+		List<AccountBalanceData> inverseDeltas = accountDeltas.stream()
+				.map(delta -> new AccountBalanceData(delta.getAddress(), delta.getAssetId(), 0 - delta.getBalance()))
+				.collect(Collectors.toList());
+
+		block.repository.getAccountRepository().modifyAssetBalances(inverseDeltas);
+	}
+
+}

src/main/java/org/qortal/block/BlockChain.java

@@ -79,7 +79,8 @@ public class BlockChain {
 		selfSponsorshipAlgoV1Height,
 		feeValidationFixTimestamp,
 		chatReferenceTimestamp,
-		arbitraryOptionalFeeTimestamp;
+		arbitraryOptionalFeeTimestamp,
+		cancelSellNameValidationTimestamp;
 	}
 
 	// Custom transaction fees
@@ -527,6 +528,10 @@ public class BlockChain {
 		return this.featureTriggers.get(FeatureTrigger.arbitraryOptionalFeeTimestamp.name()).longValue();
 	}
 
+	public long getCancelSellNameValidationTimestamp() {
+		return this.featureTriggers.get(FeatureTrigger.cancelSellNameValidationTimestamp.name()).longValue();
+	}
+
 
 	// More complex getters for aspects that change by height or timestamp
 
@@ -871,6 +876,9 @@ public class BlockChain {
 				BlockData orphanBlockData = repository.getBlockRepository().fromHeight(height);
 
 				while (height > targetHeight) {
+					if (Controller.isStopping()) {
+						return false;
+					}
 					LOGGER.info(String.format("Forcably orphaning block %d", height));
 
 					Block block = new Block(repository, orphanBlockData);

src/main/java/org/qortal/block/InvalidBalanceBlocks.java

@@ -0,0 +1,134 @@
+package org.qortal.block;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.eclipse.persistence.jaxb.JAXBContextFactory;
+import org.eclipse.persistence.jaxb.UnmarshallerProperties;
+import org.qortal.data.account.AccountBalanceData;
+import org.qortal.repository.DataException;
+
+import javax.xml.bind.JAXBContext;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.UnmarshalException;
+import javax.xml.bind.Unmarshaller;
+import javax.xml.transform.stream.StreamSource;
+import java.io.InputStream;
+import java.util.*;
+import java.util.stream.Collectors;
+
+
+/**
+ * Due to various bugs - which have been fixed - a small amount of balance drift occurred
+ * in the chainstate of running nodes and bootstraps, when compared with a clean sync from genesis.
+ * This resulted in a significant number of invalid transactions in the chain history due to
+ * subtle balance discrepancies. The sum of all discrepancies that resulted in an invalid
+ * transaction is 0.00198322 QORT, so despite the large quantity of transactions, they
+ * represent an insignificant amount when summed.
+ * <p>
+ * This class is responsible for retroactively fixing all the past transactions which
+ * are invalid due to the balance discrepancies.
+ */
+
+
+public final class InvalidBalanceBlocks {
+
+	private static final Logger LOGGER = LogManager.getLogger(InvalidBalanceBlocks.class);
+
+	private static final String ACCOUNT_DELTAS_SOURCE = "invalid-transaction-balance-deltas.json";
+
+	private static final List<AccountBalanceData> accountDeltas = readAccountDeltas();
+	private static final List<Integer> affectedHeights = getAffectedHeights();
+
+	private InvalidBalanceBlocks() {
+		/* Do not instantiate */
+	}
+
+	@SuppressWarnings("unchecked")
+	private static List<AccountBalanceData> readAccountDeltas() {
+		Unmarshaller unmarshaller;
+
+		try {
+			// Create JAXB context aware of classes we need to unmarshal
+			JAXBContext jc = JAXBContextFactory.createContext(new Class[] {
+					AccountBalanceData.class
+			}, null);
+
+			// Create unmarshaller
+			unmarshaller = jc.createUnmarshaller();
+
+			// Set the unmarshaller media type to JSON
+			unmarshaller.setProperty(UnmarshallerProperties.MEDIA_TYPE, "application/json");
+
+			// Tell unmarshaller that there's no JSON root element in the JSON input
+			unmarshaller.setProperty(UnmarshallerProperties.JSON_INCLUDE_ROOT, false);
+		} catch (JAXBException e) {
+			String message = "Failed to setup unmarshaller to read block 212937 deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		}
+
+		ClassLoader classLoader = BlockChain.class.getClassLoader();
+		InputStream in = classLoader.getResourceAsStream(ACCOUNT_DELTAS_SOURCE);
+		StreamSource jsonSource = new StreamSource(in);
+
+		try  {
+			// Attempt to unmarshal JSON stream to BlockChain config
+			return (List<AccountBalanceData>) unmarshaller.unmarshal(jsonSource, AccountBalanceData.class).getValue();
+		} catch (UnmarshalException e) {
+			String message = "Failed to parse balance deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		} catch (JAXBException e) {
+			String message = "Unexpected JAXB issue while processing balance deltas";
+			LOGGER.error(message, e);
+			throw new RuntimeException(message, e);
+		}
+	}
+
+	private static List<Integer> getAffectedHeights() {
+		List<Integer> heights = new ArrayList<>();
+		for (AccountBalanceData accountBalanceData : accountDeltas) {
+			if (!heights.contains(accountBalanceData.getHeight())) {
+				heights.add(accountBalanceData.getHeight());
+			}
+		}
+		return heights;
+	}
+
+	private static List<AccountBalanceData> getAccountDeltasAtHeight(int height) {
+		return accountDeltas.stream().filter(a -> a.getHeight() == height).collect(Collectors.toList());
+	}
+
+	public static boolean isAffectedBlock(int height) {
+		return affectedHeights.contains(Integer.valueOf(height));
+	}
+
+	public static void processFix(Block block) throws DataException {
+		Integer blockHeight = block.getBlockData().getHeight();
+		List<AccountBalanceData> deltas = getAccountDeltasAtHeight(blockHeight);
+		if (deltas == null) {
+			throw new DataException(String.format("Unable to lookup invalid balance data for block height %d", blockHeight));
+		}
+
+		block.repository.getAccountRepository().modifyAssetBalances(deltas);
+
+		LOGGER.info("Applied balance patch for block {}", blockHeight);
+	}
+
+	public static void orphanFix(Block block) throws DataException {
+		Integer blockHeight = block.getBlockData().getHeight();
+		List<AccountBalanceData> deltas = getAccountDeltasAtHeight(blockHeight);
+		if (deltas == null) {
+			throw new DataException(String.format("Unable to lookup invalid balance data for block height %d", blockHeight));
+		}
+
+		// Create inverse delta(s)
+		for (AccountBalanceData accountBalanceData : deltas) {
+			AccountBalanceData inverseBalanceData = new AccountBalanceData(accountBalanceData.getAddress(), accountBalanceData.getAssetId(), -accountBalanceData.getBalance());
+			block.repository.getAccountRepository().modifyAssetBalances(List.of(inverseBalanceData));
+		}
+
+		LOGGER.info("Reverted balance patch for block {}", blockHeight);
+	}
+
+}

src/main/java/org/qortal/controller/AutoUpdate.java

@@ -293,4 +293,77 @@ public class AutoUpdate extends Thread {
 		}
 	}
 
+	public static boolean attemptRestart() {
+		LOGGER.info(String.format("Restarting node..."));
+
+		// Give repository a chance to backup in case things go badly wrong (if enabled)
+		if (Settings.getInstance().getRepositoryBackupInterval() > 0) {
+			try {
+				// Timeout if the database isn't ready for backing up after 60 seconds
+				long timeout = 60 * 1000L;
+				RepositoryManager.backup(true, "backup", timeout);
+
+			} catch (TimeoutException e) {
+				LOGGER.info("Attempt to backup repository failed due to timeout: {}", e.getMessage());
+				// Continue with the node restart anyway...
+			}
+		}
+
+		// Call ApplyUpdate to end this process (unlocking current JAR so it can be replaced)
+		String javaHome = System.getProperty("java.home");
+		LOGGER.debug(String.format("Java home: %s", javaHome));
+
+		Path javaBinary = Paths.get(javaHome, "bin", "java");
+		LOGGER.debug(String.format("Java binary: %s", javaBinary));
+
+		try {
+			List<String> javaCmd = new ArrayList<>();
+			// Java runtime binary itself
+			javaCmd.add(javaBinary.toString());
+
+			// JVM arguments
+			javaCmd.addAll(ManagementFactory.getRuntimeMXBean().getInputArguments());
+
+			// Disable, but retain, any -agentlib JVM arg as sub-process might fail if it tries to reuse same port
+			javaCmd = javaCmd.stream()
+					.map(arg -> arg.replace("-agentlib", AGENTLIB_JVM_HOLDER_ARG))
+					.collect(Collectors.toList());
+
+			// Remove JNI options as they won't be supported by command-line 'java'
+			// These are typically added by the AdvancedInstaller Java launcher EXE
+			javaCmd.removeAll(Arrays.asList("abort", "exit", "vfprintf"));
+
+			// Call ApplyUpdate using JAR
+			javaCmd.addAll(Arrays.asList("-cp", JAR_FILENAME, ApplyUpdate.class.getCanonicalName()));
+
+			// Add command-line args saved from start-up
+			String[] savedArgs = Controller.getInstance().getSavedArgs();
+			if (savedArgs != null)
+				javaCmd.addAll(Arrays.asList(savedArgs));
+
+			LOGGER.info(String.format("Restarting node with: %s", String.join(" ", javaCmd)));
+
+			SysTray.getInstance().showMessage(Translator.INSTANCE.translate("SysTray", "AUTO_UPDATE"), //TODO
+					Translator.INSTANCE.translate("SysTray", "APPLYING_UPDATE_AND_RESTARTING"), //TODO
+					MessageType.INFO);
+
+			ProcessBuilder processBuilder = new ProcessBuilder(javaCmd);
+
+			// New process will inherit our stdout and stderr
+			processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
+			processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);
+
+			Process process = processBuilder.start();
+
+			// Nothing to pipe to new process, so close output stream (process's stdin)
+			process.getOutputStream().close();
+
+			return true; // restarting node OK
+		} catch (Exception e) {
+			LOGGER.error(String.format("Failed to restart node: %s", e.getMessage()));
+
+			return true; // repo was okay, even if applying update failed
+		}
+	}
+
 }

src/main/java/org/qortal/controller/BlockMinter.java

@@ -380,9 +380,13 @@ public class BlockMinter extends Thread {
 						parentSignatureForLastLowWeightBlock = null;
 						timeOfLastLowWeightBlock = null;
 
+						Long unconfirmedStartTime = NTP.getTime();
+
 						// Add unconfirmed transactions
 						addUnconfirmedTransactions(repository, newBlock);
 
+						LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime()-unconfirmedStartTime)));
+
 						// Sign to create block's signature
 						newBlock.sign();
 
@@ -484,6 +488,9 @@ public class BlockMinter extends Thread {
 		// Sign to create block's signature, needed by Block.isValid()
 		newBlock.sign();
 
+		// User-defined limit per block
+		int limit = Settings.getInstance().getMaxTransactionsPerBlock();
+
 		// Attempt to add transactions until block is full, or we run out
 		// If a transaction makes the block invalid then skip it and it'll either expire or be in next block.
 		for (TransactionData transactionData : unconfirmedTransactions) {
@@ -496,6 +503,12 @@ public class BlockMinter extends Thread {
 				LOGGER.debug(() -> String.format("Skipping invalid transaction %s during block minting", Base58.encode(transactionData.getSignature())));
 				newBlock.deleteTransaction(transactionData);
 			}
+
+			// User-defined limit per block
+			List<Transaction> transactions = newBlock.getTransactions();
+			if (transactions != null && transactions.size() >= limit) {
+				break;
+			}
 		}
 	}
 

src/main/java/org/qortal/controller/Controller.java

@@ -400,10 +400,13 @@ public class Controller extends Thread {
 			RepositoryFactory repositoryFactory = new HSQLDBRepositoryFactory(getRepositoryUrl());
 			RepositoryManager.setRepositoryFactory(repositoryFactory);
 			RepositoryManager.setRequestedCheckpoint(Boolean.TRUE);
-		}
-		catch (DataException e) {
-			// If exception has no cause then repository is in use by some other process.
-			if (e.getCause() == null) {
+
+			try (final Repository repository = RepositoryManager.getRepository()) {
+				RepositoryManager.rebuildTransactionSequences(repository);
+			}
+		} catch (DataException e) {
+			// If exception has no cause or message then repository is in use by some other process.
+			if (e.getCause() == null && e.getMessage() == null) {
 				LOGGER.info("Repository in use by another process?");
 				Gui.getInstance().fatalError("Repository issue", "Repository in use by another process?");
 			} else {
@@ -437,6 +440,19 @@ public class Controller extends Thread {
 			}
 		}
 
+		try (Repository repository = RepositoryManager.getRepository()) {
+			if (RepositoryManager.needsTransactionSequenceRebuild(repository)) {
+				// Don't allow the node to start if transaction sequences haven't been built yet
+				// This is needed to handle a case when bootstrapping
+				LOGGER.error("Database upgrade needed. Please restart the core to complete the upgrade process.");
+				Gui.getInstance().fatalError("Database upgrade needed", "Please restart the core to complete the upgrade process.");
+				return;
+			}
+		} catch (DataException e) {
+			LOGGER.error("Error checking transaction sequences in repository", e);
+			return;
+		}
+
 		// Import current trade bot states and minting accounts if they exist
 		Controller.importRepositoryData();
 
@@ -1262,13 +1278,6 @@ public class Controller extends Thread {
 				TransactionImporter.getInstance().onNetworkTransactionSignaturesMessage(peer, message);
 				break;
 
-			case GET_ONLINE_ACCOUNTS:
-			case ONLINE_ACCOUNTS:
-			case GET_ONLINE_ACCOUNTS_V2:
-			case ONLINE_ACCOUNTS_V2:
-				// No longer supported - to be eventually removed
-				break;
-
 			case GET_ONLINE_ACCOUNTS_V3:
 				OnlineAccountsManager.getInstance().onNetworkGetOnlineAccountsV3Message(peer, message);
 				break;

src/main/java/org/qortal/controller/DevProxyManager.java

@@ -0,0 +1,74 @@
+package org.qortal.controller;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.api.DevProxyService;
+import org.qortal.repository.DataException;
+import org.qortal.settings.Settings;
+
+public class DevProxyManager {
+
+    protected static final Logger LOGGER = LogManager.getLogger(DevProxyManager.class);
+
+    private static DevProxyManager instance;
+
+    private boolean running = false;
+
+    private String sourceHostAndPort = "127.0.0.1:5173"; // Default for React/Vite
+
+    private DevProxyManager() {
+
+    }
+
+    public static DevProxyManager getInstance() {
+        if (instance == null)
+            instance = new DevProxyManager();
+
+        return instance;
+    }
+
+    public void start() throws DataException {
+        synchronized(this) {
+            if (this.running) {
+                // Already running
+                return;
+            }
+
+            LOGGER.info(String.format("Starting developer proxy service on port %d", Settings.getInstance().getDevProxyPort()));
+            DevProxyService devProxyService = DevProxyService.getInstance();
+            devProxyService.start();
+            this.running = true;
+        }
+    }
+
+    public void stop() {
+        synchronized(this) {
+            if (!this.running) {
+                // Not running
+                return;
+            }
+
+            LOGGER.info(String.format("Shutting down developer proxy service"));
+            DevProxyService devProxyService = DevProxyService.getInstance();
+            devProxyService.stop();
+            this.running = false;
+        }
+    }
+
+    public void setSourceHostAndPort(String sourceHostAndPort) {
+        this.sourceHostAndPort = sourceHostAndPort;
+    }
+
+    public String getSourceHostAndPort() {
+        return this.sourceHostAndPort;
+    }
+
+    public Integer getPort() {
+        return Settings.getInstance().getDevProxyPort();
+    }
+
+    public boolean isRunning() {
+        return this.running;
+    }
+
+}

src/main/java/org/qortal/controller/OnlineAccountsManager.java

@@ -414,7 +414,7 @@ public class OnlineAccountsManager {
         boolean isSuperiorEntry = isOnlineAccountsDataSuperior(onlineAccountData);
         if (isSuperiorEntry)
             // Remove existing inferior entry so it can be re-added below (it's likely the existing copy is missing a nonce value)
-            onlineAccounts.remove(onlineAccountData);
+            onlineAccounts.removeIf(a -> Objects.equals(a.getPublicKey(), onlineAccountData.getPublicKey()));
 
         boolean isNewEntry = onlineAccounts.add(onlineAccountData);
 
@@ -504,110 +504,118 @@ public class OnlineAccountsManager {
         computeOurAccountsForTimestamp(onlineAccountsTimestamp);
     }
 
-    private boolean computeOurAccountsForTimestamp(long onlineAccountsTimestamp) {
-        List<MintingAccountData> mintingAccounts;
-        try (final Repository repository = RepositoryManager.getRepository()) {
-            mintingAccounts = repository.getAccountRepository().getMintingAccounts();
+    private boolean computeOurAccountsForTimestamp(Long onlineAccountsTimestamp) {
+        if (onlineAccountsTimestamp != null) {
+            List<MintingAccountData> mintingAccounts;
+            try (final Repository repository = RepositoryManager.getRepository()) {
+                mintingAccounts = repository.getAccountRepository().getMintingAccounts();
 
-            // We have no accounts to send
-            if (mintingAccounts.isEmpty())
+                // We have no accounts to send
+                if (mintingAccounts.isEmpty())
+                    return false;
+
+                // Only active reward-shares allowed
+                Iterator<MintingAccountData> iterator = mintingAccounts.iterator();
+                int i = 0;
+                while (iterator.hasNext()) {
+                    MintingAccountData mintingAccountData = iterator.next();
+
+                    RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
+                    if (rewardShareData == null) {
+                        // Reward-share doesn't even exist - probably not a good sign
+                        iterator.remove();
+                        continue;
+                    }
+
+                    Account mintingAccount = new Account(repository, rewardShareData.getMinter());
+                    if (!mintingAccount.canMint()) {
+                        // Minting-account component of reward-share can no longer mint - disregard
+                        iterator.remove();
+                        continue;
+                    }
+
+                    if (++i > 1 + 1) {
+                        iterator.remove();
+                        continue;
+                    }
+                }
+            } catch (DataException e) {
+                LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage()));
                 return false;
+            }
 
-            // Only active reward-shares allowed
-            Iterator<MintingAccountData> iterator = mintingAccounts.iterator();
-            while (iterator.hasNext()) {
-                MintingAccountData mintingAccountData = iterator.next();
+            byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
+            List<OnlineAccountData> ourOnlineAccounts = new ArrayList<>();
 
-                RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
-                if (rewardShareData == null) {
-                    // Reward-share doesn't even exist - probably not a good sign
-                    iterator.remove();
+            int remaining = mintingAccounts.size();
+            for (MintingAccountData mintingAccountData : mintingAccounts) {
+                remaining--;
+                byte[] privateKey = mintingAccountData.getPrivateKey();
+                byte[] publicKey = Crypto.toPublicKey(privateKey);
+
+                // We don't want to compute the online account nonce and signature again if it already exists
+                Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet());
+                boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey));
+                if (alreadyExists) {
+                    this.hasOurOnlineAccounts = true;
+
+                    if (remaining > 0) {
+                        // Move on to next account
+                        continue;
+                    } else {
+                        // Everything exists, so return true
+                        return true;
+                    }
+                }
+
+                // Generate bytes for mempow
+                byte[] mempowBytes;
+                try {
+                    mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp);
+                } catch (IOException e) {
+                    LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account...");
                     continue;
                 }
 
-                Account mintingAccount = new Account(repository, rewardShareData.getMinter());
-                if (!mintingAccount.canMint()) {
-                    // Minting-account component of reward-share can no longer mint - disregard
-                    iterator.remove();
-                    continue;
-                }
-            }
-        } catch (DataException e) {
-            LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage()));
-            return false;
-        }
-
-        byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
-        List<OnlineAccountData> ourOnlineAccounts = new ArrayList<>();
-
-        int remaining = mintingAccounts.size();
-        for (MintingAccountData mintingAccountData : mintingAccounts) {
-            remaining--;
-            byte[] privateKey = mintingAccountData.getPrivateKey();
-            byte[] publicKey = Crypto.toPublicKey(privateKey);
-
-            // We don't want to compute the online account nonce and signature again if it already exists
-            Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet());
-            boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey));
-            if (alreadyExists) {
-                this.hasOurOnlineAccounts = true;
-
-                if (remaining > 0) {
-                    // Move on to next account
-                    continue;
-                }
-                else {
-                    // Everything exists, so return true
-                    return true;
-                }
-            }
-
-            // Generate bytes for mempow
-            byte[] mempowBytes;
-            try {
-                mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp);
-            }
-            catch (IOException e) {
-                LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account...");
-                continue;
-            }
-
-            // Compute nonce
-            Integer nonce;
-            try {
-                nonce = this.computeMemoryPoW(mempowBytes, publicKey, onlineAccountsTimestamp);
-                if (nonce == null) {
-                    // A nonce is required
+                // Compute nonce
+                Integer nonce;
+                try {
+                    nonce = this.computeMemoryPoW(mempowBytes, publicKey, onlineAccountsTimestamp);
+                    if (nonce == null) {
+                        // A nonce is required
+                        return false;
+                    }
+                } catch (TimeoutException e) {
+                    LOGGER.info(String.format("Timed out computing nonce for account %.8s", Base58.encode(publicKey)));
                     return false;
                 }
-            } catch (TimeoutException e) {
-                LOGGER.info(String.format("Timed out computing nonce for account %.8s", Base58.encode(publicKey)));
+
+                byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes);
+
+                // Our account is online
+                OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce);
+
+                // Make sure to verify before adding
+                if (verifyMemoryPoW(ourOnlineAccountData, null)) {
+                    ourOnlineAccounts.add(ourOnlineAccountData);
+                }
+            }
+
+            this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty();
+
+            boolean hasInfoChanged = addAccounts(ourOnlineAccounts);
+
+            if (!hasInfoChanged)
                 return false;
-            }
 
-            byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes);
+            Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts));
 
-            // Our account is online
-            OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce);
+            LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp);
 
-            // Make sure to verify before adding
-            if (verifyMemoryPoW(ourOnlineAccountData, null)) {
-                ourOnlineAccounts.add(ourOnlineAccountData);
-            }
+            return true;
         }
 
-        this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty();
-
-        boolean hasInfoChanged = addAccounts(ourOnlineAccounts);
-
-        if (!hasInfoChanged)
-            return false;
-
-        Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts));
-
-        LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp);
-
-        return true;
+        return false;
     }
 
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java

@@ -11,10 +11,7 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.transaction.Transaction;
 import org.qortal.transaction.Transaction.TransactionType;
-import org.qortal.utils.ArbitraryTransactionUtils;
-import org.qortal.utils.Base58;
-import org.qortal.utils.FilesystemUtils;
-import org.qortal.utils.NTP;
+import org.qortal.utils.*;
 
 import java.io.File;
 import java.io.IOException;
@@ -239,7 +236,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 
 					// Delete random data associated with name if we're over our storage limit for this name
 					// Use the DELETION_THRESHOLD, for the same reasons as above
-					for (String followedName : storageManager.followedNames()) {
+					for (String followedName : ListUtils.followedNames()) {
 						if (isStopping) {
 							return;
 						}
@@ -349,6 +346,10 @@ public class ArbitraryDataCleanupManager extends Thread {
 	/**
 	 * Iteratively walk through given directory and delete a single random file
 	 *
+	 * TODO: public data should be prioritized over private data
+	 * (unless this node is part of a data market contract for that data).
+	 * See: Service.privateServices() for a list of services containing private data.
+	 *
 	 * @param directory - the base directory
 	 * @return boolean - whether a file was deleted
 	 */
@@ -487,7 +488,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 
 				// Delete data relating to blocked names
 				String name = directory.getName();
-				if (name != null && storageManager.isNameBlocked(name)) {
+				if (name != null && ListUtils.isNameBlocked(name)) {
 					this.safeDeleteDirectory(directory, "blocked name");
 				}
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java

@@ -20,6 +20,7 @@ import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
 import org.qortal.utils.NTP;
 import org.qortal.utils.Triple;
 
@@ -123,29 +124,29 @@ public class ArbitraryDataFileListManager {
             }
         }
 
-        // Then allow another 3 attempts, each 5 minutes apart
-        if (timeSinceLastAttempt > 5 * 60 * 1000L) {
-            // We haven't tried for at least 5 minutes
+        // Then allow another 5 attempts, each 1 minute apart
+        if (timeSinceLastAttempt > 60 * 1000L) {
+            // We haven't tried for at least 1 minute
 
-            if (networkBroadcastCount < 6) {
-                // We've made less than 6 total attempts
+            if (networkBroadcastCount < 8) {
+                // We've made less than 8 total attempts
                 return true;
             }
         }
 
-        // Then allow another 4 attempts, each 30 minutes apart
-        if (timeSinceLastAttempt > 30 * 60 * 1000L) {
-            // We haven't tried for at least 5 minutes
+        // Then allow another 8 attempts, each 15 minutes apart
+        if (timeSinceLastAttempt > 15 * 60 * 1000L) {
+            // We haven't tried for at least 15 minutes
 
-            if (networkBroadcastCount < 10) {
-                // We've made less than 10 total attempts
+            if (networkBroadcastCount < 16) {
+                // We've made less than 16 total attempts
                 return true;
             }
         }
 
-        // From then on, only try once every 24 hours, to reduce network spam
-        if (timeSinceLastAttempt > 24 * 60 * 60 * 1000L) {
-            // We haven't tried for at least 24 hours
+        // From then on, only try once every 6 hours, to reduce network spam
+        if (timeSinceLastAttempt > 6 * 60 * 60 * 1000L) {
+            // We haven't tried for at least 6 hours
             return true;
         }
 
@@ -503,7 +504,7 @@ public class ArbitraryDataFileListManager {
 
         // Forwarding
         if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
-            boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName()));
+            boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
             if (!isBlocked) {
                 Peer requestingPeer = request.getB();
                 if (requestingPeer != null) {
@@ -682,7 +683,7 @@ public class ArbitraryDataFileListManager {
         }
 
         // We may need to forward this request on
-        boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName()));
+        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
         if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
             // In relay mode - so ask our other peers if they have it
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java

@@ -231,6 +231,11 @@ public class ArbitraryDataFileManager extends Thread {
             arbitraryDataFile = existingFile;
         }
 
+        if (arbitraryDataFile == null) {
+            // We don't have a file, so give up here
+            return null;
+        }
+
         // We might want to forward the request to the peer that originally requested it
         this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
 

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java

@@ -16,7 +16,6 @@ import org.qortal.arbitrary.misc.Service;
 import org.qortal.controller.Controller;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
-import org.qortal.list.ResourceListManager;
 import org.qortal.network.Network;
 import org.qortal.network.Peer;
 import org.qortal.repository.DataException;
@@ -27,6 +26,7 @@ import org.qortal.transaction.ArbitraryTransaction;
 import org.qortal.transaction.Transaction.TransactionType;
 import org.qortal.utils.ArbitraryTransactionUtils;
 import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
 import org.qortal.utils.NTP;
 
 public class ArbitraryDataManager extends Thread {
@@ -172,7 +172,7 @@ public class ArbitraryDataManager extends Thread {
 
 	private void processNames() throws InterruptedException {
 		// Fetch latest list of followed names
-		List<String> followedNames = ResourceListManager.getInstance().getStringsInList("followedNames");
+		List<String> followedNames = ListUtils.followedNames();
 		if (followedNames == null || followedNames.isEmpty()) {
 			return;
 		}

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java

@@ -5,15 +5,11 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
-import org.qortal.list.ResourceListManager;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.settings.Settings;
 import org.qortal.transaction.Transaction;
-import org.qortal.utils.ArbitraryTransactionUtils;
-import org.qortal.utils.Base58;
-import org.qortal.utils.FilesystemUtils;
-import org.qortal.utils.NTP;
+import org.qortal.utils.*;
 
 import java.io.IOException;
 import java.io.UncheckedIOException;
@@ -61,6 +57,8 @@ public class ArbitraryDataStorageManager extends Thread {
      * This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */
     public static final double DELETION_THRESHOLD = 0.98f; // 98%
 
+    private static final long PER_NAME_STORAGE_MULTIPLIER = 4L;
+
     public ArbitraryDataStorageManager() {
     }
 
@@ -135,11 +133,11 @@ public class ArbitraryDataStorageManager extends Thread {
             case ALL:
             case VIEWED:
                 // If the policy includes viewed data, we can host it as long as it's not blocked
-                return !this.isNameBlocked(name);
+                return !ListUtils.isNameBlocked(name);
 
             case FOLLOWED:
                 // If the policy is for followed data only, we have to be following it
-                return this.isFollowingName(name);
+                return ListUtils.isFollowingName(name);
 
                 // For NONE or all else, we shouldn't host this data
             case NONE:
@@ -188,14 +186,14 @@ public class ArbitraryDataStorageManager extends Thread {
         }
 
         // Never fetch data from blocked names, even if they are followed
-        if (this.isNameBlocked(name)) {
+        if (ListUtils.isNameBlocked(name)) {
             return false;
         }
 
         switch (Settings.getInstance().getStoragePolicy()) {
             case FOLLOWED:
             case FOLLOWED_OR_VIEWED:
-                return this.isFollowingName(name);
+                return ListUtils.isFollowingName(name);
                 
             case ALL:
                 return true;
@@ -235,7 +233,7 @@ public class ArbitraryDataStorageManager extends Thread {
      * @return boolean - whether the resource is blocked or not
      */
     public boolean isBlocked(ArbitraryTransactionData arbitraryTransactionData) {
-        return isNameBlocked(arbitraryTransactionData.getName());
+        return ListUtils.isNameBlocked(arbitraryTransactionData.getName());
     }
 
     private boolean isDataTypeAllowed(ArbitraryTransactionData arbitraryTransactionData) {
@@ -253,22 +251,6 @@ public class ArbitraryDataStorageManager extends Thread {
         return true;
     }
 
-    public boolean isNameBlocked(String name) {
-        return ResourceListManager.getInstance().listContains("blockedNames", name, false);
-    }
-
-    private boolean isFollowingName(String name) {
-        return ResourceListManager.getInstance().listContains("followedNames", name, false);
-    }
-
-    public List<String> followedNames() {
-        return ResourceListManager.getInstance().getStringsInList("followedNames");
-    }
-
-    private int followedNamesCount() {
-        return ResourceListManager.getInstance().getItemCountForList("followedNames");
-    }
-
 
     public List<ArbitraryTransactionData> loadAllHostedTransactions(Repository repository) {
         
@@ -508,12 +490,17 @@ public class ArbitraryDataStorageManager extends Thread {
             return false;
         }
 
+        if (Settings.getInstance().getStoragePolicy() == StoragePolicy.ALL) {
+            // Using storage policy ALL, so don't limit anything per name
+            return true;
+        }
+
         if (name == null) {
             // This transaction doesn't have a name, so fall back to total space limitations
             return true;
         }
 
-        int followedNamesCount = this.followedNamesCount();
+        int followedNamesCount = ListUtils.followedNamesCount();
         if (followedNamesCount == 0) {
             // Not following any names, so we have space
             return true;
@@ -543,14 +530,16 @@ public class ArbitraryDataStorageManager extends Thread {
     }
 
     public long storageCapacityPerName(double threshold) {
-        int followedNamesCount = this.followedNamesCount();
+        int followedNamesCount = ListUtils.followedNamesCount();
         if (followedNamesCount == 0) {
             // Not following any names, so we have the total space available
             return this.getStorageCapacityIncludingThreshold(threshold);
         }
 
         double maxStorageCapacity = (double)this.storageCapacity * threshold;
-        long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount);
+
+        // Some names won't need/use much space, so give all names a 4x multiplier to compensate
+        long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount) * PER_NAME_STORAGE_MULTIPLIER;
 
         return maxStoragePerName;
     }

src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java

@@ -16,6 +16,7 @@ import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
 import org.qortal.utils.NTP;
 import org.qortal.utils.Triple;
 
@@ -101,7 +102,14 @@ public class ArbitraryMetadataManager {
                 if (metadataFile.exists()) {
                     // Use local copy
                     ArbitraryDataTransactionMetadata transactionMetadata = new ArbitraryDataTransactionMetadata(metadataFile.getFilePath());
-                    transactionMetadata.read();
+                    try {
+                        transactionMetadata.read();
+                    } catch (DataException e) {
+                        // Invalid file, so delete it
+                        LOGGER.info("Deleting invalid metadata file due to exception: {}", e.getMessage());
+                        transactionMetadata.delete();
+                        return null;
+                    }
                     return transactionMetadata;
                 }
             }
@@ -332,7 +340,7 @@ public class ArbitraryMetadataManager {
             }
 
             // Check if the name is blocked
-            boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName()));
+            boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
             if (!isBlocked) {
                 Peer requestingPeer = request.getB();
                 if (requestingPeer != null) {
@@ -420,7 +428,7 @@ public class ArbitraryMetadataManager {
         }
 
         // We may need to forward this request on
-        boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName()));
+        boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
         if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
             // In relay mode - so ask our other peers if they have it
 

src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java

@@ -13,7 +13,9 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.transaction.Transaction.TransactionType;
 import org.qortal.utils.Unicode;
 
+import java.math.BigInteger;
 import java.util.*;
+import java.util.stream.Collectors;
 
 public class NamesDatabaseIntegrityCheck {
 
@@ -23,21 +25,14 @@ public class NamesDatabaseIntegrityCheck {
             TransactionType.REGISTER_NAME,
             TransactionType.UPDATE_NAME,
             TransactionType.BUY_NAME,
-            TransactionType.SELL_NAME
+            TransactionType.SELL_NAME,
+            TransactionType.CANCEL_SELL_NAME
     );
 
     private List<TransactionData> nameTransactions = new ArrayList<>();
 
+
     public int rebuildName(String name, Repository repository) {
-        return this.rebuildName(name, repository, null);
-    }
-
-    public int rebuildName(String name, Repository repository, List<String> referenceNames) {
-        // "referenceNames" tracks the linked names that have already been rebuilt, to prevent circular dependencies
-        if (referenceNames == null) {
-            referenceNames = new ArrayList<>();
-        }
-
         int modificationCount = 0;
         try {
             List<TransactionData> transactions = this.fetchAllTransactionsInvolvingName(name, repository);
@@ -46,6 +41,14 @@ public class NamesDatabaseIntegrityCheck {
                 return modificationCount;
             }
 
+            // If this name has been updated at any point, we need to add transactions from the other names to the sequence
+            int added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository);
+            while (added > 0) {
+                // Keep going until all have been added
+                LOGGER.trace("{} added for {}. Looking for more transactions...", added, name);
+                added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository);
+            }
+
             // Loop through each past transaction and re-apply it to the Names table
             for (TransactionData currentTransaction : transactions) {
 
@@ -61,29 +64,14 @@ public class NamesDatabaseIntegrityCheck {
                 // Process UPDATE_NAME transactions
                 if (currentTransaction.getType() == TransactionType.UPDATE_NAME) {
                     UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) currentTransaction;
-
-                    if (Objects.equals(updateNameTransactionData.getNewName(), name) &&
-                            !Objects.equals(updateNameTransactionData.getName(), updateNameTransactionData.getNewName())) {
-                        // This renames an existing name, so we need to process that instead
-
-                        if (!referenceNames.contains(name)) {
-                            referenceNames.add(name);
-                            this.rebuildName(updateNameTransactionData.getName(), repository, referenceNames);
-                        }
-                        else {
-                            // We've already processed this name so there's nothing more to do
-                        }
-                    }
-                    else {
-                        Name nameObj = new Name(repository, name);
-                        if (nameObj != null && nameObj.getNameData() != null) {
-                            nameObj.update(updateNameTransactionData);
-                            modificationCount++;
-                            LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name);
-                        } else {
-                            // Something went wrong
-                            throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName()));
-                        }
+                    Name nameObj = new Name(repository, updateNameTransactionData.getName());
+                    if (nameObj != null && nameObj.getNameData() != null) {
+                        nameObj.update(updateNameTransactionData);
+                        modificationCount++;
+                        LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name);
+                    } else {
+                        // Something went wrong
+                        throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName()));
                     }
                 }
 
@@ -354,8 +342,8 @@ public class NamesDatabaseIntegrityCheck {
             }
         }
 
-        // Sort by lowest timestamp first
-        transactions.sort(Comparator.comparingLong(TransactionData::getTimestamp));
+        // Sort by lowest block height first
+        sortTransactions(transactions);
 
         return transactions;
     }
@@ -419,4 +407,67 @@ public class NamesDatabaseIntegrityCheck {
         return names;
     }
 
+    private int addAdditionalTransactionsRelatingToName(List<TransactionData> transactions, String name, Repository repository) throws DataException {
+        int added = 0;
+
+        // If this name has been updated at any point, we need to add transactions from the other names to the sequence
+        List<String> otherNames = new ArrayList<>();
+        List<TransactionData> updateNameTransactions = transactions.stream().filter(t -> t.getType() == TransactionType.UPDATE_NAME).collect(Collectors.toList());
+        for (TransactionData transactionData : updateNameTransactions) {
+            UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) transactionData;
+            // If the newName field isn't empty, and either the "name" or "newName" is different from our reference name,
+            // we should remember this additional name, in case it has relevant transactions associated with it.
+            if (updateNameTransactionData.getNewName() != null && !updateNameTransactionData.getNewName().isEmpty()) {
+                if (!Objects.equals(updateNameTransactionData.getName(), name)) {
+                    otherNames.add(updateNameTransactionData.getName());
+                }
+                if (!Objects.equals(updateNameTransactionData.getNewName(), name)) {
+                    otherNames.add(updateNameTransactionData.getNewName());
+                }
+            }
+        }
+
+
+        for (String otherName : otherNames) {
+            List<TransactionData> otherNameTransactions = this.fetchAllTransactionsInvolvingName(otherName, repository);
+            for (TransactionData otherNameTransactionData : otherNameTransactions) {
+                if (!transactions.contains(otherNameTransactionData)) {
+                    // Add new transaction relating to other name
+                    transactions.add(otherNameTransactionData);
+                    added++;
+                }
+            }
+        }
+
+        if (added > 0) {
+            // New transaction(s) added, so re-sort
+            sortTransactions(transactions);
+        }
+
+        return added;
+    }
+
+    private void sortTransactions(List<TransactionData> transactions) {
+        Collections.sort(transactions, new Comparator() {
+            public int compare(Object o1, Object o2) {
+                TransactionData td1 = (TransactionData) o1;
+                TransactionData td2 = (TransactionData) o2;
+
+                // Sort by block height first
+                int heightComparison = td1.getBlockHeight().compareTo(td2.getBlockHeight());
+                if (heightComparison != 0) {
+                    return heightComparison;
+                }
+
+                // Same height so compare timestamps
+                int timestampComparison = Long.compare(td1.getTimestamp(), td2.getTimestamp());
+                if (timestampComparison != 0) {
+                    return timestampComparison;
+                }
+
+                // Same timestamp so compare signatures
+                return new BigInteger(td1.getSignature()).compareTo(new BigInteger(td2.getSignature()));
+            }});
+    }
+
 }

src/main/java/org/qortal/controller/tradebot/TradeBot.java

@@ -10,6 +10,7 @@ import org.apache.logging.log4j.Logger;
 import org.bitcoinj.core.ECKey;
 import org.qortal.account.PrivateKeyAccount;
 import org.qortal.api.model.crosschain.TradeBotCreateRequest;
+import org.qortal.api.resource.TransactionsResource;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
 import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult;
@@ -19,6 +20,7 @@ import org.qortal.data.at.ATData;
 import org.qortal.data.crosschain.CrossChainTradeData;
 import org.qortal.data.crosschain.TradeBotData;
 import org.qortal.data.network.TradePresenceData;
+import org.qortal.data.transaction.TransactionData;
 import org.qortal.event.Event;
 import org.qortal.event.EventBus;
 import org.qortal.event.Listener;
@@ -33,6 +35,7 @@ import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 import org.qortal.repository.hsqldb.HSQLDBImportExport;
 import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
 import org.qortal.utils.ByteArray;
 import org.qortal.utils.NTP;
 
@@ -113,6 +116,9 @@ public class TradeBot implements Listener {
 	private Map<ByteArray, TradePresenceData> safeAllTradePresencesByPubkey = Collections.emptyMap();
 	private long nextTradePresenceBroadcastTimestamp = 0L;
 
+	private Map<String, Long> failedTrades = new HashMap<>();
+	private Map<String, Long> validTrades = new HashMap<>();
+
 	private TradeBot() {
 		EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event));
 	}
@@ -674,6 +680,78 @@ public class TradeBot implements Listener {
 				});
 	}
 
+	/** Removes any trades that have had multiple failures */
+	public List<CrossChainTradeData> removeFailedTrades(Repository repository, List<CrossChainTradeData> crossChainTrades) {
+		Long now = NTP.getTime();
+		if (now == null) {
+			return crossChainTrades;
+		}
+
+		List<CrossChainTradeData> updatedCrossChainTrades = new ArrayList<>(crossChainTrades);
+		int getMaxTradeOfferAttempts = Settings.getInstance().getMaxTradeOfferAttempts();
+
+		for (CrossChainTradeData crossChainTradeData : crossChainTrades) {
+			// We only care about trades in the OFFERING state
+			if (crossChainTradeData.mode != AcctMode.OFFERING) {
+				failedTrades.remove(crossChainTradeData.qortalAtAddress);
+				validTrades.remove(crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+
+			// Return recently cached values if they exist
+			Long failedTimestamp = failedTrades.get(crossChainTradeData.qortalAtAddress);
+			if (failedTimestamp != null && now - failedTimestamp < 60 * 60 * 1000L) {
+				updatedCrossChainTrades.remove(crossChainTradeData);
+				//LOGGER.info("Removing cached failed trade AT {}", crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+			Long validTimestamp = validTrades.get(crossChainTradeData.qortalAtAddress);
+			if (validTimestamp != null && now - validTimestamp < 60 * 60 * 1000L) {
+				//LOGGER.info("NOT removing cached valid trade AT {}", crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+
+			try {
+				List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, Arrays.asList(Transaction.TransactionType.MESSAGE), null, null, crossChainTradeData.qortalCreatorTradeAddress, TransactionsResource.ConfirmationStatus.CONFIRMED, null, null, null);
+				if (signatures.size() < getMaxTradeOfferAttempts) {
+					// Less than 3 (or user-specified number of) MESSAGE transactions relate to this trade, so assume it is ok
+					validTrades.put(crossChainTradeData.qortalAtAddress, now);
+					continue;
+				}
+
+				List<TransactionData> transactions = new ArrayList<>(signatures.size());
+				for (byte[] signature : signatures) {
+					transactions.add(repository.getTransactionRepository().fromSignature(signature));
+				}
+				transactions.sort(Transaction.getDataComparator());
+
+				// Get timestamp of the first MESSAGE transaction
+				long firstMessageTimestamp = transactions.get(0).getTimestamp();
+
+				// Treat as failed if first buy attempt was more than 60 mins ago (as it's still in the OFFERING state)
+				boolean isFailed = (now - firstMessageTimestamp > 60*60*1000L);
+				if (isFailed) {
+					failedTrades.put(crossChainTradeData.qortalAtAddress, now);
+					updatedCrossChainTrades.remove(crossChainTradeData);
+				}
+				else {
+					validTrades.put(crossChainTradeData.qortalAtAddress, now);
+				}
+
+			} catch (DataException e) {
+				LOGGER.info("Unable to determine failed state of AT {}", crossChainTradeData.qortalAtAddress);
+				continue;
+			}
+		}
+
+		return updatedCrossChainTrades;
+	}
+
+	public boolean isFailedTrade(Repository repository, CrossChainTradeData crossChainTradeData) {
+		List<CrossChainTradeData> results = removeFailedTrades(repository, Arrays.asList(crossChainTradeData));
+		return results.isEmpty();
+	}
+
 	private long generateExpiry(long timestamp) {
 		return ((timestamp - 1) / EXPIRY_ROUNDING) * EXPIRY_ROUNDING + PRESENCE_LIFETIME;
 	}

src/main/java/org/qortal/crypto/TrustlessSSLSocketFactory.java

@@ -28,7 +28,7 @@ public abstract class TrustlessSSLSocketFactory {
 	private static final SSLContext sc;
 	static {
 		try {
-			sc = SSLContext.getInstance("SSL");
+			sc = SSLContext.getInstance("TLSv1.3");
 			sc.init(null, TRUSTLESS_MANAGER, new java.security.SecureRandom());
 		} catch (Exception e) {
 			throw new RuntimeException(e);

src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java

@@ -46,7 +46,7 @@ public class ArbitraryResourceStatus {
         this.description = status.description;
         this.localChunkCount = localChunkCount;
         this.totalChunkCount = totalChunkCount;
-        this.percentLoaded = (this.localChunkCount != null && this.totalChunkCount != null) ? this.localChunkCount / (float)this.totalChunkCount * 100.0f : null;
+        this.percentLoaded = (this.localChunkCount != null && this.totalChunkCount != null && this.totalChunkCount > 0) ? this.localChunkCount / (float)this.totalChunkCount * 100.0f : null;
     }
 
     public ArbitraryResourceStatus(Status status) {

src/main/java/org/qortal/data/network/OnlineAccountData.java

@@ -1,6 +1,7 @@
 package org.qortal.data.network;
 
 import java.util.Arrays;
+import java.util.Objects;
 
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
@@ -34,10 +35,6 @@ public class OnlineAccountData {
 		this.nonce = nonce;
 	}
 
-	public OnlineAccountData(long timestamp, byte[] signature, byte[] publicKey) {
-		this(timestamp, signature, publicKey, null);
-	}
-
 	public long getTimestamp() {
 		return this.timestamp;
 	}
@@ -76,6 +73,10 @@ public class OnlineAccountData {
 		if (otherOnlineAccountData.timestamp != this.timestamp)
 			return false;
 
+		// Almost as quick
+		if (!Objects.equals(otherOnlineAccountData.nonce, this.nonce))
+			return false;
+
 		if (!Arrays.equals(otherOnlineAccountData.publicKey, this.publicKey))
 			return false;
 
@@ -88,9 +89,10 @@ public class OnlineAccountData {
 	public int hashCode() {
 		int h = this.hash;
 		if (h == 0) {
-			this.hash = h = Long.hashCode(this.timestamp)
-					^ Arrays.hashCode(this.publicKey);
+			h = Objects.hash(timestamp, nonce);
+			h = 31 * h + Arrays.hashCode(publicKey);
 			// We don't use signature because newer aggregate signatures use random nonces
+			this.hash = h;
 		}
 		return h;
 	}

src/main/java/org/qortal/data/transaction/CreatePollTransactionData.java

@@ -2,9 +2,11 @@ package org.qortal.data.transaction;
 
 import java.util.List;
 
+import javax.xml.bind.Unmarshaller;
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 
+import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorValue;
 import org.qortal.data.voting.PollOptionData;
 import org.qortal.transaction.Transaction;
 import org.qortal.transaction.Transaction.TransactionType;
@@ -14,8 +16,13 @@ import io.swagger.v3.oas.annotations.media.Schema;
 // All properties to be converted to JSON via JAXB
 @XmlAccessorType(XmlAccessType.FIELD)
 @Schema(allOf = { TransactionData.class })
+@XmlDiscriminatorValue("CREATE_POLL")
 public class CreatePollTransactionData extends TransactionData {
 
+
+	@Schema(description = "Poll creator's public key", example = "2tiMr5LTpaWCgbRvkPK8TFd7k63DyHJMMFFsz9uBf1ZP")
+	private byte[] pollCreatorPublicKey;
+	
 	// Properties
 	private String owner;
 	private String pollName;
@@ -29,10 +36,15 @@ public class CreatePollTransactionData extends TransactionData {
 		super(TransactionType.CREATE_POLL);
 	}
 
+	public void afterUnmarshal(Unmarshaller u, Object parent) {
+		this.creatorPublicKey = this.pollCreatorPublicKey;
+	}
+	
 	public CreatePollTransactionData(BaseTransactionData baseTransactionData,
 			String owner, String pollName, String description, List<PollOptionData> pollOptions) {
 		super(Transaction.TransactionType.CREATE_POLL, baseTransactionData);
 
+		this.creatorPublicKey = baseTransactionData.creatorPublicKey;
 		this.owner = owner;
 		this.pollName = pollName;
 		this.description = description;
@@ -41,6 +53,7 @@ public class CreatePollTransactionData extends TransactionData {
 
 	// Getters/setters
 
+	public byte[] getPollCreatorPublicKey() { return this.creatorPublicKey; }
 	public String getOwner() {
 		return this.owner;
 	}

src/main/java/org/qortal/data/transaction/TransactionData.java

@@ -12,6 +12,8 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
 
 import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorNode;
 import org.qortal.crypto.Crypto;
+import org.qortal.data.voting.PollData;
+import org.qortal.data.voting.VoteOnPollData;
 import org.qortal.transaction.Transaction.ApprovalStatus;
 import org.qortal.transaction.Transaction.TransactionType;
 
@@ -29,6 +31,7 @@ import io.swagger.v3.oas.annotations.media.Schema.AccessMode;
 @XmlSeeAlso({GenesisTransactionData.class, PaymentTransactionData.class, RegisterNameTransactionData.class, UpdateNameTransactionData.class,
 	SellNameTransactionData.class, CancelSellNameTransactionData.class, BuyNameTransactionData.class,
 	CreatePollTransactionData.class, VoteOnPollTransactionData.class, ArbitraryTransactionData.class,
+	PollData.class, VoteOnPollData.class,
 	IssueAssetTransactionData.class, TransferAssetTransactionData.class,
 	CreateAssetOrderTransactionData.class, CancelAssetOrderTransactionData.class,
 	MultiPaymentTransactionData.class, DeployAtTransactionData.class, MessageTransactionData.class, ATTransactionData.class,
@@ -76,6 +79,10 @@ public abstract class TransactionData {
 	@Schema(accessMode = AccessMode.READ_ONLY, hidden = true, description = "height of block containing transaction")
 	protected Integer blockHeight;
 
+	// Not always present
+	@Schema(accessMode = AccessMode.READ_ONLY, hidden = true, description = "sequence in block containing transaction")
+	protected Integer blockSequence;
+
 	// Not always present
 	@Schema(accessMode = AccessMode.READ_ONLY, description = "group-approval status")
 	protected ApprovalStatus approvalStatus;
@@ -106,6 +113,7 @@ public abstract class TransactionData {
 		this.fee = baseTransactionData.fee;
 		this.signature = baseTransactionData.signature;
 		this.blockHeight = baseTransactionData.blockHeight;
+		this.blockSequence = baseTransactionData.blockSequence;
 		this.approvalStatus = baseTransactionData.approvalStatus;
 		this.approvalHeight = baseTransactionData.approvalHeight;
 	}
@@ -174,6 +182,15 @@ public abstract class TransactionData {
 		this.blockHeight = blockHeight;
 	}
 
+	public Integer getBlockSequence() {
+		return this.blockSequence;
+	}
+
+	@XmlTransient
+	public void setBlockSequence(Integer blockSequence) {
+		this.blockSequence = blockSequence;
+	}
+
 	public ApprovalStatus getApprovalStatus() {
 		return approvalStatus;
 	}

src/main/java/org/qortal/data/transaction/VoteOnPollTransactionData.java

@@ -3,7 +3,9 @@ package org.qortal.data.transaction;
 import javax.xml.bind.Unmarshaller;
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlTransient;
 
+import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorValue;
 import org.qortal.transaction.Transaction.TransactionType;
 
 import io.swagger.v3.oas.annotations.media.Schema;
@@ -11,12 +13,17 @@ import io.swagger.v3.oas.annotations.media.Schema;
 // All properties to be converted to JSON via JAXB
 @XmlAccessorType(XmlAccessType.FIELD)
 @Schema(allOf = { TransactionData.class })
+@XmlDiscriminatorValue("VOTE_ON_POLL")
 public class VoteOnPollTransactionData extends TransactionData {
 
 	// Properties
+	@Schema(description = "Vote creator's public key", example = "2tiMr5LTpaWCgbRvkPK8TFd7k63DyHJMMFFsz9uBf1ZP")
 	private byte[] voterPublicKey;
 	private String pollName;
 	private int optionIndex;
+	// For internal use when orphaning
+	@XmlTransient
+	@Schema(hidden = true)
 	private Integer previousOptionIndex;
 
 	// Constructors

src/main/java/org/qortal/data/voting/PollData.java

@@ -14,6 +14,11 @@ public class PollData {
 
 	// Constructors
 
+	// For JAXB
+	protected PollData() {
+		super();
+	}
+
 	public PollData(byte[] creatorPublicKey, String owner, String pollName, String description, List<PollOptionData> pollOptions, long published) {
 		this.creatorPublicKey = creatorPublicKey;
 		this.owner = owner;
@@ -29,22 +34,42 @@ public class PollData {
 		return this.creatorPublicKey;
 	}
 
+	public void setCreatorPublicKey(byte[] creatorPublicKey) {
+		this.creatorPublicKey = creatorPublicKey;
+	}
+
 	public String getOwner() {
 		return this.owner;
 	}
 
+	public void setOwner(String owner) {
+		this.owner = owner;
+	}
+
 	public String getPollName() {
 		return this.pollName;
 	}
 
+	public void setPollName(String pollName) {
+		this.pollName = pollName;
+	}
+
 	public String getDescription() {
 		return this.description;
 	}
 
+	public void setDescription(String description) {
+		this.description = description;
+	}
+
 	public List<PollOptionData> getPollOptions() {
 		return this.pollOptions;
 	}
 
+	public void setPollOptions(List<PollOptionData> pollOptions) {
+		this.pollOptions = pollOptions;
+	}
+
 	public long getPublished() {
 		return this.published;
 	}

src/main/java/org/qortal/data/voting/VoteOnPollData.java

@@ -9,6 +9,11 @@ public class VoteOnPollData {
 
 	// Constructors
 
+	// For JAXB
+	protected VoteOnPollData() {
+		super();
+	}
+
 	public VoteOnPollData(String pollName, byte[] voterPublicKey, int optionIndex) {
 		this.pollName = pollName;
 		this.voterPublicKey = voterPublicKey;
@@ -21,12 +26,24 @@ public class VoteOnPollData {
 		return this.pollName;
 	}
 
+	public void setPollName(String pollName) {
+		this.pollName = pollName;
+	}
+
 	public byte[] getVoterPublicKey() {
 		return this.voterPublicKey;
 	}
 
+	public void setVoterPublicKey(byte[] voterPublicKey) {
+		this.voterPublicKey = voterPublicKey;
+	}
+
 	public int getOptionIndex() {
 		return this.optionIndex;
 	}
 
+	public void setOptionIndex(int optionIndex) {
+		this.optionIndex = optionIndex;
+	}
+
 }

src/main/java/org/qortal/list/ResourceList.java

@@ -9,6 +9,7 @@ import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileWriter;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -52,6 +53,15 @@ public class ResourceList {
         String jsonString = ResourceList.listToJSONString(this.list);
         Path filePath = this.getFilePath();
 
+        // Don't create list if it's empty
+        if (this.list != null && this.list.isEmpty()) {
+            if (filePath != null && Files.exists(filePath)) {
+                // Delete empty list
+                Files.delete(filePath);
+            }
+            return;
+        }
+
         // Create parent directory if needed
         try {
             Files.createDirectories(filePath.getParent());
@@ -72,7 +82,7 @@ public class ResourceList {
         }
 
         try {
-            String jsonString = new String(Files.readAllBytes(path));
+            String jsonString = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
             this.list = ResourceList.listFromJSONString(jsonString);
         } catch (IOException e) {
             throw new IOException(String.format("Couldn't read contents from file %s", path.toString()));
@@ -109,6 +119,13 @@ public class ResourceList {
         this.list.remove(resource);
     }
 
+    public void clear() {
+        if (this.list == null) {
+            return;
+        }
+        this.list.clear();
+    }
+
     public boolean contains(String resource, boolean caseSensitive) {
         if (resource == null || this.list == null) {
             return false;

src/main/java/org/qortal/list/ResourceListManager.java

@@ -2,8 +2,11 @@ package org.qortal.list;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.qortal.settings.Settings;
 
 import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
@@ -18,6 +21,7 @@ public class ResourceListManager {
 
 
     public ResourceListManager() {
+        this.lists = this.fetchLists();
     }
 
     public static synchronized ResourceListManager getInstance() {
@@ -27,6 +31,38 @@ public class ResourceListManager {
         return instance;
     }
 
+    public static synchronized void reset() {
+        if (instance != null) {
+            instance = null;
+        }
+    }
+
+    private List<ResourceList> fetchLists() {
+        List<ResourceList> lists = new ArrayList<>();
+        Path listsPath = Paths.get(Settings.getInstance().getListsPath());
+
+        if (listsPath.toFile().isDirectory()) {
+            String[] files = listsPath.toFile().list();
+
+            for (String fileName : files) {
+                try {
+                    // Remove .json extension
+                    if (fileName.endsWith(".json")) {
+                        fileName = fileName.substring(0, fileName.length() - 5);
+                    }
+
+                    ResourceList list = new ResourceList(fileName);
+                    if (list != null) {
+                        lists.add(list);
+                    }
+                } catch (IOException e) {
+                    // Ignore this list
+                }
+            }
+        }
+        return lists;
+    }
+
     private ResourceList getList(String listName) {
         for (ResourceList list : this.lists) {
             if (Objects.equals(list.getName(), listName)) {
@@ -48,6 +84,18 @@ public class ResourceListManager {
 
     }
 
+    private List<ResourceList> getListsByPrefix(String listNamePrefix) {
+        List<ResourceList> lists = new ArrayList<>();
+
+        for (ResourceList list : this.lists) {
+            if (list != null && list.getName() != null && list.getName().startsWith(listNamePrefix)) {
+                lists.add(list);
+            }
+        }
+
+        return lists;
+    }
+
     public boolean addToList(String listName, String item, boolean save) {
         ResourceList list = this.getList(listName);
         if (list == null) {
@@ -95,6 +143,16 @@ public class ResourceListManager {
         return list.contains(item, caseSensitive);
     }
 
+    public boolean listWithPrefixContains(String listNamePrefix, String item, boolean caseSensitive) {
+        List<ResourceList> lists = getListsByPrefix(listNamePrefix);
+        for (ResourceList list : lists) {
+            if (list.contains(item, caseSensitive)) {
+                return true;
+            }
+        }
+        return false;
+    }
+
     public void saveList(String listName) {
         ResourceList list = this.getList(listName);
         if (list == null) {
@@ -133,6 +191,15 @@ public class ResourceListManager {
         return list.getList();
     }
 
+    public List<String> getStringsInListsWithPrefix(String listNamePrefix) {
+        List<String> items = new ArrayList<>();
+        List<ResourceList> lists = getListsByPrefix(listNamePrefix);
+        for (ResourceList list : lists) {
+            items.addAll(list.getList());
+        }
+        return items;
+    }
+
     public int getItemCountForList(String listName) {
         ResourceList list = this.getList(listName);
         if (list == null) {

src/main/java/org/qortal/network/Handshake.java

@@ -265,7 +265,7 @@ public enum Handshake {
 	private static final long PEER_VERSION_131 = 0x0100030001L;
 
 	/** Minimum peer version that we are allowed to communicate with */
-	private static final String MIN_PEER_VERSION = "3.8.2";
+	private static final String MIN_PEER_VERSION = "4.1.1";
 
 	private static final int POW_BUFFER_SIZE_PRE_131 = 8 * 1024 * 1024; // bytes
 	private static final int POW_DIFFICULTY_PRE_131 = 8; // leading zero bits

src/main/java/org/qortal/network/Network.java

@@ -124,6 +124,8 @@ public class Network {
 
     private final List<PeerAddress> selfPeers = new ArrayList<>();
 
+    private String bindAddress = null;
+
     private final ExecuteProduceConsume networkEPC;
     private Selector channelSelector;
     private ServerSocketChannel serverChannel;
@@ -159,25 +161,43 @@ public class Network {
         // Grab P2P port from settings
         int listenPort = Settings.getInstance().getListenPort();
 
-        // Grab P2P bind address from settings
-        try {
-            InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
-            InetSocketAddress endpoint = new InetSocketAddress(bindAddr, listenPort);
+        // Grab P2P bind addresses from settings
+        List<String> bindAddresses = new ArrayList<>();
+        if (Settings.getInstance().getBindAddress() != null) {
+            bindAddresses.add(Settings.getInstance().getBindAddress());
+        }
+        if (Settings.getInstance().getBindAddressFallback() != null) {
+            bindAddresses.add(Settings.getInstance().getBindAddressFallback());
+        }
 
-            channelSelector = Selector.open();
+        for (int i=0; i<bindAddresses.size(); i++) {
+            try {
+                String bindAddress = bindAddresses.get(i);
+                InetAddress bindAddr = InetAddress.getByName(bindAddress);
+                InetSocketAddress endpoint = new InetSocketAddress(bindAddr, listenPort);
 
-            // Set up listen socket
-            serverChannel = ServerSocketChannel.open();
-            serverChannel.configureBlocking(false);
-            serverChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true);
-            serverChannel.bind(endpoint, LISTEN_BACKLOG);
-            serverSelectionKey = serverChannel.register(channelSelector, SelectionKey.OP_ACCEPT);
-        } catch (UnknownHostException e) {
-            LOGGER.error("Can't bind listen socket to address {}", Settings.getInstance().getBindAddress());
-            throw new IOException("Can't bind listen socket to address", e);
-        } catch (IOException e) {
-            LOGGER.error("Can't create listen socket: {}", e.getMessage());
-            throw new IOException("Can't create listen socket", e);
+                channelSelector = Selector.open();
+
+                // Set up listen socket
+                serverChannel = ServerSocketChannel.open();
+                serverChannel.configureBlocking(false);
+                serverChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true);
+                serverChannel.bind(endpoint, LISTEN_BACKLOG);
+                serverSelectionKey = serverChannel.register(channelSelector, SelectionKey.OP_ACCEPT);
+
+                this.bindAddress = bindAddress; // Store the selected address, so that it can be used by other parts of the app
+                break; // We don't want to bind to more than one address
+            } catch (UnknownHostException | UnsupportedAddressTypeException e) {
+                LOGGER.error("Can't bind listen socket to address {}", Settings.getInstance().getBindAddress());
+                if (i == bindAddresses.size()-1) { // Only throw an exception if all addresses have been tried
+                    throw new IOException("Can't bind listen socket to address", e);
+                }
+            } catch (IOException e) {
+                LOGGER.error("Can't create listen socket: {}", e.getMessage());
+                if (i == bindAddresses.size()-1) { // Only throw an exception if all addresses have been tried
+                    throw new IOException("Can't create listen socket", e);
+                }
+            }
         }
 
         // Load all known peers from repository
@@ -228,6 +248,10 @@ public class Network {
         return this.maxPeers;
     }
 
+    public String getBindAddress() {
+        return this.bindAddress;
+    }
+
     public byte[] getMessageMagic() {
         return Settings.getInstance().isTestNet() ? TESTNET_MESSAGE_MAGIC : MAINNET_MESSAGE_MAGIC;
     }
@@ -1556,7 +1580,7 @@ public class Network {
         this.isShuttingDown = true;
 
         // Close listen socket to prevent more incoming connections
-        if (this.serverChannel.isOpen()) {
+        if (this.serverChannel != null && this.serverChannel.isOpen()) {
             try {
                 this.serverChannel.close();
             } catch (IOException e) {

src/main/java/org/qortal/network/message/ArbitraryDataFileMessage.java

@@ -68,7 +68,7 @@ public class ArbitraryDataFileMessage extends Message {
 		byteBuffer.get(data);
 
 		try {
-			ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(data, signature);
+			ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(data, signature, false);
 			return new ArbitraryDataFileMessage(id, signature, arbitraryDataFile);
 		} catch (DataException e) {
 			LOGGER.info("Unable to process received file: {}", e.getMessage());

src/main/java/org/qortal/network/message/ArbitraryMetadataMessage.java

@@ -64,7 +64,7 @@ public class ArbitraryMetadataMessage extends Message {
 		byteBuffer.get(data);
 
 		try {
-			ArbitraryDataFile arbitraryMetadataFile = new ArbitraryDataFile(data, signature);
+			ArbitraryDataFile arbitraryMetadataFile = new ArbitraryDataFile(data, signature, false);
 			return new ArbitraryMetadataMessage(id, signature, arbitraryMetadataFile);
 		} catch (DataException e) {
 			throw new MessageException("Unable to process arbitrary metadata message: " + e.getMessage(), e);

src/main/java/org/qortal/network/message/GetOnlineAccountsMessage.java

@@ -1,69 +0,0 @@
-package org.qortal.network.message;
-
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.stream.Collectors;
-
-import org.qortal.data.network.OnlineAccountData;
-import org.qortal.transform.Transformer;
-
-import com.google.common.primitives.Ints;
-import com.google.common.primitives.Longs;
-
-public class GetOnlineAccountsMessage extends Message {
-	private static final int MAX_ACCOUNT_COUNT = 5000;
-
-	private List<OnlineAccountData> onlineAccounts;
-
-	public GetOnlineAccountsMessage(List<OnlineAccountData> onlineAccounts) {
-		super(MessageType.GET_ONLINE_ACCOUNTS);
-
-		ByteArrayOutputStream bytes = new ByteArrayOutputStream();
-
-		try {
-			bytes.write(Ints.toByteArray(onlineAccounts.size()));
-
-			for (OnlineAccountData onlineAccountData : onlineAccounts) {
-				bytes.write(Longs.toByteArray(onlineAccountData.getTimestamp()));
-
-				bytes.write(onlineAccountData.getPublicKey());
-			}
-		} catch (IOException e) {
-			throw new AssertionError("IOException shouldn't occur with ByteArrayOutputStream");
-		}
-
-		this.dataBytes = bytes.toByteArray();
-		this.checksumBytes = Message.generateChecksum(this.dataBytes);
-	}
-
-	private GetOnlineAccountsMessage(int id, List<OnlineAccountData> onlineAccounts) {
-		super(id, MessageType.GET_ONLINE_ACCOUNTS);
-
-		this.onlineAccounts = onlineAccounts.stream().limit(MAX_ACCOUNT_COUNT).collect(Collectors.toList());
-	}
-
-	public List<OnlineAccountData> getOnlineAccounts() {
-		return this.onlineAccounts;
-	}
-
-	public static Message fromByteBuffer(int id, ByteBuffer bytes) {
-		final int accountCount = bytes.getInt();
-
-		List<OnlineAccountData> onlineAccounts = new ArrayList<>(accountCount);
-
-		for (int i = 0; i < Math.min(MAX_ACCOUNT_COUNT, accountCount); ++i) {
-			long timestamp = bytes.getLong();
-
-			byte[] publicKey = new byte[Transformer.PUBLIC_KEY_LENGTH];
-			bytes.get(publicKey);
-
-			onlineAccounts.add(new OnlineAccountData(timestamp, null, publicKey));
-		}
-
-		return new GetOnlineAccountsMessage(id, onlineAccounts);
-	}
-
-}

src/main/java/org/qortal/network/message/GetOnlineAccountsV2Message.java

@@ -1,109 +0,0 @@
-package org.qortal.network.message;
-
-import com.google.common.primitives.Ints;
-import com.google.common.primitives.Longs;
-import org.qortal.data.network.OnlineAccountData;
-import org.qortal.transform.Transformer;
-
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-/**
- * For requesting online accounts info from remote peer, given our list of online accounts.
- *
- * Different format to V1:
- * V1 is: number of entries, then timestamp + pubkey for each entry
- * V2 is: groups of: number of entries, timestamp, then pubkey for each entry
- *
- * Also V2 only builds online accounts message once!
- */
-public class GetOnlineAccountsV2Message extends Message {
-
-	private List<OnlineAccountData> onlineAccounts;
-
-	public GetOnlineAccountsV2Message(List<OnlineAccountData> onlineAccounts) {
-		super(MessageType.GET_ONLINE_ACCOUNTS_V2);
-
-		// If we don't have ANY online accounts then it's an easier construction...
-		if (onlineAccounts.isEmpty()) {
-			// Always supply a number of accounts
-			this.dataBytes = Ints.toByteArray(0);
-			this.checksumBytes = Message.generateChecksum(this.dataBytes);
-			return;
-		}
-
-		// How many of each timestamp
-		Map<Long, Integer> countByTimestamp = new HashMap<>();
-
-		for (OnlineAccountData onlineAccountData : onlineAccounts) {
-			Long timestamp = onlineAccountData.getTimestamp();
-			countByTimestamp.compute(timestamp, (k, v) -> v == null ? 1 : ++v);
-		}
-
-		// We should know exactly how many bytes to allocate now
-		int byteSize = countByTimestamp.size() * (Transformer.INT_LENGTH + Transformer.TIMESTAMP_LENGTH)
-				+ onlineAccounts.size() * Transformer.PUBLIC_KEY_LENGTH;
-
-		ByteArrayOutputStream bytes = new ByteArrayOutputStream(byteSize);
-
-		try {
-			for (long timestamp : countByTimestamp.keySet()) {
-				bytes.write(Ints.toByteArray(countByTimestamp.get(timestamp)));
-
-				bytes.write(Longs.toByteArray(timestamp));
-
-				for (OnlineAccountData onlineAccountData : onlineAccounts) {
-					if (onlineAccountData.getTimestamp() == timestamp)
-						bytes.write(onlineAccountData.getPublicKey());
-				}
-			}
-		} catch (IOException e) {
-			throw new AssertionError("IOException shouldn't occur with ByteArrayOutputStream");
-		}
-
-		this.dataBytes = bytes.toByteArray();
-		this.checksumBytes = Message.generateChecksum(this.dataBytes);
-	}
-
-	private GetOnlineAccountsV2Message(int id, List<OnlineAccountData> onlineAccounts) {
-		super(id, MessageType.GET_ONLINE_ACCOUNTS_V2);
-
-		this.onlineAccounts = onlineAccounts;
-	}
-
-	public List<OnlineAccountData> getOnlineAccounts() {
-		return this.onlineAccounts;
-	}
-
-	public static Message fromByteBuffer(int id, ByteBuffer bytes) {
-		int accountCount = bytes.getInt();
-
-		List<OnlineAccountData> onlineAccounts = new ArrayList<>(accountCount);
-
-		while (accountCount > 0) {
-			long timestamp = bytes.getLong();
-
-			for (int i = 0; i < accountCount; ++i) {
-				byte[] publicKey = new byte[Transformer.PUBLIC_KEY_LENGTH];
-				bytes.get(publicKey);
-
-				onlineAccounts.add(new OnlineAccountData(timestamp, null, publicKey));
-			}
-
-			if (bytes.hasRemaining()) {
-				accountCount = bytes.getInt();
-			} else {
-				// we've finished
-				accountCount = 0;
-			}
-		}
-
-		return new GetOnlineAccountsV2Message(id, onlineAccounts);
-	}
-
-}

src/main/java/org/qortal/network/message/MessageType.java

@@ -43,11 +43,7 @@ public enum MessageType {
     BLOCK_SUMMARIES(70, BlockSummariesMessage::fromByteBuffer),
     GET_BLOCK_SUMMARIES(71, GetBlockSummariesMessage::fromByteBuffer),
     BLOCK_SUMMARIES_V2(72, BlockSummariesV2Message::fromByteBuffer),
-
-    ONLINE_ACCOUNTS(80, OnlineAccountsMessage::fromByteBuffer),
-    GET_ONLINE_ACCOUNTS(81, GetOnlineAccountsMessage::fromByteBuffer),
-    ONLINE_ACCOUNTS_V2(82, OnlineAccountsV2Message::fromByteBuffer),
-    GET_ONLINE_ACCOUNTS_V2(83, GetOnlineAccountsV2Message::fromByteBuffer),
+    
     ONLINE_ACCOUNTS_V3(84, OnlineAccountsV3Message::fromByteBuffer),
     GET_ONLINE_ACCOUNTS_V3(85, GetOnlineAccountsV3Message::fromByteBuffer),
 

src/main/java/org/qortal/network/message/OnlineAccountsMessage.java

@@ -1,75 +0,0 @@
-package org.qortal.network.message;
-
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.stream.Collectors;
-
-import org.qortal.data.network.OnlineAccountData;
-import org.qortal.transform.Transformer;
-
-import com.google.common.primitives.Ints;
-import com.google.common.primitives.Longs;
-
-public class OnlineAccountsMessage extends Message {
-	private static final int MAX_ACCOUNT_COUNT = 5000;
-
-	private List<OnlineAccountData> onlineAccounts;
-
-	public OnlineAccountsMessage(List<OnlineAccountData> onlineAccounts) {
-		super(MessageType.ONLINE_ACCOUNTS);
-
-		ByteArrayOutputStream bytes = new ByteArrayOutputStream();
-
-		try {
-			bytes.write(Ints.toByteArray(onlineAccounts.size()));
-
-			for (OnlineAccountData onlineAccountData : onlineAccounts) {
-				bytes.write(Longs.toByteArray(onlineAccountData.getTimestamp()));
-
-				bytes.write(onlineAccountData.getSignature());
-
-				bytes.write(onlineAccountData.getPublicKey());
-			}
-		} catch (IOException e) {
-			throw new AssertionError("IOException shouldn't occur with ByteArrayOutputStream");
-		}
-
-		this.dataBytes = bytes.toByteArray();
-		this.checksumBytes = Message.generateChecksum(this.dataBytes);
-	}
-
-	private OnlineAccountsMessage(int id, List<OnlineAccountData> onlineAccounts) {
-		super(id, MessageType.ONLINE_ACCOUNTS);
-
-		this.onlineAccounts = onlineAccounts.stream().limit(MAX_ACCOUNT_COUNT).collect(Collectors.toList());
-	}
-
-	public List<OnlineAccountData> getOnlineAccounts() {
-		return this.onlineAccounts;
-	}
-
-	public static Message fromByteBuffer(int id, ByteBuffer bytes) {
-		final int accountCount = bytes.getInt();
-
-		List<OnlineAccountData> onlineAccounts = new ArrayList<>(accountCount);
-
-		for (int i = 0; i < Math.min(MAX_ACCOUNT_COUNT, accountCount); ++i) {
-			long timestamp = bytes.getLong();
-
-			byte[] signature = new byte[Transformer.SIGNATURE_LENGTH];
-			bytes.get(signature);
-
-			byte[] publicKey = new byte[Transformer.PUBLIC_KEY_LENGTH];
-			bytes.get(publicKey);
-
-			OnlineAccountData onlineAccountData = new OnlineAccountData(timestamp, signature, publicKey);
-			onlineAccounts.add(onlineAccountData);
-		}
-
-		return new OnlineAccountsMessage(id, onlineAccounts);
-	}
-
-}

src/main/java/org/qortal/network/message/OnlineAccountsV2Message.java

@@ -1,113 +0,0 @@
-package org.qortal.network.message;
-
-import com.google.common.primitives.Ints;
-import com.google.common.primitives.Longs;
-import org.qortal.data.network.OnlineAccountData;
-import org.qortal.transform.Transformer;
-
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-/**
- * For sending online accounts info to remote peer.
- *
- * Different format to V1:
- * V1 is: number of entries, then timestamp + sig + pubkey for each entry
- * V2 is: groups of: number of entries, timestamp, then sig + pubkey for each entry
- *
- * Also V2 only builds online accounts message once!
- */
-public class OnlineAccountsV2Message extends Message {
-
-	private List<OnlineAccountData> onlineAccounts;
-
-	public OnlineAccountsV2Message(List<OnlineAccountData> onlineAccounts) {
-		super(MessageType.ONLINE_ACCOUNTS_V2);
-
-		// Shortcut in case we have no online accounts
-		if (onlineAccounts.isEmpty()) {
-			this.dataBytes = Ints.toByteArray(0);
-			this.checksumBytes = Message.generateChecksum(this.dataBytes);
-			return;
-		}
-
-		// How many of each timestamp
-		Map<Long, Integer> countByTimestamp = new HashMap<>();
-
-		for (OnlineAccountData onlineAccountData : onlineAccounts) {
-			Long timestamp = onlineAccountData.getTimestamp();
-			countByTimestamp.compute(timestamp, (k, v) -> v == null ? 1 : ++v);
-		}
-
-		// We should know exactly how many bytes to allocate now
-		int byteSize = countByTimestamp.size() * (Transformer.INT_LENGTH + Transformer.TIMESTAMP_LENGTH)
-				+ onlineAccounts.size() * (Transformer.SIGNATURE_LENGTH + Transformer.PUBLIC_KEY_LENGTH);
-
-		ByteArrayOutputStream bytes = new ByteArrayOutputStream(byteSize);
-
-		try {
-			for (long timestamp : countByTimestamp.keySet()) {
-				bytes.write(Ints.toByteArray(countByTimestamp.get(timestamp)));
-
-				bytes.write(Longs.toByteArray(timestamp));
-
-				for (OnlineAccountData onlineAccountData : onlineAccounts) {
-					if (onlineAccountData.getTimestamp() == timestamp) {
-						bytes.write(onlineAccountData.getSignature());
-						bytes.write(onlineAccountData.getPublicKey());
-					}
-				}
-			}
-		} catch (IOException e) {
-			throw new AssertionError("IOException shouldn't occur with ByteArrayOutputStream");
-		}
-
-		this.dataBytes = bytes.toByteArray();
-		this.checksumBytes = Message.generateChecksum(this.dataBytes);
-	}
-
-	private OnlineAccountsV2Message(int id, List<OnlineAccountData> onlineAccounts) {
-		super(id, MessageType.ONLINE_ACCOUNTS_V2);
-
-		this.onlineAccounts = onlineAccounts;
-	}
-
-	public List<OnlineAccountData> getOnlineAccounts() {
-		return this.onlineAccounts;
-	}
-
-	public static Message fromByteBuffer(int id, ByteBuffer bytes) throws MessageException {
-		int accountCount = bytes.getInt();
-
-		List<OnlineAccountData> onlineAccounts = new ArrayList<>(accountCount);
-
-		while (accountCount > 0) {
-			long timestamp = bytes.getLong();
-
-			for (int i = 0; i < accountCount; ++i) {
-				byte[] signature = new byte[Transformer.SIGNATURE_LENGTH];
-				bytes.get(signature);
-
-				byte[] publicKey = new byte[Transformer.PUBLIC_KEY_LENGTH];
-				bytes.get(publicKey);
-
-				onlineAccounts.add(new OnlineAccountData(timestamp, signature, publicKey));
-			}
-
-			if (bytes.hasRemaining()) {
-				accountCount = bytes.getInt();
-			} else {
-				// we've finished
-				accountCount = 0;
-			}
-		}
-
-		return new OnlineAccountsV2Message(id, onlineAccounts);
-	}
-
-}

src/main/java/org/qortal/network/message/OnlineAccountsV3Message.java

@@ -99,9 +99,10 @@ public class OnlineAccountsV3Message extends Message {
 				bytes.get(publicKey);
 
 				// Nonce is optional - will be -1 if missing
+				// ... but we should skip/ignore an online account if it has no nonce
 				Integer nonce = bytes.getInt();
 				if (nonce < 0) {
-					nonce = null;
+					continue;
 				}
 
 				onlineAccounts.add(new OnlineAccountData(timestamp, signature, publicKey, nonce));

src/main/java/org/qortal/repository/ArbitraryRepository.java

@@ -24,9 +24,9 @@ public interface ArbitraryRepository {
 	public ArbitraryTransactionData getLatestTransaction(String name, Service service, Method method, String identifier) throws DataException;
 
 
-	public List<ArbitraryResourceInfo> getArbitraryResources(Service service, String identifier, List<String> names, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
+	public List<ArbitraryResourceInfo> getArbitraryResources(Service service, String identifier, List<String> names, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, Integer limit, Integer offset, Boolean reverse) throws DataException;
 
-	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, String name, boolean prefixOnly, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
+	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, List<String> names, boolean prefixOnly, List<String> namesFilter, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, Integer limit, Integer offset, Boolean reverse) throws DataException;
 
 	public List<ArbitraryResourceNameInfo> getArbitraryResourceCreatorNames(Service service, String identifier, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
 

src/main/java/org/qortal/repository/NameRepository.java

@@ -14,10 +14,12 @@ public interface NameRepository {
 
 	public boolean reducedNameExists(String reducedName) throws DataException;
 
-	public List<NameData> getAllNames(Integer limit, Integer offset, Boolean reverse) throws DataException;
+	public List<NameData> searchNames(String query, boolean prefixOnly, Integer limit, Integer offset, Boolean reverse) throws DataException;
+
+	public List<NameData> getAllNames(Long after, Integer limit, Integer offset, Boolean reverse) throws DataException;
 
 	public default List<NameData> getAllNames() throws DataException {
-		return getAllNames(null, null, null);
+		return getAllNames(null, null, null, null);
 	}
 
 	public List<NameData> getNamesForSale(Integer limit, Integer offset, Boolean reverse) throws DataException;

src/main/java/org/qortal/repository/ReindexManager.java

@@ -0,0 +1,213 @@
+package org.qortal.repository;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.block.Block;
+import org.qortal.block.GenesisBlock;
+import org.qortal.controller.Controller;
+import org.qortal.data.block.BlockArchiveData;
+import org.qortal.data.block.BlockData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.block.BlockTransformation;
+import org.qortal.utils.Base58;
+import org.qortal.utils.NTP;
+
+import java.util.concurrent.TimeoutException;
+
+public class ReindexManager {
+
+    private static final Logger LOGGER = LogManager.getLogger(ReindexManager.class);
+
+    private Repository repository;
+
+    private final int pruneAndTrimBlockInterval = 2000;
+    private final int maintenanceBlockInterval = 50000;
+
+    private boolean resume = false;
+
+    public ReindexManager() {
+
+    }
+
+    public void reindex() throws DataException {
+        try {
+            this.runPreChecks();
+            this.rebuildRepository();
+
+            try (final Repository repository = RepositoryManager.getRepository()) {
+                this.repository = repository;
+                this.requestCheckpoint();
+                this.processGenesisBlock();
+                this.processBlocks();
+            }
+
+        } catch (InterruptedException e) {
+            throw new DataException("Interrupted before complete");
+        }
+    }
+
+    private void runPreChecks() throws DataException, InterruptedException {
+        LOGGER.info("Running pre-checks...");
+        if (Settings.getInstance().isTopOnly()) {
+            throw new DataException("Reindexing not supported in top-only mode. Please bootstrap or resync from genesis.");
+        }
+        if (Settings.getInstance().isLite()) {
+            throw new DataException("Reindexing not supported in lite mode.");
+        }
+
+        while (NTP.getTime() == null) {
+            LOGGER.info("Waiting for NTP...");
+            Thread.sleep(5000L);
+        }
+    }
+
+    private void rebuildRepository() throws DataException {
+        if (resume) {
+            return;
+        }
+
+        LOGGER.info("Rebuilding repository...");
+        RepositoryManager.rebuild();
+    }
+
+    private void requestCheckpoint() {
+        RepositoryManager.setRequestedCheckpoint(Boolean.TRUE);
+    }
+
+    private void processGenesisBlock() throws DataException, InterruptedException {
+        if (resume) {
+            return;
+        }
+
+        LOGGER.info("Processing genesis block...");
+
+        GenesisBlock genesisBlock = GenesisBlock.getInstance(repository);
+
+        // Add Genesis Block to blockchain
+        genesisBlock.process();
+
+        this.repository.saveChanges();
+    }
+
+    private void processBlocks() throws DataException {
+        LOGGER.info("Processing blocks...");
+
+        int height = this.repository.getBlockRepository().getBlockchainHeight();
+        while (true) {
+            height++;
+
+            boolean processed = this.processBlock(height);
+            if (!processed) {
+                LOGGER.info("Block {} couldn't be processed. If this is the last archived block, then the process is complete.", height);
+                break; // TODO: check if complete
+            }
+
+            // Prune and trim regularly, leaving a buffer
+            if (height >= pruneAndTrimBlockInterval*2 && height % pruneAndTrimBlockInterval == 0) {
+                int startHeight = Math.max(height - pruneAndTrimBlockInterval*2, 2);
+                int endHeight = height - pruneAndTrimBlockInterval;
+                LOGGER.info("Pruning and trimming blocks {} to {}...", startHeight, endHeight);
+                this.repository.getATRepository().rebuildLatestAtStates(height - 250);
+                this.repository.saveChanges();
+                this.prune(startHeight, endHeight);
+                this.trim(startHeight, endHeight);
+            }
+
+            // Run repository maintenance regularly, to keep blockchain.data size down
+            if (height % maintenanceBlockInterval == 0) {
+                this.runRepositoryMaintenance();
+            }
+        }
+    }
+
+    private boolean processBlock(int height) throws DataException {
+        Block block = this.fetchBlock(height);
+        if (block == null) {
+            return false;
+        }
+
+        // Transactions are stored without approval status so determine that now
+        for (Transaction transaction : block.getTransactions())
+            transaction.setInitialApprovalStatus();
+
+        // It's best not to run preProcess() until there is a reason to
+        // block.preProcess();
+
+        Block.ValidationResult validationResult = block.isValid();
+        if (validationResult != Block.ValidationResult.OK) {
+            throw new DataException(String.format("Invalid block at height %d: %s", height, validationResult));
+        }
+
+        // Save transactions attached to this block
+        for (Transaction transaction : block.getTransactions()) {
+            TransactionData transactionData = transaction.getTransactionData();
+            this.repository.getTransactionRepository().save(transactionData);
+        }
+
+        block.process();
+
+        LOGGER.info(String.format("Reindexed block height %d, sig %.8s", block.getBlockData().getHeight(), Base58.encode(block.getBlockData().getSignature())));
+
+        // Add to block archive table, since this originated from the archive but the chainstate has to be rebuilt
+        this.addToBlockArchive(block.getBlockData());
+
+        this.repository.saveChanges();
+
+        Controller.getInstance().onNewBlock(block.getBlockData());
+
+        return true;
+    }
+
+    private Block fetchBlock(int height) {
+        BlockTransformation b = BlockArchiveReader.getInstance().fetchBlockAtHeight(height);
+        if (b != null) {
+            if (b.getAtStatesHash() != null) {
+                return new Block(this.repository, b.getBlockData(), b.getTransactions(), b.getAtStatesHash());
+            }
+            else {
+                return new Block(this.repository, b.getBlockData(), b.getTransactions(), b.getAtStates());
+            }
+        }
+
+        return null;
+    }
+
+    private void addToBlockArchive(BlockData blockData) throws DataException {
+        // Write the signature and height into the BlockArchive table
+        BlockArchiveData blockArchiveData = new BlockArchiveData(blockData);
+        this.repository.getBlockArchiveRepository().save(blockArchiveData);
+        this.repository.getBlockArchiveRepository().setBlockArchiveHeight(blockData.getHeight()+1);
+        this.repository.saveChanges();
+    }
+
+    private void prune(int startHeight, int endHeight) throws DataException {
+        this.repository.getBlockRepository().pruneBlocks(startHeight, endHeight);
+        this.repository.getATRepository().pruneAtStates(startHeight, endHeight);
+        this.repository.getATRepository().setAtPruneHeight(endHeight+1);
+        this.repository.saveChanges();
+    }
+
+    private void trim(int startHeight, int endHeight) throws DataException {
+        this.repository.getBlockRepository().trimOldOnlineAccountsSignatures(startHeight, endHeight);
+
+        int count = 1; // Any number greater than 0
+        while (count > 0) {
+            count = this.repository.getATRepository().trimAtStates(startHeight, endHeight, Settings.getInstance().getAtStatesTrimLimit());
+        }
+
+        this.repository.getBlockRepository().setBlockPruneHeight(endHeight+1);
+        this.repository.getATRepository().setAtTrimHeight(endHeight+1);
+        this.repository.saveChanges();
+    }
+
+    private void runRepositoryMaintenance() throws DataException {
+        try {
+            this.repository.performPeriodicMaintenance(1000L);
+        } catch (TimeoutException e) {
+            LOGGER.info("Timed out waiting for repository before running maintenance");
+        }
+    }
+
+}

src/main/java/org/qortal/repository/RepositoryManager.java

@@ -2,9 +2,23 @@ package org.qortal.repository;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.qortal.block.Block;
+import org.qortal.crypto.Crypto;
+import org.qortal.data.at.ATData;
+import org.qortal.data.block.BlockData;
+import org.qortal.data.transaction.ATTransactionData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.gui.SplashFrame;
+import org.qortal.settings.Settings;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.block.BlockTransformation;
 
 import java.sql.SQLException;
+import java.util.*;
 import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+
+import static org.qortal.transaction.Transaction.TransactionType.AT;
 
 public abstract class RepositoryManager {
 	private static final Logger LOGGER = LogManager.getLogger(RepositoryManager.class);
@@ -56,6 +70,164 @@ public abstract class RepositoryManager {
 		}
 	}
 
+	public static boolean needsTransactionSequenceRebuild(Repository repository) throws DataException {
+		// Check if we have any transactions without a block_sequence
+		List<byte[]> testSignatures = repository.getTransactionRepository().getSignaturesMatchingCustomCriteria(
+				null, Arrays.asList("block_height IS NOT NULL AND block_sequence IS NULL"), new ArrayList<>(), 100);
+		if (testSignatures.isEmpty()) {
+			// block_sequence intact, so assume complete
+			return false;
+		}
+
+		return true;
+	}
+
+	public static boolean rebuildTransactionSequences(Repository repository) throws DataException {
+		if (Settings.getInstance().isLite()) {
+			// Lite nodes have no blockchain
+			return false;
+		}
+		if (Settings.getInstance().isTopOnly()) {
+			// topOnly nodes are unable to perform this reindex, and so are temporarily unsupported
+			throw new DataException("topOnly nodes are now unsupported, as they are missing data required for a db reshape");
+		}
+
+		try {
+			// Check if we have any unpopulated block_sequence values for the first 1000 blocks
+			if (!needsTransactionSequenceRebuild(repository)) {
+				// block_sequence already populated for the first 1000 blocks, so assume complete.
+				// We avoid checkpointing and prevent the node from starting up in the case of a rebuild failure, so
+				// we shouldn't ever be left in a partially rebuilt state.
+				return false;
+			}
+
+			LOGGER.info("Rebuilding transaction sequences - this will take a while...");
+
+			SplashFrame.getInstance().updateStatus("Rebuilding transactions - please wait...");
+
+			int blockchainHeight = repository.getBlockRepository().getBlockchainHeight();
+			int totalTransactionCount = 0;
+
+			for (int height = 1; height <= blockchainHeight; ++height) {
+				List<TransactionData> inputTransactions = new ArrayList<>();
+
+				// Fetch block and transactions
+				BlockData blockData = repository.getBlockRepository().fromHeight(height);
+				boolean loadedFromArchive = false;
+				if (blockData == null) {
+					// Get (non-AT) transactions from the archive
+					BlockTransformation blockTransformation = BlockArchiveReader.getInstance().fetchBlockAtHeight(height);
+					blockData = blockTransformation.getBlockData();
+					inputTransactions = blockTransformation.getTransactions(); // This doesn't include AT transactions
+					loadedFromArchive = true;
+				}
+				else {
+					// Get transactions from db
+					Block block = new Block(repository, blockData);
+					for (Transaction transaction : block.getTransactions()) {
+						inputTransactions.add(transaction.getTransactionData());
+					}
+				}
+
+				if (blockData == null) {
+					throw new DataException("Missing block data");
+				}
+
+				List<TransactionData> transactions = new ArrayList<>();
+
+				if (loadedFromArchive) {
+					List<TransactionData> transactionDataList = new ArrayList<>(blockData.getTransactionCount());
+					// Fetch any AT transactions in this block
+					List<byte[]> atSignatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, height, height);
+					for (byte[] s : atSignatures) {
+						TransactionData transactionData = repository.getTransactionRepository().fromSignature(s);
+						if (transactionData.getType() == AT) {
+							transactionDataList.add(transactionData);
+						}
+					}
+
+					List<ATTransactionData> atTransactions = new ArrayList<>();
+					for (TransactionData transactionData : transactionDataList) {
+						ATTransactionData atTransactionData = (ATTransactionData) transactionData;
+						atTransactions.add(atTransactionData);
+					}
+
+					// Create sorted list of ATs by creation time
+					List<ATData> ats = new ArrayList<>();
+
+					for (ATTransactionData atTransactionData : atTransactions) {
+						ATData atData = repository.getATRepository().fromATAddress(atTransactionData.getATAddress());
+						boolean hasExistingEntry = ats.stream().anyMatch(a -> Objects.equals(a.getATAddress(), atTransactionData.getATAddress()));
+						if (!hasExistingEntry) {
+							ats.add(atData);
+						}
+					}
+
+					// Sort list of ATs by creation date
+					ats.sort(Comparator.comparingLong(ATData::getCreation));
+
+					// Loop through unique ATs
+					for (ATData atData : ats) {
+						List<ATTransactionData> thisAtTransactions = atTransactions.stream()
+								.filter(t -> Objects.equals(t.getATAddress(), atData.getATAddress()))
+								.collect(Collectors.toList());
+
+						int count = thisAtTransactions.size();
+
+						if (count == 1) {
+							ATTransactionData atTransactionData = thisAtTransactions.get(0);
+							transactions.add(atTransactionData);
+						}
+						else if (count == 2) {
+							String atCreatorAddress = Crypto.toAddress(atData.getCreatorPublicKey());
+
+							ATTransactionData atTransactionData1 = thisAtTransactions.stream()
+									.filter(t -> !Objects.equals(t.getRecipient(), atCreatorAddress))
+									.findFirst().orElse(null);
+							transactions.add(atTransactionData1);
+
+							ATTransactionData atTransactionData2 = thisAtTransactions.stream()
+									.filter(t -> Objects.equals(t.getRecipient(), atCreatorAddress))
+									.findFirst().orElse(null);
+							transactions.add(atTransactionData2);
+						}
+						else if (count > 2) {
+							LOGGER.info("Error: AT has more than 2 output transactions");
+						}
+					}
+				}
+
+				// Add all the regular transactions now that AT transactions have been handled
+				transactions.addAll(inputTransactions);
+				totalTransactionCount += transactions.size();
+
+				// Loop through and update sequences
+				for (int sequence = 0; sequence < transactions.size(); ++sequence) {
+					TransactionData transactionData = transactions.get(sequence);
+
+					// Update transaction's sequence in repository
+					repository.getTransactionRepository().updateBlockSequence(transactionData.getSignature(), sequence);
+				}
+
+				if (height % 10000 == 0) {
+					LOGGER.info("Rebuilt sequences for {} blocks (total transactions: {})", height, totalTransactionCount);
+				}
+
+				repository.saveChanges();
+			}
+
+			LOGGER.info("Completed rebuild of transaction sequences.");
+			return true;
+		}
+		catch (DataException e) {
+			LOGGER.info("Unable to rebuild transaction sequences: {}. The database may have been left in an inconsistent state.", e.getMessage());
+
+			// Throw an exception so that the node startup is halted, allowing for a retry next time.
+			repository.discardChanges();
+			throw new DataException("Rebuild of transaction sequences failed.");
+		}
+	}
+
 	public static void setRequestedCheckpoint(Boolean quick) {
 		quickCheckpointRequested = quick;
 	}

src/main/java/org/qortal/repository/TransactionRepository.java

@@ -125,6 +125,23 @@ public interface TransactionRepository {
 	public List<byte[]> getSignaturesMatchingCustomCriteria(TransactionType txType, List<String> whereClauses,
 															List<Object> bindParams) throws DataException;
 
+	/**
+	 * Returns signatures for transactions that match search criteria, with optional limit.
+	 * <p>
+	 * Alternate version that allows for custom where clauses and bind params.
+	 * Only use for very specific use cases, such as the names integrity check.
+	 * Not advised to be used otherwise, given that it could be possible for
+	 * unsanitized inputs to be passed in if not careful.
+	 *
+	 * @param txType
+	 * @param whereClauses
+	 * @param bindParams
+	 * @return
+	 * @throws DataException
+	 */
+	public List<byte[]> getSignaturesMatchingCustomCriteria(TransactionType txType, List<String> whereClauses,
+															List<Object> bindParams, Integer limit) throws DataException;
+
 	/**
 	 * Returns signature for latest auto-update transaction.
 	 * <p>
@@ -297,7 +314,7 @@ public interface TransactionRepository {
 	 * @return list of transactions, or empty if none.
 	 * @throws DataException
 	 */
-	public List<TransactionData> getUnconfirmedTransactions(EnumSet<TransactionType> excludedTxTypes) throws DataException;
+	public List<TransactionData> getUnconfirmedTransactions(EnumSet<TransactionType> excludedTxTypes, Integer limit) throws DataException;
 
 	/**
 	 * Remove transaction from unconfirmed transactions pile.
@@ -309,6 +326,8 @@ public interface TransactionRepository {
 
 	public void updateBlockHeight(byte[] signature, Integer height) throws DataException;
 
+	public void updateBlockSequence(byte[] signature, Integer sequence) throws DataException;
+
 	public void updateApprovalHeight(byte[] signature, Integer approvalHeight) throws DataException;
 
 	/**

src/main/java/org/qortal/repository/VotingRepository.java

@@ -9,6 +9,8 @@ public interface VotingRepository {
 
 	// Polls
 
+	public List<PollData> getAllPolls(Integer limit, Integer offset, Boolean reverse) throws DataException;
+
 	public PollData fromPollName(String pollName) throws DataException;
 
 	public boolean pollExists(String pollName) throws DataException;

src/main/java/org/qortal/repository/hsqldb/HSQLDBATRepository.java

@@ -296,10 +296,9 @@ public class HSQLDBATRepository implements ATRepository {
 
 	@Override
 	public Integer getATCreationBlockHeight(String atAddress) throws DataException {
-		String sql = "SELECT height "
+		String sql = "SELECT block_height "
 				+ "FROM DeployATTransactions "
-				+ "JOIN BlockTransactions ON transaction_signature = signature "
-				+ "JOIN Blocks ON Blocks.signature = block_signature "
+				+ "JOIN Transactions USING (signature) "
 				+ "WHERE AT_address = ? "
 				+ "LIMIT 1";
 
@@ -877,18 +876,17 @@ public class HSQLDBATRepository implements ATRepository {
 	public NextTransactionInfo findNextTransaction(String recipient, int height, int sequence) throws DataException {
 		// We only need to search for a subset of transaction types: MESSAGE, PAYMENT or AT
 
-		String sql = "SELECT height, sequence, Transactions.signature "
+		String sql = "SELECT block_height, block_sequence, Transactions.signature "
 				+ "FROM ("
 					+ "SELECT signature FROM PaymentTransactions WHERE recipient = ? "
 					+ "UNION "
 					+ "SELECT signature FROM MessageTransactions WHERE recipient = ? "
 					+ "UNION "
 					+ "SELECT signature FROM ATTransactions WHERE recipient = ?"
-				+ ") AS Transactions "
-				+ "JOIN BlockTransactions ON BlockTransactions.transaction_signature = Transactions.signature "
-				+ "JOIN Blocks ON Blocks.signature = BlockTransactions.block_signature "
-				+ "WHERE (height > ? OR (height = ? AND sequence > ?)) "
-				+ "ORDER BY height ASC, sequence ASC "
+				+ ") AS SelectedTransactions "
+				+ "JOIN Transactions USING (signature)"
+				+ "WHERE (block_height > ? OR (block_height = ? AND block_sequence > ?)) "
+				+ "ORDER BY block_height ASC, block_sequence ASC "
 				+ "LIMIT 1";
 
 		Object[] bindParams = new Object[] { recipient, recipient, recipient, height, height, sequence };

src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java

@@ -16,6 +16,7 @@ import org.qortal.arbitrary.ArbitraryDataFile;
 import org.qortal.transaction.ArbitraryTransaction;
 import org.qortal.transaction.Transaction.ApprovalStatus;
 import org.qortal.utils.Base58;
+import org.qortal.utils.ListUtils;
 
 import java.sql.ResultSet;
 import java.sql.SQLException;
@@ -284,7 +285,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 
 	@Override
 	public List<ArbitraryResourceInfo> getArbitraryResources(Service service, String identifier, List<String> names,
-															 boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException {
+															 boolean defaultResource,  Boolean followedOnly, Boolean excludeBlocked,
+															 Integer limit, Integer offset, Boolean reverse) throws DataException {
 		StringBuilder sql = new StringBuilder(512);
 		List<Object> bindParams = new ArrayList<>();
 
@@ -319,6 +321,36 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 			sql.append(")");
 		}
 
+		// Handle "followed only"
+		if (followedOnly != null && followedOnly) {
+			List<String> followedNames = ListUtils.followedNames();
+			if (followedNames != null && !followedNames.isEmpty()) {
+				sql.append(" AND name IN (?");
+				bindParams.add(followedNames.get(0));
+
+				for (int i = 1; i < followedNames.size(); ++i) {
+					sql.append(", ?");
+					bindParams.add(followedNames.get(i));
+				}
+				sql.append(")");
+			}
+		}
+
+		// Handle "exclude blocked"
+		if (excludeBlocked != null && excludeBlocked) {
+			List<String> blockedNames = ListUtils.blockedNames();
+			if (blockedNames != null && !blockedNames.isEmpty()) {
+				sql.append(" AND name NOT IN (?");
+				bindParams.add(blockedNames.get(0));
+
+				for (int i = 1; i < blockedNames.size(); ++i) {
+					sql.append(", ?");
+					bindParams.add(blockedNames.get(i));
+				}
+				sql.append(")");
+			}
+		}
+
 		sql.append(" GROUP BY name, service, identifier ORDER BY name COLLATE SQL_TEXT_UCC_NO_PAD");
 
 		if (reverse != null && reverse) {
@@ -360,8 +392,9 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 	}
 
 	@Override
-	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, String name, boolean prefixOnly,
-															 boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException {
+	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, List<String> names, boolean prefixOnly,
+																List<String> exactMatchNames, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked,
+																Integer limit, Integer offset, Boolean reverse) throws DataException {
 		StringBuilder sql = new StringBuilder(512);
 		List<Object> bindParams = new ArrayList<>();
 
@@ -403,12 +436,60 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 			bindParams.add(queryWildcard);
 		}
 
-		// Handle name matches
-		if (name != null) {
-			// Search anywhere in the identifier, unless "prefixOnly" has been requested
-			String queryWildcard = prefixOnly ? String.format("%s%%", name.toLowerCase()) : String.format("%%%s%%", name.toLowerCase());
-			sql.append(" AND LCASE(name) LIKE ?");
-			bindParams.add(queryWildcard);
+		// Handle name searches
+		if (names != null && !names.isEmpty()) {
+			sql.append(" AND (");
+
+			for (int i = 0; i < names.size(); ++i) {
+				// Search anywhere in the name, unless "prefixOnly" has been requested
+				String queryWildcard = prefixOnly ? String.format("%s%%", names.get(i).toLowerCase()) : String.format("%%%s%%", names.get(i).toLowerCase());
+				if (i > 0) sql.append(" OR ");
+				sql.append("LCASE(name) LIKE ?");
+				bindParams.add(queryWildcard);
+			}
+			sql.append(")");
+		}
+
+		// Handle name exact matches
+		if (exactMatchNames != null && !exactMatchNames.isEmpty()) {
+			sql.append(" AND LCASE(name) IN (?");
+			bindParams.add(exactMatchNames.get(0).toLowerCase());
+
+			for (int i = 1; i < exactMatchNames.size(); ++i) {
+				sql.append(", ?");
+				bindParams.add(exactMatchNames.get(i).toLowerCase());
+			}
+			sql.append(")");
+		}
+
+		// Handle "followed only"
+		if (followedOnly != null && followedOnly) {
+			List<String> followedNames = ListUtils.followedNames();
+			if (followedNames != null && !followedNames.isEmpty()) {
+				sql.append(" AND LCASE(name) IN (?");
+				bindParams.add(followedNames.get(0).toLowerCase());
+
+				for (int i = 1; i < followedNames.size(); ++i) {
+					sql.append(", ?");
+					bindParams.add(followedNames.get(i).toLowerCase());
+				}
+				sql.append(")");
+			}
+		}
+
+		// Handle "exclude blocked"
+		if (excludeBlocked != null && excludeBlocked) {
+			List<String> blockedNames = ListUtils.blockedNames();
+			if (blockedNames != null && !blockedNames.isEmpty()) {
+				sql.append(" AND LCASE(name) NOT IN (?");
+				bindParams.add(blockedNames.get(0).toLowerCase());
+
+				for (int i = 1; i < blockedNames.size(); ++i) {
+					sql.append(", ?");
+					bindParams.add(blockedNames.get(i).toLowerCase());
+				}
+				sql.append(")");
+			}
 		}
 
 		sql.append(" GROUP BY name, service, identifier ORDER BY date_created");

src/main/java/org/qortal/repository/hsqldb/HSQLDBDatabaseUpdates.java

@@ -993,6 +993,17 @@ public class HSQLDBDatabaseUpdates {
 					stmt.execute("ALTER TABLE CancelSellNameTransactions ADD sale_price QortalAmount");
 					break;
 
+				case 47:
+					// Add `block_sequence` to the Transaction table, as the BlockTransactions table is pruned for
+					// older blocks and therefore the sequence becomes unavailable
+					LOGGER.info("Reshaping Transactions table - this can take a while...");
+					stmt.execute("ALTER TABLE Transactions ADD block_sequence INTEGER");
+
+					// For finding transactions by height and sequence
+					LOGGER.info("Adding index to Transactions table - this can take a while...");
+					stmt.execute("CREATE INDEX TransactionHeightSequenceIndex on Transactions (block_height, block_sequence)");
+					break;
+
 				default:
 					// nothing to do
 					return false;

src/main/java/org/qortal/repository/hsqldb/HSQLDBNameRepository.java

@@ -103,12 +103,18 @@ public class HSQLDBNameRepository implements NameRepository {
 		}
 	}
 
-	@Override
-	public List<NameData> getAllNames(Integer limit, Integer offset, Boolean reverse) throws DataException {
-		StringBuilder sql = new StringBuilder(256);
+	public List<NameData> searchNames(String query, boolean prefixOnly, Integer limit, Integer offset, Boolean reverse) throws DataException {
+		StringBuilder sql = new StringBuilder(512);
+		List<Object> bindParams = new ArrayList<>();
 
 		sql.append("SELECT name, reduced_name, owner, data, registered_when, updated_when, "
-				+ "is_for_sale, sale_price, reference, creation_group_id FROM Names ORDER BY name");
+				+ "is_for_sale, sale_price, reference, creation_group_id FROM Names "
+				+ "WHERE LCASE(name) LIKE ? ORDER BY name");
+
+		// Search anywhere in the name, unless "prefixOnly" has been requested
+		// Note that without prefixOnly it will bypass any indexes
+		String queryWildcard = prefixOnly ? String.format("%s%%", query.toLowerCase()) : String.format("%%%s%%", query.toLowerCase());
+		bindParams.add(queryWildcard);
 
 		if (reverse != null && reverse)
 			sql.append(" DESC");
@@ -117,7 +123,64 @@ public class HSQLDBNameRepository implements NameRepository {
 
 		List<NameData> names = new ArrayList<>();
 
-		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString())) {
+		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) {
+			if (resultSet == null)
+				return names;
+
+			do {
+				String name = resultSet.getString(1);
+				String reducedName = resultSet.getString(2);
+				String owner = resultSet.getString(3);
+				String data = resultSet.getString(4);
+				long registered = resultSet.getLong(5);
+
+				// Special handling for possibly-NULL "updated" column
+				Long updated = resultSet.getLong(6);
+				if (updated == 0 && resultSet.wasNull())
+					updated = null;
+
+				boolean isForSale = resultSet.getBoolean(7);
+
+				Long salePrice = resultSet.getLong(8);
+				if (salePrice == 0 && resultSet.wasNull())
+					salePrice = null;
+
+				byte[] reference = resultSet.getBytes(9);
+				int creationGroupId = resultSet.getInt(10);
+
+				names.add(new NameData(name, reducedName, owner, data, registered, updated, isForSale, salePrice, reference, creationGroupId));
+			} while (resultSet.next());
+
+			return names;
+		} catch (SQLException e) {
+			throw new DataException("Unable to search names in repository", e);
+		}
+	}
+
+	@Override
+	public List<NameData> getAllNames(Long after, Integer limit, Integer offset, Boolean reverse) throws DataException {
+		StringBuilder sql = new StringBuilder(256);
+		List<Object> bindParams = new ArrayList<>();
+
+		sql.append("SELECT name, reduced_name, owner, data, registered_when, updated_when, "
+				+ "is_for_sale, sale_price, reference, creation_group_id FROM Names");
+
+		if (after != null) {
+			sql.append(" WHERE registered_when > ? OR updated_when > ?");
+			bindParams.add(after);
+			bindParams.add(after);
+		}
+
+		sql.append(" ORDER BY name");
+
+		if (reverse != null && reverse)
+			sql.append(" DESC");
+
+		HSQLDBRepository.limitOffsetSql(sql, limit, offset);
+
+		List<NameData> names = new ArrayList<>();
+
+		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) {
 			if (resultSet == null)
 				return names;
 

src/main/java/org/qortal/repository/hsqldb/HSQLDBVotingRepository.java

@@ -21,6 +21,55 @@ public class HSQLDBVotingRepository implements VotingRepository {
 
 	// Polls
 
+	@Override
+	public List<PollData> getAllPolls(Integer limit, Integer offset, Boolean reverse) throws DataException {
+		StringBuilder sql = new StringBuilder(512);
+
+		sql.append("SELECT poll_name, description, creator, owner, published_when FROM Polls ORDER BY poll_name");
+
+		if (reverse != null && reverse)
+			sql.append(" DESC");
+
+		HSQLDBRepository.limitOffsetSql(sql, limit, offset);
+
+		List<PollData> polls = new ArrayList<>();
+
+		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString())) {
+			if (resultSet == null)
+				return polls;
+
+			do {
+				String pollName = resultSet.getString(1);
+				String description = resultSet.getString(2);
+				byte[] creatorPublicKey = resultSet.getBytes(3);
+				String owner = resultSet.getString(4);
+				long published = resultSet.getLong(5);
+
+				String optionsSql = "SELECT option_name FROM PollOptions WHERE poll_name = ? ORDER BY option_index ASC";
+				try (ResultSet optionsResultSet = this.repository.checkedExecute(optionsSql, pollName)) {
+					if (optionsResultSet == null)
+						return null;
+
+					List<PollOptionData> pollOptions = new ArrayList<>();
+
+					// NOTE: do-while because checkedExecute() above has already called rs.next() for us
+					do {
+						String optionName = optionsResultSet.getString(1);
+
+						pollOptions.add(new PollOptionData(optionName));
+					} while (optionsResultSet.next());
+
+					polls.add(new PollData(creatorPublicKey, owner, pollName, description, pollOptions, published));
+				}
+				
+			} while (resultSet.next());
+
+			return polls;
+		} catch (SQLException e) {
+			throw new DataException("Unable to fetch polls from repository", e);
+		}
+	}
+
 	@Override
 	public PollData fromPollName(String pollName) throws DataException {
 		String sql = "SELECT description, creator, owner, published_when FROM Polls WHERE poll_name = ?";

src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBTransactionRepository.java

@@ -194,8 +194,7 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 
 	@Override
 	public TransactionData fromHeightAndSequence(int height, int sequence) throws DataException {
-		String sql = "SELECT transaction_signature FROM BlockTransactions JOIN Blocks ON signature = block_signature "
-				+ "WHERE height = ? AND sequence = ?";
+		String sql = "SELECT signature FROM Transactions WHERE block_height = ? AND block_sequence = ?";
 		
 		try (ResultSet resultSet = this.repository.checkedExecute(sql, height, sequence)) {
 			if (resultSet == null)
@@ -657,8 +656,13 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 															List<Object> bindParams) throws DataException {
 		List<byte[]> signatures = new ArrayList<>();
 
+		String txTypeClassName = "";
+		if (txType != null) {
+			txTypeClassName = txType.className;
+		}
+
 		StringBuilder sql = new StringBuilder(1024);
-		sql.append(String.format("SELECT signature FROM %sTransactions", txType.className));
+		sql.append(String.format("SELECT signature FROM %sTransactions", txTypeClassName));
 
 		if (!whereClauses.isEmpty()) {
 			sql.append(" WHERE ");
@@ -690,6 +694,53 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 		}
 	}
 
+	public List<byte[]> getSignaturesMatchingCustomCriteria(TransactionType txType, List<String> whereClauses,
+															List<Object> bindParams, Integer limit) throws DataException {
+		List<byte[]> signatures = new ArrayList<>();
+
+		String txTypeClassName = "";
+		if (txType != null) {
+			txTypeClassName = txType.className;
+		}
+
+		StringBuilder sql = new StringBuilder(1024);
+		sql.append(String.format("SELECT signature FROM %sTransactions", txTypeClassName));
+
+		if (!whereClauses.isEmpty()) {
+			sql.append(" WHERE ");
+
+			final int whereClausesSize = whereClauses.size();
+			for (int wci = 0; wci < whereClausesSize; ++wci) {
+				if (wci != 0)
+					sql.append(" AND ");
+
+				sql.append(whereClauses.get(wci));
+			}
+		}
+
+		if (limit != null) {
+			sql.append(" LIMIT ?");
+			bindParams.add(limit);
+		}
+
+		LOGGER.trace(() -> String.format("Transaction search SQL: %s", sql));
+
+		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) {
+			if (resultSet == null)
+				return signatures;
+
+			do {
+				byte[] signature = resultSet.getBytes(1);
+
+				signatures.add(signature);
+			} while (resultSet.next());
+
+			return signatures;
+		} catch (SQLException e) {
+			throw new DataException("Unable to fetch matching transaction signatures from repository", e);
+		}
+	}
+
 	@Override
 	public byte[] getLatestAutoUpdateTransaction(TransactionType txType, int txGroupId, Integer service) throws DataException {
 		StringBuilder sql = new StringBuilder(1024);
@@ -1378,8 +1429,10 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 	}
 
 	@Override
-	public List<TransactionData> getUnconfirmedTransactions(EnumSet<TransactionType> excludedTxTypes) throws DataException {
+	public List<TransactionData> getUnconfirmedTransactions(EnumSet<TransactionType> excludedTxTypes, Integer limit) throws DataException {
 		StringBuilder sql = new StringBuilder(1024);
+		List<Object> bindParams = new ArrayList<>();
+
 		sql.append("SELECT signature FROM UnconfirmedTransactions ");
 		sql.append("JOIN Transactions USING (signature) ");
 		sql.append("WHERE type NOT IN (");
@@ -1395,12 +1448,17 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 		}
 
 		sql.append(")");
-		sql.append("ORDER BY created_when, signature");
+		sql.append("ORDER BY created_when, signature ");
+
+		if (limit != null) {
+			sql.append("LIMIT ?");
+			bindParams.add(limit);
+		}
 
 		List<TransactionData> transactions = new ArrayList<>();
 
 		// Find transactions with no corresponding row in BlockTransactions
-		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString())) {
+		try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) {
 			if (resultSet == null)
 				return transactions;
 
@@ -1444,6 +1502,19 @@ public class HSQLDBTransactionRepository implements TransactionRepository {
 		}
 	}
 
+	@Override
+	public void updateBlockSequence(byte[] signature, Integer blockSequence) throws DataException {
+		HSQLDBSaver saver = new HSQLDBSaver("Transactions");
+
+		saver.bind("signature", signature).bind("block_sequence", blockSequence);
+
+		try {
+			saver.execute(repository);
+		} catch (SQLException e) {
+			throw new DataException("Unable to update transaction's block sequence in repository", e);
+		}
+	}
+
 	@Override
 	public void updateApprovalHeight(byte[] signature, Integer approvalHeight) throws DataException {
 		HSQLDBSaver saver = new HSQLDBSaver("Transactions");

src/main/java/org/qortal/settings/Settings.java

@@ -47,6 +47,9 @@ public class Settings {
 	private static final int MAINNET_GATEWAY_PORT = 80;
 	private static final int TESTNET_GATEWAY_PORT = 8080;
 
+    private static final int MAINNET_DEV_PROXY_PORT = 12393;
+    private static final int TESTNET_DEV_PROXY_PORT = 62393;
+
 	private static final Logger LOGGER = LogManager.getLogger(Settings.class);
 	private static final String SETTINGS_FILENAME = "settings.json";
 
@@ -61,6 +64,7 @@ public class Settings {
 
 	// Common to all networking (API/P2P)
 	private String bindAddress = "::"; // Use IPv6 wildcard to listen on all local addresses
+	private String bindAddressFallback = "0.0.0.0"; // Some systems are unable to bind using IPv6
 
 	// UI servers
 	private int uiPort = 12388;
@@ -106,6 +110,11 @@ public class Settings {
 	private boolean gatewayLoggingEnabled = false;
 	private boolean gatewayLoopbackEnabled = false;
 
+    // Developer Proxy
+	private Integer devProxyPort;
+    private boolean devProxyLoggingEnabled = false;
+
+
 	// Specific to this node
 	private boolean wipeUnconfirmedOnStart = false;
 	/** Maximum number of unconfirmed transactions allowed per account */
@@ -137,6 +146,9 @@ public class Settings {
 	/* How many blocks to cache locally. Defaulted to 10, which covers a typical Synchronizer request + a few spare */
 	private int blockCacheSize = 10;
 
+	/** Maximum number of transactions for the block minter to include in a block */
+	private int maxTransactionsPerBlock = 25;
+
 	/** How long to keep old, full, AT state data (ms). */
 	private long atStatesMaxLifetime = 5 * 24 * 60 * 60 * 1000L; // milliseconds
 	/** How often to attempt AT state trimming (ms). */
@@ -180,7 +192,7 @@ public class Settings {
 	/** How often to attempt archiving (ms). */
 	private long archiveInterval = 7171L; // milliseconds
 	/** Serialization version to use when building an archive */
-	private int defaultArchiveVersion = 1;
+	private int defaultArchiveVersion = 2;
 
 
 	/** Whether to automatically bootstrap instead of syncing from genesis */
@@ -200,25 +212,25 @@ public class Settings {
 	/** Whether to attempt to open the listen port via UPnP */
 	private boolean uPnPEnabled = true;
 	/** Minimum number of peers to allow block minting / synchronization. */
-	private int minBlockchainPeers = 5;
+	private int minBlockchainPeers = 3;
 	/** Target number of outbound connections to peers we should make. */
 	private int minOutboundPeers = 16;
 	/** Maximum number of peer connections we allow. */
-	private int maxPeers = 36;
+	private int maxPeers = 40;
 	/** Number of slots to reserve for short-lived QDN data transfers */
 	private int maxDataPeers = 4;
 	/** Maximum number of threads for network engine. */
-	private int maxNetworkThreadPoolSize = 32;
+	private int maxNetworkThreadPoolSize = 120;
 	/** Maximum number of threads for network proof-of-work compute, used during handshaking. */
 	private int networkPoWComputePoolSize = 2;
 	/** Maximum number of retry attempts if a peer fails to respond with the requested data */
 	private int maxRetries = 2;
 
 	/** The number of seconds of no activity before recovery mode begins */
-	public long recoveryModeTimeout = 10 * 60 * 1000L;
+	public long recoveryModeTimeout = 24 * 60 * 60 * 1000L;
 
 	/** Minimum peer version number required in order to sync with them */
-	private String minPeerVersion = "3.8.7";
+	private String minPeerVersion = "4.1.2";
 	/** Whether to allow connections with peers below minPeerVersion
 	 * If true, we won't sync with them but they can still sync with us, and will show in the peers list
 	 * If false, sync will be blocked both ways, and they will not appear in the peers list */
@@ -252,6 +264,9 @@ public class Settings {
 	/** Whether to show SysTray pop-up notifications when trade-bot entries change state */
 	private boolean tradebotSystrayEnabled = false;
 
+	/** Maximum buy attempts for each trade offer before it is considered failed, and hidden from the list */
+	private int maxTradeOfferAttempts = 3;
+
 	/** Wallets path - used for storing encrypted wallet caches for coins that require them */
 	private String walletsPath = "wallets";
 
@@ -263,7 +278,7 @@ public class Settings {
 	/** Repository storage path. */
 	private String repositoryPath = "db";
 	/** Repository connection pool size. Needs to be a bit bigger than maxNetworkThreadPoolSize */
-	private int repositoryConnectionPoolSize = 100;
+	private int repositoryConnectionPoolSize = 240;
 	private List<String> fixedNetwork;
 
 	// Export/import
@@ -504,6 +519,9 @@ public class Settings {
 		if (this.minBlockchainPeers < 1 && !singleNodeTestnet)
 			throwValidationError("minBlockchainPeers must be at least 1");
 
+		if (this.topOnly)
+			throwValidationError("topOnly mode is no longer supported");
+
 		if (this.apiKey != null && this.apiKey.trim().length() < 8)
 			throwValidationError("apiKey must be at least 8 characters");
 
@@ -642,6 +660,18 @@ public class Settings {
 	}
 
 
+    public int getDevProxyPort() {
+        if (this.devProxyPort != null)
+            return this.devProxyPort;
+
+        return this.isTestNet ? TESTNET_DEV_PROXY_PORT : MAINNET_DEV_PROXY_PORT;
+    }
+
+    public boolean isDevProxyLoggingEnabled() {
+        return this.devProxyLoggingEnabled;
+    }
+
+
 	public boolean getWipeUnconfirmedOnStart() {
 		return this.wipeUnconfirmedOnStart;
 	}
@@ -666,6 +696,10 @@ public class Settings {
 		return this.blockCacheSize;
 	}
 
+	public int getMaxTransactionsPerBlock() {
+		return this.maxTransactionsPerBlock;
+	}
+
 	public boolean isTestNet() {
 		return this.isTestNet;
 	}
@@ -689,6 +723,10 @@ public class Settings {
 		return this.bindAddress;
 	}
 
+	public String getBindAddressFallback() {
+		return this.bindAddressFallback;
+	}
+
 	public boolean isUPnPEnabled() {
 		return this.uPnPEnabled;
 	}
@@ -766,6 +804,10 @@ public class Settings {
 		return this.pirateChainNet;
 	}
 
+	public int getMaxTradeOfferAttempts() {
+		return this.maxTradeOfferAttempts;
+	}
+
 	public String getWalletsPath() {
 		return this.walletsPath;
 	}

src/main/java/org/qortal/transaction/CancelSellNameTransaction.java

@@ -5,6 +5,7 @@ import java.util.List;
 
 import org.qortal.account.Account;
 import org.qortal.asset.Asset;
+import org.qortal.block.BlockChain;
 import org.qortal.controller.repository.NamesDatabaseIntegrityCheck;
 import org.qortal.data.naming.NameData;
 import org.qortal.data.transaction.CancelSellNameTransactionData;
@@ -65,7 +66,9 @@ public class CancelSellNameTransaction extends Transaction {
 
 		// Check name is currently for sale
 		if (!nameData.isForSale())
-			return ValidationResult.NAME_NOT_FOR_SALE;
+			// Only validate after feature-trigger timestamp, due to a small number of double cancelations in the chain history
+			if (this.cancelSellNameTransactionData.getTimestamp() > BlockChain.getInstance().getCancelSellNameValidationTimestamp())
+				return ValidationResult.NAME_NOT_FOR_SALE;
 
 		// Check transaction creator matches name's current owner
 		Account owner = getOwner();

src/main/java/org/qortal/transaction/ChatTransaction.java

@@ -8,6 +8,7 @@ import java.util.function.Predicate;
 import org.qortal.account.Account;
 import org.qortal.account.PublicKeyAccount;
 import org.qortal.asset.Asset;
+import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
 import org.qortal.crypto.Crypto;
 import org.qortal.crypto.MemoryPoW;
 import org.qortal.data.naming.NameData;
@@ -22,6 +23,7 @@ import org.qortal.settings.Settings;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.transaction.ChatTransactionTransformer;
 import org.qortal.transform.transaction.TransactionTransformer;
+import org.qortal.utils.ListUtils;
 import org.qortal.utils.NTP;
 
 public class ChatTransaction extends Transaction {
@@ -156,8 +158,7 @@ public class ChatTransaction extends Transaction {
 		}
 
 		// Check for blocked author by address
-		ResourceListManager listManager = ResourceListManager.getInstance();
-		if (listManager.listContains("blockedAddresses", this.chatTransactionData.getSender(), true)) {
+		if (ListUtils.isAddressBlocked(this.chatTransactionData.getSender())) {
 			return ValidationResult.ADDRESS_BLOCKED;
 		}
 
@@ -166,7 +167,7 @@ public class ChatTransaction extends Transaction {
 		if (names != null && names.size() > 0) {
 			for (NameData nameData : names) {
 				if (nameData != null && nameData.getName() != null) {
-					if (listManager.listContains("blockedNames", nameData.getName(), false)) {
+					if (ListUtils.isNameBlocked(nameData.getName())) {
 						return ValidationResult.NAME_BLOCKED;
 					}
 				}

src/main/java/org/qortal/transaction/Transaction.java

@@ -641,7 +641,7 @@ public abstract class Transaction {
 		BlockData latestBlockData = repository.getBlockRepository().getLastBlock();
 
 		EnumSet<TransactionType> excludedTxTypes = EnumSet.of(TransactionType.CHAT, TransactionType.PRESENCE);
-		List<TransactionData> unconfirmedTransactions = repository.getTransactionRepository().getUnconfirmedTransactions(excludedTxTypes);
+		List<TransactionData> unconfirmedTransactions = repository.getTransactionRepository().getUnconfirmedTransactions(excludedTxTypes, null);
 
 		unconfirmedTransactions.sort(getDataComparator());
 

src/main/java/org/qortal/utils/BlockArchiveUtils.java

@@ -1,5 +1,7 @@
 package org.qortal.utils;
 
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
 import org.qortal.data.at.ATStateData;
 import org.qortal.data.block.BlockData;
 import org.qortal.data.transaction.TransactionData;
@@ -12,6 +14,8 @@ import java.util.List;
 
 public class BlockArchiveUtils {
 
+    private static final Logger LOGGER = LogManager.getLogger(BlockArchiveUtils.class);
+
     /**
      * importFromArchive
      * <p>
@@ -87,7 +91,8 @@ public class BlockArchiveUtils {
 
             } catch (DataException e) {
                 repository.discardChanges();
-                throw new IllegalStateException("Unable to import blocks from archive");
+                LOGGER.info("Unable to import blocks from archive", e);
+                throw(e);
             }
         }
         repository.saveChanges();

src/main/java/org/qortal/utils/FilesystemUtils.java

@@ -228,12 +228,18 @@ public class FilesystemUtils {
      * @throws IOException
      */
     public static byte[] getSingleFileContents(Path path) throws IOException {
+        return getSingleFileContents(path, null);
+    }
+
+    public static byte[] getSingleFileContents(Path path, Integer maxLength) throws IOException {
         byte[] data = null;
         // TODO: limit the file size that can be loaded into memory
 
         // If the path is a file, read the contents directly
         if (path.toFile().isFile()) {
-            data = Files.readAllBytes(path);
+            int fileSize = (int)path.toFile().length();
+            maxLength = maxLength != null ? Math.min(maxLength, fileSize) : fileSize;
+            data = FilesystemUtils.readFromFile(path.toString(), 0, maxLength);
         }
 
         // Or if it's a directory, only load file contents if there is a single file inside it
@@ -242,7 +248,9 @@ public class FilesystemUtils {
             if (files.length == 1) {
                 Path filePath = Paths.get(path.toString(), files[0]);
                 if (filePath.toFile().isFile()) {
-                    data = Files.readAllBytes(filePath);
+                    int fileSize = (int)filePath.toFile().length();
+                    maxLength = maxLength != null ? Math.min(maxLength, fileSize) : fileSize;
+                    data = FilesystemUtils.readFromFile(filePath.toString(), 0, maxLength);
                 }
             }
         }

src/main/java/org/qortal/utils/ListUtils.java

@@ -0,0 +1,38 @@
+package org.qortal.utils;
+
+import org.qortal.list.ResourceListManager;
+
+import java.util.List;
+
+public class ListUtils {
+
+    /* Blocking */
+
+    public static List<String> blockedNames() {
+        return ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames");
+    }
+
+    public static boolean isNameBlocked(String name) {
+        return ResourceListManager.getInstance().listWithPrefixContains("blockedNames", name, false);
+    }
+
+    public static boolean isAddressBlocked(String address) {
+        return ResourceListManager.getInstance().listWithPrefixContains("blockedAddresses", address, true);
+    }
+
+
+    /* Following */
+
+    public static List<String> followedNames() {
+        return ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames");
+    }
+
+    public static boolean isFollowingName(String name) {
+        return ResourceListManager.getInstance().listWithPrefixContains("followedNames", name, false);
+    }
+
+    public static int followedNamesCount() {
+        return ListUtils.followedNames().size();
+    }
+
+}

src/main/resources/blockchain.json

@@ -86,7 +86,8 @@
 		"selfSponsorshipAlgoV1Height": 1092400,
 		"feeValidationFixTimestamp": 1671918000000,
 		"chatReferenceTimestamp": 1674316800000,
-		"arbitraryOptionalFeeTimestamp": 1680278400000
+		"arbitraryOptionalFeeTimestamp": 1680278400000,
+		"cancelSellNameValidationTimestamp": 1676986362069
 	},
 	"checkpoints": [
 		{ "height": 1136300, "signature": "3BbwawEF2uN8Ni5ofpJXkukoU8ctAPxYoFB7whq9pKfBnjfZcpfEJT4R95NvBDoTP8WDyWvsUvbfHbcr9qSZuYpSKZjUQTvdFf6eqznHGEwhZApWfvXu6zjGCxYCp65F4jsVYYJjkzbjmkCg5WAwN5voudngA23kMK6PpTNygapCzXt" }

src/main/resources/i18n/ApiError_jp.properties

@@ -0,0 +1,83 @@
+#Generated by ResourceBundle Editor (http://essiembre.github.io/eclipse-rbe/)
+# Keys are from api.ApiError enum
+
+# "localeLang": "jp",
+
+### Common ###
+JSON = JSON メッセージの解析に失敗しました
+
+INSUFFICIENT_BALANCE = 残高不足
+
+UNAUTHORIZED = APIコール未承認
+
+REPOSITORY_ISSUE = リポジトリエラー
+
+NON_PRODUCTION = この APIコールはプロダクションシステムでは許可されていません
+
+BLOCKCHAIN_NEEDS_SYNC = ブロックチェーンをまず同期する必要があります
+
+NO_TIME_SYNC = 時刻が未同期
+
+### Validation ###
+INVALID_SIGNATURE = 無効な署名
+
+INVALID_ADDRESS = 無効なアドレス
+
+INVALID_PUBLIC_KEY = 無効な公開鍵
+
+INVALID_DATA = 無効なデータ
+
+INVALID_NETWORK_ADDRESS = 無効なネットワーク アドレス
+
+ADDRESS_UNKNOWN = 不明なアカウントアドレス
+
+INVALID_CRITERIA = 無効な検索条件
+
+INVALID_REFERENCE = 無効な参照
+
+TRANSFORMATION_ERROR = JSONをトランザクションに変換出来ませんでした
+
+INVALID_PRIVATE_KEY = 無効な秘密鍵
+
+INVALID_HEIGHT = 無効なブロック高
+
+CANNOT_MINT = アカウントはミント出来ません
+
+### Blocks ###
+BLOCK_UNKNOWN = 不明なブロック
+
+### Transactions ###
+TRANSACTION_UNKNOWN = 不明なトランザクション
+
+PUBLIC_KEY_NOT_FOUND = 公開鍵が見つかりません
+
+# this one is special in that caller expected to pass two additional strings, hence the two %s
+TRANSACTION_INVALID = 無効なトランザクション: %s (%s)
+
+### Naming ###
+NAME_UNKNOWN = 不明な名前
+
+### Asset ###
+INVALID_ASSET_ID = 無効なアセット ID
+
+INVALID_ORDER_ID = 無効なアセット注文 ID
+
+ORDER_UNKNOWN = 不明なアセット注文 ID
+
+### Groups ###
+GROUP_UNKNOWN = 不明なグループ
+
+### Foreign Blockchain ###
+FOREIGN_BLOCKCHAIN_NETWORK_ISSUE = 外部ブロックチェーンまたはElectrumXネットワークの問題
+
+FOREIGN_BLOCKCHAIN_BALANCE_ISSUE = 外部ブロックチェーンの残高が不足しています
+
+FOREIGN_BLOCKCHAIN_TOO_SOON = 外部ブロックチェーン トランザクションのブロードキャストが時期尚早 (ロックタイム/ブロック時間の中央値)
+
+### Trade Portal ###
+ORDER_SIZE_TOO_SMALL = 注文金額が低すぎます
+
+### Data ###
+FILE_NOT_FOUND = ファイルが見つかりません
+
+NO_REPLY = ピアが制限時間内に応答しませんでした

src/main/resources/i18n/SysTray_jp.properties

@@ -0,0 +1,48 @@
+#Generated by ResourceBundle Editor (http://essiembre.github.io/eclipse-rbe/)
+# SysTray pop-up menu # Japanese translation by R M 2023
+
+APPLYING_UPDATE_AND_RESTARTING = 自動更新を適用して再起動しています...
+
+AUTO_UPDATE = 自動更新
+
+BLOCK_HEIGHT = ブロック高
+
+BLOCKS_REMAINING = 残りのブロック
+
+BUILD_VERSION = ビルドバージョン
+
+CHECK_TIME_ACCURACY = 時刻の精度を確認
+
+CONNECTING = 接続中
+
+CONNECTION = 接続
+
+CONNECTIONS = 接続
+
+CREATING_BACKUP_OF_DB_FILES = データベース ファイルのバックアップを作成中...
+
+DB_BACKUP = データベースのバックアップ
+
+DB_CHECKPOINT = データベースのチェックポイント
+
+DB_MAINTENANCE = データベースのメンテナンス
+
+EXIT = 終了
+
+LITE_NODE = ライトノード
+
+MINTING_DISABLED = ミント一時中止中
+
+MINTING_ENABLED = \u2714 ミント
+
+OPEN_UI = UIを開く
+
+PERFORMING_DB_CHECKPOINT = コミットされていないデータベースの変更を保存中...
+
+PERFORMING_DB_MAINTENANCE = 定期メンテナンスを実行中...
+
+SYNCHRONIZE_CLOCK = 時刻を同期
+
+SYNCHRONIZING_BLOCKCHAIN = ブロックチェーンを同期中
+
+SYNCHRONIZING_CLOCK = 時刻を同期中

src/main/resources/i18n/TransactionValidity_jp.properties

@@ -0,0 +1,195 @@
+#
+
+ACCOUNT_ALREADY_EXISTS = 既にアカウントは存在します
+
+ACCOUNT_CANNOT_REWARD_SHARE = アカウントは報酬シェアが出来ません
+
+ADDRESS_ABOVE_RATE_LIMIT = アドレスが指定されたレート制限に達しました
+
+ADDRESS_BLOCKED = このアドレスはブロックされています
+
+ALREADY_GROUP_ADMIN = 既ににグループ管理者です
+
+ALREADY_GROUP_MEMBER = 既にグループメンバーです
+
+ALREADY_VOTED_FOR_THAT_OPTION = 既にそのオプションに投票しています
+
+ASSET_ALREADY_EXISTS = 既にアセットは存在します
+
+ASSET_DOES_NOT_EXIST = アセットが存在しません
+
+ASSET_DOES_NOT_MATCH_AT = アセットがATのアセットと一致しません
+
+ASSET_NOT_SPENDABLE = 資産が使用不可です
+
+AT_ALREADY_EXISTS = 既にATが存在します
+
+AT_IS_FINISHED = ATが終了しました
+
+AT_UNKNOWN = 不明なAT
+
+BAN_EXISTS = 既にバンされてます
+
+BAN_UNKNOWN = 不明なバン
+
+BANNED_FROM_GROUP = グループからのバンされています
+
+BUYER_ALREADY_OWNER = 既に購入者が所有者です
+
+CLOCK_NOT_SYNCED = 時刻が未同期
+
+DUPLICATE_MESSAGE = このアドレスは重複メッセージを送信しました
+
+DUPLICATE_OPTION = 重複したオプション
+
+GROUP_ALREADY_EXISTS = 既にグループは存在します
+
+GROUP_APPROVAL_DECIDED = 既にグループの承認は決定されています
+
+GROUP_APPROVAL_NOT_REQUIRED = グループ承認が不必要
+
+GROUP_DOES_NOT_EXIST = グループが存在しません
+
+GROUP_ID_MISMATCH = グループ ID が不一致
+
+GROUP_OWNER_CANNOT_LEAVE = グループ所有者はグループを退会出来ません
+
+HAVE_EQUALS_WANT = 持っている資産は欲しい資産と同じです
+
+INCORRECT_NONCE = 不正な PoW ナンス
+
+INSUFFICIENT_FEE = 手数料が不十分です
+
+INVALID_ADDRESS = 無効なアドレス
+
+INVALID_AMOUNT = 無効な金額
+
+INVALID_ASSET_OWNER = 無効なアセット所有者
+
+INVALID_AT_TRANSACTION = 無効なATトランザクション
+
+INVALID_AT_TYPE_LENGTH = 無効なATの「タイプ」の長さです
+
+INVALID_BUT_OK = 無効だがOK
+
+INVALID_CREATION_BYTES = 無効な作成バイト数
+
+INVALID_DATA_LENGTH = 無効なデータ長
+
+INVALID_DESCRIPTION_LENGTH = 無効な概要の長さ
+
+INVALID_GROUP_APPROVAL_THRESHOLD = 無効なグループ承認のしきい値
+
+INVALID_GROUP_BLOCK_DELAY = 無効なグループ承認のブロック遅延
+
+INVALID_GROUP_ID = 無効なグループ ID
+
+INVALID_GROUP_OWNER = 無効なグループ所有者
+
+INVALID_LIFETIME = 無効な有効期間
+
+INVALID_NAME_LENGTH = 無効な名前の長さです
+
+INVALID_NAME_OWNER = 無効な名前の所有者
+
+INVALID_OPTION_LENGTH = 無効なオプションの長さ
+
+INVALID_OPTIONS_COUNT = 無効なオプションの数
+
+INVALID_ORDER_CREATOR = 無効な注文作成者
+
+INVALID_PAYMENTS_COUNT = 無効な入出金数
+
+INVALID_PUBLIC_KEY = 無効な公開鍵
+
+INVALID_QUANTITY = 無効な数量
+
+INVALID_REFERENCE = 無効な参照
+
+INVALID_RETURN = 無効な返品
+
+INVALID_REWARD_SHARE_PERCENT = 無効な報酬シェア率
+
+INVALID_SELLER = 無効な販売者
+
+INVALID_TAGS_LENGTH = 無効な「タグ」の長さ
+
+INVALID_TIMESTAMP_SIGNATURE = 無効なタイムスタンプ署名
+
+INVALID_TX_GROUP_ID = 無効なトランザクション グループ ID
+
+INVALID_VALUE_LENGTH = 無効な「値」の長さ
+
+INVITE_UNKNOWN = 不明なグループ招待
+
+JOIN_REQUEST_EXISTS = 既にグループ参加リクエストが存在します
+
+MAXIMUM_REWARD_SHARES = 既にこのアカウントの報酬シェアは最大です
+
+MISSING_CREATOR = 作成者が見つかりません
+
+MULTIPLE_NAMES_FORBIDDEN = アカウントごとに複数の登録名は禁止されています
+
+NAME_ALREADY_FOR_SALE = 既に名前は販売中です
+
+NAME_ALREADY_REGISTERED = 既に名前は登録されています
+
+NAME_BLOCKED = この名前はブロックされています
+
+NAME_DOES_NOT_EXIST = 名前は存在しません
+
+NAME_NOT_FOR_SALE = 名前は非売品です
+
+NAME_NOT_NORMALIZED = 名前は Unicode の「正規化」形式ではありません
+
+NEGATIVE_AMOUNT = 無効な/負の金額
+
+NEGATIVE_FEE = 無効な/負の料金
+
+NEGATIVE_PRICE = 無効な/負の価格
+
+NO_BALANCE = 残高が不足しています
+
+NO_BLOCKCHAIN_LOCK = ノードのブロックチェーンは現在ビジーです
+
+NO_FLAG_PERMISSION = アカウントにはその権限がありません
+
+NOT_GROUP_ADMIN = アカウントはグループ管理者ではありません
+
+NOT_GROUP_MEMBER = アカウントはグループメンバーではありません
+
+NOT_MINTING_ACCOUNT = アカウントはミント出来ません
+
+NOT_YET_RELEASED = 機能はまだリリースされていません
+
+OK = OK
+
+ORDER_ALREADY_CLOSED = 既に資産取引注文は終了しています
+
+ORDER_DOES_NOT_EXIST = 資産取引注文が存在しません
+
+POLL_ALREADY_EXISTS = 既に投票は存在します
+
+POLL_DOES_NOT_EXIST = 投票は存在しません
+
+POLL_OPTION_DOES_NOT_EXIST = 投票オプションが存在しません
+
+PUBLIC_KEY_UNKNOWN = 不明な公開鍵
+
+REWARD_SHARE_UNKNOWN = 不明な報酬シェア
+
+SELF_SHARE_EXISTS = 既に自己シェア(報酬シェア)が存在します
+
+TIMESTAMP_TOO_NEW = タイムスタンプが新しすぎます
+
+TIMESTAMP_TOO_OLD = タイムスタンプが古すぎます
+
+TOO_MANY_UNCONFIRMED = アカウントに保留中の未承認トランザクションが多すぎます
+
+TRANSACTION_ALREADY_CONFIRMED = 既にトランザクションは承認されています
+
+TRANSACTION_ALREADY_EXISTS = 既にトランザクションは存在します
+
+TRANSACTION_UNKNOWN = 不明なトランザクション
+
+TX_GROUP_ID_MISMATCH = トランザクションのグループIDが一致しません

src/main/resources/q-apps/q-apps-gateway.js

@@ -1,5 +1,37 @@
 console.log("Gateway mode");
 
+function qdnGatewayShowModal(message) {
+    const modalElementId = "qdnGatewayModal";
+
+    if (document.getElementById(modalElementId) != null) {
+        document.body.removeChild(document.getElementById(modalElementId));
+    }
+
+    var modalElement = document.createElement('div');
+    modalElement.style.cssText = 'position:fixed; z-index:99999; background:#fff; padding:20px; border-radius:5px; font-family:sans-serif; bottom:20px; right:20px; color:#000; max-width:400px; box-shadow:0 3px 10px rgb(0 0 0 / 0.2); font-family:arial; font-weight:normal; font-size:16px;';
+    modalElement.innerHTML = message + "<br /><br />";
+    modalElement.id = modalElementId;
+
+    var closeButton = document.createElement('button');
+    closeButton.style.cssText = 'background-color:#008CBA; border:none; color:white; cursor:pointer; float: right; margin: 10px; padding:15px; border-radius:5px; display:inline-block; text-align:center; text-decoration:none; font-family:arial; font-weight:normal; font-size:16px;';
+    closeButton.innerText = "Close";
+    closeButton.addEventListener ("click", function() {
+        document.body.removeChild(document.getElementById(modalElementId));
+    });
+    modalElement.appendChild(closeButton);
+
+    var qortalButton = document.createElement('button');
+    qortalButton.style.cssText = 'background-color:#4CAF50; border:none; color:white; cursor:pointer; float: right; margin: 10px; padding:15px; border-radius:5px; text-align:center; text-decoration:none; display:inline-block; font-family:arial; font-weight:normal; font-size:16px;';
+    qortalButton.innerText = "Learn more";
+    qortalButton.addEventListener ("click", function() {
+        document.body.removeChild(document.getElementById(modalElementId));
+        window.open("https://qortal.org");
+    });
+    modalElement.appendChild(qortalButton);
+
+    document.body.appendChild(modalElement);
+}
+
 window.addEventListener("message", (event) => {
     if (event == null || event.data == null || event.data.length == 0) {
         return;
@@ -18,14 +50,20 @@ window.addEventListener("message", (event) => {
     switch (data.action) {
         case "GET_USER_ACCOUNT":
         case "PUBLISH_QDN_RESOURCE":
+        case "PUBLISH_MULTIPLE_QDN_RESOURCES":
         case "SEND_CHAT_MESSAGE":
         case "JOIN_GROUP":
         case "DEPLOY_AT":
         case "GET_WALLET_BALANCE":
         case "SEND_COIN":
-            const errorString = "Authentication was requested, but this is not yet supported when viewing via a gateway. To use interactive features, please access using the Qortal UI desktop app. More info at: https://qortal.org";
-            alert(errorString);
+        case "GET_LIST_ITEMS":
+        case "ADD_LIST_ITEMS":
+        case "DELETE_LIST_ITEM":
+            const errorString = "Interactive features were requested, but these are not yet supported when viewing via a gateway. To use interactive features, please access using the Qortal UI desktop app. More info at: https://qortal.org";
             response = "{\"error\": \"" + errorString + "\"}"
+
+            const modalText = "This app is powered by the Qortal blockchain. You are viewing in read-only mode. To use interactive features, please access using the Qortal UI desktop app.";
+            qdnGatewayShowModal(modalText);
             break;
 
         default: