Add "percentLoaded" to resource statuses.

Max size for on-chain data is 239 bytes, due to 16-byte IV. Must be a single file resource, without .qortal folder.

.gitignore

@@ -14,7 +14,6 @@
 /.mvn.classpath
 /notes*
 /settings.json
-/testnet*
 /settings*.json
 /testchain*.json
 /run-testnet*.sh

Q-Apps.md

@@ -0,0 +1,741 @@
+# Qortal Project - Q-Apps Documentation
+
+## Introduction
+
+Q-Apps are static web apps written in javascript, HTML, CSS, and other static assets. The key difference between a Q-App and a fully static site is its ability to interact with both the logged-in user and on-chain data. This is achieved using the API described in this document.
+
+
+# Section 0: Basic QDN concepts
+
+## Introduction to QDN resources
+Each published item on QDN (Qortal Data Network) is referred to as a "resource". A resource could contain anything from a few characters of text, to a multi-layered directory structure containing thousands of files.
+
+Resources are stored on-chain, however the data payload is generally stored off-chain, and verified using an on-chain SHA-256 hash.
+
+To publish a resource, a user must first register a name, and then include that name when publishing the data. Accounts without a registered name are unable to publish to QDN from a Q-App at this time.
+
+Owning the name grants update privileges to the data. If that name is later sold or transferred, the permission to update that resource is moved to the new owner.
+
+
+## Name, service & identifier
+
+Each QDN resource has 3 important fields:
+- `name` - the registered name of the account that is publishing the data (which will hold update/edit privileges going forwards).<br /><br />
+- `service` - the type of content (e.g. IMAGE or JSON). Different services have different validation rules. See [list of available services](#services).<br /><br />
+- `identifier` - an optional string to allow more than one resource to exist for a given name/service combination. For example, the name `QortalDemo` may wish to publish multiple images. This can be achieved by using a different identifier string for each. The identifier is only unique to the name in question, and so it doesn't matter if another name is using the same service and identifier string.
+
+
+## Shared identifiers
+
+Since an identifier can be used by multiple names, this can be used to the advantage of Q-App developers as it allows for data to be stored in a deterministic location.
+
+An example of this is the user's avatar. This will always be published with service `THUMBNAIL` and identifier `qortal_avatar`, along with the user's name. So, an app can display the avatar of a user just by specifying their name when requesting the data. The same applies when publishing data.
+
+
+## "Default" resources
+
+A "default" resource refers to one without an identifier. For example, when a website is published via the UI, it will use the user's name and the service `WEBSITE`. These do not have an identifier, and are therefore the "default" website for this name. When requesting or publishing data without an identifier, apps can either omit the `identifier` key entirely, or include `"identifier": "default"` to indicate that the resource(s) being queried or published do not have an identifier.
+
+
+<a name="services"></a>
+## Available service types
+
+Here is a list of currently available services that can be used in Q-Apps:
+
+IMAGE,
+THUMBNAIL,
+VIDEO,
+AUDIO,
+ARBITRARY_DATA,
+JSON,
+DOCUMENT,
+LIST,
+PLAYLIST,
+METADATA,
+BLOG,
+BLOG_POST,
+BLOG_COMMENT,
+GIF_REPOSITORY
+WEBSITE,
+APP,
+QCHAT_ATTACHMENT,
+QCHAT_IMAGE,
+QCHAT_AUDIO,
+QCHAT_VOICE
+
+
+## Single vs multi-file resources
+
+Some resources, such as those published with the `IMAGE` or `JSON` service, consist of a single file or piece of data (the image or the JSON string). This is the most common type of QDN resource, especially in the context of Q-Apps. These can be published by supplying a base64-encoded string containing the data.
+
+Other resources, such as those published with the `WEBSITE`, `APP`, or `GIF_REPOSITORY` service, can contain multiple files and directories. Publishing these kinds of files is not yet available for Q-Apps, however it is possible to retrieve multi-file resources that are already published. When retrieving this data (via FETCH_QDN_RESOURCE), a `filepath` must be included to indicate the file that you would like to retrieve. There is no need to specify a filepath for single file resources, as these will automatically return the contents of the single file.
+
+
+## App-specific data
+
+Some apps may want to make all QDN data for a particular service available. However, others may prefer to only deal with data that has been published by their app (if a specific format/schema is being used for instance).
+
+Identifiers can be used to allow app developers to locate data that has been published by their app. The recommended approach for this is to use the app name as a prefix on all identifiers when publishing data.
+
+For instance, an app called `MyApp` could allow users to publish JSON data. The app could choose to prefix all identifiers with the string `myapp_`, and then use a random string for each published resource (resulting in identifiers such as `myapp_qR5ndZ8v`). Then, to locate data that has potentially been published by users of MyApp, it can later search the QDN database for items with `"service": "JSON"` and `"identifier": "myapp_"`. The SEARCH_QDN_RESOURCES action has a `prefix` option in order to match identifiers beginning with the supplied string.
+
+Note that QDN is a permissionless system, and therefore it's not possible to verify that a resource was actually published by the app. It is recommended that apps validate the contents of the resource to ensure it is formatted correctly, instead of making assumptions.
+
+
+## Updating a resource
+
+To update a resource, it can be overwritten by publishing with the same `name`, `service`, and `identifier` combination. Note that the authenticated account must currently own the name in order to publish an update.
+
+
+## Routing
+
+If a non-existent `filepath` is accessed, the default behaviour of QDN is to return a `404: File not found` error. This includes anything published using the `WEBSITE` service.
+
+However, routing is handled differently for anything published using the `APP` service. 
+
+For apps, QDN automatically sends all unhandled requests to the index file (generally index.html). This allows the app to use custom routing, as it is able to listen on any path. If a file exists at a path, the file itself will be served, and so the request won't be sent to the index file.
+
+It's recommended that all apps return a 404 page if a request isn't able to be routed.
+
+
+# Section 1: Simple links and image loading via HTML
+
+## Section 1a: Linking to other QDN websites / resources
+
+The `qortal://` protocol can be used to access QDN data from within Qortal websites and apps. The basic format is as follows:
+```
+<a href="qortal://{service}/{name}/{identifier}/{path}">link text</a>
+```
+
+However, the system will support the omission of the `identifier` and/or `path` components to allow for simpler URL formats.
+
+A simple link to another website can be achieved with this HTML code:
+```
+<a href="qortal://WEBSITE/QortalDemo">link text</a>
+```
+
+To link to a specific page of another website:
+```
+<a href="qortal://WEBSITE/QortalDemo/minting-leveling/index.html">link text</a>
+```
+
+To link to a standalone resource, such as an avatar
+```
+<a href="qortal://THUMBNAIL/QortalDemo/qortal_avatar">avatar</a>
+```
+
+For cases where you would prefer to explicitly include an identifier (to remove ambiguity) you can use the keyword `default` to access a resource that doesn't have an identifier. For instance:
+```
+<a href="qortal://WEBSITE/QortalDemo/default">link to root of website</a>
+<a href="qortal://WEBSITE/QortalDemo/default/minting-leveling/index.html">link to subpage of website</a>
+```
+
+
+## Section 1b: Linking to other QDN images
+
+The same applies for images, such as displaying an avatar:
+```
+<img src="qortal://THUMBNAIL/QortalDemo/qortal_avatar" />
+```
+
+...or even an image from an entirely different website:
+```
+<img src="qortal://WEBSITE/AlphaX/assets/img/logo.png" />
+```
+
+
+# Section 2: Integrating a Javascript app
+
+Javascript apps allow for much more complex integrations with Qortal's blockchain data.
+
+## Section 2a: Direct API calls
+
+The standard [Qortal Core API](http://localhost:12391/api-documentation) is available to websites and apps, and can be called directly using a standard AJAX request, such as:
+```
+async function getNameInfo(name) {
+    const response = await fetch("/names/" + name);
+    const nameData = await response.json();
+    console.log("nameData: " + JSON.stringify(nameData));
+}
+getNameInfo("QortalDemo");
+```
+
+However, this only works for read-only data, such as looking up transactions, names, balances, etc. Also, since the address of the logged in account can't be retrieved from the core, apps can't show personalized data with this approach.
+
+
+## Section 2b: User interaction via qortalRequest()
+
+To take things a step further, the qortalRequest() function can be used to interact with the user, in order to:
+
+- Request address and public key of the logged in account
+- Publish data to QDN
+- Send chat messages
+- Join groups
+- Deploy ATs (smart contracts)
+- Send QORT or any supported foreign coin
+
+In addition to the above, qortalRequest() also supports many read-only functions that are also available via direct core API calls. Using qortalRequest() helps with futureproofing, as the core APIs can be modified without breaking functionality of existing Q-Apps.
+
+
+### Making a request
+
+Qortal core will automatically inject the `qortalRequest()` javascript function to all websites/apps, which returns a Promise. This can be used to fetch data or publish data to the Qortal blockchain. This functionality supports async/await, as well as try/catch error handling.
+
+```
+async function myfunction() {
+    try {
+        let res = await qortalRequest({
+            action: "GET_ACCOUNT_DATA",
+            address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
+        });
+        console.log(JSON.stringify(res)); // Log the response to the console
+    
+    } catch(e) {
+        console.log("Error: " + JSON.stringify(e));
+    }
+}
+myfunction();
+```
+
+### Timeouts
+
+Request timeouts are handled automatically when using qortalRequest(). The timeout value will differ based on the action being used - see `getDefaultTimeout()` in [q-apps.js](src/main/resources/q-apps/q-apps.js) for the current values.
+
+If a request times out it will throw an error - `The request timed out` - which can be handled by the Q-App.
+
+It is also possible to specify a custom timeout using `qortalRequestWithTimeout(request, timeout)`, however this is discouraged. It's more reliable and futureproof to let the core handle the timeout values.
+
+
+# Section 3: qortalRequest Documentation
+
+## Supported actions
+
+Here is a list of currently supported actions:
+- GET_USER_ACCOUNT
+- GET_ACCOUNT_DATA
+- GET_ACCOUNT_NAMES
+- GET_NAME_DATA
+- LIST_QDN_RESOURCES
+- SEARCH_QDN_RESOURCES
+- GET_QDN_RESOURCE_STATUS
+- GET_QDN_RESOURCE_PROPERTIES
+- FETCH_QDN_RESOURCE
+- PUBLISH_QDN_RESOURCE
+- PUBLISH_MULTIPLE_QDN_RESOURCES
+- GET_WALLET_BALANCE
+- GET_BALANCE
+- SEND_COIN
+- SEARCH_CHAT_MESSAGES
+- SEND_CHAT_MESSAGE
+- LIST_GROUPS
+- JOIN_GROUP
+- DEPLOY_AT
+- GET_AT
+- GET_AT_DATA
+- LIST_ATS
+- FETCH_BLOCK
+- FETCH_BLOCK_RANGE
+- SEARCH_TRANSACTIONS
+- GET_PRICE
+- GET_QDN_RESOURCE_URL
+- LINK_TO_QDN_RESOURCE
+
+More functionality will be added in the future.
+
+## Example Requests
+
+Here are some example requests for each of the above:
+
+### Get address of logged in account
+_Will likely require user approval_
+```
+let account = await qortalRequest({
+     action: "GET_USER_ACCOUNT"
+});
+let address = account.address;
+```
+
+### Get public key of logged in account
+_Will likely require user approval_
+```
+let pubkey = await qortalRequest({
+     action: "GET_USER_ACCOUNT"
+});
+let publicKey = account.publicKey;
+```
+
+### Get account data
+```
+let res = await qortalRequest({
+    action: "GET_ACCOUNT_DATA",
+    address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
+});
+```
+
+### Get names owned by account
+```
+let res = await qortalRequest({
+    action: "GET_ACCOUNT_NAMES",
+    address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
+});
+```
+
+### Get name data
+```
+let res = await qortalRequest({
+    action: "GET_NAME_DATA",
+    name: "QortalDemo"
+});
+```
+
+
+### List QDN resources
+```
+let res = await qortalRequest({
+    action: "LIST_QDN_RESOURCES",
+    service: "THUMBNAIL",
+    name: "QortalDemo", // Optional (exact match)
+    identifier: "qortal_avatar", // Optional (exact match)
+    default: true, // Optional
+    includeStatus: false, // Optional - will take time to respond, so only request if necessary
+    includeMetadata: false, // Optional - will take time to respond, so only request if necessary
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Search QDN resources
+```
+let res = await qortalRequest({
+    action: "SEARCH_QDN_RESOURCES",
+    service: "THUMBNAIL",
+    query: "search query goes here", // Optional - searches both "identifier" and "name" fields
+    identifier: "search query goes here", // Optional - searches only the "identifier" field
+    name: "search query goes here", // Optional - searches only the "name" field
+    prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters
+    default: false, // Optional - if true, only resources without identifiers are returned
+    includeStatus: false, // Optional - will take time to respond, so only request if necessary
+    includeMetadata: false, // Optional - will take time to respond, so only request if necessary
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Fetch QDN single file resource
+```
+let res = await qortalRequest({
+    action: "FETCH_QDN_RESOURCE",
+    name: "QortalDemo",
+    service: "THUMBNAIL",
+    identifier: "qortal_avatar", // Optional. If omitted, the default resource is returned, or you can alternatively use the keyword "default"
+    encoding: "base64", // Optional. If omitted, data is returned in raw form
+    rebuild: false
+});
+```
+
+### Fetch file from multi file QDN resource
+Data is returned in the base64 format
+```
+let res = await qortalRequest({
+    action: "FETCH_QDN_RESOURCE",
+    name: "QortalDemo",
+    service: "WEBSITE",
+    identifier: "default", // Optional. If omitted, the default resource is returned, or you can alternatively request that using the keyword "default", as shown here
+    filepath: "index.html", // Required only for resources containing more than one file
+    rebuild: false
+});
+```
+
+### Get QDN resource status
+```
+let res = await qortalRequest({
+    action: "GET_QDN_RESOURCE_STATUS",
+    name: "QortalDemo",
+    service: "THUMBNAIL",
+    identifier: "qortal_avatar" // Optional
+});
+```
+
+### Get QDN resource properties
+```
+let res = await qortalRequest({
+    action: "GET_QDN_RESOURCE_PROPERTIES",
+    name: "QortalDemo",
+    service: "THUMBNAIL",
+    identifier: "qortal_avatar" // Optional
+});
+// Returns: filename, size, mimeType (where available)
+```
+
+### Publish a single file to QDN
+_Requires user approval_.<br />
+Note: this publishes a single, base64-encoded file. Multi-file resource publishing (such as a WEBSITE or GIF_REPOSITORY) is not yet supported via a Q-App. It will be added in a future update.
+```
+await qortalRequest({
+    action: "PUBLISH_QDN_RESOURCE",
+    name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
+    service: "IMAGE",
+    identifier: "myapp-image1234" // Optional
+    data64: "base64_encoded_data",
+    // filename: "image.jpg", // Optional - to help apps determine the file's type
+    // title: "Title", // Optional
+    // description: "Description", // Optional
+    // category: "TECHNOLOGY", // Optional
+    // tag1: "any", // Optional
+    // tag2: "strings", // Optional
+    // tag3: "can", // Optional
+    // tag4: "go", // Optional
+    // tag5: "here" // Optional
+});
+```
+
+### Publish multiple resources at once to QDN
+_Requires user approval_.<br />
+Note: each resource being published consists of a single, base64-encoded file, each in its own transaction. Useful for publishing two or more related things, such as a video and a video thumbnail.
+```
+await qortalRequest({
+    action: "PUBLISH_MULTIPLE_QDN_RESOURCES",
+    resources: [
+        name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
+        service: "IMAGE",
+        identifier: "myapp-image1234" // Optional
+        data64: "base64_encoded_data",
+        // filename: "image.jpg", // Optional - to help apps determine the file's type
+        // title: "Title", // Optional
+        // description: "Description", // Optional
+        // category: "TECHNOLOGY", // Optional
+        // tag1: "any", // Optional
+        // tag2: "strings", // Optional
+        // tag3: "can", // Optional
+        // tag4: "go", // Optional
+        // tag5: "here" // Optional
+    ],
+    [
+        ... more resources here if needed ...
+    ]
+});
+```
+
+### Get wallet balance (QORT)
+_Requires user approval_
+```
+await qortalRequest({
+    action: "GET_WALLET_BALANCE",
+    coin: "QORT"
+});
+```
+
+
+### Get address or asset balance
+```
+let res = await qortalRequest({
+    action: "GET_BALANCE",
+    address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
+});
+```
+```
+let res = await qortalRequest({
+    action: "GET_BALANCE",
+    assetId: 1,
+    address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
+});
+```
+
+### Send QORT to address
+_Requires user approval_
+```
+await qortalRequest({
+    action: "SEND_COIN",
+    coin: "QORT",
+    destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
+    amount: 1.00000000 // 1 QORT
+});
+```
+
+
+### Search or list chat messages
+```
+let res = await qortalRequest({
+    action: "SEARCH_CHAT_MESSAGES",
+    before: 999999999999999,
+    after: 0,
+    txGroupId: 0, // Optional (must specify either txGroupId or two involving addresses)
+    // involving: ["QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2", "QSefrppsDCsZebcwrqiM1gNbWq7YMDXtG2"], // Optional (must specify either txGroupId or two involving addresses)
+    // reference: "reference", // Optional
+    // chatReference: "chatreference", // Optional
+    // hasChatReference: true, // Optional
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Send a group chat message
+_Requires user approval_
+```
+await qortalRequest({
+    action: "SEND_CHAT_MESSAGE",
+    groupId: 0,
+    message: "Test"
+});
+```
+
+### Send a private chat message
+_Requires user approval_
+```
+await qortalRequest({
+    action: "SEND_CHAT_MESSAGE",
+    destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
+    message: "Test"
+});
+```
+
+### List groups
+```
+let res = await qortalRequest({
+    action: "LIST_GROUPS",
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Join a group
+_Requires user approval_
+```
+await qortalRequest({
+    action: "JOIN_GROUP",
+    groupId: 100
+});
+```
+
+
+### Deploy an AT
+_Requires user approval_
+```
+let res = await qortalRequest({
+    action: "DEPLOY_AT",
+    creationBytes: "12345", // Must be Base58 encoded
+    name: "test name",
+    description: "test description",
+    type: "test type",
+    tags: "test tags",
+    amount: 1.00000000, // 1 QORT
+    assetId: 0,
+    // fee: 0.002 // optional - will use default fee if excluded
+});
+```
+
+### Get AT info
+```
+let res = await qortalRequest({
+    action: "GET_AT",
+    atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH"
+});
+```
+
+### Get AT data bytes (base58 encoded)
+```
+let res = await qortalRequest({
+    action: "GET_AT_DATA",
+    atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH"
+});
+```
+
+### List ATs by functionality
+```
+let res = await qortalRequest({
+    action: "LIST_ATS",
+    codeHash58: "4KdJETRAdymE7dodDmJbf5d9L1bp4g5Nxky8m47TBkvA",
+    isExecutable: true,
+    limit: 100,
+    offset: 0,
+    reverse: true
+});
+```
+
+### Fetch block by signature
+```
+let res = await qortalRequest({
+    action: "FETCH_BLOCK",
+    signature: "875yGFUy1zHV2hmxNWzrhtn9S1zkeD7SQppwdXFysvTXrankCHCz4iyAUgCBM3GjvibbnyRQpriuy1cyu953U1u5uQdzuH3QjQivi9UVwz86z1Akn17MGd5Z5STjpDT7248K6vzMamuqDei57Znonr8GGgn8yyyABn35CbZUCeAuXju"
+});
+```
+
+### Fetch block by height
+```
+let res = await qortalRequest({
+    action: "FETCH_BLOCK",
+    height: "1139850"
+});
+```
+
+### Fetch a range of blocks
+```
+let res = await qortalRequest({
+    action: "FETCH_BLOCK_RANGE",
+    height: "1139800",
+    count: 20,
+    reverse: false
+});
+```
+
+### Search transactions
+```
+let res = await qortalRequest({
+    action: "SEARCH_TRANSACTIONS",
+    // startBlock: 1139000,
+    // blockLimit: 1000,
+    txGroupId: 0,
+    txType: [
+        "PAYMENT",
+        "REWARD_SHARE"
+    ],
+    confirmationStatus: "CONFIRMED",
+    limit: 10,
+    offset: 0,
+    reverse: false
+});
+```
+
+### Get an estimate of the QORT price
+```
+let res = await qortalRequest({
+    action: "GET_PRICE",
+    blockchain: "LITECOIN",
+    // maxtrades: 10,
+    inverse: true
+});
+```
+
+### Get URL to load a QDN resource
+```
+let url = await qortalRequest({
+    action: "GET_QDN_RESOURCE_URL",
+    service: "THUMBNAIL",
+    name: "QortalDemo",
+    identifier: "qortal_avatar"
+    // path: "filename.jpg" // optional - not needed if resource contains only one file
+});
+```
+
+### Get URL to load a QDN website
+```
+let url = await qortalRequest({
+    action: "GET_QDN_RESOURCE_URL",
+    service: "WEBSITE",
+    name: "QortalDemo",
+});
+```
+
+### Get URL to load a specific file from a QDN website
+```
+let url = await qortalRequest({
+    action: "GET_QDN_RESOURCE_URL",
+    service: "WEBSITE",
+    name: "AlphaX",
+    path: "/assets/img/logo.png"
+});
+```
+
+### Link/redirect to another QDN website
+Note: an alternate method is to include `<a href="qortal://WEBSITE/QortalDemo">link text</a>` within your HTML code.
+```
+let res = await qortalRequest({
+    action: "LINK_TO_QDN_RESOURCE",
+    service: "WEBSITE",
+    name: "QortalDemo",
+});
+```
+
+### Link/redirect to a specific path of another QDN website
+Note: an alternate method is to include `<a href="qortal://WEBSITE/QortalDemo/minting-leveling/index.html">link text</a>` within your HTML code.
+```
+let res = await qortalRequest({
+    action: "LINK_TO_QDN_RESOURCE",
+    service: "WEBSITE",
+    name: "QortalDemo",
+    path: "/minting-leveling/index.html"
+});
+```
+
+
+# Section 4: Examples
+
+## Sample App
+
+Here is a sample application to display the logged-in user's avatar:
+```
+<html>
+<head>
+    <script>
+        async function showAvatar() {
+            try {
+                // Get QORT address of logged in account
+                let account = await qortalRequest({
+                    action: "GET_USER_ACCOUNT"
+                });
+                let address = account.address;
+                console.log("address: " + address);
+            
+                // Get names owned by this account
+                let names = await qortalRequest({
+                    action: "GET_ACCOUNT_NAMES",
+                    address: address
+                });
+                console.log("names: " + JSON.stringify(names));
+            
+                if (names.length == 0) {
+                    console.log("User has no registered names");
+                    return;
+                }
+            
+                // Download base64-encoded avatar of the first registered name
+                let avatar = await qortalRequest({
+                    action: "FETCH_QDN_RESOURCE",
+                    name: names[0].name,
+                    service: "THUMBNAIL",
+                    identifier: "qortal_avatar",
+                    encoding: "base64"
+                });
+                console.log("Avatar size: " + avatar.length + " bytes");
+            
+                // Display the avatar image on the screen
+                document.getElementById("avatar").src = "data:image/png;base64," + avatar;
+                
+            } catch(e) {
+                console.log("Error: " + JSON.stringify(e));
+            }
+        }
+    </script>
+</head>
+<body onload="showAvatar()">
+    <img width="500" id="avatar" />
+</body>
+</html>
+```
+
+
+# Section 5: Testing and Development
+
+Publishing an in-development app to mainnet isn't recommended. There are several options for developing and testing a Q-app before publishing to mainnet:
+
+### Preview mode
+
+Select "Preview" in the UI after choosing the zip. This allows for full Q-App testing without the need to publish any data.
+
+
+### Testnets
+
+For an end-to-end test of Q-App publishing, you can use the official testnet, or set up a single node testnet of your own (often referred to as devnet) on your local machine. See [Single Node Testnet Quick Start Guide](testnet/README.md#quick-start).
+
+
+### Debugging
+
+It is recommended that you develop and test in a web browser, to allow access to the javascript console. To do this:
+1. Open the UI app, then minimise it.
+2. In a Chromium-based web browser, visit: http://localhost:12388/
+3. Log in to your account and then preview your app/website.
+4. Go to `View > Developer > JavaScript Console`. Here you can monitor console logs, errors, and network requests from your app, in the same way as any other web-app.

pom.xml

@@ -3,7 +3,7 @@
 	<modelVersion>4.0.0</modelVersion>
 	<groupId>org.qortal</groupId>
 	<artifactId>qortal</artifactId>
-	<version>3.8.8</version>
+	<version>3.9.0</version>
 	<packaging>jar</packaging>
 	<properties>
 		<skipTests>true</skipTests>
@@ -36,6 +36,7 @@
 		<java-diff-utils.version>4.10</java-diff-utils.version>
 		<grpc.version>1.45.1</grpc.version>
 		<protobuf.version>3.19.4</protobuf.version>
+		<simplemagic.version>1.17</simplemagic.version>
 	</properties>
 	<build>
 		<sourceDirectory>src/main/java</sourceDirectory>
@@ -728,5 +729,10 @@
 			<artifactId>protobuf-java</artifactId>
 			<version>${protobuf.version}</version>
 		</dependency>
+		<dependency>
+			<groupId>com.j256.simplemagic</groupId>
+			<artifactId>simplemagic</artifactId>
+			<version>${simplemagic.version}</version>
+		</dependency>
 	</dependencies>
 </project>

src/main/java/org/qortal/api/ApiRequest.java

@@ -3,6 +3,7 @@ package org.qortal.api;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.UnsupportedEncodingException;
+import java.io.Writer;
 import java.net.HttpURLConnection;
 import java.net.InetSocketAddress;
 import java.net.Socket;
@@ -20,14 +21,12 @@ import javax.net.ssl.SNIHostName;
 import javax.net.ssl.SNIServerName;
 import javax.net.ssl.SSLParameters;
 import javax.net.ssl.SSLSocket;
-import javax.xml.bind.JAXBContext;
-import javax.xml.bind.JAXBException;
-import javax.xml.bind.UnmarshalException;
-import javax.xml.bind.Unmarshaller;
+import javax.xml.bind.*;
 import javax.xml.transform.stream.StreamSource;
 
 import org.eclipse.persistence.exceptions.XMLMarshalException;
 import org.eclipse.persistence.jaxb.JAXBContextFactory;
+import org.eclipse.persistence.jaxb.MarshallerProperties;
 import org.eclipse.persistence.jaxb.UnmarshallerProperties;
 
 public class ApiRequest {
@@ -107,6 +106,36 @@ public class ApiRequest {
 		}
 	}
 
+	private static Marshaller createMarshaller(Class<?> objectClass) {
+		try {
+			// Create JAXB context aware of object's class
+			JAXBContext jc = JAXBContextFactory.createContext(new Class[] { objectClass }, null);
+
+			// Create marshaller
+			Marshaller marshaller = jc.createMarshaller();
+
+			// Set the marshaller media type to JSON
+			marshaller.setProperty(MarshallerProperties.MEDIA_TYPE, "application/json");
+
+			// Tell marshaller not to include JSON root element in the output
+			marshaller.setProperty(MarshallerProperties.JSON_INCLUDE_ROOT, false);
+
+			return marshaller;
+		} catch (JAXBException e) {
+			throw new RuntimeException("Unable to create API marshaller", e);
+		}
+	}
+
+	public static void marshall(Writer writer, Object object) throws IOException {
+		Marshaller marshaller = createMarshaller(object.getClass());
+
+		try {
+			marshaller.marshal(object, writer);
+		} catch (JAXBException e) {
+			throw new IOException("Unable to create marshall object for API", e);
+		}
+	}
+
 	public static String getParamsString(Map<String, String> params) {
 		StringBuilder result = new StringBuilder();
 

src/main/java/org/qortal/api/ApiService.java

@@ -13,8 +13,8 @@ import java.security.SecureRandom;
 
 import javax.net.ssl.KeyManagerFactory;
 import javax.net.ssl.SSLContext;
+import javax.servlet.http.HttpServletRequest;
 
-import org.checkerframework.checker.units.qual.A;
 import org.eclipse.jetty.http.HttpVersion;
 import org.eclipse.jetty.rewrite.handler.RedirectPatternRule;
 import org.eclipse.jetty.rewrite.handler.RewriteHandler;
@@ -51,9 +51,11 @@ public class ApiService {
 	private Server server;
 	private ApiKey apiKey;
 
+	public static final String API_VERSION_HEADER = "X-API-VERSION";
+
 	private ApiService() {
 		this.config = new ResourceConfig();
-		this.config.packages("org.qortal.api.resource");
+		this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource");
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);
@@ -230,4 +232,19 @@ public class ApiService {
 		this.server = null;
 	}
 
+	public static int getApiVersion(HttpServletRequest request) {
+		// Get API version
+		String apiVersionString = request.getHeader(API_VERSION_HEADER);
+		if (apiVersionString == null) {
+			// Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141
+			apiVersionString = request.getParameter("apiVersion");
+		}
+
+		int apiVersion = 1;
+		if (apiVersionString != null) {
+			apiVersion = Integer.parseInt(apiVersionString);
+		}
+		return apiVersion;
+	}
+
 }

src/main/java/org/qortal/api/DomainMapService.java

@@ -3,7 +3,6 @@ package org.qortal.api;
 import io.swagger.v3.jaxrs2.integration.resources.OpenApiResource;
 import org.eclipse.jetty.http.HttpVersion;
 import org.eclipse.jetty.rewrite.handler.RewriteHandler;
-import org.eclipse.jetty.rewrite.handler.RewritePatternRule;
 import org.eclipse.jetty.server.*;
 import org.eclipse.jetty.server.handler.ErrorHandler;
 import org.eclipse.jetty.server.handler.InetAccessHandler;
@@ -38,7 +37,7 @@ public class DomainMapService {
 
 	private DomainMapService() {
 		this.config = new ResourceConfig();
-		this.config.packages("org.qortal.api.domainmap.resource");
+		this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource");
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);

src/main/java/org/qortal/api/GatewayService.java

@@ -37,7 +37,7 @@ public class GatewayService {
 
 	private GatewayService() {
 		this.config = new ResourceConfig();
-		this.config.packages("org.qortal.api.gateway.resource");
+		this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource");
 		this.config.register(OpenApiResource.class);
 		this.config.register(ApiDefinition.class);
 		this.config.register(AnnotationPostProcessor.class);

src/main/java/org/qortal/api/HTMLParser.java

@@ -5,6 +5,9 @@ import org.apache.logging.log4j.Logger;
 import org.jsoup.Jsoup;
 import org.jsoup.nodes.Document;
 import org.jsoup.select.Elements;
+import org.qortal.arbitrary.misc.Service;
+
+import java.util.Objects;
 
 public class HTMLParser {
 
@@ -12,11 +15,24 @@ public class HTMLParser {
 
     private String linkPrefix;
     private byte[] data;
+    private String qdnContext;
+    private String resourceId;
+    private Service service;
+    private String identifier;
+    private String path;
+    private String theme;
 
-    public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data) {
-        String inPathWithoutFilename = inPath.substring(0, inPath.lastIndexOf('/'));
+    public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data,
+                      String qdnContext, Service service, String identifier, String theme) {
+        String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : "";
         this.linkPrefix = usePrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : "";
         this.data = data;
+        this.qdnContext = qdnContext;
+        this.resourceId = resourceId;
+        this.service = service;
+        this.identifier = identifier;
+        this.path = inPath;
+        this.theme = theme;
     }
 
     public void addAdditionalHeaderTags() {
@@ -25,6 +41,25 @@ public class HTMLParser {
         String baseUrl = this.linkPrefix + "/";
         Elements head = document.getElementsByTag("head");
         if (!head.isEmpty()) {
+            // Add q-apps script tag
+            String qAppsScriptElement = String.format("<script src=\"/apps/q-apps.js?time=%d\">", System.currentTimeMillis());
+            head.get(0).prepend(qAppsScriptElement);
+
+            // Add q-apps gateway script tag if in gateway mode
+            if (Objects.equals(this.qdnContext, "gateway")) {
+                String qAppsGatewayScriptElement = String.format("<script src=\"/apps/q-apps-gateway.js?time=%d\">", System.currentTimeMillis());
+                head.get(0).prepend(qAppsGatewayScriptElement);
+            }
+
+            // Escape and add vars
+            String service = this.service.toString().replace("\"","\\\"");
+            String name = this.resourceId != null ? this.resourceId.replace("\"","\\\"") : "";
+            String identifier = this.identifier != null ? this.identifier.replace("\"","\\\"") : "";
+            String path = this.path != null ? this.path.replace("\"","\\\"") : "";
+            String theme = this.theme != null ? this.theme.replace("\"","\\\"") : "";
+            String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\";</script>", this.qdnContext, theme, service, name, identifier, path);
+            head.get(0).prepend(qdnContextVar);
+
             // Add base href tag
             String baseElement = String.format("<base href=\"%s\">", baseUrl);
             head.get(0).prepend(baseElement);

src/main/java/org/qortal/api/Security.java

@@ -15,7 +15,21 @@ public abstract class Security {
 
 	public static final String API_KEY_HEADER = "X-API-KEY";
 
+	/**
+	 * Check API call is allowed, retrieving the API key from the request header or GET/POST parameters where required
+	 * @param request
+	 */
 	public static void checkApiCallAllowed(HttpServletRequest request) {
+		checkApiCallAllowed(request, null);
+	}
+
+	/**
+	 * Check API call is allowed, retrieving the API key first from the passedApiKey parameter, with a fallback
+	 * to the request header or GET/POST parameters when null.
+	 * @param request
+	 * @param passedApiKey - the API key to test, or null if it should be retrieved from the request headers.
+	 */
+	public static void checkApiCallAllowed(HttpServletRequest request, String passedApiKey) {
 		// We may want to allow automatic authentication for local requests, if enabled in settings
 		boolean localAuthBypassEnabled = Settings.getInstance().isLocalAuthBypassEnabled();
 		if (localAuthBypassEnabled) {
@@ -38,7 +52,10 @@ public abstract class Security {
 		}
 
 		// We require an API key to be passed
-		String passedApiKey = request.getHeader(API_KEY_HEADER);
+		if (passedApiKey == null) {
+			// API call not passed as a parameter, so try the header
+			passedApiKey = request.getHeader(API_KEY_HEADER);
+		}
 		if (passedApiKey == null) {
 			// Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141
 			passedApiKey = request.getParameter("apiKey");
@@ -56,7 +73,7 @@ public abstract class Security {
 	public static void disallowLoopbackRequests(HttpServletRequest request) {
 		try {
 			InetAddress remoteAddr = InetAddress.getByName(request.getRemoteAddr());
-			if (remoteAddr.isLoopbackAddress()) {
+			if (remoteAddr.isLoopbackAddress() && !Settings.getInstance().isGatewayLoopbackEnabled()) {
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.UNAUTHORIZED, "Local requests not allowed");
 			}
 		} catch (UnknownHostException e) {
@@ -84,9 +101,9 @@ public abstract class Security {
 		}
 	}
 
-	public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier) {
+	public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier, String apiKey) {
 		try {
-			Security.checkApiCallAllowed(request);
+			Security.checkApiCallAllowed(request, apiKey);
 
 		} catch (ApiException e) {
 			// API call wasn't allowed, but maybe it was pre-authorized

src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java

@@ -42,16 +42,16 @@ public class DomainMapResource {
             // Build synchronously, so that we don't need to make the summary API endpoints available over
             // the domain map server. This means that there will be no loading screen, but this is potentially
             // preferred in this situation anyway (e.g. to avoid confusing search engine robots).
-            return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", false, false);
+            return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, null, inPath, null, "", false, false);
         }
         return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
     }
 
-    private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
-                                    String secret58, String prefix, boolean usePrefix, boolean async) {
+    private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
+                                    String inPath, String secret58, String prefix, boolean usePrefix, boolean async) {
 
-        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
-                secret58, prefix, usePrefix, async, request, response, context);
+        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
+                secret58, prefix, usePrefix, async, "domainMap", request, response, context);
         return renderer.render();
     }
 

src/main/java/org/qortal/api/gateway/resource/GatewayResource.java

@@ -2,6 +2,7 @@ package org.qortal.api.gateway.resource;
 
 import io.swagger.v3.oas.annotations.security.SecurityRequirement;
 import io.swagger.v3.oas.annotations.tags.Tag;
+import org.apache.commons.lang3.StringUtils;
 import org.qortal.api.Security;
 import org.qortal.arbitrary.ArbitraryDataFile;
 import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType;
@@ -16,6 +17,10 @@ import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
 
 
 @Path("/")
@@ -76,50 +81,83 @@ public class GatewayResource {
 
 
     @GET
-    @Path("{name}/{path:.*}")
+    @Path("{path:.*}")
     @SecurityRequirement(name = "apiKey")
-    public HttpServletResponse getPathByName(@PathParam("name") String name,
-                                             @PathParam("path") String inPath) {
+    public HttpServletResponse getPath(@PathParam("path") String inPath) {
         // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
         Security.disallowLoopbackRequests(request);
-        return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", true, true);
-    }
-
-    @GET
-    @Path("{name}")
-    @SecurityRequirement(name = "apiKey")
-    public HttpServletResponse getIndexByName(@PathParam("name") String name) {
-        // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
-        Security.disallowLoopbackRequests(request);
-        return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "", true, true);
-    }
-
-
-    // Optional /site alternative for backwards support
-
-    @GET
-    @Path("/site/{name}/{path:.*}")
-    public HttpServletResponse getSitePathByName(@PathParam("name") String name,
-                                                 @PathParam("path") String inPath) {
-        // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
-        Security.disallowLoopbackRequests(request);
-        return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "/site", true, true);
-    }
-
-    @GET
-    @Path("/site/{name}")
-    public HttpServletResponse getSiteIndexByName(@PathParam("name") String name) {
-        // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
-        Security.disallowLoopbackRequests(request);
-        return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "/site", true, true);
+        return this.parsePath(inPath, "gateway", null, true, true);
     }
 
     
-    private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
-                                    String secret58, String prefix, boolean usePrefix, boolean async) {
+    private HttpServletResponse parsePath(String inPath, String qdnContext, String secret58, boolean usePrefix, boolean async) {
 
-        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
-                secret58, prefix, usePrefix, async, request, response, context);
+        if (inPath == null || inPath.equals("")) {
+            // Assume not a real file
+            return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
+        }
+
+        // Default service is WEBSITE
+        Service service = Service.WEBSITE;
+        String name = null;
+        String identifier = null;
+        String outPath = "";
+        List<String> prefixParts = new ArrayList<>();
+
+        if (!inPath.contains("/")) {
+            // Assume entire inPath is a registered name
+            name = inPath;
+        }
+        else {
+            // Parse the path to determine what we need to load
+            List<String> parts = new LinkedList<>(Arrays.asList(inPath.split("/")));
+
+            // Check if the first element is a service
+            try {
+                Service parsedService = Service.valueOf(parts.get(0).toUpperCase());
+                if (parsedService != null) {
+                    // First element matches a service, so we can assume it is one
+                    service = parsedService;
+                    parts.remove(0);
+                    prefixParts.add(service.name());
+                }
+            } catch (IllegalArgumentException e) {
+                // Not a service
+            }
+
+            if (parts.isEmpty()) {
+                // We need more than just a service
+                return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
+            }
+
+            // Service is removed, so assume first element is now a registered name
+            name = parts.get(0);
+            parts.remove(0);
+
+            if (!parts.isEmpty()) {
+                // Name is removed, so check if the first element is now an identifier
+                ArbitraryResourceStatus status = this.getStatus(service, name, parts.get(0), false);
+                if (status.getTotalChunkCount() > 0) {
+                    // Matched service, name and identifier combination - so assume this is an identifier and can be removed
+                    identifier = parts.get(0);
+                    parts.remove(0);
+                    prefixParts.add(identifier);
+                }
+            }
+
+            if (!parts.isEmpty()) {
+                // outPath can be built by combining any remaining parts
+                outPath = String.join("/", parts);
+            }
+        }
+
+        String prefix = StringUtils.join(prefixParts, "/");
+        if (prefix != null && prefix.length() > 0) {
+            prefix = "/" + prefix;
+        }
+
+        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(name, ResourceIdType.NAME, service, identifier, outPath,
+                secret58, prefix, usePrefix, async, qdnContext, request, response, context);
         return renderer.render();
     }
 

src/main/java/org/qortal/api/model/FileProperties.java

@@ -0,0 +1,16 @@
+package org.qortal.api.model;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+
+@XmlAccessorType(XmlAccessType.FIELD)
+public class FileProperties {
+
+	public String filename;
+	public String mimeType;
+	public Long size;
+
+	public FileProperties() {
+	}
+
+}

src/main/java/org/qortal/api/resource/AppsResource.java

@@ -0,0 +1,83 @@
+package org.qortal.api.resource;
+
+import com.google.common.io.Resources;
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.media.Schema;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+import io.swagger.v3.oas.annotations.Hidden;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import org.qortal.api.*;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.*;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MediaType;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+
+
+@Path("/apps")
+@Tag(name = "Apps")
+public class AppsResource {
+
+    @Context HttpServletRequest request;
+    @Context HttpServletResponse response;
+    @Context ServletContext context;
+
+    @GET
+    @Path("/q-apps.js")
+    @Hidden // For internal Q-App API use only
+    @Operation(
+            summary = "Javascript interface for Q-Apps",
+            responses = {
+                    @ApiResponse(
+                            description = "javascript",
+                            content = @Content(
+                                    mediaType = MediaType.TEXT_PLAIN,
+                                    schema = @Schema(
+                                            type = "string"
+                                    )
+                            )
+                    )
+            }
+    )
+    public String getQAppsJs() {
+        URL url = Resources.getResource("q-apps/q-apps.js");
+        try {
+            return Resources.toString(url, StandardCharsets.UTF_8);
+        } catch (IOException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
+        }
+    }
+
+    @GET
+    @Path("/q-apps-gateway.js")
+    @Hidden // For internal Q-App API use only
+    @Operation(
+            summary = "Gateway-specific interface for Q-Apps",
+            responses = {
+                    @ApiResponse(
+                            description = "javascript",
+                            content = @Content(
+                                    mediaType = MediaType.TEXT_PLAIN,
+                                    schema = @Schema(
+                                            type = "string"
+                                    )
+                            )
+                    )
+            }
+    )
+    public String getQAppsGatewayJs() {
+        URL url = Resources.getResource("q-apps/q-apps-gateway.js");
+        try {
+            return Resources.toString(url, StandardCharsets.UTF_8);
+        } catch (IOException e) {
+            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
+        }
+    }
+
+}

src/main/java/org/qortal/api/resource/ArbitraryResource.java

@@ -1,6 +1,8 @@
 package org.qortal.api.resource;
 
 import com.google.common.primitives.Bytes;
+import com.j256.simplemagic.ContentInfo;
+import com.j256.simplemagic.ContentInfoUtil;
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.Parameter;
 import io.swagger.v3.oas.annotations.media.ArraySchema;
@@ -12,11 +14,14 @@ import io.swagger.v3.oas.annotations.security.SecurityRequirement;
 import io.swagger.v3.oas.annotations.tags.Tag;
 
 import java.io.*;
+import java.net.FileNameMap;
+import java.net.URLConnection;
 import java.nio.file.Files;
 import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
+import java.util.Objects;
 
 import javax.servlet.ServletContext;
 import javax.servlet.http.HttpServletRequest;
@@ -25,11 +30,13 @@ import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
 
+import org.apache.commons.io.FileUtils;
 import org.apache.commons.lang3.ArrayUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.bouncycastle.util.encoders.Base64;
 import org.qortal.api.*;
+import org.qortal.api.model.FileProperties;
 import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
 import org.qortal.arbitrary.*;
 import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType;
@@ -38,6 +45,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
 import org.qortal.arbitrary.misc.Category;
 import org.qortal.arbitrary.misc.Service;
 import org.qortal.controller.Controller;
+import org.qortal.controller.arbitrary.ArbitraryDataRenderManager;
 import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
 import org.qortal.controller.arbitrary.ArbitraryMetadataManager;
 import org.qortal.data.account.AccountData;
@@ -88,6 +96,7 @@ public class ArbitraryResource {
 	@ApiErrors({ApiError.REPOSITORY_ISSUE})
 	public List<ArbitraryResourceInfo> getResources(
 			@QueryParam("service") Service service,
+			@QueryParam("name") String name,
 			@QueryParam("identifier") String identifier,
 			@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
 			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@@ -110,9 +119,14 @@ public class ArbitraryResource {
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource");
 			}
 
-			// Load filter from list if needed
+			// Set up name filters if supplied
 			List<String> names = null;
-			if (nameFilter != null) {
+			if (name != null) {
+				// Filter using single name
+				names = Arrays.asList(name);
+			}
+			else if (nameFilter != null) {
+				// Filter using supplied list of names
 				names = ResourceListManager.getInstance().getStringsInList(nameFilter);
 				if (names.isEmpty()) {
 					// List doesn't exist or is empty - so there will be no matches
@@ -128,10 +142,10 @@ public class ArbitraryResource {
 			}
 
 			if (includeStatus != null && includeStatus) {
-				resources = this.addStatusToResources(resources);
+				resources = ArbitraryTransactionUtils.addStatusToResources(resources);
 			}
 			if (includeMetadata != null && includeMetadata) {
-				resources = this.addMetadataToResources(resources);
+				resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
 			}
 
 			return resources;
@@ -155,7 +169,10 @@ public class ArbitraryResource {
 	@ApiErrors({ApiError.REPOSITORY_ISSUE})
 	public List<ArbitraryResourceInfo> searchResources(
 			@QueryParam("service") Service service,
-			@QueryParam("query") String query,
+			@Parameter(description = "Query (searches both name and identifier fields)") @QueryParam("query") String query,
+			@Parameter(description = "Identifier (searches identifier field only)") @QueryParam("identifier") String identifier,
+			@Parameter(description = "Name (searches name field only)") @QueryParam("name") String name,
+			@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
 			@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
 			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
 			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@@ -166,19 +183,20 @@ public class ArbitraryResource {
 		try (final Repository repository = RepositoryManager.getRepository()) {
 
 			boolean defaultRes = Boolean.TRUE.equals(defaultResource);
+			boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
 
 			List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
-					.searchArbitraryResources(service, query, defaultRes, limit, offset, reverse);
+					.searchArbitraryResources(service, query, identifier, name, usePrefixOnly, defaultRes, limit, offset, reverse);
 
 			if (resources == null) {
 				return new ArrayList<>();
 			}
 
 			if (includeStatus != null && includeStatus) {
-				resources = this.addStatusToResources(resources);
+				resources = ArbitraryTransactionUtils.addStatusToResources(resources);
 			}
 			if (includeMetadata != null && includeMetadata) {
-				resources = this.addMetadataToResources(resources);
+				resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
 			}
 
 			return resources;
@@ -232,10 +250,10 @@ public class ArbitraryResource {
 							.getArbitraryResources(service, identifier, Arrays.asList(name), defaultRes, null, null, reverse);
 
 					if (includeStatus != null && includeStatus) {
-						resources = this.addStatusToResources(resources);
+						resources = ArbitraryTransactionUtils.addStatusToResources(resources);
 					}
 					if (includeMetadata != null && includeMetadata) {
-						resources = this.addMetadataToResources(resources);
+						resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
 					}
 
 					creatorName.resources = resources;
@@ -266,10 +284,35 @@ public class ArbitraryResource {
 															@PathParam("name") String name,
 															@QueryParam("build") Boolean build) {
 
-		Security.requirePriorAuthorizationOrApiKey(request, name, service, null);
+		if (!Settings.getInstance().isQDNAuthBypassEnabled())
+			Security.requirePriorAuthorizationOrApiKey(request, name, service, null, apiKey);
+
 		return ArbitraryTransactionUtils.getStatus(service, name, null, build);
 	}
 
+	@GET
+	@Path("/resource/properties/{service}/{name}/{identifier}")
+	@Operation(
+			summary = "Get properties of a QDN resource",
+			description = "This attempts a download of the data if it's not available locally. A filename will only be returned for single file resources. mimeType is only returned when it can be determined.",
+			responses = {
+					@ApiResponse(
+							content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = FileProperties.class))
+					)
+			}
+	)
+	@SecurityRequirement(name = "apiKey")
+	public FileProperties getResourceProperties(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
+											  @PathParam("service") Service service,
+											  @PathParam("name") String name,
+											  @PathParam("identifier") String identifier) {
+
+		if (!Settings.getInstance().isQDNAuthBypassEnabled())
+			Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey);
+
+		return this.getFileProperties(service, name, identifier);
+	}
+
 	@GET
 	@Path("/resource/status/{service}/{name}/{identifier}")
 	@Operation(
@@ -288,7 +331,9 @@ public class ArbitraryResource {
 													 @PathParam("identifier") String identifier,
 													 @QueryParam("build") Boolean build) {
 
-		Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier);
+		if (!Settings.getInstance().isQDNAuthBypassEnabled())
+			Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey);
+
 		return ArbitraryTransactionUtils.getStatus(service, name, identifier, build);
 	}
 
@@ -501,6 +546,9 @@ public class ArbitraryResource {
 			}
 
 			for (ArbitraryTransactionData transactionData : transactionDataList) {
+				if (transactionData.getService() == null) {
+					continue;
+				}
 				ArbitraryResourceInfo arbitraryResourceInfo = new ArbitraryResourceInfo();
 				arbitraryResourceInfo.name = transactionData.getName();
 				arbitraryResourceInfo.service = transactionData.getService();
@@ -511,10 +559,10 @@ public class ArbitraryResource {
 			}
 
 			if (includeStatus != null && includeStatus) {
-				resources = this.addStatusToResources(resources);
+				resources = ArbitraryTransactionUtils.addStatusToResources(resources);
 			}
 			if (includeMetadata != null && includeMetadata) {
-				resources = this.addMetadataToResources(resources);
+				resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
 			}
 
 			return resources;
@@ -544,7 +592,7 @@ public class ArbitraryResource {
 
 		Security.checkApiCallAllowed(request);
 		ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier);
-		return resource.delete();
+		return resource.delete(false);
 	}
 
 	@POST
@@ -641,6 +689,7 @@ public class ArbitraryResource {
 								   @PathParam("service") Service service,
 								   @PathParam("name") String name,
 								   @QueryParam("filepath") String filepath,
+								   @QueryParam("encoding") String encoding,
 								   @QueryParam("rebuild") boolean rebuild,
 								   @QueryParam("async") boolean async,
 								   @QueryParam("attempts") Integer attempts) {
@@ -650,7 +699,7 @@ public class ArbitraryResource {
 			Security.checkApiCallAllowed(request);
 		}
 
-		return this.download(service, name, null, filepath, rebuild, async, attempts);
+		return this.download(service, name, null, filepath, encoding, rebuild, async, attempts);
 	}
 
 	@GET
@@ -676,16 +725,17 @@ public class ArbitraryResource {
 								   @PathParam("name") String name,
 								   @PathParam("identifier") String identifier,
 								   @QueryParam("filepath") String filepath,
+								   @QueryParam("encoding") String encoding,
 								   @QueryParam("rebuild") boolean rebuild,
 								   @QueryParam("async") boolean async,
 								   @QueryParam("attempts") Integer attempts) {
 
 		// Authentication can be bypassed in the settings, for those running public QDN nodes
 		if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
-			Security.checkApiCallAllowed(request);
+			Security.checkApiCallAllowed(request, apiKey);
 		}
 
-		return this.download(service, name, identifier, filepath, rebuild, async, attempts);
+		return this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts);
 	}
 
 
@@ -733,7 +783,7 @@ public class ArbitraryResource {
 			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
 		}
 
-		throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
+		throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
 	}
 
 
@@ -773,6 +823,8 @@ public class ArbitraryResource {
 					   @QueryParam("description") String description,
 					   @QueryParam("tags") List<String> tags,
 					   @QueryParam("category") Category category,
+					   @QueryParam("fee") Long fee,
+					   @QueryParam("preview") Boolean preview,
 					   String path) {
 		Security.checkApiCallAllowed(request);
 
@@ -781,7 +833,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, null, path, null, null, false,
-				title, description, tags, category);
+				fee, null, title, description, tags, category, preview);
 	}
 
 	@POST
@@ -818,6 +870,8 @@ public class ArbitraryResource {
 					   @QueryParam("description") String description,
 					   @QueryParam("tags") List<String> tags,
 					   @QueryParam("category") Category category,
+					   @QueryParam("fee") Long fee,
+					   @QueryParam("preview") Boolean preview,
 					   String path) {
 		Security.checkApiCallAllowed(request);
 
@@ -826,7 +880,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, identifier, path, null, null, false,
-				title, description, tags, category);
+				fee, null, title, description, tags, category, preview);
 	}
 
 
@@ -864,6 +918,9 @@ public class ArbitraryResource {
 										@QueryParam("description") String description,
 										@QueryParam("tags") List<String> tags,
 										@QueryParam("category") Category category,
+										@QueryParam("filename") String filename,
+										@QueryParam("fee") Long fee,
+										@QueryParam("preview") Boolean preview,
 										String base64) {
 		Security.checkApiCallAllowed(request);
 
@@ -872,7 +929,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, null, null, null, base64, false,
-				title, description, tags, category);
+				fee, filename, title, description, tags, category, preview);
 	}
 
 	@POST
@@ -907,6 +964,9 @@ public class ArbitraryResource {
 										@QueryParam("description") String description,
 										@QueryParam("tags") List<String> tags,
 										@QueryParam("category") Category category,
+										@QueryParam("filename") String filename,
+										@QueryParam("fee") Long fee,
+										@QueryParam("preview") Boolean preview,
 										String base64) {
 		Security.checkApiCallAllowed(request);
 
@@ -915,7 +975,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64, false,
-				title, description, tags, category);
+				fee, filename, title, description, tags, category, preview);
 	}
 
 
@@ -952,6 +1012,8 @@ public class ArbitraryResource {
 								 @QueryParam("description") String description,
 								 @QueryParam("tags") List<String> tags,
 								 @QueryParam("category") Category category,
+								 @QueryParam("fee") Long fee,
+								 @QueryParam("preview") Boolean preview,
 								 String base64Zip) {
 		Security.checkApiCallAllowed(request);
 
@@ -960,7 +1022,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, null, null, null, base64Zip, true,
-				title, description, tags, category);
+				fee, null, title, description, tags, category, preview);
 	}
 
 	@POST
@@ -995,6 +1057,8 @@ public class ArbitraryResource {
 								 @QueryParam("description") String description,
 								 @QueryParam("tags") List<String> tags,
 								 @QueryParam("category") Category category,
+								 @QueryParam("fee") Long fee,
+								 @QueryParam("preview") Boolean preview,
 								 String base64Zip) {
 		Security.checkApiCallAllowed(request);
 
@@ -1003,7 +1067,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64Zip, true,
-				title, description, tags, category);
+				fee, null, title, description, tags, category, preview);
 	}
 
 
@@ -1043,6 +1107,9 @@ public class ArbitraryResource {
 							 @QueryParam("description") String description,
 							 @QueryParam("tags") List<String> tags,
 							 @QueryParam("category") Category category,
+							 @QueryParam("filename") String filename,
+							 @QueryParam("fee") Long fee,
+							 @QueryParam("preview") Boolean preview,
 							 String string) {
 		Security.checkApiCallAllowed(request);
 
@@ -1051,7 +1118,7 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, null, null, string, null, false,
-				title, description, tags, category);
+				fee, filename, title, description, tags, category, preview);
 	}
 
 	@POST
@@ -1088,6 +1155,9 @@ public class ArbitraryResource {
 							 @QueryParam("description") String description,
 							 @QueryParam("tags") List<String> tags,
 							 @QueryParam("category") Category category,
+							 @QueryParam("filename") String filename,
+							 @QueryParam("fee") Long fee,
+							 @QueryParam("preview") Boolean preview,
 							 String string) {
 		Security.checkApiCallAllowed(request);
 
@@ -1096,15 +1166,48 @@ public class ArbitraryResource {
 		}
 
 		return this.upload(Service.valueOf(serviceString), name, identifier, null, string, null, false,
-				title, description, tags, category);
+				fee, filename, title, description, tags, category, preview);
 	}
 
 
 	// Shared methods
 
+	private String preview(String directoryPath, Service service) {
+		Security.checkApiCallAllowed(request);
+		ArbitraryTransactionData.Method method = ArbitraryTransactionData.Method.PUT;
+		ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.ZIP;
+
+		ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath),
+				null, service, null, method, compression,
+				null, null, null, null);
+		try {
+			arbitraryDataWriter.save();
+		} catch (IOException | DataException | InterruptedException | MissingDataException e) {
+			LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
+			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
+		} catch (RuntimeException e) {
+			LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
+			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
+		}
+
+		ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile();
+		if (arbitraryDataFile != null) {
+			String digest58 = arbitraryDataFile.digest58();
+			if (digest58 != null) {
+				// Pre-authorize resource
+				ArbitraryDataResource resource = new ArbitraryDataResource(digest58, null, null, null);
+				ArbitraryDataRenderManager.getInstance().addToAuthorizedResources(resource);
+
+				return "/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret());
+			}
+		}
+		return "Unable to generate preview URL";
+	}
+
 	private String upload(Service service, String name, String identifier,
-						  String path, String string, String base64, boolean zipped,
-						  String title, String description, List<String> tags, Category category) {
+						  String path, String string, String base64, boolean zipped, Long fee, String filename,
+						  String title, String description, List<String> tags, Category category,
+						  Boolean preview) {
 		// Fetch public key from registered name
 		try (final Repository repository = RepositoryManager.getRepository()) {
 			NameData nameData = repository.getNameRepository().fromName(name);
@@ -1128,7 +1231,12 @@ public class ArbitraryResource {
 			if (path == null) {
 				// See if we have a string instead
 				if (string != null) {
-					File tempFile = File.createTempFile("qortal-", "");
+					if (filename == null) {
+						// Use current time as filename
+						filename = String.format("qortal-%d", NTP.getTime());
+					}
+					java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-");
+					File tempFile = Paths.get(tempDirectory.toString(), filename).toFile();
 					tempFile.deleteOnExit();
 					BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile.toPath().toString()));
 					writer.write(string);
@@ -1138,7 +1246,12 @@ public class ArbitraryResource {
 				}
 				// ... or base64 encoded raw data
 				else if (base64 != null) {
-					File tempFile = File.createTempFile("qortal-", "");
+					if (filename == null) {
+						// Use current time as filename
+						filename = String.format("qortal-%d", NTP.getTime());
+					}
+					java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-");
+					File tempFile = Paths.get(tempDirectory.toString(), filename).toFile();
 					tempFile.deleteOnExit();
 					Files.write(tempFile.toPath(), Base64.decode(base64));
 					path = tempFile.toPath().toString();
@@ -1167,9 +1280,19 @@ public class ArbitraryResource {
 				}
 			}
 
+			// Finish here if user has requested a preview
+			if (preview != null && preview == true) {
+				return this.preview(path, service);
+			}
+
+			// Default to zero fee if not specified
+			if (fee == null) {
+				fee = 0L;
+			}
+
 			try {
 				ArbitraryDataTransactionBuilder transactionBuilder = new ArbitraryDataTransactionBuilder(
-						repository, publicKey58, Paths.get(path), name, null, service, identifier,
+						repository, publicKey58, fee, Paths.get(path), name, null, service, identifier,
 						title, description, tags, category
 				);
 
@@ -1188,7 +1311,7 @@ public class ArbitraryResource {
 		}
 	}
 
-	private HttpServletResponse download(Service service, String name, String identifier, String filepath, boolean rebuild, boolean async, Integer maxAttempts) {
+	private HttpServletResponse download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts) {
 
 		ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
 		try {
@@ -1247,7 +1370,14 @@ public class ArbitraryResource {
 				String message = String.format("No file exists at filepath: %s", filepath);
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message);
 			}
+
 			byte[] data = Files.readAllBytes(path);
+
+			// Encode the data if requested
+			if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) {
+				data = Base64.encode(data);
+			}
+
 			response.setContentType(context.getMimeType(path.toString()));
 			response.setContentLength(data.length);
 			response.getOutputStream().write(data);
@@ -1259,41 +1389,44 @@ public class ArbitraryResource {
 		}
 	}
 
+	private FileProperties getFileProperties(Service service, String name, String identifier) {
+		ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
+		try {
+			arbitraryDataReader.loadSynchronously(false);
+			java.nio.file.Path outputPath = arbitraryDataReader.getFilePath();
+			if (outputPath == null) {
+				// Assume the resource doesn't exist
+				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, "File not found");
+			}
 
-	private List<ArbitraryResourceInfo> addStatusToResources(List<ArbitraryResourceInfo> resources) {
-		// Determine and add the status of each resource
-		List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
-		for (ArbitraryResourceInfo resourceInfo : resources) {
-			try {
-				ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME,
-						resourceInfo.service, resourceInfo.identifier);
-				ArbitraryResourceStatus status = resource.getStatus(true);
-				if (status != null) {
-					resourceInfo.status = status;
+			FileProperties fileProperties = new FileProperties();
+			fileProperties.size = FileUtils.sizeOfDirectory(outputPath.toFile());
+
+			String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal");
+			if (files.length == 1) {
+				String filename = files[0];
+				java.nio.file.Path filePath = Paths.get(outputPath.toString(), files[0]);
+				ContentInfoUtil util = new ContentInfoUtil();
+				ContentInfo info = util.findMatch(filePath.toFile());
+				String mimeType;
+				if (info != null) {
+					// Attempt to extract MIME type from file contents
+					mimeType = info.getMimeType();
 				}
-				updatedResources.add(resourceInfo);
-
-			} catch (Exception e) {
-				// Catch and log all exceptions, since some systems are experiencing 500 errors when including statuses
-				LOGGER.info("Caught exception when adding status to resource %s: %s", resourceInfo, e.toString());
+				else {
+					// Fall back to using the filename
+					FileNameMap fileNameMap = URLConnection.getFileNameMap();
+					mimeType = fileNameMap.getContentTypeFor(filename);
+				}
+				fileProperties.filename = filename;
+				fileProperties.mimeType = mimeType;
 			}
-		}
-		return updatedResources;
-	}
 
-	private List<ArbitraryResourceInfo> addMetadataToResources(List<ArbitraryResourceInfo> resources) {
-		// Add metadata fields to each resource if they exist
-		List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
-		for (ArbitraryResourceInfo resourceInfo : resources) {
-			ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME,
-					resourceInfo.service, resourceInfo.identifier);
-			ArbitraryDataTransactionMetadata transactionMetadata = resource.getLatestTransactionMetadata();
-			ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata, false);
-			if (resourceMetadata != null) {
-				resourceInfo.metadata = resourceMetadata;
-			}
-			updatedResources.add(resourceInfo);
+			return fileProperties;
+
+		} catch (Exception e) {
+			LOGGER.debug(String.format("Unable to load %s %s: %s", service, name, e.getMessage()));
+			throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
 		}
-		return updatedResources;
 	}
 }

src/main/java/org/qortal/api/resource/BlocksResource.java

@@ -48,6 +48,7 @@ import org.qortal.repository.RepositoryManager;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.block.BlockTransformer;
 import org.qortal.utils.Base58;
+import org.qortal.utils.Triple;
 
 @Path("/blocks")
 @Tag(name = "Blocks")
@@ -165,10 +166,13 @@ public class BlocksResource {
             }
 
             // Not found, so try the block archive
-            byte[] bytes = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, false, repository);
-            if (bytes != null) {
-				if (version != 1) {
-					throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Archived blocks require version 1");
+            Triple<byte[], Integer, Integer> serializedBlock = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, false, repository);
+            if (serializedBlock != null) {
+				byte[] bytes = serializedBlock.getA();
+				Integer serializationVersion = serializedBlock.getB();
+				if (version != serializationVersion) {
+					// TODO: we could quite easily reserialize the block with the requested version
+					throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Block is not stored using requested serialization version.");
 				}
 				return Base58.encode(bytes);
             }

src/main/java/org/qortal/api/resource/ChatResource.java

@@ -40,6 +40,8 @@ import org.qortal.utils.Base58;
 
 import com.google.common.primitives.Bytes;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 @Path("/chat")
 @Tag(name = "Chat")
 public class ChatResource {
@@ -72,6 +74,8 @@ public class ChatResource {
 			@QueryParam("reference") String reference,
 			@QueryParam("chatreference") String chatReference,
 			@QueryParam("haschatreference") Boolean hasChatReference,
+			@QueryParam("sender") String sender,
+			@QueryParam("encoding") Encoding encoding,
 			@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
 			@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
 			@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
@@ -107,6 +111,8 @@ public class ChatResource {
 					chatReferenceBytes,
 					hasChatReference,
 					involvingAddresses,
+					sender,
+					encoding,
 					limit, offset, reverse);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
@@ -129,7 +135,7 @@ public class ChatResource {
 			}
 	)
 	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
-	public ChatMessage getMessageBySignature(@PathParam("signature") String signature58) {
+	public ChatMessage getMessageBySignature(@PathParam("signature") String signature58, @QueryParam("encoding") Encoding encoding) {
 		byte[] signature = Base58.decode(signature58);
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
@@ -139,7 +145,7 @@ public class ChatResource {
 				throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Message not found");
 			}
 
-			return repository.getChatRepository().toChatMessage(chatTransactionData);
+			return repository.getChatRepository().toChatMessage(chatTransactionData, encoding);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		}
@@ -162,12 +168,12 @@ public class ChatResource {
 		}
 	)
 	@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
-	public ActiveChats getActiveChats(@PathParam("address") String address) {
+	public ActiveChats getActiveChats(@PathParam("address") String address, @QueryParam("encoding") Encoding encoding) {
 		if (address == null || !Crypto.isValidAddress(address))
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			return repository.getChatRepository().getActiveChats(address);
+			return repository.getChatRepository().getActiveChats(address, encoding);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		}

src/main/java/org/qortal/api/resource/TransactionsResource.java

@@ -9,6 +9,8 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
 import io.swagger.v3.oas.annotations.responses.ApiResponse;
 import io.swagger.v3.oas.annotations.tags.Tag;
 
+import java.io.IOException;
+import java.io.StringWriter;
 import java.lang.reflect.Constructor;
 import java.lang.reflect.InvocationTargetException;
 import java.util.ArrayList;
@@ -18,19 +20,12 @@ import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.ReentrantLock;
 
 import javax.servlet.http.HttpServletRequest;
-import javax.ws.rs.GET;
-import javax.ws.rs.POST;
-import javax.ws.rs.Path;
-import javax.ws.rs.PathParam;
-import javax.ws.rs.QueryParam;
+import javax.ws.rs.*;
 import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
 
 import org.qortal.account.PrivateKeyAccount;
-import org.qortal.api.ApiError;
-import org.qortal.api.ApiErrors;
-import org.qortal.api.ApiException;
-import org.qortal.api.ApiExceptionFactory;
+import org.qortal.api.*;
 import org.qortal.api.model.SimpleTransactionSignRequest;
 import org.qortal.controller.Controller;
 import org.qortal.controller.LiteNode;
@@ -709,7 +704,7 @@ public class TransactionsResource {
 		),
 		responses = {
 			@ApiResponse(
-				description = "true if accepted, false otherwise",
+				description = "For API version 1, this returns true if accepted.\nFor API version 2, the transactionData is returned as a JSON string if accepted.",
 				content = @Content(
 					mediaType = MediaType.TEXT_PLAIN,
 					schema = @Schema(
@@ -722,7 +717,9 @@ public class TransactionsResource {
 	@ApiErrors({
 		ApiError.BLOCKCHAIN_NEEDS_SYNC, ApiError.INVALID_SIGNATURE, ApiError.INVALID_DATA, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE
 	})
-	public String processTransaction(String rawBytes58) {
+	public String processTransaction(String rawBytes58, @HeaderParam(ApiService.API_VERSION_HEADER) String apiVersionHeader) {
+		int apiVersion = ApiService.getApiVersion(request);
+
 		// Only allow a transaction to be processed if our latest block is less than 60 minutes old
 		// If older than this, we should first wait until the blockchain is synced
 		final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
@@ -759,13 +756,27 @@ public class TransactionsResource {
 				blockchainLock.unlock();
 			}
 
-			return "true";
+			switch (apiVersion) {
+				case 1:
+					return "true";
+
+				case 2:
+				default:
+					// Marshall transactionData to string
+					StringWriter stringWriter = new StringWriter();
+					ApiRequest.marshall(stringWriter, transactionData);
+					return stringWriter.toString();
+			}
+
+
 		} catch (NumberFormatException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA, e);
 		} catch (DataException e) {
 			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
 		} catch (InterruptedException e) {
 			throw createTransactionInvalidException(request, ValidationResult.NO_BLOCKCHAIN_LOCK);
+		} catch (IOException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
 		}
 	}
 

src/main/java/org/qortal/api/restricted/resource/AdminResource.java

@@ -1,4 +1,4 @@
-package org.qortal.api.resource;
+package org.qortal.api.restricted.resource;
 
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.Parameter;
@@ -45,6 +45,7 @@ import org.qortal.block.BlockChain;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
 import org.qortal.controller.Synchronizer.SynchronizationResult;
+import org.qortal.controller.repository.BlockArchiveRebuilder;
 import org.qortal.data.account.MintingAccountData;
 import org.qortal.data.account.RewardShareData;
 import org.qortal.network.Network;
@@ -734,6 +735,64 @@ public class AdminResource {
 		}
 	}
 
+	@POST
+	@Path("/repository/archive/rebuild")
+	@Operation(
+			summary = "Rebuild archive",
+			description = "Rebuilds archive files, using the specified serialization version",
+			requestBody = @RequestBody(
+					required = true,
+					content = @Content(
+							mediaType = MediaType.TEXT_PLAIN,
+							schema = @Schema(
+									type = "number", example = "2"
+							)
+					)
+			),
+			responses = {
+					@ApiResponse(
+							description = "\"true\"",
+							content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
+					)
+			}
+	)
+	@ApiErrors({ApiError.REPOSITORY_ISSUE})
+	@SecurityRequirement(name = "apiKey")
+	public String rebuildArchive(@HeaderParam(Security.API_KEY_HEADER) String apiKey, Integer serializationVersion) {
+		Security.checkApiCallAllowed(request);
+
+		// Default serialization version to value specified in settings
+		if (serializationVersion == null) {
+			serializationVersion = Settings.getInstance().getDefaultArchiveVersion();
+		}
+
+		try {
+			// We don't actually need to lock the blockchain here, but we'll do it anyway so that
+			// the node can focus on rebuilding rather than synchronizing / minting.
+			ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
+
+			blockchainLock.lockInterruptibly();
+
+			try {
+				BlockArchiveRebuilder blockArchiveRebuilder = new BlockArchiveRebuilder(serializationVersion);
+				blockArchiveRebuilder.start();
+
+				return "true";
+
+			} catch (IOException e) {
+				throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA, e);
+
+			} finally {
+				blockchainLock.unlock();
+			}
+		} catch (InterruptedException e) {
+			// We couldn't lock blockchain to perform rebuild
+			return "false";
+		} catch (DataException e) {
+			throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
+		}
+	}
+
 	@DELETE
 	@Path("/repository")
 	@Operation(

src/main/java/org/qortal/api/restricted/resource/BootstrapResource.java

@@ -1,4 +1,4 @@
-package org.qortal.api.resource;
+package org.qortal.api.restricted.resource;
 
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.media.Content;

src/main/java/org/qortal/api/restricted/resource/RenderResource.java

@@ -1,4 +1,4 @@
-package org.qortal.api.resource;
+package org.qortal.api.restricted.resource;
 
 import javax.servlet.ServletContext;
 import javax.servlet.http.HttpServletRequest;
@@ -8,7 +8,6 @@ import javax.ws.rs.core.Context;
 import javax.ws.rs.core.MediaType;
 import java.io.*;
 import java.nio.file.Paths;
-import java.util.Map;
 
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.media.Content;
@@ -28,8 +27,8 @@ import org.qortal.arbitrary.exception.MissingDataException;
 import org.qortal.controller.arbitrary.ArbitraryDataRenderManager;
 import org.qortal.data.transaction.ArbitraryTransactionData.*;
 import org.qortal.repository.DataException;
-import org.qortal.settings.Settings;
 import org.qortal.arbitrary.ArbitraryDataFile.*;
+import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
 
 
@@ -43,60 +42,6 @@ public class RenderResource {
     @Context HttpServletResponse response;
     @Context ServletContext context;
 
-    @POST
-    @Path("/preview")
-    @Operation(
-            summary = "Generate preview URL based on a user-supplied path and service",
-            requestBody = @RequestBody(
-                    required = true,
-                    content = @Content(
-                            mediaType = MediaType.TEXT_PLAIN,
-                            schema = @Schema(
-                                    type = "string", example = "/Users/user/Documents/MyStaticWebsite"
-                            )
-                    )
-            ),
-            responses = {
-                    @ApiResponse(
-                            description = "a temporary URL to preview the website",
-                            content = @Content(
-                                    mediaType = MediaType.TEXT_PLAIN,
-                                    schema = @Schema(
-                                            type = "string"
-                                    )
-                            )
-                    )
-            }
-    )
-    @SecurityRequirement(name = "apiKey")
-    public String preview(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String directoryPath) {
-        Security.checkApiCallAllowed(request);
-        Method method = Method.PUT;
-        Compression compression = Compression.ZIP;
-
-        ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath),
-                null, Service.WEBSITE, null, method, compression,
-                null, null, null, null);
-        try {
-            arbitraryDataWriter.save();
-        } catch (IOException | DataException | InterruptedException | MissingDataException e) {
-            LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
-            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE);
-        } catch (RuntimeException e) {
-            LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
-            throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
-        }
-
-        ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile();
-        if (arbitraryDataFile != null) {
-            String digest58 = arbitraryDataFile.digest58();
-            if (digest58 != null) {
-                return "http://localhost:12393/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret());
-            }
-        }
-        return "Unable to generate preview URL";
-    }
-
     @POST
     @Path("/authorize/{resourceId}")
     @SecurityRequirement(name = "apiKey")
@@ -140,8 +85,10 @@ public class RenderResource {
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature,
                                                    @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
-        return this.get(signature, ResourceIdType.SIGNATURE, null, "/", null, "/render/signature", true, true, theme);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
+
+        return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme);
     }
 
     @GET
@@ -149,8 +96,10 @@ public class RenderResource {
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath,
                                                   @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
-        return this.get(signature, ResourceIdType.SIGNATURE, null, inPath,null, "/render/signature", true, true, theme);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
+
+        return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme);
     }
 
     @GET
@@ -158,8 +107,10 @@ public class RenderResource {
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58,
                                               @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
-        return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, "/", secret58, "/render/hash", true, false, theme);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
+
+        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme);
     }
 
     @GET
@@ -168,8 +119,10 @@ public class RenderResource {
     public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath,
                                              @QueryParam("secret") String secret58,
                                              @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
-        return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, inPath, secret58, "/render/hash", true, false, theme);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
+
+        return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme);
     }
 
     @GET
@@ -178,10 +131,13 @@ public class RenderResource {
     public HttpServletResponse getPathByName(@PathParam("service") Service service,
                                              @PathParam("name") String name,
                                              @PathParam("path") String inPath,
+                                             @QueryParam("identifier") String identifier,
                                              @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, name, service, null);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, name, service, null);
+
         String prefix = String.format("/render/%s", service);
-        return this.get(name, ResourceIdType.NAME, service, inPath, null, prefix, true, true, theme);
+        return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme);
     }
 
     @GET
@@ -189,19 +145,22 @@ public class RenderResource {
     @SecurityRequirement(name = "apiKey")
     public HttpServletResponse getIndexByName(@PathParam("service") Service service,
                                               @PathParam("name") String name,
+                                              @QueryParam("identifier") String identifier,
                                               @QueryParam("theme") String theme) {
-        Security.requirePriorAuthorization(request, name, service, null);
+        if (!Settings.getInstance().isQDNAuthBypassEnabled())
+            Security.requirePriorAuthorization(request, name, service, null);
+
         String prefix = String.format("/render/%s", service);
-        return this.get(name, ResourceIdType.NAME, service, "/", null, prefix, true, true, theme);
+        return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme);
     }
 
 
 
-    private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
-                                    String secret58, String prefix, boolean usePrefix, boolean async, String theme) {
+    private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
+                                    String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String theme) {
 
-        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
-                secret58, prefix, usePrefix, async, request, response, context);
+        ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
+                secret58, prefix, usePrefix, async, "render", request, response, context);
 
         if (theme != null) {
             renderer.setTheme(theme);

src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java

@@ -2,7 +2,9 @@ package org.qortal.api.websocket;
 
 import java.io.IOException;
 import java.io.StringWriter;
+import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.concurrent.atomic.AtomicReference;
 
 import org.eclipse.jetty.websocket.api.Session;
@@ -21,6 +23,8 @@ import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 @WebSocket
 @SuppressWarnings("serial")
 public class ActiveChatsWebSocket extends ApiWebSocket {
@@ -62,7 +66,9 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
 
 	@OnWebSocketMessage
 	public void onWebSocketMessage(Session session, String message) {
-		/* ignored */
+		if (Objects.equals(message, "ping")) {
+			session.getRemote().sendStringByFuture("pong");
+		}
 	}
 
 	private void onNotify(Session session, ChatTransactionData chatTransactionData, String ourAddress, AtomicReference<String> previousOutput) {
@@ -75,7 +81,7 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
 		}
 
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress);
+			ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session));
 
 			StringWriter stringWriter = new StringWriter();
 
@@ -93,4 +99,12 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
 		}
 	}
 
+	private Encoding getTargetEncoding(Session session) {
+		// Default to Base58 if not specified, for backwards support
+		Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
+		List<String> encodingList = queryParams.get("encoding");
+		String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58";
+		return Encoding.valueOf(encoding);
+	}
+
 }

src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java

@@ -2,10 +2,7 @@ package org.qortal.api.websocket;
 
 import java.io.IOException;
 import java.io.StringWriter;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 import org.eclipse.jetty.websocket.api.Session;
 import org.eclipse.jetty.websocket.api.WebSocketException;
@@ -22,6 +19,8 @@ import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 @WebSocket
 @SuppressWarnings("serial")
 public class ChatMessagesWebSocket extends ApiWebSocket {
@@ -35,6 +34,16 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 	@Override
 	public void onWebSocketConnect(Session session) {
 		Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
+		Encoding encoding = getTargetEncoding(session);
+
+		List<String> limitList = queryParams.get("limit");
+		Integer limit = (limitList != null && limitList.size() == 1) ? Integer.parseInt(limitList.get(0)) : null;
+
+		List<String> offsetList = queryParams.get("offset");
+		Integer offset = (offsetList != null && offsetList.size() == 1) ? Integer.parseInt(offsetList.get(0)) : null;
+
+		List<String> reverseList = queryParams.get("offset");
+		Boolean reverse = (reverseList != null && reverseList.size() == 1) ? Boolean.getBoolean(reverseList.get(0)) : null;
 
 		List<String> txGroupIds = queryParams.get("txGroupId");
 		if (txGroupIds != null && txGroupIds.size() == 1) {
@@ -49,7 +58,9 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 						null,
 						null,
 						null,
-						null, null, null);
+						null,
+						encoding,
+						limit, offset, reverse);
 
 				sendMessages(session, chatMessages);
 			} catch (DataException e) {
@@ -79,7 +90,9 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 					null,
 					null,
 					involvingAddresses,
-					null, null, null);
+					null,
+					encoding,
+					limit, offset, reverse);
 
 			sendMessages(session, chatMessages);
 		} catch (DataException e) {
@@ -105,7 +118,9 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 
 	@OnWebSocketMessage
 	public void onWebSocketMessage(Session session, String message) {
-		/* ignored */
+		if (Objects.equals(message, "ping")) {
+			session.getRemote().sendStringByFuture("pong");
+		}
 	}
 
 	private void onNotify(Session session, ChatTransactionData chatTransactionData, int txGroupId) {
@@ -153,7 +168,7 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 		// Convert ChatTransactionData to ChatMessage
 		ChatMessage chatMessage;
 		try (final Repository repository = RepositoryManager.getRepository()) {
-			chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData);
+			chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData, getTargetEncoding(session));
 		} catch (DataException e) {
 			// No output this time?
 			return;
@@ -162,4 +177,12 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
 		sendMessages(session, Collections.singletonList(chatMessage));
 	}
 
+	private Encoding getTargetEncoding(Session session) {
+		// Default to Base58 if not specified, for backwards support
+		Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
+		List<String> encodingList = queryParams.get("encoding");
+		String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58";
+		return Encoding.valueOf(encoding);
+	}
+
 }

src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java

@@ -4,6 +4,7 @@ import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
 import org.qortal.crypto.Crypto;
+import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.repository.DataException;
 import org.qortal.settings.Settings;
 import org.qortal.utils.Base58;
@@ -15,7 +16,6 @@ import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.nio.file.StandardCopyOption;
 import java.util.*;
-import java.util.stream.Stream;
 
 import static java.util.Arrays.stream;
 import static java.util.stream.Collectors.toMap;
@@ -85,6 +85,7 @@ public class ArbitraryDataFile {
             return;
         }
 
+        this.chunks = new ArrayList<>();
         this.hash58 = Base58.encode(Crypto.digest(fileContent));
         this.signature = signature;
         LOGGER.trace(String.format("File digest: %s, size: %d bytes", this.hash58, fileContent.length));
@@ -111,6 +112,41 @@ public class ArbitraryDataFile {
         return ArbitraryDataFile.fromHash58(Base58.encode(hash), signature);
     }
 
+    public static ArbitraryDataFile fromRawData(byte[] data, byte[] signature) throws DataException {
+        if (data == null) {
+            return null;
+        }
+        return new ArbitraryDataFile(data, signature);
+    }
+
+    public static ArbitraryDataFile fromTransactionData(ArbitraryTransactionData transactionData) throws DataException {
+        ArbitraryDataFile arbitraryDataFile = null;
+        byte[] signature = transactionData.getSignature();
+        byte[] data = transactionData.getData();
+
+        if (data == null) {
+            return null;
+        }
+
+        // Create data file
+        switch (transactionData.getDataType()) {
+            case DATA_HASH:
+                arbitraryDataFile = ArbitraryDataFile.fromHash(data, signature);
+                break;
+
+            case RAW_DATA:
+                arbitraryDataFile = ArbitraryDataFile.fromRawData(data, signature);
+                break;
+        }
+
+        // Set metadata hash
+        if (arbitraryDataFile != null) {
+            arbitraryDataFile.setMetadataHash(transactionData.getMetadataHash());
+        }
+
+        return arbitraryDataFile;
+    }
+
     public static ArbitraryDataFile fromPath(Path path, byte[] signature) {
         if (path == null) {
             return null;
@@ -260,6 +296,11 @@ public class ArbitraryDataFile {
             this.chunks = new ArrayList<>();
 
             if (file != null) {
+                if (file.exists() && file.length() <= chunkSize) {
+                    // No need to split into chunks if we're already below the chunk size
+                    return 0;
+                }
+
                 try (FileInputStream fileInputStream = new FileInputStream(file);
                      BufferedInputStream bis = new BufferedInputStream(fileInputStream)) {
 
@@ -388,12 +429,15 @@ public class ArbitraryDataFile {
         return false;
     }
 
-    public boolean deleteAll() {
+    public boolean deleteAll(boolean deleteMetadata) {
         // Delete the complete file
         boolean fileDeleted = this.delete();
 
-        // Delete the metadata file
-        boolean metadataDeleted = this.deleteMetadata();
+        // Delete the metadata file if requested
+        boolean metadataDeleted = false;
+        if (deleteMetadata) {
+            metadataDeleted = this.deleteMetadata();
+        }
 
         // Delete the individual chunks
         boolean chunksDeleted = this.deleteAllChunks();
@@ -612,6 +656,22 @@ public class ArbitraryDataFile {
         return this.chunks.size();
     }
 
+    public int fileCount() {
+        int fileCount = this.chunkCount();
+        
+        if (fileCount == 0) {
+            // Transactions without any chunks can already be treated as a complete file
+            fileCount++;
+        }
+
+        if (this.getMetadataHash() != null) {
+            // Add the metadata file
+            fileCount++;
+        }
+
+        return fileCount;
+    }
+
     public List<ArbitraryDataFileChunk> getChunks() {
         return this.chunks;
     }

src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java

@@ -362,11 +362,6 @@ public class ArbitraryDataReader {
             throw new DataException(String.format("Transaction data not found for signature %s", this.resourceId));
         }
 
-        // Load hashes
-        byte[] digest = transactionData.getData();
-        byte[] metadataHash = transactionData.getMetadataHash();
-        byte[] signature = transactionData.getSignature();
-
         // Load secret
         byte[] secret = transactionData.getSecret();
         if (secret != null) {
@@ -374,16 +369,14 @@ public class ArbitraryDataReader {
         }
 
         // Load data file(s)
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
         ArbitraryTransactionUtils.checkAndRelocateMiscFiles(transactionData);
-        arbitraryDataFile.setMetadataHash(metadataHash);
 
         if (!arbitraryDataFile.allFilesExist()) {
             if (ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())) {
                 throw new DataException(
                         String.format("Unable to request missing data for file %s because the name is blocked", arbitraryDataFile));
-            }
-            else {
+            } else {
                 // Ask the arbitrary data manager to fetch data for this transaction
                 String message;
                 if (this.canRequestMissingFiles) {
@@ -394,8 +387,7 @@ public class ArbitraryDataReader {
                     } else {
                         message = String.format("Unable to reissue request for missing file %s for signature %s due to rate limit. Please try again later.", arbitraryDataFile, Base58.encode(transactionData.getSignature()));
                     }
-                }
-                else {
+                } else {
                     message = String.format("Missing data for file %s", arbitraryDataFile);
                 }
 
@@ -405,21 +397,25 @@ public class ArbitraryDataReader {
             }
         }
 
-        if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) {
-            // We have all the chunks but not the complete file, so join them
-            arbitraryDataFile.join();
+        // Data hashes need some extra processing
+        if (transactionData.getDataType() == DataType.DATA_HASH) {
+            if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) {
+                // We have all the chunks but not the complete file, so join them
+                arbitraryDataFile.join();
+            }
+
+            // If the complete file still doesn't exist then something went wrong
+            if (!arbitraryDataFile.exists()) {
+                throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile));
+            }
+            // Ensure the complete hash matches the joined chunks
+            if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) {
+                // Delete the invalid file
+                arbitraryDataFile.delete();
+                throw new DataException("Unable to validate complete file hash");
+            }
         }
 
-        // If the complete file still doesn't exist then something went wrong
-        if (!arbitraryDataFile.exists()) {
-            throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile));
-        }
-        // Ensure the complete hash matches the joined chunks
-        if (!Arrays.equals(arbitraryDataFile.digest(), digest)) {
-            // Delete the invalid file
-            arbitraryDataFile.delete();
-            throw new DataException("Unable to validate complete file hash");
-        }
         // Ensure the file's size matches the size reported by the transaction (throws a DataException if not)
         arbitraryDataFile.validateFileSize(transactionData.getSize());
 

src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java

@@ -2,6 +2,7 @@ package org.qortal.arbitrary;
 
 import com.google.common.io.Resources;
 import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.ArrayUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 import org.qortal.api.HTMLParser;
@@ -34,28 +35,32 @@ public class ArbitraryDataRenderer {
     private final String resourceId;
     private final ResourceIdType resourceIdType;
     private final Service service;
+    private final String identifier;
     private String theme = "light";
     private String inPath;
     private final String secret58;
     private final String prefix;
     private final boolean usePrefix;
     private final boolean async;
+    private final String qdnContext;
     private final HttpServletRequest request;
     private final HttpServletResponse response;
     private final ServletContext context;
 
-    public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
-                                 String secret58, String prefix, boolean usePrefix, boolean async,
+    public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
+                                 String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String qdnContext,
                                  HttpServletRequest request, HttpServletResponse response, ServletContext context) {
 
         this.resourceId = resourceId;
         this.resourceIdType = resourceIdType;
         this.service = service;
+        this.identifier = identifier != null ? identifier : "default";
         this.inPath = inPath;
         this.secret58 = secret58;
         this.prefix = prefix;
         this.usePrefix = usePrefix;
         this.async = async;
+        this.qdnContext = qdnContext;
         this.request = request;
         this.response = response;
         this.context = context;
@@ -71,14 +76,14 @@ public class ArbitraryDataRenderer {
             return ArbitraryDataRenderer.getResponse(response, 500, "QDN is disabled in settings");
         }
 
-        ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, null);
+        ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, identifier);
         arbitraryDataReader.setSecret58(secret58); // Optional, used for loading encrypted file hashes only
         try {
             if (!arbitraryDataReader.isCachedDataAvailable()) {
                 // If async is requested, show a loading screen whilst build is in progress
                 if (async) {
                     arbitraryDataReader.loadAsynchronously(false, 10);
-                    return this.getLoadingResponse(service, resourceId, theme);
+                    return this.getLoadingResponse(service, resourceId, identifier, theme);
                 }
 
                 // Otherwise, loop until we have data
@@ -111,23 +116,57 @@ public class ArbitraryDataRenderer {
         }
         String unzippedPath = path.toString();
 
+        // Set path automatically for single file resources (except for apps, which handle routing differently)
+        String[] files = ArrayUtils.removeElement(new File(unzippedPath).list(), ".qortal");
+        if (files.length == 1 && this.service != Service.APP) {
+            // This is a single file resource
+            inPath = files[0];
+        }
+
         try {
             String filename = this.getFilename(unzippedPath, inPath);
-            String filePath = Paths.get(unzippedPath, filename).toString();
+            Path filePath = Paths.get(unzippedPath, filename);
+
+            // If the file doesn't exist, we may need to route the request elsewhere, or cleanup
+            if (!Files.exists(filePath)) {
+                if (inPath.equals("/")) {
+                    // Delete the unzipped folder if no index file was found
+                    try {
+                        FileUtils.deleteDirectory(new File(unzippedPath));
+                    } catch (IOException e) {
+                        LOGGER.debug("Unable to delete directory: {}", unzippedPath, e);
+                    }
+                }
+
+                // If this is an app, then forward all unhandled requests to the index, to give the app the option to route it
+                if (this.service == Service.APP) {
+                    // Locate index file
+                    List<String> indexFiles = ArbitraryDataRenderer.indexFiles();
+                    for (String indexFile : indexFiles) {
+                        Path indexPath = Paths.get(unzippedPath, indexFile);
+                        if (Files.exists(indexPath)) {
+                            // Forward request to index file
+                            filePath = indexPath;
+                            filename = indexFile;
+                            break;
+                        }
+                    }
+                }
+            }
 
             if (HTMLParser.isHtmlFile(filename)) {
                 // HTML file - needs to be parsed
-                byte[] data = Files.readAllBytes(Paths.get(filePath)); // TODO: limit file size that can be read into memory
-                HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data);
+                byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory
+                HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data, qdnContext, service, identifier, theme);
                 htmlParser.addAdditionalHeaderTags();
-                response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' blob:; img-src 'self' data: blob:;");
+                response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;");
                 response.setContentType(context.getMimeType(filename));
                 response.setContentLength(htmlParser.getData().length);
                 response.getOutputStream().write(htmlParser.getData());
             }
             else {
                 // Regular file - can be streamed directly
-                File file = new File(filePath);
+                File file = filePath.toFile();
                 FileInputStream inputStream = new FileInputStream(file);
                 response.addHeader("Content-Security-Policy", "default-src 'self'");
                 response.setContentType(context.getMimeType(filename));
@@ -143,14 +182,6 @@ public class ArbitraryDataRenderer {
             return response;
         } catch (FileNotFoundException | NoSuchFileException e) {
             LOGGER.info("Unable to serve file: {}", e.getMessage());
-            if (inPath.equals("/")) {
-                // Delete the unzipped folder if no index file was found
-                try {
-                    FileUtils.deleteDirectory(new File(unzippedPath));
-                } catch (IOException ioException) {
-                    LOGGER.debug("Unable to delete directory: {}", unzippedPath, e);
-                }
-            }
         } catch (IOException e) {
             LOGGER.info("Unable to serve file at path {}: {}", inPath, e.getMessage());
         }
@@ -172,7 +203,7 @@ public class ArbitraryDataRenderer {
         return userPath;
     }
 
-    private HttpServletResponse getLoadingResponse(Service service, String name, String theme) {
+    private HttpServletResponse getLoadingResponse(Service service, String name, String identifier, String theme) {
         String responseString = "";
         URL url = Resources.getResource("loading/index.html");
         try {
@@ -181,6 +212,7 @@ public class ArbitraryDataRenderer {
             // Replace vars
             responseString = responseString.replace("%%SERVICE%%", service.toString());
             responseString = responseString.replace("%%NAME%%", name);
+            responseString = responseString.replace("%%IDENTIFIER%%", identifier);
             responseString = responseString.replace("%%THEME%%", theme);
 
         } catch (IOException e) {

src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java

@@ -43,6 +43,7 @@ public class ArbitraryDataResource {
     private int layerCount;
     private Integer localChunkCount = null;
     private Integer totalChunkCount = null;
+    private boolean exists = false;
 
     public ArbitraryDataResource(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) {
         this.resourceId = resourceId.toLowerCase();
@@ -61,6 +62,10 @@ public class ArbitraryDataResource {
         // Avoid this for "quick" statuses, to speed things up
         if (!quick) {
             this.calculateChunkCounts();
+
+            if (!this.exists) {
+                return new ArbitraryResourceStatus(Status.NOT_PUBLISHED, this.localChunkCount, this.totalChunkCount);
+            }
         }
 
         if (resourceIdType != ResourceIdType.NAME) {
@@ -135,21 +140,20 @@ public class ArbitraryDataResource {
         return null;
     }
 
-    public boolean delete() {
+    public boolean delete(boolean deleteMetadata) {
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                return false;
+            }
 
             List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
 
             for (ArbitraryTransactionData transactionData : transactionDataList) {
-                byte[] hash = transactionData.getData();
-                byte[] metadataHash = transactionData.getMetadataHash();
-                byte[] signature = transactionData.getSignature();
-                ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
-                arbitraryDataFile.setMetadataHash(metadataHash);
+                ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
                 // Delete any chunks or complete files from each transaction
-                arbitraryDataFile.deleteAll();
+                arbitraryDataFile.deleteAll(deleteMetadata);
             }
 
             // Also delete cached data for the entire resource
@@ -193,6 +197,9 @@ public class ArbitraryDataResource {
 
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                return false;
+            }
 
             List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
 
@@ -212,6 +219,14 @@ public class ArbitraryDataResource {
     private void calculateChunkCounts() {
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                this.exists = false;
+                this.localChunkCount = 0;
+                this.totalChunkCount = 0;
+                return;
+            }
+
+            this.exists = true;
 
             List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
             int localChunkCount = 0;
@@ -231,6 +246,9 @@ public class ArbitraryDataResource {
     private boolean isRateLimited() {
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                return true;
+            }
 
             List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
 
@@ -254,6 +272,10 @@ public class ArbitraryDataResource {
     private boolean isDataPotentiallyAvailable() {
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                return false;
+            }
+
             Long now = NTP.getTime();
             if (now == null) {
                 return false;
@@ -285,6 +307,10 @@ public class ArbitraryDataResource {
     private boolean isDownloading() {
         try {
             this.fetchTransactions();
+            if (this.transactions == null) {
+                return false;
+            }
+
             Long now = NTP.getTime();
             if (now == null) {
                 return false;
@@ -337,7 +363,10 @@ public class ArbitraryDataResource {
             this.transactions = transactionDataList;
             this.layerCount = transactionDataList.size();
 
-        } catch (DataException e) {
+        } catch (DataNotPublishedException e) {
+            // Ignore without logging
+        }
+        catch (DataException e) {
             LOGGER.info(String.format("Repository error when fetching transactions for resource %s: %s", this, e.getMessage()));
         }
     }

src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java

@@ -9,6 +9,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataMetadataPatch;
 import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
 import org.qortal.arbitrary.misc.Category;
 import org.qortal.arbitrary.misc.Service;
+import org.qortal.crypto.AES;
 import org.qortal.crypto.Crypto;
 import org.qortal.data.PaymentData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
@@ -46,6 +47,7 @@ public class ArbitraryDataTransactionBuilder {
     private static final double MAX_FILE_DIFF = 0.5f;
 
     private final String publicKey58;
+    private final long fee;
     private final Path path;
     private final String name;
     private Method method;
@@ -64,11 +66,12 @@ public class ArbitraryDataTransactionBuilder {
     private ArbitraryTransactionData arbitraryTransactionData;
     private ArbitraryDataFile arbitraryDataFile;
 
-    public ArbitraryDataTransactionBuilder(Repository repository, String publicKey58, Path path, String name,
+    public ArbitraryDataTransactionBuilder(Repository repository, String publicKey58, long fee, Path path, String name,
                                            Method method, Service service, String identifier,
                                            String title, String description, List<String> tags, Category category) {
         this.repository = repository;
         this.publicKey58 = publicKey58;
+        this.fee = fee;
         this.path = path;
         this.name = name;
         this.method = method;
@@ -179,6 +182,7 @@ public class ArbitraryDataTransactionBuilder {
                 for (ModifiedPath path : metadata.getModifiedPaths()) {
                     if (path.getDiffType() != DiffType.COMPLETE_FILE) {
                         atLeastOnePatch = true;
+                        break;
                     }
                 }
             }
@@ -227,10 +231,12 @@ public class ArbitraryDataTransactionBuilder {
                 random.nextBytes(lastReference);
             }
 
-            Compression compression = Compression.ZIP;
+            // Single file resources are handled differently, especially for very small data payloads, as these go on chain
+            final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false);
+            final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
 
-            // FUTURE? Use zip compression for directories, or no compression for single files
-            // Compression compression = (path.toFile().isDirectory()) ? Compression.ZIP : Compression.NONE;
+            // Use zip compression if data isn't going on chain
+            Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP;
 
             ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(path, name, service, identifier, method,
                     compression, title, description, tags, category);
@@ -248,40 +254,44 @@ public class ArbitraryDataTransactionBuilder {
                 throw new DataException("Arbitrary data file is null");
             }
 
-            // Get chunks metadata file
+            // Get metadata file
             ArbitraryDataFile metadataFile = arbitraryDataFile.getMetadataFile();
             if (metadataFile == null && arbitraryDataFile.chunkCount() > 1) {
                 throw new DataException(String.format("Chunks metadata data file is null but there are %d chunks", arbitraryDataFile.chunkCount()));
             }
 
-            String digest58 = arbitraryDataFile.digest58();
-            if (digest58 == null) {
-                LOGGER.error("Unable to calculate file digest");
-                throw new DataException("Unable to calculate file digest");
+            // Default to using a data hash, with data held off-chain
+            ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH;
+            byte[] data = arbitraryDataFile.digest();
+
+            // For small, single-chunk resources, we can store the data directly on chain
+            if (shouldUseOnChainData && arbitraryDataFile.getBytes().length <= ArbitraryTransaction.MAX_DATA_SIZE && arbitraryDataFile.chunkCount() == 0) {
+                // Within allowed on-chain data size
+                dataType = DataType.RAW_DATA;
+                data = arbitraryDataFile.getBytes();
             }
 
             final BaseTransactionData baseTransactionData = new BaseTransactionData(now, Group.NO_GROUP,
-                    lastReference, creatorPublicKey, 0L, null);
+                    lastReference, creatorPublicKey, fee, null);
             final int size = (int) arbitraryDataFile.size();
             final int version = 5;
             final int nonce = 0;
             byte[] secret = arbitraryDataFile.getSecret();
-            final ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH;
-            final byte[] digest = arbitraryDataFile.digest();
+
             final byte[] metadataHash = (metadataFile != null) ? metadataFile.getHash() : null;
             final List<PaymentData> payments = new ArrayList<>();
 
             ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData,
-                    version, service, nonce, size, name, identifier, method,
-                    secret, compression, digest, dataType, metadataHash, payments);
+                    version, service.value, nonce, size, name, identifier, method,
+                    secret, compression, data, dataType, metadataHash, payments);
 
             this.arbitraryTransactionData = transactionData;
 
-        } catch (DataException e) {
+        } catch (DataException | IOException e) {
             if (arbitraryDataFile != null) {
-                arbitraryDataFile.deleteAll();
+                arbitraryDataFile.deleteAll(true);
             }
-            throw(e);
+            throw new DataException(e);
         }
 
     }
@@ -313,7 +323,7 @@ public class ArbitraryDataTransactionBuilder {
 
         Transaction.ValidationResult result = transaction.isValidUnconfirmed();
         if (result != Transaction.ValidationResult.OK) {
-            arbitraryDataFile.deleteAll();
+            arbitraryDataFile.deleteAll(true);
             throw new DataException(String.format("Arbitrary transaction invalid: %s", result));
         }
         LOGGER.info("Transaction is valid");

src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java

@@ -1,5 +1,7 @@
 package org.qortal.arbitrary;
 
+import com.j256.simplemagic.ContentInfo;
+import com.j256.simplemagic.ContentInfoUtil;
 import org.apache.commons.io.FileUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
@@ -23,6 +25,8 @@ import javax.crypto.NoSuchPaddingException;
 import javax.crypto.SecretKey;
 import java.io.File;
 import java.io.IOException;
+import java.net.FileNameMap;
+import java.net.URLConnection;
 import java.nio.file.*;
 import java.security.InvalidAlgorithmParameterException;
 import java.security.InvalidKeyException;
@@ -48,6 +52,7 @@ public class ArbitraryDataWriter {
     private final List<String> tags;
     private final Category category;
     private List<String> files;
+    private String mimeType;
 
     private int chunkSize = ArbitraryDataFile.CHUNK_SIZE;
 
@@ -79,6 +84,7 @@ public class ArbitraryDataWriter {
         this.tags = ArbitraryDataTransactionMetadata.limitTags(tags);
         this.category = category;
         this.files = new ArrayList<>(); // Populated in buildFileList()
+        this.mimeType = null; // Populated in buildFileList()
     }
 
     public void save() throws IOException, DataException, InterruptedException, MissingDataException {
@@ -101,10 +107,9 @@ public class ArbitraryDataWriter {
     private void preExecute() throws DataException {
         this.checkEnabled();
 
-        // Enforce compression when uploading a directory
-        File file = new File(this.filePath.toString());
-        if (file.isDirectory() && compression == Compression.NONE) {
-            throw new DataException("Unable to upload a directory without compression");
+        // Enforce compression when uploading multiple files
+        if (!FilesystemUtils.isSingleFileResource(this.filePath, false) && compression == Compression.NONE) {
+            throw new DataException("Unable to publish multiple files without compression");
         }
 
         // Create temporary working directory
@@ -144,20 +149,44 @@ public class ArbitraryDataWriter {
     }
 
     private void buildFileList() throws IOException {
-        // Single file resources consist of a single element in the file list
+        // Check if the path already points to a single file
         boolean isSingleFile = this.filePath.toFile().isFile();
+        Path singleFilePath = null;
         if (isSingleFile) {
             this.files.add(this.filePath.getFileName().toString());
-            return;
+            singleFilePath = this.filePath;
+        }
+        else {
+            // Multi file resources (or a single file in a directory) require a walk through the directory tree
+            try (Stream<Path> stream = Files.walk(this.filePath)) {
+                this.files = stream
+                        .filter(Files::isRegularFile)
+                        .map(p -> this.filePath.relativize(p).toString())
+                        .filter(s -> !s.isEmpty())
+                        .collect(Collectors.toList());
+
+                if (this.files.size() == 1) {
+                    singleFilePath = Paths.get(this.filePath.toString(), this.files.get(0));
+
+                    // Update filePath to point to the single file (instead of the directory containing the file)
+                    this.filePath = singleFilePath;
+                }
+            }
         }
 
-        // Multi file resources require a walk through the directory tree
-        try (Stream<Path> stream = Files.walk(this.filePath)) {
-            this.files = stream
-                    .filter(Files::isRegularFile)
-                    .map(p -> this.filePath.relativize(p).toString())
-                    .filter(s -> !s.isEmpty())
-                    .collect(Collectors.toList());
+        if (singleFilePath != null) {
+            // Single file resource, so try and determine the MIME type
+            ContentInfoUtil util = new ContentInfoUtil();
+            ContentInfo info = util.findMatch(singleFilePath.toFile());
+            if (info != null) {
+                // Attempt to extract MIME type from file contents
+                this.mimeType = info.getMimeType();
+            }
+            else {
+                // Fall back to using the filename
+                FileNameMap fileNameMap = URLConnection.getFileNameMap();
+                this.mimeType = fileNameMap.getContentTypeFor(singleFilePath.toFile().getName());
+            }
         }
     }
 
@@ -287,9 +316,6 @@ public class ArbitraryDataWriter {
         if (chunkCount > 0) {
             LOGGER.info(String.format("Successfully split into %d chunk%s", chunkCount, (chunkCount == 1 ? "" : "s")));
         }
-        else {
-            throw new DataException("Unable to split file into chunks");
-        }
     }
 
     private void createMetadataFile() throws IOException, DataException {
@@ -304,6 +330,7 @@ public class ArbitraryDataWriter {
             metadata.setCategory(this.category);
             metadata.setChunks(this.arbitraryDataFile.chunkHashList());
             metadata.setFiles(this.files);
+            metadata.setMimeType(this.mimeType);
             metadata.write();
 
             // Create an ArbitraryDataFile from the JSON file (we don't have a signature yet)

src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java

@@ -20,9 +20,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
     private List<String> tags;
     private Category category;
     private List<String> files;
+    private String mimeType;
 
     private static int MAX_TITLE_LENGTH = 80;
-    private static int MAX_DESCRIPTION_LENGTH = 500;
+    private static int MAX_DESCRIPTION_LENGTH = 240;
     private static int MAX_TAG_LENGTH = 20;
     private static int MAX_TAGS_COUNT = 5;
 
@@ -92,6 +93,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
             }
             this.files = filesList;
         }
+
+        if (metadata.has("mimeType")) {
+            this.mimeType = metadata.getString("mimeType");
+        }
     }
 
     @Override
@@ -134,6 +139,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
         }
         outer.put("files", files);
 
+        if (this.mimeType != null && !this.mimeType.isEmpty()) {
+            outer.put("mimeType", this.mimeType);
+        }
+
         this.jsonString = outer.toString(2);
         LOGGER.trace("Transaction metadata: {}", this.jsonString);
     }
@@ -187,6 +196,14 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
         return this.files;
     }
 
+    public void setMimeType(String mimeType) {
+        this.mimeType = mimeType;
+    }
+
+    public String getMimeType() {
+        return this.mimeType;
+    }
+
     public boolean containsChunk(byte[] chunk) {
         for (byte[] c : this.chunks) {
             if (Arrays.equals(c, chunk)) {

src/main/java/org/qortal/arbitrary/misc/Service.java

@@ -8,17 +8,21 @@ import org.qortal.utils.FilesystemUtils;
 
 import java.io.File;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.*;
 
+import com.fasterxml.jackson.databind.ObjectMapper;
+
 import static java.util.Arrays.stream;
 import static java.util.stream.Collectors.toMap;
 
 public enum Service {
-    AUTO_UPDATE(1, false, null, null),
-    ARBITRARY_DATA(100, false, null, null),
-    QCHAT_ATTACHMENT(120, true, 1024*1024L, null) {
+    AUTO_UPDATE(1, false, null, false, null),
+    ARBITRARY_DATA(100, false, null, false, null),
+    QCHAT_ATTACHMENT(120, true, 1024*1024L, true, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -26,37 +30,24 @@ public enum Service {
                 return superclassResult;
             }
 
-            // Custom validation function to require a single file, with a whitelisted extension
-            int fileCount = 0;
             File[] files = path.toFile().listFiles();
             // If already a single file, replace the list with one that contains that file only
             if (files == null && path.toFile().isFile()) {
                 files = new File[] { path.toFile() };
             }
-            if (files != null) {
-                for (File file : files) {
-                    if (file.getName().equals(".qortal")) {
-                        continue;
-                    }
-                    if (file.isDirectory()) {
-                        return ValidationResult.DIRECTORIES_NOT_ALLOWED;
-                    }
-                    final String extension = FilenameUtils.getExtension(file.getName()).toLowerCase();
-                    // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string
-                    final List<String> allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
-                    if (extension == null || !allowedExtensions.contains(extension)) {
-                        return ValidationResult.INVALID_FILE_EXTENSION;
-                    }
-                    fileCount++;
+            // Now validate the file's extension
+            if (files != null && files[0] != null) {
+                final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase();
+                // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string
+                final List<String> allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
+                if (extension == null || !allowedExtensions.contains(extension)) {
+                    return ValidationResult.INVALID_FILE_EXTENSION;
                 }
             }
-            if (fileCount != 1) {
-                return ValidationResult.INVALID_FILE_COUNT;
-            }
             return ValidationResult.OK;
         }
     },
-    WEBSITE(200, true, null, null) {
+    WEBSITE(200, true, null, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -78,23 +69,42 @@ public enum Service {
             return ValidationResult.MISSING_INDEX_FILE;
         }
     },
-    GIT_REPOSITORY(300, false, null, null),
-    IMAGE(400, true, 10*1024*1024L, null),
-    THUMBNAIL(410, true, 500*1024L, null),
-    QCHAT_IMAGE(420, true, 500*1024L, null),
-    VIDEO(500, false, null, null),
-    AUDIO(600, false, null, null),
-    QCHAT_AUDIO(610, true, 10*1024*1024L, null),
-    QCHAT_VOICE(620, true, 10*1024*1024L, null),
-    BLOG(700, false, null, null),
-    BLOG_POST(777, false, null, null),
-    BLOG_COMMENT(778, false, null, null),
-    DOCUMENT(800, false, null, null),
-    LIST(900, true, null, null),
-    PLAYLIST(910, true, null, null),
-    APP(1000, false, null, null),
-    METADATA(1100, false, null, null),
-    GIF_REPOSITORY(1200, true, 25*1024*1024L, null) {
+    GIT_REPOSITORY(300, false, null, false, null),
+    IMAGE(400, true, 10*1024*1024L, true, null),
+    THUMBNAIL(410, true, 500*1024L, true, null),
+    QCHAT_IMAGE(420, true, 500*1024L, true, null),
+    VIDEO(500, false, null, true, null),
+    AUDIO(600, false, null, true, null),
+    QCHAT_AUDIO(610, true, 10*1024*1024L, true, null),
+    QCHAT_VOICE(620, true, 10*1024*1024L, true, null),
+    BLOG(700, false, null, false, null),
+    BLOG_POST(777, false, null, true, null),
+    BLOG_COMMENT(778, false, null, true, null),
+    DOCUMENT(800, false, null, true, null),
+    LIST(900, true, null, true, null),
+    PLAYLIST(910, true, null, true, null),
+    APP(1000, true, 50*1024*1024L, false, null),
+    METADATA(1100, false, null, true, null),
+    JSON(1110, true, 25*1024L, true, null) {
+        @Override
+        public ValidationResult validate(Path path) throws IOException {
+            ValidationResult superclassResult = super.validate(path);
+            if (superclassResult != ValidationResult.OK) {
+                return superclassResult;
+            }
+
+            // Require valid JSON
+            byte[] data = FilesystemUtils.getSingleFileContents(path);
+            String json = new String(data, StandardCharsets.UTF_8);
+            try {
+                objectMapper.readTree(json);
+                return ValidationResult.OK;
+            } catch (IOException e) {
+                return ValidationResult.INVALID_CONTENT;
+            }
+        }
+    },
+    GIF_REPOSITORY(1200, true, 25*1024*1024L, false, null) {
         @Override
         public ValidationResult validate(Path path) throws IOException {
             ValidationResult superclassResult = super.validate(path);
@@ -134,15 +144,20 @@ public enum Service {
     public final int value;
     private final boolean requiresValidation;
     private final Long maxSize;
+    private final boolean single;
     private final List<String> requiredKeys;
 
     private static final Map<Integer, Service> map = stream(Service.values())
             .collect(toMap(service -> service.value, service -> service));
 
-    Service(int value, boolean requiresValidation, Long maxSize, List<String> requiredKeys) {
+    // For JSON validation
+    private static final ObjectMapper objectMapper = new ObjectMapper();
+
+    Service(int value, boolean requiresValidation, Long maxSize, boolean single, List<String> requiredKeys) {
         this.value = value;
         this.requiresValidation = requiresValidation;
         this.maxSize = maxSize;
+        this.single = single;
         this.requiredKeys = requiredKeys;
     }
 
@@ -161,6 +176,11 @@ public enum Service {
             }
         }
 
+        // Validate file count if needed
+        if (this.single && data == null) {
+            return ValidationResult.INVALID_FILE_COUNT;
+        }
+
         // Validate required keys if needed
         if (this.requiredKeys != null) {
             if (data == null) {
@@ -199,7 +219,8 @@ public enum Service {
         DIRECTORIES_NOT_ALLOWED(5),
         INVALID_FILE_EXTENSION(6),
         MISSING_DATA(7),
-        INVALID_FILE_COUNT(8);
+        INVALID_FILE_COUNT(8),
+        INVALID_CONTENT(9);
 
         public final int value;
 

src/main/java/org/qortal/block/Block.java

@@ -657,6 +657,10 @@ public class Block {
 		return this.atStates;
 	}
 
+	public byte[] getAtStatesHash() {
+		return this.atStatesHash;
+	}
+
 	/**
 	 * Return expanded info on block's online accounts.
 	 * <p>

src/main/java/org/qortal/block/BlockChain.java

@@ -78,7 +78,8 @@ public class BlockChain {
 		onlineAccountMinterLevelValidationHeight,
 		selfSponsorshipAlgoV1Height,
 		feeValidationFixTimestamp,
-		chatReferenceTimestamp;
+		chatReferenceTimestamp,
+		arbitraryOptionalFeeTimestamp;
 	}
 
 	// Custom transaction fees
@@ -522,6 +523,10 @@ public class BlockChain {
 		return this.featureTriggers.get(FeatureTrigger.chatReferenceTimestamp.name()).longValue();
 	}
 
+	public long getArbitraryOptionalFeeTimestamp() {
+		return this.featureTriggers.get(FeatureTrigger.arbitraryOptionalFeeTimestamp.name()).longValue();
+	}
+
 
 	// More complex getters for aspects that change by height or timestamp
 

src/main/java/org/qortal/controller/BlockMinter.java

@@ -432,6 +432,10 @@ public class BlockMinter extends Thread {
 							// Unable to process block - report and discard
 							LOGGER.error("Unable to process newly minted block?", e);
 							newBlocks.clear();
+						} catch (ArithmeticException e) {
+							// Unable to process block - report and discard
+							LOGGER.error("Unable to process newly minted block?", e);
+							newBlocks.clear();
 						}
 					} finally {
 						blockchainLock.unlock();

src/main/java/org/qortal/controller/Controller.java

@@ -400,12 +400,8 @@ public class Controller extends Thread {
 			RepositoryFactory repositoryFactory = new HSQLDBRepositoryFactory(getRepositoryUrl());
 			RepositoryManager.setRepositoryFactory(repositoryFactory);
 			RepositoryManager.setRequestedCheckpoint(Boolean.TRUE);
-
-			try (final Repository repository = RepositoryManager.getRepository()) {
-				RepositoryManager.archive(repository);
-				RepositoryManager.prune(repository);
-			}
-		} catch (DataException e) {
+		}
+		catch (DataException e) {
 			// If exception has no cause then repository is in use by some other process.
 			if (e.getCause() == null) {
 				LOGGER.info("Repository in use by another process?");
@@ -1379,9 +1375,24 @@ public class Controller extends Thread {
 			// If we have no block data, we should check the archive in case it's there
 			if (blockData == null) {
 				if (Settings.getInstance().isArchiveEnabled()) {
-					byte[] bytes = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, true, repository);
-					if (bytes != null) {
-						CachedBlockMessage blockMessage = new CachedBlockMessage(bytes);
+					Triple<byte[], Integer, Integer> serializedBlock = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, true, repository);
+					if (serializedBlock != null) {
+						byte[] bytes = serializedBlock.getA();
+						Integer serializationVersion = serializedBlock.getB();
+
+						Message blockMessage;
+						switch (serializationVersion) {
+							case 1:
+								blockMessage = new CachedBlockMessage(bytes);
+								break;
+
+							case 2:
+								blockMessage = new CachedBlockV2Message(bytes);
+								break;
+
+							default:
+								return;
+						}
 						blockMessage.setId(message.getId());
 
 						// This call also causes the other needed data to be pulled in from repository

src/main/java/org/qortal/controller/PirateChainWalletController.java

@@ -163,7 +163,7 @@ public class PirateChainWalletController extends Thread {
 
             // Library not found, so check if we've fetched the resource from QDN
             ArbitraryTransactionData t = this.getTransactionData(repository);
-            if (t == null) {
+            if (t == null || t.getService() == null) {
                 // Can't find the transaction - maybe on a different chain?
                 return;
             }

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java

@@ -137,7 +137,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 
 						// Fetch the transaction data
 						ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
-						if (arbitraryTransactionData == null) {
+						if (arbitraryTransactionData == null || arbitraryTransactionData.getService() == null) {
 							continue;
 						}
 
@@ -204,7 +204,7 @@ public class ArbitraryDataCleanupManager extends Thread {
 
 						if (completeFileExists && !allChunksExist) {
 							// We have the complete file but not the chunks, so let's convert it
-							LOGGER.info(String.format("Transaction %s has complete file but no chunks",
+							LOGGER.debug(String.format("Transaction %s has complete file but no chunks",
 									Base58.encode(arbitraryTransactionData.getSignature())));
 
 							ArbitraryTransactionUtils.convertFileToChunks(arbitraryTransactionData, now, STALE_FILE_TIMEOUT);

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java

@@ -258,8 +258,6 @@ public class ArbitraryDataFileListManager {
     // Lookup file lists by signature (and optionally hashes)
 
     public boolean fetchArbitraryDataFileList(ArbitraryTransactionData arbitraryTransactionData) {
-        byte[] digest = arbitraryTransactionData.getData();
-        byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
         byte[] signature = arbitraryTransactionData.getSignature();
         String signature58 = Base58.encode(signature);
 
@@ -286,8 +284,7 @@ public class ArbitraryDataFileListManager {
 
         // Find hashes that we are missing
         try {
-            ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-            arbitraryDataFile.setMetadataHash(metadataHash);
+            ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
             missingHashes = arbitraryDataFile.missingHashes();
         } catch (DataException e) {
             // Leave missingHashes as null, so that all hashes are requested
@@ -460,10 +457,9 @@ public class ArbitraryDataFileListManager {
 
             arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
 
-            // Load data file(s)
-            ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature);
-            arbitraryDataFile.setMetadataHash(arbitraryTransactionData.getMetadataHash());
-
+//          // Load data file(s)
+//          ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
+//
 //			// Check all hashes exist
 //			for (byte[] hash : hashes) {
 //				//LOGGER.debug("Received hash {}", Base58.encode(hash));
@@ -594,12 +590,8 @@ public class ArbitraryDataFileListManager {
                 // Check if we're even allowed to serve data for this transaction
                 if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
 
-                    byte[] hash = transactionData.getData();
-                    byte[] metadataHash = transactionData.getMetadataHash();
-
                     // Load file(s) and add any that exist to the list of hashes
-                    ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
-                    arbitraryDataFile.setMetadataHash(metadataHash);
+                    ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
                     // If the peer didn't supply a hash list, we need to return all hashes for this transaction
                     if (requestedHashes == null || requestedHashes.isEmpty()) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java

@@ -132,9 +132,7 @@ public class ArbitraryDataFileManager extends Thread {
                                            List<byte[]> hashes) throws DataException {
 
         // Load data file(s)
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature);
-        byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
-        arbitraryDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
         boolean receivedAtLeastOneFile = false;
 
         // Now fetch actual data from this peer
@@ -148,10 +146,10 @@ public class ArbitraryDataFileManager extends Thread {
                 if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) {
                     LOGGER.debug("Requesting data file {} from peer {}", hash58, peer);
                     Long startTime = NTP.getTime();
-                    ArbitraryDataFileMessage receivedArbitraryDataFileMessage = fetchArbitraryDataFile(peer, null, signature, hash, null);
+                    ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, null, signature, hash, null);
                     Long endTime = NTP.getTime();
-                    if (receivedArbitraryDataFileMessage != null && receivedArbitraryDataFileMessage.getArbitraryDataFile() != null) {
-                        LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFileMessage.getArbitraryDataFile().getHash58(), peer, (endTime-startTime));
+                    if (receivedArbitraryDataFile != null) {
+                        LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime));
                         receivedAtLeastOneFile = true;
 
                         // Remove this hash from arbitraryDataFileHashResponses now that we have received it
@@ -193,11 +191,11 @@ public class ArbitraryDataFileManager extends Thread {
         return receivedAtLeastOneFile;
     }
 
-    private ArbitraryDataFileMessage fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
+    private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
         ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
         boolean fileAlreadyExists = existingFile.exists();
         String hash58 = Base58.encode(hash);
-        ArbitraryDataFileMessage arbitraryDataFileMessage;
+        ArbitraryDataFile arbitraryDataFile;
 
         // Fetch the file if it doesn't exist locally
         if (!fileAlreadyExists) {
@@ -227,28 +225,27 @@ public class ArbitraryDataFileManager extends Thread {
             }
 
             ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
-            arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, peersArbitraryDataFileMessage.getArbitraryDataFile());
+            arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
         } else {
             LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
-            arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, existingFile);
+            arbitraryDataFile = existingFile;
         }
 
         // We might want to forward the request to the peer that originally requested it
-        this.handleArbitraryDataFileForwarding(requestingPeer, arbitraryDataFileMessage, originalMessage);
+        this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
 
         boolean isRelayRequest = (requestingPeer != null);
         if (isRelayRequest) {
             if (!fileAlreadyExists) {
                 // File didn't exist locally before the request, and it's a forwarding request, so delete it
                 LOGGER.debug("Deleting file {} because it was needed for forwarding only", Base58.encode(hash));
-                ArbitraryDataFile dataFile = arbitraryDataFileMessage.getArbitraryDataFile();
 
                 // Keep trying to delete the data until it is deleted, or we reach 10 attempts
-                dataFile.delete(10);
+                arbitraryDataFile.delete(10);
             }
         }
 
-        return arbitraryDataFileMessage;
+        return arbitraryDataFile;
     }
 
     private void handleFileListRequests(byte[] signature) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java

@@ -114,7 +114,7 @@ public class ArbitraryDataFileRequestThread implements Runnable {
                 return;
             }
 
-            LOGGER.debug("Fetching file {} from peer {} via request thread...", hash58, peer);
+            LOGGER.trace("Fetching file {} from peer {} via request thread...", hash58, peer);
             arbitraryDataFileManager.fetchArbitraryDataFiles(repository, peer, signature, arbitraryTransactionData, Arrays.asList(hash));
 
         } catch (DataException e) {

src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java

@@ -398,6 +398,11 @@ public class ArbitraryDataManager extends Thread {
 	// Entrypoint to request new metadata from peers
 	public ArbitraryDataTransactionMetadata fetchMetadata(ArbitraryTransactionData arbitraryTransactionData) {
 
+		if (arbitraryTransactionData.getService() == null) {
+			// Can't fetch metadata without a valid service
+			return null;
+		}
+
 		ArbitraryDataResource resource = new ArbitraryDataResource(
 				arbitraryTransactionData.getName(),
 				ArbitraryDataFile.ResourceIdType.NAME,
@@ -489,7 +494,7 @@ public class ArbitraryDataManager extends Thread {
 	public void invalidateCache(ArbitraryTransactionData arbitraryTransactionData) {
 		String signature58 = Base58.encode(arbitraryTransactionData.getSignature());
 
-		if (arbitraryTransactionData.getName() != null) {
+		if (arbitraryTransactionData.getName() != null && arbitraryTransactionData.getService() != null) {
 			String resourceId = arbitraryTransactionData.getName().toLowerCase();
 			Service service = arbitraryTransactionData.getService();
 			String identifier = arbitraryTransactionData.getIdentifier();

src/main/java/org/qortal/controller/repository/BlockArchiveRebuilder.java

@@ -0,0 +1,121 @@
+package org.qortal.controller.repository;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.qortal.controller.Controller;
+import org.qortal.controller.Synchronizer;
+import org.qortal.repository.*;
+import org.qortal.settings.Settings;
+import org.qortal.transform.TransformationException;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+
+public class BlockArchiveRebuilder {
+
+    private static final Logger LOGGER = LogManager.getLogger(BlockArchiveRebuilder.class);
+
+    private final int serializationVersion;
+
+    public BlockArchiveRebuilder(int serializationVersion) {
+        this.serializationVersion = serializationVersion;
+    }
+
+    public void start() throws DataException, IOException {
+        if (!Settings.getInstance().isArchiveEnabled() || Settings.getInstance().isLite()) {
+            return;
+        }
+
+        // New archive path is in a different location from original archive path, to avoid conflicts.
+        // It will be moved later, once the process is complete.
+        final Path newArchivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive-rebuild");
+        final Path originalArchivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive");
+
+        // Delete archive-rebuild if it exists from a previous attempt
+        FileUtils.deleteDirectory(newArchivePath.toFile());
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            int startHeight = 1; // We need to rebuild the entire archive
+
+            LOGGER.info("Rebuilding block archive from height {}...", startHeight);
+
+            while (!Controller.isStopping()) {
+                repository.discardChanges();
+
+                Thread.sleep(1000L);
+
+                // Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
+                if (Synchronizer.getInstance().isSynchronizing()) {
+                    continue;
+                }
+
+                // Rebuild archive
+                try {
+                    final int maximumArchiveHeight = BlockArchiveReader.getInstance().getHeightOfLastArchivedBlock();
+                    if (startHeight >= maximumArchiveHeight) {
+                        // We've finished.
+                        // Delete existing archive and move the newly built one into its place
+                        FileUtils.deleteDirectory(originalArchivePath.toFile());
+                        FileUtils.moveDirectory(newArchivePath.toFile(), originalArchivePath.toFile());
+                        BlockArchiveReader.getInstance().invalidateFileListCache();
+                        LOGGER.info("Block archive successfully rebuilt");
+                        return;
+                    }
+
+                    BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, serializationVersion, newArchivePath, repository);
+
+                    // Set data source to BLOCK_ARCHIVE as we are rebuilding
+                    writer.setDataSource(BlockArchiveWriter.BlockArchiveDataSource.BLOCK_ARCHIVE);
+
+                    // We can't enforce the 100MB file size target, as the final file needs to contain all blocks
+                    // that exist in the current archive. Otherwise, the final blocks in the archive will be lost.
+                    writer.setShouldEnforceFileSizeTarget(false);
+
+                    // We want to log the rebuild progress
+                    writer.setShouldLogProgress(true);
+
+                    BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+                    switch (result) {
+                        case OK:
+                            // Increment block archive height
+                            startHeight += writer.getWrittenCount();
+                            repository.saveChanges();
+                            break;
+
+                        case STOPPING:
+                            return;
+
+                        // We've reached the limit of the blocks we can archive
+                        // Sleep for a while to allow more to become available
+                        case NOT_ENOUGH_BLOCKS:
+                            // This shouldn't happen, as we're not enforcing minimum file sizes
+                            repository.discardChanges();
+                            throw new DataException("Unable to rebuild archive due to unexpected NOT_ENOUGH_BLOCKS response.");
+
+                        case BLOCK_NOT_FOUND:
+                            // We tried to archive a block that didn't exist. This is a major failure and likely means
+                            // that a bootstrap or re-sync is needed. Try again every minute until then.
+                            LOGGER.info("Error: block not found when rebuilding archive. If this error persists, " +
+                                    "a bootstrap or re-sync may be needed.");
+                            repository.discardChanges();
+                            throw new DataException("Unable to rebuild archive because a block is missing.");
+                    }
+
+                } catch (IOException | TransformationException e) {
+                    LOGGER.info("Caught exception when rebuilding block archive", e);
+                    throw new DataException("Unable to rebuild block archive");
+                }
+
+            }
+        } catch (InterruptedException e) {
+            // Do nothing
+        } finally {
+            // Delete archive-rebuild if it still exists, as that means something went wrong
+            FileUtils.deleteDirectory(newArchivePath.toFile());
+        }
+    }
+
+}

src/main/java/org/qortal/crypto/AES.java

@@ -202,4 +202,12 @@ public class AES {
                 .decode(cipherText)));
     }
 
+    public static long getEncryptedFileSize(long inFileSize) {
+        // To calculate the resulting file size, add 16 (for the IV), then round up to the nearest multiple of 16
+        final int ivSize = 16;
+        final int chunkSize = 16;
+        final int expectedSize = Math.round((inFileSize + ivSize) / chunkSize) * chunkSize + chunkSize;
+        return expectedSize;
+    }
+
 }

src/main/java/org/qortal/data/arbitrary/ArbitraryResourceInfo.java

@@ -16,10 +16,17 @@ public class ArbitraryResourceInfo {
 	public ArbitraryResourceMetadata metadata;
 
 	public Long size;
+	public Long created;
+	public Long updated;
 
 	public ArbitraryResourceInfo() {
 	}
 
+	@Override
+	public String toString() {
+		return String.format("%s %s %s", name, service, identifier);
+	}
+
 	@Override
 	public boolean equals(Object o) {
 		if (o == this)

src/main/java/org/qortal/data/arbitrary/ArbitraryResourceMetadata.java

@@ -16,16 +16,18 @@ public class ArbitraryResourceMetadata {
     private Category category;
     private String categoryName;
     private List<String> files;
+    private String mimeType;
 
     public ArbitraryResourceMetadata() {
     }
 
-    public ArbitraryResourceMetadata(String title, String description, List<String> tags, Category category, List<String> files) {
+    public ArbitraryResourceMetadata(String title, String description, List<String> tags, Category category, List<String> files, String mimeType) {
         this.title = title;
         this.description = description;
         this.tags = tags;
         this.category = category;
         this.files = files;
+        this.mimeType = mimeType;
 
         if (category != null) {
             this.categoryName = category.getName();
@@ -40,6 +42,7 @@ public class ArbitraryResourceMetadata {
         String description = transactionMetadata.getDescription();
         List<String> tags = transactionMetadata.getTags();
         Category category = transactionMetadata.getCategory();
+        String mimeType = transactionMetadata.getMimeType();
 
         // We don't always want to include the file list as it can be too verbose
         List<String> files = null;
@@ -47,11 +50,11 @@ public class ArbitraryResourceMetadata {
             files = transactionMetadata.getFiles();
         }
 
-        if (title == null && description == null && tags == null && category == null && files == null) {
+        if (title == null && description == null && tags == null && category == null && files == null && mimeType == null) {
             return null;
         }
 
-        return new ArbitraryResourceMetadata(title, description, tags, category, files);
+        return new ArbitraryResourceMetadata(title, description, tags, category, files, mimeType);
     }
 
     public List<String> getFiles() {

src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java

@@ -8,6 +8,7 @@ public class ArbitraryResourceStatus {
 
     public enum Status {
         PUBLISHED("Published", "Published but not yet downloaded"),
+        NOT_PUBLISHED("Not published", "Resource does not exist"),
         DOWNLOADING("Downloading", "Locating and downloading files..."),
         DOWNLOADED("Downloaded", "Files downloaded"),
         BUILDING("Building", "Building..."),
@@ -33,6 +34,7 @@ public class ArbitraryResourceStatus {
 
     private Integer localChunkCount;
     private Integer totalChunkCount;
+    private Float percentLoaded;
 
     public ArbitraryResourceStatus() {
     }
@@ -44,6 +46,7 @@ public class ArbitraryResourceStatus {
         this.description = status.description;
         this.localChunkCount = localChunkCount;
         this.totalChunkCount = totalChunkCount;
+        this.percentLoaded = (this.localChunkCount != null && this.totalChunkCount != null) ? this.localChunkCount / (float)this.totalChunkCount * 100.0f : null;
     }
 
     public ArbitraryResourceStatus(Status status) {

src/main/java/org/qortal/data/chat/ActiveChats.java

@@ -1,10 +1,15 @@
 package org.qortal.data.chat;
 
+import org.bouncycastle.util.encoders.Base64;
+import org.qortal.utils.Base58;
+
 import java.util.List;
 
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 @XmlAccessorType(XmlAccessType.FIELD)
 public class ActiveChats {
 
@@ -18,20 +23,38 @@ public class ActiveChats {
 		private String sender;
 		private String senderName;
 		private byte[] signature;
-		private byte[] data;
+		private Encoding encoding;
+		private String data;
 
 		protected GroupChat() {
 			/* JAXB */
 		}
 
-		public GroupChat(int groupId, String groupName, Long timestamp, String sender, String senderName, byte[] signature, byte[] data) {
+		public GroupChat(int groupId, String groupName, Long timestamp, String sender, String senderName,
+						 byte[] signature, Encoding encoding, byte[] data) {
 			this.groupId = groupId;
 			this.groupName = groupName;
 			this.timestamp = timestamp;
 			this.sender = sender;
 			this.senderName = senderName;
 			this.signature = signature;
-			this.data = data;
+			this.encoding = encoding != null ? encoding : Encoding.BASE58;
+
+			if (data != null) {
+				switch (this.encoding) {
+					case BASE64:
+						this.data = Base64.toBase64String(data);
+						break;
+
+					case BASE58:
+					default:
+						this.data = Base58.encode(data);
+						break;
+				}
+			}
+			else {
+				this.data = null;
+			}
 		}
 
 		public int getGroupId() {
@@ -58,7 +81,7 @@ public class ActiveChats {
 			return this.signature;
 		}
 
-		public byte[] getData() {
+		public String getData() {
 			return this.data;
 		}
 	}

src/main/java/org/qortal/data/chat/ChatMessage.java

@@ -1,11 +1,19 @@
 package org.qortal.data.chat;
 
+import org.bouncycastle.util.encoders.Base64;
+import org.qortal.utils.Base58;
+
 import javax.xml.bind.annotation.XmlAccessType;
 import javax.xml.bind.annotation.XmlAccessorType;
 
 @XmlAccessorType(XmlAccessType.FIELD)
 public class ChatMessage {
 
+	public enum Encoding {
+		BASE58,
+		BASE64
+	}
+
 	// Properties
 
 	private long timestamp;
@@ -29,7 +37,9 @@ public class ChatMessage {
 
 	private byte[] chatReference;
 
-	private byte[] data;
+	private Encoding encoding;
+
+	private String data;
 
 	private boolean isText;
 	private boolean isEncrypted;
@@ -44,8 +54,8 @@ public class ChatMessage {
 
 	// For repository use
 	public ChatMessage(long timestamp, int txGroupId, byte[] reference, byte[] senderPublicKey, String sender,
-			String senderName, String recipient, String recipientName, byte[] chatReference, byte[] data,
-			boolean isText, boolean isEncrypted, byte[] signature) {
+					   String senderName, String recipient, String recipientName, byte[] chatReference,
+					   Encoding encoding, byte[] data, boolean isText, boolean isEncrypted, byte[] signature) {
 		this.timestamp = timestamp;
 		this.txGroupId = txGroupId;
 		this.reference = reference;
@@ -55,7 +65,24 @@ public class ChatMessage {
 		this.recipient = recipient;
 		this.recipientName = recipientName;
 		this.chatReference = chatReference;
-		this.data = data;
+		this.encoding = encoding != null ? encoding : Encoding.BASE58;
+
+		if (data != null) {
+			switch (this.encoding) {
+				case BASE64:
+					this.data = Base64.toBase64String(data);
+					break;
+
+				case BASE58:
+				default:
+					this.data = Base58.encode(data);
+					break;
+			}
+		}
+		else {
+			this.data = null;
+		}
+
 		this.isText = isText;
 		this.isEncrypted = isEncrypted;
 		this.signature = signature;
@@ -97,7 +124,7 @@ public class ChatMessage {
 		return this.chatReference;
 	}
 
-	public byte[] getData() {
+	public String getData() {
 		return this.data;
 	}
 

src/main/java/org/qortal/data/transaction/ArbitraryTransactionData.java

@@ -73,7 +73,7 @@ public class ArbitraryTransactionData extends TransactionData {
 	@Schema(example = "sender_public_key")
 	private byte[] senderPublicKey;
 
-	private Service service;
+	private int service;
 	private int nonce;
 	private int size;
 
@@ -103,7 +103,7 @@ public class ArbitraryTransactionData extends TransactionData {
 	}
 
 	public ArbitraryTransactionData(BaseTransactionData baseTransactionData,
-									int version, Service service, int nonce, int size,
+									int version, int service, int nonce, int size,
 									String name, String identifier, Method method, byte[] secret, Compression compression,
 									byte[] data, DataType dataType, byte[] metadataHash, List<PaymentData> payments) {
 		super(TransactionType.ARBITRARY, baseTransactionData);
@@ -135,6 +135,10 @@ public class ArbitraryTransactionData extends TransactionData {
 	}
 
 	public Service getService() {
+		return Service.valueOf(this.service);
+	}
+
+	public int getServiceInt() {
 		return this.service;
 	}
 

src/main/java/org/qortal/naming/Name.java

@@ -16,6 +16,8 @@ import org.qortal.repository.Repository;
 import org.qortal.transaction.Transaction.TransactionType;
 import org.qortal.utils.Unicode;
 
+import java.util.Objects;
+
 public class Name {
 
 	// Properties
@@ -116,7 +118,7 @@ public class Name {
 
 		this.repository.getNameRepository().save(this.nameData);
 
-		if (!updateNameTransactionData.getNewName().isEmpty())
+		if (!updateNameTransactionData.getNewName().isEmpty() && !Objects.equals(updateNameTransactionData.getName(), updateNameTransactionData.getNewName()))
 			// Name has changed, delete old entry
 			this.repository.getNameRepository().delete(updateNameTransactionData.getNewName());
 

src/main/java/org/qortal/network/message/CachedBlockV2Message.java

@@ -0,0 +1,43 @@
+package org.qortal.network.message;
+
+import com.google.common.primitives.Ints;
+import org.qortal.block.Block;
+import org.qortal.transform.TransformationException;
+import org.qortal.transform.block.BlockTransformer;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+// This is an OUTGOING-only Message which more readily lends itself to being cached
+public class CachedBlockV2Message extends Message implements Cloneable {
+
+	public CachedBlockV2Message(Block block) throws TransformationException {
+		super(MessageType.BLOCK_V2);
+
+		ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+
+		try {
+			bytes.write(Ints.toByteArray(block.getBlockData().getHeight()));
+
+			bytes.write(BlockTransformer.toBytes(block));
+		} catch (IOException e) {
+			throw new AssertionError("IOException shouldn't occur with ByteArrayOutputStream");
+		}
+
+		this.dataBytes = bytes.toByteArray();
+		this.checksumBytes = Message.generateChecksum(this.dataBytes);
+	}
+
+	public CachedBlockV2Message(byte[] cachedBytes) {
+		super(MessageType.BLOCK_V2);
+
+		this.dataBytes = cachedBytes;
+		this.checksumBytes = Message.generateChecksum(this.dataBytes);
+	}
+
+	public static Message fromByteBuffer(int id, ByteBuffer byteBuffer) {
+		throw new UnsupportedOperationException("CachedBlockMessageV2 is for outgoing messages only");
+	}
+
+}

src/main/java/org/qortal/repository/ArbitraryRepository.java

@@ -26,7 +26,7 @@ public interface ArbitraryRepository {
 
 	public List<ArbitraryResourceInfo> getArbitraryResources(Service service, String identifier, List<String> names, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
 
-	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
+	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, String name, boolean prefixOnly, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
 
 	public List<ArbitraryResourceNameInfo> getArbitraryResourceCreatorNames(Service service, String identifier, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException;
 

src/main/java/org/qortal/repository/BlockArchiveReader.java

@@ -3,10 +3,7 @@ package org.qortal.repository;
 import com.google.common.primitives.Ints;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.qortal.data.at.ATStateData;
 import org.qortal.data.block.BlockArchiveData;
-import org.qortal.data.block.BlockData;
-import org.qortal.data.transaction.TransactionData;
 import org.qortal.settings.Settings;
 import org.qortal.transform.TransformationException;
 import org.qortal.transform.block.BlockTransformation;
@@ -67,20 +64,51 @@ public class BlockArchiveReader {
         this.fileListCache = Map.copyOf(map);
     }
 
+    public Integer fetchSerializationVersionForHeight(int height) {
+        if (this.fileListCache == null) {
+            this.fetchFileList();
+        }
+
+        Triple<byte[], Integer, Integer> serializedBlock = this.fetchSerializedBlockBytesForHeight(height);
+        if (serializedBlock == null) {
+            return null;
+        }
+        Integer serializationVersion = serializedBlock.getB();
+        return serializationVersion;
+    }
+
     public BlockTransformation fetchBlockAtHeight(int height) {
         if (this.fileListCache == null) {
             this.fetchFileList();
         }
 
-        byte[] serializedBytes = this.fetchSerializedBlockBytesForHeight(height);
-        if (serializedBytes == null) {
+        Triple<byte[], Integer, Integer> serializedBlock = this.fetchSerializedBlockBytesForHeight(height);
+        if (serializedBlock == null) {
+            return null;
+        }
+        byte[] serializedBytes = serializedBlock.getA();
+        Integer serializationVersion = serializedBlock.getB();
+        if (serializedBytes == null || serializationVersion == null) {
             return null;
         }
 
         ByteBuffer byteBuffer = ByteBuffer.wrap(serializedBytes);
         BlockTransformation blockInfo = null;
         try {
-            blockInfo = BlockTransformer.fromByteBuffer(byteBuffer);
+            switch (serializationVersion) {
+                case 1:
+                    blockInfo = BlockTransformer.fromByteBuffer(byteBuffer);
+                    break;
+
+                case 2:
+                    blockInfo = BlockTransformer.fromByteBufferV2(byteBuffer);
+                    break;
+
+                default:
+                    // Invalid serialization version
+                    return null;
+            }
+
             if (blockInfo != null && blockInfo.getBlockData() != null) {
                 // Block height is stored outside of the main serialized bytes, so it
                 // won't be set automatically.
@@ -168,15 +196,20 @@ public class BlockArchiveReader {
         return null;
     }
 
-    public byte[] fetchSerializedBlockBytesForSignature(byte[] signature, boolean includeHeightPrefix, Repository repository) {
+    public Triple<byte[], Integer, Integer> fetchSerializedBlockBytesForSignature(byte[] signature, boolean includeHeightPrefix, Repository repository) {
         if (this.fileListCache == null) {
             this.fetchFileList();
         }
 
         Integer height = this.fetchHeightForSignature(signature, repository);
         if (height != null) {
-            byte[] blockBytes = this.fetchSerializedBlockBytesForHeight(height);
-            if (blockBytes == null) {
+            Triple<byte[], Integer, Integer> serializedBlock = this.fetchSerializedBlockBytesForHeight(height);
+            if (serializedBlock == null) {
+                return null;
+            }
+            byte[] blockBytes = serializedBlock.getA();
+            Integer version = serializedBlock.getB();
+            if (blockBytes == null || version == null) {
                 return null;
             }
 
@@ -187,18 +220,18 @@ public class BlockArchiveReader {
                 try {
                     bytes.write(Ints.toByteArray(height));
                     bytes.write(blockBytes);
-                    return bytes.toByteArray();
+                    return new Triple<>(bytes.toByteArray(), version, height);
 
                 } catch (IOException e) {
                     return null;
                 }
             }
-            return blockBytes;
+            return new Triple<>(blockBytes, version, height);
         }
         return null;
     }
 
-    public byte[] fetchSerializedBlockBytesForHeight(int height) {
+    public Triple<byte[], Integer, Integer> fetchSerializedBlockBytesForHeight(int height) {
         String filename = this.getFilenameForHeight(height);
         if (filename == null) {
             // We don't have this block in the archive
@@ -221,7 +254,7 @@ public class BlockArchiveReader {
             // End of fixed length header
 
             // Make sure the version is one we recognize
-            if (version != 1) {
+            if (version != 1 && version != 2) {
                 LOGGER.info("Error: unknown version in file {}: {}", filename, version);
                 return null;
             }
@@ -258,7 +291,7 @@ public class BlockArchiveReader {
             byte[] blockBytes = new byte[blockLength];
             file.read(blockBytes);
 
-            return blockBytes;
+            return new Triple<>(blockBytes, version, height);
 
         } catch (FileNotFoundException e) {
             LOGGER.info("File {} not found: {}", filename, e.getMessage());
@@ -279,6 +312,30 @@ public class BlockArchiveReader {
         }
     }
 
+    public int getHeightOfLastArchivedBlock() {
+        if (this.fileListCache == null) {
+            this.fetchFileList();
+        }
+
+        int maxEndHeight = 0;
+
+        Iterator it = this.fileListCache.entrySet().iterator();
+        while (it.hasNext()) {
+            Map.Entry pair = (Map.Entry) it.next();
+            if (pair == null && pair.getKey() == null && pair.getValue() == null) {
+                continue;
+            }
+            Triple<Integer, Integer, Integer> heightInfo = (Triple<Integer, Integer, Integer>) pair.getValue();
+            Integer endHeight = heightInfo.getB();
+
+            if (endHeight != null && endHeight > maxEndHeight) {
+                maxEndHeight = endHeight;
+            }
+        }
+
+        return maxEndHeight;
+    }
+
     public void invalidateFileListCache() {
         this.fileListCache = null;
     }

src/main/java/org/qortal/repository/BlockArchiveWriter.java

@@ -6,10 +6,13 @@ import org.apache.logging.log4j.Logger;
 import org.qortal.block.Block;
 import org.qortal.controller.Controller;
 import org.qortal.controller.Synchronizer;
+import org.qortal.data.at.ATStateData;
 import org.qortal.data.block.BlockArchiveData;
 import org.qortal.data.block.BlockData;
+import org.qortal.data.transaction.TransactionData;
 import org.qortal.settings.Settings;
 import org.qortal.transform.TransformationException;
+import org.qortal.transform.block.BlockTransformation;
 import org.qortal.transform.block.BlockTransformer;
 
 import java.io.ByteArrayOutputStream;
@@ -18,6 +21,7 @@ import java.io.IOException;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
+import java.util.List;
 
 public class BlockArchiveWriter {
 
@@ -28,25 +32,78 @@ public class BlockArchiveWriter {
         BLOCK_NOT_FOUND
     }
 
+    public enum BlockArchiveDataSource {
+        BLOCK_REPOSITORY, // To build an archive from the Blocks table
+        BLOCK_ARCHIVE // To build a new archive from an existing archive
+    }
+
     private static final Logger LOGGER = LogManager.getLogger(BlockArchiveWriter.class);
 
-    public static final long DEFAULT_FILE_SIZE_TARGET = 100 * 1024 * 1024; // 100MiB
+    public static final long DEFAULT_FILE_SIZE_TARGET_V1 = 100 * 1024 * 1024; // 100MiB
+    public static final long DEFAULT_FILE_SIZE_TARGET_V2 = 10 * 1024 * 1024; // 10MiB
 
     private int startHeight;
     private final int endHeight;
+    private final Integer serializationVersion;
+    private final Path archivePath;
     private final Repository repository;
 
-    private long fileSizeTarget = DEFAULT_FILE_SIZE_TARGET;
+    private long fileSizeTarget = DEFAULT_FILE_SIZE_TARGET_V1;
     private boolean shouldEnforceFileSizeTarget = true;
 
+    // Default data source to BLOCK_REPOSITORY; can optionally be overridden
+    private BlockArchiveDataSource dataSource = BlockArchiveDataSource.BLOCK_REPOSITORY;
+
+    private boolean shouldLogProgress = false;
+
     private int writtenCount;
     private int lastWrittenHeight;
     private Path outputPath;
 
-    public BlockArchiveWriter(int startHeight, int endHeight, Repository repository) {
+    /**
+     * Instantiate a BlockArchiveWriter using a custom archive path
+     * @param startHeight
+     * @param endHeight
+     * @param repository
+     */
+    public BlockArchiveWriter(int startHeight, int endHeight, Integer serializationVersion, Path archivePath, Repository repository) {
         this.startHeight = startHeight;
         this.endHeight = endHeight;
+        this.archivePath = archivePath.toAbsolutePath();
         this.repository = repository;
+
+        if (serializationVersion == null) {
+            // When serialization version isn't specified, fetch it from the existing archive
+            serializationVersion = this.findSerializationVersion();
+        }
+
+        // Reduce default file size target if we're using V2, as the average block size is over 90% smaller
+        if (serializationVersion == 2) {
+            this.setFileSizeTarget(DEFAULT_FILE_SIZE_TARGET_V2);
+        }
+
+        this.serializationVersion = serializationVersion;
+    }
+
+    /**
+     * Instantiate a BlockArchiveWriter using the default archive path and version
+     * @param startHeight
+     * @param endHeight
+     * @param repository
+     */
+    public BlockArchiveWriter(int startHeight, int endHeight, Repository repository) {
+        this(startHeight, endHeight, null, Paths.get(Settings.getInstance().getRepositoryPath(), "archive"), repository);
+    }
+
+    private int findSerializationVersion() {
+        // Attempt to fetch the serialization version from the existing archive
+        Integer block2SerializationVersion = BlockArchiveReader.getInstance().fetchSerializationVersionForHeight(2);
+        if (block2SerializationVersion != null) {
+            return block2SerializationVersion;
+        }
+
+        // Default to version specified in settings
+        return Settings.getInstance().getDefaultArchiveVersion();
     }
 
     public static int getMaxArchiveHeight(Repository repository) throws DataException {
@@ -72,8 +129,7 @@ public class BlockArchiveWriter {
 
     public BlockArchiveWriteResult write() throws DataException, IOException, TransformationException, InterruptedException {
         // Create the archive folder if it doesn't exist
-        // This is a subfolder of the db directory, to make bootstrapping easier
-        Path archivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive").toAbsolutePath();
+        // This is generally a subfolder of the db directory, to make bootstrapping easier
         try {
             Files.createDirectories(archivePath);
         } catch (IOException e) {
@@ -95,13 +151,13 @@ public class BlockArchiveWriter {
 
         LOGGER.info(String.format("Fetching blocks from height %d...", startHeight));
         int i = 0;
-        while (headerBytes.size() + bytes.size() < this.fileSizeTarget
-                || this.shouldEnforceFileSizeTarget == false) {
+        while (headerBytes.size() + bytes.size() < this.fileSizeTarget) {
 
             if (Controller.isStopping()) {
                 return BlockArchiveWriteResult.STOPPING;
             }
             if (Synchronizer.getInstance().isSynchronizing()) {
+                Thread.sleep(1000L);
                 continue;
             }
 
@@ -112,7 +168,28 @@ public class BlockArchiveWriter {
 
             //LOGGER.info("Fetching block {}...", currentHeight);
 
-            BlockData blockData = repository.getBlockRepository().fromHeight(currentHeight);
+            BlockData blockData = null;
+            List<TransactionData> transactions = null;
+            List<ATStateData> atStates = null;
+            byte[] atStatesHash = null;
+
+            switch (this.dataSource) {
+                case BLOCK_ARCHIVE:
+                    BlockTransformation archivedBlock = BlockArchiveReader.getInstance().fetchBlockAtHeight(currentHeight);
+                    if (archivedBlock != null) {
+                        blockData = archivedBlock.getBlockData();
+                        transactions = archivedBlock.getTransactions();
+                        atStates = archivedBlock.getAtStates();
+                        atStatesHash = archivedBlock.getAtStatesHash();
+                    }
+                    break;
+
+                case BLOCK_REPOSITORY:
+                default:
+                    blockData = repository.getBlockRepository().fromHeight(currentHeight);
+                    break;
+            }
+
             if (blockData == null) {
                 return BlockArchiveWriteResult.BLOCK_NOT_FOUND;
             }
@@ -122,18 +199,50 @@ public class BlockArchiveWriter {
             repository.getBlockArchiveRepository().save(blockArchiveData);
             repository.saveChanges();
 
+            // Build the block
+            Block block;
+            if (atStatesHash != null) {
+                block = new Block(repository, blockData, transactions, atStatesHash);
+            }
+            else if (atStates != null) {
+                block = new Block(repository, blockData, transactions, atStates);
+            }
+            else {
+                block = new Block(repository, blockData);
+            }
+
             // Write the block data to some byte buffers
-            Block block = new Block(repository, blockData);
             int blockIndex = bytes.size();
             // Write block index to header
             headerBytes.write(Ints.toByteArray(blockIndex));
             // Write block height
             bytes.write(Ints.toByteArray(block.getBlockData().getHeight()));
-            byte[] blockBytes = BlockTransformer.toBytes(block);
+
+            // Get serialized block bytes
+            byte[] blockBytes;
+            switch (serializationVersion) {
+                case 1:
+                    blockBytes = BlockTransformer.toBytes(block);
+                    break;
+
+                case 2:
+                    blockBytes = BlockTransformer.toBytesV2(block);
+                    break;
+
+                default:
+                    throw new DataException("Invalid serialization version");
+            }
+
             // Write block length
             bytes.write(Ints.toByteArray(blockBytes.length));
             // Write block bytes
             bytes.write(blockBytes);
+
+            // Log every 1000 blocks
+            if (this.shouldLogProgress && i % 1000 == 0) {
+                LOGGER.info("Archived up to block height {}. Size of current file: {} bytes", currentHeight, (headerBytes.size() + bytes.size()));
+            }
+
             i++;
 
         }
@@ -147,11 +256,10 @@ public class BlockArchiveWriter {
 
         // We have enough blocks to create a new file
         int endHeight = startHeight + i - 1;
-        int version = 1;
         String filePath = String.format("%s/%d-%d.dat", archivePath.toString(), startHeight, endHeight);
         FileOutputStream fileOutputStream = new FileOutputStream(filePath);
         // Write version number
-        fileOutputStream.write(Ints.toByteArray(version));
+        fileOutputStream.write(Ints.toByteArray(serializationVersion));
         // Write start height
         fileOutputStream.write(Ints.toByteArray(startHeight));
         // Write end height
@@ -199,4 +307,12 @@ public class BlockArchiveWriter {
         this.shouldEnforceFileSizeTarget = shouldEnforceFileSizeTarget;
     }
 
+    public void setDataSource(BlockArchiveDataSource dataSource) {
+        this.dataSource = dataSource;
+    }
+
+    public void setShouldLogProgress(boolean shouldLogProgress) {
+        this.shouldLogProgress = shouldLogProgress;
+    }
+
 }

src/main/java/org/qortal/repository/ChatRepository.java

@@ -6,6 +6,8 @@ import org.qortal.data.chat.ActiveChats;
 import org.qortal.data.chat.ChatMessage;
 import org.qortal.data.transaction.ChatTransactionData;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 public interface ChatRepository {
 
 	/**
@@ -15,10 +17,11 @@ public interface ChatRepository {
 	 */
 	public List<ChatMessage> getMessagesMatchingCriteria(Long before, Long after,
 			Integer txGroupId, byte[] reference, byte[] chatReferenceBytes, Boolean hasChatReference,
-			List<String> involving, Integer limit, Integer offset, Boolean reverse) throws DataException;
+			List<String> involving, String senderAddress, Encoding encoding,
+			Integer limit, Integer offset, Boolean reverse) throws DataException;
 
-	public ChatMessage toChatMessage(ChatTransactionData chatTransactionData) throws DataException;
+	public ChatMessage toChatMessage(ChatTransactionData chatTransactionData, Encoding encoding) throws DataException;
 
-	public ActiveChats getActiveChats(String address) throws DataException;
+	public ActiveChats getActiveChats(String address, Encoding encoding) throws DataException;
 
 }

src/main/java/org/qortal/repository/RepositoryManager.java

@@ -2,11 +2,6 @@ package org.qortal.repository;
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.qortal.gui.SplashFrame;
-import org.qortal.repository.hsqldb.HSQLDBDatabaseArchiving;
-import org.qortal.repository.hsqldb.HSQLDBDatabasePruning;
-import org.qortal.repository.hsqldb.HSQLDBRepository;
-import org.qortal.settings.Settings;
 
 import java.sql.SQLException;
 import java.util.concurrent.TimeoutException;
@@ -61,62 +56,6 @@ public abstract class RepositoryManager {
 		}
 	}
 
-	public static boolean archive(Repository repository) {
-		if (Settings.getInstance().isLite()) {
-			// Lite nodes have no blockchain
-			return false;
-		}
-
-		// Bulk archive the database the first time we use archive mode
-		if (Settings.getInstance().isArchiveEnabled()) {
-			if (RepositoryManager.canArchiveOrPrune()) {
-				try {
-					return HSQLDBDatabaseArchiving.buildBlockArchive(repository, BlockArchiveWriter.DEFAULT_FILE_SIZE_TARGET);
-
-				} catch (DataException e) {
-					LOGGER.info("Unable to build block archive. The database may have been left in an inconsistent state.");
-				}
-			}
-			else {
-				LOGGER.info("Unable to build block archive due to missing ATStatesHeightIndex. Bootstrapping is recommended.");
-				LOGGER.info("To bootstrap, stop the core and delete the db folder, then start the core again.");
-				SplashFrame.getInstance().updateStatus("Missing index. Bootstrapping is recommended.");
-			}
-		}
-		return false;
-	}
-
-	public static boolean prune(Repository repository) {
-		if (Settings.getInstance().isLite()) {
-			// Lite nodes have no blockchain
-			return false;
-		}
-
-		// Bulk prune the database the first time we use top-only or block archive mode
-		if (Settings.getInstance().isTopOnly() ||
-			Settings.getInstance().isArchiveEnabled()) {
-			if (RepositoryManager.canArchiveOrPrune()) {
-				try {
-					boolean prunedATStates = HSQLDBDatabasePruning.pruneATStates((HSQLDBRepository) repository);
-					boolean prunedBlocks = HSQLDBDatabasePruning.pruneBlocks((HSQLDBRepository) repository);
-
-					// Perform repository maintenance to shrink the db size down
-					if (prunedATStates && prunedBlocks) {
-						HSQLDBDatabasePruning.performMaintenance(repository);
-						return true;
-					}
-
-				} catch (SQLException | DataException e) {
-					LOGGER.info("Unable to bulk prune AT states. The database may have been left in an inconsistent state.");
-				}
-			}
-			else {
-				LOGGER.info("Unable to prune blocks due to missing ATStatesHeightIndex. Bootstrapping is recommended.");
-			}
-		}
-		return false;
-	}
-
 	public static void setRequestedCheckpoint(Boolean quick) {
 		quickCheckpointRequested = quick;
 	}

src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java

@@ -5,9 +5,7 @@ import org.apache.logging.log4j.Logger;
 import org.bouncycastle.util.Longs;
 import org.qortal.arbitrary.misc.Service;
 import org.qortal.data.arbitrary.ArbitraryResourceInfo;
-import org.qortal.crypto.Crypto;
 import org.qortal.data.arbitrary.ArbitraryResourceNameInfo;
-import org.qortal.data.network.ArbitraryPeerData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.ArbitraryTransactionData.*;
 import org.qortal.data.transaction.BaseTransactionData;
@@ -15,6 +13,7 @@ import org.qortal.data.transaction.TransactionData;
 import org.qortal.repository.ArbitraryRepository;
 import org.qortal.repository.DataException;
 import org.qortal.arbitrary.ArbitraryDataFile;
+import org.qortal.transaction.ArbitraryTransaction;
 import org.qortal.transaction.Transaction.ApprovalStatus;
 import org.qortal.utils.Base58;
 
@@ -27,8 +26,6 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 
 	private static final Logger LOGGER = LogManager.getLogger(HSQLDBArbitraryRepository.class);
 
-	private static final int MAX_RAW_DATA_SIZE = 255; // size of VARBINARY
-
 	protected HSQLDBRepository repository;
 	
 	public HSQLDBArbitraryRepository(HSQLDBRepository repository) {
@@ -55,13 +52,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 			return true;
 		}
 
-		// Load hashes
-		byte[] hash = transactionData.getData();
-		byte[] metadataHash = transactionData.getMetadataHash();
-
 		// Load data file(s)
-		ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
-		arbitraryDataFile.setMetadataHash(metadataHash);
+		ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
 		// Check if we already have the complete data file or all chunks
 		if (arbitraryDataFile.allFilesExist()) {
@@ -84,13 +76,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 				return transactionData.getData();
 			}
 
-			// Load hashes
-			byte[] digest = transactionData.getData();
-			byte[] metadataHash = transactionData.getMetadataHash();
-
 			// Load data file(s)
-			ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-			arbitraryDataFile.setMetadataHash(metadataHash);
+			ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
 			// If we have the complete data file, return it
 			if (arbitraryDataFile.exists()) {
@@ -105,6 +92,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 				arbitraryDataFile.join();
 
 				// Verify that the combined hash matches the expected hash
+				byte[] digest = transactionData.getData();
 				if (!digest.equals(arbitraryDataFile.digest())) {
 					LOGGER.info(String.format("Hash mismatch for transaction: %s", Base58.encode(signature)));
 					return null;
@@ -132,11 +120,11 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 		}
 
 		// Trivial-sized payloads can remain in raw form
-		if (arbitraryTransactionData.getDataType() == DataType.RAW_DATA && arbitraryTransactionData.getData().length <= MAX_RAW_DATA_SIZE) {
+		if (arbitraryTransactionData.getDataType() == DataType.RAW_DATA && arbitraryTransactionData.getData().length <= ArbitraryTransaction.MAX_DATA_SIZE) {
 			return;
 		}
 
-		throw new IllegalStateException(String.format("Supplied data is larger than maximum size (%d bytes). Please use ArbitraryDataWriter.", MAX_RAW_DATA_SIZE));
+		throw new IllegalStateException(String.format("Supplied data is larger than maximum size (%d bytes). Please use ArbitraryDataWriter.", ArbitraryTransaction.MAX_DATA_SIZE));
 	}
 
 	@Override
@@ -146,17 +134,11 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 			return;
 		}
 
-		// Load hashes
-		byte[] hash = arbitraryTransactionData.getData();
-		byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
-
 		// Load data file(s)
-		byte[] signature = arbitraryTransactionData.getSignature();
-		ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
-		arbitraryDataFile.setMetadataHash(metadataHash);
+		ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
 
-		// Delete file and chunks
-		arbitraryDataFile.deleteAll();
+		// Delete file, chunks, and metadata
+		arbitraryDataFile.deleteAll(true);
 	}
 
 	@Override
@@ -202,7 +184,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 
 				int version = resultSet.getInt(11);
 				int nonce = resultSet.getInt(12);
-				Service serviceResult = Service.valueOf(resultSet.getInt(13));
+				int serviceInt = resultSet.getInt(13);
 				int size = resultSet.getInt(14);
 				boolean isDataRaw = resultSet.getBoolean(15); // NOT NULL, so no null to false
 				DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH;
@@ -216,7 +198,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 				// FUTURE: get payments from signature if needed. Avoiding for now to reduce database calls.
 
 				ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData,
-						version, serviceResult, nonce, size, nameResult, identifierResult, method, secret,
+						version, serviceInt, nonce, size, nameResult, identifierResult, method, secret,
 						compression, data, dataType, metadataHash, null);
 
 				arbitraryTransactionData.add(transactionData);
@@ -277,7 +259,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 
 			int version = resultSet.getInt(11);
 			int nonce = resultSet.getInt(12);
-			Service serviceResult = Service.valueOf(resultSet.getInt(13));
+			int serviceInt = resultSet.getInt(13);
 			int size = resultSet.getInt(14);
 			boolean isDataRaw = resultSet.getBoolean(15); // NOT NULL, so no null to false
 			DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH;
@@ -291,7 +273,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 			// FUTURE: get payments from signature if needed. Avoiding for now to reduce database calls.
 
 			ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData,
-					version, serviceResult, nonce, size, nameResult, identifierResult, methodResult, secret,
+					version, serviceInt, nonce, size, nameResult, identifierResult, methodResult, secret,
 					compression, data, dataType, metadataHash, null);
 
 			return transactionData;
@@ -378,37 +360,58 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 	}
 
 	@Override
-	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query,
+	public List<ArbitraryResourceInfo> searchArbitraryResources(Service service, String query, String identifier, String name, boolean prefixOnly,
 															 boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException {
 		StringBuilder sql = new StringBuilder(512);
 		List<Object> bindParams = new ArrayList<>();
 
-		// For now we are searching anywhere in the fields
-		// Note that this will bypass any indexes so may not scale well
-		// Longer term we probably want to copy resources to their own table anyway
-		String queryWildcard = String.format("%%%s%%", query.toLowerCase());
-
-		sql.append("SELECT name, service, identifier, MAX(size) AS max_size FROM ArbitraryTransactions WHERE 1=1");
+		sql.append("SELECT name, service, identifier, MAX(size) AS max_size, MIN(created_when) AS date_created, MAX(created_when) AS date_updated " +
+				"FROM ArbitraryTransactions " +
+				"JOIN Transactions USING (signature) " +
+				"WHERE 1=1");
 
 		if (service != null) {
 			sql.append(" AND service = ");
 			sql.append(service.value);
 		}
 
-		if (defaultResource) {
-			// Default resource requested - use NULL identifier and search name only
-			sql.append(" AND LCASE(name) LIKE ? AND identifier IS NULL");
-			bindParams.add(queryWildcard);
+		// Handle general query matches
+		if (query != null) {
+			// Search anywhere in the fields, unless "prefixOnly" has been requested
+			// Note that without prefixOnly it will bypass any indexes so may not scale well
+			// Longer term we probably want to copy resources to their own table anyway
+			String queryWildcard = prefixOnly ? String.format("%s%%", query.toLowerCase()) : String.format("%%%s%%", query.toLowerCase());
+
+			if (defaultResource) {
+				// Default resource requested - use NULL identifier and search name only
+				sql.append(" AND LCASE(name) LIKE ? AND identifier IS NULL");
+				bindParams.add(queryWildcard);
+			} else {
+				// Non-default resource requested
+				// In this case we search the identifier as well as the name
+				sql.append(" AND (LCASE(name) LIKE ? OR LCASE(identifier) LIKE ?)");
+				bindParams.add(queryWildcard);
+				bindParams.add(queryWildcard);
+			}
 		}
-		else {
-			// Non-default resource requested
-			// In this case we search the identifier as well as the name
-			sql.append(" AND (LCASE(name) LIKE ? OR LCASE(identifier) LIKE ?)");
-			bindParams.add(queryWildcard);
+
+		// Handle identifier matches
+		if (identifier != null) {
+			// Search anywhere in the identifier, unless "prefixOnly" has been requested
+			String queryWildcard = prefixOnly ? String.format("%s%%", identifier.toLowerCase()) : String.format("%%%s%%", identifier.toLowerCase());
+			sql.append(" AND LCASE(identifier) LIKE ?");
 			bindParams.add(queryWildcard);
 		}
 
-		sql.append(" GROUP BY name, service, identifier ORDER BY name COLLATE SQL_TEXT_UCC_NO_PAD");
+		// Handle name matches
+		if (name != null) {
+			// Search anywhere in the identifier, unless "prefixOnly" has been requested
+			String queryWildcard = prefixOnly ? String.format("%s%%", name.toLowerCase()) : String.format("%%%s%%", name.toLowerCase());
+			sql.append(" AND LCASE(name) LIKE ?");
+			bindParams.add(queryWildcard);
+		}
+
+		sql.append(" GROUP BY name, service, identifier ORDER BY date_created");
 
 		if (reverse != null && reverse) {
 			sql.append(" DESC");
@@ -427,6 +430,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 				Service serviceResult = Service.valueOf(resultSet.getInt(2));
 				String identifierResult = resultSet.getString(3);
 				Integer sizeResult = resultSet.getInt(4);
+				long dateCreated = resultSet.getLong(5);
+				long dateUpdated = resultSet.getLong(6);
 
 				// We should filter out resources without names
 				if (nameResult == null) {
@@ -438,6 +443,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository {
 				arbitraryResourceInfo.service = serviceResult;
 				arbitraryResourceInfo.identifier = identifierResult;
 				arbitraryResourceInfo.size = Longs.valueOf(sizeResult);
+				arbitraryResourceInfo.created = dateCreated;
+				arbitraryResourceInfo.updated = dateUpdated;
 
 				arbitraryResources.add(arbitraryResourceInfo);
 			} while (resultSet.next());

src/main/java/org/qortal/repository/hsqldb/HSQLDBChatRepository.java

@@ -14,6 +14,8 @@ import org.qortal.repository.ChatRepository;
 import org.qortal.repository.DataException;
 import org.qortal.transaction.Transaction.TransactionType;
 
+import static org.qortal.data.chat.ChatMessage.Encoding;
+
 public class HSQLDBChatRepository implements ChatRepository {
 
 	protected HSQLDBRepository repository;
@@ -24,8 +26,8 @@ public class HSQLDBChatRepository implements ChatRepository {
 
 	@Override
 	public List<ChatMessage> getMessagesMatchingCriteria(Long before, Long after, Integer txGroupId, byte[] referenceBytes,
-			byte[] chatReferenceBytes, Boolean hasChatReference, List<String> involving,
-			Integer limit, Integer offset, Boolean reverse) throws DataException {
+														 byte[] chatReferenceBytes, Boolean hasChatReference, List<String> involving, String senderAddress,
+														 Encoding encoding, Integer limit, Integer offset, Boolean reverse) throws DataException {
 		// Check args meet expectations
 		if ((txGroupId != null && involving != null && !involving.isEmpty())
 				|| (txGroupId == null && (involving == null || involving.size() != 2)))
@@ -74,6 +76,11 @@ public class HSQLDBChatRepository implements ChatRepository {
 			whereClauses.add("chat_reference IS NULL");
 		}
 
+		if (senderAddress != null) {
+			whereClauses.add("sender = ?");
+			bindParams.add(senderAddress);
+		}
+
 		if (txGroupId != null) {
 			whereClauses.add("tx_group_id = " + txGroupId); // int safe to use literally
 			whereClauses.add("recipient IS NULL");
@@ -122,7 +129,7 @@ public class HSQLDBChatRepository implements ChatRepository {
 				byte[] signature = resultSet.getBytes(13);
 
 				ChatMessage chatMessage = new ChatMessage(timestamp, groupId, reference, senderPublicKey, sender,
-						senderName, recipient, recipientName, chatReference, data, isText, isEncrypted, signature);
+						senderName, recipient, recipientName, chatReference, encoding, data, isText, isEncrypted, signature);
 
 				chatMessages.add(chatMessage);
 			} while (resultSet.next());
@@ -134,7 +141,7 @@ public class HSQLDBChatRepository implements ChatRepository {
 	}
 
 	@Override
-	public ChatMessage toChatMessage(ChatTransactionData chatTransactionData) throws DataException {
+	public ChatMessage toChatMessage(ChatTransactionData chatTransactionData, Encoding encoding) throws DataException {
 		String sql = "SELECT SenderNames.name, RecipientNames.name "
 				+ "FROM ChatTransactions "
 				+ "LEFT OUTER JOIN Names AS SenderNames ON SenderNames.owner = sender "
@@ -161,21 +168,22 @@ public class HSQLDBChatRepository implements ChatRepository {
 			byte[] signature = chatTransactionData.getSignature();
 
 			return new ChatMessage(timestamp, groupId, reference, senderPublicKey, sender,
-					senderName, recipient, recipientName, chatReference, data, isText, isEncrypted, signature);
+					senderName, recipient, recipientName, chatReference, encoding, data,
+					isText, isEncrypted, signature);
 		} catch (SQLException e) {
 			throw new DataException("Unable to fetch convert chat transaction from repository", e);
 		}
 	}
 
 	@Override
-	public ActiveChats getActiveChats(String address) throws DataException {
-		List<GroupChat> groupChats = getActiveGroupChats(address);
+	public ActiveChats getActiveChats(String address, Encoding encoding) throws DataException {
+		List<GroupChat> groupChats = getActiveGroupChats(address, encoding);
 		List<DirectChat> directChats = getActiveDirectChats(address);
 
 		return new ActiveChats(groupChats, directChats);
 	}
 
-	private List<GroupChat> getActiveGroupChats(String address) throws DataException {
+	private List<GroupChat> getActiveGroupChats(String address, Encoding encoding) throws DataException {
 		// Find groups where address is a member and potential latest message details
 		String groupsSql = "SELECT group_id, group_name, latest_timestamp, sender, sender_name, signature, data "
 				+ "FROM GroupMembers "
@@ -208,7 +216,7 @@ public class HSQLDBChatRepository implements ChatRepository {
 					byte[] signature = resultSet.getBytes(6);
 					byte[] data = resultSet.getBytes(7);
 
-					GroupChat groupChat = new GroupChat(groupId, groupName, timestamp, sender, senderName, signature, data);
+					GroupChat groupChat = new GroupChat(groupId, groupName, timestamp, sender, senderName, signature, encoding, data);
 					groupChats.add(groupChat);
 				} while (resultSet.next());
 			}
@@ -242,7 +250,7 @@ public class HSQLDBChatRepository implements ChatRepository {
 				data = resultSet.getBytes(5);
 			}
 
-			GroupChat groupChat = new GroupChat(0, null, timestamp, sender, senderName, signature, data);
+			GroupChat groupChat = new GroupChat(0, null, timestamp, sender, senderName, signature, encoding, data);
 			groupChats.add(groupChat);
 		} catch (SQLException e) {
 			throw new DataException("Unable to fetch active group chats from repository", e);

src/main/java/org/qortal/repository/hsqldb/HSQLDBDatabaseArchiving.java

@@ -1,88 +0,0 @@
-package org.qortal.repository.hsqldb;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-import org.qortal.controller.Controller;
-import org.qortal.gui.SplashFrame;
-import org.qortal.repository.BlockArchiveWriter;
-import org.qortal.repository.DataException;
-import org.qortal.repository.Repository;
-import org.qortal.repository.RepositoryManager;
-import org.qortal.transform.TransformationException;
-
-import java.io.IOException;
-
-/**
- *
- * When switching to an archiving node, we need to archive most of the database contents.
- * This involves copying its data into flat files.
- * If we do this entirely as a background process, it is very slow and can interfere with syncing.
- * However, if we take the approach of doing this in bulk, before starting up the rest of the
- * processes, this makes it much faster and less invasive.
- *
- * From that point, the original background archiving process will run, but can be dialled right down
- * so not to interfere with syncing.
- *
- */
-
-
-public class HSQLDBDatabaseArchiving {
-
-    private static final Logger LOGGER = LogManager.getLogger(HSQLDBDatabaseArchiving.class);
-
-
-    public static boolean buildBlockArchive(Repository repository, long fileSizeTarget) throws DataException {
-
-        // Only build the archive if we haven't already got one that is up to date
-        boolean upToDate = BlockArchiveWriter.isArchiverUpToDate(repository);
-        if (upToDate) {
-            // Already archived
-            return false;
-        }
-
-        LOGGER.info("Building block archive - this process could take a while...");
-        SplashFrame.getInstance().updateStatus("Building block archive...");
-
-        final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
-        int startHeight = 0;
-
-        while (!Controller.isStopping()) {
-            try {
-                BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, repository);
-                writer.setFileSizeTarget(fileSizeTarget);
-                BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
-                switch (result) {
-                    case OK:
-                        // Increment block archive height
-                        startHeight = writer.getLastWrittenHeight() + 1;
-                        repository.getBlockArchiveRepository().setBlockArchiveHeight(startHeight);
-                        repository.saveChanges();
-                        break;
-
-                    case STOPPING:
-                        return false;
-
-                    case NOT_ENOUGH_BLOCKS:
-                        // We've reached the limit of the blocks we can archive
-                        // Return from the whole method
-                        return true;
-
-                    case BLOCK_NOT_FOUND:
-                        // We tried to archive a block that didn't exist. This is a major failure and likely means
-                        // that a bootstrap or re-sync is needed. Return rom the method
-                        LOGGER.info("Error: block not found when building archive. If this error persists, " +
-                                "a bootstrap or re-sync may be needed.");
-                        return false;
-                }
-
-            } catch (IOException | TransformationException | InterruptedException e) {
-                LOGGER.info("Caught exception when creating block cache", e);
-                return false;
-            }
-        }
-
-        // If we got this far then something went wrong (most likely the app is stopping)
-        return false;
-    }
-
-}

src/main/java/org/qortal/repository/hsqldb/HSQLDBDatabasePruning.java

@@ -1,332 +0,0 @@
-package org.qortal.repository.hsqldb;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-import org.qortal.controller.Controller;
-import org.qortal.data.block.BlockData;
-import org.qortal.gui.SplashFrame;
-import org.qortal.repository.BlockArchiveWriter;
-import org.qortal.repository.DataException;
-import org.qortal.repository.Repository;
-import org.qortal.settings.Settings;
-
-import java.sql.ResultSet;
-import java.sql.SQLException;
-import java.util.concurrent.TimeoutException;
-
-/**
- *
- * When switching from a full node to a pruning node, we need to delete most of the database contents.
- * If we do this entirely as a background process, it is very slow and can interfere with syncing.
- * However, if we take the approach of transferring only the necessary rows to a new table and then
- * deleting the original table, this makes the process much faster. It was taking several days to
- * delete the AT states in the background, but only a couple of minutes to copy them to a new table.
- *
- * The trade off is that we have to go through a form of "reshape" when starting the app for the first
- * time after enabling pruning mode. But given that this is an opt-in mode, I don't think it will be
- * a problem.
- *
- * Once the pruning is complete, it automatically performs a CHECKPOINT DEFRAG in order to
- * shrink the database file size down to a fraction of what it was before.
- *
- * From this point, the original background process will run, but can be dialled right down so not
- * to interfere with syncing.
- *
- */
-
-
-public class HSQLDBDatabasePruning {
-
-    private static final Logger LOGGER = LogManager.getLogger(HSQLDBDatabasePruning.class);
-
-
-    public static boolean pruneATStates(HSQLDBRepository repository) throws SQLException, DataException {
-
-        // Only bulk prune AT states if we have never done so before
-        int pruneHeight = repository.getATRepository().getAtPruneHeight();
-        if (pruneHeight > 0) {
-            // Already pruned AT states
-            return false;
-        }
-
-        if (Settings.getInstance().isArchiveEnabled()) {
-            // Only proceed if we can see that the archiver has already finished
-            // This way, if the archiver failed for any reason, we can prune once it has had
-            // some opportunities to try again
-            boolean upToDate = BlockArchiveWriter.isArchiverUpToDate(repository);
-            if (!upToDate) {
-                return false;
-            }
-        }
-
-        LOGGER.info("Starting bulk prune of AT states - this process could take a while... " +
-                "(approx. 2 mins on high spec, or upwards of 30 mins in some cases)");
-        SplashFrame.getInstance().updateStatus("Pruning database (takes up to 30 mins)...");
-
-        // Create new AT-states table to hold smaller dataset
-        repository.executeCheckedUpdate("DROP TABLE IF EXISTS ATStatesNew");
-        repository.executeCheckedUpdate("CREATE TABLE ATStatesNew ("
-                + "AT_address QortalAddress, height INTEGER NOT NULL, state_hash ATStateHash NOT NULL, "
-                + "fees QortalAmount NOT NULL, is_initial BOOLEAN NOT NULL, sleep_until_message_timestamp BIGINT, "
-                + "PRIMARY KEY (AT_address, height), "
-                + "FOREIGN KEY (AT_address) REFERENCES ATs (AT_address) ON DELETE CASCADE)");
-        repository.executeCheckedUpdate("SET TABLE ATStatesNew NEW SPACE");
-        repository.executeCheckedUpdate("CHECKPOINT");
-
-        // Add a height index
-        LOGGER.info("Adding index to AT states table...");
-        repository.executeCheckedUpdate("CREATE INDEX IF NOT EXISTS ATStatesNewHeightIndex ON ATStatesNew (height)");
-        repository.executeCheckedUpdate("CHECKPOINT");
-
-
-        // Find our latest block
-        BlockData latestBlock = repository.getBlockRepository().getLastBlock();
-        if (latestBlock == null) {
-            LOGGER.info("Unable to determine blockchain height, necessary for bulk block pruning");
-            return false;
-        }
-
-        // Calculate some constants for later use
-        final int blockchainHeight = latestBlock.getHeight();
-        int maximumBlockToTrim = blockchainHeight - Settings.getInstance().getPruneBlockLimit();
-        if (Settings.getInstance().isArchiveEnabled()) {
-            // Archive mode - don't prune anything that hasn't been archived yet
-            maximumBlockToTrim = Math.min(maximumBlockToTrim, repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1);
-        }
-        final int endHeight = blockchainHeight;
-        final int blockStep = 10000;
-
-
-        // It's essential that we rebuild the latest AT states here, as we are using this data in the next query.
-        // Failing to do this will result in important AT states being deleted, rendering the database unusable.
-        repository.getATRepository().rebuildLatestAtStates(endHeight);
-
-
-        // Loop through all the LatestATStates and copy them to the new table
-        LOGGER.info("Copying AT states...");
-        for (int height = 0; height < endHeight; height += blockStep) {
-            final int batchEndHeight = height + blockStep - 1;
-            //LOGGER.info(String.format("Copying AT states between %d and %d...", height, batchEndHeight));
-
-            String sql = "SELECT height, AT_address FROM LatestATStates WHERE height BETWEEN ? AND ?";
-            try (ResultSet latestAtStatesResultSet = repository.checkedExecute(sql, height, batchEndHeight)) {
-                if (latestAtStatesResultSet != null) {
-                    do {
-                        int latestAtHeight = latestAtStatesResultSet.getInt(1);
-                        String latestAtAddress = latestAtStatesResultSet.getString(2);
-
-                        // Copy this latest ATState to the new table
-                        //LOGGER.info(String.format("Copying AT %s at height %d...", latestAtAddress, latestAtHeight));
-                        try {
-                            String updateSql = "INSERT INTO ATStatesNew ("
-                                    + "SELECT AT_address, height, state_hash, fees, is_initial, sleep_until_message_timestamp "
-                                    + "FROM ATStates "
-                                    + "WHERE height = ? AND AT_address = ?)";
-                            repository.executeCheckedUpdate(updateSql, latestAtHeight, latestAtAddress);
-                        } catch (SQLException e) {
-                            repository.examineException(e);
-                            throw new DataException("Unable to copy ATStates", e);
-                        }
-
-                        // If this batch includes blocks after the maximum block to trim, we will need to copy
-                        // each of its AT states above maximumBlockToTrim as they are considered "recent". We
-                        // need to do this for _all_ AT states in these blocks, regardless of their latest state.
-                        if (batchEndHeight >= maximumBlockToTrim) {
-                            // Now copy this AT's states for each recent block they are present in
-                            for (int i = maximumBlockToTrim; i < endHeight; i++) {
-                                if (latestAtHeight < i) {
-                                    // This AT finished before this block so there is nothing to copy
-                                    continue;
-                                }
-
-                                //LOGGER.info(String.format("Copying recent AT %s at height %d...", latestAtAddress, i));
-                                try {
-                                    // Copy each LatestATState to the new table
-                                    String updateSql = "INSERT IGNORE INTO ATStatesNew ("
-                                            + "SELECT AT_address, height, state_hash, fees, is_initial, sleep_until_message_timestamp "
-                                            + "FROM ATStates "
-                                            + "WHERE height = ? AND AT_address = ?)";
-                                    repository.executeCheckedUpdate(updateSql, i, latestAtAddress);
-                                } catch (SQLException e) {
-                                    repository.examineException(e);
-                                    throw new DataException("Unable to copy ATStates", e);
-                                }
-                            }
-                        }
-                        repository.saveChanges();
-
-                    } while (latestAtStatesResultSet.next());
-                }
-            } catch (SQLException e) {
-                throw new DataException("Unable to copy AT states", e);
-            }
-        }
-
-
-        // Finally, drop the original table and rename
-        LOGGER.info("Deleting old AT states...");
-        repository.executeCheckedUpdate("DROP TABLE ATStates");
-        repository.executeCheckedUpdate("ALTER TABLE ATStatesNew RENAME TO ATStates");
-        repository.executeCheckedUpdate("ALTER INDEX ATStatesNewHeightIndex RENAME TO ATStatesHeightIndex");
-        repository.executeCheckedUpdate("CHECKPOINT");
-
-        // Update the prune height
-        int nextPruneHeight = maximumBlockToTrim + 1;
-        repository.getATRepository().setAtPruneHeight(nextPruneHeight);
-        repository.saveChanges();
-
-        repository.executeCheckedUpdate("CHECKPOINT");
-
-        // Now prune/trim the ATStatesData, as this currently goes back over a month
-        return HSQLDBDatabasePruning.pruneATStateData(repository);
-    }
-
-    /*
-     * Bulk prune ATStatesData to catch up with the now pruned ATStates table
-     * This uses the existing AT States trimming code but with a much higher end block
-     */
-    private static boolean pruneATStateData(Repository repository) throws DataException {
-
-        if (Settings.getInstance().isArchiveEnabled()) {
-            // Don't prune ATStatesData in archive mode
-            return true;
-        }
-
-        BlockData latestBlock = repository.getBlockRepository().getLastBlock();
-        if (latestBlock == null) {
-            LOGGER.info("Unable to determine blockchain height, necessary for bulk ATStatesData pruning");
-            return false;
-        }
-        final int blockchainHeight = latestBlock.getHeight();
-        int upperPrunableHeight = blockchainHeight - Settings.getInstance().getPruneBlockLimit();
-        // ATStateData is already trimmed - so carry on from where we left off in the past
-        int pruneStartHeight = repository.getATRepository().getAtTrimHeight();
-
-        LOGGER.info("Starting bulk prune of AT states data - this process could take a while... (approx. 3 mins on high spec)");
-
-        while (pruneStartHeight < upperPrunableHeight) {
-            // Prune all AT state data up until our latest minus pruneBlockLimit (or our archive height)
-
-            if (Controller.isStopping()) {
-                return false;
-            }
-
-            // Override batch size in the settings because this is a one-off process
-            final int batchSize = 1000;
-            final int rowLimitPerBatch = 50000;
-            int upperBatchHeight = pruneStartHeight + batchSize;
-            int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
-
-            LOGGER.trace(String.format("Pruning AT states data between %d and %d...", pruneStartHeight, upperPruneHeight));
-
-            int numATStatesPruned = repository.getATRepository().trimAtStates(pruneStartHeight, upperPruneHeight, rowLimitPerBatch);
-            repository.saveChanges();
-
-            if (numATStatesPruned > 0) {
-                LOGGER.trace(String.format("Pruned %d AT states data rows between blocks %d and %d",
-                        numATStatesPruned, pruneStartHeight, upperPruneHeight));
-            } else {
-                repository.getATRepository().setAtTrimHeight(upperBatchHeight);
-                // No need to rebuild the latest AT states as we aren't currently synchronizing
-                repository.saveChanges();
-                LOGGER.debug(String.format("Bumping AT states trim height to %d", upperBatchHeight));
-
-                // Can we move onto next batch?
-                if (upperPrunableHeight > upperBatchHeight) {
-                    pruneStartHeight = upperBatchHeight;
-                }
-                else {
-                    // We've finished pruning
-                    break;
-                }
-            }
-        }
-
-        return true;
-    }
-
-    public static boolean pruneBlocks(Repository repository) throws SQLException, DataException {
-
-        // Only bulk prune AT states if we have never done so before
-        int pruneHeight = repository.getBlockRepository().getBlockPruneHeight();
-        if (pruneHeight > 0) {
-            // Already pruned blocks
-            return false;
-        }
-
-        if (Settings.getInstance().isArchiveEnabled()) {
-            // Only proceed if we can see that the archiver has already finished
-            // This way, if the archiver failed for any reason, we can prune once it has had
-            // some opportunities to try again
-            boolean upToDate = BlockArchiveWriter.isArchiverUpToDate(repository);
-            if (!upToDate) {
-                return false;
-            }
-        }
-
-        BlockData latestBlock = repository.getBlockRepository().getLastBlock();
-        if (latestBlock == null) {
-            LOGGER.info("Unable to determine blockchain height, necessary for bulk block pruning");
-            return false;
-        }
-        final int blockchainHeight = latestBlock.getHeight();
-        int upperPrunableHeight = blockchainHeight - Settings.getInstance().getPruneBlockLimit();
-        int pruneStartHeight = 0;
-
-        if (Settings.getInstance().isArchiveEnabled()) {
-            // Archive mode - don't prune anything that hasn't been archived yet
-            upperPrunableHeight = Math.min(upperPrunableHeight, repository.getBlockArchiveRepository().getBlockArchiveHeight() - 1);
-        }
-
-        LOGGER.info("Starting bulk prune of blocks - this process could take a while... (approx. 5 mins on high spec)");
-
-        while (pruneStartHeight < upperPrunableHeight) {
-            // Prune all blocks up until our latest minus pruneBlockLimit
-
-            int upperBatchHeight = pruneStartHeight + Settings.getInstance().getBlockPruneBatchSize();
-            int upperPruneHeight = Math.min(upperBatchHeight, upperPrunableHeight);
-
-            LOGGER.info(String.format("Pruning blocks between %d and %d...", pruneStartHeight, upperPruneHeight));
-
-            int numBlocksPruned = repository.getBlockRepository().pruneBlocks(pruneStartHeight, upperPruneHeight);
-            repository.saveChanges();
-
-            if (numBlocksPruned > 0) {
-                LOGGER.info(String.format("Pruned %d block%s between %d and %d",
-                        numBlocksPruned, (numBlocksPruned != 1 ? "s" : ""),
-                        pruneStartHeight, upperPruneHeight));
-            } else {
-                final int nextPruneHeight = upperPruneHeight + 1;
-                repository.getBlockRepository().setBlockPruneHeight(nextPruneHeight);
-                repository.saveChanges();
-                LOGGER.debug(String.format("Bumping block base prune height to %d", nextPruneHeight));
-
-                // Can we move onto next batch?
-                if (upperPrunableHeight > nextPruneHeight) {
-                    pruneStartHeight = nextPruneHeight;
-                }
-                else {
-                    // We've finished pruning
-                    break;
-                }
-            }
-        }
-
-        return true;
-    }
-
-    public static void performMaintenance(Repository repository) throws SQLException, DataException {
-        try {
-            SplashFrame.getInstance().updateStatus("Performing maintenance...");
-
-            // Timeout if the database isn't ready for backing up after 5 minutes
-            // Nothing else should be using the db at this point, so a timeout shouldn't happen
-            long timeout = 5 * 60 * 1000L;
-            repository.performPeriodicMaintenance(timeout);
-
-        } catch (TimeoutException e) {
-            LOGGER.info("Attempt to perform maintenance failed due to timeout: {}", e.getMessage());
-        }
-    }
-
-}

src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBArbitraryTransactionRepository.java

@@ -31,7 +31,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos
 
 			int version = resultSet.getInt(1);
 			int nonce = resultSet.getInt(2);
-			Service service = Service.valueOf(resultSet.getInt(3));
+			int serviceInt = resultSet.getInt(3);
 			int size = resultSet.getInt(4);
 			boolean isDataRaw = resultSet.getBoolean(5); // NOT NULL, so no null to false
 			DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH;
@@ -44,7 +44,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos
 			ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.valueOf(resultSet.getInt(12));
 
 			List<PaymentData> payments = this.getPaymentsFromSignature(baseTransactionData.getSignature());
-			return new ArbitraryTransactionData(baseTransactionData, version, service, nonce, size, name,
+			return new ArbitraryTransactionData(baseTransactionData, version, serviceInt, nonce, size, name,
 					identifier, method, secret, compression, data, dataType, metadataHash, payments);
 		} catch (SQLException e) {
 			throw new DataException("Unable to fetch arbitrary transaction from repository", e);
@@ -66,7 +66,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos
 		HSQLDBSaver saveHelper = new HSQLDBSaver("ArbitraryTransactions");
 
 		saveHelper.bind("signature", arbitraryTransactionData.getSignature()).bind("sender", arbitraryTransactionData.getSenderPublicKey())
-				.bind("version", arbitraryTransactionData.getVersion()).bind("service", arbitraryTransactionData.getService().value)
+				.bind("version", arbitraryTransactionData.getVersion()).bind("service", arbitraryTransactionData.getServiceInt())
 				.bind("nonce", arbitraryTransactionData.getNonce()).bind("size", arbitraryTransactionData.getSize())
 				.bind("is_data_raw", arbitraryTransactionData.getDataType() == DataType.RAW_DATA).bind("data", arbitraryTransactionData.getData())
 				.bind("metadata_hash", arbitraryTransactionData.getMetadataHash()).bind("name", arbitraryTransactionData.getName())

src/main/java/org/qortal/settings/Settings.java

@@ -104,6 +104,7 @@ public class Settings {
 	private Integer gatewayPort;
 	private boolean gatewayEnabled = false;
 	private boolean gatewayLoggingEnabled = false;
+	private boolean gatewayLoopbackEnabled = false;
 
 	// Specific to this node
 	private boolean wipeUnconfirmedOnStart = false;
@@ -178,6 +179,8 @@ public class Settings {
 	private boolean archiveEnabled = true;
 	/** How often to attempt archiving (ms). */
 	private long archiveInterval = 7171L; // milliseconds
+	/** Serialization version to use when building an archive */
+	private int defaultArchiveVersion = 1;
 
 
 	/** Whether to automatically bootstrap instead of syncing from genesis */
@@ -273,6 +276,7 @@ public class Settings {
 	private String[] bootstrapHosts = new String[] {
 			"http://bootstrap.qortal.org",
 			"http://bootstrap2.qortal.org",
+			"http://bootstrap3.qortal.org",
 			"http://bootstrap.qortal.online"
 	};
 
@@ -350,7 +354,7 @@ public class Settings {
 	private Long maxStorageCapacity = null;
 
 	/** Whether to serve QDN data without authentication */
-	private boolean qdnAuthBypassEnabled = false;
+	private boolean qdnAuthBypassEnabled = true;
 
 	// Domain mapping
 	public static class DomainMap {
@@ -633,6 +637,10 @@ public class Settings {
 		return this.gatewayLoggingEnabled;
 	}
 
+	public boolean isGatewayLoopbackEnabled() {
+		return this.gatewayLoopbackEnabled;
+	}
+
 
 	public boolean getWipeUnconfirmedOnStart() {
 		return this.wipeUnconfirmedOnStart;
@@ -926,6 +934,10 @@ public class Settings {
 		return this.archiveInterval;
 	}
 
+	public int getDefaultArchiveVersion() {
+		return this.defaultArchiveVersion;
+	}
+
 
 	public boolean getBootstrap() {
 		return this.bootstrap;
@@ -994,6 +1006,10 @@ public class Settings {
 	}
 
 	public boolean isQDNAuthBypassEnabled() {
+		if (this.gatewayEnabled) {
+			// We must always bypass QDN authentication in gateway mode, in order for it to function properly
+			return true;
+		}
 		return this.qdnAuthBypassEnabled;
 	}
 }

src/main/java/org/qortal/transaction/ArbitraryTransaction.java

@@ -9,6 +9,7 @@ import org.qortal.account.Account;
 import org.qortal.block.BlockChain;
 import org.qortal.controller.arbitrary.ArbitraryDataManager;
 import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
+import org.qortal.controller.repository.NamesDatabaseIntegrityCheck;
 import org.qortal.crypto.Crypto;
 import org.qortal.crypto.MemoryPoW;
 import org.qortal.data.PaymentData;
@@ -32,7 +33,7 @@ public class ArbitraryTransaction extends Transaction {
 	private ArbitraryTransactionData arbitraryTransactionData;
 
 	// Other useful constants
-	public static final int MAX_DATA_SIZE = 4000;
+	public static final int MAX_DATA_SIZE = 256;
 	public static final int MAX_METADATA_LENGTH = 32;
 	public static final int HASH_LENGTH = TransactionTransformer.SHA256_LENGTH;
 	public static final int MAX_IDENTIFIER_LENGTH = 64;
@@ -87,6 +88,12 @@ public class ArbitraryTransaction extends Transaction {
 		if (this.transactionData.getFee() < 0)
 			return ValidationResult.NEGATIVE_FEE;
 
+		// After the feature trigger, we require the fee to be sufficient if it's not 0.
+		// If the fee is zero, then the nonce is validated in isSignatureValid() as an alternative to a fee
+		if (this.arbitraryTransactionData.getTimestamp() >= BlockChain.getInstance().getArbitraryOptionalFeeTimestamp() && this.arbitraryTransactionData.getFee() != 0L) {
+			return super.isFeeValid();
+		}
+
 		return ValidationResult.OK;
 	}
 
@@ -207,10 +214,14 @@ public class ArbitraryTransaction extends Transaction {
 			// Clear nonce from transactionBytes
 			ArbitraryTransactionTransformer.clearNonce(transactionBytes);
 
-			// We only need to check nonce for recent transactions due to PoW verification overhead
-			if (NTP.getTime() - this.arbitraryTransactionData.getTimestamp() < HISTORIC_THRESHOLD) {
-				int difficulty = ArbitraryDataManager.getInstance().getPowDifficulty();
-				return MemoryPoW.verify2(transactionBytes, POW_BUFFER_SIZE, difficulty, nonce);
+			// As of feature-trigger timestamp, we only require a nonce when the fee is zero
+			boolean beforeFeatureTrigger = this.arbitraryTransactionData.getTimestamp() < BlockChain.getInstance().getArbitraryOptionalFeeTimestamp();
+			if (beforeFeatureTrigger || this.arbitraryTransactionData.getFee() == 0L) {
+				// We only need to check nonce for recent transactions due to PoW verification overhead
+				if (NTP.getTime() - this.arbitraryTransactionData.getTimestamp() < HISTORIC_THRESHOLD) {
+					int difficulty = ArbitraryDataManager.getInstance().getPowDifficulty();
+					return MemoryPoW.verify2(transactionBytes, POW_BUFFER_SIZE, difficulty, nonce);
+				}
 			}
 		}
 
@@ -241,7 +252,15 @@ public class ArbitraryTransaction extends Transaction {
 
 	@Override
 	public void preProcess() throws DataException {
-		// Nothing to do
+		ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
+		if (arbitraryTransactionData.getName() == null)
+			return;
+
+		// Rebuild this name in the Names table from the transaction history
+		// This is necessary because in some rare cases names can be missing from the Names table after registration
+		// but we have been unable to reproduce the issue and track down the root cause
+		NamesDatabaseIntegrityCheck namesDatabaseIntegrityCheck = new NamesDatabaseIntegrityCheck();
+		namesDatabaseIntegrityCheck.rebuildName(arbitraryTransactionData.getName(), this.repository);
 	}
 
 	@Override

src/main/java/org/qortal/transform/block/BlockTransformer.java

@@ -312,16 +312,24 @@ public class BlockTransformer extends Transformer {
 				ByteArrayOutputStream atHashBytes = new ByteArrayOutputStream(atBytesLength);
 				long atFees = 0;
 
-				for (ATStateData atStateData : block.getATStates()) {
-					// Skip initial states generated by DEPLOY_AT transactions in the same block
-					if (atStateData.isInitial())
-						continue;
+				if (block.getAtStatesHash() != null) {
+					// We already have the AT states hash
+					atFees = blockData.getATFees();
+					atHashBytes.write(block.getAtStatesHash());
+				}
+				else {
+					// We need to build the AT states hash
+					for (ATStateData atStateData : block.getATStates()) {
+						// Skip initial states generated by DEPLOY_AT transactions in the same block
+						if (atStateData.isInitial())
+							continue;
 
-					atHashBytes.write(atStateData.getATAddress().getBytes(StandardCharsets.UTF_8));
-					atHashBytes.write(atStateData.getStateHash());
-					atHashBytes.write(Longs.toByteArray(atStateData.getFees()));
+						atHashBytes.write(atStateData.getATAddress().getBytes(StandardCharsets.UTF_8));
+						atHashBytes.write(atStateData.getStateHash());
+						atHashBytes.write(Longs.toByteArray(atStateData.getFees()));
 
-					atFees += atStateData.getFees();
+						atFees += atStateData.getFees();
+					}
 				}
 
 				bytes.write(Ints.toByteArray(blockData.getATCount()));

src/main/java/org/qortal/transform/transaction/ArbitraryTransactionTransformer.java

@@ -7,7 +7,6 @@ import java.util.ArrayList;
 import java.util.List;
 
 import com.google.common.base.Utf8;
-import org.qortal.arbitrary.misc.Service;
 import org.qortal.crypto.Crypto;
 import org.qortal.data.PaymentData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
@@ -131,7 +130,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer {
 				payments.add(PaymentTransformer.fromByteBuffer(byteBuffer));
 		}
 
-		Service service = Service.valueOf(byteBuffer.getInt());
+		int service = byteBuffer.getInt();
 
 		// We might be receiving hash of data instead of actual raw data
 		boolean isRaw = byteBuffer.get() != 0;
@@ -226,7 +225,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer {
 			for (PaymentData paymentData : payments)
 				bytes.write(PaymentTransformer.toBytes(paymentData));
 
-			bytes.write(Ints.toByteArray(arbitraryTransactionData.getService().value));
+			bytes.write(Ints.toByteArray(arbitraryTransactionData.getServiceInt()));
 
 			bytes.write((byte) (arbitraryTransactionData.getDataType() == DataType.RAW_DATA ? 1 : 0));
 
@@ -299,7 +298,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer {
 					bytes.write(PaymentTransformer.toBytes(paymentData));
 			}
 
-			bytes.write(Ints.toByteArray(arbitraryTransactionData.getService().value));
+			bytes.write(Ints.toByteArray(arbitraryTransactionData.getServiceInt()));
 
 			bytes.write(Ints.toByteArray(arbitraryTransactionData.getData().length));
 

src/main/java/org/qortal/utils/ArbitraryTransactionUtils.java

@@ -3,11 +3,11 @@ package org.qortal.utils;
 import org.apache.commons.lang3.ArrayUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
-import org.qortal.arbitrary.ArbitraryDataFile;
-import org.qortal.arbitrary.ArbitraryDataFileChunk;
-import org.qortal.arbitrary.ArbitraryDataReader;
-import org.qortal.arbitrary.ArbitraryDataResource;
+import org.qortal.arbitrary.*;
+import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
 import org.qortal.arbitrary.misc.Service;
+import org.qortal.data.arbitrary.ArbitraryResourceInfo;
+import org.qortal.data.arbitrary.ArbitraryResourceMetadata;
 import org.qortal.data.arbitrary.ArbitraryResourceStatus;
 import org.qortal.data.transaction.ArbitraryTransactionData;
 import org.qortal.data.transaction.TransactionData;
@@ -110,13 +110,8 @@ public class ArbitraryTransactionUtils {
             return false;
         }
 
-        byte[] digest = transactionData.getData();
-        byte[] metadataHash = transactionData.getMetadataHash();
-        byte[] signature = transactionData.getSignature();
-
         // Load complete file and chunks
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-        arbitraryDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
         return arbitraryDataFile.allChunksExist();
     }
@@ -126,18 +121,13 @@ public class ArbitraryTransactionUtils {
             return false;
         }
 
-        byte[] digest = transactionData.getData();
-        byte[] metadataHash = transactionData.getMetadataHash();
-        byte[] signature = transactionData.getSignature();
-
-        if (metadataHash == null) {
+        if (transactionData.getMetadataHash() == null) {
             // This file doesn't have any metadata/chunks, which means none exist
             return false;
         }
 
         // Load complete file and chunks
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-        arbitraryDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
         return arbitraryDataFile.anyChunksExist();
     }
@@ -147,12 +137,7 @@ public class ArbitraryTransactionUtils {
             return 0;
         }
 
-        byte[] digest = transactionData.getData();
-        byte[] metadataHash = transactionData.getMetadataHash();
-        byte[] signature = transactionData.getSignature();
-
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-        arbitraryDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
         // Find the folder containing the files
         Path parentPath = arbitraryDataFile.getFilePath().getParent();
@@ -180,20 +165,15 @@ public class ArbitraryTransactionUtils {
             return 0;
         }
 
-        byte[] digest = transactionData.getData();
-        byte[] metadataHash = transactionData.getMetadataHash();
-        byte[] signature = transactionData.getSignature();
-
-        if (metadataHash == null) {
+        if (transactionData.getMetadataHash() == null) {
             // This file doesn't have any metadata, therefore it has a single (complete) chunk
             return 1;
         }
 
         // Load complete file and chunks
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
-        arbitraryDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
 
-        return arbitraryDataFile.chunkCount() + 1; // +1 for the metadata file
+        return arbitraryDataFile.fileCount();
     }
 
     public static boolean isFileRecent(Path filePath, long now, long cleanupAfter) {
@@ -243,31 +223,24 @@ public class ArbitraryTransactionUtils {
     }
 
     public static void deleteCompleteFileAndChunks(ArbitraryTransactionData arbitraryTransactionData) throws DataException {
-        byte[] completeHash = arbitraryTransactionData.getData();
-        byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
-        byte[] signature = arbitraryTransactionData.getSignature();
-
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(completeHash, signature);
-        arbitraryDataFile.setMetadataHash(metadataHash);
-        arbitraryDataFile.deleteAll();
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
+        arbitraryDataFile.deleteAll(true);
     }
 
     public static void convertFileToChunks(ArbitraryTransactionData arbitraryTransactionData, long now, long cleanupAfter) throws DataException {
-        byte[] completeHash = arbitraryTransactionData.getData();
-        byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
-        byte[] signature = arbitraryTransactionData.getSignature();
-
         // Find the expected chunk hashes
-        ArbitraryDataFile expectedDataFile = ArbitraryDataFile.fromHash(completeHash, signature);
-        expectedDataFile.setMetadataHash(metadataHash);
+        ArbitraryDataFile expectedDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
 
-        if (metadataHash == null || !expectedDataFile.getMetadataFile().exists()) {
+        if (arbitraryTransactionData.getMetadataHash() == null || !expectedDataFile.getMetadataFile().exists()) {
             // We don't have the metadata file, or this transaction doesn't have one - nothing to do
             return;
         }
 
+        byte[] completeHash = arbitraryTransactionData.getData();
+        byte[] signature = arbitraryTransactionData.getSignature();
+
         // Split the file into chunks
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(completeHash, signature);
+        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
         int chunkCount = arbitraryDataFile.split(ArbitraryDataFile.CHUNK_SIZE);
         if (chunkCount > 1) {
             LOGGER.info(String.format("Successfully split %s into %d chunk%s",
@@ -426,7 +399,7 @@ public class ArbitraryTransactionUtils {
 
         // If "build" has been specified, build the resource before returning its status
         if (build != null && build == true) {
-            ArbitraryDataReader reader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, null);
+            ArbitraryDataReader reader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
             try {
                 if (!reader.isBuilding()) {
                     reader.loadSynchronously(false);
@@ -440,4 +413,41 @@ public class ArbitraryTransactionUtils {
         return resource.getStatus(false);
     }
 
+    public static List<ArbitraryResourceInfo> addStatusToResources(List<ArbitraryResourceInfo> resources) {
+        // Determine and add the status of each resource
+        List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
+        for (ArbitraryResourceInfo resourceInfo : resources) {
+            try {
+                ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ArbitraryDataFile.ResourceIdType.NAME,
+                        resourceInfo.service, resourceInfo.identifier);
+                ArbitraryResourceStatus status = resource.getStatus(true);
+                if (status != null) {
+                    resourceInfo.status = status;
+                }
+                updatedResources.add(resourceInfo);
+
+            } catch (Exception e) {
+                // Catch and log all exceptions, since some systems are experiencing 500 errors when including statuses
+                LOGGER.info("Caught exception when adding status to resource {}: {}", resourceInfo, e.toString());
+            }
+        }
+        return updatedResources;
+    }
+
+    public static List<ArbitraryResourceInfo> addMetadataToResources(List<ArbitraryResourceInfo> resources) {
+        // Add metadata fields to each resource if they exist
+        List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
+        for (ArbitraryResourceInfo resourceInfo : resources) {
+            ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ArbitraryDataFile.ResourceIdType.NAME,
+                    resourceInfo.service, resourceInfo.identifier);
+            ArbitraryDataTransactionMetadata transactionMetadata = resource.getLatestTransactionMetadata();
+            ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata, false);
+            if (resourceMetadata != null) {
+                resourceInfo.metadata = resourceMetadata;
+            }
+            updatedResources.add(resourceInfo);
+        }
+        return updatedResources;
+    }
+
 }

src/main/java/org/qortal/utils/BlockArchiveUtils.java

@@ -21,6 +21,16 @@ public class BlockArchiveUtils {
      * into the HSQLDB, in order to make it SQL-compatible
      * again.
      * <p>
+     * This is only fully compatible with archives that use
+     * serialization version 1. For version 2 (or above),
+     * we are unable to import individual AT states as we
+     * only have a single combined hash, so the use cases
+     * for this are greatly limited.
+     * <p>
+     * A version 1 archive should ultimately be rebuildable
+     * via a resync or reindex from genesis, allowing
+     * access to this feature once again.
+     * <p>
      * Note: calls discardChanges() and saveChanges(), so
      * make sure that you commit any existing repository
      * changes before calling this method.
@@ -61,9 +71,18 @@ public class BlockArchiveUtils {
                 repository.getBlockRepository().save(blockInfo.getBlockData());
 
                 // Save AT state data hashes
-                for (ATStateData atStateData : blockInfo.getAtStates()) {
-                    atStateData.setHeight(blockInfo.getBlockData().getHeight());
-                    repository.getATRepository().save(atStateData);
+                if (blockInfo.getAtStates() != null) {
+                    for (ATStateData atStateData : blockInfo.getAtStates()) {
+                        atStateData.setHeight(blockInfo.getBlockData().getHeight());
+                        repository.getATRepository().save(atStateData);
+                    }
+                }
+                else {
+                    // We don't have AT state hashes, so we are only importing a partial state.
+                    // This can still be useful to allow orphaning to very old blocks, when we
+                    // need to access other chainstate info (such as balances) at an earlier block.
+                    // In order to do this, the orphan process must be temporarily adjusted to avoid
+                    // orphaning AT states, as it will otherwise fail due to having no previous state.
                 }
 
             } catch (DataException e) {

src/main/java/org/qortal/utils/FilesystemUtils.java

@@ -241,13 +241,48 @@ public class FilesystemUtils {
             String[] files = ArrayUtils.removeElement(path.toFile().list(), ".qortal");
             if (files.length == 1) {
                 Path filePath = Paths.get(path.toString(), files[0]);
-                data = Files.readAllBytes(filePath);
+                if (filePath.toFile().isFile()) {
+                    data = Files.readAllBytes(filePath);
+                }
             }
         }
 
         return data;
     }
 
+    /**
+     * isSingleFileResource
+     * Returns true if the path points to a file, or a
+     * directory containing a single file only.
+     *
+     * @param path to file or directory
+     * @param excludeQortalDirectory - if true, a directory containing a single file and a .qortal directory is considered a single file resource
+     * @return
+     * @throws IOException
+     */
+    public static boolean isSingleFileResource(Path path, boolean excludeQortalDirectory) {
+        // If the path is a file, read the contents directly
+        if (path.toFile().isFile()) {
+            return true;
+        }
+
+        // Or if it's a directory, only load file contents if there is a single file inside it
+        else if (path.toFile().isDirectory()) {
+            String[] files = path.toFile().list();
+            if (excludeQortalDirectory) {
+                files = ArrayUtils.removeElement(files, ".qortal");
+            }
+            if (files.length == 1) {
+                Path filePath = Paths.get(path.toString(), files[0]);
+                if (filePath.toFile().isFile()) {
+                    return true;
+                }
+            }
+        }
+
+        return false;
+    }
+
     public static byte[] readFromFile(String filePath, long position, int size) throws IOException {
         RandomAccessFile file = new RandomAccessFile(filePath, "r");
         file.seek(position);

src/main/resources/blockchain.json

@@ -85,7 +85,8 @@
 		"onlineAccountMinterLevelValidationHeight": 1092000,
 		"selfSponsorshipAlgoV1Height": 1092400,
 		"feeValidationFixTimestamp": 1671918000000,
-		"chatReferenceTimestamp": 1674316800000
+		"chatReferenceTimestamp": 1674316800000,
+		"arbitraryOptionalFeeTimestamp": 1680278400000
 	},
 	"checkpoints": [
 		{ "height": 1136300, "signature": "3BbwawEF2uN8Ni5ofpJXkukoU8ctAPxYoFB7whq9pKfBnjfZcpfEJT4R95NvBDoTP8WDyWvsUvbfHbcr9qSZuYpSKZjUQTvdFf6eqznHGEwhZApWfvXu6zjGCxYCp65F4jsVYYJjkzbjmkCg5WAwN5voudngA23kMK6PpTNygapCzXt" }

src/main/resources/loading/index.html

@@ -43,8 +43,9 @@
       var host = location.protocol + '//' + location.host;
       var service = "%%SERVICE%%"
       var name = "%%NAME%%"
+      var identifier = "%%IDENTIFIER%%"
 
-      var url = host + '/arbitrary/resource/status/' + service + '/' + name + '?build=true';
+      var url = host + '/arbitrary/resource/status/' + service + '/' + name + '/' + identifier + '?build=true';
       var textStatus = "Loading...";
       var textProgress = "";
       var retryInterval = 2500;
@@ -74,18 +75,18 @@
       }
       else if (status.id == "BUILDING") {
         textStatus = status.description;
-        retryInterval = 1000;
+        retryInterval = 2000;
       }
       else if (status.id == "BUILD_FAILED") {
         textStatus = status.description;
       }
       else if (status.id == "NOT_STARTED") {
         textStatus = status.description;
-        retryInterval = 1000;
+        retryInterval = 2000;
       }
       else if (status.id == "DOWNLOADING") {
         textStatus = status.description;
-        retryInterval = 1000;
+        retryInterval = 2000;
       }
       else if (status.id == "MISSING_DATA") {
         textStatus = status.description;
@@ -96,8 +97,14 @@
       else if (status.id == "DOWNLOADED") {
         textStatus = status.description;
       }
+      else if (status.id == "NOT_PUBLISHED") {
+        document.getElementById("title").innerHTML = "File not found";
+        document.getElementById("description").innerHTML = "";
+        document.getElementById("c").style.opacity = "0.5";
+        textStatus = status.description;
+      }
 
-      if (status.localChunkCount != null && status.totalChunkCount != null) {
+      if (status.localChunkCount != null && status.totalChunkCount != null && status.totalChunkCount > 0) {
         textProgress = "Files downloaded: " + status.localChunkCount + " / " + status.totalChunkCount;
       }
 
@@ -275,8 +282,8 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
 
 <div id="panel-outer">
   <div id="panel">
-    <h2>Loading</h2>
-    <p>
+    <h2 id="title">Loading</h2>
+    <p id="description">
       Files are being retrieved from the Qortal Data Network.
       This page will refresh automatically when the content becomes available.
     </p>

src/main/resources/q-apps/q-apps-gateway.js

@@ -0,0 +1,38 @@
+console.log("Gateway mode");
+
+window.addEventListener("message", (event) => {
+    if (event == null || event.data == null || event.data.length == 0) {
+        return;
+    }
+    if (event.data.action == null || event.data.requestedHandler == null) {
+        return;
+    }
+    if (event.data.requestedHandler != "UI") {
+        // Gateway mode only cares about requests that were intended for the UI
+        return;
+    }
+
+    let response;
+    let data = event.data;
+
+    switch (data.action) {
+        case "GET_USER_ACCOUNT":
+        case "PUBLISH_QDN_RESOURCE":
+        case "SEND_CHAT_MESSAGE":
+        case "JOIN_GROUP":
+        case "DEPLOY_AT":
+        case "GET_WALLET_BALANCE":
+        case "SEND_COIN":
+            const errorString = "Authentication was requested, but this is not yet supported when viewing via a gateway. To use interactive features, please access using the Qortal UI desktop app. More info at: https://qortal.org";
+            alert(errorString);
+            response = "{\"error\": \"" + errorString + "\"}"
+            break;
+
+        default:
+            console.log('Unhandled gateway message: ' + JSON.stringify(data));
+            return;
+    }
+
+    handleResponse(event, response);
+
+}, false);

src/main/resources/q-apps/q-apps.js

@@ -0,0 +1,483 @@
+function httpGet(url) {
+    var request = new XMLHttpRequest();
+    request.open("GET", url, false);
+    request.send(null);
+    return request.responseText;
+}
+
+function handleResponse(event, response) {
+    if (event == null) {
+        return;
+    }
+
+    // Handle empty or missing responses
+    if (response == null || response.length == 0) {
+        response = "{\"error\": \"Empty response\"}"
+    }
+
+    // Parse response
+    let responseObj;
+    try {
+        responseObj = JSON.parse(response);
+    } catch (e) {
+        // Not all responses will be JSON
+        responseObj = response;
+    }
+
+    // Respond to app
+    if (responseObj.error != null) {
+        event.ports[0].postMessage({
+            result: null,
+            error: responseObj
+        });
+    }
+    else {
+        event.ports[0].postMessage({
+            result: responseObj,
+            error: null
+        });
+    }
+}
+
+function buildResourceUrl(service, name, identifier, path, isLink) {
+    if (isLink == false) {
+        // If this URL isn't being used as a link, then we need to fetch the data
+        // synchronously, instead of showing the loading screen.
+        url = "/arbitrary/" + service + "/" + name;
+        if (identifier != null) url = url.concat("/" + identifier);
+        if (path != null) url = url.concat("?filepath=" + path);
+    }
+    else if (_qdnContext == "render") {
+        url = "/render/" + service + "/" + name;
+        if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path);
+        if (identifier != null) url = url.concat("?identifier=" + identifier);
+    }
+    else if (_qdnContext == "gateway") {
+        url = "/" + service + "/" + name;
+        if (identifier != null) url = url.concat("/" + identifier);
+        if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path);
+    }
+    else {
+        // domainMap only serves websites right now
+        url = "/" + name;
+        if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path);
+    }
+
+    if (isLink) url = url.concat((url.includes("?") ? "" : "?") + "&theme=" + _qdnTheme);
+
+    return url;
+}
+
+function extractComponents(url) {
+    if (!url.startsWith("qortal://")) {
+        return null;
+    }
+
+    url = url.replace(/^(qortal\:\/\/)/,"");
+    if (url.includes("/")) {
+        let parts = url.split("/");
+        const service = parts[0].toUpperCase();
+        parts.shift();
+        const name = parts[0];
+        parts.shift();
+        let identifier;
+
+        if (parts.length > 0) {
+            identifier = parts[0]; // Do not shift yet
+            // Check if a resource exists with this service, name and identifier combination
+            const url = "/arbitrary/resource/status/" + service + "/" + name + "/" + identifier;
+            const response = httpGet(url);
+            const responseObj = JSON.parse(response);
+            if (responseObj.totalChunkCount > 0) {
+                // Identifier exists, so don't include it in the path
+                parts.shift();
+            }
+            else {
+                identifier = null;
+            }
+        }
+
+        const path = parts.join("/");
+
+        const components = {};
+        components["service"] = service;
+        components["name"] = name;
+        components["identifier"] = identifier;
+        components["path"] = path;
+        return components;
+    }
+
+    return null;
+}
+
+function convertToResourceUrl(url, isLink) {
+    if (!url.startsWith("qortal://")) {
+        return null;
+    }
+    const c = extractComponents(url);
+    if (c == null) {
+        return null;
+    }
+
+    return buildResourceUrl(c.service, c.name, c.identifier, c.path, isLink);
+}
+
+window.addEventListener("message", (event) => {
+    if (event == null || event.data == null || event.data.length == 0) {
+        return;
+    }
+    if (event.data.action == null) {
+        // This could be a response from the UI
+        handleResponse(event, event.data);
+    }
+    if (event.data.requestedHandler != null && event.data.requestedHandler === "UI") {
+        // This request was destined for the UI, so ignore it
+        return;
+    }
+
+    console.log("Core received event: " + JSON.stringify(event.data));
+
+    let url;
+    let response;
+    let data = event.data;
+
+    switch (data.action) {
+        case "GET_ACCOUNT_DATA":
+            response = httpGet("/addresses/" + data.address);
+            break;
+
+        case "GET_ACCOUNT_NAMES":
+            response = httpGet("/names/address/" + data.address);
+            break;
+
+        case "GET_NAME_DATA":
+            response = httpGet("/names/" + data.name);
+            break;
+
+        case "GET_QDN_RESOURCE_URL":
+            response = buildResourceUrl(data.service, data.name, data.identifier, data.path, false);
+            break;
+
+        case "LINK_TO_QDN_RESOURCE":
+            if (data.service == null) data.service = "WEBSITE"; // Default to WEBSITE
+            window.location = buildResourceUrl(data.service, data.name, data.identifier, data.path, true);
+            break;
+
+        case "LIST_QDN_RESOURCES":
+            url = "/arbitrary/resources?";
+            if (data.service != null) url = url.concat("&service=" + data.service);
+            if (data.name != null) url = url.concat("&name=" + data.name);
+            if (data.identifier != null) url = url.concat("&identifier=" + data.identifier);
+            if (data.default != null) url = url.concat("&default=" + new Boolean(data.default).toString());
+            if (data.includeStatus != null) url = url.concat("&includestatus=" + new Boolean(data.includeStatus).toString());
+            if (data.includeMetadata != null) url = url.concat("&includemetadata=" + new Boolean(data.includeMetadata).toString());
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "SEARCH_QDN_RESOURCES":
+            url = "/arbitrary/resources/search?";
+            if (data.service != null) url = url.concat("&service=" + data.service);
+            if (data.query != null) url = url.concat("&query=" + data.query);
+            if (data.identifier != null) url = url.concat("&identifier=" + data.identifier);
+            if (data.name != null) url = url.concat("&name=" + data.name);
+            if (data.prefix != null) url = url.concat("&prefix=" + new Boolean(data.prefix).toString());
+            if (data.default != null) url = url.concat("&default=" + new Boolean(data.default).toString());
+            if (data.includeStatus != null) url = url.concat("&includestatus=" + new Boolean(data.includeStatus).toString());
+            if (data.includeMetadata != null) url = url.concat("&includemetadata=" + new Boolean(data.includeMetadata).toString());
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "FETCH_QDN_RESOURCE":
+            url = "/arbitrary/" + data.service + "/" + data.name;
+            if (data.identifier != null) url = url.concat("/" + data.identifier);
+            url = url.concat("?");
+            if (data.filepath != null) url = url.concat("&filepath=" + data.filepath);
+            if (data.rebuild != null) url = url.concat("&rebuild=" + new Boolean(data.rebuild).toString())
+            if (data.encoding != null) url = url.concat("&encoding=" + data.encoding);
+            response = httpGet(url);
+            break;
+
+        case "GET_QDN_RESOURCE_STATUS":
+            url = "/arbitrary/resource/status/" + data.service + "/" + data.name;
+            if (data.identifier != null) url = url.concat("/" + data.identifier);
+            response = httpGet(url);
+            break;
+
+        case "GET_QDN_RESOURCE_PROPERTIES":
+            let identifier = (data.identifier != null) ? data.identifier : "default";
+            url = "/arbitrary/resource/properties/" + data.service + "/" + data.name + "/" + identifier;
+            response = httpGet(url);
+            break;
+
+        case "SEARCH_CHAT_MESSAGES":
+            url = "/chat/messages?";
+            if (data.before != null) url = url.concat("&before=" + data.before);
+            if (data.after != null) url = url.concat("&after=" + data.after);
+            if (data.txGroupId != null) url = url.concat("&txGroupId=" + data.txGroupId);
+            if (data.involving != null) data.involving.forEach((x, i) => url = url.concat("&involving=" + x));
+            if (data.reference != null) url = url.concat("&reference=" + data.reference);
+            if (data.chatReference != null) url = url.concat("&chatreference=" + data.chatReference);
+            if (data.hasChatReference != null) url = url.concat("&haschatreference=" + new Boolean(data.hasChatReference).toString());
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "LIST_GROUPS":
+            url = "/groups?";
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "GET_BALANCE":
+            url = "/addresses/balance/" + data.address;
+            if (data.assetId != null) url = url.concat("&assetId=" + data.assetId);
+            response = httpGet(url);
+            break;
+
+        case "GET_AT":
+            url = "/at" + data.atAddress;
+            response = httpGet(url);
+            break;
+
+        case "GET_AT_DATA":
+            url = "/at/" + data.atAddress + "/data";
+            response = httpGet(url);
+            break;
+
+        case "LIST_ATS":
+            url = "/at/byfunction/" + data.codeHash58 + "?";
+            if (data.isExecutable != null) url = url.concat("&isExecutable=" + data.isExecutable);
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "FETCH_BLOCK":
+            if (data.signature != null) {
+                url = "/blocks/" + data.signature;
+            }
+            else if (data.height != null) {
+                url = "/blocks/byheight/" + data.height;
+            }
+            url = url.concat("?");
+            if (data.includeOnlineSignatures != null) url = url.concat("&includeOnlineSignatures=" + data.includeOnlineSignatures);
+            response = httpGet(url);
+            break;
+
+        case "FETCH_BLOCK_RANGE":
+            url = "/blocks/range/" + data.height + "?";
+            if (data.count != null) url = url.concat("&count=" + data.count);
+            if (data.reverse != null) url = url.concat("&reverse=" + data.reverse);
+            if (data.includeOnlineSignatures != null) url = url.concat("&includeOnlineSignatures=" + data.includeOnlineSignatures);
+            response = httpGet(url);
+            break;
+
+        case "SEARCH_TRANSACTIONS":
+            url = "/transactions/search?";
+            if (data.startBlock != null) url = url.concat("&startBlock=" + data.startBlock);
+            if (data.blockLimit != null) url = url.concat("&blockLimit=" + data.blockLimit);
+            if (data.txGroupId != null) url = url.concat("&txGroupId=" + data.txGroupId);
+            if (data.txType != null) data.txType.forEach((x, i) => url = url.concat("&txType=" + x));
+            if (data.address != null) url = url.concat("&address=" + data.address);
+            if (data.confirmationStatus != null) url = url.concat("&confirmationStatus=" + data.confirmationStatus);
+            if (data.limit != null) url = url.concat("&limit=" + data.limit);
+            if (data.offset != null) url = url.concat("&offset=" + data.offset);
+            if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString());
+            response = httpGet(url);
+            break;
+
+        case "GET_PRICE":
+            url = "/crosschain/price/" + data.blockchain + "?";
+            if (data.maxtrades != null) url = url.concat("&maxtrades=" + data.maxtrades);
+            if (data.inverse != null) url = url.concat("&inverse=" + data.inverse);
+            response = httpGet(url);
+            break;
+
+        default:
+            // Pass to parent (UI), in case they can fulfil this request
+            event.data.requestedHandler = "UI";
+            parent.postMessage(event.data, '*', [event.ports[0]]);
+            return;
+    }
+
+    if (response == null) {
+        // Pass to parent (UI), in case they can fulfil this request
+        event.data.requestedHandler = "UI";
+        parent.postMessage(event.data, '*', [event.ports[0]]);
+        return;
+    }
+
+    handleResponse(event, response);
+
+}, false);
+
+
+/**
+ * Listen for and intercept all link click events
+ */
+function interceptClickEvent(e) {
+    var target = e.target || e.srcElement;
+    if (target.tagName !== 'A') {
+        target = target.closest('A');
+    }
+    if (target == null || target.getAttribute('href') == null) {
+        return;
+    }
+    let href = target.getAttribute('href');
+    if (href.startsWith("qortal://")) {
+        const c = extractComponents(href);
+        if (c != null) {
+            qortalRequest({
+                action: "LINK_TO_QDN_RESOURCE",
+                service: c.service,
+                name: c.name,
+                identifier: c.identifier,
+                path: c.path
+            });
+        }
+        e.preventDefault();
+    }
+    else if (href.startsWith("http://") || href.startsWith("https://") || href.startsWith("//")) {
+        // Block external links
+        e.preventDefault();
+    }
+}
+if (document.addEventListener) {
+    document.addEventListener('click', interceptClickEvent);
+}
+else if (document.attachEvent) {
+    document.attachEvent('onclick', interceptClickEvent);
+}
+
+/**
+ * Send current page details to UI
+ */
+document.addEventListener('DOMContentLoaded', () => {
+    qortalRequest({
+        action: "QDN_RESOURCE_DISPLAYED",
+        service: _qdnService,
+        name: _qdnName,
+        identifier: _qdnIdentifier,
+        path: _qdnPath
+    });
+});
+
+/**
+ * Intercept image loads from the DOM
+ */
+document.addEventListener('DOMContentLoaded', () => {
+    const imgElements = document.querySelectorAll('img');
+    imgElements.forEach((img) => {
+        let url = img.src;
+        const newUrl = convertToResourceUrl(url, false);
+        if (newUrl != null) {
+            document.querySelector('img').src = newUrl;
+        }
+    });
+});
+
+/**
+ * Intercept img src updates
+ */
+document.addEventListener('DOMContentLoaded', () => {
+    const imgElements = document.querySelectorAll('img');
+    imgElements.forEach((img) => {
+        let observer = new MutationObserver((changes) => {
+            changes.forEach(change => {
+                if (change.attributeName.includes('src')) {
+                    const newUrl = convertToResourceUrl(img.src, false);
+                    if (newUrl != null) {
+                        document.querySelector('img').src = newUrl;
+                    }
+                }
+            });
+        });
+        observer.observe(img, {attributes: true});
+    });
+});
+
+
+
+const awaitTimeout = (timeout, reason) =>
+    new Promise((resolve, reject) =>
+        setTimeout(
+            () => (reason === undefined ? resolve() : reject(reason)),
+            timeout
+        )
+    );
+
+function getDefaultTimeout(action) {
+    if (action != null) {
+        // Some actions need longer default timeouts, especially those that create transactions
+        switch (action) {
+            case "GET_USER_ACCOUNT":
+                // User may take a long time to accept/deny the popup
+                return 60 * 60 * 1000;
+
+            case "FETCH_QDN_RESOURCE":
+                // Fetching data can take a while, especially if the status hasn't been checked first
+                return 60 * 1000;
+
+            case "PUBLISH_QDN_RESOURCE":
+                // Publishing could take a very long time on slow system, due to the proof-of-work computation
+                // It's best not to timeout
+                return 60 * 60 * 1000;
+
+            case "SEND_CHAT_MESSAGE":
+                // Chat messages rely on PoW computations, so allow extra time
+                return 60 * 1000;
+
+            case "JOIN_GROUP":
+            case "DEPLOY_AT":
+            case "SEND_COIN":
+                // Allow extra time for other actions that create transactions, even if there is no PoW
+                return 5 * 60 * 1000;
+
+            default:
+                break;
+        }
+    }
+    return 10 * 1000;
+}
+
+/**
+ * Make a Qortal (Q-Apps) request with no timeout
+ */
+const qortalRequestWithNoTimeout = (request) => new Promise((res, rej) => {
+    const channel = new MessageChannel();
+
+    channel.port1.onmessage = ({data}) => {
+        channel.port1.close();
+
+        if (data.error) {
+            rej(data.error);
+        } else {
+            res(data.result);
+        }
+    };
+
+    window.postMessage(request, '*', [channel.port2]);
+});
+
+/**
+ * Make a Qortal (Q-Apps) request with the default timeout (10 seconds)
+ */
+const qortalRequest = (request) =>
+    Promise.race([qortalRequestWithNoTimeout(request), awaitTimeout(getDefaultTimeout(request.action), "The request timed out")]);
+
+/**
+ * Make a Qortal (Q-Apps) request with a custom timeout, specified in milliseconds
+ */
+const qortalRequestWithTimeout = (request, timeout) =>
+    Promise.race([qortalRequestWithNoTimeout(request), awaitTimeout(timeout, "The request timed out")]);

src/test/java/org/qortal/test/BlockArchiveV1Tests.java

@@ -1,6 +1,7 @@
 package org.qortal.test;
 
 import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
@@ -10,8 +11,6 @@ import org.qortal.data.at.ATStateData;
 import org.qortal.data.block.BlockData;
 import org.qortal.data.transaction.TransactionData;
 import org.qortal.repository.*;
-import org.qortal.repository.hsqldb.HSQLDBDatabaseArchiving;
-import org.qortal.repository.hsqldb.HSQLDBDatabasePruning;
 import org.qortal.repository.hsqldb.HSQLDBRepository;
 import org.qortal.settings.Settings;
 import org.qortal.test.common.AtUtils;
@@ -26,7 +25,6 @@ import org.qortal.utils.NTP;
 
 import java.io.File;
 import java.io.IOException;
-import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.sql.SQLException;
@@ -34,13 +32,16 @@ import java.util.List;
 
 import static org.junit.Assert.*;
 
-public class BlockArchiveTests extends Common {
+public class BlockArchiveV1Tests extends Common {
 
 	@Before
-	public void beforeTest() throws DataException {
+	public void beforeTest() throws DataException, IllegalAccessException {
 		Common.useSettings("test-settings-v2-block-archive.json");
 		NTP.setFixedOffset(Settings.getInstance().getTestNtpOffset());
 		this.deleteArchiveDirectory();
+
+		// Set default archive version to 1, so that archive builds in these tests use V2
+		FieldUtils.writeField(Settings.getInstance(), "defaultArchiveVersion", 1, true);
 	}
 
 	@After
@@ -333,212 +334,6 @@ public class BlockArchiveTests extends Common {
 		}
 	}
 
-	@Test
-	public void testBulkArchiveAndPrune() throws DataException, SQLException {
-		try (final Repository repository = RepositoryManager.getRepository()) {
-			HSQLDBRepository hsqldb = (HSQLDBRepository) repository;
-
-			// Deploy an AT so that we have AT state data
-			PrivateKeyAccount deployer = Common.getTestAccount(repository, "alice");
-			byte[] creationBytes = AtUtils.buildSimpleAT();
-			long fundingAmount = 1_00000000L;
-			AtUtils.doDeployAT(repository, deployer, creationBytes, fundingAmount);
-
-			// Mint some blocks so that we are able to archive them later
-			for (int i = 0; i < 1000; i++) {
-				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
-			}
-
-			// Assume 900 blocks are trimmed (this specifies the first untrimmed height)
-			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(901);
-			repository.getATRepository().setAtTrimHeight(901);
-
-			// Check the max archive height - this should be one less than the first untrimmed height
-			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
-			assertEquals(900, maximumArchiveHeight);
-
-			// Check the current archive height
-			assertEquals(0, repository.getBlockArchiveRepository().getBlockArchiveHeight());
-
-			// Write blocks 2-900 to the archive (using bulk method)
-			int fileSizeTarget = 428600; // Pre-calculated size of 900 blocks
-			assertTrue(HSQLDBDatabaseArchiving.buildBlockArchive(repository, fileSizeTarget));
-
-			// Ensure the block archive height has increased
-			assertEquals(901, repository.getBlockArchiveRepository().getBlockArchiveHeight());
-
-			// Ensure the SQL repository contains blocks 2 and 900...
-			assertNotNull(repository.getBlockRepository().fromHeight(2));
-			assertNotNull(repository.getBlockRepository().fromHeight(900));
-
-			// Check the current prune heights
-			assertEquals(0, repository.getBlockRepository().getBlockPruneHeight());
-			assertEquals(0, repository.getATRepository().getAtPruneHeight());
-
-			// Prior to archiving or pruning, ensure blocks 2 to 1002 and their AT states are available in the db
-			for (int i=2; i<=1002; i++) {
-				assertNotNull(repository.getBlockRepository().fromHeight(i));
-				List<ATStateData> atStates = repository.getATRepository().getBlockATStatesAtHeight(i);
-				assertNotNull(atStates);
-				assertEquals(1, atStates.size());
-			}
-
-			// Prune all the archived blocks and AT states (using bulk method)
-			assertTrue(HSQLDBDatabasePruning.pruneBlocks(hsqldb));
-			assertTrue(HSQLDBDatabasePruning.pruneATStates(hsqldb));
-
-			// Ensure the current prune heights have increased
-			assertEquals(901, repository.getBlockRepository().getBlockPruneHeight());
-			assertEquals(901, repository.getATRepository().getAtPruneHeight());
-
-			// Now ensure the SQL repository is missing blocks 2 and 900...
-			assertNull(repository.getBlockRepository().fromHeight(2));
-			assertNull(repository.getBlockRepository().fromHeight(900));
-
-			// ... but it's not missing blocks 1 and 901 (we don't prune the genesis block)
-			assertNotNull(repository.getBlockRepository().fromHeight(1));
-			assertNotNull(repository.getBlockRepository().fromHeight(901));
-
-			// Validate the latest block height in the repository
-			assertEquals(1002, (int) repository.getBlockRepository().getLastBlock().getHeight());
-
-			// Ensure blocks 2-900 are all available in the archive
-			for (int i=2; i<=900; i++) {
-				assertNotNull(repository.getBlockArchiveRepository().fromHeight(i));
-			}
-
-			// Ensure blocks 2-900 are NOT available in the db
-			for (int i=2; i<=900; i++) {
-				assertNull(repository.getBlockRepository().fromHeight(i));
-			}
-
-			// Ensure blocks 901 to 1002 and their AT states are available in the db
-			for (int i=901; i<=1002; i++) {
-				assertNotNull(repository.getBlockRepository().fromHeight(i));
-				List<ATStateData> atStates = repository.getATRepository().getBlockATStatesAtHeight(i);
-				assertNotNull(atStates);
-				assertEquals(1, atStates.size());
-			}
-
-			// Ensure blocks 901 to 1002 are not available in the archive
-			for (int i=901; i<=1002; i++) {
-				assertNull(repository.getBlockArchiveRepository().fromHeight(i));
-			}
-		}
-	}
-
-	@Test
-	public void testBulkArchiveAndPruneMultipleFiles() throws DataException, SQLException {
-		try (final Repository repository = RepositoryManager.getRepository()) {
-			HSQLDBRepository hsqldb = (HSQLDBRepository) repository;
-
-			// Deploy an AT so that we have AT state data
-			PrivateKeyAccount deployer = Common.getTestAccount(repository, "alice");
-			byte[] creationBytes = AtUtils.buildSimpleAT();
-			long fundingAmount = 1_00000000L;
-			AtUtils.doDeployAT(repository, deployer, creationBytes, fundingAmount);
-
-			// Mint some blocks so that we are able to archive them later
-			for (int i = 0; i < 1000; i++) {
-				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
-			}
-
-			// Assume 900 blocks are trimmed (this specifies the first untrimmed height)
-			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(901);
-			repository.getATRepository().setAtTrimHeight(901);
-
-			// Check the max archive height - this should be one less than the first untrimmed height
-			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
-			assertEquals(900, maximumArchiveHeight);
-
-			// Check the current archive height
-			assertEquals(0, repository.getBlockArchiveRepository().getBlockArchiveHeight());
-
-			// Write blocks 2-900 to the archive (using bulk method)
-			int fileSizeTarget = 42360; // Pre-calculated size of approx 90 blocks
-			assertTrue(HSQLDBDatabaseArchiving.buildBlockArchive(repository, fileSizeTarget));
-
-			// Ensure 10 archive files have been created
-			Path archivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive");
-			assertEquals(10, new File(archivePath.toString()).list().length);
-
-			// Check the files exist
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "2-90.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "91-179.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "180-268.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "269-357.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "358-446.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "447-535.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "536-624.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "625-713.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "714-802.dat")));
-			assertTrue(Files.exists(Paths.get(archivePath.toString(), "803-891.dat")));
-
-			// Ensure the block archive height has increased
-			// It won't be as high as 901, because blocks 892-901 were too small to reach the file size
-			// target of the 11th file
-			assertEquals(892, repository.getBlockArchiveRepository().getBlockArchiveHeight());
-
-			// Ensure the SQL repository contains blocks 2 and 891...
-			assertNotNull(repository.getBlockRepository().fromHeight(2));
-			assertNotNull(repository.getBlockRepository().fromHeight(891));
-
-			// Check the current prune heights
-			assertEquals(0, repository.getBlockRepository().getBlockPruneHeight());
-			assertEquals(0, repository.getATRepository().getAtPruneHeight());
-
-			// Prior to archiving or pruning, ensure blocks 2 to 1002 and their AT states are available in the db
-			for (int i=2; i<=1002; i++) {
-				assertNotNull(repository.getBlockRepository().fromHeight(i));
-				List<ATStateData> atStates = repository.getATRepository().getBlockATStatesAtHeight(i);
-				assertNotNull(atStates);
-				assertEquals(1, atStates.size());
-			}
-
-			// Prune all the archived blocks and AT states (using bulk method)
-			assertTrue(HSQLDBDatabasePruning.pruneBlocks(hsqldb));
-			assertTrue(HSQLDBDatabasePruning.pruneATStates(hsqldb));
-
-			// Ensure the current prune heights have increased
-			assertEquals(892, repository.getBlockRepository().getBlockPruneHeight());
-			assertEquals(892, repository.getATRepository().getAtPruneHeight());
-
-			// Now ensure the SQL repository is missing blocks 2 and 891...
-			assertNull(repository.getBlockRepository().fromHeight(2));
-			assertNull(repository.getBlockRepository().fromHeight(891));
-
-			// ... but it's not missing blocks 1 and 901 (we don't prune the genesis block)
-			assertNotNull(repository.getBlockRepository().fromHeight(1));
-			assertNotNull(repository.getBlockRepository().fromHeight(892));
-
-			// Validate the latest block height in the repository
-			assertEquals(1002, (int) repository.getBlockRepository().getLastBlock().getHeight());
-
-			// Ensure blocks 2-891 are all available in the archive
-			for (int i=2; i<=891; i++) {
-				assertNotNull(repository.getBlockArchiveRepository().fromHeight(i));
-			}
-
-			// Ensure blocks 2-891 are NOT available in the db
-			for (int i=2; i<=891; i++) {
-				assertNull(repository.getBlockRepository().fromHeight(i));
-			}
-
-			// Ensure blocks 892 to 1002 and their AT states are available in the db
-			for (int i=892; i<=1002; i++) {
-				assertNotNull(repository.getBlockRepository().fromHeight(i));
-				List<ATStateData> atStates = repository.getATRepository().getBlockATStatesAtHeight(i);
-				assertNotNull(atStates);
-				assertEquals(1, atStates.size());
-			}
-
-			// Ensure blocks 892 to 1002 are not available in the archive
-			for (int i=892; i<=1002; i++) {
-				assertNull(repository.getBlockArchiveRepository().fromHeight(i));
-			}
-		}
-	}
-
 	@Test
 	public void testTrimArchivePruneAndOrphan() throws DataException, InterruptedException, TransformationException, IOException {
 		try (final Repository repository = RepositoryManager.getRepository()) {

src/test/java/org/qortal/test/BlockArchiveV2Tests.java

@@ -0,0 +1,504 @@
+package org.qortal.test;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.reflect.FieldUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.qortal.account.PrivateKeyAccount;
+import org.qortal.controller.BlockMinter;
+import org.qortal.data.at.ATStateData;
+import org.qortal.data.block.BlockData;
+import org.qortal.data.transaction.TransactionData;
+import org.qortal.repository.*;
+import org.qortal.repository.hsqldb.HSQLDBRepository;
+import org.qortal.settings.Settings;
+import org.qortal.test.common.AtUtils;
+import org.qortal.test.common.BlockUtils;
+import org.qortal.test.common.Common;
+import org.qortal.transaction.DeployAtTransaction;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.TransformationException;
+import org.qortal.transform.block.BlockTransformation;
+import org.qortal.utils.BlockArchiveUtils;
+import org.qortal.utils.NTP;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.SQLException;
+import java.util.List;
+
+import static org.junit.Assert.*;
+
+public class BlockArchiveV2Tests extends Common {
+
+	@Before
+	public void beforeTest() throws DataException, IllegalAccessException {
+		Common.useSettings("test-settings-v2-block-archive.json");
+		NTP.setFixedOffset(Settings.getInstance().getTestNtpOffset());
+		this.deleteArchiveDirectory();
+
+		// Set default archive version to 2, so that archive builds in these tests use V2
+		FieldUtils.writeField(Settings.getInstance(), "defaultArchiveVersion", 2, true);
+	}
+
+	@After
+	public void afterTest() throws DataException {
+		this.deleteArchiveDirectory();
+	}
+
+
+	@Test
+	public void testWriter() throws DataException, InterruptedException, TransformationException, IOException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// Mint some blocks so that we are able to archive them later
+			for (int i = 0; i < 1000; i++) {
+				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
+			}
+
+			// 900 blocks are trimmed (this specifies the first untrimmed height)
+			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(901);
+			repository.getATRepository().setAtTrimHeight(901);
+
+			// Check the max archive height - this should be one less than the first untrimmed height
+			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+			assertEquals(900, maximumArchiveHeight);
+
+			// Write blocks 2-900 to the archive
+			BlockArchiveWriter writer = new BlockArchiveWriter(0, maximumArchiveHeight, repository);
+			writer.setShouldEnforceFileSizeTarget(false); // To avoid the need to pre-calculate file sizes
+			BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+			assertEquals(BlockArchiveWriter.BlockArchiveWriteResult.OK, result);
+
+			// Make sure that the archive contains the correct number of blocks
+			assertEquals(900 - 1, writer.getWrittenCount());
+
+			// Increment block archive height
+			repository.getBlockArchiveRepository().setBlockArchiveHeight(writer.getWrittenCount());
+			repository.saveChanges();
+			assertEquals(900 - 1, repository.getBlockArchiveRepository().getBlockArchiveHeight());
+
+			// Ensure the file exists
+			File outputFile = writer.getOutputPath().toFile();
+			assertTrue(outputFile.exists());
+		}
+	}
+
+	@Test
+	public void testWriterAndReader() throws DataException, InterruptedException, TransformationException, IOException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// Mint some blocks so that we are able to archive them later
+			for (int i = 0; i < 1000; i++) {
+				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
+			}
+
+			// 900 blocks are trimmed (this specifies the first untrimmed height)
+			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(901);
+			repository.getATRepository().setAtTrimHeight(901);
+
+			// Check the max archive height - this should be one less than the first untrimmed height
+			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+			assertEquals(900, maximumArchiveHeight);
+
+			// Write blocks 2-900 to the archive
+			BlockArchiveWriter writer = new BlockArchiveWriter(0, maximumArchiveHeight, repository);
+			writer.setShouldEnforceFileSizeTarget(false); // To avoid the need to pre-calculate file sizes
+			BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+			assertEquals(BlockArchiveWriter.BlockArchiveWriteResult.OK, result);
+
+			// Make sure that the archive contains the correct number of blocks
+			assertEquals(900 - 1, writer.getWrittenCount());
+
+			// Increment block archive height
+			repository.getBlockArchiveRepository().setBlockArchiveHeight(writer.getWrittenCount());
+			repository.saveChanges();
+			assertEquals(900 - 1, repository.getBlockArchiveRepository().getBlockArchiveHeight());
+
+			// Ensure the file exists
+			File outputFile = writer.getOutputPath().toFile();
+			assertTrue(outputFile.exists());
+
+			// Read block 2 from the archive
+			BlockArchiveReader reader = BlockArchiveReader.getInstance();
+			BlockTransformation block2Info = reader.fetchBlockAtHeight(2);
+			BlockData block2ArchiveData = block2Info.getBlockData();
+
+			// Read block 2 from the repository
+			BlockData block2RepositoryData = repository.getBlockRepository().fromHeight(2);
+
+			// Ensure the values match
+			assertEquals(block2ArchiveData.getHeight(), block2RepositoryData.getHeight());
+			assertArrayEquals(block2ArchiveData.getSignature(), block2RepositoryData.getSignature());
+
+			// Test some values in the archive
+			assertEquals(1, block2ArchiveData.getOnlineAccountsCount());
+
+			// Read block 900 from the archive
+			BlockTransformation block900Info = reader.fetchBlockAtHeight(900);
+			BlockData block900ArchiveData = block900Info.getBlockData();
+
+			// Read block 900 from the repository
+			BlockData block900RepositoryData = repository.getBlockRepository().fromHeight(900);
+
+			// Ensure the values match
+			assertEquals(block900ArchiveData.getHeight(), block900RepositoryData.getHeight());
+			assertArrayEquals(block900ArchiveData.getSignature(), block900RepositoryData.getSignature());
+
+			// Test some values in the archive
+			assertEquals(1, block900ArchiveData.getOnlineAccountsCount());
+
+		}
+	}
+
+	@Test
+	public void testArchivedAtStates() throws DataException, InterruptedException, TransformationException, IOException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// Deploy an AT so that we have AT state data
+			PrivateKeyAccount deployer = Common.getTestAccount(repository, "alice");
+			byte[] creationBytes = AtUtils.buildSimpleAT();
+			long fundingAmount = 1_00000000L;
+			DeployAtTransaction deployAtTransaction =  AtUtils.doDeployAT(repository, deployer, creationBytes, fundingAmount);
+			String atAddress = deployAtTransaction.getATAccount().getAddress();
+
+			// Mint some blocks so that we are able to archive them later
+			for (int i = 0; i < 1000; i++) {
+				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
+			}
+
+			// 9 blocks are trimmed (this specifies the first untrimmed height)
+			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(10);
+			repository.getATRepository().setAtTrimHeight(10);
+
+			// Check the max archive height
+			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+			assertEquals(9, maximumArchiveHeight);
+
+			// Write blocks 2-9 to the archive
+			BlockArchiveWriter writer = new BlockArchiveWriter(0, maximumArchiveHeight, repository);
+			writer.setShouldEnforceFileSizeTarget(false); // To avoid the need to pre-calculate file sizes
+			BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+			assertEquals(BlockArchiveWriter.BlockArchiveWriteResult.OK, result);
+
+			// Make sure that the archive contains the correct number of blocks
+			assertEquals(9 - 1, writer.getWrittenCount());
+
+			// Increment block archive height
+			repository.getBlockArchiveRepository().setBlockArchiveHeight(writer.getWrittenCount());
+			repository.saveChanges();
+			assertEquals(9 - 1, repository.getBlockArchiveRepository().getBlockArchiveHeight());
+
+			// Ensure the file exists
+			File outputFile = writer.getOutputPath().toFile();
+			assertTrue(outputFile.exists());
+
+			// Check blocks 3-9
+			for (Integer testHeight = 2; testHeight <= 9; testHeight++) {
+
+				// Read a block from the archive
+				BlockArchiveReader reader = BlockArchiveReader.getInstance();
+				BlockTransformation blockInfo = reader.fetchBlockAtHeight(testHeight);
+				BlockData archivedBlockData = blockInfo.getBlockData();
+				byte[] archivedAtStateHash = blockInfo.getAtStatesHash();
+				List<TransactionData> archivedTransactions = blockInfo.getTransactions();
+
+				// Read the same block from the repository
+				BlockData repositoryBlockData = repository.getBlockRepository().fromHeight(testHeight);
+				ATStateData repositoryAtStateData = repository.getATRepository().getATStateAtHeight(atAddress, testHeight);
+
+				// Ensure the repository has full AT state data
+				assertNotNull(repositoryAtStateData.getStateHash());
+				assertNotNull(repositoryAtStateData.getStateData());
+
+				// Check the archived AT state
+				if (testHeight == 2) {
+					assertEquals(1, archivedTransactions.size());
+					assertEquals(Transaction.TransactionType.DEPLOY_AT, archivedTransactions.get(0).getType());
+				}
+				else {
+					// Blocks 3+ shouldn't have any transactions
+					assertTrue(archivedTransactions.isEmpty());
+				}
+
+				// Ensure the archive has the AT states hash
+				assertNotNull(archivedAtStateHash);
+
+				// Also check the online accounts count and height
+				assertEquals(1, archivedBlockData.getOnlineAccountsCount());
+				assertEquals(testHeight, archivedBlockData.getHeight());
+
+				// Ensure the values match
+				assertEquals(archivedBlockData.getHeight(), repositoryBlockData.getHeight());
+				assertArrayEquals(archivedBlockData.getSignature(), repositoryBlockData.getSignature());
+				assertEquals(archivedBlockData.getOnlineAccountsCount(), repositoryBlockData.getOnlineAccountsCount());
+				assertArrayEquals(archivedBlockData.getMinterSignature(), repositoryBlockData.getMinterSignature());
+				assertEquals(archivedBlockData.getATCount(), repositoryBlockData.getATCount());
+				assertEquals(archivedBlockData.getOnlineAccountsCount(), repositoryBlockData.getOnlineAccountsCount());
+				assertArrayEquals(archivedBlockData.getReference(), repositoryBlockData.getReference());
+				assertEquals(archivedBlockData.getTimestamp(), repositoryBlockData.getTimestamp());
+				assertEquals(archivedBlockData.getATFees(), repositoryBlockData.getATFees());
+				assertEquals(archivedBlockData.getTotalFees(), repositoryBlockData.getTotalFees());
+				assertEquals(archivedBlockData.getTransactionCount(), repositoryBlockData.getTransactionCount());
+				assertArrayEquals(archivedBlockData.getTransactionsSignature(), repositoryBlockData.getTransactionsSignature());
+
+				// TODO: build atStatesHash and compare against value in archive
+			}
+
+			// Check block 10 (unarchived)
+			BlockArchiveReader reader = BlockArchiveReader.getInstance();
+			BlockTransformation blockInfo = reader.fetchBlockAtHeight(10);
+			assertNull(blockInfo);
+
+		}
+
+	}
+
+	@Test
+	public void testArchiveAndPrune() throws DataException, InterruptedException, TransformationException, IOException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// Deploy an AT so that we have AT state data
+			PrivateKeyAccount deployer = Common.getTestAccount(repository, "alice");
+			byte[] creationBytes = AtUtils.buildSimpleAT();
+			long fundingAmount = 1_00000000L;
+			AtUtils.doDeployAT(repository, deployer, creationBytes, fundingAmount);
+
+			// Mint some blocks so that we are able to archive them later
+			for (int i = 0; i < 1000; i++) {
+				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
+			}
+
+			// Assume 900 blocks are trimmed (this specifies the first untrimmed height)
+			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(901);
+			repository.getATRepository().setAtTrimHeight(901);
+
+			// Check the max archive height - this should be one less than the first untrimmed height
+			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+			assertEquals(900, maximumArchiveHeight);
+
+			// Write blocks 2-900 to the archive
+			BlockArchiveWriter writer = new BlockArchiveWriter(0, maximumArchiveHeight, repository);
+			writer.setShouldEnforceFileSizeTarget(false); // To avoid the need to pre-calculate file sizes
+			BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+			assertEquals(BlockArchiveWriter.BlockArchiveWriteResult.OK, result);
+
+			// Make sure that the archive contains the correct number of blocks
+			assertEquals(900 - 1, writer.getWrittenCount());
+
+			// Increment block archive height
+			repository.getBlockArchiveRepository().setBlockArchiveHeight(901);
+			repository.saveChanges();
+			assertEquals(901, repository.getBlockArchiveRepository().getBlockArchiveHeight());
+
+			// Ensure the file exists
+			File outputFile = writer.getOutputPath().toFile();
+			assertTrue(outputFile.exists());
+
+			// Ensure the SQL repository contains blocks 2 and 900...
+			assertNotNull(repository.getBlockRepository().fromHeight(2));
+			assertNotNull(repository.getBlockRepository().fromHeight(900));
+
+			// Prune all the archived blocks
+			int numBlocksPruned = repository.getBlockRepository().pruneBlocks(0, 900);
+			assertEquals(900-1, numBlocksPruned);
+			repository.getBlockRepository().setBlockPruneHeight(901);
+
+			// Prune the AT states for the archived blocks
+			repository.getATRepository().rebuildLatestAtStates(900);
+			repository.saveChanges();
+			int numATStatesPruned = repository.getATRepository().pruneAtStates(0, 900);
+			assertEquals(900-2, numATStatesPruned); // Minus 1 for genesis block, and another for the latest AT state
+			repository.getATRepository().setAtPruneHeight(901);
+
+			// Now ensure the SQL repository is missing blocks 2 and 900...
+			assertNull(repository.getBlockRepository().fromHeight(2));
+			assertNull(repository.getBlockRepository().fromHeight(900));
+
+			// ... but it's not missing blocks 1 and 901 (we don't prune the genesis block)
+			assertNotNull(repository.getBlockRepository().fromHeight(1));
+			assertNotNull(repository.getBlockRepository().fromHeight(901));
+
+			// Validate the latest block height in the repository
+			assertEquals(1002, (int) repository.getBlockRepository().getLastBlock().getHeight());
+
+		}
+	}
+
+	@Test
+	public void testTrimArchivePruneAndOrphan() throws DataException, InterruptedException, TransformationException, IOException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+
+			// Deploy an AT so that we have AT state data
+			PrivateKeyAccount deployer = Common.getTestAccount(repository, "alice");
+			byte[] creationBytes = AtUtils.buildSimpleAT();
+			long fundingAmount = 1_00000000L;
+			AtUtils.doDeployAT(repository, deployer, creationBytes, fundingAmount);
+
+			// Mint some blocks so that we are able to archive them later
+			for (int i = 0; i < 1000; i++) {
+				BlockMinter.mintTestingBlock(repository, Common.getTestAccount(repository, "alice-reward-share"));
+			}
+
+			// Make sure that block 500 has full AT state data and data hash
+			List<ATStateData> block500AtStatesData = repository.getATRepository().getBlockATStatesAtHeight(500);
+			ATStateData atStatesData = repository.getATRepository().getATStateAtHeight(block500AtStatesData.get(0).getATAddress(), 500);
+			assertNotNull(atStatesData.getStateHash());
+			assertNotNull(atStatesData.getStateData());
+
+			// Trim the first 500 blocks
+			repository.getBlockRepository().trimOldOnlineAccountsSignatures(0, 500);
+			repository.getBlockRepository().setOnlineAccountsSignaturesTrimHeight(501);
+			repository.getATRepository().rebuildLatestAtStates(500);
+			repository.getATRepository().trimAtStates(0, 500, 1000);
+			repository.getATRepository().setAtTrimHeight(501);
+
+			// Now block 499 should only have the AT state data hash
+			List<ATStateData> block499AtStatesData = repository.getATRepository().getBlockATStatesAtHeight(499);
+			atStatesData = repository.getATRepository().getATStateAtHeight(block499AtStatesData.get(0).getATAddress(), 499);
+			assertNotNull(atStatesData.getStateHash());
+			assertNull(atStatesData.getStateData());
+
+			// ... but block 500 should have the full data (due to being retained as the "latest" AT state in the trimmed range
+			block500AtStatesData = repository.getATRepository().getBlockATStatesAtHeight(500);
+			atStatesData = repository.getATRepository().getATStateAtHeight(block500AtStatesData.get(0).getATAddress(), 500);
+			assertNotNull(atStatesData.getStateHash());
+			assertNotNull(atStatesData.getStateData());
+
+			// ... and block 501 should also have the full data
+			List<ATStateData> block501AtStatesData = repository.getATRepository().getBlockATStatesAtHeight(501);
+			atStatesData = repository.getATRepository().getATStateAtHeight(block501AtStatesData.get(0).getATAddress(), 501);
+			assertNotNull(atStatesData.getStateHash());
+			assertNotNull(atStatesData.getStateData());
+
+			// Check the max archive height - this should be one less than the first untrimmed height
+			final int maximumArchiveHeight = BlockArchiveWriter.getMaxArchiveHeight(repository);
+			assertEquals(500, maximumArchiveHeight);
+
+			BlockData block3DataPreArchive = repository.getBlockRepository().fromHeight(3);
+
+			// Write blocks 2-500 to the archive
+			BlockArchiveWriter writer = new BlockArchiveWriter(0, maximumArchiveHeight, repository);
+			writer.setShouldEnforceFileSizeTarget(false); // To avoid the need to pre-calculate file sizes
+			BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
+			assertEquals(BlockArchiveWriter.BlockArchiveWriteResult.OK, result);
+
+			// Make sure that the archive contains the correct number of blocks
+			assertEquals(500 - 1, writer.getWrittenCount()); // -1 for the genesis block
+
+			// Increment block archive height
+			repository.getBlockArchiveRepository().setBlockArchiveHeight(writer.getWrittenCount());
+			repository.saveChanges();
+			assertEquals(500 - 1, repository.getBlockArchiveRepository().getBlockArchiveHeight());
+
+			// Ensure the file exists
+			File outputFile = writer.getOutputPath().toFile();
+			assertTrue(outputFile.exists());
+
+			// Ensure the SQL repository contains blocks 2 and 500...
+			assertNotNull(repository.getBlockRepository().fromHeight(2));
+			assertNotNull(repository.getBlockRepository().fromHeight(500));
+
+			// Prune all the archived blocks
+			int numBlocksPruned = repository.getBlockRepository().pruneBlocks(0, 500);
+			assertEquals(500-1, numBlocksPruned);
+			repository.getBlockRepository().setBlockPruneHeight(501);
+
+			// Prune the AT states for the archived blocks
+			repository.getATRepository().rebuildLatestAtStates(500);
+			repository.saveChanges();
+			int numATStatesPruned = repository.getATRepository().pruneAtStates(2, 500);
+			assertEquals(498, numATStatesPruned); // Minus 1 for genesis block, and another for the latest AT state
+			repository.getATRepository().setAtPruneHeight(501);
+
+			// Now ensure the SQL repository is missing blocks 2 and 500...
+			assertNull(repository.getBlockRepository().fromHeight(2));
+			assertNull(repository.getBlockRepository().fromHeight(500));
+
+			// ... but it's not missing blocks 1 and 501 (we don't prune the genesis block)
+			assertNotNull(repository.getBlockRepository().fromHeight(1));
+			assertNotNull(repository.getBlockRepository().fromHeight(501));
+
+			// Validate the latest block height in the repository
+			assertEquals(1002, (int) repository.getBlockRepository().getLastBlock().getHeight());
+
+			// Now orphan some unarchived blocks.
+			BlockUtils.orphanBlocks(repository, 500);
+			assertEquals(502, (int) repository.getBlockRepository().getLastBlock().getHeight());
+
+			// We're close to the lower limit of the SQL database now, so
+			// we need to import some blocks from the archive
+			BlockArchiveUtils.importFromArchive(401, 500, repository);
+
+			// Ensure the SQL repository now contains block 401 but not 400...
+			assertNotNull(repository.getBlockRepository().fromHeight(401));
+			assertNull(repository.getBlockRepository().fromHeight(400));
+
+			// Import the remaining 399 blocks
+			BlockArchiveUtils.importFromArchive(2, 400, repository);
+
+			// Verify that block 3 matches the original
+			BlockData block3DataPostArchive = repository.getBlockRepository().fromHeight(3);
+			assertArrayEquals(block3DataPreArchive.getSignature(), block3DataPostArchive.getSignature());
+			assertEquals(block3DataPreArchive.getHeight(), block3DataPostArchive.getHeight());
+
+			// Orphan 2 more block, which should be the last one that is possible to be orphaned
+			// TODO: figure out why this is 1 block more than in the equivalent block archive V1 test
+			BlockUtils.orphanBlocks(repository, 2);
+
+			// Orphan another block, which should fail
+			Exception exception = null;
+			try {
+				BlockUtils.orphanBlocks(repository, 1);
+			} catch (DataException e) {
+				exception = e;
+			}
+
+			// Ensure that a DataException is thrown because there is no more AT states data available
+			assertNotNull(exception);
+			assertEquals(DataException.class, exception.getClass());
+
+			// FUTURE: we may be able to retain unique AT states when trimming, to avoid this exception
+			// and allow orphaning back through blocks with trimmed AT states.
+
+		}
+	}
+
+
+	/**
+	 * Many nodes are missing an ATStatesHeightIndex due to an earlier bug
+	 * In these cases we disable archiving and pruning as this index is a
+	 * very essential component in these processes.
+	 */
+	@Test
+	public void testMissingAtStatesHeightIndex() throws DataException, SQLException {
+		try (final HSQLDBRepository repository = (HSQLDBRepository) RepositoryManager.getRepository()) {
+
+			// Firstly check that we're able to prune or archive when the index exists
+			assertTrue(repository.getATRepository().hasAtStatesHeightIndex());
+			assertTrue(RepositoryManager.canArchiveOrPrune());
+
+			// Delete the index
+			repository.prepareStatement("DROP INDEX ATSTATESHEIGHTINDEX").execute();
+
+			// Ensure check that we're unable to prune or archive when the index doesn't exist
+			assertFalse(repository.getATRepository().hasAtStatesHeightIndex());
+			assertFalse(RepositoryManager.canArchiveOrPrune());
+		}
+	}
+
+
+	private void deleteArchiveDirectory() {
+		// Delete archive directory if exists
+		Path archivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive").toAbsolutePath();
+		try {
+			FileUtils.deleteDirectory(archivePath.toFile());
+		} catch (IOException e) {
+
+		}
+	}
+
+}

src/test/java/org/qortal/test/RepositoryTests.java

@@ -9,6 +9,7 @@ import org.qortal.crosschain.BitcoinACCTv1;
 import org.qortal.crypto.Crypto;
 import org.qortal.data.account.AccountBalanceData;
 import org.qortal.data.account.AccountData;
+import org.qortal.data.chat.ChatMessage;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
@@ -417,7 +418,7 @@ public class RepositoryTests extends Common {
 		try (final HSQLDBRepository hsqldb = (HSQLDBRepository) RepositoryManager.getRepository()) {
 			String address = Crypto.toAddress(new byte[32]);
 
-			hsqldb.getChatRepository().getActiveChats(address);
+			hsqldb.getChatRepository().getActiveChats(address, ChatMessage.Encoding.BASE58);
 		} catch (DataException e) {
 			fail("HSQLDB bug #1580");
 		}

src/test/java/org/qortal/test/api/AdminApiTests.java

@@ -5,7 +5,7 @@ import static org.junit.Assert.*;
 import org.apache.commons.lang3.reflect.FieldUtils;
 import org.junit.Before;
 import org.junit.Test;
-import org.qortal.api.resource.AdminResource;
+import org.qortal.api.restricted.resource.AdminResource;
 import org.qortal.repository.DataException;
 import org.qortal.settings.Settings;
 import org.qortal.test.common.ApiCommon;

src/test/java/org/qortal/test/arbitrary/ArbitraryDataStoragePolicyTests.java

@@ -246,7 +246,7 @@ public class ArbitraryDataStoragePolicyTests extends Common {
         Path path = Paths.get("src/test/resources/arbitrary/demo1");
 
         ArbitraryDataTransactionBuilder txnBuilder = new ArbitraryDataTransactionBuilder(
-                repository, publicKey58, path, name, Method.PUT, Service.ARBITRARY_DATA, null,
+                repository, publicKey58, 0L, path, name, Method.PUT, Service.ARBITRARY_DATA, null,
                 null, null, null, null);
 
         txnBuilder.build();

src/test/java/org/qortal/test/arbitrary/ArbitraryDataTests.java

@@ -111,8 +111,8 @@ public class ArbitraryDataTests extends Common {
                 fail("Creating transaction should fail due to nonexistent PUT transaction");
 
             } catch (DataException expectedException) {
-                assertEquals(String.format("Couldn't find PUT transaction for " +
-                        "name %s, service %s and identifier ", name, service), expectedException.getMessage());
+                assertTrue(expectedException.getMessage().contains(String.format("Couldn't find PUT transaction for " +
+                        "name %s, service %s and identifier ", name, service)));
             }
 
         }
@@ -358,7 +358,7 @@ public class ArbitraryDataTests extends Common {
             byte[] path1FileDigest = Crypto.digest(path1.toFile());
             ArbitraryDataDigest path1DirectoryDigest = new ArbitraryDataDigest(path1.getParent());
             path1DirectoryDigest.compute();
-            ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, Method.PUT, service, alice);
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, Method.PUT, service, alice);
 
             // Now build the latest data state for this name
             ArbitraryDataReader arbitraryDataReader1 = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier);

src/test/java/org/qortal/test/arbitrary/ArbitraryEncryptionTests.java

@@ -0,0 +1,135 @@
+package org.qortal.test.arbitrary;
+
+import org.junit.Before;
+import org.junit.Test;
+import org.qortal.arbitrary.ArbitraryDataDigest;
+import org.qortal.crypto.AES;
+import org.qortal.crypto.Crypto;
+import org.qortal.repository.DataException;
+import org.qortal.test.common.Common;
+import org.qortal.utils.ZipUtils;
+
+import javax.crypto.BadPaddingException;
+import javax.crypto.IllegalBlockSizeException;
+import javax.crypto.NoSuchPaddingException;
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardOpenOption;
+import java.security.InvalidAlgorithmParameterException;
+import java.security.InvalidKeyException;
+import java.security.NoSuchAlgorithmException;
+import java.util.Arrays;
+import java.util.Random;
+
+import static org.junit.Assert.*;
+
+public class ArbitraryEncryptionTests extends Common {
+
+    @Before
+    public void beforeTest() throws DataException {
+        Common.useDefaultSettings();
+    }
+
+    @Test
+    public void testEncryption() throws IOException, NoSuchAlgorithmException, InvalidAlgorithmParameterException, NoSuchPaddingException, IllegalBlockSizeException, BadPaddingException, InvalidKeyException {
+        String enclosingFolderName = "data";
+        Path inputFilePath = Files.createTempFile("inputFile", null);
+        Path outputDirectory = Files.createTempDirectory("outputDirectory");
+        Path outputFilePath = Paths.get(outputDirectory.toString(), enclosingFolderName);
+        inputFilePath.toFile().deleteOnExit();
+        outputDirectory.toFile().deleteOnExit();
+
+        // Write random data to the input file
+        byte[] data = new byte[10];
+        new Random().nextBytes(data);
+        Files.write(inputFilePath, data, StandardOpenOption.CREATE);
+
+        assertTrue(Files.exists(inputFilePath));
+        assertFalse(Files.exists(outputFilePath));
+
+        // Encrypt...
+        String algorithm = "AES/CBC/PKCS5Padding";
+        SecretKey aesKey = AES.generateKey(256);
+        AES.encryptFile(algorithm, aesKey, inputFilePath.toString(), outputFilePath.toString());
+
+        assertTrue(Files.exists(inputFilePath));
+        assertTrue(Files.exists(outputFilePath));
+
+        // Ensure encrypted file's hash differs from the original
+        assertFalse(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(outputFilePath.toFile())));
+
+        // Create paths for decrypting
+        Path decryptedDirectory = Files.createTempDirectory("decryptedDirectory");
+        Path decryptedFile = Paths.get(decryptedDirectory.toString(), enclosingFolderName, inputFilePath.getFileName().toString());
+        decryptedDirectory.toFile().deleteOnExit();
+        assertFalse(Files.exists(decryptedFile));
+
+        // Now decrypt...
+        AES.decryptFile(algorithm, aesKey, outputFilePath.toString(), decryptedFile.toString());
+
+        // Ensure resulting file exists
+        assertTrue(Files.exists(decryptedFile));
+
+        // And make sure it matches the original input file
+        assertTrue(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(decryptedFile.toFile())));
+    }
+
+    @Test
+    public void testEncryptionSizeOverhead() throws IOException, NoSuchAlgorithmException, InvalidAlgorithmParameterException, NoSuchPaddingException, IllegalBlockSizeException, BadPaddingException, InvalidKeyException {
+        for (int size = 1; size < 256; size++) {
+            String enclosingFolderName = "data";
+            Path inputFilePath = Files.createTempFile("inputFile", null);
+            Path outputDirectory = Files.createTempDirectory("outputDirectory");
+            Path outputFilePath = Paths.get(outputDirectory.toString(), enclosingFolderName);
+            inputFilePath.toFile().deleteOnExit();
+            outputDirectory.toFile().deleteOnExit();
+
+            // Write random data to the input file
+            byte[] data = new byte[size];
+            new Random().nextBytes(data);
+            Files.write(inputFilePath, data, StandardOpenOption.CREATE);
+
+            assertTrue(Files.exists(inputFilePath));
+            assertFalse(Files.exists(outputFilePath));
+
+            // Ensure input file is the same size as the data
+            assertEquals(size, inputFilePath.toFile().length());
+
+            // Encrypt...
+            String algorithm = "AES/CBC/PKCS5Padding";
+            SecretKey aesKey = AES.generateKey(256);
+            AES.encryptFile(algorithm, aesKey, inputFilePath.toString(), outputFilePath.toString());
+
+            assertTrue(Files.exists(inputFilePath));
+            assertTrue(Files.exists(outputFilePath));
+
+            final long expectedSize = AES.getEncryptedFileSize(inputFilePath.toFile().length());
+            System.out.println(String.format("Plaintext size: %d bytes, Ciphertext size: %d bytes", inputFilePath.toFile().length(), outputFilePath.toFile().length()));
+
+            // Ensure encryption added a fixed amount of space to the output file
+            assertEquals(expectedSize, outputFilePath.toFile().length());
+
+            // Ensure encrypted file's hash differs from the original
+            assertFalse(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(outputFilePath.toFile())));
+
+            // Create paths for decrypting
+            Path decryptedDirectory = Files.createTempDirectory("decryptedDirectory");
+            Path decryptedFile = Paths.get(decryptedDirectory.toString(), enclosingFolderName, inputFilePath.getFileName().toString());
+            decryptedDirectory.toFile().deleteOnExit();
+            assertFalse(Files.exists(decryptedFile));
+
+            // Now decrypt...
+            AES.decryptFile(algorithm, aesKey, outputFilePath.toString(), decryptedFile.toString());
+
+            // Ensure resulting file exists
+            assertTrue(Files.exists(decryptedFile));
+
+            // And make sure it matches the original input file
+            assertTrue(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(decryptedFile.toFile())));
+        }
+    }
+
+}

src/test/java/org/qortal/test/arbitrary/ArbitraryServiceTests.java

@@ -22,6 +22,8 @@ import org.qortal.test.common.transaction.TestTransaction;
 import org.qortal.transaction.RegisterNameTransaction;
 import org.qortal.utils.Base58;
 
+import java.io.BufferedWriter;
+import java.io.FileWriter;
 import java.io.IOException;
 import java.nio.file.Files;
 import java.nio.file.Path;
@@ -317,17 +319,15 @@ public class ArbitraryServiceTests extends Common {
         // Write the data to several files in a temp path
         Path path = Files.createTempDirectory("testValidateMultiLayerQChatAttachment");
         path.toFile().deleteOnExit();
-        Files.write(Paths.get(path.toString(), "file1.txt"), data, StandardOpenOption.CREATE);
 
         Path subdirectory = Paths.get(path.toString(), "subdirectory");
         Files.createDirectories(subdirectory);
-        Files.write(Paths.get(subdirectory.toString(), "file2.txt"), data, StandardOpenOption.CREATE);
-        Files.write(Paths.get(subdirectory.toString(), "file3.txt"), data, StandardOpenOption.CREATE);
+        Files.write(Paths.get(subdirectory.toString(), "file.txt"), data, StandardOpenOption.CREATE);
 
         Service service = Service.QCHAT_ATTACHMENT;
         assertTrue(service.isValidationRequired());
 
-        assertEquals(ValidationResult.DIRECTORIES_NOT_ALLOWED, service.validate(path));
+        assertEquals(ValidationResult.INVALID_FILE_COUNT, service.validate(path));
     }
 
     @Test
@@ -389,4 +389,51 @@ public class ArbitraryServiceTests extends Common {
         }
     }
 
+    @Test
+    public void testValidateValidJson() throws IOException {
+        String invalidJsonString = "{\"test\": true, \"test2\": \"valid\"}";
+
+        // Write the data a single file in a temp path
+        Path path = Files.createTempDirectory("testValidateValidJson");
+        Path filePath = Paths.get(path.toString(), "test.json");
+        filePath.toFile().deleteOnExit();
+
+        BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile()));
+        writer.write(invalidJsonString);
+        writer.close();
+
+        Service service = Service.JSON;
+        assertTrue(service.isValidationRequired());
+
+        assertEquals(ValidationResult.OK, service.validate(filePath));
+    }
+    @Test
+    public void testValidateInvalidJson() throws IOException {
+        String invalidJsonString = "{\"test\": true, \"test2\": invalid}";
+
+        // Write the data a single file in a temp path
+        Path path = Files.createTempDirectory("testValidateInvalidJson");
+        Path filePath = Paths.get(path.toString(), "test.json");
+        filePath.toFile().deleteOnExit();
+
+        BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile()));
+        writer.write(invalidJsonString);
+        writer.close();
+
+        Service service = Service.JSON;
+        assertTrue(service.isValidationRequired());
+
+        assertEquals(ValidationResult.INVALID_CONTENT, service.validate(filePath));
+    }
+
+    @Test
+    public void testValidateEmptyJson() throws IOException {
+        Path path = Files.createTempDirectory("testValidateEmptyJson");
+
+        Service service = Service.JSON;
+        assertTrue(service.isValidationRequired());
+
+        assertEquals(ValidationResult.INVALID_FILE_COUNT, service.validate(path));
+    }
+
 }

src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionMetadataTests.java

@@ -91,8 +91,8 @@ public class ArbitraryTransactionMetadataTests extends Common {
             String name = "TEST"; // Can be anything for this test
             String identifier = null; // Not used for this test
             Service service = Service.ARBITRARY_DATA;
-            int chunkSize = 1000;
-            int dataLength = 10; // Actual data length will be longer due to encryption
+            int chunkSize = 10000;
+            int dataLength = 1000; // Actual data length will be longer due to encryption
 
             String title = "Test title";
             String description = "Test description";
@@ -107,7 +107,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             // Create PUT transaction
             Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
             ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
-                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
                     title, description, tags, category);
 
             // Check the chunk count is correct
@@ -118,6 +118,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             assertEquals(description, arbitraryDataFile.getMetadata().getDescription());
             assertEquals(tags, arbitraryDataFile.getMetadata().getTags());
             assertEquals(category, arbitraryDataFile.getMetadata().getCategory());
+            assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType());
 
             // Now build the latest data state for this name
             ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier);
@@ -141,8 +142,8 @@ public class ArbitraryTransactionMetadataTests extends Common {
             String name = "TEST"; // Can be anything for this test
             String identifier = null; // Not used for this test
             Service service = Service.ARBITRARY_DATA;
-            int chunkSize = 1000;
-            int dataLength = 10; // Actual data length will be longer due to encryption
+            int chunkSize = 10000;
+            int dataLength = 1000; // Actual data length will be longer due to encryption
 
             String title = "Test title";
             String description = "Test description";
@@ -157,7 +158,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             // Create PUT transaction
             Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
             ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
-                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
                     title, description, tags, category);
 
             // Check the chunk count is correct
@@ -168,6 +169,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             assertEquals(description, arbitraryDataFile.getMetadata().getDescription());
             assertEquals(tags, arbitraryDataFile.getMetadata().getTags());
             assertEquals(category, arbitraryDataFile.getMetadata().getCategory());
+            assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType());
 
             // Delete the file, to simulate that it hasn't been fetched from the network yet
             arbitraryDataFile.delete();
@@ -219,7 +221,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             // Create PUT transaction
             Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
             ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
-                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
                     title, description, tags, category);
 
             // Check the chunk count is correct
@@ -230,6 +232,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             assertEquals(description, arbitraryDataFile.getMetadata().getDescription());
             assertEquals(tags, arbitraryDataFile.getMetadata().getTags());
             assertEquals(category, arbitraryDataFile.getMetadata().getCategory());
+            assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType());
 
             // Now build the latest data state for this name
             ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier);
@@ -262,7 +265,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             Category category = Category.CRYPTOCURRENCY;
 
             String expectedTitle = "title Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat "; // 80 chars
-            String expectedDescription = "description Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat pretium massa, non pulvinar mi pretium id. Ut gravida sapien vitae dui posuere tincidunt. Quisque in nibh est. Curabitur at blandit nunc, id aliquet neque. Nulla condimentum eget dolor a egestas. Vestibulum vel tincidunt ex. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Cras congue lacus in risus mattis suscipit. Quisque nisl eros, facilisis a lorem quis, vehicula biben"; // 500 chars
+            String expectedDescription = "description Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat pretium massa, non pulvinar mi pretium id. Ut gravida sapien vitae dui posuere tincidunt. Quisque in nibh est. Curabitur at blandit nunc, id aliquet neque"; // 240 chars
             List<String> expectedTags = Arrays.asList("tag 1", "tag 2", "tag 4", "tag 5", "tag 6");
 
             // Register the name to Alice
@@ -273,7 +276,7 @@ public class ArbitraryTransactionMetadataTests extends Common {
             // Create PUT transaction
             Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
             ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
-                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
                     title, description, tags, category);
 
             // Check the metadata is correct
@@ -318,9 +321,11 @@ public class ArbitraryTransactionMetadataTests extends Common {
             assertTrue(resourceMetadata.getFiles().contains("file.txt"));
 
             // Ensure it's not returned when specified to be excluded
-            // The entire object will be null because there is no metadata
             ArbitraryResourceMetadata resourceMetadataSimple = ArbitraryResourceMetadata.fromTransactionMetadata(arbitraryDataFile.getMetadata(), false);
-            assertNull(resourceMetadataSimple);
+            assertNull(resourceMetadataSimple.getFiles());
+
+            // Single-file resources should have a MIME type
+            assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType());
         }
     }
 
@@ -369,6 +374,9 @@ public class ArbitraryTransactionMetadataTests extends Common {
             // The entire object will be null because there is no metadata
             ArbitraryResourceMetadata resourceMetadataSimple = ArbitraryResourceMetadata.fromTransactionMetadata(arbitraryDataFile.getMetadata(), false);
             assertNull(resourceMetadataSimple);
+
+            // Multi-file resources won't have a MIME type
+            assertEquals(null, arbitraryDataFile.getMetadata().getMimeType());
         }
     }
 

src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionTests.java

@@ -5,12 +5,19 @@ import org.junit.Before;
 import org.junit.Test;
 import org.qortal.account.PrivateKeyAccount;
 import org.qortal.arbitrary.ArbitraryDataFile;
+import org.qortal.arbitrary.ArbitraryDataReader;
+import org.qortal.arbitrary.ArbitraryDataTransactionBuilder;
 import org.qortal.arbitrary.exception.MissingDataException;
+import org.qortal.arbitrary.misc.Category;
 import org.qortal.arbitrary.misc.Service;
 import org.qortal.controller.arbitrary.ArbitraryDataManager;
+import org.qortal.crypto.Crypto;
+import org.qortal.data.PaymentData;
 import org.qortal.data.transaction.ArbitraryTransactionData;
+import org.qortal.data.transaction.BaseTransactionData;
 import org.qortal.data.transaction.RegisterNameTransactionData;
 import org.qortal.data.transaction.TransactionData;
+import org.qortal.group.Group;
 import org.qortal.repository.DataException;
 import org.qortal.repository.Repository;
 import org.qortal.repository.RepositoryManager;
@@ -20,11 +27,19 @@ import org.qortal.test.common.TransactionUtils;
 import org.qortal.test.common.transaction.TestTransaction;
 import org.qortal.transaction.ArbitraryTransaction;
 import org.qortal.transaction.RegisterNameTransaction;
+import org.qortal.transaction.Transaction;
+import org.qortal.transform.transaction.TransactionTransformer;
 import org.qortal.utils.Base58;
 import org.qortal.utils.NTP;
 
+import java.io.File;
 import java.io.IOException;
 import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
 
 import static org.junit.Assert.*;
 
@@ -36,7 +51,7 @@ public class ArbitraryTransactionTests extends Common {
     }
 
     @Test
-    public void testDifficultyTooLow() throws IllegalAccessException, DataException, IOException, MissingDataException {
+    public void testDifficultyTooLow() throws IllegalAccessException, DataException, IOException {
         try (final Repository repository = RepositoryManager.getRepository()) {
             PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
             String publicKey58 = Base58.encode(alice.getPublicKey());
@@ -78,7 +93,555 @@ public class ArbitraryTransactionTests extends Common {
             assertTrue(transaction.isSignatureValid());
 
         }
-
     }
 
+    @Test
+    public void testNonceAndFee() throws IllegalAccessException, DataException, IOException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 10000000; // sufficient
+            boolean computeNonce = true;
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+
+            // Check that nonce validation succeeds
+            byte[] signature = arbitraryDataFile.getSignature();
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            ArbitraryTransaction transaction = new ArbitraryTransaction(repository, transactionData);
+            assertTrue(transaction.isSignatureValid());
+
+            // Increase difficulty to 15
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 15, true);
+
+            // Make sure that nonce validation still succeeds, as the fee has allowed us to avoid including a nonce
+            assertTrue(transaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testNonceAndLowFee() throws IllegalAccessException, DataException, IOException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee that is too low
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 9999999; // insufficient
+            boolean computeNonce = true;
+            boolean insufficientFeeDetected = false;
+            try {
+                ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+            }
+            catch (DataException e) {
+                if (e.getMessage().contains("INSUFFICIENT_FEE")) {
+                    insufficientFeeDetected = true;
+                }
+            }
+
+            // Transaction should be invalid due to an insufficient fee
+            assertTrue(insufficientFeeDetected);
+        }
+    }
+
+    @Test
+    public void testFeeNoNonce() throws IllegalAccessException, DataException, IOException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 10000000; // sufficient
+            boolean computeNonce = false;
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+
+            // Check that nonce validation succeeds, even though it wasn't computed. This is because we have included a sufficient fee.
+            byte[] signature = arbitraryDataFile.getSignature();
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            ArbitraryTransaction transaction = new ArbitraryTransaction(repository, transactionData);
+            assertTrue(transaction.isSignatureValid());
+
+            // Increase difficulty to 15
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 15, true);
+
+            // Make sure that nonce validation still succeeds, as the fee has allowed us to avoid including a nonce
+            assertTrue(transaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testLowFeeNoNonce() throws IllegalAccessException, DataException, IOException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee that is too low. Also, don't compute a nonce.
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 9999999; // insufficient
+
+            ArbitraryDataTransactionBuilder txnBuilder = new ArbitraryDataTransactionBuilder(
+                    repository, publicKey58, fee, path1, name, ArbitraryTransactionData.Method.PUT, service, identifier, null, null, null, null);
+
+            txnBuilder.setChunkSize(chunkSize);
+            txnBuilder.build();
+            ArbitraryTransactionData transactionData = txnBuilder.getArbitraryTransactionData();
+            Transaction.ValidationResult result = TransactionUtils.signAndImport(repository, transactionData, alice);
+
+            // Transaction should be invalid due to an insufficient fee
+            assertEquals(Transaction.ValidationResult.INSUFFICIENT_FEE, result);
+        }
+    }
+
+    @Test
+    public void testZeroFeeNoNonce() throws IllegalAccessException, DataException, IOException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee that is too low. Also, don't compute a nonce.
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 0L;
+
+            ArbitraryDataTransactionBuilder txnBuilder = new ArbitraryDataTransactionBuilder(
+                    repository, publicKey58, fee, path1, name, ArbitraryTransactionData.Method.PUT, service, identifier, null, null, null, null);
+
+            txnBuilder.setChunkSize(chunkSize);
+            txnBuilder.build();
+            ArbitraryTransactionData transactionData = txnBuilder.getArbitraryTransactionData();
+            ArbitraryTransaction arbitraryTransaction = new ArbitraryTransaction(repository, transactionData);
+
+            // Transaction should be invalid
+            assertFalse(arbitraryTransaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testNonceAndFeeBeforeFeatureTrigger() throws IllegalAccessException, DataException, IOException {
+        // Use v2-minting settings, as these are pre-feature-trigger
+        Common.useSettings("test-settings-v2-minting.json");
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 10000000; // sufficient
+            boolean computeNonce = true;
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+
+            // Check that nonce validation succeeds
+            byte[] signature = arbitraryDataFile.getSignature();
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            ArbitraryTransaction transaction = new ArbitraryTransaction(repository, transactionData);
+            assertTrue(transaction.isSignatureValid());
+
+            // Increase difficulty to 15
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 15, true);
+
+            // Make sure the nonce validation fails, as we aren't allowing a fee to replace a nonce yet.
+            // Note: there is a very tiny chance this could succeed due to being extremely lucky
+            // and finding a high difficulty nonce in the first couple of cycles. It will be rare
+            // enough that we shouldn't need to account for it.
+            assertFalse(transaction.isSignatureValid());
+
+            // Reduce difficulty back to 1, to double check
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+            assertTrue(transaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testNonceAndInsufficientFeeBeforeFeatureTrigger() throws IllegalAccessException, DataException, IOException {
+        // Use v2-minting settings, as these are pre-feature-trigger
+        Common.useSettings("test-settings-v2-minting.json");
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 9999999; // insufficient
+            boolean computeNonce = true;
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+
+            // Check that nonce validation succeeds
+            byte[] signature = arbitraryDataFile.getSignature();
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            ArbitraryTransaction transaction = new ArbitraryTransaction(repository, transactionData);
+            assertTrue(transaction.isSignatureValid());
+
+            // The transaction should be valid because we don't care about the fee (before the feature trigger)
+            assertEquals(Transaction.ValidationResult.OK, transaction.isValidUnconfirmed());
+
+            // Increase difficulty to 15
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 15, true);
+
+            // Make sure the nonce validation fails, as we aren't allowing a fee to replace a nonce yet (and it was insufficient anyway)
+            // Note: there is a very tiny chance this could succeed due to being extremely lucky
+            // and finding a high difficulty nonce in the first couple of cycles. It will be rare
+            // enough that we shouldn't need to account for it.
+            assertFalse(transaction.isSignatureValid());
+
+            // Reduce difficulty back to 1, to double check
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+            assertTrue(transaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testNonceAndZeroFeeBeforeFeatureTrigger() throws IllegalAccessException, DataException, IOException {
+        // Use v2-minting settings, as these are pre-feature-trigger
+        Common.useSettings("test-settings-v2-minting.json");
+
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 100;
+            int dataLength = 900; // Actual data length will be longer due to encryption
+
+            // Register the name to Alice
+            RegisterNameTransactionData registerNameTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            registerNameTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(registerNameTransactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, registerNameTransactionData, alice);
+
+            // Set difficulty to 1
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            // Create PUT transaction, with a fee
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength);
+            long fee = 0L;
+            boolean computeNonce = true;
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, fee, computeNonce, null, null, null, null);
+
+            // Check that nonce validation succeeds
+            byte[] signature = arbitraryDataFile.getSignature();
+            TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
+            ArbitraryTransaction transaction = new ArbitraryTransaction(repository, transactionData);
+            assertTrue(transaction.isSignatureValid());
+
+            // The transaction should be valid because we don't care about the fee (before the feature trigger)
+            assertEquals(Transaction.ValidationResult.OK, transaction.isValidUnconfirmed());
+
+            // Increase difficulty to 15
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 15, true);
+
+            // Make sure the nonce validation fails, as we aren't allowing a fee to replace a nonce yet (and it was insufficient anyway)
+            // Note: there is a very tiny chance this could succeed due to being extremely lucky
+            // and finding a high difficulty nonce in the first couple of cycles. It will be rare
+            // enough that we shouldn't need to account for it.
+            assertFalse(transaction.isSignatureValid());
+
+            // Reduce difficulty back to 1, to double check
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+            assertTrue(transaction.isSignatureValid());
+        }
+    }
+
+    @Test
+    public void testInvalidService() {
+        byte[] randomHash = new byte[32];
+        new Random().nextBytes(randomHash);
+
+        byte[] lastReference = new byte[64];
+        new Random().nextBytes(lastReference);
+
+        Long now = NTP.getTime();
+
+        final BaseTransactionData baseTransactionData = new BaseTransactionData(now, Group.NO_GROUP,
+                lastReference, randomHash, 0L, null);
+        final String name = "test";
+        final String identifier = "test";
+        final ArbitraryTransactionData.Method method = ArbitraryTransactionData.Method.PUT;
+        final ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.ZIP;
+        final int size = 999;
+        final int version = 5;
+        final int nonce = 0;
+        final byte[] secret = randomHash;
+        final ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH;
+        final byte[] digest = randomHash;
+        final byte[] metadataHash = null;
+        final List<PaymentData> payments = new ArrayList<>();
+        final int validService = Service.IMAGE.value;
+        final int invalidService = 99999999;
+
+        // Try with valid service
+        ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData,
+                version, validService, nonce, size, name, identifier, method,
+                secret, compression, digest, dataType, metadataHash, payments);
+        assertEquals(Service.IMAGE, transactionData.getService());
+
+        // Try with invalid service
+        transactionData = new ArbitraryTransactionData(baseTransactionData,
+                version, invalidService, nonce, size, name, identifier, method,
+                secret, compression, digest, dataType, metadataHash, payments);
+        assertNull(transactionData.getService());
+    }
+
+    @Test
+    public void testOnChainData() throws DataException, IOException, MissingDataException, IllegalAccessException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Set difficulty to 1 to speed up the tests
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 1000;
+            int dataLength = 239; // Max possible size. Becomes 256 bytes after encryption.
+
+            // Register the name to Alice
+            RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, transactionData, alice);
+
+            // Create PUT transaction
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true);
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
+                    null, null, null, null);
+
+            byte[] signature = arbitraryDataFile.getSignature();
+            ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature);
+
+            // Check that the data is published on chain
+            assertEquals(ArbitraryTransactionData.DataType.RAW_DATA, arbitraryTransactionData.getDataType());
+            assertEquals(arbitraryDataFile.getBytes().length, arbitraryTransactionData.getData().length);
+            assertArrayEquals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData());
+
+            // Check that we have no chunks because the complete file is already less than the chunk size
+            assertEquals(0, arbitraryDataFile.chunkCount());
+
+            // Check that we have one file total - just the complete file (no chunks or metadata)
+            assertEquals(1, arbitraryDataFile.fileCount());
+
+            // Check the metadata isn't present
+            assertNull(arbitraryDataFile.getMetadata());
+
+            // Now build the latest data state for this name
+            ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
+            arbitraryDataReader.loadSynchronously(true);
+
+            // Filename will be "data" because it's been held as raw bytes in the transaction,
+            // so there is nowhere to store the original filename
+            File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "data").toFile();
+
+            assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile()));
+        }
+    }
+
+    @Test
+    public void testOnChainDataWithMetadata() throws DataException, IOException, MissingDataException, IllegalAccessException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Set difficulty to 1 to speed up the tests
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 1000;
+            int dataLength = 239; // Max possible size. Becomes 256 bytes after encryption.
+
+            String title = "Test title";
+            String description = "Test description";
+            List<String> tags = Arrays.asList("Test", "tag", "another tag");
+            Category category = Category.QORTAL;
+
+            // Register the name to Alice
+            RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, transactionData, alice);
+
+            // Create PUT transaction
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true);
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
+                    title, description, tags, category);
+
+            byte[] signature = arbitraryDataFile.getSignature();
+            ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature);
+
+            // Check that the data is published on chain
+            assertEquals(ArbitraryTransactionData.DataType.RAW_DATA, arbitraryTransactionData.getDataType());
+            assertEquals(arbitraryDataFile.getBytes().length, arbitraryTransactionData.getData().length);
+            assertArrayEquals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData());
+
+            // Check that we have no chunks because the complete file is already less than the chunk size
+            assertEquals(0, arbitraryDataFile.chunkCount());
+
+            // Check that we have two files total - one for the complete file, and the other for the metadata
+            assertEquals(2, arbitraryDataFile.fileCount());
+
+            // Check the metadata is correct
+            assertEquals(title, arbitraryDataFile.getMetadata().getTitle());
+            assertEquals(description, arbitraryDataFile.getMetadata().getDescription());
+            assertEquals(tags, arbitraryDataFile.getMetadata().getTags());
+            assertEquals(category, arbitraryDataFile.getMetadata().getCategory());
+            assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType());
+
+            // Now build the latest data state for this name
+            ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
+            arbitraryDataReader.loadSynchronously(true);
+
+            // Filename will be "data" because it's been held as raw bytes in the transaction,
+            // so there is nowhere to store the original filename
+            File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "data").toFile();
+
+            assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile()));
+        }
+    }
+
+    @Test
+    public void testOffChainData() throws DataException, IOException, MissingDataException, IllegalAccessException {
+        try (final Repository repository = RepositoryManager.getRepository()) {
+            // Set difficulty to 1 to speed up the tests
+            FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true);
+
+            PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+            String publicKey58 = Base58.encode(alice.getPublicKey());
+            String name = "TEST"; // Can be anything for this test
+            String identifier = null; // Not used for this test
+            Service service = Service.ARBITRARY_DATA;
+            int chunkSize = 1000;
+            int dataLength = 240; // Min possible size. Becomes 257 bytes after encryption.
+
+            // Register the name to Alice
+            RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, "");
+            transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp()));
+            TransactionUtils.signAndMint(repository, transactionData, alice);
+
+            // Create PUT transaction
+            Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true);
+            ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name,
+                    identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true,
+                    null, null, null, null);
+
+            byte[] signature = arbitraryDataFile.getSignature();
+            ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature);
+
+            // Check that the data is published on chain
+            assertEquals(ArbitraryTransactionData.DataType.DATA_HASH, arbitraryTransactionData.getDataType());
+            assertEquals(TransactionTransformer.SHA256_LENGTH, arbitraryTransactionData.getData().length);
+            assertFalse(Arrays.equals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData()));
+
+            // Check that we have no chunks because the complete file is already less than the chunk size
+            assertEquals(0, arbitraryDataFile.chunkCount());
+
+            // Check that we have one file total - just the complete file (no chunks or metadata)
+            assertEquals(1, arbitraryDataFile.fileCount());
+
+            // Check the metadata isn't present
+            assertNull(arbitraryDataFile.getMetadata());
+
+            // Now build the latest data state for this name
+            ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
+            arbitraryDataReader.loadSynchronously(true);
+
+            // File content should match original file
+            File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "file.txt").toFile();
+            assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile()));
+        }
+    }
 }

src/test/java/org/qortal/test/common/ArbitraryUtils.java

@@ -29,31 +29,30 @@ public class ArbitraryUtils {
                                                      int chunkSize) throws DataException {
 
         return ArbitraryUtils.createAndMintTxn(repository, publicKey58, path, name, identifier, method, service,
-                account, chunkSize, null, null, null, null);
+                account, chunkSize, 0L, true, null, null, null, null);
     }
 
     public static ArbitraryDataFile createAndMintTxn(Repository repository, String publicKey58, Path path, String name, String identifier,
                                                      ArbitraryTransactionData.Method method, Service service, PrivateKeyAccount account,
-                                                     int chunkSize, String title, String description, List<String> tags, Category category) throws DataException {
+                                                     int chunkSize, long fee, boolean computeNonce,
+                                                     String title, String description, List<String> tags, Category category) throws DataException {
 
         ArbitraryDataTransactionBuilder txnBuilder = new ArbitraryDataTransactionBuilder(
-                repository, publicKey58, path, name, method, service, identifier, title, description, tags, category);
+                repository, publicKey58, fee, path, name, method, service, identifier, title, description, tags, category);
 
         txnBuilder.setChunkSize(chunkSize);
         txnBuilder.build();
-        txnBuilder.computeNonce();
+        if (computeNonce) {
+            txnBuilder.computeNonce();
+        }
         ArbitraryTransactionData transactionData = txnBuilder.getArbitraryTransactionData();
         Transaction.ValidationResult result = TransactionUtils.signAndImport(repository, transactionData, account);
         assertEquals(Transaction.ValidationResult.OK, result);
         BlockUtils.mintBlock(repository);
 
         // We need a new ArbitraryDataFile instance because the files will have been moved to the signature's folder
-        byte[] hash = txnBuilder.getArbitraryDataFile().getHash();
-        byte[] signature = transactionData.getSignature();
-        ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
-        arbitraryDataFile.setMetadataHash(transactionData.getMetadataHash());
-
-        return arbitraryDataFile;
+        // Or, it may now be using RAW_DATA instead of a hash
+        return ArbitraryDataFile.fromTransactionData(transactionData);
     }
 
     public static ArbitraryDataFile createAndMintTxn(Repository repository, String publicKey58, Path path, String name, String identifier,
@@ -65,6 +64,17 @@ public class ArbitraryUtils {
     }
 
     public static Path generateRandomDataPath(int length) throws IOException {
+        return generateRandomDataPath(length, false);
+    }
+
+    /**
+     * Generate random data, held in a single file within a directory
+     * @param length - size of file to create
+     * @param returnFilePath - if true, the file's path is returned. If false, the outer directory's path is returned.
+     * @return - path to file or directory, depending on the "returnFilePath" boolean
+     * @throws IOException
+     */
+    public static Path generateRandomDataPath(int length, boolean returnFilePath) throws IOException {
         // Create a file in a random temp directory
         Path tempDir = Files.createTempDirectory("generateRandomDataPath");
         File file = new File(Paths.get(tempDir.toString(), "file.txt").toString());
@@ -81,6 +91,10 @@ public class ArbitraryUtils {
         file1Writer.newLine();
         file1Writer.close();
 
+        if (returnFilePath) {
+            return file.toPath();
+        }
+
         return tempDir;
     }
 

src/test/java/org/qortal/test/common/transaction/ArbitraryTestTransaction.java

@@ -45,7 +45,7 @@ public class ArbitraryTestTransaction extends TestTransaction {
 		List<PaymentData> payments = new ArrayList<>();
 		payments.add(new PaymentData(recipient, assetId, amount));
 
-		return new ArbitraryTransactionData(generateBase(account), version, service, nonce, size,name, identifier,
+		return new ArbitraryTransactionData(generateBase(account), version, service.value, nonce, size,name, identifier,
 				method, secret, compression, data, dataType, metadataHash, payments);
 	}
 

src/test/java/org/qortal/test/naming/UpdateTests.java

@@ -219,6 +219,65 @@ public class UpdateTests extends Common {
 		}
 	}
 
+	// Test that multiple UPDATE_NAME transactions work as expected, when using a matching name and newName string
+	@Test
+	public void testDoubleUpdateNameWithMatchingNewName() throws DataException {
+		try (final Repository repository = RepositoryManager.getRepository()) {
+			// Register-name
+			PrivateKeyAccount alice = Common.getTestAccount(repository, "alice");
+			String name = "name";
+			String reducedName = "name";
+			String data = "{\"age\":30}";
+
+			TransactionData initialTransactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, data);
+			initialTransactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(initialTransactionData.getTimestamp()));
+			TransactionUtils.signAndMint(repository, initialTransactionData, alice);
+
+			// Check name exists
+			assertTrue(repository.getNameRepository().nameExists(name));
+			assertNotNull(repository.getNameRepository().fromReducedName(reducedName));
+
+			// Update name
+			TransactionData middleTransactionData = new UpdateNameTransactionData(TestTransaction.generateBase(alice), name, name, data);
+			TransactionUtils.signAndMint(repository, middleTransactionData, alice);
+
+			// Check name still exists
+			assertTrue(repository.getNameRepository().nameExists(name));
+			assertNotNull(repository.getNameRepository().fromReducedName(reducedName));
+
+			// Update name again
+			TransactionData newestTransactionData = new UpdateNameTransactionData(TestTransaction.generateBase(alice), name, name, data);
+			TransactionUtils.signAndMint(repository, newestTransactionData, alice);
+
+			// Check name still exists
+			assertTrue(repository.getNameRepository().nameExists(name));
+			assertNotNull(repository.getNameRepository().fromReducedName(reducedName));
+
+			// Check updated timestamp is correct
+			assertEquals((Long) newestTransactionData.getTimestamp(), repository.getNameRepository().fromName(name).getUpdated());
+
+			// orphan and recheck
+			BlockUtils.orphanLastBlock(repository);
+
+			// Check name still exists
+			assertTrue(repository.getNameRepository().nameExists(name));
+			assertNotNull(repository.getNameRepository().fromReducedName(reducedName));
+
+			// Check updated timestamp is correct
+			assertEquals((Long) middleTransactionData.getTimestamp(), repository.getNameRepository().fromName(name).getUpdated());
+
+			// orphan and recheck
+			BlockUtils.orphanLastBlock(repository);
+
+			// Check name still exists
+			assertTrue(repository.getNameRepository().nameExists(name));
+			assertNotNull(repository.getNameRepository().fromReducedName(reducedName));
+
+			// Check updated timestamp is empty
+			assertNull(repository.getNameRepository().fromName(name).getUpdated());
+		}
+	}
+
 	// Test that reverting using previous UPDATE_NAME works as expected
 	@Test
 	public void testIntermediateUpdateName() throws DataException {

src/test/resources/test-chain-v2-block-timestamps.json

@@ -75,7 +75,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-disable-reference.json

@@ -78,7 +78,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-founder-rewards.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-leftover-reward.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-minting.json

@@ -74,12 +74,13 @@
 		"calcChainWeightTimestamp": 0,
 		"transactionV5Timestamp": 0,
 		"transactionV6Timestamp": 0,
-		"disableReferenceTimestamp": 9999999999999,
+		"disableReferenceTimestamp": 0,
 		"increaseOnlineAccountsDifficultyTimestamp": 9999999999999,
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 9999999999999
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-qora-holder-extremes.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-qora-holder-reduction.json

@@ -80,7 +80,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-qora-holder.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-reward-levels.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-reward-scaling.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-reward-shares.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2-self-sponsorship-algo.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 20,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-chain-v2.json

@@ -79,7 +79,8 @@
 		"onlineAccountMinterLevelValidationHeight": 0,
 		"selfSponsorshipAlgoV1Height": 999999999,
 		"feeValidationFixTimestamp": 0,
-		"chatReferenceTimestamp": 0
+		"chatReferenceTimestamp": 0,
+		"arbitraryOptionalFeeTimestamp": 0
 	},
 	"genesisInfo": {
 		"version": 4,

src/test/resources/test-settings-v2-block-archive.json

@@ -9,5 +9,6 @@
   "testNtpOffset": 0,
   "minPeers": 0,
   "pruneBlockLimit": 100,
-  "repositoryPath": "dbtest"
+  "repositoryPath": "dbtest",
+  "defaultArchiveVersion": 1
 }

testnet/README.md

@@ -2,9 +2,10 @@
 
 ## Create testnet blockchain config
 
-- You can begin by copying the mainnet blockchain config `src/main/resources/blockchain.json`
+- The simplest option is to use the testchain.json included in this folder.
+- Alternatively, you can create one by copying the mainnet blockchain config `src/main/resources/blockchain.json`
 - Insert `"isTestChain": true,` after the opening `{`
-- Modify testnet genesis block
+- Modify testnet genesis block, feature triggers etc
 
 ### Testnet genesis block
 
@@ -25,6 +26,7 @@
 - Make sure to reference testnet blockchain config file: `"blockchainConfig": "testchain.json",`
 - It is a good idea to use a separate database: `"repositoryPath": "db-testnet",`
 - You might also need to add `"bitcoinNet": "TEST3",` and `"litecoinNet": "TEST3",`
+- Also make sure to use a custom `listenPort` (not 62391 or 12391) to ensure that transactions remain isolated to your testnet.
 
 ## Other nodes
 
@@ -58,7 +60,7 @@ Just add this setting:
 "singleNodeTestnet": true
 ```
 This will automatically allow multiple consecutive blocks to be minted, as well as setting minBlockchainPeers to 0.
-Remember to put these values back after introducing other nodes
+Remember to put these values back after introducing other nodes.
 
 ## Fixed network
 
@@ -97,6 +99,10 @@ Your options are:
 {
   "isTestNet": true,
   "bitcoinNet": "TEST3",
+  "litecoinNet": "TEST3",
+  "dogecoinNet": "TEST3",
+  "digibyteNet": "TEST3",
+  "ravencoinNet": "TEST3",
   "repositoryPath": "db-testnet",
   "blockchainConfig": "testchain.json",
   "minBlockchainPeers": 1,
@@ -110,14 +116,16 @@ Your options are:
 }
 ```
 
+<a name="quick-start"></a>
 ## Quick start
 Here are some steps to quickly get a single node testnet up and running with a generic minting account:
-1. Start with template `settings-test.json`, and create a `testchain.json` based on mainnet's blockchain.json (or obtain one from Qortal developers). These should be in the same directory as the jar.
-2. Make sure feature triggers and other timestamp/height activations are correctly set. Generally these would be `0` so that they are enabled from the start.
-3. Set a recent genesis `timestamp` in testchain.json, and add this reward share entry:
+1. Start with template `settings-test.json`, and `testchain.json` which can be found in this folder. Copy/move them to the same directory as the jar.
+2. Set a custom `listenPort` in settings-test.json (not 62391 or 12391) to ensure that transactions remain isolated to your testnet.
+3. Make sure feature triggers and other timestamp/height activations are correctly set. Generally these would be `0` so that they are enabled from the start.
+4. Set a recent genesis `timestamp` in testchain.json, and add this reward share entry:
 `{ "type": "REWARD_SHARE", "minterPublicKey": "DwcUnhxjamqppgfXCLgbYRx8H9XFPUc2qYRy3CEvQWEw", "recipient": "QbTDMss7NtRxxQaSqBZtSLSNdSYgvGaqFf", "rewardSharePublicKey": "CRvQXxFfUMfr4q3o1PcUZPA4aPCiubBsXkk47GzRo754", "sharePercent": 0 },`
-4. Start the node, passing in settings-test.json, e.g: `java -jar qortal.jar settings-test.json`
-5. Once started, add the corresponding minting key to the node:
+5. Start the node, passing in settings-test.json, e.g: `java -jar qortal.jar settings-test.json`
+6. Once started, add the corresponding minting key to the node:
 `curl -X POST "http://localhost:62391/admin/mintingaccounts" -d "F48mYJycFgRdqtc58kiovwbcJgVukjzRE4qRRtRsK9ix"`
-6. Alternatively you can use your own minting account instead of the generic one above.
-7. After a short while, blocks should be minted from the genesis timestamp until the current time.
+7. Alternatively you can use your own minting account instead of the generic one above.
+8. After a short while, blocks should be minted from the genesis timestamp until the current time.

testnet/settings-test.json

@@ -0,0 +1,18 @@
+{
+  "isTestNet": true,
+  "bitcoinNet": "TEST3",
+  "litecoinNet": "TEST3",
+  "dogecoinNet": "TEST3",
+  "digibyteNet": "TEST3",
+  "ravencoinNet": "TEST3",
+  "repositoryPath": "db-testnet",
+  "blockchainConfig": "testchain.json",
+  "minBlockchainPeers": 1,
+  "apiDocumentationEnabled": true,
+  "apiRestricted": false,
+  "bootstrap": false,
+  "maxPeerConnectionTime": 999999999,
+  "localAuthBypassEnabled": true,
+  "singleNodeTestnet": false,
+  "recoveryModeTimeout": 0
+}