crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity

Merge branch 'test-german' into german-translation

Browse Source
This commit is contained in:
QuickMythril 2023-09-08 02:29:07 -04:00 committed by GitHub
commit f74c9672f6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
303 changed files with 19589 additions and 9614 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.github/workflows/pr-testing.yml vendored
View File

@ -8,16 +8,16 @@ jobs:
mavenTesting:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Cache local Maven repository
uses: actions/cache@v2
uses: actions/cache@v3
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Set up the Java JDK
uses: actions/setup-java@v2
uses: actions/setup-java@v3
with:
java-version: '11'
distribution: 'adopt'

1
.gitignore vendored
View File

@ -14,7 +14,6 @@
/.mvn.classpath
/notes*
/settings.json
/testnet*
/settings*.json
/testchain*.json
/run-testnet*.sh

911
Q-Apps.md Normal file
View File

@ -0,0 +1,911 @@
# Qortal Project - Q-Apps Documentation
## Introduction
Q-Apps are static web apps written in javascript, HTML, CSS, and other static assets. The key difference between a Q-App and a fully static site is its ability to interact with both the logged-in user and on-chain data. This is achieved using the API described in this document.
# Section 0: Basic QDN concepts
## Introduction to QDN resources
Each published item on QDN (Qortal Data Network) is referred to as a "resource". A resource could contain anything from a few characters of text, to a multi-layered directory structure containing thousands of files.
Resources are stored on-chain, however the data payload is generally stored off-chain, and verified using an on-chain SHA-256 hash.
To publish a resource, a user must first register a name, and then include that name when publishing the data. Accounts without a registered name are unable to publish to QDN from a Q-App at this time.
Owning the name grants update privileges to the data. If that name is later sold or transferred, the permission to update that resource is moved to the new owner.
## Name, service & identifier
Each QDN resource has 3 important fields:
- `name` - the registered name of the account that is publishing the data (which will hold update/edit privileges going forwards).<br /><br />
- `service` - the type of content (e.g. IMAGE or JSON). Different services have different validation rules. See [list of available services](#services).<br /><br />
- `identifier` - an optional string to allow more than one resource to exist for a given name/service combination. For example, the name `QortalDemo` may wish to publish multiple images. This can be achieved by using a different identifier string for each. The identifier is only unique to the name in question, and so it doesn't matter if another name is using the same service and identifier string.
## Shared identifiers
Since an identifier can be used by multiple names, this can be used to the advantage of Q-App developers as it allows for data to be stored in a deterministic location.
An example of this is the user's avatar. This will always be published with service `THUMBNAIL` and identifier `qortal_avatar`, along with the user's name. So, an app can display the avatar of a user just by specifying their name when requesting the data. The same applies when publishing data.
## "Default" resources
A "default" resource refers to one without an identifier. For example, when a website is published via the UI, it will use the user's name and the service `WEBSITE`. These do not have an identifier, and are therefore the "default" website for this name. When requesting or publishing data without an identifier, apps can either omit the `identifier` key entirely, or include `"identifier": "default"` to indicate that the resource(s) being queried or published do not have an identifier.
<a name="services"></a>
## Available service types
Here is a list of currently available services that can be used in Q-Apps:
### Public services ###
The services below are intended to be used for publicly accessible data.
IMAGE,
THUMBNAIL,
VIDEO,
AUDIO,
PODCAST,
VOICE,
ARBITRARY_DATA,
JSON,
DOCUMENT,
LIST,
PLAYLIST,
METADATA,
BLOG,
BLOG_POST,
BLOG_COMMENT,
GIF_REPOSITORY,
ATTACHMENT,
FILE,
FILES,
CHAIN_DATA,
STORE,
PRODUCT,
OFFER,
COUPON,
CODE,
PLUGIN,
EXTENSION,
GAME,
ITEM,
NFT,
DATABASE,
SNAPSHOT,
COMMENT,
CHAIN_COMMENT,
WEBSITE,
APP,
QCHAT_ATTACHMENT,
QCHAT_IMAGE,
QCHAT_AUDIO,
QCHAT_VOICE
### Private services ###
For the services below, data is encrypted for a single recipient, and can only be decrypted using the private key of the recipient's wallet.
QCHAT_ATTACHMENT_PRIVATE,
ATTACHMENT_PRIVATE,
FILE_PRIVATE,
IMAGE_PRIVATE,
VIDEO_PRIVATE,
AUDIO_PRIVATE,
VOICE_PRIVATE,
DOCUMENT_PRIVATE,
MAIL_PRIVATE,
MESSAGE_PRIVATE
## Single vs multi-file resources
Some resources, such as those published with the `IMAGE` or `JSON` service, consist of a single file or piece of data (the image or the JSON string). This is the most common type of QDN resource, especially in the context of Q-Apps. These can be published by supplying a base64-encoded string containing the data.
Other resources, such as those published with the `WEBSITE`, `APP`, or `GIF_REPOSITORY` service, can contain multiple files and directories. Publishing these kinds of files is not yet available for Q-Apps, however it is possible to retrieve multi-file resources that are already published. When retrieving this data (via FETCH_QDN_RESOURCE), a `filepath` must be included to indicate the file that you would like to retrieve. There is no need to specify a filepath for single file resources, as these will automatically return the contents of the single file.
## App-specific data
Some apps may want to make all QDN data for a particular service available. However, others may prefer to only deal with data that has been published by their app (if a specific format/schema is being used for instance).
Identifiers can be used to allow app developers to locate data that has been published by their app. The recommended approach for this is to use the app name as a prefix on all identifiers when publishing data.
For instance, an app called `MyApp` could allow users to publish JSON data. The app could choose to prefix all identifiers with the string `myapp_`, and then use a random string for each published resource (resulting in identifiers such as `myapp_qR5ndZ8v`). Then, to locate data that has potentially been published by users of MyApp, it can later search the QDN database for items with `"service": "JSON"` and `"identifier": "myapp_"`. The SEARCH_QDN_RESOURCES action has a `prefix` option in order to match identifiers beginning with the supplied string.
Note that QDN is a permissionless system, and therefore it's not possible to verify that a resource was actually published by the app. It is recommended that apps validate the contents of the resource to ensure it is formatted correctly, instead of making assumptions.
## Updating a resource
To update a resource, it can be overwritten by publishing with the same `name`, `service`, and `identifier` combination. Note that the authenticated account must currently own the name in order to publish an update.
## Routing
If a non-existent `filepath` is accessed, the default behaviour of QDN is to return a `404: File not found` error. This includes anything published using the `WEBSITE` service.
However, routing is handled differently for anything published using the `APP` service.
For apps, QDN automatically sends all unhandled requests to the index file (generally index.html). This allows the app to use custom routing, as it is able to listen on any path. If a file exists at a path, the file itself will be served, and so the request won't be sent to the index file.
It's recommended that all apps return a 404 page if a request isn't able to be routed.
# Section 1: Simple links and image loading via HTML
## Section 1a: Linking to other QDN websites / resources
The `qortal://` protocol can be used to access QDN data from within Qortal websites and apps. The basic format is as follows:
```
<a href="qortal://{service}/{name}/{identifier}/{path}">link text</a>
```
However, the system will support the omission of the `identifier` and/or `path` components to allow for simpler URL formats.
A simple link to another website can be achieved with this HTML code:
```
<a href="qortal://WEBSITE/QortalDemo">link text</a>
```
To link to a specific page of another website:
```
<a href="qortal://WEBSITE/QortalDemo/minting-leveling/index.html">link text</a>
```
To link to a standalone resource, such as an avatar
```
<a href="qortal://THUMBNAIL/QortalDemo/qortal_avatar">avatar</a>
```
For cases where you would prefer to explicitly include an identifier (to remove ambiguity) you can use the keyword `default` to access a resource that doesn't have an identifier. For instance:
```
<a href="qortal://WEBSITE/QortalDemo/default">link to root of website</a>
<a href="qortal://WEBSITE/QortalDemo/default/minting-leveling/index.html">link to subpage of website</a>
```
## Section 1b: Linking to other QDN images
The same applies for images, such as displaying an avatar:
```
<img src="qortal://THUMBNAIL/QortalDemo/qortal_avatar" />
```
...or even an image from an entirely different website:
```
<img src="qortal://WEBSITE/AlphaX/assets/img/logo.png" />
```
# Section 2: Integrating a Javascript app
Javascript apps allow for much more complex integrations with Qortal's blockchain data.
## Section 2a: Direct API calls
The standard [Qortal Core API](http://localhost:12391/api-documentation) is available to websites and apps, and can be called directly using a standard AJAX request, such as:
```
async function getNameInfo(name) {
const response = await fetch("/names/" + name);
const nameData = await response.json();
console.log("nameData: " + JSON.stringify(nameData));
}
getNameInfo("QortalDemo");
```
However, this only works for read-only data, such as looking up transactions, names, balances, etc. Also, since the address of the logged in account can't be retrieved from the core, apps can't show personalized data with this approach.
## Section 2b: User interaction via qortalRequest()
To take things a step further, the qortalRequest() function can be used to interact with the user, in order to:
- Request address and public key of the logged in account
- Publish data to QDN
- Send chat messages
- Join groups
- Deploy ATs (smart contracts)
- Send QORT or any supported foreign coin
- Add/remove items from lists
In addition to the above, qortalRequest() also supports many read-only functions that are also available via direct core API calls. Using qortalRequest() helps with futureproofing, as the core APIs can be modified without breaking functionality of existing Q-Apps.
### Making a request
Qortal core will automatically inject the `qortalRequest()` javascript function to all websites/apps, which returns a Promise. This can be used to fetch data or publish data to the Qortal blockchain. This functionality supports async/await, as well as try/catch error handling.
```
async function myfunction() {
try {
let res = await qortalRequest({
action: "GET_ACCOUNT_DATA",
address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
});
console.log(JSON.stringify(res)); // Log the response to the console
} catch(e) {
console.log("Error: " + JSON.stringify(e));
}
}
myfunction();
```
### Timeouts
Request timeouts are handled automatically when using qortalRequest(). The timeout value will differ based on the action being used - see `getDefaultTimeout()` in [q-apps.js](src/main/resources/q-apps/q-apps.js) for the current values.
If a request times out it will throw an error - `The request timed out` - which can be handled by the Q-App.
It is also possible to specify a custom timeout using `qortalRequestWithTimeout(request, timeout)`, however this is discouraged. It's more reliable and futureproof to let the core handle the timeout values.
# Section 3: qortalRequest Documentation
## Supported actions
Here is a list of currently supported actions:
- GET_USER_ACCOUNT
- GET_ACCOUNT_DATA
- GET_ACCOUNT_NAMES
- SEARCH_NAMES
- GET_NAME_DATA
- LIST_QDN_RESOURCES
- SEARCH_QDN_RESOURCES
- GET_QDN_RESOURCE_STATUS
- GET_QDN_RESOURCE_PROPERTIES
- GET_QDN_RESOURCE_METADATA
- GET_QDN_RESOURCE_URL
- LINK_TO_QDN_RESOURCE
- FETCH_QDN_RESOURCE
- PUBLISH_QDN_RESOURCE
- PUBLISH_MULTIPLE_QDN_RESOURCES
- DECRYPT_DATA
- SAVE_FILE
- GET_WALLET_BALANCE
- GET_BALANCE
- SEND_COIN
- SEARCH_CHAT_MESSAGES
- SEND_CHAT_MESSAGE
- LIST_GROUPS
- JOIN_GROUP
- DEPLOY_AT
- GET_AT
- GET_AT_DATA
- LIST_ATS
- FETCH_BLOCK
- FETCH_BLOCK_RANGE
- SEARCH_TRANSACTIONS
- GET_PRICE
- GET_LIST_ITEMS
- ADD_LIST_ITEMS
- DELETE_LIST_ITEM
More functionality will be added in the future.
## Example Requests
Here are some example requests for each of the above:
### Get address of logged in account
_Will likely require user approval_
```
let account = await qortalRequest({
action: "GET_USER_ACCOUNT"
});
let address = account.address;
```
### Get public key of logged in account
_Will likely require user approval_
```
let pubkey = await qortalRequest({
action: "GET_USER_ACCOUNT"
});
let publicKey = account.publicKey;
```
### Get account data
```
let res = await qortalRequest({
action: "GET_ACCOUNT_DATA",
address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
});
```
### Get names owned by account
```
let res = await qortalRequest({
action: "GET_ACCOUNT_NAMES",
address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
});
```
### Search names
```
let res = await qortalRequest({
action: "SEARCH_NAMES",
query: "search query goes here",
prefix: false, // Optional - if true, only the beginning of the name is matched
limit: 100,
offset: 0,
reverse: false
});
```
### Get name data
```
let res = await qortalRequest({
action: "GET_NAME_DATA",
name: "QortalDemo"
});
```
### List QDN resources
```
let res = await qortalRequest({
action: "LIST_QDN_RESOURCES",
service: "THUMBNAIL",
name: "QortalDemo", // Optional (exact match)
identifier: "qortal_avatar", // Optional (exact match)
default: true, // Optional
includeStatus: false, // Optional - will take time to respond, so only request if necessary
includeMetadata: false, // Optional - will take time to respond, so only request if necessary
followedOnly: false, // Optional - include followed names only
excludeBlocked: false, // Optional - exclude blocked content
limit: 100,
offset: 0,
reverse: true
});
```
### Search QDN resources
```
let res = await qortalRequest({
action: "SEARCH_QDN_RESOURCES",
service: "THUMBNAIL",
query: "search query goes here", // Optional - searches both "identifier" and "name" fields
identifier: "search query goes here", // Optional - searches only the "identifier" field
name: "search query goes here", // Optional - searches only the "name" field
prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters
exactMatchNames: true, // Optional - if true, partial name matches are excluded
default: false, // Optional - if true, only resources without identifiers are returned
includeStatus: false, // Optional - will take time to respond, so only request if necessary
includeMetadata: false, // Optional - will take time to respond, so only request if necessary
nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list
followedOnly: false, // Optional - include followed names only
excludeBlocked: false, // Optional - exclude blocked content
limit: 100,
offset: 0,
reverse: true
});
```
### Search QDN resources (multiple names)
```
let res = await qortalRequest({
action: "SEARCH_QDN_RESOURCES",
service: "THUMBNAIL",
query: "search query goes here", // Optional - searches both "identifier" and "name" fields
identifier: "search query goes here", // Optional - searches only the "identifier" field
names: ["QortalDemo", "crowetic", "AlphaX"], // Optional - searches only the "name" field for any of the supplied names
prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters
default: false, // Optional - if true, only resources without identifiers are returned
includeStatus: false, // Optional - will take time to respond, so only request if necessary
includeMetadata: false, // Optional - will take time to respond, so only request if necessary
nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list
followedOnly: false, // Optional - include followed names only
excludeBlocked: false, // Optional - exclude blocked content
limit: 100,
offset: 0,
reverse: true
});
```
### Fetch QDN single file resource
```
let res = await qortalRequest({
action: "FETCH_QDN_RESOURCE",
name: "QortalDemo",
service: "THUMBNAIL",
identifier: "qortal_avatar", // Optional. If omitted, the default resource is returned, or you can alternatively use the keyword "default"
encoding: "base64", // Optional. If omitted, data is returned in raw form
rebuild: false
});
```
### Fetch file from multi file QDN resource
Data is returned in the base64 format
```
let res = await qortalRequest({
action: "FETCH_QDN_RESOURCE",
name: "QortalDemo",
service: "WEBSITE",
identifier: "default", // Optional. If omitted, the default resource is returned, or you can alternatively request that using the keyword "default", as shown here
filepath: "index.html", // Required only for resources containing more than one file
rebuild: false
});
```
### Get QDN resource status
```
let res = await qortalRequest({
action: "GET_QDN_RESOURCE_STATUS",
name: "QortalDemo",
service: "THUMBNAIL",
identifier: "qortal_avatar", // Optional
build: true // Optional - request that the resource is fetched & built in the background
});
```
### Get QDN resource properties
```
let res = await qortalRequest({
action: "GET_QDN_RESOURCE_PROPERTIES",
name: "QortalDemo",
service: "THUMBNAIL",
identifier: "qortal_avatar" // Optional
});
// Returns: filename, size, mimeType (where available)
```
### Get QDN resource metadata
```
let res = await qortalRequest({
action: "GET_QDN_RESOURCE_METADATA",
name: "QortalDemo",
service: "THUMBNAIL",
identifier: "qortal_avatar" // Optional
});
```
### Publish a single file to QDN
_Requires user approval_.<br />
Note: this publishes a single, base64-encoded file. Multi-file resource publishing (such as a WEBSITE or GIF_REPOSITORY) is not yet supported via a Q-App. It will be added in a future update.
```
let res = await qortalRequest({
action: "PUBLISH_QDN_RESOURCE",
name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
service: "IMAGE",
identifier: "myapp-image1234" // Optional
data64: "base64_encoded_data",
// filename: "image.jpg", // Optional - to help apps determine the file's type
// title: "Title", // Optional
// description: "Description", // Optional
// category: "TECHNOLOGY", // Optional
// tag1: "any", // Optional
// tag2: "strings", // Optional
// tag3: "can", // Optional
// tag4: "go", // Optional
// tag5: "here", // Optional
// encrypt: true, // Optional - to be used with a private service
// recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true
});
```
### Publish multiple resources at once to QDN
_Requires user approval_.<br />
Note: each resource being published consists of a single, base64-encoded file, each in its own transaction. Useful for publishing two or more related things, such as a video and a video thumbnail.
```
let res = await qortalRequest({
action: "PUBLISH_MULTIPLE_QDN_RESOURCES",
resources: [
name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list
service: "IMAGE",
identifier: "myapp-image1234" // Optional
data64: "base64_encoded_data",
// filename: "image.jpg", // Optional - to help apps determine the file's type
// title: "Title", // Optional
// description: "Description", // Optional
// category: "TECHNOLOGY", // Optional
// tag1: "any", // Optional
// tag2: "strings", // Optional
// tag3: "can", // Optional
// tag4: "go", // Optional
// tag5: "here", // Optional
// encrypt: true, // Optional - to be used with a private service
// recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true
],
[
... more resources here if needed ...
]
});
```
### Decrypt encrypted/private data
```
let res = await qortalRequest({
action: "DECRYPT_DATA",
encryptedData: 'qortalEncryptedDatabMx4fELNTV+ifJxmv4+GcuOIJOTo+3qAvbWKNY2L1r',
publicKey: 'publickeygoeshere'
});
// Returns base64 encoded string of plaintext data
```
### Prompt user to save a file to disk
Note: mimeType not required but recommended. If not specified, saving will fail if the mimeType is unable to be derived from the Blob.
```
let res = await qortalRequest({
action: "SAVE_FILE",
blob: dataBlob,
filename: "myfile.pdf",
mimeType: "application/pdf" // Optional but recommended
});
```
### Get wallet balance (QORT)
_Requires user approval_
```
let res = await qortalRequest({
action: "GET_WALLET_BALANCE",
coin: "QORT"
});
```
### Get address or asset balance
```
let res = await qortalRequest({
action: "GET_BALANCE",
address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
});
```
```
let res = await qortalRequest({
action: "GET_BALANCE",
assetId: 1,
address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2"
});
```
### Send QORT to address
_Requires user approval_
```
let res = await qortalRequest({
action: "SEND_COIN",
coin: "QORT",
destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
amount: 1.00000000 // 1 QORT
});
```
### Send foreign coin to address
_Requires user approval_<br />
Note: default fees can be found [here](https://github.com/Qortal/qortal-ui/blob/master/plugins/plugins/core/qdn/browser/browser.src.js#L205-L209).
```
let res = await qortalRequest({
action: "SEND_COIN",
coin: "LTC",
destinationAddress: "LSdTvMHRm8sScqwCi6x9wzYQae8JeZhx6y",
amount: 1.00000000, // 1 LTC
fee: 0.00000020 // Optional fee per byte (default fee used if omitted, recommended) - not used for QORT or ARRR
});
```
### Search or list chat messages
```
let res = await qortalRequest({
action: "SEARCH_CHAT_MESSAGES",
before: 999999999999999,
after: 0,
txGroupId: 0, // Optional (must specify either txGroupId or two involving addresses)
// involving: ["QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2", "QSefrppsDCsZebcwrqiM1gNbWq7YMDXtG2"], // Optional (must specify either txGroupId or two involving addresses)
// reference: "reference", // Optional
// chatReference: "chatreference", // Optional
// hasChatReference: true, // Optional
encoding: "BASE64", // Optional (defaults to BASE58 if omitted)
limit: 100,
offset: 0,
reverse: true
});
```
### Send a group chat message
_Requires user approval_
```
let res = await qortalRequest({
action: "SEND_CHAT_MESSAGE",
groupId: 0,
message: "Test"
});
```
### Send a private chat message
_Requires user approval_
```
let res = await qortalRequest({
action: "SEND_CHAT_MESSAGE",
destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2",
message: "Test"
});
```
### List groups
```
let res = await qortalRequest({
action: "LIST_GROUPS",
limit: 100,
offset: 0,
reverse: true
});
```
### Join a group
_Requires user approval_
```
let res = await qortalRequest({
action: "JOIN_GROUP",
groupId: 100
});
```
### Deploy an AT
_Requires user approval_
```
let res = await qortalRequest({
action: "DEPLOY_AT",
creationBytes: "12345", // Must be Base58 encoded
name: "test name",
description: "test description",
type: "test type",
tags: "test tags",
amount: 1.00000000, // 1 QORT
assetId: 0,
// fee: 0.002 // optional - will use default fee if excluded
});
```
### Get AT info
```
let res = await qortalRequest({
action: "GET_AT",
atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH"
});
```
### Get AT data bytes (base58 encoded)
```
let res = await qortalRequest({
action: "GET_AT_DATA",
atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH"
});
```
### List ATs by functionality
```
let res = await qortalRequest({
action: "LIST_ATS",
codeHash58: "4KdJETRAdymE7dodDmJbf5d9L1bp4g5Nxky8m47TBkvA",
isExecutable: true,
limit: 100,
offset: 0,
reverse: true
});
```
### Fetch block by signature
```
let res = await qortalRequest({
action: "FETCH_BLOCK",
signature: "875yGFUy1zHV2hmxNWzrhtn9S1zkeD7SQppwdXFysvTXrankCHCz4iyAUgCBM3GjvibbnyRQpriuy1cyu953U1u5uQdzuH3QjQivi9UVwz86z1Akn17MGd5Z5STjpDT7248K6vzMamuqDei57Znonr8GGgn8yyyABn35CbZUCeAuXju"
});
```
### Fetch block by height
```
let res = await qortalRequest({
action: "FETCH_BLOCK",
height: "1139850"
});
```
### Fetch a range of blocks
```
let res = await qortalRequest({
action: "FETCH_BLOCK_RANGE",
height: "1139800",
count: 20,
reverse: false
});
```
### Search transactions
```
let res = await qortalRequest({
action: "SEARCH_TRANSACTIONS",
// startBlock: 1139000,
// blockLimit: 1000,
txGroupId: 0,
txType: [
"PAYMENT",
"REWARD_SHARE"
],
confirmationStatus: "CONFIRMED",
limit: 10,
offset: 0,
reverse: false
});
```
### Get an estimate of the QORT price
```
let res = await qortalRequest({
action: "GET_PRICE",
blockchain: "LITECOIN",
// maxtrades: 10,
inverse: true
});
```
### Get URL to load a QDN resource
Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
```
let url = await qortalRequest({
action: "GET_QDN_RESOURCE_URL",
service: "THUMBNAIL",
name: "QortalDemo",
identifier: "qortal_avatar"
// path: "filename.jpg" // optional - not needed if resource contains only one file
});
```
### Get URL to load a QDN website
Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
```
let url = await qortalRequest({
action: "GET_QDN_RESOURCE_URL",
service: "WEBSITE",
name: "QortalDemo",
});
```
### Get URL to load a specific file from a QDN website
Note: this returns a "Resource does not exist" error if a non-existent resource is requested.
```
let url = await qortalRequest({
action: "GET_QDN_RESOURCE_URL",
service: "WEBSITE",
name: "AlphaX",
path: "/assets/img/logo.png"
});
```
### Link/redirect to another QDN website
Note: an alternate method is to include `<a href="qortal://WEBSITE/QortalDemo">link text</a>` within your HTML code.
```
let res = await qortalRequest({
action: "LINK_TO_QDN_RESOURCE",
service: "WEBSITE",
name: "QortalDemo",
});
```
### Link/redirect to a specific path of another QDN website
Note: an alternate method is to include `<a href="qortal://WEBSITE/QortalDemo/minting-leveling/index.html">link text</a>` within your HTML code.
```
let res = await qortalRequest({
action: "LINK_TO_QDN_RESOURCE",
service: "WEBSITE",
name: "QortalDemo",
path: "/minting-leveling/index.html"
});
```
### Get the contents of a list
_Requires user approval_
```
let res = await qortalRequest({
action: "GET_LIST_ITEMS",
list_name: "followedNames"
});
```
### Add one or more items to a list
_Requires user approval_
```
let res = await qortalRequest({
action: "ADD_LIST_ITEMS",
list_name: "blockedNames",
items: ["QortalDemo"]
});
```
### Delete a single item from a list
_Requires user approval_.
Items must be deleted one at a time.
```
let res = await qortalRequest({
action: "DELETE_LIST_ITEM",
list_name: "blockedNames",
item: "QortalDemo"
});
```
# Section 4: Examples
Some example projects can be found [here](https://github.com/Qortal/Q-Apps). These can be cloned and modified, or used as a reference when creating a new app.
## Sample App
Here is a sample application to display the logged-in user's avatar:
```
<html>
<head>
<script>
async function showAvatar() {
try {
// Get QORT address of logged in account
let account = await qortalRequest({
action: "GET_USER_ACCOUNT"
});
let address = account.address;
console.log("address: " + address);
// Get names owned by this account
let names = await qortalRequest({
action: "GET_ACCOUNT_NAMES",
address: address
});
console.log("names: " + JSON.stringify(names));
if (names.length == 0) {
console.log("User has no registered names");
return;
}
// Download base64-encoded avatar of the first registered name
let avatar = await qortalRequest({
action: "FETCH_QDN_RESOURCE",
name: names[0].name,
service: "THUMBNAIL",
identifier: "qortal_avatar",
encoding: "base64"
});
console.log("Avatar size: " + avatar.length + " bytes");
// Display the avatar image on the screen
document.getElementById("avatar").src = "data:image/png;base64," + avatar;
} catch(e) {
console.log("Error: " + JSON.stringify(e));
}
}
</script>
</head>
<body onload="showAvatar()">
<img width="500" id="avatar" />
</body>
</html>
```
# Section 5: Testing and Development
Publishing an in-development app to mainnet isn't recommended. There are several options for developing and testing a Q-app before publishing to mainnet:
### Preview mode
Select "Preview" in the UI after choosing the zip. This allows for full Q-App testing without the need to publish any data.
### Testnets
For an end-to-end test of Q-App publishing, you can use the official testnet, or set up a single node testnet of your own (often referred to as devnet) on your local machine. See [Single Node Testnet Quick Start Guide](testnet/README.md#quick-start).
### Debugging
It is recommended that you develop and test in a web browser, to allow access to the javascript console. To do this:
1. Open the UI app, then minimise it.
2. In a Chromium-based web browser, visit: http://localhost:12388/
3. Log in to your account and then preview your app/website.
4. Go to `View > Developer > JavaScript Console`. Here you can monitor console logs, errors, and network requests from your app, in the same way as any other web-app.

8
WindowsInstaller/Qortal.aip
View File

@ -17,10 +17,10 @@
<ROW Property="Manufacturer" Value="Qortal"/>
<ROW Property="MsiLogging" MultiBuildValue="DefaultBuild:vp"/>
<ROW Property="NTP_GOOD" Value="false"/>
<ROW Property="ProductCode" Value="1033:{E5597539-098E-4BA6-99DF-4D22018BC0D3} 1049:{2B5E55A2-142A-4BED-B3B9-5657162282B7} 2052:{6F19171F-4743-4127-B191-AAFA3FA885D2} 2057:{A1B3108D-EC5D-47A1-AEE4-DBD956E682FB} " Type="16"/>
<ROW Property="ProductCode" Value="1033:{CB85115E-ECCE-4B3D-BB7F-6251A2764922} 1049:{09AC1C62-4E33-4312-826A-38F597ED1B17} 2052:{3CF701B3-E118-4A31-A4B7-156CEA19FBCC} 2057:{468F337D-0EF8-41D1-B5DE-4EEE66BA2AF6} " Type="16"/>
<ROW Property="ProductLanguage" Value="2057"/>
<ROW Property="ProductName" Value="Qortal"/>
<ROW Property="ProductVersion" Value="3.4.3" Type="32"/>
<ROW Property="ProductVersion" Value="3.8.5" Type="32"/>
<ROW Property="RECONFIG_NTP" Value="true"/>
<ROW Property="REMOVE_BLOCKCHAIN" Value="YES" Type="4"/>
<ROW Property="REPAIR_BLOCKCHAIN" Value="YES" Type="4"/>
@ -212,7 +212,7 @@
<ROW Component="ADDITIONAL_LICENSE_INFO_71" ComponentId="{12A3ADBE-BB7A-496C-8869-410681E6232F}" Directory_="jdk.zipfs_Dir" Attributes="0" KeyPath="ADDITIONAL_LICENSE_INFO_71" Type="0"/>
<ROW Component="ADDITIONAL_LICENSE_INFO_8" ComponentId="{D53AD95E-CF96-4999-80FC-5812277A7456}" Directory_="java.naming_Dir" Attributes="0" KeyPath="ADDITIONAL_LICENSE_INFO_8" Type="0"/>
<ROW Component="ADDITIONAL_LICENSE_INFO_9" ComponentId="{6B7EA9B0-5D17-47A8-B78C-FACE86D15E01}" Directory_="java.net.http_Dir" Attributes="0" KeyPath="ADDITIONAL_LICENSE_INFO_9" Type="0"/>
<ROW Component="AI_CustomARPName" ComponentId="{F17029E8-CCC4-456D-B4AC-1854C81C46B6}" Directory_="APPDIR" Attributes="260" KeyPath="DisplayName" Options="1"/>
<ROW Component="AI_CustomARPName" ComponentId="{094B5D07-2258-4A39-9917-2E2F7F6E210B}" Directory_="APPDIR" Attributes="260" KeyPath="DisplayName" Options="1"/>
<ROW Component="AI_ExePath" ComponentId="{3644948D-AE0B-41BB-9FAF-A79E70490A08}" Directory_="APPDIR" Attributes="260" KeyPath="AI_ExePath"/>
<ROW Component="APPDIR" ComponentId="{680DFDDE-3FB4-47A5-8FF5-934F576C6F91}" Directory_="APPDIR" Attributes="0"/>
<ROW Component="AccessBridgeCallbacks.h" ComponentId="{288055D1-1062-47A3-AA44-5601B4E38AED}" Directory_="bridge_Dir" Attributes="0" KeyPath="AccessBridgeCallbacks.h" Type="0"/>
@ -1173,7 +1173,7 @@
<ROW Action="AI_STORE_LOCATION" Type="51" Source="ARPINSTALLLOCATION" Target="[APPDIR]"/>
<ROW Action="AI_SetPermissions" Type="11265" Source="userAccounts.dll" Target="OnSetPermissions" WithoutSeq="true"/>
<ROW Action="CustomizeLog4j2PropertiesScript" Type="3109" Target="Script Text" TargetUnformatted="var actionData = Session.Property(&quot;CustomActionData&quot;);&#13;&#10;var actionDataArray = actionData.split(&quot;|&quot;);&#13;&#10;var appDir = actionDataArray[0];&#13;&#10;var dataFolder = actionDataArray[1] + actionDataArray[2] + &quot;\\&quot;;&#13;&#10;&#13;&#10;var ForReading = 1, ForWriting = 2, ForAppending = 8;&#13;&#10;var fso = new ActiveXObject(&quot;Scripting.FileSystemObject&quot;);&#13;&#10;&#13;&#10;// Make copy&#13;&#10;fso.CopyFile(appDir + &quot;log4j2.properties&quot;, appDir + &quot;log4j2-orig.properties&quot;, true); // overwrite&#13;&#10;&#13;&#10;// Rewrite %AppDir%\log4j2.properties to update logfile storage path&#13;&#10;var fin = fso.OpenTextFile(appDir + &quot;log4j2-orig.properties&quot;, ForReading, false); // no create&#13;&#10;var fout = fso.OpenTextFile(appDir + &quot;log4j2.properties&quot;, ForWriting, true); // can create&#13;&#10;&#13;&#10;// Copy lines with rewriting where necessary&#13;&#10;while( !fin.AtEndOfStream ) {&#13;&#10;&#9;var line = fin.ReadLine();&#13;&#10;&#13;&#10;&#9;var start = line.indexOf(&quot;property.dirname&quot;);&#13;&#10;&#9;if (start &gt; 0) {&#13;&#10;&#9;&#9;// line: # property.dirname = ...appdata...&#13;&#10;&#9;&#9;// uncomment/replace this line for Windows&#13;&#10;&#9;&#9;fout.WriteLine( &quot;property.dirname = &quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) );&#13;&#10;&#9;} else {&#13;&#10;&#9;&#9;// not found - output verbatim&#13;&#10;&#9;&#9;fout.WriteLine( line );&#13;&#10;&#9;}&#13;&#10;}&#13;&#10;&#13;&#10;fin.Close();&#13;&#10;fout.Close();&#13;&#10;" AdditionalSeq="AI_DATA_SETTER_4"/>
<ROW Action="CustomizeSettingsJsonScript" Type="3109" Target="Script Text" TargetUnformatted="var actionData = Session.Property(&quot;CustomActionData&quot;);&#13;&#10;var actionDataArray = actionData.split(&quot;|&quot;);&#13;&#10;var appDir = actionDataArray[0];&#13;&#10;var dataFolder = actionDataArray[1] + actionDataArray[2] + &quot;\\&quot;;&#13;&#10;&#13;&#10;var ForReading = 1, ForWriting = 2, ForAppending = 8;&#13;&#10;var fso = new ActiveXObject(&quot;Scripting.FileSystemObject&quot;);&#13;&#10;&#13;&#10;// Create basic %APPDIR%\settings.json with path to real settings.json in dataFolder&#13;&#10;var fts = fso.OpenTextFile(appDir + &quot;settings.json&quot;, ForWriting, true);&#13;&#10;&#13;&#10;fts.WriteLine( &quot;{&quot; );&#13;&#10;// We need to escape Windows path backslashes to keep JSON valid&#13;&#10;fts.WriteLine( &quot; \&quot;userPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;\&quot;&quot; );&#13;&#10;fts.WriteLine( &quot;}&quot; );&#13;&#10;&#13;&#10;fts.Close();&#13;&#10;&#13;&#10;// Make copy&#13;&#10;fso.CopyFile(dataFolder + &quot;settings.json&quot;, dataFolder + &quot;settings-orig.json&quot;, true); // overwrite&#13;&#10;&#13;&#10;// Rewrite settings.json to update repository path&#13;&#10;var fin = fso.OpenTextFile(dataFolder + &quot;settings-orig.json&quot;, ForReading, false);&#13;&#10;var fout = fso.OpenTextFile(dataFolder + &quot;settings.json&quot;, ForWriting, true);&#13;&#10;&#13;&#10;// First line should contain opening brace&#13;&#10;fout.WriteLine( fin.ReadLine() );&#13;&#10;&#13;&#10;// Append our entries&#13;&#10;fout.WriteLine( &quot; \&quot;repositoryPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;db\&quot;,&quot; );&#13;&#10;fout.WriteLine( &quot; \&quot;dataPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;data\&quot;,&quot; );&#13;&#10;&#13;&#10;// copy rest of settings&#13;&#10;while( !fin.AtEndOfStream ) {&#13;&#10;&#9;fout.WriteLine( fin.ReadLine() );&#13;&#10;}&#13;&#10;&#13;&#10;fin.Close();&#13;&#10;fout.Close();&#13;&#10;" AdditionalSeq="AI_DATA_SETTER_3"/>
<ROW Action="CustomizeSettingsJsonScript" Type="3109" Target="Script Text" TargetUnformatted="var actionData = Session.Property(&quot;CustomActionData&quot;);&#13;&#10;var actionDataArray = actionData.split(&quot;|&quot;);&#13;&#10;var appDir = actionDataArray[0];&#13;&#10;var dataFolder = actionDataArray[1] + actionDataArray[2] + &quot;\\&quot;;&#13;&#10;&#13;&#10;var ForReading = 1, ForWriting = 2, ForAppending = 8;&#13;&#10;var fso = new ActiveXObject(&quot;Scripting.FileSystemObject&quot;);&#13;&#10;&#13;&#10;// Create basic %APPDIR%\settings.json with path to real settings.json in dataFolder&#13;&#10;var fts = fso.OpenTextFile(appDir + &quot;settings.json&quot;, ForWriting, true);&#13;&#10;&#13;&#10;fts.WriteLine( &quot;{&quot; );&#13;&#10;// We need to escape Windows path backslashes to keep JSON valid&#13;&#10;fts.WriteLine( &quot; \&quot;userPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;\&quot;&quot; );&#13;&#10;fts.WriteLine( &quot;}&quot; );&#13;&#10;&#13;&#10;fts.Close();&#13;&#10;&#13;&#10;// Make copy&#13;&#10;fso.CopyFile(dataFolder + &quot;settings.json&quot;, dataFolder + &quot;settings-orig.json&quot;, true); // overwrite&#13;&#10;&#13;&#10;// Rewrite settings.json to update repository path&#13;&#10;var fin = fso.OpenTextFile(dataFolder + &quot;settings-orig.json&quot;, ForReading, false);&#13;&#10;var fout = fso.OpenTextFile(dataFolder + &quot;settings.json&quot;, ForWriting, true);&#13;&#10;&#13;&#10;// First line should contain opening brace&#13;&#10;fout.WriteLine( fin.ReadLine() );&#13;&#10;&#13;&#10;// Append our entries&#13;&#10;fout.WriteLine( &quot; \&quot;repositoryPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;db\&quot;,&quot; );&#13;&#10;fout.WriteLine( &quot; \&quot;dataPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;data\&quot;,&quot; );&#13;&#10;fout.WriteLine( &quot; \&quot;walletsPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;wallets\&quot;,&quot; );&#13;&#10;fout.WriteLine( &quot; \&quot;listsPath\&quot;: \&quot;&quot; + dataFolder.split(&apos;\\&apos;).join(&apos;\\\\&apos;) + &quot;lists\&quot;,&quot; );&#13;&#10;&#13;&#10;// copy rest of settings&#13;&#10;while( !fin.AtEndOfStream ) {&#13;&#10;&#9;fout.WriteLine( fin.ReadLine() );&#13;&#10;}&#13;&#10;&#13;&#10;fin.Close();&#13;&#10;fout.Close();&#13;&#10;" AdditionalSeq="AI_DATA_SETTER_3"/>
<ROW Action="DetectRunningProcess" Type="1" Source="aicustact.dll" Target="DetectProcess" Options="3" AdditionalSeq="AI_DATA_SETTER_8"/>
<ROW Action="DetectW32Time" Type="1" Source="aicustact.dll" Target="DetectService" Options="3" AdditionalSeq="AI_DATA_SETTER_11"/>
<ROW Action="NTP_config" Type="3090" Source="ntpcfg.bat"/>

BIN
lib/org/ciyam/AT/1.4.0/AT-1.4.0.jar Normal file
View File

Binary file not shown.

9
lib/org/ciyam/AT/1.4.0/AT-1.4.0.pom Normal file
View File

@ -0,0 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>org.ciyam</groupId>
<artifactId>AT</artifactId>
<version>1.4.0</version>
<description>POM was created from install:install-file</description>
</project>

5
lib/org/ciyam/AT/maven-metadata-local.xml
View File

@ -3,14 +3,15 @@
<groupId>org.ciyam</groupId>
<artifactId>AT</artifactId>
<versioning>
<release>1.3.8</release>
<release>1.4.0</release>
<versions>
<version>1.3.4</version>
<version>1.3.5</version>
<version>1.3.6</version>
<version>1.3.7</version>
<version>1.3.8</version>
<version>1.4.0</version>
</versions>
<lastUpdated>20200925114415</lastUpdated>
<lastUpdated>20221105114346</lastUpdated>
</versioning>
</metadata>

12
pom.xml
View File

@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>org.qortal</groupId>
<artifactId>qortal</artifactId>
<version>3.6.1</version>
<version>4.3.0</version>
<packaging>jar</packaging>
<properties>
<skipTests>true</skipTests>
@ -11,7 +11,7 @@
<bitcoinj.version>0.15.10</bitcoinj.version>
<bouncycastle.version>1.69</bouncycastle.version>
<build.timestamp>${maven.build.timestamp}</build.timestamp>
<ciyam-at.version>1.3.8</ciyam-at.version>
<ciyam-at.version>1.4.0</ciyam-at.version>
<commons-net.version>3.6</commons-net.version>
<commons-text.version>1.8</commons-text.version>
<commons-io.version>2.6</commons-io.version>
@ -36,6 +36,7 @@
<java-diff-utils.version>4.10</java-diff-utils.version>
<grpc.version>1.45.1</grpc.version>
<protobuf.version>3.19.4</protobuf.version>
<simplemagic.version>1.17</simplemagic.version>
</properties>
<build>
<sourceDirectory>src/main/java</sourceDirectory>
@ -147,6 +148,7 @@
tagsSorter: "alpha",
operationsSorter:
"alpha",
validatorUrl: false,
</value>
</replacement>
</replacements>
@ -304,6 +306,7 @@
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.qortal.controller.Controller</mainClass>
<manifestEntries>
<Multi-Release>true</Multi-Release>
<Class-Path>. ..</Class-Path>
</manifestEntries>
</transformer>
@ -727,5 +730,10 @@
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
<dependency>
<groupId>com.j256.simplemagic</groupId>
<artifactId>simplemagic</artifactId>
<version>${simplemagic.version}</version>
</dependency>
</dependencies>
</project>

21
src/main/java/org/qortal/account/Account.java
View File

@ -211,7 +211,8 @@ public class Account {
if (level != null && level >= BlockChain.getInstance().getMinAccountLevelToMint())
return true;
if (Account.isFounder(accountData.getFlags()))
// Founders can always mint, unless they have a penalty
if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
return true;
return false;
@ -222,6 +223,11 @@ public class Account {
return this.repository.getAccountRepository().getMintedBlockCount(this.address);
}
/** Returns account's blockMintedPenalty or null if account not found in repository. */
public Integer getBlocksMintedPenalty() throws DataException {
return this.repository.getAccountRepository().getBlocksMintedPenaltyCount(this.address);
}
/** Returns whether account can build reward-shares.
* <p>
@ -243,7 +249,7 @@ public class Account {
if (level != null && level >= BlockChain.getInstance().getMinAccountLevelToRewardShare())
return true;
if (Account.isFounder(accountData.getFlags()))
if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
return true;
return false;
@ -271,7 +277,7 @@ public class Account {
/**
* Returns 'effective' minting level, or zero if account does not exist/cannot mint.
* <p>
* For founder accounts, this returns "founderEffectiveMintingLevel" from blockchain config.
* For founder accounts with no penalty, this returns "founderEffectiveMintingLevel" from blockchain config.
*
* @return 0+
* @throws DataException
@ -281,7 +287,8 @@ public class Account {
if (accountData == null)
return 0;
if (Account.isFounder(accountData.getFlags()))
// Founders are assigned a different effective minting level, as long as they have no penalty
if (Account.isFounder(accountData.getFlags()) && accountData.getBlocksMintedPenalty() == 0)
return BlockChain.getInstance().getFounderEffectiveMintingLevel();
return accountData.getLevel();
@ -289,8 +296,6 @@ public class Account {
/**
* Returns 'effective' minting level, or zero if reward-share does not exist.
* <p>
* this is being used on src/main/java/org/qortal/api/resource/AddressesResource.java to fulfil the online accounts api call
*
* @param repository
* @param rewardSharePublicKey
@ -309,7 +314,7 @@ public class Account {
/**
* Returns 'effective' minting level, with a fix for the zero level.
* <p>
* For founder accounts, this returns "founderEffectiveMintingLevel" from blockchain config.
* For founder accounts with no penalty, this returns "founderEffectiveMintingLevel" from blockchain config.
*
* @param repository
* @param rewardSharePublicKey
@ -322,7 +327,7 @@ public class Account {
if (rewardShareData == null)
return 0;
else if(!rewardShareData.getMinter().equals(rewardShareData.getRecipient()))//the minter is different than the recipient this means sponsorship
else if (!rewardShareData.getMinter().equals(rewardShareData.getRecipient())) // Sponsorship reward share
return 0;
Account rewardShareMinter = new Account(repository, rewardShareData.getMinter());

367
src/main/java/org/qortal/account/SelfSponsorshipAlgoV1.java Normal file
View File

@ -0,0 +1,367 @@
package org.qortal.account;
import org.qortal.api.resource.TransactionsResource;
import org.qortal.asset.Asset;
import org.qortal.data.account.AccountData;
import org.qortal.data.naming.NameData;
import org.qortal.data.transaction.*;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.transaction.Transaction.TransactionType;
import java.util.*;
import java.util.stream.Collectors;
public class SelfSponsorshipAlgoV1 {
private final Repository repository;
private final String address;
private final AccountData accountData;
private final long snapshotTimestamp;
private final boolean override;
private int registeredNameCount = 0;
private int suspiciousCount = 0;
private int suspiciousPercent = 0;
private int consolidationCount = 0;
private int bulkIssuanceCount = 0;
private int recentSponsorshipCount = 0;
private List<RewardShareTransactionData> sponsorshipRewardShares = new ArrayList<>();
private final Map<String, List<TransactionData>> paymentsByAddress = new HashMap<>();
private final Set<String> sponsees = new LinkedHashSet<>();
private Set<String> consolidatedAddresses = new LinkedHashSet<>();
private final Set<String> zeroTransactionAddreses = new LinkedHashSet<>();
private final Set<String> penaltyAddresses = new LinkedHashSet<>();
public SelfSponsorshipAlgoV1(Repository repository, String address, long snapshotTimestamp, boolean override) throws DataException {
this.repository = repository;
this.address = address;
this.accountData = this.repository.getAccountRepository().getAccount(this.address);
this.snapshotTimestamp = snapshotTimestamp;
this.override = override;
}
public String getAddress() {
return this.address;
}
public Set<String> getPenaltyAddresses() {
return this.penaltyAddresses;
}
public void run() throws DataException {
if (this.accountData == null) {
// Nothing to do
return;
}
this.fetchSponsorshipRewardShares();
if (this.sponsorshipRewardShares.isEmpty()) {
// Nothing to do
return;
}
this.findConsolidatedRewards();
this.findBulkIssuance();
this.findRegisteredNameCount();
this.findRecentSponsorshipCount();
int score = this.calculateScore();
if (score <= 0 && !override) {
return;
}
String newAddress = this.getDestinationAccount(this.address);
while (newAddress != null) {
// Found destination account
this.penaltyAddresses.add(newAddress);
// Run algo for this address, but in "override" mode because it has already been flagged
SelfSponsorshipAlgoV1 algoV1 = new SelfSponsorshipAlgoV1(this.repository, newAddress, this.snapshotTimestamp, true);
algoV1.run();
this.penaltyAddresses.addAll(algoV1.getPenaltyAddresses());
newAddress = this.getDestinationAccount(newAddress);
}
this.penaltyAddresses.add(this.address);
if (this.override || this.recentSponsorshipCount < 20) {
this.penaltyAddresses.addAll(this.consolidatedAddresses);
this.penaltyAddresses.addAll(this.zeroTransactionAddreses);
}
else {
this.penaltyAddresses.addAll(this.sponsees);
}
}
private String getDestinationAccount(String address) throws DataException {
List<TransactionData> transferPrivsTransactions = fetchTransferPrivsForAddress(address);
if (transferPrivsTransactions.isEmpty()) {
// No TRANSFER_PRIVS transactions for this address
return null;
}
AccountData accountData = this.repository.getAccountRepository().getAccount(address);
if (accountData == null) {
return null;
}
for (TransactionData transactionData : transferPrivsTransactions) {
TransferPrivsTransactionData transferPrivsTransactionData = (TransferPrivsTransactionData) transactionData;
if (Arrays.equals(transferPrivsTransactionData.getSenderPublicKey(), accountData.getPublicKey())) {
return transferPrivsTransactionData.getRecipient();
}
}
return null;
}
private void findConsolidatedRewards() throws DataException {
List<String> sponseesThatSentRewards = new ArrayList<>();
Map<String, Integer> paymentRecipients = new HashMap<>();
// Collect outgoing payments of each sponsee
for (String sponseeAddress : this.sponsees) {
// Firstly fetch all payments for address, since the functions below depend on this data
this.fetchPaymentsForAddress(sponseeAddress);
// Check if the address has zero relevant transactions
if (this.hasZeroTransactions(sponseeAddress)) {
this.zeroTransactionAddreses.add(sponseeAddress);
}
// Get payment recipients
List<String> allPaymentRecipients = this.fetchOutgoingPaymentRecipientsForAddress(sponseeAddress);
if (allPaymentRecipients.isEmpty()) {
continue;
}
sponseesThatSentRewards.add(sponseeAddress);
List<String> addressesPaidByThisSponsee = new ArrayList<>();
for (String paymentRecipient : allPaymentRecipients) {
if (addressesPaidByThisSponsee.contains(paymentRecipient)) {
// We already tracked this association - don't allow multiple to stack up
continue;
}
addressesPaidByThisSponsee.add(paymentRecipient);
// Increment count for this recipient, or initialize to 1 if not present
if (paymentRecipients.computeIfPresent(paymentRecipient, (k, v) -> v + 1) == null) {
paymentRecipients.put(paymentRecipient, 1);
}
}
}
// Exclude addresses with a low number of payments
Map<String, Integer> filteredPaymentRecipients = paymentRecipients.entrySet().stream()
.filter(p -> p.getValue() != null && p.getValue() >= 10)
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
// Now check how many sponsees have sent to this subset of addresses
Map<String, Integer> sponseesThatConsolidatedRewards = new HashMap<>();
for (String sponseeAddress : sponseesThatSentRewards) {
List<String> allPaymentRecipients = this.fetchOutgoingPaymentRecipientsForAddress(sponseeAddress);
// Remove any that aren't to one of the flagged recipients (i.e. consolidation)
allPaymentRecipients.removeIf(r -> !filteredPaymentRecipients.containsKey(r));
int count = allPaymentRecipients.size();
if (count == 0) {
continue;
}
if (sponseesThatConsolidatedRewards.computeIfPresent(sponseeAddress, (k, v) -> v + count) == null) {
sponseesThatConsolidatedRewards.put(sponseeAddress, count);
}
}
// Remove sponsees that have only sent a low number of payments to the filtered addresses
Map<String, Integer> filteredSponseesThatConsolidatedRewards = sponseesThatConsolidatedRewards.entrySet().stream()
.filter(p -> p.getValue() != null && p.getValue() >= 2)
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
this.consolidationCount = sponseesThatConsolidatedRewards.size();
this.consolidatedAddresses = new LinkedHashSet<>(filteredSponseesThatConsolidatedRewards.keySet());
this.suspiciousCount = this.consolidationCount + this.zeroTransactionAddreses.size();
this.suspiciousPercent = (int)(this.suspiciousCount / (float) this.sponsees.size() * 100);
}
private void findBulkIssuance() {
Long lastTimestamp = null;
for (RewardShareTransactionData rewardShareTransactionData : sponsorshipRewardShares) {
long timestamp = rewardShareTransactionData.getTimestamp();
if (timestamp >= this.snapshotTimestamp) {
continue;
}
if (lastTimestamp != null) {
if (timestamp - lastTimestamp < 3*60*1000L) {
this.bulkIssuanceCount++;
}
}
lastTimestamp = timestamp;
}
}
private void findRegisteredNameCount() throws DataException {
int registeredNameCount = 0;
for (String sponseeAddress : sponsees) {
List<NameData> names = repository.getNameRepository().getNamesByOwner(sponseeAddress);
for (NameData name : names) {
if (name.getRegistered() < this.snapshotTimestamp) {
registeredNameCount++;
break;
}
}
}
this.registeredNameCount = registeredNameCount;
}
private void findRecentSponsorshipCount() {
final long referenceTimestamp = this.snapshotTimestamp - (365 * 24 * 60 * 60 * 1000L);
int recentSponsorshipCount = 0;
for (RewardShareTransactionData rewardShare : sponsorshipRewardShares) {
if (rewardShare.getTimestamp() >= referenceTimestamp) {
recentSponsorshipCount++;
}
}
this.recentSponsorshipCount = recentSponsorshipCount;
}
private int calculateScore() {
final int suspiciousMultiplier = (this.suspiciousCount >= 100) ? this.suspiciousPercent : 1;
final int nameMultiplier = (this.sponsees.size() >= 50 && this.registeredNameCount == 0) ? 2 : 1;
final int consolidationMultiplier = Math.max(this.consolidationCount, 1);
final int bulkIssuanceMultiplier = Math.max(this.bulkIssuanceCount / 2, 1);
final int offset = 9;
return suspiciousMultiplier * nameMultiplier * consolidationMultiplier * bulkIssuanceMultiplier - offset;
}
private void fetchSponsorshipRewardShares() throws DataException {
List<RewardShareTransactionData> sponsorshipRewardShares = new ArrayList<>();
// Define relevant transactions
List<TransactionType> txTypes = List.of(TransactionType.REWARD_SHARE);
List<TransactionData> transactionDataList = fetchTransactions(repository, txTypes, this.address, false);
for (TransactionData transactionData : transactionDataList) {
if (transactionData.getType() != TransactionType.REWARD_SHARE) {
continue;
}
RewardShareTransactionData rewardShareTransactionData = (RewardShareTransactionData) transactionData;
// Skip removals
if (rewardShareTransactionData.getSharePercent() < 0) {
continue;
}
// Skip if not sponsored by this account
if (!Arrays.equals(rewardShareTransactionData.getCreatorPublicKey(), accountData.getPublicKey())) {
continue;
}
// Skip self shares
if (Objects.equals(rewardShareTransactionData.getRecipient(), this.address)) {
continue;
}
boolean duplicateFound = false;
for (RewardShareTransactionData existingRewardShare : sponsorshipRewardShares) {
if (Objects.equals(existingRewardShare.getRecipient(), rewardShareTransactionData.getRecipient())) {
// Duplicate
duplicateFound = true;
break;
}
}
if (!duplicateFound) {
sponsorshipRewardShares.add(rewardShareTransactionData);
this.sponsees.add(rewardShareTransactionData.getRecipient());
}
}
this.sponsorshipRewardShares = sponsorshipRewardShares;
}
private List<TransactionData> fetchTransferPrivsForAddress(String address) throws DataException {
return fetchTransactions(repository,
List.of(TransactionType.TRANSFER_PRIVS),
address, true);
}
private void fetchPaymentsForAddress(String address) throws DataException {
List<TransactionData> payments = fetchTransactions(repository,
Arrays.asList(TransactionType.PAYMENT, TransactionType.TRANSFER_ASSET),
address, false);
this.paymentsByAddress.put(address, payments);
}
private List<String> fetchOutgoingPaymentRecipientsForAddress(String address) {
List<String> outgoingPaymentRecipients = new ArrayList<>();
List<TransactionData> transactionDataList = this.paymentsByAddress.get(address);
if (transactionDataList == null) transactionDataList = new ArrayList<>();
transactionDataList.removeIf(t -> t.getTimestamp() >= this.snapshotTimestamp);
for (TransactionData transactionData : transactionDataList) {
switch (transactionData.getType()) {
case PAYMENT:
PaymentTransactionData paymentTransactionData = (PaymentTransactionData) transactionData;
if (!Objects.equals(paymentTransactionData.getRecipient(), address)) {
// Outgoing payment from this account
outgoingPaymentRecipients.add(paymentTransactionData.getRecipient());
}
break;
case TRANSFER_ASSET:
TransferAssetTransactionData transferAssetTransactionData = (TransferAssetTransactionData) transactionData;
if (transferAssetTransactionData.getAssetId() == Asset.QORT) {
if (!Objects.equals(transferAssetTransactionData.getRecipient(), address)) {
// Outgoing payment from this account
outgoingPaymentRecipients.add(transferAssetTransactionData.getRecipient());
}
}
break;
default:
break;
}
}
return outgoingPaymentRecipients;
}
private boolean hasZeroTransactions(String address) {
List<TransactionData> transactionDataList = this.paymentsByAddress.get(address);
if (transactionDataList == null) {
return true;
}
transactionDataList.removeIf(t -> t.getTimestamp() >= this.snapshotTimestamp);
return transactionDataList.size() == 0;
}
private static List<TransactionData> fetchTransactions(Repository repository, List<TransactionType> txTypes, String address, boolean reverse) throws DataException {
// Fetch all relevant transactions for this account
List<byte[]> signatures = repository.getTransactionRepository()
.getSignaturesMatchingCriteria(null, null, null, txTypes,
null, null, address, TransactionsResource.ConfirmationStatus.CONFIRMED,
null, null, reverse);
List<TransactionData> transactionDataList = new ArrayList<>();
for (byte[] signature : signatures) {
// Fetch transaction data
TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
if (transactionData == null) {
continue;
}
transactionDataList.add(transactionData);
}
return transactionDataList;
}
}

2
src/main/java/org/qortal/api/ApiError.java
View File

@ -79,7 +79,7 @@ public enum ApiError {
// BUYER_ALREADY_OWNER(411, 422),
// POLLS
// POLL_NO_EXISTS(501, 404),
POLL_NO_EXISTS(501, 404),
// POLL_ALREADY_EXISTS(502, 422),
// DUPLICATE_OPTION(503, 422),
// POLL_OPTION_NO_EXISTS(504, 404),

37
src/main/java/org/qortal/api/ApiRequest.java
View File

@ -3,6 +3,7 @@ package org.qortal.api;
import java.io.IOException;
import java.io.InputStream;
import java.io.UnsupportedEncodingException;
import java.io.Writer;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.Socket;
@ -20,14 +21,12 @@ import javax.net.ssl.SNIHostName;
import javax.net.ssl.SNIServerName;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLSocket;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.UnmarshalException;
import javax.xml.bind.Unmarshaller;
import javax.xml.bind.*;
import javax.xml.transform.stream.StreamSource;
import org.eclipse.persistence.exceptions.XMLMarshalException;
import org.eclipse.persistence.jaxb.JAXBContextFactory;
import org.eclipse.persistence.jaxb.MarshallerProperties;
import org.eclipse.persistence.jaxb.UnmarshallerProperties;
public class ApiRequest {
@ -107,6 +106,36 @@ public class ApiRequest {
}
}
private static Marshaller createMarshaller(Class<?> objectClass) {
try {
// Create JAXB context aware of object's class
JAXBContext jc = JAXBContextFactory.createContext(new Class[] { objectClass }, null);
// Create marshaller
Marshaller marshaller = jc.createMarshaller();
// Set the marshaller media type to JSON
marshaller.setProperty(MarshallerProperties.MEDIA_TYPE, "application/json");
// Tell marshaller not to include JSON root element in the output
marshaller.setProperty(MarshallerProperties.JSON_INCLUDE_ROOT, false);
return marshaller;
} catch (JAXBException e) {
throw new RuntimeException("Unable to create API marshaller", e);
}
}
public static void marshall(Writer writer, Object object) throws IOException {
Marshaller marshaller = createMarshaller(object.getClass());
try {
marshaller.marshal(object, writer);
} catch (JAXBException e) {
throw new IOException("Unable to create marshall object for API", e);
}
}
public static String getParamsString(Map<String, String> params) {
StringBuilder result = new StringBuilder();

28
src/main/java/org/qortal/api/ApiService.java
View File

@ -13,8 +13,8 @@ import java.security.SecureRandom;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.servlet.http.HttpServletRequest;
import org.checkerframework.checker.units.qual.A;
import org.eclipse.jetty.http.HttpVersion;
import org.eclipse.jetty.rewrite.handler.RedirectPatternRule;
import org.eclipse.jetty.rewrite.handler.RewriteHandler;
@ -41,6 +41,7 @@ import org.glassfish.jersey.servlet.ServletContainer;
import org.qortal.api.resource.AnnotationPostProcessor;
import org.qortal.api.resource.ApiDefinition;
import org.qortal.api.websocket.*;
import org.qortal.network.Network;
import org.qortal.settings.Settings;
public class ApiService {
@ -51,9 +52,11 @@ public class ApiService {
private Server server;
private ApiKey apiKey;
public static final String API_VERSION_HEADER = "X-API-VERSION";
private ApiService() {
this.config = new ResourceConfig();
this.config.packages("org.qortal.api.resource");
this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource");
this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class);
@ -93,7 +96,7 @@ public class ApiService {
throw new RuntimeException("Failed to start SSL API due to broken keystore");
// BouncyCastle-specific SSLContext build
SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@ -123,13 +126,13 @@ public class ApiService {
ServerConnector portUnifiedConnector = new ServerConnector(this.server,
new DetectorConnectionFactory(sslConnectionFactory),
httpConnectionFactory);
portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
portUnifiedConnector.setPort(Settings.getInstance().getApiPort());
this.server.addConnector(portUnifiedConnector);
} else {
// Non-SSL
InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getApiPort());
this.server = new Server(endpoint);
}
@ -230,4 +233,19 @@ public class ApiService {
this.server = null;
}
public static int getApiVersion(HttpServletRequest request) {
// Get API version
String apiVersionString = request.getHeader(API_VERSION_HEADER);
if (apiVersionString == null) {
// Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141
apiVersionString = request.getParameter("apiVersion");
}
int apiVersion = 1;
if (apiVersionString != null) {
apiVersion = Integer.parseInt(apiVersionString);
}
return apiVersion;
}
}

173
src/main/java/org/qortal/api/DevProxyService.java Normal file
View File

@ -0,0 +1,173 @@
package org.qortal.api;
import io.swagger.v3.jaxrs2.integration.resources.OpenApiResource;
import org.eclipse.jetty.http.HttpVersion;
import org.eclipse.jetty.rewrite.handler.RewriteHandler;
import org.eclipse.jetty.server.*;
import org.eclipse.jetty.server.handler.ErrorHandler;
import org.eclipse.jetty.server.handler.InetAccessHandler;
import org.eclipse.jetty.servlet.FilterHolder;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.eclipse.jetty.servlets.CrossOriginFilter;
import org.eclipse.jetty.util.ssl.SslContextFactory;
import org.glassfish.jersey.server.ResourceConfig;
import org.glassfish.jersey.servlet.ServletContainer;
import org.qortal.api.resource.AnnotationPostProcessor;
import org.qortal.api.resource.ApiDefinition;
import org.qortal.network.Network;
import org.qortal.repository.DataException;
import org.qortal.settings.Settings;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import java.io.InputStream;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.KeyStore;
import java.security.SecureRandom;
public class DevProxyService {
private static DevProxyService instance;
private final ResourceConfig config;
private Server server;
private DevProxyService() {
this.config = new ResourceConfig();
this.config.packages("org.qortal.api.proxy.resource", "org.qortal.api.resource");
this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class);
}
public static DevProxyService getInstance() {
if (instance == null)
instance = new DevProxyService();
return instance;
}
public Iterable<Class<?>> getResources() {
return this.config.getClasses();
}
public void start() throws DataException {
try {
// Create API server
// SSL support if requested
String keystorePathname = Settings.getInstance().getSslKeystorePathname();
String keystorePassword = Settings.getInstance().getSslKeystorePassword();
if (keystorePathname != null && keystorePassword != null) {
// SSL version
if (!Files.isReadable(Path.of(keystorePathname)))
throw new RuntimeException("Failed to start SSL API due to broken keystore");
// BouncyCastle-specific SSLContext build
SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
try (InputStream keystoreStream = Files.newInputStream(Paths.get(keystorePathname))) {
keyStore.load(keystoreStream, keystorePassword.toCharArray());
}
keyManagerFactory.init(keyStore, keystorePassword.toCharArray());
sslContext.init(keyManagerFactory.getKeyManagers(), null, new SecureRandom());
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setSslContext(sslContext);
this.server = new Server();
HttpConfiguration httpConfig = new HttpConfiguration();
httpConfig.setSecureScheme("https");
httpConfig.setSecurePort(Settings.getInstance().getDevProxyPort());
SecureRequestCustomizer src = new SecureRequestCustomizer();
httpConfig.addCustomizer(src);
HttpConnectionFactory httpConnectionFactory = new HttpConnectionFactory(httpConfig);
SslConnectionFactory sslConnectionFactory = new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString());
ServerConnector portUnifiedConnector = new ServerConnector(this.server,
new DetectorConnectionFactory(sslConnectionFactory),
httpConnectionFactory);
portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
portUnifiedConnector.setPort(Settings.getInstance().getDevProxyPort());
this.server.addConnector(portUnifiedConnector);
} else {
// Non-SSL
InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getDevProxyPort());
this.server = new Server(endpoint);
}
// Error handler
ErrorHandler errorHandler = new ApiErrorHandler();
this.server.setErrorHandler(errorHandler);
// Request logging
if (Settings.getInstance().isDevProxyLoggingEnabled()) {
RequestLogWriter logWriter = new RequestLogWriter("devproxy-requests.log");
logWriter.setAppend(true);
logWriter.setTimeZone("UTC");
RequestLog requestLog = new CustomRequestLog(logWriter, CustomRequestLog.EXTENDED_NCSA_FORMAT);
this.server.setRequestLog(requestLog);
}
// Access handler (currently no whitelist is used)
InetAccessHandler accessHandler = new InetAccessHandler();
this.server.setHandler(accessHandler);
// URL rewriting
RewriteHandler rewriteHandler = new RewriteHandler();
accessHandler.setHandler(rewriteHandler);
// Context
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
context.setContextPath("/");
rewriteHandler.setHandler(context);
// Cross-origin resource sharing
FilterHolder corsFilterHolder = new FilterHolder(CrossOriginFilter.class);
corsFilterHolder.setInitParameter(CrossOriginFilter.ALLOWED_ORIGINS_PARAM, "*");
corsFilterHolder.setInitParameter(CrossOriginFilter.ALLOWED_METHODS_PARAM, "GET, POST, DELETE");
corsFilterHolder.setInitParameter(CrossOriginFilter.CHAIN_PREFLIGHT_PARAM, "false");
context.addFilter(corsFilterHolder, "/*", null);
// API servlet
ServletContainer container = new ServletContainer(this.config);
ServletHolder apiServlet = new ServletHolder(container);
apiServlet.setInitOrder(1);
context.addServlet(apiServlet, "/*");
// Start server
this.server.start();
} catch (Exception e) {
// Failed to start
throw new DataException("Failed to start developer proxy", e);
}
}
public void stop() {
try {
// Stop server
this.server.stop();
} catch (Exception e) {
// Failed to stop
}
this.server = null;
instance = null;
}
}

10
src/main/java/org/qortal/api/DomainMapService.java
View File

@ -3,7 +3,6 @@ package org.qortal.api;
import io.swagger.v3.jaxrs2.integration.resources.OpenApiResource;
import org.eclipse.jetty.http.HttpVersion;
import org.eclipse.jetty.rewrite.handler.RewriteHandler;
import org.eclipse.jetty.rewrite.handler.RewritePatternRule;
import org.eclipse.jetty.server.*;
import org.eclipse.jetty.server.handler.ErrorHandler;
import org.eclipse.jetty.server.handler.InetAccessHandler;
@ -16,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig;
import org.glassfish.jersey.servlet.ServletContainer;
import org.qortal.api.resource.AnnotationPostProcessor;
import org.qortal.api.resource.ApiDefinition;
import org.qortal.network.Network;
import org.qortal.settings.Settings;
import javax.net.ssl.KeyManagerFactory;
@ -38,7 +38,7 @@ public class DomainMapService {
private DomainMapService() {
this.config = new ResourceConfig();
this.config.packages("org.qortal.api.domainmap.resource");
this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource");
this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class);
@ -69,7 +69,7 @@ public class DomainMapService {
throw new RuntimeException("Failed to start SSL API due to broken keystore");
// BouncyCastle-specific SSLContext build
SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@ -99,13 +99,13 @@ public class DomainMapService {
ServerConnector portUnifiedConnector = new ServerConnector(this.server,
new DetectorConnectionFactory(sslConnectionFactory),
httpConnectionFactory);
portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
portUnifiedConnector.setPort(Settings.getInstance().getDomainMapPort());
this.server.addConnector(portUnifiedConnector);
} else {
// Non-SSL
InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getDomainMapPort());
this.server = new Server(endpoint);
}

9
src/main/java/org/qortal/api/GatewayService.java
View File

@ -15,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig;
import org.glassfish.jersey.servlet.ServletContainer;
import org.qortal.api.resource.AnnotationPostProcessor;
import org.qortal.api.resource.ApiDefinition;
import org.qortal.network.Network;
import org.qortal.settings.Settings;
import javax.net.ssl.KeyManagerFactory;
@ -37,7 +38,7 @@ public class GatewayService {
private GatewayService() {
this.config = new ResourceConfig();
this.config.packages("org.qortal.api.gateway.resource");
this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource");
this.config.register(OpenApiResource.class);
this.config.register(ApiDefinition.class);
this.config.register(AnnotationPostProcessor.class);
@ -68,7 +69,7 @@ public class GatewayService {
throw new RuntimeException("Failed to start SSL API due to broken keystore");
// BouncyCastle-specific SSLContext build
SSLContext sslContext = SSLContext.getInstance("TLS", "BCJSSE");
SSLContext sslContext = SSLContext.getInstance("TLSv1.3", "BCJSSE");
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX", "BCJSSE");
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType(), "BC");
@ -98,13 +99,13 @@ public class GatewayService {
ServerConnector portUnifiedConnector = new ServerConnector(this.server,
new DetectorConnectionFactory(sslConnectionFactory),
httpConnectionFactory);
portUnifiedConnector.setHost(Settings.getInstance().getBindAddress());
portUnifiedConnector.setHost(Network.getInstance().getBindAddress());
portUnifiedConnector.setPort(Settings.getInstance().getGatewayPort());
this.server.addConnector(portUnifiedConnector);
} else {
// Non-SSL
InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress());
InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress());
InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getGatewayPort());
this.server = new Server(endpoint);
}

57
src/main/java/org/qortal/api/HTMLParser.java
View File

@ -5,28 +5,71 @@ import org.apache.logging.log4j.Logger;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.qortal.arbitrary.misc.Service;
import java.util.Objects;
public class HTMLParser {
private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class);
private String linkPrefix;
private String qdnBase;
private String qdnBaseWithPath;
private byte[] data;
private String qdnContext;
private String resourceId;
private Service service;
private String identifier;
private String path;
private String theme;
private boolean usingCustomRouting;
public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data) {
String inPathWithoutFilename = inPath.substring(0, inPath.lastIndexOf('/'));
this.linkPrefix = usePrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : "";
public HTMLParser(String resourceId, String inPath, String prefix, boolean includeResourceIdInPrefix, byte[] data,
String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting) {
String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : String.format("/%s",inPath);
this.qdnBase = includeResourceIdInPrefix ? String.format("%s/%s", prefix, resourceId) : prefix;
this.qdnBaseWithPath = includeResourceIdInPrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : String.format("%s%s", prefix, inPathWithoutFilename);
this.data = data;
this.qdnContext = qdnContext;
this.resourceId = resourceId;
this.service = service;
this.identifier = identifier;
this.path = inPath;
this.theme = theme;
this.usingCustomRouting = usingCustomRouting;
}
public void addAdditionalHeaderTags() {
String fileContents = new String(data);
Document document = Jsoup.parse(fileContents);
String baseUrl = this.linkPrefix + "/";
Elements head = document.getElementsByTag("head");
if (!head.isEmpty()) {
// Add q-apps script tag
String qAppsScriptElement = String.format("<script src=\"/apps/q-apps.js?time=%d\">", System.currentTimeMillis());
head.get(0).prepend(qAppsScriptElement);
// Add q-apps gateway script tag if in gateway mode
if (Objects.equals(this.qdnContext, "gateway")) {
String qAppsGatewayScriptElement = String.format("<script src=\"/apps/q-apps-gateway.js?time=%d\">", System.currentTimeMillis());
head.get(0).prepend(qAppsGatewayScriptElement);
}
// Escape and add vars
String qdnContext = this.qdnContext != null ? this.qdnContext.replace("\\", "").replace("\"","\\\"") : "";
String service = this.service.toString().replace("\\", "").replace("\"","\\\"");
String name = this.resourceId != null ? this.resourceId.replace("\\", "").replace("\"","\\\"") : "";
String identifier = this.identifier != null ? this.identifier.replace("\\", "").replace("\"","\\\"") : "";
String path = this.path != null ? this.path.replace("\\", "").replace("\"","\\\"") : "";
String theme = this.theme != null ? this.theme.replace("\\", "").replace("\"","\\\"") : "";
String qdnBase = this.qdnBase != null ? this.qdnBase.replace("\\", "").replace("\"","\\\"") : "";
String qdnBaseWithPath = this.qdnBaseWithPath != null ? this.qdnBaseWithPath.replace("\\", "").replace("\"","\\\"") : "";
String qdnContextVar = String.format("<script>var _qdnContext=\"%s\"; var _qdnTheme=\"%s\"; var _qdnService=\"%s\"; var _qdnName=\"%s\"; var _qdnIdentifier=\"%s\"; var _qdnPath=\"%s\"; var _qdnBase=\"%s\"; var _qdnBaseWithPath=\"%s\";</script>", qdnContext, theme, service, name, identifier, path, qdnBase, qdnBaseWithPath);
head.get(0).prepend(qdnContextVar);
// Add base href tag
String baseElement = String.format("<base href=\"%s\">", baseUrl);
// Exclude the path if this request was routed back to the index automatically
String baseHref = this.usingCustomRouting ? this.qdnBase : this.qdnBaseWithPath;
String baseElement = String.format("<base href=\"%s/\">", baseHref);
head.get(0).prepend(baseElement);
// Add meta charset tag
@ -39,7 +82,7 @@ public class HTMLParser {
}
public static boolean isHtmlFile(String path) {
if (path.endsWith(".html") || path.endsWith(".htm")) {
if (path.endsWith(".html") || path.endsWith(".htm") || path.equals("")) {
return true;
}
return false;

25
src/main/java/org/qortal/api/Security.java
View File

@ -15,7 +15,21 @@ public abstract class Security {
public static final String API_KEY_HEADER = "X-API-KEY";
/**
* Check API call is allowed, retrieving the API key from the request header or GET/POST parameters where required
* @param request
*/
public static void checkApiCallAllowed(HttpServletRequest request) {
checkApiCallAllowed(request, null);
}
/**
* Check API call is allowed, retrieving the API key first from the passedApiKey parameter, with a fallback
* to the request header or GET/POST parameters when null.
* @param request
* @param passedApiKey - the API key to test, or null if it should be retrieved from the request headers.
*/
public static void checkApiCallAllowed(HttpServletRequest request, String passedApiKey) {
// We may want to allow automatic authentication for local requests, if enabled in settings
boolean localAuthBypassEnabled = Settings.getInstance().isLocalAuthBypassEnabled();
if (localAuthBypassEnabled) {
@ -38,7 +52,10 @@ public abstract class Security {
}
// We require an API key to be passed
String passedApiKey = request.getHeader(API_KEY_HEADER);
if (passedApiKey == null) {
// API call not passed as a parameter, so try the header
passedApiKey = request.getHeader(API_KEY_HEADER);
}
if (passedApiKey == null) {
// Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141
passedApiKey = request.getParameter("apiKey");
@ -56,7 +73,7 @@ public abstract class Security {
public static void disallowLoopbackRequests(HttpServletRequest request) {
try {
InetAddress remoteAddr = InetAddress.getByName(request.getRemoteAddr());
if (remoteAddr.isLoopbackAddress()) {
if (remoteAddr.isLoopbackAddress() && !Settings.getInstance().isGatewayLoopbackEnabled()) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.UNAUTHORIZED, "Local requests not allowed");
}
} catch (UnknownHostException e) {
@ -84,9 +101,9 @@ public abstract class Security {
}
}
public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier) {
public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier, String apiKey) {
try {
Security.checkApiCallAllowed(request);
Security.checkApiCallAllowed(request, apiKey);
} catch (ApiException e) {
// API call wasn't allowed, but maybe it was pre-authorized

10
src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java
View File

@ -42,16 +42,16 @@ public class DomainMapResource {
// Build synchronously, so that we don't need to make the summary API endpoints available over
// the domain map server. This means that there will be no loading screen, but this is potentially
// preferred in this situation anyway (e.g. to avoid confusing search engine robots).
return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", false, false);
return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, null, inPath, null, "", false, false);
}
return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
}
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
String secret58, String prefix, boolean usePrefix, boolean async) {
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async) {
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
secret58, prefix, usePrefix, async, request, response, context);
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
secret58, prefix, includeResourceIdInPrefix, async, "domainMap", request, response, context);
return renderer.render();
}

112
src/main/java/org/qortal/api/gateway/resource/GatewayResource.java
View File

@ -2,6 +2,7 @@ package org.qortal.api.gateway.resource;
import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.commons.lang3.StringUtils;
import org.qortal.api.Security;
import org.qortal.arbitrary.ArbitraryDataFile;
import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType;
@ -16,6 +17,10 @@ import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
@Path("/")
@ -76,50 +81,83 @@ public class GatewayResource {
@GET
@Path("{name}/{path:.*}")
@Path("{path:.*}")
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getPathByName(@PathParam("name") String name,
@PathParam("path") String inPath) {
public HttpServletResponse getPath(@PathParam("path") String inPath) {
// Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
Security.disallowLoopbackRequests(request);
return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", true, true);
}
@GET
@Path("{name}")
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexByName(@PathParam("name") String name) {
// Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
Security.disallowLoopbackRequests(request);
return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "", true, true);
}
// Optional /site alternative for backwards support
@GET
@Path("/site/{name}/{path:.*}")
public HttpServletResponse getSitePathByName(@PathParam("name") String name,
@PathParam("path") String inPath) {
// Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
Security.disallowLoopbackRequests(request);
return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "/site", true, true);
}
@GET
@Path("/site/{name}")
public HttpServletResponse getSiteIndexByName(@PathParam("name") String name) {
// Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data
Security.disallowLoopbackRequests(request);
return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "/site", true, true);
return this.parsePath(inPath, "gateway", null, true, true);
}
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
String secret58, String prefix, boolean usePrefix, boolean async) {
private HttpServletResponse parsePath(String inPath, String qdnContext, String secret58, boolean includeResourceIdInPrefix, boolean async) {
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
secret58, prefix, usePrefix, async, request, response, context);
if (inPath == null || inPath.equals("")) {
// Assume not a real file
return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
}
// Default service is WEBSITE
Service service = Service.WEBSITE;
String name = null;
String identifier = null;
String outPath = "";
List<String> prefixParts = new ArrayList<>();
if (!inPath.contains("/")) {
// Assume entire inPath is a registered name
name = inPath;
}
else {
// Parse the path to determine what we need to load
List<String> parts = new LinkedList<>(Arrays.asList(inPath.split("/")));
// Check if the first element is a service
try {
Service parsedService = Service.valueOf(parts.get(0).toUpperCase());
if (parsedService != null) {
// First element matches a service, so we can assume it is one
service = parsedService;
parts.remove(0);
prefixParts.add(service.name());
}
} catch (IllegalArgumentException e) {
// Not a service
}
if (parts.isEmpty()) {
// We need more than just a service
return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found");
}
// Service is removed, so assume first element is now a registered name
name = parts.get(0);
parts.remove(0);
if (!parts.isEmpty()) {
// Name is removed, so check if the first element is now an identifier
ArbitraryResourceStatus status = this.getStatus(service, name, parts.get(0), false);
if (status.getTotalChunkCount() > 0) {
// Matched service, name and identifier combination - so assume this is an identifier and can be removed
identifier = parts.get(0);
parts.remove(0);
prefixParts.add(identifier);
}
}
if (!parts.isEmpty()) {
// outPath can be built by combining any remaining parts
outPath = String.join("/", parts);
}
}
String prefix = StringUtils.join(prefixParts, "/");
if (prefix != null && prefix.length() > 0) {
prefix = "/" + prefix;
}
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(name, ResourceIdType.NAME, service, identifier, outPath,
secret58, prefix, includeResourceIdInPrefix, async, qdnContext, request, response, context);
return renderer.render();
}

56
src/main/java/org/qortal/api/model/AccountPenaltyStats.java Normal file
View File

@ -0,0 +1,56 @@
package org.qortal.api.model;
import org.qortal.block.SelfSponsorshipAlgoV1Block;
import org.qortal.data.account.AccountData;
import org.qortal.data.naming.NameData;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import java.util.ArrayList;
import java.util.List;
@XmlAccessorType(XmlAccessType.FIELD)
public class AccountPenaltyStats {
public Integer totalPenalties;
public Integer maxPenalty;
public Integer minPenalty;
public String penaltyHash;
protected AccountPenaltyStats() {
}
public AccountPenaltyStats(Integer totalPenalties, Integer maxPenalty, Integer minPenalty, String penaltyHash) {
this.totalPenalties = totalPenalties;
this.maxPenalty = maxPenalty;
this.minPenalty = minPenalty;
this.penaltyHash = penaltyHash;
}
public static AccountPenaltyStats fromAccounts(List<AccountData> accounts) {
int totalPenalties = 0;
Integer maxPenalty = null;
Integer minPenalty = null;
List<String> addresses = new ArrayList<>();
for (AccountData accountData : accounts) {
int penalty = accountData.getBlocksMintedPenalty();
addresses.add(accountData.getAddress());
totalPenalties++;
// Penalties are expressed as a negative number, so the min and the max are reversed here
if (maxPenalty == null || penalty < maxPenalty) maxPenalty = penalty;
if (minPenalty == null || penalty > minPenalty) minPenalty = penalty;
}
String penaltyHash = SelfSponsorshipAlgoV1Block.getHash(addresses);
return new AccountPenaltyStats(totalPenalties, maxPenalty, minPenalty, penaltyHash);
}
@Override
public String toString() {
return String.format("totalPenalties: %d, maxPenalty: %d, minPenalty: %d, penaltyHash: %s", totalPenalties, maxPenalty, minPenalty, penaltyHash == null ? "null" : penaltyHash);
}
}

102
src/main/java/org/qortal/api/model/AtCreationRequest.java Normal file
View File

@ -0,0 +1,102 @@
package org.qortal.api.model;
import io.swagger.v3.oas.annotations.media.Schema;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlTransient;
import org.bouncycastle.util.encoders.Base64;
import org.bouncycastle.util.encoders.DecoderException;
@XmlAccessorType(XmlAccessType.FIELD)
public class AtCreationRequest {
@Schema(description = "CIYAM AT version", example = "2")
private short ciyamAtVersion;
@Schema(description = "base64-encoded code bytes", example = "")
private String codeBytesBase64;
@Schema(description = "base64-encoded data bytes", example = "")
private String dataBytesBase64;
private short numCallStackPages;
private short numUserStackPages;
private long minActivationAmount;
// Default constructor for JSON deserialization
public AtCreationRequest() {}
// Getters and setters
public short getCiyamAtVersion() {
return ciyamAtVersion;
}
public void setCiyamAtVersion(short ciyamAtVersion) {
this.ciyamAtVersion = ciyamAtVersion;
}
public String getCodeBytesBase64() {
return this.codeBytesBase64;
}
@XmlTransient
@Schema(hidden = true)
public byte[] getCodeBytes() {
if (this.codeBytesBase64 != null) {
try {
return Base64.decode(this.codeBytesBase64);
}
catch (DecoderException e) {
return null;
}
}
return null;
}
public String getDataBytesBase64() {
return this.dataBytesBase64;
}
@XmlTransient
@Schema(hidden = true)
public byte[] getDataBytes() {
if (this.dataBytesBase64 != null) {
try {
return Base64.decode(this.dataBytesBase64);
}
catch (DecoderException e) {
return null;
}
}
return null;
}
public short getNumCallStackPages() {
return numCallStackPages;
}
public void setNumCallStackPages(short numCallStackPages) {
this.numCallStackPages = numCallStackPages;
}
public short getNumUserStackPages() {
return numUserStackPages;
}
public void setNumUserStackPages(short numUserStackPages) {
this.numUserStackPages = numUserStackPages;
}
public long getMinActivationAmount() {
return minActivationAmount;
}
public void setMinActivationAmount(long minActivationAmount) {
this.minActivationAmount = minActivationAmount;
}
}

7
src/main/java/org/qortal/api/model/ConnectedPeer.java
View File

@ -1,6 +1,7 @@
package org.qortal.api.model;
import io.swagger.v3.oas.annotations.media.Schema;
import org.qortal.controller.Controller;
import org.qortal.data.block.BlockSummaryData;
import org.qortal.data.network.PeerData;
import org.qortal.network.Handshake;
@ -36,6 +37,7 @@ public class ConnectedPeer {
public Long lastBlockTimestamp;
public UUID connectionId;
public String age;
public Boolean isTooDivergent;
protected ConnectedPeer() {
}
@ -69,6 +71,11 @@ public class ConnectedPeer {
this.lastBlockSignature = peerChainTipData.getSignature();
this.lastBlockTimestamp = peerChainTipData.getTimestamp();
}
// Only include isTooDivergent decision if we've had the opportunity to request block summaries this peer
if (peer.getLastTooDivergentTime() != null) {
this.isTooDivergent = Controller.wasRecentlyTooDivergent.test(peer);
}
}
}

16
src/main/java/org/qortal/api/model/FileProperties.java Normal file
View File

@ -0,0 +1,16 @@
package org.qortal.api.model;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
@XmlAccessorType(XmlAccessType.FIELD)
public class FileProperties {
public String filename;
public String mimeType;
public Long size;
public FileProperties() {
}
}

56
src/main/java/org/qortal/api/model/PollVotes.java Normal file
View File

@ -0,0 +1,56 @@
package org.qortal.api.model;
import java.util.List;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import io.swagger.v3.oas.annotations.media.Schema;
import org.qortal.data.voting.VoteOnPollData;
@Schema(description = "Poll vote info, including voters")
// All properties to be converted to JSON via JAX-RS
@XmlAccessorType(XmlAccessType.FIELD)
public class PollVotes {
@Schema(description = "List of individual votes")
@XmlElement(name = "votes")
public List<VoteOnPollData> votes;
@Schema(description = "Total number of votes")
public Integer totalVotes;
@Schema(description = "List of vote counts for each option")
public List<OptionCount> voteCounts;
// For JAX-RS
protected PollVotes() {
}
public PollVotes(List<VoteOnPollData> votes, Integer totalVotes, List<OptionCount> voteCounts) {
this.votes = votes;
this.totalVotes = totalVotes;
this.voteCounts = voteCounts;
}
@Schema(description = "Vote info")
// All properties to be converted to JSON via JAX-RS
@XmlAccessorType(XmlAccessType.FIELD)
public static class OptionCount {
@Schema(description = "Option name")
public String optionName;
@Schema(description = "Vote count")
public Integer voteCount;
// For JAX-RS
protected OptionCount() {
}
public OptionCount(String optionName, Integer voteCount) {
this.optionName = optionName;
this.voteCount = voteCount;
}
}
}

164
src/main/java/org/qortal/api/proxy/resource/DevProxyServerResource.java Normal file
View File

@ -0,0 +1,164 @@
package org.qortal.api.proxy.resource;
import org.qortal.api.ApiError;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.api.HTMLParser;
import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.DevProxyManager;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.core.Context;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.ConnectException;
import java.net.HttpURLConnection;
import java.net.ProtocolException;
import java.net.URL;
import java.util.Enumeration;
@Path("/")
public class DevProxyServerResource {
@Context HttpServletRequest request;
@Context HttpServletResponse response;
@Context ServletContext context;
@GET
public HttpServletResponse getProxyIndex() {
return this.proxy("/");
}
@GET
@Path("{path:.*}")
public HttpServletResponse getProxyPath(@PathParam("path") String inPath) {
return this.proxy(inPath);
}
private HttpServletResponse proxy(String inPath) {
try {
String source = DevProxyManager.getInstance().getSourceHostAndPort();
if (!inPath.startsWith("/")) {
inPath = "/" + inPath;
}
String queryString = request.getQueryString() != null ? "?" + request.getQueryString() : "";
// Open URL
URL url = new URL(String.format("http://%s%s%s", source, inPath, queryString));
HttpURLConnection con = (HttpURLConnection) url.openConnection();
// Proxy the request data
this.proxyRequestToConnection(request, con);
try {
// Make the request and proxy the response code
response.setStatus(con.getResponseCode());
}
catch (ConnectException e) {
// Tey converting localhost / 127.0.0.1 to IPv6 [::1]
if (source.startsWith("localhost") || source.startsWith("127.0.0.1")) {
int port = 80;
String[] parts = source.split(":");
if (parts.length > 1) {
port = Integer.parseInt(parts[1]);
}
source = String.format("[::1]:%d", port);
}
// Retry connection
url = new URL(String.format("http://%s%s%s", source, inPath, queryString));
con = (HttpURLConnection) url.openConnection();
this.proxyRequestToConnection(request, con);
response.setStatus(con.getResponseCode());
}
// Proxy the response data back to the caller
this.proxyConnectionToResponse(con, response, inPath);
} catch (IOException e) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, e.getMessage());
}
return response;
}
private void proxyRequestToConnection(HttpServletRequest request, HttpURLConnection con) throws ProtocolException {
// Proxy the request method
con.setRequestMethod(request.getMethod());
// Proxy the request headers
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = request.getHeader(headerName);
con.setRequestProperty(headerName, headerValue);
}
// TODO: proxy any POST parameters from "request" to "con"
}
private void proxyConnectionToResponse(HttpURLConnection con, HttpServletResponse response, String inPath) throws IOException {
// Proxy the response headers
for (int i = 0; ; i++) {
String headerKey = con.getHeaderFieldKey(i);
String headerValue = con.getHeaderField(i);
if (headerKey != null && headerValue != null) {
response.addHeader(headerKey, headerValue);
continue;
}
break;
}
// Read the response body
InputStream inputStream = con.getInputStream();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
byte[] data = outputStream.toByteArray(); // TODO: limit file size that can be read into memory
// Close the streams
outputStream.close();
inputStream.close();
// Extract filename
String filename = "";
if (inPath.contains("/")) {
String[] parts = inPath.split("/");
if (parts.length > 0) {
filename = parts[parts.length - 1];
}
}
// Parse and modify output if needed
if (HTMLParser.isHtmlFile(filename)) {
// HTML file - needs to be parsed
HTMLParser htmlParser = new HTMLParser("", inPath, "", false, data, "proxy", Service.APP, null, "light", true);
htmlParser.addAdditionalHeaderTags();
response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:; connect-src 'self' ws:; font-src 'self' data:;");
response.setContentType(con.getContentType());
response.setContentLength(htmlParser.getData().length);
response.getOutputStream().write(htmlParser.getData());
}
else {
// Regular file - can be streamed directly
response.addHeader("Content-Security-Policy", "default-src 'self'");
response.setContentType(con.getContentType());
response.setContentLength(data.length);
response.getOutputStream().write(data);
}
}
}

51
src/main/java/org/qortal/api/resource/AddressesResource.java
View File

@ -14,6 +14,7 @@ import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.*;
@ -27,6 +28,7 @@ import org.qortal.api.ApiErrors;
import org.qortal.api.ApiException;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.api.Security;
import org.qortal.api.model.AccountPenaltyStats;
import org.qortal.api.model.ApiOnlineAccount;
import org.qortal.api.model.RewardShareKeyRequest;
import org.qortal.asset.Asset;
@ -34,6 +36,7 @@ import org.qortal.controller.LiteNode;
import org.qortal.controller.OnlineAccountsManager;
import org.qortal.crypto.Crypto;
import org.qortal.data.account.AccountData;
import org.qortal.data.account.AccountPenaltyData;
import org.qortal.data.account.RewardShareData;
import org.qortal.data.network.OnlineAccountData;
import org.qortal.data.network.OnlineAccountLevel;
@ -471,6 +474,54 @@ public class AddressesResource {
}
}
@GET
@Path("/penalties")
@Operation(
summary = "Get addresses with penalties",
description = "Returns a list of accounts with a blocksMintedPenalty",
responses = {
@ApiResponse(
description = "accounts with penalties",
content = @Content(mediaType = MediaType.APPLICATION_JSON, array = @ArraySchema(schema = @Schema(implementation = AccountPenaltyData.class)))
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public List<AccountPenaltyData> getAccountsWithPenalties() {
try (final Repository repository = RepositoryManager.getRepository()) {
List<AccountData> accounts = repository.getAccountRepository().getPenaltyAccounts();
List<AccountPenaltyData> penalties = accounts.stream().map(a -> new AccountPenaltyData(a.getAddress(), a.getBlocksMintedPenalty())).collect(Collectors.toList());
return penalties;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/penalties/stats")
@Operation(
summary = "Get stats about current penalties",
responses = {
@ApiResponse(
description = "aggregated stats about accounts with penalties",
content = @Content(mediaType = MediaType.APPLICATION_JSON, array = @ArraySchema(schema = @Schema(implementation = AccountPenaltyStats.class)))
)
}
)
@ApiErrors({ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public AccountPenaltyStats getPenaltyStats() {
try (final Repository repository = RepositoryManager.getRepository()) {
List<AccountData> accounts = repository.getAccountRepository().getPenaltyAccounts();
return AccountPenaltyStats.fromAccounts(accounts);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@POST
@Path("/publicize")
@Operation(

83
src/main/java/org/qortal/api/resource/AppsResource.java Normal file
View File

@ -0,0 +1,83 @@
package org.qortal.api.resource;
import com.google.common.io.Resources;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.Hidden;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.qortal.api.*;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import java.io.IOException;
import java.net.URL;
import java.nio.charset.StandardCharsets;
@Path("/apps")
@Tag(name = "Apps")
public class AppsResource {
@Context HttpServletRequest request;
@Context HttpServletResponse response;
@Context ServletContext context;
@GET
@Path("/q-apps.js")
@Hidden // For internal Q-App API use only
@Operation(
summary = "Javascript interface for Q-Apps",
responses = {
@ApiResponse(
description = "javascript",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
public String getQAppsJs() {
URL url = Resources.getResource("q-apps/q-apps.js");
try {
return Resources.toString(url, StandardCharsets.UTF_8);
} catch (IOException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
}
}
@GET
@Path("/q-apps-gateway.js")
@Hidden // For internal Q-App API use only
@Operation(
summary = "Gateway-specific interface for Q-Apps",
responses = {
@ApiResponse(
description = "javascript",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
public String getQAppsGatewayJs() {
URL url = Resources.getResource("q-apps/q-apps-gateway.js");
try {
return Resources.toString(url, StandardCharsets.UTF_8);
} catch (IOException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
}
}
}

421
src/main/java/org/qortal/api/resource/ArbitraryResource.java
View File

@ -1,6 +1,8 @@
package org.qortal.api.resource;
import com.google.common.primitives.Bytes;
import com.j256.simplemagic.ContentInfo;
import com.j256.simplemagic.ContentInfoUtil;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.ArraySchema;
@ -12,11 +14,14 @@ import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.io.*;
import java.net.FileNameMap;
import java.net.URLConnection;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
@ -25,11 +30,13 @@ import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.bouncycastle.util.encoders.Base64;
import org.qortal.api.*;
import org.qortal.api.model.FileProperties;
import org.qortal.api.resource.TransactionsResource.ConfirmationStatus;
import org.qortal.arbitrary.*;
import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType;
@ -38,6 +45,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
import org.qortal.arbitrary.misc.Category;
import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.Controller;
import org.qortal.controller.arbitrary.ArbitraryDataRenderManager;
import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
import org.qortal.controller.arbitrary.ArbitraryMetadataManager;
import org.qortal.data.account.AccountData;
@ -57,10 +65,7 @@ import org.qortal.transaction.Transaction.ValidationResult;
import org.qortal.transform.TransformationException;
import org.qortal.transform.transaction.ArbitraryTransactionTransformer;
import org.qortal.transform.transaction.TransactionTransformer;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58;
import org.qortal.utils.NTP;
import org.qortal.utils.ZipUtils;
import org.qortal.utils.*;
@Path("/arbitrary")
@Tag(name = "Arbitrary")
@ -88,12 +93,15 @@ public class ArbitraryResource {
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<ArbitraryResourceInfo> getResources(
@QueryParam("service") Service service,
@QueryParam("name") String name,
@QueryParam("identifier") String identifier,
@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
@Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameFilter,
@Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly,
@Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked,
@Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameListFilter,
@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
@ -110,28 +118,33 @@ public class ArbitraryResource {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource");
}
// Load filter from list if needed
// Set up name filters if supplied
List<String> names = null;
if (nameFilter != null) {
names = ResourceListManager.getInstance().getStringsInList(nameFilter);
if (name != null) {
// Filter using single name
names = Arrays.asList(name);
}
else if (nameListFilter != null) {
// Filter using supplied list of names
names = ResourceListManager.getInstance().getStringsInList(nameListFilter);
if (names.isEmpty()) {
// List doesn't exist or is empty - so there will be no matches
// If list is empty (or doesn't exist) we can shortcut with empty response
return new ArrayList<>();
}
}
List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
.getArbitraryResources(service, identifier, names, defaultRes, limit, offset, reverse);
.getArbitraryResources(service, identifier, names, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse);
if (resources == null) {
return new ArrayList<>();
}
if (includeStatus != null && includeStatus) {
resources = this.addStatusToResources(resources);
resources = ArbitraryTransactionUtils.addStatusToResources(resources);
}
if (includeMetadata != null && includeMetadata) {
resources = this.addMetadataToResources(resources);
resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
}
return resources;
@ -155,30 +168,56 @@ public class ArbitraryResource {
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<ArbitraryResourceInfo> searchResources(
@QueryParam("service") Service service,
@QueryParam("query") String query,
@Parameter(description = "Query (searches both name and identifier fields)") @QueryParam("query") String query,
@Parameter(description = "Identifier (searches identifier field only)") @QueryParam("identifier") String identifier,
@Parameter(description = "Name (searches name field only)") @QueryParam("name") List<String> names,
@Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly,
@Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly,
@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
@Parameter(description = "Filter names by list (exact matches only)") @QueryParam("namefilter") String nameListFilter,
@Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly,
@Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked,
@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
try (final Repository repository = RepositoryManager.getRepository()) {
boolean defaultRes = Boolean.TRUE.equals(defaultResource);
boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
List<String> exactMatchNames = new ArrayList<>();
if (nameListFilter != null) {
// Load names from supplied list of names
exactMatchNames.addAll(ResourceListManager.getInstance().getStringsInList(nameListFilter));
// If list is empty (or doesn't exist) we can shortcut with empty response
if (exactMatchNames.isEmpty()) {
return new ArrayList<>();
}
}
// Move names to exact match list, if requested
if (exactMatchNamesOnly != null && exactMatchNamesOnly && names != null) {
exactMatchNames.addAll(names);
names = null;
}
List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
.searchArbitraryResources(service, query, defaultRes, limit, offset, reverse);
.searchArbitraryResources(service, query, identifier, names, usePrefixOnly, exactMatchNames, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse);
if (resources == null) {
return new ArrayList<>();
}
if (includeStatus != null && includeStatus) {
resources = this.addStatusToResources(resources);
resources = ArbitraryTransactionUtils.addStatusToResources(resources);
}
if (includeMetadata != null && includeMetadata) {
resources = this.addMetadataToResources(resources);
resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
}
return resources;
@ -188,67 +227,6 @@ public class ArbitraryResource {
}
}
@GET
@Path("/resources/names")
@Operation(
summary = "List arbitrary resources available on chain, grouped by creator's name",
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = ArbitraryResourceInfo.class))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<ArbitraryResourceNameInfo> getResourcesGroupedByName(
@QueryParam("service") Service service,
@QueryParam("identifier") String identifier,
@Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
@Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus,
@Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) {
try (final Repository repository = RepositoryManager.getRepository()) {
// Treat empty identifier as null
if (identifier != null && identifier.isEmpty()) {
identifier = null;
}
// Ensure that "default" and "identifier" parameters cannot coexist
boolean defaultRes = Boolean.TRUE.equals(defaultResource);
if (defaultRes == true && identifier != null) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource");
}
List<ArbitraryResourceNameInfo> creatorNames = repository.getArbitraryRepository()
.getArbitraryResourceCreatorNames(service, identifier, defaultRes, limit, offset, reverse);
for (ArbitraryResourceNameInfo creatorName : creatorNames) {
String name = creatorName.name;
if (name != null) {
List<ArbitraryResourceInfo> resources = repository.getArbitraryRepository()
.getArbitraryResources(service, identifier, Arrays.asList(name), defaultRes, null, null, reverse);
if (includeStatus != null && includeStatus) {
resources = this.addStatusToResources(resources);
}
if (includeMetadata != null && includeMetadata) {
resources = this.addMetadataToResources(resources);
}
creatorName.resources = resources;
}
}
return creatorNames;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/resource/status/{service}/{name}")
@Operation(
@ -266,10 +244,35 @@ public class ArbitraryResource {
@PathParam("name") String name,
@QueryParam("build") Boolean build) {
Security.requirePriorAuthorizationOrApiKey(request, name, service, null);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorizationOrApiKey(request, name, service, null, apiKey);
return ArbitraryTransactionUtils.getStatus(service, name, null, build);
}
@GET
@Path("/resource/properties/{service}/{name}/{identifier}")
@Operation(
summary = "Get properties of a QDN resource",
description = "This attempts a download of the data if it's not available locally. A filename will only be returned for single file resources. mimeType is only returned when it can be determined.",
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = FileProperties.class))
)
}
)
@SecurityRequirement(name = "apiKey")
public FileProperties getResourceProperties(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") Service service,
@PathParam("name") String name,
@PathParam("identifier") String identifier) {
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey);
return this.getFileProperties(service, name, identifier);
}
@GET
@Path("/resource/status/{service}/{name}/{identifier}")
@Operation(
@ -288,7 +291,9 @@ public class ArbitraryResource {
@PathParam("identifier") String identifier,
@QueryParam("build") Boolean build) {
Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey);
return ArbitraryTransactionUtils.getStatus(service, name, identifier, build);
}
@ -501,6 +506,9 @@ public class ArbitraryResource {
}
for (ArbitraryTransactionData transactionData : transactionDataList) {
if (transactionData.getService() == null) {
continue;
}
ArbitraryResourceInfo arbitraryResourceInfo = new ArbitraryResourceInfo();
arbitraryResourceInfo.name = transactionData.getName();
arbitraryResourceInfo.service = transactionData.getService();
@ -511,10 +519,10 @@ public class ArbitraryResource {
}
if (includeStatus != null && includeStatus) {
resources = this.addStatusToResources(resources);
resources = ArbitraryTransactionUtils.addStatusToResources(resources);
}
if (includeMetadata != null && includeMetadata) {
resources = this.addMetadataToResources(resources);
resources = ArbitraryTransactionUtils.addMetadataToResources(resources);
}
return resources;
@ -544,7 +552,7 @@ public class ArbitraryResource {
Security.checkApiCallAllowed(request);
ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier);
return resource.delete();
return resource.delete(false);
}
@POST
@ -641,6 +649,7 @@ public class ArbitraryResource {
@PathParam("service") Service service,
@PathParam("name") String name,
@QueryParam("filepath") String filepath,
@QueryParam("encoding") String encoding,
@QueryParam("rebuild") boolean rebuild,
@QueryParam("async") boolean async,
@QueryParam("attempts") Integer attempts) {
@ -650,7 +659,7 @@ public class ArbitraryResource {
Security.checkApiCallAllowed(request);
}
return this.download(service, name, null, filepath, rebuild, async, attempts);
return this.download(service, name, null, filepath, encoding, rebuild, async, attempts);
}
@GET
@ -676,16 +685,17 @@ public class ArbitraryResource {
@PathParam("name") String name,
@PathParam("identifier") String identifier,
@QueryParam("filepath") String filepath,
@QueryParam("encoding") String encoding,
@QueryParam("rebuild") boolean rebuild,
@QueryParam("async") boolean async,
@QueryParam("attempts") Integer attempts) {
// Authentication can be bypassed in the settings, for those running public QDN nodes
if (!Settings.getInstance().isQDNAuthBypassEnabled()) {
Security.checkApiCallAllowed(request);
Security.checkApiCallAllowed(request, apiKey);
}
return this.download(service, name, identifier, filepath, rebuild, async, attempts);
return this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts);
}
@ -708,18 +718,15 @@ public class ArbitraryResource {
}
)
@SecurityRequirement(name = "apiKey")
public ArbitraryResourceMetadata getMetadata(@HeaderParam(Security.API_KEY_HEADER) String apiKey,
@PathParam("service") Service service,
@PathParam("name") String name,
@PathParam("identifier") String identifier) {
Security.checkApiCallAllowed(request);
public ArbitraryResourceMetadata getMetadata(@PathParam("service") Service service,
@PathParam("name") String name,
@PathParam("identifier") String identifier) {
ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier);
try {
ArbitraryDataTransactionMetadata transactionMetadata = ArbitraryMetadataManager.getInstance().fetchMetadata(resource, false);
ArbitraryDataTransactionMetadata transactionMetadata = ArbitraryMetadataManager.getInstance().fetchMetadata(resource, true);
if (transactionMetadata != null) {
ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata);
ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata, true);
if (resourceMetadata != null) {
return resourceMetadata;
}
@ -733,7 +740,7 @@ public class ArbitraryResource {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
}
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND);
}
@ -773,6 +780,8 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String path) {
Security.checkApiCallAllowed(request);
@ -781,7 +790,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, null, path, null, null, false,
title, description, tags, category);
fee, null, title, description, tags, category, preview);
}
@POST
@ -818,6 +827,8 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String path) {
Security.checkApiCallAllowed(request);
@ -826,7 +837,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, identifier, path, null, null, false,
title, description, tags, category);
fee, null, title, description, tags, category, preview);
}
@ -864,6 +875,9 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String base64) {
Security.checkApiCallAllowed(request);
@ -872,7 +886,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, null, null, null, base64, false,
title, description, tags, category);
fee, filename, title, description, tags, category, preview);
}
@POST
@ -907,6 +921,9 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String base64) {
Security.checkApiCallAllowed(request);
@ -915,7 +932,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64, false,
title, description, tags, category);
fee, filename, title, description, tags, category, preview);
}
@ -952,6 +969,8 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String base64Zip) {
Security.checkApiCallAllowed(request);
@ -960,7 +979,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, null, null, null, base64Zip, true,
title, description, tags, category);
fee, null, title, description, tags, category, preview);
}
@POST
@ -995,6 +1014,8 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String base64Zip) {
Security.checkApiCallAllowed(request);
@ -1003,7 +1024,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64Zip, true,
title, description, tags, category);
fee, null, title, description, tags, category, preview);
}
@ -1043,6 +1064,9 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String string) {
Security.checkApiCallAllowed(request);
@ -1051,7 +1075,7 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, null, null, string, null, false,
title, description, tags, category);
fee, filename, title, description, tags, category, preview);
}
@POST
@ -1088,6 +1112,9 @@ public class ArbitraryResource {
@QueryParam("description") String description,
@QueryParam("tags") List<String> tags,
@QueryParam("category") Category category,
@QueryParam("filename") String filename,
@QueryParam("fee") Long fee,
@QueryParam("preview") Boolean preview,
String string) {
Security.checkApiCallAllowed(request);
@ -1096,15 +1123,48 @@ public class ArbitraryResource {
}
return this.upload(Service.valueOf(serviceString), name, identifier, null, string, null, false,
title, description, tags, category);
fee, filename, title, description, tags, category, preview);
}
// Shared methods
private String preview(String directoryPath, Service service) {
Security.checkApiCallAllowed(request);
ArbitraryTransactionData.Method method = ArbitraryTransactionData.Method.PUT;
ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.ZIP;
ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath),
null, service, null, method, compression,
null, null, null, null);
try {
arbitraryDataWriter.save();
} catch (IOException | DataException | InterruptedException | MissingDataException e) {
LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
} catch (RuntimeException e) {
LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
}
ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile();
if (arbitraryDataFile != null) {
String digest58 = arbitraryDataFile.digest58();
if (digest58 != null) {
// Pre-authorize resource
ArbitraryDataResource resource = new ArbitraryDataResource(digest58, null, null, null);
ArbitraryDataRenderManager.getInstance().addToAuthorizedResources(resource);
return "/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret());
}
}
return "Unable to generate preview URL";
}
private String upload(Service service, String name, String identifier,
String path, String string, String base64, boolean zipped,
String title, String description, List<String> tags, Category category) {
String path, String string, String base64, boolean zipped, Long fee, String filename,
String title, String description, List<String> tags, Category category,
Boolean preview) {
// Fetch public key from registered name
try (final Repository repository = RepositoryManager.getRepository()) {
NameData nameData = repository.getNameRepository().fromName(name);
@ -1113,7 +1173,11 @@ public class ArbitraryResource {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, error);
}
final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
final Long now = NTP.getTime();
if (now == null) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NO_TIME_SYNC);
}
final Long minLatestBlockTimestamp = now - (60 * 60 * 1000L);
if (!Controller.getInstance().isUpToDate(minLatestBlockTimestamp)) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCKCHAIN_NEEDS_SYNC);
}
@ -1128,7 +1192,12 @@ public class ArbitraryResource {
if (path == null) {
// See if we have a string instead
if (string != null) {
File tempFile = File.createTempFile("qortal-", ".tmp");
if (filename == null) {
// Use current time as filename
filename = String.format("qortal-%d", NTP.getTime());
}
java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-");
File tempFile = Paths.get(tempDirectory.toString(), filename).toFile();
tempFile.deleteOnExit();
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile.toPath().toString()));
writer.write(string);
@ -1138,7 +1207,12 @@ public class ArbitraryResource {
}
// ... or base64 encoded raw data
else if (base64 != null) {
File tempFile = File.createTempFile("qortal-", ".tmp");
if (filename == null) {
// Use current time as filename
filename = String.format("qortal-%d", NTP.getTime());
}
java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-");
File tempFile = Paths.get(tempDirectory.toString(), filename).toFile();
tempFile.deleteOnExit();
Files.write(tempFile.toPath(), Base64.decode(base64));
path = tempFile.toPath().toString();
@ -1161,15 +1235,25 @@ public class ArbitraryResource {
// The actual data will be in a randomly-named subfolder of tempDirectory
// Remove hidden folders, i.e. starting with "_", as some systems can add them, e.g. "__MACOSX"
String[] files = tempDirectory.toFile().list((parent, child) -> !child.startsWith("_"));
if (files.length == 1) { // Single directory or file only
if (files != null && files.length == 1) { // Single directory or file only
path = Paths.get(tempDirectory.toString(), files[0]).toString();
}
}
}
// Finish here if user has requested a preview
if (preview != null && preview == true) {
return this.preview(path, service);
}
// Default to zero fee if not specified
if (fee == null) {
fee = 0L;
}
try {
ArbitraryDataTransactionBuilder transactionBuilder = new ArbitraryDataTransactionBuilder(
repository, publicKey58, Paths.get(path), name, null, service, identifier,
repository, publicKey58, fee, Paths.get(path), name, null, service, identifier,
title, description, tags, category
);
@ -1183,12 +1267,13 @@ public class ArbitraryResource {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage());
}
} catch (DataException | IOException e) {
} catch (Exception e) {
LOGGER.info("Exception when publishing data: ", e);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
}
}
private HttpServletResponse download(Service service, String name, String identifier, String filepath, boolean rebuild, boolean async, Integer maxAttempts) {
private HttpServletResponse download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts) {
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
try {
@ -1231,7 +1316,7 @@ public class ArbitraryResource {
if (filepath == null || filepath.isEmpty()) {
// No file path supplied - so check if this is a single file resource
String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal");
if (files.length == 1) {
if (files != null && files.length == 1) {
// This is a single file resource
filepath = files[0];
}
@ -1241,13 +1326,50 @@ public class ArbitraryResource {
}
}
// TODO: limit file size that can be read into memory
java.nio.file.Path path = Paths.get(outputPath.toString(), filepath);
if (!Files.exists(path)) {
String message = String.format("No file exists at filepath: %s", filepath);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message);
}
byte[] data = Files.readAllBytes(path);
byte[] data;
int fileSize = (int)path.toFile().length();
int length = fileSize;
// Parse "Range" header
Integer rangeStart = null;
Integer rangeEnd = null;
String range = request.getHeader("Range");
if (range != null) {
range = range.replace("bytes=", "");
String[] parts = range.split("-");
rangeStart = (parts != null && parts.length > 0) ? Integer.parseInt(parts[0]) : null;
rangeEnd = (parts != null && parts.length > 1) ? Integer.parseInt(parts[1]) : fileSize;
}
if (rangeStart != null && rangeEnd != null) {
// We have a range, so update the requested length
length = rangeEnd - rangeStart;
}
if (length < fileSize && encoding == null) {
// Partial content requested, and not encoding the data
response.setStatus(206);
response.addHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd-1, fileSize));
data = FilesystemUtils.readFromFile(path.toString(), rangeStart, length);
}
else {
// Full content requested (or encoded data)
response.setStatus(200);
data = Files.readAllBytes(path); // TODO: limit file size that can be read into memory
}
// Encode the data if requested
if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) {
data = Base64.encode(data);
}
response.addHeader("Accept-Ranges", "bytes");
response.setContentType(context.getMimeType(path.toString()));
response.setContentLength(data.length);
response.getOutputStream().write(data);
@ -1259,41 +1381,44 @@ public class ArbitraryResource {
}
}
private FileProperties getFileProperties(Service service, String name, String identifier) {
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier);
try {
arbitraryDataReader.loadSynchronously(false);
java.nio.file.Path outputPath = arbitraryDataReader.getFilePath();
if (outputPath == null) {
// Assume the resource doesn't exist
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, "File not found");
}
private List<ArbitraryResourceInfo> addStatusToResources(List<ArbitraryResourceInfo> resources) {
// Determine and add the status of each resource
List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
for (ArbitraryResourceInfo resourceInfo : resources) {
try {
ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME,
resourceInfo.service, resourceInfo.identifier);
ArbitraryResourceStatus status = resource.getStatus(true);
if (status != null) {
resourceInfo.status = status;
FileProperties fileProperties = new FileProperties();
fileProperties.size = FileUtils.sizeOfDirectory(outputPath.toFile());
String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal");
if (files.length == 1) {
String filename = files[0];
java.nio.file.Path filePath = Paths.get(outputPath.toString(), files[0]);
ContentInfoUtil util = new ContentInfoUtil();
ContentInfo info = util.findMatch(filePath.toFile());
String mimeType;
if (info != null) {
// Attempt to extract MIME type from file contents
mimeType = info.getMimeType();
}
updatedResources.add(resourceInfo);
} catch (Exception e) {
// Catch and log all exceptions, since some systems are experiencing 500 errors when including statuses
LOGGER.info("Caught exception when adding status to resource %s: %s", resourceInfo, e.toString());
else {
// Fall back to using the filename
FileNameMap fileNameMap = URLConnection.getFileNameMap();
mimeType = fileNameMap.getContentTypeFor(filename);
}
fileProperties.filename = filename;
fileProperties.mimeType = mimeType;
}
}
return updatedResources;
}
private List<ArbitraryResourceInfo> addMetadataToResources(List<ArbitraryResourceInfo> resources) {
// Add metadata fields to each resource if they exist
List<ArbitraryResourceInfo> updatedResources = new ArrayList<>();
for (ArbitraryResourceInfo resourceInfo : resources) {
ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME,
resourceInfo.service, resourceInfo.identifier);
ArbitraryDataTransactionMetadata transactionMetadata = resource.getLatestTransactionMetadata();
ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata);
if (resourceMetadata != null) {
resourceInfo.metadata = resourceMetadata;
}
updatedResources.add(resourceInfo);
return fileProperties;
} catch (Exception e) {
LOGGER.debug(String.format("Unable to load %s %s: %s", service, name, e.getMessage()));
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage());
}
return updatedResources;
}
}

52
src/main/java/org/qortal/api/resource/AtResource.java
View File

@ -27,6 +27,7 @@ import org.qortal.api.ApiException;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.data.at.ATData;
import org.qortal.data.at.ATStateData;
import org.qortal.api.model.AtCreationRequest;
import org.qortal.data.transaction.DeployAtTransactionData;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
@ -38,9 +39,14 @@ import org.qortal.transform.TransformationException;
import org.qortal.transform.transaction.DeployAtTransactionTransformer;
import org.qortal.utils.Base58;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Path("/at")
@Tag(name = "Automated Transactions")
public class AtResource {
private static final Logger logger = LoggerFactory.getLogger(AtResource.class);
@Context
HttpServletRequest request;
@ -156,6 +162,52 @@ public class AtResource {
}
}
@POST
@Path("/create")
@Operation(
summary = "Create base58-encoded AT creation bytes from the provided parameters",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(
implementation = AtCreationRequest.class
)
)
),
responses = {
@ApiResponse(
description = "AT creation bytes suitable for use in a DEPLOY_AT transaction",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
public String create(AtCreationRequest atCreationRequest) {
if (atCreationRequest.getCiyamAtVersion() < 2) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "ciyamAtVersion must be at least 2");
}
if (atCreationRequest.getCodeBytes() == null) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Valid codeBytesBase64 must be supplied");
}
if (atCreationRequest.getDataBytes() == null) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Valid dataBytesBase64 must be supplied");
}
byte[] creationBytes = MachineState.toCreationBytes(
atCreationRequest.getCiyamAtVersion(),
atCreationRequest.getCodeBytes(),
atCreationRequest.getDataBytes(),
atCreationRequest.getNumCallStackPages(),
atCreationRequest.getNumUserStackPages(),
atCreationRequest.getMinActivationAmount()
);
return Base58.encode(creationBytes);
}
@POST
@Operation(
summary = "Build raw, unsigned, DEPLOY_AT transaction",

54
src/main/java/org/qortal/api/resource/BlocksResource.java
View File

@ -48,6 +48,7 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.transform.TransformationException;
import org.qortal.transform.block.BlockTransformer;
import org.qortal.utils.Base58;
import org.qortal.utils.Triple;
@Path("/blocks")
@Tag(name = "Blocks")
@ -165,10 +166,13 @@ public class BlocksResource {
}
// Not found, so try the block archive
byte[] bytes = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, false, repository);
if (bytes != null) {
if (version != 1) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Archived blocks require version 1");
Triple<byte[], Integer, Integer> serializedBlock = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, false, repository);
if (serializedBlock != null) {
byte[] bytes = serializedBlock.getA();
Integer serializationVersion = serializedBlock.getB();
if (version != serializationVersion) {
// TODO: we could quite easily reserialize the block with the requested version
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Block is not stored using requested serialization version.");
}
return Base58.encode(bytes);
}
@ -218,14 +222,25 @@ public class BlocksResource {
}
try (final Repository repository = RepositoryManager.getRepository()) {
// Check if the block exists in either the database or archive
if (repository.getBlockRepository().getHeightFromSignature(signature) == 0 &&
repository.getBlockArchiveRepository().getHeightFromSignature(signature) == 0) {
// Not found in either the database or archive
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
}
// Check if the block exists in either the database or archive
int height = repository.getBlockRepository().getHeightFromSignature(signature);
if (height == 0) {
height = repository.getBlockArchiveRepository().getHeightFromSignature(signature);
if (height == 0) {
// Not found in either the database or archive
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
}
}
return repository.getBlockRepository().getTransactionsFromSignature(signature, limit, offset, reverse);
List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, height, height);
// Expand signatures to transactions
List<TransactionData> transactions = new ArrayList<>(signatures.size());
for (byte[] s : signatures) {
transactions.add(repository.getTransactionRepository().fromSignature(s));
}
return transactions;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
@ -634,13 +649,16 @@ public class BlocksResource {
@ApiErrors({
ApiError.REPOSITORY_ISSUE
})
public List<BlockData> getBlockRange(@PathParam("height") int height, @Parameter(
ref = "count"
) @QueryParam("count") int count) {
public List<BlockData> getBlockRange(@PathParam("height") int height,
@Parameter(ref = "count") @QueryParam("count") int count,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse,
@QueryParam("includeOnlineSignatures") Boolean includeOnlineSignatures) {
try (final Repository repository = RepositoryManager.getRepository()) {
List<BlockData> blocks = new ArrayList<>();
boolean shouldReverse = (reverse != null && reverse == true);
for (/* count already set */; count > 0; --count, ++height) {
int i = 0;
while (i < count) {
BlockData blockData = repository.getBlockRepository().fromHeight(height);
if (blockData == null) {
// Not found - try the archive
@ -650,8 +668,14 @@ public class BlocksResource {
break;
}
}
if (includeOnlineSignatures == null || includeOnlineSignatures == false) {
blockData.setOnlineAccountsSignatures(null);
}
blocks.add(blockData);
height = shouldReverse ? height - 1 : height + 1;
i++;
}
return blocks;

91
src/main/java/org/qortal/api/resource/ChatResource.java
View File

@ -40,6 +40,8 @@ import org.qortal.utils.Base58;
import com.google.common.primitives.Bytes;
import static org.qortal.data.chat.ChatMessage.Encoding;
@Path("/chat")
@Tag(name = "Chat")
public class ChatResource {
@ -70,6 +72,10 @@ public class ChatResource {
@QueryParam("txGroupId") Integer txGroupId,
@QueryParam("involving") List<String> involvingAddresses,
@QueryParam("reference") String reference,
@QueryParam("chatreference") String chatReference,
@QueryParam("haschatreference") Boolean hasChatReference,
@QueryParam("sender") String sender,
@QueryParam("encoding") Encoding encoding,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
@ -92,19 +98,96 @@ public class ChatResource {
if (reference != null)
referenceBytes = Base58.decode(reference);
byte[] chatReferenceBytes = null;
if (chatReference != null)
chatReferenceBytes = Base58.decode(chatReference);
try (final Repository repository = RepositoryManager.getRepository()) {
return repository.getChatRepository().getMessagesMatchingCriteria(
before,
after,
txGroupId,
referenceBytes,
chatReferenceBytes,
hasChatReference,
involvingAddresses,
sender,
encoding,
limit, offset, reverse);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/messages/count")
@Operation(
summary = "Count chat messages",
description = "Returns count of CHAT messages that match criteria. Must provide EITHER 'txGroupId' OR two 'involving' addresses.",
responses = {
@ApiResponse(
description = "count of messages",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "integer"
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public int countChatMessages(@QueryParam("before") Long before, @QueryParam("after") Long after,
@QueryParam("txGroupId") Integer txGroupId,
@QueryParam("involving") List<String> involvingAddresses,
@QueryParam("reference") String reference,
@QueryParam("chatreference") String chatReference,
@QueryParam("haschatreference") Boolean hasChatReference,
@QueryParam("sender") String sender,
@QueryParam("encoding") Encoding encoding,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) {
// Check args meet expectations
if ((txGroupId == null && involvingAddresses.size() != 2)
|| (txGroupId != null && !involvingAddresses.isEmpty()))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
// Check any provided addresses are valid
if (involvingAddresses.stream().anyMatch(address -> !Crypto.isValidAddress(address)))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
if (before != null && before < 1500000000000L)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
if (after != null && after < 1500000000000L)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
byte[] referenceBytes = null;
if (reference != null)
referenceBytes = Base58.decode(reference);
byte[] chatReferenceBytes = null;
if (chatReference != null)
chatReferenceBytes = Base58.decode(chatReference);
try (final Repository repository = RepositoryManager.getRepository()) {
return repository.getChatRepository().getMessagesMatchingCriteria(
before,
after,
txGroupId,
referenceBytes,
chatReferenceBytes,
hasChatReference,
involvingAddresses,
sender,
encoding,
limit, offset, reverse).size();
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/message/{signature}")
@Operation(
@ -121,7 +204,7 @@ public class ChatResource {
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public ChatMessage getMessageBySignature(@PathParam("signature") String signature58) {
public ChatMessage getMessageBySignature(@PathParam("signature") String signature58, @QueryParam("encoding") Encoding encoding) {
byte[] signature = Base58.decode(signature58);
try (final Repository repository = RepositoryManager.getRepository()) {
@ -131,7 +214,7 @@ public class ChatResource {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Message not found");
}
return repository.getChatRepository().toChatMessage(chatTransactionData);
return repository.getChatRepository().toChatMessage(chatTransactionData, encoding);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
@ -154,12 +237,12 @@ public class ChatResource {
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE})
public ActiveChats getActiveChats(@PathParam("address") String address) {
public ActiveChats getActiveChats(@PathParam("address") String address, @QueryParam("encoding") Encoding encoding) {
if (address == null || !Crypto.isValidAddress(address))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS);
try (final Repository repository = RepositoryManager.getRepository()) {
return repository.getChatRepository().getActiveChats(address);
return repository.getChatRepository().getActiveChats(address, encoding);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}

73
src/main/java/org/qortal/api/resource/CrossChainBitcoinResource.java
View File

@ -14,6 +14,7 @@ import java.util.List;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -35,6 +36,37 @@ public class CrossChainBitcoinResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current Bitcoin block height",
description = "Returns the height of the most recent block in the Bitcoin chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getBitcoinHeight() {
Bitcoin bitcoin = Bitcoin.getInstance();
try {
Integer height = bitcoin.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(
@ -68,7 +100,7 @@ public class CrossChainBitcoinResource {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
Long balance = bitcoin.getWalletBalanceFromTransactions(key58);
Long balance = bitcoin.getWalletBalance(key58);
if (balance == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
@ -118,6 +150,45 @@ public class CrossChainBitcoinResource {
}
}
@POST
@Path("/unusedaddress")
@Operation(
summary = "Returns first unused address for hierarchical, deterministic BIP32 wallet",
description = "Supply BIP32 'm' private/public key in base58, starting with 'xprv'/'xpub' for mainnet, 'tprv'/'tpub' for testnet",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
description = "BIP32 'm' private/public key in base58",
example = "tpubD6NzVbkrYhZ4XTPc4btCZ6SMgn8CxmWkj6VBVZ1tfcJfMq4UwAjZbG8U74gGSypL9XBYk2R2BLbDBe8pcEyBKM1edsGQEPKXNbEskZozeZc"
)
)
),
responses = {
@ApiResponse(
content = @Content(array = @ArraySchema( schema = @Schema( implementation = SimpleTransaction.class ) ) )
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
@SecurityRequirement(name = "apiKey")
public String getUnusedBitcoinReceiveAddress(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String key58) {
Security.checkApiCallAllowed(request);
Bitcoin bitcoin = Bitcoin.getInstance();
if (!bitcoin.isValidDeterministicKey(key58))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
return bitcoin.getUnusedReceiveAddress(key58);
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/send")
@Operation(

73
src/main/java/org/qortal/api/resource/CrossChainDigibyteResource.java
View File

@ -14,6 +14,7 @@ import java.util.List;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -35,6 +36,37 @@ public class CrossChainDigibyteResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current Digibyte block height",
description = "Returns the height of the most recent block in the Digibyte chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getDigibyteHeight() {
Digibyte digibyte = Digibyte.getInstance();
try {
Integer height = digibyte.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(
@ -68,7 +100,7 @@ public class CrossChainDigibyteResource {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
Long balance = digibyte.getWalletBalanceFromTransactions(key58);
Long balance = digibyte.getWalletBalance(key58);
if (balance == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
@ -118,6 +150,45 @@ public class CrossChainDigibyteResource {
}
}
@POST
@Path("/unusedaddress")
@Operation(
summary = "Returns first unused address for hierarchical, deterministic BIP32 wallet",
description = "Supply BIP32 'm' private/public key in base58, starting with 'xprv'/'xpub' for mainnet, 'tprv'/'tpub' for testnet",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
description = "BIP32 'm' private/public key in base58",
example = "tpubD6NzVbkrYhZ4XTPc4btCZ6SMgn8CxmWkj6VBVZ1tfcJfMq4UwAjZbG8U74gGSypL9XBYk2R2BLbDBe8pcEyBKM1edsGQEPKXNbEskZozeZc"
)
)
),
responses = {
@ApiResponse(
content = @Content(array = @ArraySchema( schema = @Schema( implementation = SimpleTransaction.class ) ) )
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
@SecurityRequirement(name = "apiKey")
public String getUnusedDigibyteReceiveAddress(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String key58) {
Security.checkApiCallAllowed(request);
Digibyte digibyte = Digibyte.getInstance();
if (!digibyte.isValidDeterministicKey(key58))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
return digibyte.getUnusedReceiveAddress(key58);
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/send")
@Operation(

73
src/main/java/org/qortal/api/resource/CrossChainDogecoinResource.java
View File

@ -21,6 +21,7 @@ import org.qortal.crosschain.SimpleTransaction;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -33,6 +34,37 @@ public class CrossChainDogecoinResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current Dogecoin block height",
description = "Returns the height of the most recent block in the Dogecoin chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getDogecoinHeight() {
Dogecoin dogecoin = Dogecoin.getInstance();
try {
Integer height = dogecoin.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(
@ -66,7 +98,7 @@ public class CrossChainDogecoinResource {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
Long balance = dogecoin.getWalletBalanceFromTransactions(key58);
Long balance = dogecoin.getWalletBalance(key58);
if (balance == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
@ -116,6 +148,45 @@ public class CrossChainDogecoinResource {
}
}
@POST
@Path("/unusedaddress")
@Operation(
summary = "Returns first unused address for hierarchical, deterministic BIP32 wallet",
description = "Supply BIP32 'm' private/public key in base58, starting with 'xprv'/'xpub' for mainnet, 'tprv'/'tpub' for testnet",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
description = "BIP32 'm' private/public key in base58",
example = "tpubD6NzVbkrYhZ4XTPc4btCZ6SMgn8CxmWkj6VBVZ1tfcJfMq4UwAjZbG8U74gGSypL9XBYk2R2BLbDBe8pcEyBKM1edsGQEPKXNbEskZozeZc"
)
)
),
responses = {
@ApiResponse(
content = @Content(array = @ArraySchema( schema = @Schema( implementation = SimpleTransaction.class ) ) )
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
@SecurityRequirement(name = "apiKey")
public String getUnusedDogecoinReceiveAddress(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String key58) {
Security.checkApiCallAllowed(request);
Dogecoin dogecoin = Dogecoin.getInstance();
if (!dogecoin.isValidDeterministicKey(key58))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
return dogecoin.getUnusedReceiveAddress(key58);
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/send")
@Operation(

163
src/main/java/org/qortal/api/resource/CrossChainHtlcResource.java
View File

@ -8,11 +8,10 @@ import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.io.IOException;
import java.math.BigDecimal;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.*;
@ -25,7 +24,6 @@ import org.bitcoinj.core.*;
import org.bitcoinj.script.Script;
import org.qortal.api.*;
import org.qortal.api.model.CrossChainBitcoinyHTLCStatus;
import org.qortal.controller.Controller;
import org.qortal.crosschain.*;
import org.qortal.crypto.Crypto;
import org.qortal.data.at.ATData;
@ -586,98 +584,103 @@ public class CrossChainHtlcResource {
}
List<TradeBotData> allTradeBotData = repository.getCrossChainRepository().getAllTradeBotData();
TradeBotData tradeBotData = allTradeBotData.stream().filter(tradeBotDataItem -> tradeBotDataItem.getAtAddress().equals(atAddress)).findFirst().orElse(null);
if (tradeBotData == null)
List<TradeBotData> tradeBotDataList = allTradeBotData.stream().filter(tradeBotDataItem -> tradeBotDataItem.getAtAddress().equals(atAddress)).collect(Collectors.toList());
if (tradeBotDataList == null || tradeBotDataList.isEmpty())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
Bitcoiny bitcoiny = (Bitcoiny) acct.getBlockchain();
int lockTime = tradeBotData.getLockTimeA();
// Loop through all matching entries for this AT address, as there might be more than one
for (TradeBotData tradeBotData : tradeBotDataList) {
// We can't refund P2SH-A until lockTime-A has passed
if (NTP.getTime() <= lockTime * 1000L)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_TOO_SOON);
if (tradeBotData == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
// We can't refund P2SH-A until median block time has passed lockTime-A (see BIP113)
int medianBlockTime = bitcoiny.getMedianBlockTime();
if (medianBlockTime <= lockTime)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_TOO_SOON);
Bitcoiny bitcoiny = (Bitcoiny) acct.getBlockchain();
int lockTime = tradeBotData.getLockTimeA();
// Fee for redeem/refund is subtracted from P2SH-A balance.
long feeTimestamp = calcFeeTimestamp(lockTime, crossChainTradeData.tradeTimeout);
long p2shFee = bitcoiny.getP2shFee(feeTimestamp);
long minimumAmountA = crossChainTradeData.expectedForeignAmount + p2shFee;
// We can't refund P2SH-A until lockTime-A has passed
if (NTP.getTime() <= lockTime * 1000L)
continue;
// Create redeem script based on destination chain
byte[] redeemScriptA;
String p2shAddressA;
BitcoinyHTLC.Status htlcStatusA;
if (Objects.equals(bitcoiny.getCurrencyCode(), "ARRR")) {
redeemScriptA = PirateChainHTLC.buildScript(tradeBotData.getTradeForeignPublicKey(), lockTime, crossChainTradeData.creatorForeignPKH, tradeBotData.getHashOfSecret());
p2shAddressA = PirateChain.getInstance().deriveP2shAddressBPrefix(redeemScriptA);
htlcStatusA = PirateChainHTLC.determineHtlcStatus(bitcoiny.getBlockchainProvider(), p2shAddressA, minimumAmountA);
}
else {
redeemScriptA = BitcoinyHTLC.buildScript(tradeBotData.getTradeForeignPublicKeyHash(), lockTime, crossChainTradeData.creatorForeignPKH, tradeBotData.getHashOfSecret());
p2shAddressA = bitcoiny.deriveP2shAddress(redeemScriptA);
htlcStatusA = BitcoinyHTLC.determineHtlcStatus(bitcoiny.getBlockchainProvider(), p2shAddressA, minimumAmountA);
}
LOGGER.info(String.format("Refunding P2SH address: %s", p2shAddressA));
// We can't refund P2SH-A until median block time has passed lockTime-A (see BIP113)
int medianBlockTime = bitcoiny.getMedianBlockTime();
if (medianBlockTime <= lockTime)
continue;
switch (htlcStatusA) {
case UNFUNDED:
case FUNDING_IN_PROGRESS:
// Still waiting for P2SH-A to be funded...
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_TOO_SOON);
// Fee for redeem/refund is subtracted from P2SH-A balance.
long feeTimestamp = calcFeeTimestamp(lockTime, crossChainTradeData.tradeTimeout);
long p2shFee = bitcoiny.getP2shFee(feeTimestamp);
long minimumAmountA = crossChainTradeData.expectedForeignAmount + p2shFee;
case REDEEM_IN_PROGRESS:
case REDEEMED:
case REFUND_IN_PROGRESS:
case REFUNDED:
// Too late!
return false;
// Create redeem script based on destination chain
byte[] redeemScriptA;
String p2shAddressA;
BitcoinyHTLC.Status htlcStatusA;
if (Objects.equals(bitcoiny.getCurrencyCode(), "ARRR")) {
redeemScriptA = PirateChainHTLC.buildScript(tradeBotData.getTradeForeignPublicKey(), lockTime, crossChainTradeData.creatorForeignPKH, tradeBotData.getHashOfSecret());
p2shAddressA = PirateChain.getInstance().deriveP2shAddressBPrefix(redeemScriptA);
htlcStatusA = PirateChainHTLC.determineHtlcStatus(bitcoiny.getBlockchainProvider(), p2shAddressA, minimumAmountA);
} else {
redeemScriptA = BitcoinyHTLC.buildScript(tradeBotData.getTradeForeignPublicKeyHash(), lockTime, crossChainTradeData.creatorForeignPKH, tradeBotData.getHashOfSecret());
p2shAddressA = bitcoiny.deriveP2shAddress(redeemScriptA);
htlcStatusA = BitcoinyHTLC.determineHtlcStatus(bitcoiny.getBlockchainProvider(), p2shAddressA, minimumAmountA);
}
LOGGER.info(String.format("Refunding P2SH address: %s", p2shAddressA));
case FUNDED:{
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
switch (htlcStatusA) {
case UNFUNDED:
case FUNDING_IN_PROGRESS:
// Still waiting for P2SH-A to be funded...
continue;
if (Objects.equals(bitcoiny.getCurrencyCode(), "ARRR")) {
// Pirate Chain custom integration
case REDEEM_IN_PROGRESS:
case REDEEMED:
case REFUND_IN_PROGRESS:
case REFUNDED:
// Too late!
continue;
PirateChain pirateChain = PirateChain.getInstance();
String p2shAddressT3 = pirateChain.deriveP2shAddress(redeemScriptA);
case FUNDED: {
Coin refundAmount = Coin.valueOf(crossChainTradeData.expectedForeignAmount);
// Get funding txid
String fundingTxidHex = PirateChainHTLC.getUnspentFundingTxid(pirateChain.getBlockchainProvider(), p2shAddressA, minimumAmountA);
if (fundingTxidHex == null) {
throw new ForeignBlockchainException("Missing funding txid when refunding P2SH");
if (Objects.equals(bitcoiny.getCurrencyCode(), "ARRR")) {
// Pirate Chain custom integration
PirateChain pirateChain = PirateChain.getInstance();
String p2shAddressT3 = pirateChain.deriveP2shAddress(redeemScriptA);
// Get funding txid
String fundingTxidHex = PirateChainHTLC.getUnspentFundingTxid(pirateChain.getBlockchainProvider(), p2shAddressA, minimumAmountA);
if (fundingTxidHex == null) {
throw new ForeignBlockchainException("Missing funding txid when refunding P2SH");
}
String fundingTxid58 = Base58.encode(HashCode.fromString(fundingTxidHex).asBytes());
byte[] privateKey = tradeBotData.getTradePrivateKey();
String privateKey58 = Base58.encode(privateKey);
String redeemScript58 = Base58.encode(redeemScriptA);
String txid = PirateChain.getInstance().refundP2sh(p2shAddressT3,
receiveAddress, refundAmount.value, redeemScript58, fundingTxid58, lockTime, privateKey58);
LOGGER.info("Refund txid: {}", txid);
} else {
// ElectrumX coins
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA);
// Validate the destination foreign blockchain address
Address receiving = Address.fromString(bitcoiny.getNetworkParameters(), receiveAddress);
if (receiving.getOutputScriptType() != Script.ScriptType.P2PKH)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
Transaction p2shRefundTransaction = BitcoinyHTLC.buildRefundTransaction(bitcoiny.getNetworkParameters(), refundAmount, refundKey,
fundingOutputs, redeemScriptA, lockTime, receiving.getHash());
bitcoiny.broadcastTransaction(p2shRefundTransaction);
}
String fundingTxid58 = Base58.encode(HashCode.fromString(fundingTxidHex).asBytes());
byte[] privateKey = tradeBotData.getTradePrivateKey();
String privateKey58 = Base58.encode(privateKey);
String redeemScript58 = Base58.encode(redeemScriptA);
String txid = PirateChain.getInstance().refundP2sh(p2shAddressT3,
receiveAddress, refundAmount.value, redeemScript58, fundingTxid58, lockTime, privateKey58);
LOGGER.info("Refund txid: {}", txid);
return true;
}
else {
// ElectrumX coins
ECKey refundKey = ECKey.fromPrivate(tradeBotData.getTradePrivateKey());
List<TransactionOutput> fundingOutputs = bitcoiny.getUnspentOutputs(p2shAddressA);
// Validate the destination foreign blockchain address
Address receiving = Address.fromString(bitcoiny.getNetworkParameters(), receiveAddress);
if (receiving.getOutputScriptType() != Script.ScriptType.P2PKH)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
Transaction p2shRefundTransaction = BitcoinyHTLC.buildRefundTransaction(bitcoiny.getNetworkParameters(), refundAmount, refundKey,
fundingOutputs, redeemScriptA, lockTime, receiving.getHash());
bitcoiny.broadcastTransaction(p2shRefundTransaction);
}
return true;
}
}

73
src/main/java/org/qortal/api/resource/CrossChainLitecoinResource.java
View File

@ -14,6 +14,7 @@ import java.util.List;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -35,6 +36,37 @@ public class CrossChainLitecoinResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current Litecoin block height",
description = "Returns the height of the most recent block in the Litecoin chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getLitecoinHeight() {
Litecoin litecoin = Litecoin.getInstance();
try {
Integer height = litecoin.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(
@ -68,7 +100,7 @@ public class CrossChainLitecoinResource {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
Long balance = litecoin.getWalletBalanceFromTransactions(key58);
Long balance = litecoin.getWalletBalance(key58);
if (balance == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
@ -118,6 +150,45 @@ public class CrossChainLitecoinResource {
}
}
@POST
@Path("/unusedaddress")
@Operation(
summary = "Returns first unused address for hierarchical, deterministic BIP32 wallet",
description = "Supply BIP32 'm' private/public key in base58, starting with 'xprv'/'xpub' for mainnet, 'tprv'/'tpub' for testnet",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
description = "BIP32 'm' private/public key in base58",
example = "tpubD6NzVbkrYhZ4XTPc4btCZ6SMgn8CxmWkj6VBVZ1tfcJfMq4UwAjZbG8U74gGSypL9XBYk2R2BLbDBe8pcEyBKM1edsGQEPKXNbEskZozeZc"
)
)
),
responses = {
@ApiResponse(
content = @Content(array = @ArraySchema( schema = @Schema( implementation = SimpleTransaction.class ) ) )
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
@SecurityRequirement(name = "apiKey")
public String getUnusedLitecoinReceiveAddress(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String key58) {
Security.checkApiCallAllowed(request);
Litecoin litecoin = Litecoin.getInstance();
if (!litecoin.isValidDeterministicKey(key58))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
return litecoin.getUnusedReceiveAddress(key58);
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/send")
@Operation(

32
src/main/java/org/qortal/api/resource/CrossChainPirateChainResource.java
View File

@ -20,6 +20,7 @@ import org.qortal.crosschain.SimpleTransaction;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -32,6 +33,37 @@ public class CrossChainPirateChainResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current PirateChain block height",
description = "Returns the height of the most recent block in the PirateChain chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getPirateChainHeight() {
PirateChain pirateChain = PirateChain.getInstance();
try {
Integer height = pirateChain.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(

73
src/main/java/org/qortal/api/resource/CrossChainRavencoinResource.java
View File

@ -14,6 +14,7 @@ import java.util.List;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.HeaderParam;
import javax.ws.rs.POST;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@ -35,6 +36,37 @@ public class CrossChainRavencoinResource {
@Context
HttpServletRequest request;
@GET
@Path("/height")
@Operation(
summary = "Returns current Ravencoin block height",
description = "Returns the height of the most recent block in the Ravencoin chain.",
responses = {
@ApiResponse(
content = @Content(
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
public String getRavencoinHeight() {
Ravencoin ravencoin = Ravencoin.getInstance();
try {
Integer height = ravencoin.getBlockchainHeight();
if (height == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
return height.toString();
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/walletbalance")
@Operation(
@ -68,7 +100,7 @@ public class CrossChainRavencoinResource {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
Long balance = ravencoin.getWalletBalanceFromTransactions(key58);
Long balance = ravencoin.getWalletBalance(key58);
if (balance == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
@ -118,6 +150,45 @@ public class CrossChainRavencoinResource {
}
}
@POST
@Path("/unusedaddress")
@Operation(
summary = "Returns first unused address for hierarchical, deterministic BIP32 wallet",
description = "Supply BIP32 'm' private/public key in base58, starting with 'xprv'/'xpub' for mainnet, 'tprv'/'tpub' for testnet",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
description = "BIP32 'm' private/public key in base58",
example = "tpubD6NzVbkrYhZ4XTPc4btCZ6SMgn8CxmWkj6VBVZ1tfcJfMq4UwAjZbG8U74gGSypL9XBYk2R2BLbDBe8pcEyBKM1edsGQEPKXNbEskZozeZc"
)
)
),
responses = {
@ApiResponse(
content = @Content(array = @ArraySchema( schema = @Schema( implementation = SimpleTransaction.class ) ) )
)
}
)
@ApiErrors({ApiError.INVALID_PRIVATE_KEY, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE})
@SecurityRequirement(name = "apiKey")
public String getUnusedRavencoinReceiveAddress(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String key58) {
Security.checkApiCallAllowed(request);
Ravencoin ravencoin = Ravencoin.getInstance();
if (!ravencoin.isValidDeterministicKey(key58))
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_PRIVATE_KEY);
try {
return ravencoin.getUnusedReceiveAddress(key58);
} catch (ForeignBlockchainException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FOREIGN_BLOCKCHAIN_NETWORK_ISSUE);
}
}
@POST
@Path("/send")
@Operation(

61
src/main/java/org/qortal/api/resource/CrossChainResource.java
View File

@ -115,6 +115,9 @@ public class CrossChainResource {
crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp));
}
// Remove any trades that have had too many failures
crossChainTrades = TradeBot.getInstance().removeFailedTrades(repository, crossChainTrades);
if (limit != null && limit > 0) {
// Make sure to not return more than the limit
int upperLimit = Math.min(limit, crossChainTrades.size());
@ -129,6 +132,64 @@ public class CrossChainResource {
}
}
@GET
@Path("/tradeoffers/hidden")
@Operation(
summary = "Find cross-chain trade offers that have been hidden due to too many failures",
responses = {
@ApiResponse(
content = @Content(
array = @ArraySchema(
schema = @Schema(
implementation = CrossChainTradeData.class
)
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE})
public List<CrossChainTradeData> getHiddenTradeOffers(
@Parameter(
description = "Limit to specific blockchain",
example = "LITECOIN",
schema = @Schema(implementation = SupportedBlockchain.class)
) @QueryParam("foreignBlockchain") SupportedBlockchain foreignBlockchain) {
final boolean isExecutable = true;
List<CrossChainTradeData> crossChainTrades = new ArrayList<>();
try (final Repository repository = RepositoryManager.getRepository()) {
Map<ByteArray, Supplier<ACCT>> acctsByCodeHash = SupportedBlockchain.getFilteredAcctMap(foreignBlockchain);
for (Map.Entry<ByteArray, Supplier<ACCT>> acctInfo : acctsByCodeHash.entrySet()) {
byte[] codeHash = acctInfo.getKey().value;
ACCT acct = acctInfo.getValue().get();
List<ATData> atsData = repository.getATRepository().getATsByFunctionality(codeHash, isExecutable, null, null, null);
for (ATData atData : atsData) {
CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atData);
if (crossChainTradeData.mode == AcctMode.OFFERING) {
crossChainTrades.add(crossChainTradeData);
}
}
}
// Sort the trades by timestamp
crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp));
// Remove trades that haven't failed
crossChainTrades.removeIf(t -> !TradeBot.getInstance().isFailedTrade(repository, t));
crossChainTrades.stream().forEach(CrossChainResource::decorateTradeDataWithPresence);
return crossChainTrades;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/trade/{ataddress}")
@Operation(

15
src/main/java/org/qortal/api/resource/CrossChainTradeBotResource.java
View File

@ -11,6 +11,7 @@ import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import javax.servlet.http.HttpServletRequest;
@ -38,9 +39,12 @@ import org.qortal.crypto.Crypto;
import org.qortal.data.at.ATData;
import org.qortal.data.crosschain.CrossChainTradeData;
import org.qortal.data.crosschain.TradeBotData;
import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.Transaction;
import org.qortal.utils.Base58;
import org.qortal.utils.NTP;
@ -223,6 +227,17 @@ public class CrossChainTradeBotResource {
if (crossChainTradeData.mode != AcctMode.OFFERING)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA);
// Check if there is a buy or a cancel request in progress for this trade
List<Transaction.TransactionType> txTypes = List.of(Transaction.TransactionType.MESSAGE);
List<TransactionData> unconfirmed = repository.getTransactionRepository().getUnconfirmedTransactions(txTypes, null, 0, 0, false);
for (TransactionData transactionData : unconfirmed) {
MessageTransactionData messageTransactionData = (MessageTransactionData) transactionData;
if (Objects.equals(messageTransactionData.getRecipient(), atAddress)) {
// There is a pending request for this trade, so block this buy attempt to reduce the risk of refunds
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Trade has an existing buy request or is pending cancellation.");
}
}
AcctTradeBot.ResponseResult result = TradeBot.getInstance().startResponse(repository, atData, acct, crossChainTradeData,
tradeBotRespondRequest.foreignKey, tradeBotRespondRequest.receivingAddress);

96
src/main/java/org/qortal/api/resource/DeveloperResource.java Normal file
View File

@ -0,0 +1,96 @@
package org.qortal.api.resource;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.qortal.api.ApiError;
import org.qortal.api.ApiErrors;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.controller.DevProxyManager;
import org.qortal.repository.DataException;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
@Path("/developer")
@Tag(name = "Developer Tools")
public class DeveloperResource {
@Context HttpServletRequest request;
@Context HttpServletResponse response;
@Context ServletContext context;
@POST
@Path("/proxy/start")
@Operation(
summary = "Start proxy server, for real time QDN app/website development",
requestBody = @RequestBody(
description = "Host and port of source webserver to be proxied",
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string",
example = "127.0.0.1:5173"
)
)
),
responses = {
@ApiResponse(
description = "Port number of running server",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "number"
)
)
)
}
)
@ApiErrors({ApiError.INVALID_CRITERIA})
public Integer startProxy(String sourceHostAndPort) {
// TODO: API key
DevProxyManager devProxyManager = DevProxyManager.getInstance();
try {
devProxyManager.setSourceHostAndPort(sourceHostAndPort);
devProxyManager.start();
return devProxyManager.getPort();
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, e.getMessage());
}
}
@POST
@Path("/proxy/stop")
@Operation(
summary = "Stop proxy server",
responses = {
@ApiResponse(
description = "true if stopped",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "boolean"
)
)
)
}
)
public boolean stopProxy() {
DevProxyManager devProxyManager = DevProxyManager.getInstance();
devProxyManager.stop();
return !devProxyManager.isRunning();
}
}

53
src/main/java/org/qortal/api/resource/NamesResource.java
View File

@ -47,6 +47,7 @@ import org.qortal.transform.transaction.RegisterNameTransactionTransformer;
import org.qortal.transform.transaction.SellNameTransactionTransformer;
import org.qortal.transform.transaction.UpdateNameTransactionTransformer;
import org.qortal.utils.Base58;
import org.qortal.utils.Unicode;
@Path("/names")
@Tag(name = "Names")
@ -63,19 +64,19 @@ public class NamesResource {
description = "registered name info",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
array = @ArraySchema(schema = @Schema(implementation = NameSummary.class))
array = @ArraySchema(schema = @Schema(implementation = NameData.class))
)
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<NameSummary> getAllNames(@Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
public List<NameData> getAllNames(@Parameter(description = "Return only names registered or updated after timestamp") @QueryParam("after") Long after,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
try (final Repository repository = RepositoryManager.getRepository()) {
List<NameData> names = repository.getNameRepository().getAllNames(limit, offset, reverse);
// Convert to summary
return names.stream().map(NameSummary::new).collect(Collectors.toList());
return repository.getNameRepository().getAllNames(after, limit, offset, reverse);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
@ -135,12 +136,13 @@ public class NamesResource {
public NameData getName(@PathParam("name") String name) {
try (final Repository repository = RepositoryManager.getRepository()) {
NameData nameData;
String reducedName = Unicode.sanitize(name);
if (Settings.getInstance().isLite()) {
nameData = LiteNode.getInstance().fetchNameData(name);
}
else {
nameData = repository.getNameRepository().fromName(name);
nameData = repository.getNameRepository().fromReducedName(reducedName);
}
if (nameData == null) {
@ -155,6 +157,41 @@ public class NamesResource {
}
}
@GET
@Path("/search")
@Operation(
summary = "Search registered names",
responses = {
@ApiResponse(
description = "registered name info",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
array = @ArraySchema(schema = @Schema(implementation = NameData.class))
)
)
}
)
@ApiErrors({ApiError.NAME_UNKNOWN, ApiError.REPOSITORY_ISSUE})
public List<NameData> searchNames(@QueryParam("query") String query,
@Parameter(description = "Prefix only (if true, only the beginning of the name is matched)") @QueryParam("prefix") Boolean prefixOnly,
@Parameter(ref = "limit") @QueryParam("limit") Integer limit,
@Parameter(ref = "offset") @QueryParam("offset") Integer offset,
@Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) {
try (final Repository repository = RepositoryManager.getRepository()) {
if (query == null) {
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Missing query");
}
boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly);
return repository.getNameRepository().searchNames(query, usePrefixOnly, limit, offset, reverse);
} catch (ApiException e) {
throw e;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@POST
@Path("/register")
@ -410,4 +447,4 @@ public class NamesResource {
}
}
}
}

258
src/main/java/org/qortal/api/resource/PollsResource.java Normal file
View File

@ -0,0 +1,258 @@
package org.qortal.api.resource;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.qortal.api.ApiError;
import org.qortal.api.ApiErrors;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.data.transaction.CreatePollTransactionData;
import org.qortal.data.transaction.PaymentTransactionData;
import org.qortal.data.transaction.VoteOnPollTransactionData;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.transaction.Transaction;
import org.qortal.transform.TransformationException;
import org.qortal.transform.transaction.CreatePollTransactionTransformer;
import org.qortal.transform.transaction.PaymentTransactionTransformer;
import org.qortal.transform.transaction.VoteOnPollTransactionTransformer;
import org.qortal.utils.Base58;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.media.ArraySchema;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import javax.ws.rs.GET;
import javax.ws.rs.PathParam;
import javax.ws.rs.QueryParam;
import org.qortal.api.ApiException;
import org.qortal.api.model.PollVotes;
import org.qortal.data.voting.PollData;
import org.qortal.data.voting.PollOptionData;
import org.qortal.data.voting.VoteOnPollData;
@Path("/polls")
@Tag(name = "Polls")
public class PollsResource {
@Context
HttpServletRequest request;
@GET
@Operation(
summary = "List all polls",
responses = {
@ApiResponse(
description = "poll info",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
array = @ArraySchema(schema = @Schema(implementation = PollData.class))
)
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public List<PollData> getAllPolls(@Parameter(
ref = "limit"
) @QueryParam("limit") Integer limit, @Parameter(
ref = "offset"
) @QueryParam("offset") Integer offset, @Parameter(
ref = "reverse"
) @QueryParam("reverse") Boolean reverse) {
try (final Repository repository = RepositoryManager.getRepository()) {
List<PollData> allPollData = repository.getVotingRepository().getAllPolls(limit, offset, reverse);
return allPollData;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/{pollName}")
@Operation(
summary = "Info on poll",
responses = {
@ApiResponse(
description = "poll info",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(implementation = PollData.class)
)
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public PollData getPollData(@PathParam("pollName") String pollName) {
try (final Repository repository = RepositoryManager.getRepository()) {
PollData pollData = repository.getVotingRepository().fromPollName(pollName);
if (pollData == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS);
return pollData;
} catch (ApiException e) {
throw e;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/votes/{pollName}")
@Operation(
summary = "Votes on poll",
responses = {
@ApiResponse(
description = "poll votes",
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(implementation = PollVotes.class)
)
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
public PollVotes getPollVotes(@PathParam("pollName") String pollName, @QueryParam("onlyCounts") Boolean onlyCounts) {
try (final Repository repository = RepositoryManager.getRepository()) {
PollData pollData = repository.getVotingRepository().fromPollName(pollName);
if (pollData == null)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS);
List<VoteOnPollData> votes = repository.getVotingRepository().getVotes(pollName);
// Initialize map for counting votes
Map<String, Integer> voteCountMap = new HashMap<>();
for (PollOptionData optionData : pollData.getPollOptions()) {
voteCountMap.put(optionData.getOptionName(), 0);
}
int totalVotes = 0;
for (VoteOnPollData vote : votes) {
String selectedOption = pollData.getPollOptions().get(vote.getOptionIndex()).getOptionName();
if (voteCountMap.containsKey(selectedOption)) {
voteCountMap.put(selectedOption, voteCountMap.get(selectedOption) + 1);
totalVotes++;
}
}
// Convert map to list of VoteInfo
List<PollVotes.OptionCount> voteCounts = voteCountMap.entrySet().stream()
.map(entry -> new PollVotes.OptionCount(entry.getKey(), entry.getValue()))
.collect(Collectors.toList());
if (onlyCounts != null && onlyCounts) {
return new PollVotes(null, totalVotes, voteCounts);
} else {
return new PollVotes(votes, totalVotes, voteCounts);
}
} catch (ApiException e) {
throw e;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@POST
@Path("/create")
@Operation(
summary = "Build raw, unsigned, CREATE_POLL transaction",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(
implementation = CreatePollTransactionData.class
)
)
),
responses = {
@ApiResponse(
description = "raw, unsigned, CREATE_POLL transaction encoded in Base58",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
@ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE})
public String CreatePoll(CreatePollTransactionData transactionData) {
if (Settings.getInstance().isApiRestricted())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION);
try (final Repository repository = RepositoryManager.getRepository()) {
Transaction transaction = Transaction.fromData(repository, transactionData);
Transaction.ValidationResult result = transaction.isValidUnconfirmed();
if (result != Transaction.ValidationResult.OK)
throw TransactionsResource.createTransactionInvalidException(request, result);
byte[] bytes = CreatePollTransactionTransformer.toBytes(transactionData);
return Base58.encode(bytes);
} catch (TransformationException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@POST
@Path("/vote")
@Operation(
summary = "Build raw, unsigned, VOTE_ON_POLL transaction",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.APPLICATION_JSON,
schema = @Schema(
implementation = VoteOnPollTransactionData.class
)
)
),
responses = {
@ApiResponse(
description = "raw, unsigned, VOTE_ON_POLL transaction encoded in Base58",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
@ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE})
public String VoteOnPoll(VoteOnPollTransactionData transactionData) {
if (Settings.getInstance().isApiRestricted())
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION);
try (final Repository repository = RepositoryManager.getRepository()) {
Transaction transaction = Transaction.fromData(repository, transactionData);
Transaction.ValidationResult result = transaction.isValidUnconfirmed();
if (result != Transaction.ValidationResult.OK)
throw TransactionsResource.createTransactionInvalidException(request, result);
byte[] bytes = VoteOnPollTransactionTransformer.toBytes(transactionData);
return Base58.encode(bytes);
} catch (TransformationException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
}

70
src/main/java/org/qortal/api/resource/StatsResource.java Normal file
View File

@ -0,0 +1,70 @@
package org.qortal.api.resource;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.api.*;
import org.qortal.block.BlockChain;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.utils.Amounts;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import java.math.BigDecimal;
import java.util.List;
@Path("/stats")
@Tag(name = "Stats")
public class StatsResource {
private static final Logger LOGGER = LogManager.getLogger(StatsResource.class);
@Context
HttpServletRequest request;
@GET
@Path("/supply/circulating")
@Operation(
summary = "Fetch circulating QORT supply",
responses = {
@ApiResponse(
description = "circulating supply of QORT",
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string", format = "number"))
)
}
)
public BigDecimal circulatingSupply() {
long total = 0L;
try (final Repository repository = RepositoryManager.getRepository()) {
int currentHeight = repository.getBlockRepository().getBlockchainHeight();
List<BlockChain.RewardByHeight> rewardsByHeight = BlockChain.getInstance().getBlockRewardsByHeight();
int rewardIndex = rewardsByHeight.size() - 1;
BlockChain.RewardByHeight rewardInfo = rewardsByHeight.get(rewardIndex);
for (int height = currentHeight; height > 1; --height) {
if (height < rewardInfo.height) {
--rewardIndex;
rewardInfo = rewardsByHeight.get(rewardIndex);
}
total += rewardInfo.reward;
}
return Amounts.toBigDecimal(total);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
}

56
src/main/java/org/qortal/api/resource/TransactionsResource.java
View File

@ -9,6 +9,8 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import java.io.IOException;
import java.io.StringWriter;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
@ -18,19 +20,12 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.QueryParam;
import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import org.qortal.account.PrivateKeyAccount;
import org.qortal.api.ApiError;
import org.qortal.api.ApiErrors;
import org.qortal.api.ApiException;
import org.qortal.api.ApiExceptionFactory;
import org.qortal.api.*;
import org.qortal.api.model.SimpleTransactionSignRequest;
import org.qortal.controller.Controller;
import org.qortal.controller.LiteNode;
@ -220,10 +215,25 @@ public class TransactionsResource {
}
try (final Repository repository = RepositoryManager.getRepository()) {
if (repository.getBlockRepository().getHeightFromSignature(signature) == 0)
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
// Check if the block exists in either the database or archive
int height = repository.getBlockRepository().getHeightFromSignature(signature);
if (height == 0) {
height = repository.getBlockArchiveRepository().getHeightFromSignature(signature);
if (height == 0) {
// Not found in either the database or archive
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCK_UNKNOWN);
}
}
return repository.getBlockRepository().getTransactionsFromSignature(signature, limit, offset, reverse);
List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, height, height);
// Expand signatures to transactions
List<TransactionData> transactions = new ArrayList<>(signatures.size());
for (byte[] s : signatures) {
transactions.add(repository.getTransactionRepository().fromSignature(s));
}
return transactions;
} catch (ApiException e) {
throw e;
} catch (DataException e) {
@ -709,7 +719,7 @@ public class TransactionsResource {
),
responses = {
@ApiResponse(
description = "true if accepted, false otherwise",
description = "For API version 1, this returns true if accepted.\nFor API version 2, the transactionData is returned as a JSON string if accepted.",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
@ -722,7 +732,9 @@ public class TransactionsResource {
@ApiErrors({
ApiError.BLOCKCHAIN_NEEDS_SYNC, ApiError.INVALID_SIGNATURE, ApiError.INVALID_DATA, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE
})
public String processTransaction(String rawBytes58) {
public String processTransaction(String rawBytes58, @HeaderParam(ApiService.API_VERSION_HEADER) String apiVersionHeader) {
int apiVersion = ApiService.getApiVersion(request);
// Only allow a transaction to be processed if our latest block is less than 60 minutes old
// If older than this, we should first wait until the blockchain is synced
final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
@ -759,13 +771,27 @@ public class TransactionsResource {
blockchainLock.unlock();
}
return "true";
switch (apiVersion) {
case 1:
return "true";
case 2:
default:
// Marshall transactionData to string
StringWriter stringWriter = new StringWriter();
ApiRequest.marshall(stringWriter, transactionData);
return stringWriter.toString();
}
} catch (NumberFormatException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA, e);
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
} catch (InterruptedException e) {
throw createTransactionInvalidException(request, ValidationResult.NO_BLOCKCHAIN_LOCK);
} catch (IOException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e);
}
}

180
src/main/java/org/qortal/api/resource/AdminResource.java → src/main/java/org/qortal/api/restricted/resource/AdminResource.java
View File

@ -1,4 +1,4 @@
package org.qortal.api.resource;
package org.qortal.api.restricted.resource;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
@ -20,6 +20,7 @@ import java.time.LocalDate;
import java.time.LocalTime;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
@ -31,10 +32,13 @@ import javax.ws.rs.*;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import org.apache.commons.lang3.reflect.FieldUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.LoggerContext;
import org.apache.logging.log4j.core.appender.RollingFileAppender;
import org.json.JSONArray;
import org.json.JSONObject;
import org.qortal.account.Account;
import org.qortal.account.PrivateKeyAccount;
import org.qortal.api.*;
@ -42,9 +46,11 @@ import org.qortal.api.model.ActivitySummary;
import org.qortal.api.model.NodeInfo;
import org.qortal.api.model.NodeStatus;
import org.qortal.block.BlockChain;
import org.qortal.controller.AutoUpdate;
import org.qortal.controller.Controller;
import org.qortal.controller.Synchronizer;
import org.qortal.controller.Synchronizer.SynchronizationResult;
import org.qortal.controller.repository.BlockArchiveRebuilder;
import org.qortal.data.account.MintingAccountData;
import org.qortal.data.account.RewardShareData;
import org.qortal.network.Network;
@ -152,6 +158,53 @@ public class AdminResource {
return nodeStatus;
}
@GET
@Path("/settings")
@Operation(
summary = "Fetch node settings",
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = Settings.class))
)
}
)
public Settings settings() {
Settings nodeSettings = Settings.getInstance();
return nodeSettings;
}
@GET
@Path("/settings/{setting}")
@Operation(
summary = "Fetch a single node setting",
responses = {
@ApiResponse(
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
)
}
)
public String setting(@PathParam("setting") String setting) {
try {
Object settingValue = FieldUtils.readField(Settings.getInstance(), setting, true);
if (settingValue == null) {
return "null";
}
else if (settingValue instanceof String[]) {
JSONArray array = new JSONArray(settingValue);
return array.toString(4);
}
else if (settingValue instanceof List) {
JSONArray array = new JSONArray((List<Object>) settingValue);
return array.toString(4);
}
return settingValue.toString();
} catch (IllegalAccessException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA, e);
}
}
@GET
@Path("/stop")
@Operation(
@ -182,6 +235,37 @@ public class AdminResource {
return "true";
}
@GET
@Path("/restart")
@Operation(
summary = "Restart",
description = "Restart",
responses = {
@ApiResponse(
description = "\"true\"",
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
)
}
)
@SecurityRequirement(name = "apiKey")
public String restart(@HeaderParam(Security.API_KEY_HEADER) String apiKey) {
Security.checkApiCallAllowed(request);
new Thread(() -> {
// Short sleep to allow HTTP response body to be emitted
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// Not important
}
AutoUpdate.attemptRestart();
}).start();
return "true";
}
@GET
@Path("/summary")
@Operation(
@ -222,6 +306,42 @@ public class AdminResource {
}
}
@GET
@Path("/summary/alltime")
@Operation(
summary = "Summary of activity since genesis",
responses = {
@ApiResponse(
content = @Content(schema = @Schema(implementation = ActivitySummary.class))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
@SecurityRequirement(name = "apiKey")
public ActivitySummary allTimeSummary(@HeaderParam(Security.API_KEY_HEADER) String apiKey) {
Security.checkApiCallAllowed(request);
ActivitySummary summary = new ActivitySummary();
try (final Repository repository = RepositoryManager.getRepository()) {
int startHeight = 1;
long start = repository.getBlockRepository().fromHeight(startHeight).getTimestamp();
int endHeight = repository.getBlockRepository().getBlockchainHeight();
summary.setBlockCount(endHeight - startHeight);
summary.setTransactionCountByType(repository.getTransactionRepository().getTransactionSummary(startHeight + 1, endHeight));
summary.setAssetsIssued(repository.getAssetRepository().getRecentAssetIds(start).size());
summary.setNamesRegistered (repository.getNameRepository().getRecentNames(start).size());
return summary;
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@GET
@Path("/enginestats")
@Operation(
@ -698,6 +818,64 @@ public class AdminResource {
}
}
@POST
@Path("/repository/archive/rebuild")
@Operation(
summary = "Rebuild archive",
description = "Rebuilds archive files, using the specified serialization version",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "number", example = "2"
)
)
),
responses = {
@ApiResponse(
description = "\"true\"",
content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string"))
)
}
)
@ApiErrors({ApiError.REPOSITORY_ISSUE})
@SecurityRequirement(name = "apiKey")
public String rebuildArchive(@HeaderParam(Security.API_KEY_HEADER) String apiKey, Integer serializationVersion) {
Security.checkApiCallAllowed(request);
// Default serialization version to value specified in settings
if (serializationVersion == null) {
serializationVersion = Settings.getInstance().getDefaultArchiveVersion();
}
try {
// We don't actually need to lock the blockchain here, but we'll do it anyway so that
// the node can focus on rebuilding rather than synchronizing / minting.
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
blockchainLock.lockInterruptibly();
try {
BlockArchiveRebuilder blockArchiveRebuilder = new BlockArchiveRebuilder(serializationVersion);
blockArchiveRebuilder.start();
return "true";
} catch (IOException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA, e);
} finally {
blockchainLock.unlock();
}
} catch (InterruptedException e) {
// We couldn't lock blockchain to perform rebuild
return "false";
} catch (DataException e) {
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e);
}
}
@DELETE
@Path("/repository")
@Operation(

4
src/main/java/org/qortal/api/resource/BootstrapResource.java → src/main/java/org/qortal/api/restricted/resource/BootstrapResource.java
View File

@ -1,4 +1,4 @@
package org.qortal.api.resource;
package org.qortal.api.restricted.resource;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
@ -60,7 +60,7 @@ public class BootstrapResource {
bootstrap.validateBlockchain();
return bootstrap.create();
} catch (DataException | InterruptedException | IOException e) {
} catch (Exception e) {
LOGGER.info("Unable to create bootstrap", e);
throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage());
}

105
src/main/java/org/qortal/api/resource/RenderResource.java → src/main/java/org/qortal/api/restricted/resource/RenderResource.java
View File

@ -1,4 +1,4 @@
package org.qortal.api.resource;
package org.qortal.api.restricted.resource;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
@ -8,7 +8,6 @@ import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import java.io.*;
import java.nio.file.Paths;
import java.util.Map;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
@ -28,8 +27,8 @@ import org.qortal.arbitrary.exception.MissingDataException;
import org.qortal.controller.arbitrary.ArbitraryDataRenderManager;
import org.qortal.data.transaction.ArbitraryTransactionData.*;
import org.qortal.repository.DataException;
import org.qortal.settings.Settings;
import org.qortal.arbitrary.ArbitraryDataFile.*;
import org.qortal.settings.Settings;
import org.qortal.utils.Base58;
@ -43,60 +42,6 @@ public class RenderResource {
@Context HttpServletResponse response;
@Context ServletContext context;
@POST
@Path("/preview")
@Operation(
summary = "Generate preview URL based on a user-supplied path and service",
requestBody = @RequestBody(
required = true,
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string", example = "/Users/user/Documents/MyStaticWebsite"
)
)
),
responses = {
@ApiResponse(
description = "a temporary URL to preview the website",
content = @Content(
mediaType = MediaType.TEXT_PLAIN,
schema = @Schema(
type = "string"
)
)
)
}
)
@SecurityRequirement(name = "apiKey")
public String preview(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String directoryPath) {
Security.checkApiCallAllowed(request);
Method method = Method.PUT;
Compression compression = Compression.ZIP;
ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath),
null, Service.WEBSITE, null, method, compression,
null, null, null, null);
try {
arbitraryDataWriter.save();
} catch (IOException | DataException | InterruptedException | MissingDataException e) {
LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE);
} catch (RuntimeException e) {
LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage());
throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA);
}
ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile();
if (arbitraryDataFile != null) {
String digest58 = arbitraryDataFile.digest58();
if (digest58 != null) {
return "http://localhost:12393/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret());
}
}
return "Unable to generate preview URL";
}
@POST
@Path("/authorize/{resourceId}")
@SecurityRequirement(name = "apiKey")
@ -140,8 +85,10 @@ public class RenderResource {
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, "/", null, "/render/signature", true, true, theme);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme);
}
@GET
@ -149,8 +96,10 @@ public class RenderResource {
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, inPath,null, "/render/signature", true, true, theme);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null);
return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme);
}
@GET
@ -158,8 +107,10 @@ public class RenderResource {
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, "/", secret58, "/render/hash", true, false, theme);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme);
}
@GET
@ -168,8 +119,10 @@ public class RenderResource {
public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath,
@QueryParam("secret") String secret58,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, inPath, secret58, "/render/hash", true, false, theme);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null);
return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme);
}
@GET
@ -178,10 +131,13 @@ public class RenderResource {
public HttpServletResponse getPathByName(@PathParam("service") Service service,
@PathParam("name") String name,
@PathParam("path") String inPath,
@QueryParam("identifier") String identifier,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, name, service, null);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, name, service, null);
String prefix = String.format("/render/%s", service);
return this.get(name, ResourceIdType.NAME, service, inPath, null, prefix, true, true, theme);
return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme);
}
@GET
@ -189,19 +145,22 @@ public class RenderResource {
@SecurityRequirement(name = "apiKey")
public HttpServletResponse getIndexByName(@PathParam("service") Service service,
@PathParam("name") String name,
@QueryParam("identifier") String identifier,
@QueryParam("theme") String theme) {
Security.requirePriorAuthorization(request, name, service, null);
if (!Settings.getInstance().isQDNAuthBypassEnabled())
Security.requirePriorAuthorization(request, name, service, null);
String prefix = String.format("/render/%s", service);
return this.get(name, ResourceIdType.NAME, service, "/", null, prefix, true, true, theme);
return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme);
}
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
String secret58, String prefix, boolean usePrefix, boolean async, String theme) {
private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String theme) {
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath,
secret58, prefix, usePrefix, async, request, response, context);
ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath,
secret58, prefix, includeResourceIdInPrefix, async, "render", request, response, context);
if (theme != null) {
renderer.setTheme(theme);

18
src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java
View File

@ -2,7 +2,9 @@ package org.qortal.api.websocket;
import java.io.IOException;
import java.io.StringWriter;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicReference;
import org.eclipse.jetty.websocket.api.Session;
@ -21,6 +23,8 @@ import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import static org.qortal.data.chat.ChatMessage.Encoding;
@WebSocket
@SuppressWarnings("serial")
public class ActiveChatsWebSocket extends ApiWebSocket {
@ -62,7 +66,9 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
@OnWebSocketMessage
public void onWebSocketMessage(Session session, String message) {
/* ignored */
if (Objects.equals(message, "ping")) {
session.getRemote().sendStringByFuture("pong");
}
}
private void onNotify(Session session, ChatTransactionData chatTransactionData, String ourAddress, AtomicReference<String> previousOutput) {
@ -75,7 +81,7 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
}
try (final Repository repository = RepositoryManager.getRepository()) {
ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress);
ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session));
StringWriter stringWriter = new StringWriter();
@ -93,4 +99,12 @@ public class ActiveChatsWebSocket extends ApiWebSocket {
}
}
private Encoding getTargetEncoding(Session session) {
// Default to Base58 if not specified, for backwards support
Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
List<String> encodingList = queryParams.get("encoding");
String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58";
return Encoding.valueOf(encoding);
}
}

43
src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java
View File

@ -2,10 +2,7 @@ package org.qortal.api.websocket;
import java.io.IOException;
import java.io.StringWriter;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.*;
import org.eclipse.jetty.websocket.api.Session;
import org.eclipse.jetty.websocket.api.WebSocketException;
@ -22,6 +19,8 @@ import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import static org.qortal.data.chat.ChatMessage.Encoding;
@WebSocket
@SuppressWarnings("serial")
public class ChatMessagesWebSocket extends ApiWebSocket {
@ -35,6 +34,16 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
@Override
public void onWebSocketConnect(Session session) {
Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
Encoding encoding = getTargetEncoding(session);
List<String> limitList = queryParams.get("limit");
Integer limit = (limitList != null && limitList.size() == 1) ? Integer.parseInt(limitList.get(0)) : null;
List<String> offsetList = queryParams.get("offset");
Integer offset = (offsetList != null && offsetList.size() == 1) ? Integer.parseInt(offsetList.get(0)) : null;
List<String> reverseList = queryParams.get("offset");
Boolean reverse = (reverseList != null && reverseList.size() == 1) ? Boolean.getBoolean(reverseList.get(0)) : null;
List<String> txGroupIds = queryParams.get("txGroupId");
if (txGroupIds != null && txGroupIds.size() == 1) {
@ -47,7 +56,11 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
txGroupId,
null,
null,
null, null, null);
null,
null,
null,
encoding,
limit, offset, reverse);
sendMessages(session, chatMessages);
} catch (DataException e) {
@ -74,8 +87,12 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
null,
null,
null,
null,
null,
involvingAddresses,
null, null, null);
null,
encoding,
limit, offset, reverse);
sendMessages(session, chatMessages);
} catch (DataException e) {
@ -101,7 +118,9 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
@OnWebSocketMessage
public void onWebSocketMessage(Session session, String message) {
/* ignored */
if (Objects.equals(message, "ping")) {
session.getRemote().sendStringByFuture("pong");
}
}
private void onNotify(Session session, ChatTransactionData chatTransactionData, int txGroupId) {
@ -149,7 +168,7 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
// Convert ChatTransactionData to ChatMessage
ChatMessage chatMessage;
try (final Repository repository = RepositoryManager.getRepository()) {
chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData);
chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData, getTargetEncoding(session));
} catch (DataException e) {
// No output this time?
return;
@ -158,4 +177,12 @@ public class ChatMessagesWebSocket extends ApiWebSocket {
sendMessages(session, Collections.singletonList(chatMessage));
}
private Encoding getTargetEncoding(Session session) {
// Default to Base58 if not specified, for backwards support
Map<String, List<String>> queryParams = session.getUpgradeRequest().getParameterMap();
List<String> encodingList = queryParams.get("encoding");
String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58";
return Encoding.valueOf(encoding);
}
}

20
src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java
View File

@ -24,6 +24,7 @@ import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory;
import org.qortal.api.model.CrossChainOfferSummary;
import org.qortal.controller.Controller;
import org.qortal.controller.Synchronizer;
import org.qortal.controller.tradebot.TradeBot;
import org.qortal.crosschain.SupportedBlockchain;
import org.qortal.crosschain.ACCT;
import org.qortal.crosschain.AcctMode;
@ -315,7 +316,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
throw new DataException("Couldn't fetch historic trades from repository");
for (ATStateData historicAtState : historicAtStates) {
CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null);
CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null, null);
if (!isHistoric.test(historicOfferSummary))
continue;
@ -330,8 +331,10 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
}
}
private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, Long timestamp) throws DataException {
CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, CrossChainTradeData crossChainTradeData, Long timestamp) throws DataException {
if (crossChainTradeData == null) {
crossChainTradeData = acct.populateTradeData(repository, atState);
}
long atStateTimestamp;
@ -346,9 +349,16 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener {
private static List<CrossChainOfferSummary> produceSummaries(Repository repository, ACCT acct, List<ATStateData> atStates, Long timestamp) throws DataException {
List<CrossChainOfferSummary> offerSummaries = new ArrayList<>();
for (ATStateData atState : atStates) {
CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState);
for (ATStateData atState : atStates)
offerSummaries.add(produceSummary(repository, acct, atState, timestamp));
// Ignore trade if it has failed
if (TradeBot.getInstance().isFailedTrade(repository, crossChainTradeData)) {
continue;
}
offerSummaries.add(produceSummary(repository, acct, atState, crossChainTradeData, timestamp));
}
return offerSummaries;
}

11
src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java
View File

@ -2,6 +2,7 @@ package org.qortal.arbitrary;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.arbitrary.exception.DataNotPublishedException;
import org.qortal.arbitrary.exception.MissingDataException;
import org.qortal.arbitrary.metadata.ArbitraryDataMetadataCache;
import org.qortal.arbitrary.misc.Service;
@ -53,10 +54,6 @@ public class ArbitraryDataBuilder {
/**
* Process transactions, but do not build anything
* This is useful for checking the status of a given resource
*
* @throws DataException
* @throws IOException
* @throws MissingDataException
*/
public void process() throws DataException, IOException, MissingDataException {
this.fetchTransactions();
@ -68,10 +65,6 @@ public class ArbitraryDataBuilder {
/**
* Build the latest state of a given resource
*
* @throws DataException
* @throws IOException
* @throws MissingDataException
*/
public void build() throws DataException, IOException, MissingDataException {
this.process();
@ -88,7 +81,7 @@ public class ArbitraryDataBuilder {
if (latestPut == null) {
String message = String.format("Couldn't find PUT transaction for name %s, service %s and identifier %s",
this.name, this.service, this.identifierString());
throw new DataException(message);
throw new DataNotPublishedException(message);
}
this.latestPutTransaction = latestPut;

85
src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java
View File

@ -4,6 +4,7 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
import org.qortal.crypto.Crypto;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.repository.DataException;
import org.qortal.settings.Settings;
import org.qortal.utils.Base58;
@ -15,7 +16,6 @@ import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.*;
import java.util.stream.Stream;
import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
@ -79,17 +79,31 @@ public class ArbitraryDataFile {
this.signature = signature;
}
public ArbitraryDataFile(byte[] fileContent, byte[] signature) throws DataException {
public ArbitraryDataFile(byte[] fileContent, byte[] signature, boolean useTemporaryFile) throws DataException {
if (fileContent == null) {
LOGGER.error("fileContent is null");
return;
}
this.chunks = new ArrayList<>();
this.hash58 = Base58.encode(Crypto.digest(fileContent));
this.signature = signature;
LOGGER.trace(String.format("File digest: %s, size: %d bytes", this.hash58, fileContent.length));
Path outputFilePath = getOutputFilePath(this.hash58, signature, true);
Path outputFilePath;
if (useTemporaryFile) {
try {
outputFilePath = Files.createTempFile("qortalRawData", null);
outputFilePath.toFile().deleteOnExit();
}
catch (IOException e) {
throw new DataException(String.format("Unable to write data with hash %s to temporary file: %s", this.hash58, e.getMessage()));
}
}
else {
outputFilePath = getOutputFilePath(this.hash58, signature, true);
}
File outputFile = outputFilePath.toFile();
try (FileOutputStream outputStream = new FileOutputStream(outputFile)) {
outputStream.write(fileContent);
@ -111,6 +125,41 @@ public class ArbitraryDataFile {
return ArbitraryDataFile.fromHash58(Base58.encode(hash), signature);
}
public static ArbitraryDataFile fromRawData(byte[] data, byte[] signature) throws DataException {
if (data == null) {
return null;
}
return new ArbitraryDataFile(data, signature, true);
}
public static ArbitraryDataFile fromTransactionData(ArbitraryTransactionData transactionData) throws DataException {
ArbitraryDataFile arbitraryDataFile = null;
byte[] signature = transactionData.getSignature();
byte[] data = transactionData.getData();
if (data == null) {
return null;
}
// Create data file
switch (transactionData.getDataType()) {
case DATA_HASH:
arbitraryDataFile = ArbitraryDataFile.fromHash(data, signature);
break;
case RAW_DATA:
arbitraryDataFile = ArbitraryDataFile.fromRawData(data, signature);
break;
}
// Set metadata hash
if (arbitraryDataFile != null) {
arbitraryDataFile.setMetadataHash(transactionData.getMetadataHash());
}
return arbitraryDataFile;
}
public static ArbitraryDataFile fromPath(Path path, byte[] signature) {
if (path == null) {
return null;
@ -260,6 +309,11 @@ public class ArbitraryDataFile {
this.chunks = new ArrayList<>();
if (file != null) {
if (file.exists() && file.length() <= chunkSize) {
// No need to split into chunks if we're already below the chunk size
return 0;
}
try (FileInputStream fileInputStream = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fileInputStream)) {
@ -388,12 +442,15 @@ public class ArbitraryDataFile {
return false;
}
public boolean deleteAll() {
public boolean deleteAll(boolean deleteMetadata) {
// Delete the complete file
boolean fileDeleted = this.delete();
// Delete the metadata file
boolean metadataDeleted = this.deleteMetadata();
// Delete the metadata file if requested
boolean metadataDeleted = false;
if (deleteMetadata) {
metadataDeleted = this.deleteMetadata();
}
// Delete the individual chunks
boolean chunksDeleted = this.deleteAllChunks();
@ -612,6 +669,22 @@ public class ArbitraryDataFile {
return this.chunks.size();
}
public int fileCount() {
int fileCount = this.chunkCount();
if (fileCount == 0) {
// Transactions without any chunks can already be treated as a complete file
fileCount++;
}
if (this.getMetadataHash() != null) {
// Add the metadata file
fileCount++;
}
return fileCount;
}
public List<ArbitraryDataFileChunk> getChunks() {
return this.chunks;
}

2
src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java
View File

@ -18,7 +18,7 @@ public class ArbitraryDataFileChunk extends ArbitraryDataFile {
}
public ArbitraryDataFileChunk(byte[] fileContent, byte[] signature) throws DataException {
super(fileContent, signature);
super(fileContent, signature, false);
}
public static ArbitraryDataFileChunk fromHash58(String hash58, byte[] signature) throws DataException {

119
src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java
View File

@ -4,11 +4,11 @@ import org.apache.commons.io.FileUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.arbitrary.exception.DataNotPublishedException;
import org.qortal.arbitrary.exception.MissingDataException;
import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.arbitrary.ArbitraryDataBuildManager;
import org.qortal.controller.arbitrary.ArbitraryDataManager;
import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
import org.qortal.crypto.AES;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.ArbitraryTransactionData.*;
@ -18,10 +18,7 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.arbitrary.ArbitraryDataFile.*;
import org.qortal.settings.Settings;
import org.qortal.transform.Transformer;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58;
import org.qortal.utils.FilesystemUtils;
import org.qortal.utils.ZipUtils;
import org.qortal.utils.*;
import javax.crypto.BadPaddingException;
import javax.crypto.IllegalBlockSizeException;
@ -37,6 +34,9 @@ import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
public class ArbitraryDataReader {
@ -59,6 +59,13 @@ public class ArbitraryDataReader {
private int layerCount;
private byte[] latestSignature;
// The resource being read
ArbitraryDataResource arbitraryDataResource = null;
// Track resources that are currently being loaded, to avoid duplicate concurrent builds
// TODO: all builds could be handled by the build queue (even synchronous ones), to avoid the need for this
private static Map<String, Long> inProgress = Collections.synchronizedMap(new HashMap<>());
public ArbitraryDataReader(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) {
// Ensure names are always lowercase
if (resourceIdType == ResourceIdType.NAME) {
@ -115,6 +122,11 @@ public class ArbitraryDataReader {
return new ArbitraryDataBuildQueueItem(this.resourceId, this.resourceIdType, this.service, this.identifier);
}
private ArbitraryDataResource createArbitraryDataResource() {
return new ArbitraryDataResource(this.resourceId, this.resourceIdType, this.service, this.identifier);
}
/**
* loadAsynchronously
*
@ -148,9 +160,6 @@ public class ArbitraryDataReader {
* If no exception is thrown, you can then use getFilePath() to access the data immediately after returning
*
* @param overwrite - set to true to force rebuild an existing cache
* @throws IOException
* @throws DataException
* @throws MissingDataException
*/
public void loadSynchronously(boolean overwrite) throws DataException, IOException, MissingDataException {
try {
@ -162,6 +171,14 @@ public class ArbitraryDataReader {
return;
}
this.arbitraryDataResource = this.createArbitraryDataResource();
// Don't allow duplicate loads
if (!this.canStartLoading()) {
LOGGER.debug("Skipping duplicate load of {}", this.arbitraryDataResource);
return;
}
this.preExecute();
this.deleteExistingFiles();
this.fetch();
@ -169,10 +186,18 @@ public class ArbitraryDataReader {
this.uncompress();
this.validate();
} catch (DataNotPublishedException e) {
if (e.getMessage() != null) {
// Log the message only, to avoid spamming the logs with a full stack trace
LOGGER.debug("DataNotPublishedException when trying to load QDN resource: {}", e.getMessage());
}
this.deleteWorkingDirectory();
throw e;
} catch (DataException e) {
LOGGER.info("DataException when trying to load QDN resource", e);
this.deleteWorkingDirectory();
throw new DataException(e.getMessage());
throw e;
} finally {
this.postExecute();
@ -181,6 +206,7 @@ public class ArbitraryDataReader {
private void preExecute() throws DataException {
ArbitraryDataBuildManager.getInstance().setBuildInProgress(true);
this.checkEnabled();
this.createWorkingDirectory();
this.createUncompressedDirectory();
@ -188,6 +214,9 @@ public class ArbitraryDataReader {
private void postExecute() {
ArbitraryDataBuildManager.getInstance().setBuildInProgress(false);
this.arbitraryDataResource = this.createArbitraryDataResource();
ArbitraryDataReader.inProgress.remove(this.arbitraryDataResource.getUniqueKey());
}
private void checkEnabled() throws DataException {
@ -196,6 +225,17 @@ public class ArbitraryDataReader {
}
}
private boolean canStartLoading() {
// Avoid duplicate builds if we're already loading this resource
String uniqueKey = this.arbitraryDataResource.getUniqueKey();
if (ArbitraryDataReader.inProgress.containsKey(uniqueKey)) {
return false;
}
ArbitraryDataReader.inProgress.put(uniqueKey, NTP.getTime());
return true;
}
private void createWorkingDirectory() throws DataException {
try {
Files.createDirectories(this.workingPath);
@ -207,7 +247,6 @@ public class ArbitraryDataReader {
/**
* Working directory should only be deleted on failure, since it is currently used to
* serve a cached version of the resource for subsequent requests.
* @throws IOException
*/
private void deleteWorkingDirectory() {
try {
@ -287,7 +326,7 @@ public class ArbitraryDataReader {
break;
default:
throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType.toString()));
throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType));
}
}
@ -343,11 +382,6 @@ public class ArbitraryDataReader {
throw new DataException(String.format("Transaction data not found for signature %s", this.resourceId));
}
// Load hashes
byte[] digest = transactionData.getData();
byte[] metadataHash = transactionData.getMetadataHash();
byte[] signature = transactionData.getSignature();
// Load secret
byte[] secret = transactionData.getSecret();
if (secret != null) {
@ -355,16 +389,17 @@ public class ArbitraryDataReader {
}
// Load data file(s)
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
ArbitraryTransactionUtils.checkAndRelocateMiscFiles(transactionData);
arbitraryDataFile.setMetadataHash(metadataHash);
if (arbitraryDataFile == null) {
throw new DataException(String.format("arbitraryDataFile is null"));
}
if (!arbitraryDataFile.allFilesExist()) {
if (ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())) {
if (ListUtils.isNameBlocked(transactionData.getName())) {
throw new DataException(
String.format("Unable to request missing data for file %s because the name is blocked", arbitraryDataFile));
}
else {
} else {
// Ask the arbitrary data manager to fetch data for this transaction
String message;
if (this.canRequestMissingFiles) {
@ -375,8 +410,7 @@ public class ArbitraryDataReader {
} else {
message = String.format("Unable to reissue request for missing file %s for signature %s due to rate limit. Please try again later.", arbitraryDataFile, Base58.encode(transactionData.getSignature()));
}
}
else {
} else {
message = String.format("Missing data for file %s", arbitraryDataFile);
}
@ -386,21 +420,25 @@ public class ArbitraryDataReader {
}
}
if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) {
// We have all the chunks but not the complete file, so join them
arbitraryDataFile.join();
// Data hashes need some extra processing
if (transactionData.getDataType() == DataType.DATA_HASH) {
if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) {
// We have all the chunks but not the complete file, so join them
arbitraryDataFile.join();
}
// If the complete file still doesn't exist then something went wrong
if (!arbitraryDataFile.exists()) {
throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile));
}
// Ensure the complete hash matches the joined chunks
if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) {
// Delete the invalid file
arbitraryDataFile.delete();
throw new DataException("Unable to validate complete file hash");
}
}
// If the complete file still doesn't exist then something went wrong
if (!arbitraryDataFile.exists()) {
throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile));
}
// Ensure the complete hash matches the joined chunks
if (!Arrays.equals(arbitraryDataFile.digest(), digest)) {
// Delete the invalid file
arbitraryDataFile.delete();
throw new DataException("Unable to validate complete file hash");
}
// Ensure the file's size matches the size reported by the transaction (throws a DataException if not)
arbitraryDataFile.validateFileSize(transactionData.getSize());
@ -427,10 +465,11 @@ public class ArbitraryDataReader {
byte[] secret = this.secret58 != null ? Base58.decode(this.secret58) : null;
if (secret != null && secret.length == Transformer.AES256_LENGTH) {
try {
LOGGER.info("Decrypting using algorithm {}...", algorithm);
LOGGER.debug("Decrypting {} using algorithm {}...", this.arbitraryDataResource, algorithm);
Path unencryptedPath = Paths.get(this.workingPath.toString(), "zipped.zip");
SecretKey aesKey = new SecretKeySpec(secret, 0, secret.length, "AES");
AES.decryptFile(algorithm, aesKey, this.filePath.toString(), unencryptedPath.toString());
LOGGER.debug("Finished decrypting {} using algorithm {}", this.arbitraryDataResource, algorithm);
// Replace filePath pointer with the encrypted file path
// Don't delete the original ArbitraryDataFile, as this is handled in the cleanup phase
@ -438,7 +477,7 @@ public class ArbitraryDataReader {
} catch (NoSuchAlgorithmException | InvalidAlgorithmParameterException | NoSuchPaddingException
| BadPaddingException | IllegalBlockSizeException | IOException | InvalidKeyException e) {
LOGGER.info(String.format("Exception when decrypting using algorithm %s", algorithm), e);
LOGGER.info(String.format("Exception when decrypting %s using algorithm %s", this.arbitraryDataResource, algorithm), e);
throw new DataException(String.format("Unable to decrypt file at path %s using algorithm %s: %s", this.filePath, algorithm, e.getMessage()));
}
} else {
@ -465,7 +504,9 @@ public class ArbitraryDataReader {
// Handle each type of compression
if (compression == Compression.ZIP) {
LOGGER.debug("Unzipping {}...", this.arbitraryDataResource);
ZipUtils.unzip(this.filePath.toString(), this.uncompressedPath.getParent().toString());
LOGGER.debug("Finished unzipping {}", this.arbitraryDataResource);
}
else if (compression == Compression.NONE) {
Files.createDirectories(this.uncompressedPath);
@ -501,10 +542,12 @@ public class ArbitraryDataReader {
private void validate() throws IOException, DataException {
if (this.service.isValidationRequired()) {
LOGGER.debug("Validating {}...", this.arbitraryDataResource);
Service.ValidationResult result = this.service.validate(this.filePath);
if (result != Service.ValidationResult.OK) {
throw new DataException(String.format("Validation of %s failed: %s", this.service, result.toString()));
}
LOGGER.debug("Finished validating {}", this.arbitraryDataResource);
}
}

83
src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java
View File

@ -2,6 +2,7 @@ package org.qortal.arbitrary;
import com.google.common.io.Resources;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.api.HTMLParser;
@ -34,36 +35,40 @@ public class ArbitraryDataRenderer {
private final String resourceId;
private final ResourceIdType resourceIdType;
private final Service service;
private final String identifier;
private String theme = "light";
private String inPath;
private final String secret58;
private final String prefix;
private final boolean usePrefix;
private final boolean includeResourceIdInPrefix;
private final boolean async;
private final String qdnContext;
private final HttpServletRequest request;
private final HttpServletResponse response;
private final ServletContext context;
public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String inPath,
String secret58, String prefix, boolean usePrefix, boolean async,
public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String identifier,
String inPath, String secret58, String prefix, boolean includeResourceIdInPrefix, boolean async, String qdnContext,
HttpServletRequest request, HttpServletResponse response, ServletContext context) {
this.resourceId = resourceId;
this.resourceIdType = resourceIdType;
this.service = service;
this.identifier = identifier != null ? identifier : "default";
this.inPath = inPath;
this.secret58 = secret58;
this.prefix = prefix;
this.usePrefix = usePrefix;
this.includeResourceIdInPrefix = includeResourceIdInPrefix;
this.async = async;
this.qdnContext = qdnContext;
this.request = request;
this.response = response;
this.context = context;
}
public HttpServletResponse render() {
if (!inPath.startsWith(File.separator)) {
inPath = File.separator + inPath;
if (!inPath.startsWith("/")) {
inPath = "/" + inPath;
}
// Don't render data if QDN is disabled
@ -71,14 +76,14 @@ public class ArbitraryDataRenderer {
return ArbitraryDataRenderer.getResponse(response, 500, "QDN is disabled in settings");
}
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, null);
ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, identifier);
arbitraryDataReader.setSecret58(secret58); // Optional, used for loading encrypted file hashes only
try {
if (!arbitraryDataReader.isCachedDataAvailable()) {
// If async is requested, show a loading screen whilst build is in progress
if (async) {
arbitraryDataReader.loadAsynchronously(false, 10);
return this.getLoadingResponse(service, resourceId, theme);
return this.getLoadingResponse(service, resourceId, identifier, theme);
}
// Otherwise, loop until we have data
@ -111,23 +116,64 @@ public class ArbitraryDataRenderer {
}
String unzippedPath = path.toString();
// Set path automatically for single file resources (except for apps, which handle routing differently)
String[] files = ArrayUtils.removeElement(new File(unzippedPath).list(), ".qortal");
if (files.length == 1 && this.service != Service.APP) {
// This is a single file resource
inPath = files[0];
}
try {
String filename = this.getFilename(unzippedPath, inPath);
String filePath = Paths.get(unzippedPath, filename).toString();
Path filePath = Paths.get(unzippedPath, filename);
boolean usingCustomRouting = false;
if (Files.isDirectory(filePath) && (!inPath.endsWith("/"))) {
inPath = inPath + "/";
filename = this.getFilename(unzippedPath, inPath);
filePath = Paths.get(unzippedPath, filename);
}
// If the file doesn't exist, we may need to route the request elsewhere, or cleanup
if (!Files.exists(filePath)) {
if (inPath.equals("/")) {
// Delete the unzipped folder if no index file was found
try {
FileUtils.deleteDirectory(new File(unzippedPath));
} catch (IOException e) {
LOGGER.debug("Unable to delete directory: {}", unzippedPath, e);
}
}
// If this is an app, then forward all unhandled requests to the index, to give the app the option to route it
if (this.service == Service.APP) {
// Locate index file
List<String> indexFiles = ArbitraryDataRenderer.indexFiles();
for (String indexFile : indexFiles) {
Path indexPath = Paths.get(unzippedPath, indexFile);
if (Files.exists(indexPath)) {
// Forward request to index file
filePath = indexPath;
filename = indexFile;
usingCustomRouting = true;
break;
}
}
}
}
if (HTMLParser.isHtmlFile(filename)) {
// HTML file - needs to be parsed
byte[] data = Files.readAllBytes(Paths.get(filePath)); // TODO: limit file size that can be read into memory
HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data);
byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory
HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, includeResourceIdInPrefix, data, qdnContext, service, identifier, theme, usingCustomRouting);
htmlParser.addAdditionalHeaderTags();
response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' blob:; img-src 'self' data: blob:;");
response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;");
response.setContentType(context.getMimeType(filename));
response.setContentLength(htmlParser.getData().length);
response.getOutputStream().write(htmlParser.getData());
}
else {
// Regular file - can be streamed directly
File file = new File(filePath);
File file = filePath.toFile();
FileInputStream inputStream = new FileInputStream(file);
response.addHeader("Content-Security-Policy", "default-src 'self'");
response.setContentType(context.getMimeType(filename));
@ -143,14 +189,6 @@ public class ArbitraryDataRenderer {
return response;
} catch (FileNotFoundException | NoSuchFileException e) {
LOGGER.info("Unable to serve file: {}", e.getMessage());
if (inPath.equals("/")) {
// Delete the unzipped folder if no index file was found
try {
FileUtils.deleteDirectory(new File(unzippedPath));
} catch (IOException ioException) {
LOGGER.debug("Unable to delete directory: {}", unzippedPath, e);
}
}
} catch (IOException e) {
LOGGER.info("Unable to serve file at path {}: {}", inPath, e.getMessage());
}
@ -172,7 +210,7 @@ public class ArbitraryDataRenderer {
return userPath;
}
private HttpServletResponse getLoadingResponse(Service service, String name, String theme) {
private HttpServletResponse getLoadingResponse(Service service, String name, String identifier, String theme) {
String responseString = "";
URL url = Resources.getResource("loading/index.html");
try {
@ -181,6 +219,7 @@ public class ArbitraryDataRenderer {
// Replace vars
responseString = responseString.replace("%%SERVICE%%", service.toString());
responseString = responseString.replace("%%NAME%%", name);
responseString = responseString.replace("%%IDENTIFIER%%", identifier);
responseString = responseString.replace("%%THEME%%", theme);
} catch (IOException e) {

56
src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java
View File

@ -3,6 +3,7 @@ package org.qortal.arbitrary;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType;
import org.qortal.arbitrary.exception.DataNotPublishedException;
import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.arbitrary.ArbitraryDataBuildManager;
@ -10,13 +11,13 @@ import org.qortal.controller.arbitrary.ArbitraryDataManager;
import org.qortal.controller.arbitrary.ArbitraryDataStorageManager;
import org.qortal.data.arbitrary.ArbitraryResourceStatus;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.list.ResourceListManager;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.FilesystemUtils;
import org.qortal.utils.ListUtils;
import org.qortal.utils.NTP;
import java.io.IOException;
@ -42,6 +43,7 @@ public class ArbitraryDataResource {
private int layerCount;
private Integer localChunkCount = null;
private Integer totalChunkCount = null;
private boolean exists = false;
public ArbitraryDataResource(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) {
this.resourceId = resourceId.toLowerCase();
@ -60,6 +62,10 @@ public class ArbitraryDataResource {
// Avoid this for "quick" statuses, to speed things up
if (!quick) {
this.calculateChunkCounts();
if (!this.exists) {
return new ArbitraryResourceStatus(Status.NOT_PUBLISHED, this.localChunkCount, this.totalChunkCount);
}
}
if (resourceIdType != ResourceIdType.NAME) {
@ -68,8 +74,7 @@ public class ArbitraryDataResource {
}
// Check if the name is blocked
if (ResourceListManager.getInstance()
.listContains("blockedNames", this.resourceId, false)) {
if (ListUtils.isNameBlocked(this.resourceId)) {
return new ArbitraryResourceStatus(Status.BLOCKED, this.localChunkCount, this.totalChunkCount);
}
@ -134,21 +139,23 @@ public class ArbitraryDataResource {
return null;
}
public boolean delete() {
public boolean delete(boolean deleteMetadata) {
try {
this.fetchTransactions();
if (this.transactions == null) {
return false;
}
List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
for (ArbitraryTransactionData transactionData : transactionDataList) {
byte[] hash = transactionData.getData();
byte[] metadataHash = transactionData.getMetadataHash();
byte[] signature = transactionData.getSignature();
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
arbitraryDataFile.setMetadataHash(metadataHash);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
if (arbitraryDataFile == null) {
continue;
}
// Delete any chunks or complete files from each transaction
arbitraryDataFile.deleteAll();
arbitraryDataFile.deleteAll(deleteMetadata);
}
// Also delete cached data for the entire resource
@ -192,6 +199,9 @@ public class ArbitraryDataResource {
try {
this.fetchTransactions();
if (this.transactions == null) {
return false;
}
List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
@ -211,6 +221,14 @@ public class ArbitraryDataResource {
private void calculateChunkCounts() {
try {
this.fetchTransactions();
if (this.transactions == null) {
this.exists = false;
this.localChunkCount = 0;
this.totalChunkCount = 0;
return;
}
this.exists = true;
List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
int localChunkCount = 0;
@ -230,6 +248,9 @@ public class ArbitraryDataResource {
private boolean isRateLimited() {
try {
this.fetchTransactions();
if (this.transactions == null) {
return true;
}
List<ArbitraryTransactionData> transactionDataList = new ArrayList<>(this.transactions);
@ -253,6 +274,10 @@ public class ArbitraryDataResource {
private boolean isDataPotentiallyAvailable() {
try {
this.fetchTransactions();
if (this.transactions == null) {
return false;
}
Long now = NTP.getTime();
if (now == null) {
return false;
@ -284,6 +309,10 @@ public class ArbitraryDataResource {
private boolean isDownloading() {
try {
this.fetchTransactions();
if (this.transactions == null) {
return false;
}
Long now = NTP.getTime();
if (now == null) {
return false;
@ -325,7 +354,7 @@ public class ArbitraryDataResource {
if (latestPut == null) {
String message = String.format("Couldn't find PUT transaction for name %s, service %s and identifier %s",
this.resourceId, this.service, this.identifierString());
throw new DataException(message);
throw new DataNotPublishedException(message);
}
this.latestPutTransaction = latestPut;
@ -336,7 +365,10 @@ public class ArbitraryDataResource {
this.transactions = transactionDataList;
this.layerCount = transactionDataList.size();
} catch (DataException e) {
} catch (DataNotPublishedException e) {
// Ignore without logging
}
catch (DataException e) {
LOGGER.info(String.format("Repository error when fetching transactions for resource %s: %s", this, e.getMessage()));
}
}

61
src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java
View File

@ -9,6 +9,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataMetadataPatch;
import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata;
import org.qortal.arbitrary.misc.Category;
import org.qortal.arbitrary.misc.Service;
import org.qortal.crypto.AES;
import org.qortal.crypto.Crypto;
import org.qortal.data.PaymentData;
import org.qortal.data.transaction.ArbitraryTransactionData;
@ -46,6 +47,7 @@ public class ArbitraryDataTransactionBuilder {
private static final double MAX_FILE_DIFF = 0.5f;
private final String publicKey58;
private final long fee;
private final Path path;
private final String name;
private Method method;
@ -64,11 +66,12 @@ public class ArbitraryDataTransactionBuilder {
private ArbitraryTransactionData arbitraryTransactionData;
private ArbitraryDataFile arbitraryDataFile;
public ArbitraryDataTransactionBuilder(Repository repository, String publicKey58, Path path, String name,
public ArbitraryDataTransactionBuilder(Repository repository, String publicKey58, long fee, Path path, String name,
Method method, Service service, String identifier,
String title, String description, List<String> tags, Category category) {
this.repository = repository;
this.publicKey58 = publicKey58;
this.fee = fee;
this.path = path;
this.name = name;
this.method = method;
@ -179,6 +182,7 @@ public class ArbitraryDataTransactionBuilder {
for (ModifiedPath path : metadata.getModifiedPaths()) {
if (path.getDiffType() != DiffType.COMPLETE_FILE) {
atLeastOnePatch = true;
break;
}
}
}
@ -187,6 +191,14 @@ public class ArbitraryDataTransactionBuilder {
return Method.PUT;
}
// We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain
final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false);
final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
if (shouldUseOnChainData) {
LOGGER.info("Data size is small enough to go on chain - using PUT");
return Method.PUT;
}
// State is appropriate for a PATCH transaction
return Method.PATCH;
}
@ -227,10 +239,12 @@ public class ArbitraryDataTransactionBuilder {
random.nextBytes(lastReference);
}
Compression compression = Compression.ZIP;
// Single file resources are handled differently, especially for very small data payloads, as these go on chain
final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false);
final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE);
// FUTURE? Use zip compression for directories, or no compression for single files
// Compression compression = (path.toFile().isDirectory()) ? Compression.ZIP : Compression.NONE;
// Use zip compression if data isn't going on chain
Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP;
ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(path, name, service, identifier, method,
compression, title, description, tags, category);
@ -248,45 +262,52 @@ public class ArbitraryDataTransactionBuilder {
throw new DataException("Arbitrary data file is null");
}
// Get chunks metadata file
// Get metadata file
ArbitraryDataFile metadataFile = arbitraryDataFile.getMetadataFile();
if (metadataFile == null && arbitraryDataFile.chunkCount() > 1) {
throw new DataException(String.format("Chunks metadata data file is null but there are %d chunks", arbitraryDataFile.chunkCount()));
}
String digest58 = arbitraryDataFile.digest58();
if (digest58 == null) {
LOGGER.error("Unable to calculate file digest");
throw new DataException("Unable to calculate file digest");
// Default to using a data hash, with data held off-chain
ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH;
byte[] data = arbitraryDataFile.digest();
// For small, single-chunk resources, we can store the data directly on chain
if (shouldUseOnChainData && arbitraryDataFile.getBytes().length <= ArbitraryTransaction.MAX_DATA_SIZE && arbitraryDataFile.chunkCount() == 0) {
// Within allowed on-chain data size
dataType = DataType.RAW_DATA;
data = arbitraryDataFile.getBytes();
}
final BaseTransactionData baseTransactionData = new BaseTransactionData(now, Group.NO_GROUP,
lastReference, creatorPublicKey, 0L, null);
lastReference, creatorPublicKey, fee, null);
final int size = (int) arbitraryDataFile.size();
final int version = 5;
final int nonce = 0;
byte[] secret = arbitraryDataFile.getSecret();
final ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH;
final byte[] digest = arbitraryDataFile.digest();
final byte[] metadataHash = (metadataFile != null) ? metadataFile.getHash() : null;
final List<PaymentData> payments = new ArrayList<>();
ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData,
version, service, nonce, size, name, identifier, method,
secret, compression, digest, dataType, metadataHash, payments);
version, service.value, nonce, size, name, identifier, method,
secret, compression, data, dataType, metadataHash, payments);
this.arbitraryTransactionData = transactionData;
} catch (DataException e) {
} catch (DataException | IOException e) {
if (arbitraryDataFile != null) {
arbitraryDataFile.deleteAll();
arbitraryDataFile.deleteAll(true);
}
throw(e);
throw new DataException(e);
}
}
private boolean isMetadataEqual(ArbitraryDataTransactionMetadata existingMetadata) {
if (existingMetadata == null) {
return !this.hasMetadata();
}
if (!Objects.equals(existingMetadata.getTitle(), this.title)) {
return false;
}
@ -302,6 +323,10 @@ public class ArbitraryDataTransactionBuilder {
return true;
}
private boolean hasMetadata() {
return (this.title != null || this.description != null || this.category != null || this.tags != null);
}
public void computeNonce() throws DataException {
if (this.arbitraryTransactionData == null) {
throw new DataException("Arbitrary transaction data is required to compute nonce");
@ -313,7 +338,7 @@ public class ArbitraryDataTransactionBuilder {
Transaction.ValidationResult result = transaction.isValidUnconfirmed();
if (result != Transaction.ValidationResult.OK) {
arbitraryDataFile.deleteAll();
arbitraryDataFile.deleteAll(true);
throw new DataException(String.format("Arbitrary transaction invalid: %s", result));
}
LOGGER.info("Transaction is valid");

74
src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java
View File

@ -1,5 +1,7 @@
package org.qortal.arbitrary;
import com.j256.simplemagic.ContentInfo;
import com.j256.simplemagic.ContentInfoUtil;
import org.apache.commons.io.FileUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@ -23,16 +25,15 @@ import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.net.FileNameMap;
import java.net.URLConnection;
import java.nio.file.*;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Objects;
import java.util.*;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class ArbitraryDataWriter {
@ -50,6 +51,8 @@ public class ArbitraryDataWriter {
private final String description;
private final List<String> tags;
private final Category category;
private List<String> files;
private String mimeType;
private int chunkSize = ArbitraryDataFile.CHUNK_SIZE;
@ -80,12 +83,15 @@ public class ArbitraryDataWriter {
this.description = ArbitraryDataTransactionMetadata.limitDescription(description);
this.tags = ArbitraryDataTransactionMetadata.limitTags(tags);
this.category = category;
this.files = new ArrayList<>(); // Populated in buildFileList()
this.mimeType = null; // Populated in buildFileList()
}
public void save() throws IOException, DataException, InterruptedException, MissingDataException {
try {
this.preExecute();
this.validateService();
this.buildFileList();
this.process();
this.compress();
this.encrypt();
@ -101,10 +107,9 @@ public class ArbitraryDataWriter {
private void preExecute() throws DataException {
this.checkEnabled();
// Enforce compression when uploading a directory
File file = new File(this.filePath.toString());
if (file.isDirectory() && compression == Compression.NONE) {
throw new DataException("Unable to upload a directory without compression");
// Enforce compression when uploading multiple files
if (!FilesystemUtils.isSingleFileResource(this.filePath, false) && compression == Compression.NONE) {
throw new DataException("Unable to publish multiple files without compression");
}
// Create temporary working directory
@ -143,6 +148,48 @@ public class ArbitraryDataWriter {
}
}
private void buildFileList() throws IOException {
// Check if the path already points to a single file
boolean isSingleFile = this.filePath.toFile().isFile();
Path singleFilePath = null;
if (isSingleFile) {
this.files.add(this.filePath.getFileName().toString());
singleFilePath = this.filePath;
}
else {
// Multi file resources (or a single file in a directory) require a walk through the directory tree
try (Stream<Path> stream = Files.walk(this.filePath)) {
this.files = stream
.filter(Files::isRegularFile)
.map(p -> this.filePath.relativize(p).toString())
.filter(s -> !s.isEmpty())
.collect(Collectors.toList());
if (this.files.size() == 1) {
singleFilePath = Paths.get(this.filePath.toString(), this.files.get(0));
// Update filePath to point to the single file (instead of the directory containing the file)
this.filePath = singleFilePath;
}
}
}
if (singleFilePath != null) {
// Single file resource, so try and determine the MIME type
ContentInfoUtil util = new ContentInfoUtil();
ContentInfo info = util.findMatch(singleFilePath.toFile());
if (info != null) {
// Attempt to extract MIME type from file contents
this.mimeType = info.getMimeType();
}
else {
// Fall back to using the filename
FileNameMap fileNameMap = URLConnection.getFileNameMap();
this.mimeType = fileNameMap.getContentTypeFor(singleFilePath.toFile().getName());
}
}
}
private void process() throws DataException, IOException, MissingDataException {
switch (this.method) {
@ -269,9 +316,6 @@ public class ArbitraryDataWriter {
if (chunkCount > 0) {
LOGGER.info(String.format("Successfully split into %d chunk%s", chunkCount, (chunkCount == 1 ? "" : "s")));
}
else {
throw new DataException("Unable to split file into chunks");
}
}
private void createMetadataFile() throws IOException, DataException {
@ -285,6 +329,8 @@ public class ArbitraryDataWriter {
metadata.setTags(this.tags);
metadata.setCategory(this.category);
metadata.setChunks(this.arbitraryDataFile.chunkHashList());
metadata.setFiles(this.files);
metadata.setMimeType(this.mimeType);
metadata.write();
// Create an ArbitraryDataFile from the JSON file (we don't have a signature yet)

22
src/main/java/org/qortal/arbitrary/exception/DataNotPublishedException.java Normal file
View File

@ -0,0 +1,22 @@
package org.qortal.arbitrary.exception;
import org.qortal.repository.DataException;
public class DataNotPublishedException extends DataException {
public DataNotPublishedException() {
}
public DataNotPublishedException(String message) {
super(message);
}
public DataNotPublishedException(String message, Throwable cause) {
super(message, cause);
}
public DataNotPublishedException(Throwable cause) {
super(cause);
}
}

19
src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java
View File

@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONException;
import org.qortal.repository.DataException;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
@ -34,7 +36,7 @@ public class ArbitraryDataMetadata {
this.filePath = filePath;
}
protected void readJson() throws DataException {
protected void readJson() throws DataException, JSONException {
// To be overridden
}
@ -44,8 +46,13 @@ public class ArbitraryDataMetadata {
public void read() throws IOException, DataException {
this.loadJson();
this.readJson();
try {
this.loadJson();
this.readJson();
} catch (JSONException e) {
throw new DataException(String.format("Unable to read JSON at path %s: %s", this.filePath, e.getMessage()));
}
}
public void write() throws IOException, DataException {
@ -58,6 +65,10 @@ public class ArbitraryDataMetadata {
writer.close();
}
public void delete() throws IOException {
Files.delete(this.filePath);
}
protected void loadJson() throws IOException {
File metadataFile = new File(this.filePath.toString());
@ -65,7 +76,7 @@ public class ArbitraryDataMetadata {
throw new IOException(String.format("Metadata file doesn't exist: %s", this.filePath.toString()));
}
this.jsonString = new String(Files.readAllBytes(this.filePath));
this.jsonString = new String(Files.readAllBytes(this.filePath), StandardCharsets.UTF_8);
}

3
src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java
View File

@ -1,5 +1,6 @@
package org.qortal.arbitrary.metadata;
import org.json.JSONException;
import org.json.JSONObject;
import org.qortal.repository.DataException;
import org.qortal.utils.Base58;
@ -22,7 +23,7 @@ public class ArbitraryDataMetadataCache extends ArbitraryDataQortalMetadata {
}
@Override
protected void readJson() throws DataException {
protected void readJson() throws DataException, JSONException {
if (this.jsonString == null) {
throw new DataException("Patch JSON string is null");
}

3
src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java
View File

@ -3,6 +3,7 @@ package org.qortal.arbitrary.metadata;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.qortal.arbitrary.ArbitraryDataDiff.*;
import org.qortal.repository.DataException;
@ -40,7 +41,7 @@ public class ArbitraryDataMetadataPatch extends ArbitraryDataQortalMetadata {
}
@Override
protected void readJson() throws DataException {
protected void readJson() throws DataException, JSONException {
if (this.jsonString == null) {
throw new DataException("Patch JSON string is null");
}

23
src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java
View File

@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONException;
import org.qortal.repository.DataException;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
@ -46,20 +48,6 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
return null;
}
protected void readJson() throws DataException {
// To be overridden
}
protected void buildJson() {
// To be overridden
}
@Override
public void read() throws IOException, DataException {
this.loadJson();
this.readJson();
}
@Override
public void write() throws IOException, DataException {
@ -82,7 +70,7 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
throw new IOException(String.format("Patch file doesn't exist: %s", path.toString()));
}
this.jsonString = new String(Files.readAllBytes(path));
this.jsonString = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
}
@ -94,9 +82,4 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata {
}
}
public String getJsonString() {
return this.jsonString;
}
}

77
src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java
View File

@ -1,11 +1,13 @@
package org.qortal.arbitrary.metadata;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import org.qortal.arbitrary.misc.Category;
import org.qortal.repository.DataException;
import org.qortal.utils.Base58;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
@ -19,9 +21,11 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
private String description;
private List<String> tags;
private Category category;
private List<String> files;
private String mimeType;
private static int MAX_TITLE_LENGTH = 80;
private static int MAX_DESCRIPTION_LENGTH = 500;
private static int MAX_DESCRIPTION_LENGTH = 240;
private static int MAX_TAG_LENGTH = 20;
private static int MAX_TAGS_COUNT = 5;
@ -31,7 +35,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
}
@Override
protected void readJson() throws DataException {
protected void readJson() throws DataException, JSONException {
if (this.jsonString == null) {
throw new DataException("Transaction metadata JSON string is null");
}
@ -77,6 +81,24 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
}
this.chunks = chunksList;
}
List<String> filesList = new ArrayList<>();
if (metadata.has("files")) {
JSONArray files = metadata.getJSONArray("files");
if (files != null) {
for (int i=0; i<files.length(); i++) {
String tag = files.getString(i);
if (tag != null) {
filesList.add(tag);
}
}
}
this.files = filesList;
}
if (metadata.has("mimeType")) {
this.mimeType = metadata.getString("mimeType");
}
}
@Override
@ -111,6 +133,18 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
}
outer.put("chunks", chunks);
JSONArray files = new JSONArray();
if (this.files != null) {
for (String file : this.files) {
files.put(file);
}
}
outer.put("files", files);
if (this.mimeType != null && !this.mimeType.isEmpty()) {
outer.put("mimeType", this.mimeType);
}
this.jsonString = outer.toString(2);
LOGGER.trace("Transaction metadata: {}", this.jsonString);
}
@ -156,6 +190,22 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
return this.category;
}
public void setFiles(List<String> files) {
this.files = files;
}
public List<String> getFiles() {
return this.files;
}
public void setMimeType(String mimeType) {
this.mimeType = mimeType;
}
public String getMimeType() {
return this.mimeType;
}
public boolean containsChunk(byte[] chunk) {
for (byte[] c : this.chunks) {
if (Arrays.equals(c, chunk)) {
@ -168,6 +218,25 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
// Static helper methods
public static String trimUTF8String(String string, int maxLength) {
byte[] inputBytes = string.getBytes(StandardCharsets.UTF_8);
int length = Math.min(inputBytes.length, maxLength);
byte[] outputBytes = new byte[length];
System.arraycopy(inputBytes, 0, outputBytes, 0, length);
String result = new String(outputBytes, StandardCharsets.UTF_8);
// check if last character is truncated
int lastIndex = result.length() - 1;
if (lastIndex > 0 && result.charAt(lastIndex) != string.charAt(lastIndex)) {
// last character is truncated so remove the last character
return result.substring(0, lastIndex);
}
return result;
}
public static String limitTitle(String title) {
if (title == null) {
return null;
@ -176,7 +245,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
return null;
}
return title.substring(0, Math.min(title.length(), MAX_TITLE_LENGTH));
return trimUTF8String(title, MAX_TITLE_LENGTH);
}
public static String limitDescription(String description) {
@ -187,7 +256,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata {
return null;
}
return description.substring(0, Math.min(description.length(), MAX_DESCRIPTION_LENGTH));
return trimUTF8String(description, MAX_DESCRIPTION_LENGTH);
}
public static List<String> limitTags(List<String> tags) {

245
src/main/java/org/qortal/arbitrary/misc/Service.java
View File

@ -1,26 +1,66 @@
package org.qortal.arbitrary.misc;
import org.apache.commons.io.FilenameUtils;
import org.json.JSONObject;
import org.qortal.arbitrary.ArbitraryDataRenderer;
import org.qortal.transaction.Transaction;
import org.qortal.utils.FilesystemUtils;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import static java.util.Arrays.stream;
import static java.util.stream.Collectors.toMap;
public enum Service {
AUTO_UPDATE(1, false, null, null),
ARBITRARY_DATA(100, false, null, null),
WEBSITE(200, true, null, null) {
AUTO_UPDATE(1, false, null, false, false, null),
ARBITRARY_DATA(100, false, null, false, false, null),
QCHAT_ATTACHMENT(120, true, 1024*1024L, true, false, null) {
@Override
public ValidationResult validate(Path path) {
public ValidationResult validate(Path path) throws IOException {
ValidationResult superclassResult = super.validate(path);
if (superclassResult != ValidationResult.OK) {
return superclassResult;
}
File[] files = path.toFile().listFiles();
// If already a single file, replace the list with one that contains that file only
if (files == null && path.toFile().isFile()) {
files = new File[] { path.toFile() };
}
// Now validate the file's extension
if (files != null && files[0] != null) {
final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase();
// We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string
final List<String> allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", "");
if (extension == null || !allowedExtensions.contains(extension)) {
return ValidationResult.INVALID_FILE_EXTENSION;
}
}
return ValidationResult.OK;
}
},
QCHAT_ATTACHMENT_PRIVATE(121, true, 1024*1024L, true, true, null),
ATTACHMENT(130, false, 50*1024*1024L, true, false, null),
ATTACHMENT_PRIVATE(131, true, 50*1024*1024L, true, true, null),
FILE(140, false, null, true, false, null),
FILE_PRIVATE(141, true, null, true, true, null),
FILES(150, false, null, false, false, null),
CHAIN_DATA(160, true, 239L, true, false, null),
WEBSITE(200, true, null, false, false, null) {
@Override
public ValidationResult validate(Path path) throws IOException {
ValidationResult superclassResult = super.validate(path);
if (superclassResult != ValidationResult.OK) {
return superclassResult;
}
// Custom validation function to require an index HTML file in the root directory
List<String> fileNames = ArbitraryDataRenderer.indexFiles();
String[] files = path.toFile().list();
@ -35,33 +75,125 @@ public enum Service {
return ValidationResult.MISSING_INDEX_FILE;
}
},
GIT_REPOSITORY(300, false, null, null),
IMAGE(400, true, 10*1024*1024L, null),
THUMBNAIL(410, true, 500*1024L, null),
VIDEO(500, false, null, null),
AUDIO(600, false, null, null),
BLOG(700, false, null, null),
BLOG_POST(777, false, null, null),
BLOG_COMMENT(778, false, null, null),
DOCUMENT(800, false, null, null),
LIST(900, true, null, null),
PLAYLIST(910, true, null, null),
APP(1000, false, null, null),
METADATA(1100, false, null, null),
QORTAL_METADATA(1111, true, 10*1024L, Arrays.asList("title", "description", "tags"));
GIT_REPOSITORY(300, false, null, false, false, null),
IMAGE(400, true, 10*1024*1024L, true, false, null),
IMAGE_PRIVATE(401, true, 10*1024*1024L, true, true, null),
THUMBNAIL(410, true, 500*1024L, true, false, null),
QCHAT_IMAGE(420, true, 500*1024L, true, false, null),
VIDEO(500, false, null, true, false, null),
VIDEO_PRIVATE(501, true, null, true, true, null),
AUDIO(600, false, null, true, false, null),
AUDIO_PRIVATE(601, true, null, true, true, null),
QCHAT_AUDIO(610, true, 10*1024*1024L, true, false, null),
QCHAT_VOICE(620, true, 10*1024*1024L, true, false, null),
VOICE(630, true, 10*1024*1024L, true, false, null),
VOICE_PRIVATE(631, true, 10*1024*1024L, true, true, null),
PODCAST(640, false, null, true, false, null),
BLOG(700, false, null, false, false, null),
BLOG_POST(777, false, null, true, false, null),
BLOG_COMMENT(778, true, 500*1024L, true, false, null),
DOCUMENT(800, false, null, true, false, null),
DOCUMENT_PRIVATE(801, true, null, true, true, null),
LIST(900, true, null, true, false, null),
PLAYLIST(910, true, null, true, false, null),
APP(1000, true, 50*1024*1024L, false, false, null),
METADATA(1100, false, null, true, false, null),
JSON(1110, true, 25*1024L, true, false, null) {
@Override
public ValidationResult validate(Path path) throws IOException {
ValidationResult superclassResult = super.validate(path);
if (superclassResult != ValidationResult.OK) {
return superclassResult;
}
// Require valid JSON
byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024);
String json = new String(data, StandardCharsets.UTF_8);
try {
objectMapper.readTree(json);
return ValidationResult.OK;
} catch (IOException e) {
return ValidationResult.INVALID_CONTENT;
}
}
},
GIF_REPOSITORY(1200, true, 25*1024*1024L, false, false, null) {
@Override
public ValidationResult validate(Path path) throws IOException {
ValidationResult superclassResult = super.validate(path);
if (superclassResult != ValidationResult.OK) {
return superclassResult;
}
// Custom validation function to require .gif files only, and at least 1
int gifCount = 0;
File[] files = path.toFile().listFiles();
// If already a single file, replace the list with one that contains that file only
if (files == null && path.toFile().isFile()) {
files = new File[] { path.toFile() };
}
if (files != null) {
for (File file : files) {
if (file.getName().equals(".qortal")) {
continue;
}
if (file.isDirectory()) {
return ValidationResult.DIRECTORIES_NOT_ALLOWED;
}
String extension = FilenameUtils.getExtension(file.getName()).toLowerCase();
if (!Objects.equals(extension, "gif")) {
return ValidationResult.INVALID_FILE_EXTENSION;
}
gifCount++;
}
}
if (gifCount == 0) {
return ValidationResult.MISSING_DATA;
}
return ValidationResult.OK;
}
},
STORE(1300, false, null, true, false, null),
PRODUCT(1310, false, null, true, false, null),
OFFER(1330, false, null, true, false, null),
COUPON(1340, false, null, true, false, null),
CODE(1400, false, null, true, false, null),
PLUGIN(1410, false, null, true, false, null),
EXTENSION(1420, false, null, true, false, null),
GAME(1500, false, null, false, false, null),
ITEM(1510, false, null, true, false, null),
NFT(1600, false, null, true, false, null),
DATABASE(1700, false, null, false, false, null),
SNAPSHOT(1710, false, null, false, false, null),
COMMENT(1800, true, 500*1024L, true, false, null),
CHAIN_COMMENT(1810, true, 239L, true, false, null),
MAIL(1900, true, 1024*1024L, true, false, null),
MAIL_PRIVATE(1901, true, 1024*1024L, true, true, null),
MESSAGE(1910, true, 1024*1024L, true, false, null),
MESSAGE_PRIVATE(1911, true, 1024*1024L, true, true, null);
public final int value;
private final boolean requiresValidation;
private final Long maxSize;
private final boolean single;
private final boolean isPrivate;
private final List<String> requiredKeys;
private static final Map<Integer, Service> map = stream(Service.values())
.collect(toMap(service -> service.value, service -> service));
Service(int value, boolean requiresValidation, Long maxSize, List<String> requiredKeys) {
// For JSON validation
private static final ObjectMapper objectMapper = new ObjectMapper();
private static final String encryptedDataPrefix = "qortalEncryptedData";
private static final String encryptedGroupDataPrefix = "qortalGroupEncryptedData";
Service(int value, boolean requiresValidation, Long maxSize, boolean single, boolean isPrivate, List<String> requiredKeys) {
this.value = value;
this.requiresValidation = requiresValidation;
this.maxSize = maxSize;
this.single = single;
this.isPrivate = isPrivate;
this.requiredKeys = requiredKeys;
}
@ -70,7 +202,9 @@ public enum Service {
return ValidationResult.OK;
}
byte[] data = FilesystemUtils.getSingleFileContents(path);
// Load the first 25KB of data. This only needs to be long enough to check the prefix
// and also to allow for possible additional future validation of smaller files.
byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024);
long size = FilesystemUtils.getDirectorySize(path);
// Validate max size if needed
@ -80,6 +214,22 @@ public enum Service {
}
}
// Validate file count if needed
if (this.single && data == null) {
return ValidationResult.INVALID_FILE_COUNT;
}
// Validate private data for single file resources
if (this.single) {
String dataString = new String(data, StandardCharsets.UTF_8);
if (this.isPrivate && !dataString.startsWith(encryptedDataPrefix) && !dataString.startsWith(encryptedGroupDataPrefix)) {
return ValidationResult.DATA_NOT_ENCRYPTED;
}
if (!this.isPrivate && (dataString.startsWith(encryptedDataPrefix) || dataString.startsWith(encryptedGroupDataPrefix))) {
return ValidationResult.DATA_ENCRYPTED;
}
}
// Validate required keys if needed
if (this.requiredKeys != null) {
if (data == null) {
@ -98,7 +248,12 @@ public enum Service {
}
public boolean isValidationRequired() {
return this.requiresValidation;
// We must always validate single file resources, to ensure they are actually a single file
return this.requiresValidation || this.single;
}
public boolean isPrivate() {
return this.isPrivate;
}
public static Service valueOf(int value) {
@ -106,15 +261,53 @@ public enum Service {
}
public static JSONObject toJsonObject(byte[] data) {
String dataString = new String(data);
String dataString = new String(data, StandardCharsets.UTF_8);
return new JSONObject(dataString);
}
public static List<Service> publicServices() {
List<Service> privateServices = new ArrayList<>();
for (Service service : Service.values()) {
if (!service.isPrivate) {
privateServices.add(service);
}
}
return privateServices;
}
/**
* Fetch a list of Service objects that require encrypted data.
*
* These can ultimately be used to help inform the cleanup manager
* on the best order to delete files when the node runs out of space.
* Public data should be given priority over private data (unless
* this node is part of a data market contract for that data - this
* isn't developed yet).
*
* @return a list of Service objects that require encrypted data.
*/
public static List<Service> privateServices() {
List<Service> privateServices = new ArrayList<>();
for (Service service : Service.values()) {
if (service.isPrivate) {
privateServices.add(service);
}
}
return privateServices;
}
public enum ValidationResult {
OK(1),
MISSING_KEYS(2),
EXCEEDS_SIZE_LIMIT(3),
MISSING_INDEX_FILE(4);
MISSING_INDEX_FILE(4),
DIRECTORIES_NOT_ALLOWED(5),
INVALID_FILE_EXTENSION(6),
MISSING_DATA(7),
INVALID_FILE_COUNT(8),
INVALID_CONTENT(9),
DATA_NOT_ENCRYPTED(10),
DATA_ENCRYPTED(10);
public final int value;

205
src/main/java/org/qortal/block/Block.java
View File

@ -84,6 +84,7 @@ public class Block {
TRANSACTION_PROCESSING_FAILED(53),
TRANSACTION_ALREADY_PROCESSED(54),
TRANSACTION_NEEDS_APPROVAL(55),
TRANSACTION_NOT_CONFIRMABLE(56),
AT_STATES_MISMATCH(61),
ONLINE_ACCOUNTS_INVALID(70),
ONLINE_ACCOUNT_UNKNOWN(71),
@ -130,13 +131,16 @@ public class Block {
/** Locally-generated AT fees */
protected long ourAtFees; // Generated locally
/** Cached online accounts validation decision, to avoid revalidating when true */
private boolean onlineAccountsAlreadyValid = false;
@FunctionalInterface
private interface BlockRewardDistributor {
long distribute(long amount, Map<String, Long> balanceChanges) throws DataException;
}
/** Lazy-instantiated expanded info on block's online accounts. */
private static class ExpandedAccount {
public static class ExpandedAccount {
private final RewardShareData rewardShareData;
private final int sharePercent;
private final boolean isRecipientAlsoMinter;
@ -169,6 +173,13 @@ public class Block {
}
}
public Account getMintingAccount() {
return this.mintingAccount;
}
public Account getRecipientAccount() {
return this.recipientAccount;
}
/**
* Returns share bin for expanded account.
* <p>
@ -363,15 +374,24 @@ public class Block {
return null;
}
int height = parentBlockData.getHeight() + 1;
long timestamp = calcTimestamp(parentBlockData, minter.getPublicKey(), minterLevel);
long onlineAccountsTimestamp = OnlineAccountsManager.getCurrentOnlineAccountTimestamp();
// Fetch our list of online accounts
// Fetch our list of online accounts, removing any that are missing a nonce
List<OnlineAccountData> onlineAccounts = OnlineAccountsManager.getInstance().getOnlineAccounts(onlineAccountsTimestamp);
onlineAccounts.removeIf(a -> a.getNonce() == null || a.getNonce() < 0);
// If mempow is active, remove any legacy accounts that are missing a nonce
if (timestamp >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp()) {
onlineAccounts.removeIf(a -> a.getNonce() == null || a.getNonce() < 0);
// After feature trigger, remove any online accounts that are level 0
if (height >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
onlineAccounts.removeIf(a -> {
try {
return Account.getRewardShareEffectiveMintingLevel(repository, a.getPublicKey()) == 0;
} catch (DataException e) {
// Something went wrong, so remove the account
return true;
}
});
}
if (onlineAccounts.isEmpty()) {
@ -412,29 +432,27 @@ public class Block {
// Aggregated, single signature
byte[] onlineAccountsSignatures = Qortal25519Extras.aggregateSignatures(signaturesToAggregate);
// Add nonces to the end of the online accounts signatures if mempow is active
if (timestamp >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp()) {
try {
// Create ordered list of nonce values
List<Integer> nonces = new ArrayList<>();
for (int i = 0; i < onlineAccountsCount; ++i) {
Integer accountIndex = accountIndexes.get(i);
OnlineAccountData onlineAccountData = indexedOnlineAccounts.get(accountIndex);
nonces.add(onlineAccountData.getNonce());
}
// Encode the nonces to a byte array
byte[] encodedNonces = BlockTransformer.encodeOnlineAccountNonces(nonces);
// Append the encoded nonces to the encoded online account signatures
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
outputStream.write(onlineAccountsSignatures);
outputStream.write(encodedNonces);
onlineAccountsSignatures = outputStream.toByteArray();
}
catch (TransformationException | IOException e) {
return null;
// Add nonces to the end of the online accounts signatures
try {
// Create ordered list of nonce values
List<Integer> nonces = new ArrayList<>();
for (int i = 0; i < onlineAccountsCount; ++i) {
Integer accountIndex = accountIndexes.get(i);
OnlineAccountData onlineAccountData = indexedOnlineAccounts.get(accountIndex);
nonces.add(onlineAccountData.getNonce());
}
// Encode the nonces to a byte array
byte[] encodedNonces = BlockTransformer.encodeOnlineAccountNonces(nonces);
// Append the encoded nonces to the encoded online account signatures
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
outputStream.write(onlineAccountsSignatures);
outputStream.write(encodedNonces);
onlineAccountsSignatures = outputStream.toByteArray();
}
catch (TransformationException | IOException e) {
return null;
}
byte[] minterSignature = minter.sign(BlockTransformer.getBytesForMinterSignature(parentBlockData,
@ -442,7 +460,6 @@ public class Block {
int transactionCount = 0;
byte[] transactionsSignature = null;
int height = parentBlockData.getHeight() + 1;
int atCount = 0;
long atFees = 0;
@ -550,6 +567,13 @@ public class Block {
}
/**
* Force online accounts to be revalidated, e.g. at final stage of block minting.
*/
public void clearOnlineAccountsValidationCache() {
this.onlineAccountsAlreadyValid = false;
}
// More information
/**
@ -644,6 +668,10 @@ public class Block {
return this.atStates;
}
public byte[] getAtStatesHash() {
return this.atStatesHash;
}
/**
* Return expanded info on block's online accounts.
* <p>
@ -1026,6 +1054,10 @@ public class Block {
if (this.blockData.getHeight() != null && this.blockData.getHeight() == 1)
return ValidationResult.OK;
// Don't bother revalidating if accounts have already been validated in this block
if (this.onlineAccountsAlreadyValid)
return ValidationResult.OK;
// Expand block's online accounts indexes into actual accounts
ConciseSet accountIndexes = BlockTransformer.decodeOnlineAccounts(this.blockData.getEncodedOnlineAccounts());
// We use count of online accounts to validate decoded account indexes
@ -1036,6 +1068,15 @@ public class Block {
if (onlineRewardShares == null)
return ValidationResult.ONLINE_ACCOUNT_UNKNOWN;
// After feature trigger, require all online account minters to be greater than level 0
if (this.getBlockData().getHeight() >= BlockChain.getInstance().getOnlineAccountMinterLevelValidationHeight()) {
List<ExpandedAccount> expandedAccounts = this.getExpandedAccounts();
for (ExpandedAccount account : expandedAccounts) {
if (account.getMintingAccount().getEffectiveMintingLevel() == 0)
return ValidationResult.ONLINE_ACCOUNTS_INVALID;
}
}
// If block is past a certain age then we simply assume the signatures were correct
long signatureRequirementThreshold = NTP.getTime() - BlockChain.getInstance().getOnlineAccountSignaturesMinLifetime();
if (this.blockData.getTimestamp() < signatureRequirementThreshold)
@ -1047,14 +1088,9 @@ public class Block {
final int signaturesLength = Transformer.SIGNATURE_LENGTH;
final int noncesLength = onlineRewardShares.size() * Transformer.INT_LENGTH;
if (this.blockData.getTimestamp() >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp()) {
// We expect nonces to be appended to the online accounts signatures
if (this.blockData.getOnlineAccountsSignatures().length != signaturesLength + noncesLength)
return ValidationResult.ONLINE_ACCOUNT_SIGNATURES_MALFORMED;
} else {
if (this.blockData.getOnlineAccountsSignatures().length != signaturesLength)
return ValidationResult.ONLINE_ACCOUNT_SIGNATURES_MALFORMED;
}
// We expect nonces to be appended to the online accounts signatures
if (this.blockData.getOnlineAccountsSignatures().length != signaturesLength + noncesLength)
return ValidationResult.ONLINE_ACCOUNT_SIGNATURES_MALFORMED;
// Check signatures
long onlineTimestamp = this.blockData.getOnlineAccountsTimestamp();
@ -1063,32 +1099,33 @@ public class Block {
byte[] encodedOnlineAccountSignatures = this.blockData.getOnlineAccountsSignatures();
// Split online account signatures into signature(s) + nonces, then validate the nonces
if (this.blockData.getTimestamp() >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp()) {
byte[] extractedSignatures = BlockTransformer.extract(encodedOnlineAccountSignatures, 0, signaturesLength);
byte[] extractedNonces = BlockTransformer.extract(encodedOnlineAccountSignatures, signaturesLength, onlineRewardShares.size() * Transformer.INT_LENGTH);
encodedOnlineAccountSignatures = extractedSignatures;
byte[] extractedSignatures = BlockTransformer.extract(encodedOnlineAccountSignatures, 0, signaturesLength);
byte[] extractedNonces = BlockTransformer.extract(encodedOnlineAccountSignatures, signaturesLength, onlineRewardShares.size() * Transformer.INT_LENGTH);
encodedOnlineAccountSignatures = extractedSignatures;
List<Integer> nonces = BlockTransformer.decodeOnlineAccountNonces(extractedNonces);
List<Integer> nonces = BlockTransformer.decodeOnlineAccountNonces(extractedNonces);
// Build block's view of online accounts (without signatures, as we don't need them here)
Set<OnlineAccountData> onlineAccounts = new HashSet<>();
for (int i = 0; i < onlineRewardShares.size(); ++i) {
Integer nonce = nonces.get(i);
byte[] publicKey = onlineRewardShares.get(i).getRewardSharePublicKey();
// Build block's view of online accounts (without signatures, as we don't need them here)
Set<OnlineAccountData> onlineAccounts = new HashSet<>();
for (int i = 0; i < onlineRewardShares.size(); ++i) {
Integer nonce = nonces.get(i);
byte[] publicKey = onlineRewardShares.get(i).getRewardSharePublicKey();
OnlineAccountData onlineAccountData = new OnlineAccountData(onlineTimestamp, null, publicKey, nonce);
onlineAccounts.add(onlineAccountData);
}
// Remove those already validated & cached by online accounts manager - no need to re-validate them
OnlineAccountsManager.getInstance().removeKnown(onlineAccounts, onlineTimestamp);
// Validate the rest
for (OnlineAccountData onlineAccount : onlineAccounts)
if (!OnlineAccountsManager.getInstance().verifyMemoryPoW(onlineAccount, this.blockData.getTimestamp()))
return ValidationResult.ONLINE_ACCOUNT_NONCE_INCORRECT;
OnlineAccountData onlineAccountData = new OnlineAccountData(onlineTimestamp, null, publicKey, nonce);
onlineAccounts.add(onlineAccountData);
}
// Remove those already validated & cached by online accounts manager - no need to re-validate them
OnlineAccountsManager.getInstance().removeKnown(onlineAccounts, onlineTimestamp);
// Validate the rest
for (OnlineAccountData onlineAccount : onlineAccounts)
if (!OnlineAccountsManager.getInstance().verifyMemoryPoW(onlineAccount, null))
return ValidationResult.ONLINE_ACCOUNT_NONCE_INCORRECT;
// Cache the valid online accounts as they will likely be needed for the next block
OnlineAccountsManager.getInstance().addBlocksOnlineAccounts(onlineAccounts, onlineTimestamp);
// Extract online accounts' timestamp signatures from block data. Only one signature if aggregated.
List<byte[]> onlineAccountsSignatures = BlockTransformer.decodeTimestampSignatures(encodedOnlineAccountSignatures);
@ -1108,6 +1145,9 @@ public class Block {
// All online accounts valid, so save our list of online accounts for potential later use
this.cachedOnlineRewardShares = onlineRewardShares;
// Remember that the accounts are valid, to speed up subsequent checks
this.onlineAccountsAlreadyValid = true;
return ValidationResult.OK;
}
@ -1212,6 +1252,13 @@ public class Block {
|| transaction.getDeadline() <= this.blockData.getTimestamp())
return ValidationResult.TRANSACTION_TIMESTAMP_INVALID;
// After feature trigger, check that this transaction is confirmable
if (transactionData.getTimestamp() >= BlockChain.getInstance().getMemPoWTransactionUpdatesTimestamp()) {
if (!transaction.isConfirmable()) {
return ValidationResult.TRANSACTION_NOT_CONFIRMABLE;
}
}
// Check transaction isn't already included in a block
if (this.repository.getTransactionRepository().isConfirmed(transactionData.getSignature()))
return ValidationResult.TRANSACTION_ALREADY_PROCESSED;
@ -1445,6 +1492,9 @@ public class Block {
if (this.blockData.getHeight() == 212937)
// Apply fix for block 212937
Block212937.processFix(this);
else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height())
SelfSponsorshipAlgoV1Block.processAccountPenalties(this);
}
// We're about to (test-)process a batch of transactions,
@ -1501,19 +1551,23 @@ public class Block {
// Batch update in repository
repository.getAccountRepository().modifyMintedBlockCounts(allUniqueExpandedAccounts.stream().map(AccountData::getAddress).collect(Collectors.toList()), +1);
// Keep track of level bumps in case we need to apply to other entries
Map<String, Integer> bumpedAccounts = new HashMap<>();
// Local changes and also checks for level bump
for (AccountData accountData : allUniqueExpandedAccounts) {
// Adjust count locally (in Java)
accountData.setBlocksMinted(accountData.getBlocksMinted() + 1);
LOGGER.trace(() -> String.format("Block minter %s up to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment();
final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
if (newLevel > accountData.getLevel()) {
// Account has increased in level!
accountData.setLevel(newLevel);
bumpedAccounts.put(accountData.getAddress(), newLevel);
repository.getAccountRepository().setLevel(accountData);
LOGGER.trace(() -> String.format("Block minter %s bumped to level %d", accountData.getAddress(), accountData.getLevel()));
}
@ -1521,6 +1575,25 @@ public class Block {
break;
}
}
// Also bump other entries if need be
if (!bumpedAccounts.isEmpty()) {
for (ExpandedAccount expandedAccount : expandedAccounts) {
Integer newLevel = bumpedAccounts.get(expandedAccount.mintingAccountData.getAddress());
if (newLevel != null && expandedAccount.mintingAccountData.getLevel() != newLevel) {
expandedAccount.mintingAccountData.setLevel(newLevel);
LOGGER.trace("Also bumped {} to level {}", expandedAccount.mintingAccountData.getAddress(), newLevel);
}
if (!expandedAccount.isRecipientAlsoMinter) {
newLevel = bumpedAccounts.get(expandedAccount.recipientAccountData.getAddress());
if (newLevel != null && expandedAccount.recipientAccountData.getLevel() != newLevel) {
expandedAccount.recipientAccountData.setLevel(newLevel);
LOGGER.trace("Also bumped {} to level {}", expandedAccount.recipientAccountData.getAddress(), newLevel);
}
}
}
}
}
protected void processBlockRewards() throws DataException {
@ -1638,12 +1711,14 @@ public class Block {
transactionData.getSignature());
this.repository.getBlockRepository().save(blockTransactionData);
// Update transaction's height in repository
// Update transaction's height in repository and local transactionData
transactionRepository.updateBlockHeight(transactionData.getSignature(), this.blockData.getHeight());
// Update local transactionData's height too
transaction.getTransactionData().setBlockHeight(this.blockData.getHeight());
// Update transaction's sequence in repository and local transactionData
transactionRepository.updateBlockSequence(transactionData.getSignature(), sequence);
transaction.getTransactionData().setBlockSequence(sequence);
// No longer unconfirmed
transactionRepository.confirmTransaction(transactionData.getSignature());
@ -1680,6 +1755,9 @@ public class Block {
// Revert fix for block 212937
Block212937.orphanFix(this);
else if (this.blockData.getHeight() == BlockChain.getInstance().getSelfSponsorshipAlgoV1Height())
SelfSponsorshipAlgoV1Block.orphanAccountPenalties(this);
// Block rewards, including transaction fees, removed after transactions undone
orphanBlockRewards();
@ -1727,6 +1805,9 @@ public class Block {
// Unset height
transactionRepository.updateBlockHeight(transactionData.getSignature(), null);
// Unset sequence
transactionRepository.updateBlockSequence(transactionData.getSignature(), null);
}
transactionRepository.deleteParticipants(transactionData);
@ -1808,7 +1889,7 @@ public class Block {
accountData.setBlocksMinted(accountData.getBlocksMinted() - 1);
LOGGER.trace(() -> String.format("Block minter %s down to %d minted block%s", accountData.getAddress(), accountData.getBlocksMinted(), (accountData.getBlocksMinted() != 1 ? "s" : "")));
final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment();
final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
for (int newLevel = maximumLevel; newLevel >= 0; --newLevel)
if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {

138
src/main/java/org/qortal/block/BlockChain.java
View File

@ -48,9 +48,6 @@ public class BlockChain {
/** Transaction expiry period, starting from transaction's timestamp, in milliseconds. */
private long transactionExpiryPeriod;
@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
private long unitFee;
private int maxBytesPerUnitFee;
/** Maximum acceptable timestamp disagreement offset in milliseconds. */
@ -73,7 +70,13 @@ public class BlockChain {
calcChainWeightTimestamp,
transactionV5Timestamp,
transactionV6Timestamp,
disableReferenceTimestamp;
disableReferenceTimestamp,
increaseOnlineAccountsDifficultyTimestamp,
onlineAccountMinterLevelValidationHeight,
selfSponsorshipAlgoV1Height,
feeValidationFixTimestamp,
chatReferenceTimestamp,
arbitraryOptionalFeeTimestamp;
}
// Custom transaction fees
@ -83,6 +86,7 @@ public class BlockChain {
@XmlJavaTypeAdapter(value = org.qortal.api.AmountTypeAdapter.class)
public long fee;
}
private List<UnitFeesByTimestamp> unitFees;
private List<UnitFeesByTimestamp> nameRegistrationUnitFees;
/** Map of which blockchain features are enabled when (height/timestamp) */
@ -95,6 +99,13 @@ public class BlockChain {
/** Whether only one registered name is allowed per account. */
private boolean oneNamePerAccount = false;
/** Checkpoints */
public static class Checkpoint {
public int height;
public String signature;
}
private List<Checkpoint> checkpoints;
/** Block rewards by block height */
public static class RewardByHeight {
public int height;
@ -195,9 +206,11 @@ public class BlockChain {
* featureTriggers because unit tests need to set this value via Reflection. */
private long onlineAccountsModulusV2Timestamp;
/** Feature trigger timestamp for online accounts mempow verification. Can't use featureTriggers
* because unit tests need to set this value via Reflection. */
private long onlineAccountsMemoryPoWTimestamp;
/** Snapshot timestamp for self sponsorship algo V1 */
private long selfSponsorshipAlgoV1SnapshotTimestamp;
/** Feature-trigger timestamp to modify behaviour of various transactions that support mempow */
private long mempowTransactionUpdatesTimestamp;
/** Max reward shares by block height */
public static class MaxRewardSharesByTimestamp {
@ -334,10 +347,6 @@ public class BlockChain {
return this.isTestChain;
}
public long getUnitFee() {
return this.unitFee;
}
public int getMaxBytesPerUnitFee() {
return this.maxBytesPerUnitFee;
}
@ -359,8 +368,14 @@ public class BlockChain {
return this.onlineAccountsModulusV2Timestamp;
}
public long getOnlineAccountsMemoryPoWTimestamp() {
return this.onlineAccountsMemoryPoWTimestamp;
// Self sponsorship algo
public long getSelfSponsorshipAlgoV1SnapshotTimestamp() {
return this.selfSponsorshipAlgoV1SnapshotTimestamp;
}
// Feature-trigger timestamp to modify behaviour of various transactions that support mempow
public long getMemPoWTransactionUpdatesTimestamp() {
return this.mempowTransactionUpdatesTimestamp;
}
/** Returns true if approval-needing transaction types require a txGroupId other than NO_GROUP. */
@ -376,6 +391,10 @@ public class BlockChain {
return this.oneNamePerAccount;
}
public List<Checkpoint> getCheckpoints() {
return this.checkpoints;
}
public List<RewardByHeight> getBlockRewardsByHeight() {
return this.rewardsByHeight;
}
@ -486,6 +505,30 @@ public class BlockChain {
return this.featureTriggers.get(FeatureTrigger.disableReferenceTimestamp.name()).longValue();
}
public long getIncreaseOnlineAccountsDifficultyTimestamp() {
return this.featureTriggers.get(FeatureTrigger.increaseOnlineAccountsDifficultyTimestamp.name()).longValue();
}
public int getSelfSponsorshipAlgoV1Height() {
return this.featureTriggers.get(FeatureTrigger.selfSponsorshipAlgoV1Height.name()).intValue();
}
public long getOnlineAccountMinterLevelValidationHeight() {
return this.featureTriggers.get(FeatureTrigger.onlineAccountMinterLevelValidationHeight.name()).intValue();
}
public long getFeeValidationFixTimestamp() {
return this.featureTriggers.get(FeatureTrigger.feeValidationFixTimestamp.name()).longValue();
}
public long getChatReferenceTimestamp() {
return this.featureTriggers.get(FeatureTrigger.chatReferenceTimestamp.name()).longValue();
}
public long getArbitraryOptionalFeeTimestamp() {
return this.featureTriggers.get(FeatureTrigger.arbitraryOptionalFeeTimestamp.name()).longValue();
}
// More complex getters for aspects that change by height or timestamp
@ -506,13 +549,22 @@ public class BlockChain {
throw new IllegalStateException(String.format("No block timing info available for height %d", ourHeight));
}
public long getUnitFeeAtTimestamp(long ourTimestamp) {
for (int i = unitFees.size() - 1; i >= 0; --i)
if (unitFees.get(i).timestamp <= ourTimestamp)
return unitFees.get(i).fee;
// Shouldn't happen, but set a sensible default just in case
return 100000;
}
public long getNameRegistrationUnitFeeAtTimestamp(long ourTimestamp) {
for (int i = nameRegistrationUnitFees.size() - 1; i >= 0; --i)
if (nameRegistrationUnitFees.get(i).timestamp <= ourTimestamp)
return nameRegistrationUnitFees.get(i).fee;
// Default to system-wide unit fee
return this.getUnitFee();
// Shouldn't happen, but set a sensible default just in case
return 100000;
}
public int getMaxRewardSharesAtTimestamp(long ourTimestamp) {
@ -654,6 +706,7 @@ public class BlockChain {
boolean isTopOnly = Settings.getInstance().isTopOnly();
boolean archiveEnabled = Settings.getInstance().isArchiveEnabled();
boolean isLite = Settings.getInstance().isLite();
boolean canBootstrap = Settings.getInstance().getBootstrap();
boolean needsArchiveRebuild = false;
BlockData chainTip;
@ -674,22 +727,44 @@ public class BlockChain {
}
}
}
// Validate checkpoints
// Limited to topOnly nodes for now, in order to reduce risk, and to solve a real-world problem with divergent topOnly nodes
// TODO: remove the isTopOnly conditional below once this feature has had more testing time
if (isTopOnly && !isLite) {
List<Checkpoint> checkpoints = BlockChain.getInstance().getCheckpoints();
for (Checkpoint checkpoint : checkpoints) {
BlockData blockData = repository.getBlockRepository().fromHeight(checkpoint.height);
if (blockData == null) {
// Try the archive
blockData = repository.getBlockArchiveRepository().fromHeight(checkpoint.height);
}
if (blockData == null) {
LOGGER.trace("Couldn't find block for height {}", checkpoint.height);
// This is likely due to the block being pruned, so is safe to ignore.
// Continue, as there might be other blocks we can check more definitively.
continue;
}
byte[] signature = Base58.decode(checkpoint.signature);
if (!Arrays.equals(signature, blockData.getSignature())) {
LOGGER.info("Error: block at height {} with signature {} doesn't match checkpoint sig: {}. Bootstrapping...", checkpoint.height, Base58.encode(blockData.getSignature()), checkpoint.signature);
needsArchiveRebuild = true;
break;
}
LOGGER.info("Block at height {} matches checkpoint signature", blockData.getHeight());
}
}
}
boolean hasBlocks = (chainTip != null && chainTip.getHeight() > 1);
// Check first block is Genesis Block
if (!isGenesisBlockValid() || needsArchiveRebuild) {
try {
rebuildBlockchain();
if (isTopOnly && hasBlocks) {
// Top-only mode is enabled and we have blocks, so it's possible that the genesis block has been pruned
// It's best not to validate it, and there's no real need to
} else {
// Check first block is Genesis Block
if (!isGenesisBlockValid() || needsArchiveRebuild) {
try {
rebuildBlockchain();
} catch (InterruptedException e) {
throw new DataException(String.format("Interrupted when trying to rebuild blockchain: %s", e.getMessage()));
}
} catch (InterruptedException e) {
throw new DataException(String.format("Interrupted when trying to rebuild blockchain: %s", e.getMessage()));
}
}
@ -698,9 +773,7 @@ public class BlockChain {
try (final Repository repository = RepositoryManager.getRepository()) {
repository.checkConsistency();
// Set the number of blocks to validate based on the pruned state of the chain
// If pruned, subtract an extra 10 to allow room for error
int blocksToValidate = (isTopOnly || archiveEnabled) ? Settings.getInstance().getPruneBlockLimit() - 10 : 1440;
int blocksToValidate = Math.min(Settings.getInstance().getPruneBlockLimit() - 10, 1440);
int startHeight = Math.max(repository.getBlockRepository().getBlockchainHeight() - blocksToValidate, 1);
BlockData detachedBlockData = repository.getBlockRepository().getDetachedBlockSignature(startHeight);
@ -809,6 +882,9 @@ public class BlockChain {
BlockData orphanBlockData = repository.getBlockRepository().fromHeight(height);
while (height > targetHeight) {
if (Controller.isStopping()) {
return false;
}
LOGGER.info(String.format("Forcably orphaning block %d", height));
Block block = new Block(repository, orphanBlockData);

133
src/main/java/org/qortal/block/SelfSponsorshipAlgoV1Block.java Normal file
View File

@ -0,0 +1,133 @@
package org.qortal.block;
import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.account.SelfSponsorshipAlgoV1;
import org.qortal.api.model.AccountPenaltyStats;
import org.qortal.crypto.Crypto;
import org.qortal.data.account.AccountData;
import org.qortal.data.account.AccountPenaltyData;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.utils.Base58;
import java.nio.charset.StandardCharsets;
import java.util.*;
import java.util.stream.Collectors;
/**
* Self Sponsorship AlgoV1 Block
* <p>
* Selected block for the initial run on the "self sponsorship detection algorithm"
*/
public final class SelfSponsorshipAlgoV1Block {
private static final Logger LOGGER = LogManager.getLogger(SelfSponsorshipAlgoV1Block.class);
private SelfSponsorshipAlgoV1Block() {
/* Do not instantiate */
}
public static void processAccountPenalties(Block block) throws DataException {
LOGGER.info("Running algo for block processing - this will take a while...");
logPenaltyStats(block.repository);
long startTime = System.currentTimeMillis();
Set<AccountPenaltyData> penalties = getAccountPenalties(block.repository, -5000000);
block.repository.getAccountRepository().updateBlocksMintedPenalties(penalties);
long totalTime = System.currentTimeMillis() - startTime;
String hash = getHash(penalties.stream().map(p -> p.getAddress()).collect(Collectors.toList()));
LOGGER.info("{} penalty addresses processed (hash: {}). Total time taken: {} seconds", penalties.size(), hash, (int)(totalTime / 1000.0f));
logPenaltyStats(block.repository);
int updatedCount = updateAccountLevels(block.repository, penalties);
LOGGER.info("Account levels updated for {} penalty addresses", updatedCount);
}
public static void orphanAccountPenalties(Block block) throws DataException {
LOGGER.info("Running algo for block orphaning - this will take a while...");
logPenaltyStats(block.repository);
long startTime = System.currentTimeMillis();
Set<AccountPenaltyData> penalties = getAccountPenalties(block.repository, 5000000);
block.repository.getAccountRepository().updateBlocksMintedPenalties(penalties);
long totalTime = System.currentTimeMillis() - startTime;
String hash = getHash(penalties.stream().map(p -> p.getAddress()).collect(Collectors.toList()));
LOGGER.info("{} penalty addresses orphaned (hash: {}). Total time taken: {} seconds", penalties.size(), hash, (int)(totalTime / 1000.0f));
logPenaltyStats(block.repository);
int updatedCount = updateAccountLevels(block.repository, penalties);
LOGGER.info("Account levels updated for {} penalty addresses", updatedCount);
}
public static Set<AccountPenaltyData> getAccountPenalties(Repository repository, int penalty) throws DataException {
final long snapshotTimestamp = BlockChain.getInstance().getSelfSponsorshipAlgoV1SnapshotTimestamp();
Set<AccountPenaltyData> penalties = new LinkedHashSet<>();
List<String> addresses = repository.getTransactionRepository().getConfirmedRewardShareCreatorsExcludingSelfShares();
for (String address : addresses) {
//System.out.println(String.format("address: %s", address));
SelfSponsorshipAlgoV1 selfSponsorshipAlgoV1 = new SelfSponsorshipAlgoV1(repository, address, snapshotTimestamp, false);
selfSponsorshipAlgoV1.run();
//System.out.println(String.format("Penalty addresses: %d", selfSponsorshipAlgoV1.getPenaltyAddresses().size()));
for (String penaltyAddress : selfSponsorshipAlgoV1.getPenaltyAddresses()) {
penalties.add(new AccountPenaltyData(penaltyAddress, penalty));
}
}
return penalties;
}
private static int updateAccountLevels(Repository repository, Set<AccountPenaltyData> accountPenalties) throws DataException {
final List<Integer> cumulativeBlocksByLevel = BlockChain.getInstance().getCumulativeBlocksByLevel();
final int maximumLevel = cumulativeBlocksByLevel.size() - 1;
int updatedCount = 0;
for (AccountPenaltyData penaltyData : accountPenalties) {
AccountData accountData = repository.getAccountRepository().getAccount(penaltyData.getAddress());
final int effectiveBlocksMinted = accountData.getBlocksMinted() + accountData.getBlocksMintedAdjustment() + accountData.getBlocksMintedPenalty();
// Shortcut for penalties
if (effectiveBlocksMinted < 0) {
accountData.setLevel(0);
repository.getAccountRepository().setLevel(accountData);
updatedCount++;
LOGGER.trace(() -> String.format("Block minter %s dropped to level %d", accountData.getAddress(), accountData.getLevel()));
continue;
}
for (int newLevel = maximumLevel; newLevel >= 0; --newLevel) {
if (effectiveBlocksMinted >= cumulativeBlocksByLevel.get(newLevel)) {
accountData.setLevel(newLevel);
repository.getAccountRepository().setLevel(accountData);
updatedCount++;
LOGGER.trace(() -> String.format("Block minter %s increased to level %d", accountData.getAddress(), accountData.getLevel()));
break;
}
}
}
return updatedCount;
}
private static void logPenaltyStats(Repository repository) {
try {
LOGGER.info(getPenaltyStats(repository));
} catch (DataException e) {}
}
private static AccountPenaltyStats getPenaltyStats(Repository repository) throws DataException {
List<AccountData> accounts = repository.getAccountRepository().getPenaltyAccounts();
return AccountPenaltyStats.fromAccounts(accounts);
}
public static String getHash(List<String> penaltyAddresses) {
if (penaltyAddresses == null || penaltyAddresses.isEmpty()) {
return null;
}
Collections.sort(penaltyAddresses);
return Base58.encode(Crypto.digest(StringUtils.join(penaltyAddresses).getBytes(StandardCharsets.UTF_8)));
}
}

73
src/main/java/org/qortal/controller/AutoUpdate.java
View File

@ -293,4 +293,77 @@ public class AutoUpdate extends Thread {
}
}
public static boolean attemptRestart() {
LOGGER.info(String.format("Restarting node..."));
// Give repository a chance to backup in case things go badly wrong (if enabled)
if (Settings.getInstance().getRepositoryBackupInterval() > 0) {
try {
// Timeout if the database isn't ready for backing up after 60 seconds
long timeout = 60 * 1000L;
RepositoryManager.backup(true, "backup", timeout);
} catch (TimeoutException e) {
LOGGER.info("Attempt to backup repository failed due to timeout: {}", e.getMessage());
// Continue with the node restart anyway...
}
}
// Call ApplyUpdate to end this process (unlocking current JAR so it can be replaced)
String javaHome = System.getProperty("java.home");
LOGGER.debug(String.format("Java home: %s", javaHome));
Path javaBinary = Paths.get(javaHome, "bin", "java");
LOGGER.debug(String.format("Java binary: %s", javaBinary));
try {
List<String> javaCmd = new ArrayList<>();
// Java runtime binary itself
javaCmd.add(javaBinary.toString());
// JVM arguments
javaCmd.addAll(ManagementFactory.getRuntimeMXBean().getInputArguments());
// Disable, but retain, any -agentlib JVM arg as sub-process might fail if it tries to reuse same port
javaCmd = javaCmd.stream()
.map(arg -> arg.replace("-agentlib", AGENTLIB_JVM_HOLDER_ARG))
.collect(Collectors.toList());
// Remove JNI options as they won't be supported by command-line 'java'
// These are typically added by the AdvancedInstaller Java launcher EXE
javaCmd.removeAll(Arrays.asList("abort", "exit", "vfprintf"));
// Call ApplyUpdate using JAR
javaCmd.addAll(Arrays.asList("-cp", JAR_FILENAME, ApplyUpdate.class.getCanonicalName()));
// Add command-line args saved from start-up
String[] savedArgs = Controller.getInstance().getSavedArgs();
if (savedArgs != null)
javaCmd.addAll(Arrays.asList(savedArgs));
LOGGER.info(String.format("Restarting node with: %s", String.join(" ", javaCmd)));
SysTray.getInstance().showMessage(Translator.INSTANCE.translate("SysTray", "AUTO_UPDATE"), //TODO
Translator.INSTANCE.translate("SysTray", "APPLYING_UPDATE_AND_RESTARTING"), //TODO
MessageType.INFO);
ProcessBuilder processBuilder = new ProcessBuilder(javaCmd);
// New process will inherit our stdout and stderr
processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT);
processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT);
Process process = processBuilder.start();
// Nothing to pipe to new process, so close output stream (process's stdin)
process.getOutputStream().close();
return true; // restarting node OK
} catch (Exception e) {
LOGGER.error(String.format("Failed to restart node: %s", e.getMessage()));
return true; // repo was okay, even if applying update failed
}
}
}

58
src/main/java/org/qortal/controller/BlockMinter.java
View File

@ -26,9 +26,6 @@ import org.qortal.data.block.CommonBlockData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.network.Network;
import org.qortal.network.Peer;
import org.qortal.network.message.BlockSummariesV2Message;
import org.qortal.network.message.HeightV2Message;
import org.qortal.network.message.Message;
import org.qortal.repository.BlockRepository;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
@ -38,6 +35,8 @@ import org.qortal.transaction.Transaction;
import org.qortal.utils.Base58;
import org.qortal.utils.NTP;
import static org.junit.Assert.assertNotNull;
// Minting new blocks
public class BlockMinter extends Thread {
@ -64,8 +63,8 @@ public class BlockMinter extends Thread {
public void run() {
Thread.currentThread().setName("BlockMinter");
if (Settings.getInstance().isLite()) {
// Lite nodes do not mint
if (Settings.getInstance().isTopOnly() || Settings.getInstance().isLite()) {
// Top only and lite nodes do not sign blocks
return;
}
if (Settings.getInstance().getWipeUnconfirmedOnStart()) {
@ -93,6 +92,8 @@ public class BlockMinter extends Thread {
List<Block> newBlocks = new ArrayList<>();
final boolean isSingleNodeTestnet = Settings.getInstance().isSingleNodeTestnet();
try (final Repository repository = RepositoryManager.getRepository()) {
// Going to need this a lot...
BlockRepository blockRepository = repository.getBlockRepository();
@ -111,8 +112,9 @@ public class BlockMinter extends Thread {
// Free up any repository locks
repository.discardChanges();
// Sleep for a while
Thread.sleep(1000);
// Sleep for a while.
// It's faster on single node testnets, to allow lots of blocks to be minted quickly.
Thread.sleep(isSingleNodeTestnet ? 50 : 1000);
isMintingPossible = false;
@ -223,9 +225,10 @@ public class BlockMinter extends Thread {
List<PrivateKeyAccount> newBlocksMintingAccounts = mintingAccountsData.stream().map(accountData -> new PrivateKeyAccount(repository, accountData.getPrivateKey())).collect(Collectors.toList());
// We might need to sit the next block out, if one of our minting accounts signed the previous one
// Skip this check for single node testnets, since they definitely need to mint every block
byte[] previousBlockMinter = previousBlockData.getMinterPublicKey();
boolean mintedLastBlock = mintingAccountsData.stream().anyMatch(mintingAccount -> Arrays.equals(mintingAccount.getPublicKey(), previousBlockMinter));
if (mintedLastBlock) {
if (mintedLastBlock && !isSingleNodeTestnet) {
LOGGER.trace(String.format("One of our keys signed the last block, so we won't sign the next one"));
continue;
}
@ -244,7 +247,7 @@ public class BlockMinter extends Thread {
Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
if (newBlock == null) {
// For some reason we can't mint right now
moderatedLog(() -> LOGGER.error("Couldn't build a to-be-minted block"));
moderatedLog(() -> LOGGER.info("Couldn't build a to-be-minted block"));
continue;
}
@ -377,9 +380,13 @@ public class BlockMinter extends Thread {
parentSignatureForLastLowWeightBlock = null;
timeOfLastLowWeightBlock = null;
Long unconfirmedStartTime = NTP.getTime();
// Add unconfirmed transactions
addUnconfirmedTransactions(repository, newBlock);
LOGGER.info(String.format("Adding %d unconfirmed transactions took %d ms", newBlock.getTransactions().size(), (NTP.getTime()-unconfirmedStartTime)));
// Sign to create block's signature
newBlock.sign();
@ -429,6 +436,10 @@ public class BlockMinter extends Thread {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
} catch (ArithmeticException e) {
// Unable to process block - report and discard
LOGGER.error("Unable to process newly minted block?", e);
newBlocks.clear();
}
} finally {
blockchainLock.unlock();
@ -477,6 +488,9 @@ public class BlockMinter extends Thread {
// Sign to create block's signature, needed by Block.isValid()
newBlock.sign();
// User-defined limit per block
int limit = Settings.getInstance().getMaxTransactionsPerBlock();
// Attempt to add transactions until block is full, or we run out
// If a transaction makes the block invalid then skip it and it'll either expire or be in next block.
for (TransactionData transactionData : unconfirmedTransactions) {
@ -489,6 +503,12 @@ public class BlockMinter extends Thread {
LOGGER.debug(() -> String.format("Skipping invalid transaction %s during block minting", Base58.encode(transactionData.getSignature())));
newBlock.deleteTransaction(transactionData);
}
// User-defined limit per block
List<Transaction> transactions = newBlock.getTransactions();
if (transactions != null && transactions.size() >= limit) {
break;
}
}
}
@ -507,6 +527,21 @@ public class BlockMinter extends Thread {
PrivateKeyAccount mintingAccount = mintingAndOnlineAccounts[0];
Block block = mintTestingBlockRetainingTimestamps(repository, mintingAccount);
assertNotNull("Minted block must not be null", block);
return block;
}
public static Block mintTestingBlockUnvalidated(Repository repository, PrivateKeyAccount... mintingAndOnlineAccounts) throws DataException {
if (!BlockChain.getInstance().isTestChain())
throw new DataException("Ignoring attempt to mint testing block for non-test chain!");
// Ensure mintingAccount is 'online' so blocks can be minted
OnlineAccountsManager.getInstance().ensureTestingAccountsOnline(mintingAndOnlineAccounts);
PrivateKeyAccount mintingAccount = mintingAndOnlineAccounts[0];
return mintTestingBlockRetainingTimestamps(repository, mintingAccount);
}
@ -514,6 +549,8 @@ public class BlockMinter extends Thread {
BlockData previousBlockData = repository.getBlockRepository().getLastBlock();
Block newBlock = Block.mint(repository, previousBlockData, mintingAccount);
if (newBlock == null)
return null;
// Make sure we're the only thread modifying the blockchain
ReentrantLock blockchainLock = Controller.getInstance().getBlockchainLock();
@ -525,6 +562,9 @@ public class BlockMinter extends Thread {
// Sign to create block's signature
newBlock.sign();
// Ensure online accounts are fully re-validated in this final check
newBlock.clearOnlineAccountsValidationCache();
// Is newBlock still valid?
ValidationResult validationResult = newBlock.isValid();
if (validationResult != ValidationResult.OK)

92
src/main/java/org/qortal/controller/Controller.java
View File

@ -29,6 +29,7 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.jsse.provider.BouncyCastleJsseProvider;
import org.qortal.account.Account;
import org.qortal.api.ApiService;
import org.qortal.api.DomainMapService;
import org.qortal.api.GatewayService;
@ -401,12 +402,11 @@ public class Controller extends Thread {
RepositoryManager.setRequestedCheckpoint(Boolean.TRUE);
try (final Repository repository = RepositoryManager.getRepository()) {
RepositoryManager.archive(repository);
RepositoryManager.prune(repository);
RepositoryManager.rebuildTransactionSequences(repository);
}
} catch (DataException e) {
// If exception has no cause then repository is in use by some other process.
if (e.getCause() == null) {
// If exception has no cause or message then repository is in use by some other process.
if (e.getCause() == null && e.getMessage() == null) {
LOGGER.info("Repository in use by another process?");
Gui.getInstance().fatalError("Repository issue", "Repository in use by another process?");
} else {
@ -440,6 +440,19 @@ public class Controller extends Thread {
}
}
try (Repository repository = RepositoryManager.getRepository()) {
if (RepositoryManager.needsTransactionSequenceRebuild(repository)) {
// Don't allow the node to start if transaction sequences haven't been built yet
// This is needed to handle a case when bootstrapping
LOGGER.error("Database upgrade needed. Please restart the core to complete the upgrade process.");
Gui.getInstance().fatalError("Database upgrade needed", "Please restart the core to complete the upgrade process.");
return;
}
} catch (DataException e) {
LOGGER.error("Error checking transaction sequences in repository", e);
return;
}
// Import current trade bot states and minting accounts if they exist
Controller.importRepositoryData();
@ -756,6 +769,28 @@ public class Controller extends Thread {
return peer.isAtLeastVersion(minPeerVersion) == false;
};
public static final Predicate<Peer> hasInvalidSigner = peer -> {
final BlockSummaryData peerChainTipData = peer.getChainTipData();
if (peerChainTipData == null)
return true;
try (Repository repository = RepositoryManager.getRepository()) {
return Account.getRewardShareEffectiveMintingLevel(repository, peerChainTipData.getMinterPublicKey()) == 0;
} catch (DataException e) {
return true;
}
};
public static final Predicate<Peer> wasRecentlyTooDivergent = peer -> {
Long now = NTP.getTime();
Long peerLastTooDivergentTime = peer.getLastTooDivergentTime();
if (now == null || peerLastTooDivergentTime == null)
return false;
// Exclude any peers that were TOO_DIVERGENT in the last 5 mins
return (now - peerLastTooDivergentTime < 5 * 60 * 1000L);
};
private long getRandomRepositoryMaintenanceInterval() {
final long minInterval = Settings.getInstance().getRepositoryMaintenanceMinInterval();
final long maxInterval = Settings.getInstance().getRepositoryMaintenanceMaxInterval();
@ -838,6 +873,12 @@ public class Controller extends Thread {
String tooltip = String.format("%s - %d %s", actionText, numberOfPeers, connectionsText);
if (!Settings.getInstance().isLite()) {
tooltip = tooltip.concat(String.format(" - %s %d", heightText, height));
final Integer blocksRemaining = Synchronizer.getInstance().getBlocksRemaining();
if (blocksRemaining != null && blocksRemaining > 0) {
String blocksRemainingText = Translator.INSTANCE.translate("SysTray", "BLOCKS_REMAINING");
tooltip = tooltip.concat(String.format(" - %d %s", blocksRemaining, blocksRemainingText));
}
}
tooltip = tooltip.concat(String.format("\n%s: %s", Translator.INSTANCE.translate("SysTray", "BUILD_VERSION"), this.buildVersion));
SysTray.getInstance().setToolTipText(tooltip);
@ -1237,13 +1278,6 @@ public class Controller extends Thread {
TransactionImporter.getInstance().onNetworkTransactionSignaturesMessage(peer, message);
break;
case GET_ONLINE_ACCOUNTS:
case ONLINE_ACCOUNTS:
case GET_ONLINE_ACCOUNTS_V2:
case ONLINE_ACCOUNTS_V2:
// No longer supported - to be eventually removed
break;
case GET_ONLINE_ACCOUNTS_V3:
OnlineAccountsManager.getInstance().onNetworkGetOnlineAccountsV3Message(peer, message);
break;
@ -1350,9 +1384,24 @@ public class Controller extends Thread {
// If we have no block data, we should check the archive in case it's there
if (blockData == null) {
if (Settings.getInstance().isArchiveEnabled()) {
byte[] bytes = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, true, repository);
if (bytes != null) {
CachedBlockMessage blockMessage = new CachedBlockMessage(bytes);
Triple<byte[], Integer, Integer> serializedBlock = BlockArchiveReader.getInstance().fetchSerializedBlockBytesForSignature(signature, true, repository);
if (serializedBlock != null) {
byte[] bytes = serializedBlock.getA();
Integer serializationVersion = serializedBlock.getB();
Message blockMessage;
switch (serializationVersion) {
case 1:
blockMessage = new CachedBlockMessage(bytes);
break;
case 2:
blockMessage = new CachedBlockV2Message(bytes);
break;
default:
return;
}
blockMessage.setId(message.getId());
// This call also causes the other needed data to be pulled in from repository
@ -1603,6 +1652,17 @@ public class Controller extends Thread {
}
}
if (message.hasId()) {
/*
* Experimental proof-of-concept: discard messages with ID
* These are 'late' reply messages received after timeout has expired,
* having been passed upwards from Peer to Network to Controller.
* Hence, these are NOT simple "here's my chain tip" broadcasts from other peers.
*/
LOGGER.debug("Discarding late {} message with ID {} from {}", message.getType().name(), message.getId(), peer);
return;
}
// Update peer chain tip data
peer.setChainTipSummaries(blockSummariesV2Message.getBlockSummaries());
@ -1861,6 +1921,10 @@ public class Controller extends Thread {
if (latestBlockData == null || latestBlockData.getTimestamp() < minLatestBlockTimestamp)
return false;
if (Settings.getInstance().isSingleNodeTestnet())
// Single node testnets won't have peers, so we can assume up to date from this point
return true;
// Needs a mutable copy of the unmodifiableList
List<Peer> peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
if (peers == null)

74
src/main/java/org/qortal/controller/DevProxyManager.java Normal file
View File

@ -0,0 +1,74 @@
package org.qortal.controller;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.api.DevProxyService;
import org.qortal.repository.DataException;
import org.qortal.settings.Settings;
public class DevProxyManager {
protected static final Logger LOGGER = LogManager.getLogger(DevProxyManager.class);
private static DevProxyManager instance;
private boolean running = false;
private String sourceHostAndPort = "127.0.0.1:5173"; // Default for React/Vite
private DevProxyManager() {
}
public static DevProxyManager getInstance() {
if (instance == null)
instance = new DevProxyManager();
return instance;
}
public void start() throws DataException {
synchronized(this) {
if (this.running) {
// Already running
return;
}
LOGGER.info(String.format("Starting developer proxy service on port %d", Settings.getInstance().getDevProxyPort()));
DevProxyService devProxyService = DevProxyService.getInstance();
devProxyService.start();
this.running = true;
}
}
public void stop() {
synchronized(this) {
if (!this.running) {
// Not running
return;
}
LOGGER.info(String.format("Shutting down developer proxy service"));
DevProxyService devProxyService = DevProxyService.getInstance();
devProxyService.stop();
this.running = false;
}
}
public void setSourceHostAndPort(String sourceHostAndPort) {
this.sourceHostAndPort = sourceHostAndPort;
}
public String getSourceHostAndPort() {
return this.sourceHostAndPort;
}
public Integer getPort() {
return Settings.getInstance().getDevProxyPort();
}
public boolean isRunning() {
return this.running;
}
}

279
src/main/java/org/qortal/controller/OnlineAccountsManager.java
View File

@ -64,9 +64,19 @@ public class OnlineAccountsManager {
private static final long ONLINE_ACCOUNTS_COMPUTE_INITIAL_SLEEP_INTERVAL = 30 * 1000L; // ms
// MemoryPoW
public final int POW_BUFFER_SIZE = 1 * 1024 * 1024; // bytes
public int POW_DIFFICULTY = 18; // leading zero bits
// MemoryPoW - mainnet
public static final int POW_BUFFER_SIZE = 1 * 1024 * 1024; // bytes
public static final int POW_DIFFICULTY_V1 = 18; // leading zero bits
public static final int POW_DIFFICULTY_V2 = 19; // leading zero bits
// MemoryPoW - testnet
public static final int POW_BUFFER_SIZE_TESTNET = 1 * 1024 * 1024; // bytes
public static final int POW_DIFFICULTY_TESTNET = 5; // leading zero bits
// IMPORTANT: if we ever need to dynamically modify the buffer size using a feature trigger, the
// pre-allocated buffer below will NOT work, and we should instead use a dynamically allocated
// one for the transition period.
private static long[] POW_VERIFY_WORK_BUFFER = new long[getPoWBufferSize() / 8];
private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(4, new NamedThreadFactory("OnlineAccounts"));
private volatile boolean isStopping = false;
@ -112,6 +122,23 @@ public class OnlineAccountsManager {
return (timestamp / getOnlineTimestampModulus()) * getOnlineTimestampModulus();
}
private static int getPoWBufferSize() {
if (Settings.getInstance().isTestNet())
return POW_BUFFER_SIZE_TESTNET;
return POW_BUFFER_SIZE;
}
private static int getPoWDifficulty(long timestamp) {
if (Settings.getInstance().isTestNet())
return POW_DIFFICULTY_TESTNET;
if (timestamp >= BlockChain.getInstance().getIncreaseOnlineAccountsDifficultyTimestamp())
return POW_DIFFICULTY_V2;
return POW_DIFFICULTY_V1;
}
private OnlineAccountsManager() {
}
@ -156,7 +183,6 @@ public class OnlineAccountsManager {
return;
byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
final boolean mempowActive = onlineAccountsTimestamp >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp();
Set<OnlineAccountData> replacementAccounts = new HashSet<>();
for (PrivateKeyAccount onlineAccount : onlineAccounts) {
@ -165,7 +191,7 @@ public class OnlineAccountsManager {
byte[] signature = Qortal25519Extras.signForAggregation(onlineAccount.getPrivateKey(), timestampBytes);
byte[] publicKey = onlineAccount.getPublicKey();
Integer nonce = mempowActive ? new Random().nextInt(500000) : null;
Integer nonce = new Random().nextInt(500000);
OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce);
replacementAccounts.add(ourOnlineAccountData);
@ -192,8 +218,8 @@ public class OnlineAccountsManager {
return;
// Skip this account if it's already validated
Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountData.getTimestamp(), k -> ConcurrentHashMap.newKeySet());
if (onlineAccounts.contains(onlineAccountData)) {
Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.get(onlineAccountData.getTimestamp());
if (onlineAccounts != null && onlineAccounts.contains(onlineAccountData)) {
// We have already validated this online account
onlineAccountsImportQueue.remove(onlineAccountData);
continue;
@ -214,8 +240,8 @@ public class OnlineAccountsManager {
if (!onlineAccountsToAdd.isEmpty()) {
LOGGER.debug("Merging {} validated online accounts from import queue", onlineAccountsToAdd.size());
addAccounts(onlineAccountsToAdd);
onlineAccountsImportQueue.removeAll(onlineAccountsToRemove);
}
onlineAccountsImportQueue.removeAll(onlineAccountsToRemove);
}
}
@ -321,13 +347,10 @@ public class OnlineAccountsManager {
return false;
}
// Validate mempow if feature trigger is active (or if online account's timestamp is past the trigger timestamp)
long memoryPoWStartTimestamp = BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp();
if (now >= memoryPoWStartTimestamp || onlineAccountTimestamp >= memoryPoWStartTimestamp) {
if (!getInstance().verifyMemoryPoW(onlineAccountData, now)) {
LOGGER.trace(() -> String.format("Rejecting online reward-share for account %s due to invalid PoW nonce", mintingAccount.getAddress()));
return false;
}
// Validate mempow
if (!getInstance().verifyMemoryPoW(onlineAccountData, POW_VERIFY_WORK_BUFFER)) {
LOGGER.trace(() -> String.format("Rejecting online reward-share for account %s due to invalid PoW nonce", mintingAccount.getAddress()));
return false;
}
return true;
@ -391,7 +414,7 @@ public class OnlineAccountsManager {
boolean isSuperiorEntry = isOnlineAccountsDataSuperior(onlineAccountData);
if (isSuperiorEntry)
// Remove existing inferior entry so it can be re-added below (it's likely the existing copy is missing a nonce value)
onlineAccounts.remove(onlineAccountData);
onlineAccounts.removeIf(a -> Objects.equals(a.getPublicKey(), onlineAccountData.getPublicKey()));
boolean isNewEntry = onlineAccounts.add(onlineAccountData);
@ -471,89 +494,91 @@ public class OnlineAccountsManager {
// 'next' timestamp (prioritize this as it's the most important, if mempow active)
final long nextOnlineAccountsTimestamp = toOnlineAccountTimestamp(now) + getOnlineTimestampModulus();
if (isMemoryPoWActive(now)) {
boolean success = computeOurAccountsForTimestamp(nextOnlineAccountsTimestamp);
if (!success) {
// We didn't compute the required nonce value(s), and so can't proceed until they have been retried
return;
}
boolean success = computeOurAccountsForTimestamp(nextOnlineAccountsTimestamp);
if (!success) {
// We didn't compute the required nonce value(s), and so can't proceed until they have been retried
return;
}
// 'current' timestamp
computeOurAccountsForTimestamp(onlineAccountsTimestamp);
}
private boolean computeOurAccountsForTimestamp(long onlineAccountsTimestamp) {
List<MintingAccountData> mintingAccounts;
try (final Repository repository = RepositoryManager.getRepository()) {
mintingAccounts = repository.getAccountRepository().getMintingAccounts();
private boolean computeOurAccountsForTimestamp(Long onlineAccountsTimestamp) {
if (onlineAccountsTimestamp != null) {
List<MintingAccountData> mintingAccounts;
try (final Repository repository = RepositoryManager.getRepository()) {
mintingAccounts = repository.getAccountRepository().getMintingAccounts();
// We have no accounts to send
if (mintingAccounts.isEmpty())
// We have no accounts to send
if (mintingAccounts.isEmpty())
return false;
// Only active reward-shares allowed
Iterator<MintingAccountData> iterator = mintingAccounts.iterator();
int i = 0;
while (iterator.hasNext()) {
MintingAccountData mintingAccountData = iterator.next();
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
if (rewardShareData == null) {
// Reward-share doesn't even exist - probably not a good sign
iterator.remove();
continue;
}
Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint()) {
// Minting-account component of reward-share can no longer mint - disregard
iterator.remove();
continue;
}
if (++i > 1 + 1) {
iterator.remove();
continue;
}
}
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage()));
return false;
}
// Only active reward-shares allowed
Iterator<MintingAccountData> iterator = mintingAccounts.iterator();
while (iterator.hasNext()) {
MintingAccountData mintingAccountData = iterator.next();
byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
List<OnlineAccountData> ourOnlineAccounts = new ArrayList<>();
RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey());
if (rewardShareData == null) {
// Reward-share doesn't even exist - probably not a good sign
iterator.remove();
int remaining = mintingAccounts.size();
for (MintingAccountData mintingAccountData : mintingAccounts) {
remaining--;
byte[] privateKey = mintingAccountData.getPrivateKey();
byte[] publicKey = Crypto.toPublicKey(privateKey);
// We don't want to compute the online account nonce and signature again if it already exists
Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet());
boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey));
if (alreadyExists) {
this.hasOurOnlineAccounts = true;
if (remaining > 0) {
// Move on to next account
continue;
} else {
// Everything exists, so return true
return true;
}
}
// Generate bytes for mempow
byte[] mempowBytes;
try {
mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp);
} catch (IOException e) {
LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account...");
continue;
}
Account mintingAccount = new Account(repository, rewardShareData.getMinter());
if (!mintingAccount.canMint()) {
// Minting-account component of reward-share can no longer mint - disregard
iterator.remove();
continue;
}
}
} catch (DataException e) {
LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage()));
return false;
}
byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
List<OnlineAccountData> ourOnlineAccounts = new ArrayList<>();
int remaining = mintingAccounts.size();
for (MintingAccountData mintingAccountData : mintingAccounts) {
remaining--;
byte[] privateKey = mintingAccountData.getPrivateKey();
byte[] publicKey = Crypto.toPublicKey(privateKey);
// We don't want to compute the online account nonce and signature again if it already exists
Set<OnlineAccountData> onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet());
boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey));
if (alreadyExists) {
this.hasOurOnlineAccounts = true;
if (remaining > 0) {
// Move on to next account
continue;
}
else {
// Everything exists, so return true
return true;
}
}
// Generate bytes for mempow
byte[] mempowBytes;
try {
mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp);
}
catch (IOException e) {
LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account...");
continue;
}
// Compute nonce
Integer nonce;
if (isMemoryPoWActive(NTP.getTime())) {
// Compute nonce
Integer nonce;
try {
nonce = this.computeMemoryPoW(mempowBytes, publicKey, onlineAccountsTimestamp);
if (nonce == null) {
@ -564,47 +589,39 @@ public class OnlineAccountsManager {
LOGGER.info(String.format("Timed out computing nonce for account %.8s", Base58.encode(publicKey)));
return false;
}
}
else {
// Send -1 if we haven't computed a nonce due to feature trigger timestamp
nonce = -1;
byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes);
// Our account is online
OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce);
// Make sure to verify before adding
if (verifyMemoryPoW(ourOnlineAccountData, null)) {
ourOnlineAccounts.add(ourOnlineAccountData);
}
}
byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes);
this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty();
// Our account is online
OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce);
boolean hasInfoChanged = addAccounts(ourOnlineAccounts);
// Make sure to verify before adding
if (verifyMemoryPoW(ourOnlineAccountData, NTP.getTime())) {
ourOnlineAccounts.add(ourOnlineAccountData);
}
if (!hasInfoChanged)
return false;
Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts));
LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp);
return true;
}
this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty();
boolean hasInfoChanged = addAccounts(ourOnlineAccounts);
if (!hasInfoChanged)
return false;
Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts));
LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp);
return true;
return false;
}
// MemoryPoW
private boolean isMemoryPoWActive(Long timestamp) {
if (timestamp >= BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp() || Settings.getInstance().isOnlineAccountsMemPoWEnabled()) {
return true;
}
return false;
}
private byte[] getMemoryPoWBytes(byte[] publicKey, long onlineAccountsTimestamp) throws IOException {
byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp);
@ -616,11 +633,6 @@ public class OnlineAccountsManager {
}
private Integer computeMemoryPoW(byte[] bytes, byte[] publicKey, long onlineAccountsTimestamp) throws TimeoutException {
if (!isMemoryPoWActive(NTP.getTime())) {
LOGGER.info("Mempow start timestamp not yet reached, and onlineAccountsMemPoWEnabled not enabled in settings");
return null;
}
LOGGER.info(String.format("Computing nonce for account %.8s and timestamp %d...", Base58.encode(publicKey), onlineAccountsTimestamp));
// Calculate the time until the next online timestamp and use it as a timeout when computing the nonce
@ -628,7 +640,8 @@ public class OnlineAccountsManager {
final long nextOnlineAccountsTimestamp = toOnlineAccountTimestamp(startTime) + getOnlineTimestampModulus();
long timeUntilNextTimestamp = nextOnlineAccountsTimestamp - startTime;
Integer nonce = MemoryPoW.compute2(bytes, POW_BUFFER_SIZE, POW_DIFFICULTY, timeUntilNextTimestamp);
int difficulty = getPoWDifficulty(onlineAccountsTimestamp);
Integer nonce = MemoryPoW.compute2(bytes, getPoWBufferSize(), difficulty, timeUntilNextTimestamp);
double totalSeconds = (NTP.getTime() - startTime) / 1000.0f;
int minutes = (int) ((totalSeconds % 3600) / 60);
@ -637,16 +650,15 @@ public class OnlineAccountsManager {
LOGGER.info(String.format("Computed nonce for timestamp %d and account %.8s: %d. Buffer size: %d. Difficulty: %d. " +
"Time taken: %02d:%02d. Hashrate: %f", onlineAccountsTimestamp, Base58.encode(publicKey),
nonce, POW_BUFFER_SIZE, POW_DIFFICULTY, minutes, seconds, hashRate));
nonce, getPoWBufferSize(), difficulty, minutes, seconds, hashRate));
return nonce;
}
public boolean verifyMemoryPoW(OnlineAccountData onlineAccountData, Long timestamp) {
long memoryPoWStartTimestamp = BlockChain.getInstance().getOnlineAccountsMemoryPoWTimestamp();
if (timestamp < memoryPoWStartTimestamp && onlineAccountData.getTimestamp() < memoryPoWStartTimestamp) {
// Not active yet, so treat it as valid
return true;
public boolean verifyMemoryPoW(OnlineAccountData onlineAccountData, long[] workBuffer) {
// Require a valid nonce value
if (onlineAccountData.getNonce() == null || onlineAccountData.getNonce() < 0) {
return false;
}
int nonce = onlineAccountData.getNonce();
@ -659,7 +671,7 @@ public class OnlineAccountsManager {
}
// Verify the nonce
return MemoryPoW.verify2(mempowBytes, POW_BUFFER_SIZE, POW_DIFFICULTY, nonce);
return MemoryPoW.verify2(mempowBytes, workBuffer, getPoWBufferSize(), getPoWDifficulty(onlineAccountData.getTimestamp()), nonce);
}
@ -697,7 +709,7 @@ public class OnlineAccountsManager {
*/
// Block::mint() - only wants online accounts with (online) timestamp that matches block's (online) timestamp so they can be added to new block
public List<OnlineAccountData> getOnlineAccounts(long onlineTimestamp) {
LOGGER.info(String.format("caller's timestamp: %d, our timestamps: %s", onlineTimestamp, String.join(", ", this.currentOnlineAccounts.keySet().stream().map(l -> Long.toString(l)).collect(Collectors.joining(", ")))));
LOGGER.debug(String.format("caller's timestamp: %d, our timestamps: %s", onlineTimestamp, String.join(", ", this.currentOnlineAccounts.keySet().stream().map(l -> Long.toString(l)).collect(Collectors.joining(", ")))));
return new ArrayList<>(Set.copyOf(this.currentOnlineAccounts.getOrDefault(onlineTimestamp, Collections.emptySet())));
}
@ -743,11 +755,12 @@ public class OnlineAccountsManager {
* Typically called by {@link Block#areOnlineAccountsValid()}
*/
public void addBlocksOnlineAccounts(Set<OnlineAccountData> blocksOnlineAccounts, Long timestamp) {
// We want to add to 'current' in preference if possible
if (this.currentOnlineAccounts.containsKey(timestamp)) {
addAccounts(blocksOnlineAccounts);
// If these are current accounts, then there is no need to cache them, and should instead rely
// on the more complete entries we already have in self.currentOnlineAccounts.
// Note: since sig-agg, we no longer have individual signatures included in blocks, so we
// mustn't add anything to currentOnlineAccounts from here.
if (this.currentOnlineAccounts.containsKey(timestamp))
return;
}
// Add to block cache instead
this.latestBlocksOnlineAccounts.computeIfAbsent(timestamp, k -> ConcurrentHashMap.newKeySet())

2
src/main/java/org/qortal/controller/PirateChainWalletController.java
View File

@ -163,7 +163,7 @@ public class PirateChainWalletController extends Thread {
// Library not found, so check if we've fetched the resource from QDN
ArbitraryTransactionData t = this.getTransactionData(repository);
if (t == null) {
if (t == null || t.getService() == null) {
// Can't find the transaction - maybe on a different chain?
return;
}

108
src/main/java/org/qortal/controller/Synchronizer.java
View File

@ -8,7 +8,6 @@ import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@ -44,7 +43,7 @@ public class Synchronizer extends Thread {
private static final int SYNC_BATCH_SIZE = 1000; // XXX move to Settings?
/** Initial jump back of block height when searching for common block with peer */
private static final int INITIAL_BLOCK_STEP = 7;
private static final int INITIAL_BLOCK_STEP = 8;
/** Maximum jump back of block height when searching for common block with peer */
private static final int MAXIMUM_BLOCK_STEP = 128;
@ -54,7 +53,8 @@ public class Synchronizer extends Thread {
/** Maximum number of block signatures we ask from peer in one go */
private static final int MAXIMUM_REQUEST_SIZE = 200; // XXX move to Settings?
private static final long RECOVERY_MODE_TIMEOUT = 10 * 60 * 1000L; // ms
/** Maximum number of consecutive failed sync attempts before marking peer as misbehaved */
private static final int MAX_CONSECUTIVE_FAILED_SYNC_ATTEMPTS = 3;
private boolean running;
@ -76,6 +76,8 @@ public class Synchronizer extends Thread {
private volatile boolean isSynchronizing = false;
/** Temporary estimate of synchronization progress for SysTray use. */
private volatile int syncPercent = 0;
/** Temporary estimate of blocks remaining for SysTray use. */
private volatile int blocksRemaining = 0;
private static volatile boolean requestSync = false;
private boolean syncRequestPending = false;
@ -181,6 +183,18 @@ public class Synchronizer extends Thread {
}
}
public Integer getBlocksRemaining() {
synchronized (this.syncLock) {
// Report as 0 blocks remaining if the latest block is within the last 60 mins
final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L);
if (Controller.getInstance().isUpToDate(minLatestBlockTimestamp)) {
return 0;
}
return this.isSynchronizing ? this.blocksRemaining : null;
}
}
public void requestSync() {
requestSync = true;
}
@ -215,13 +229,6 @@ public class Synchronizer extends Thread {
peers.removeIf(Controller.hasOldVersion);
checkRecoveryModeForPeers(peers);
if (recoveryMode) {
// Needs a mutable copy of the unmodifiableList
peers = new ArrayList<>(Network.getInstance().getImmutableHandshakedPeers());
peers.removeIf(Controller.hasOnlyGenesisBlock);
peers.removeIf(Controller.hasMisbehaved);
peers.removeIf(Controller.hasOldVersion);
}
// Check we have enough peers to potentially synchronize
if (peers.size() < Settings.getInstance().getMinBlockchainPeers())
@ -233,6 +240,9 @@ public class Synchronizer extends Thread {
// Disregard peers that are on the same block as last sync attempt and we didn't like their chain
peers.removeIf(Controller.hasInferiorChainTip);
// Disregard peers that have a block with an invalid signer
peers.removeIf(Controller.hasInvalidSigner);
final int peersBeforeComparison = peers.size();
// Request recent block summaries from the remaining peers, and locate our common block with each
@ -245,10 +255,7 @@ public class Synchronizer extends Thread {
peers.removeIf(Controller.hasInferiorChainTip);
// Remove any peers that are no longer on a recent block since the last check
// Except for times when we're in recovery mode, in which case we need to keep them
if (!recoveryMode) {
peers.removeIf(Controller.hasNoRecentBlock);
}
peers.removeIf(Controller.hasNoRecentBlock);
final int peersRemoved = peersBeforeComparison - peers.size();
if (peersRemoved > 0 && peers.size() > 0)
@ -397,9 +404,10 @@ public class Synchronizer extends Thread {
timePeersLastAvailable = NTP.getTime();
// If enough time has passed, enter recovery mode, which lifts some restrictions on who we can sync with and when we can mint
if (NTP.getTime() - timePeersLastAvailable > RECOVERY_MODE_TIMEOUT) {
long recoveryModeTimeout = Settings.getInstance().getRecoveryModeTimeout();
if (NTP.getTime() - timePeersLastAvailable > recoveryModeTimeout) {
if (recoveryMode == false) {
LOGGER.info(String.format("Peers have been unavailable for %d minutes. Entering recovery mode...", RECOVERY_MODE_TIMEOUT/60/1000));
LOGGER.info(String.format("Peers have been unavailable for %d minutes. Entering recovery mode...", recoveryModeTimeout/60/1000));
recoveryMode = true;
}
}
@ -1103,6 +1111,7 @@ public class Synchronizer extends Thread {
// If common block is too far behind us then we're on massively different forks so give up.
if (!force && testHeight < ourHeight - MAXIMUM_COMMON_DELTA) {
LOGGER.info(String.format("Blockchain too divergent with peer %s", peer));
peer.setLastTooDivergentTime(NTP.getTime());
return SynchronizationResult.TOO_DIVERGENT;
}
@ -1112,6 +1121,9 @@ public class Synchronizer extends Thread {
testHeight = Math.max(testHeight - step, 1);
}
// Peer not considered too divergent
peer.setLastTooDivergentTime(0L);
// Prepend test block's summary as first block summary, as summaries returned are *after* test block
BlockSummaryData testBlockSummary = new BlockSummaryData(testBlockData);
blockSummariesFromCommon.add(0, testBlockSummary);
@ -1318,8 +1330,8 @@ public class Synchronizer extends Thread {
return SynchronizationResult.INVALID_DATA;
}
// Final check to make sure the peer isn't out of date (except for when we're in recovery mode)
if (!recoveryMode && peer.getChainTipData() != null) {
// Final check to make sure the peer isn't out of date
if (peer.getChainTipData() != null) {
final Long minLatestBlockTimestamp = Controller.getMinimumLatestBlockTimestamp();
final Long peerLastBlockTimestamp = peer.getChainTipData().getTimestamp();
if (peerLastBlockTimestamp == null || peerLastBlockTimestamp < minLatestBlockTimestamp) {
@ -1456,6 +1468,12 @@ public class Synchronizer extends Thread {
repository.saveChanges();
synchronized (this.syncLock) {
if (peer.getChainTipData() != null) {
this.blocksRemaining = peer.getChainTipData().getHeight() - newBlock.getBlockData().getHeight();
}
}
Controller.getInstance().onNewBlock(newBlock.getBlockData());
}
@ -1551,47 +1569,19 @@ public class Synchronizer extends Thread {
repository.saveChanges();
synchronized (this.syncLock) {
if (peer.getChainTipData() != null) {
this.blocksRemaining = peer.getChainTipData().getHeight() - newBlock.getBlockData().getHeight();
}
}
Controller.getInstance().onNewBlock(newBlock.getBlockData());
}
return SynchronizationResult.OK;
}
private List<BlockSummaryData> getBlockSummariesFromCache(Peer peer, byte[] parentSignature, int numberRequested) {
List<BlockSummaryData> peerSummaries = peer.getChainTipSummaries();
if (peerSummaries == null)
return null;
// Check if the requested parent block exists in peer's summaries cache
int parentIndex = IntStream.range(0, peerSummaries.size()).filter(i -> Arrays.equals(peerSummaries.get(i).getSignature(), parentSignature)).findFirst().orElse(-1);
if (parentIndex < 0)
return null;
// Peer's summaries contains the requested parent, so return summaries after that
// Make sure we have at least one block after the parent block
int summariesAvailable = peerSummaries.size() - parentIndex - 1;
if (summariesAvailable <= 0)
return null;
// Don't try and return more summaries than we have, or more than were requested
int summariesToReturn = Math.min(numberRequested, summariesAvailable);
int startIndex = parentIndex + 1;
int endIndex = startIndex + summariesToReturn - 1;
if (endIndex > peerSummaries.size() - 1)
return null;
LOGGER.trace("Serving {} block summaries from cache", summariesToReturn);
return peerSummaries.subList(startIndex, endIndex);
}
private List<BlockSummaryData> getBlockSummaries(Peer peer, byte[] parentSignature, int numberRequested) throws InterruptedException {
// We might be able to shortcut the response if we already have the summaries in the peer's chain tip data
List<BlockSummaryData> cachedSummaries = this.getBlockSummariesFromCache(peer, parentSignature, numberRequested);
if (cachedSummaries != null && !cachedSummaries.isEmpty())
return cachedSummaries;
LOGGER.trace("Requesting {} block summaries from peer {}", numberRequested, peer);
Message getBlockSummariesMessage = new GetBlockSummariesMessage(parentSignature, numberRequested);
Message message = peer.getResponse(getBlockSummariesMessage);
@ -1626,8 +1616,20 @@ public class Synchronizer extends Thread {
Message getBlockMessage = new GetBlockMessage(signature);
Message message = peer.getResponse(getBlockMessage);
if (message == null)
if (message == null) {
peer.getPeerData().incrementFailedSyncCount();
if (peer.getPeerData().getFailedSyncCount() >= MAX_CONSECUTIVE_FAILED_SYNC_ATTEMPTS) {
// Several failed attempts, so mark peer as misbehaved
LOGGER.info("Marking peer {} as misbehaved due to {} failed sync attempts", peer, peer.getPeerData().getFailedSyncCount());
Network.getInstance().peerMisbehaved(peer);
}
return null;
}
// Reset failed sync count now that we have a block response
// FUTURE: we could move this to the end of the sync process, but to reduce risk this can be done
// at a later stage. For now we are only defending against serialization errors or no responses.
peer.getPeerData().setFailedSyncCount(0);
switch (message.getType()) {
case BLOCK: {

36
src/main/java/org/qortal/controller/TransactionImporter.java
View File

@ -47,6 +47,9 @@ public class TransactionImporter extends Thread {
/** Map of recent invalid unconfirmed transactions. Key is base58 transaction signature, value is do-not-request expiry timestamp. */
private final Map<String, Long> invalidUnconfirmedTransactions = Collections.synchronizedMap(new HashMap<>());
/** Cached list of unconfirmed transactions, used when counting per creator. This is replaced regularly */
public static List<TransactionData> unconfirmedTransactionsCache = null;
public static synchronized TransactionImporter getInstance() {
if (instance == null) {
@ -215,12 +218,6 @@ public class TransactionImporter extends Thread {
LOGGER.debug("Finished validating signatures in incoming transactions queue (valid this round: {}, total pending import: {})...", validatedCount, sigValidTransactions.size());
}
if (!newlyValidSignatures.isEmpty()) {
LOGGER.debug("Broadcasting {} newly valid signatures ahead of import", newlyValidSignatures.size());
Message newTransactionSignatureMessage = new TransactionSignaturesMessage(newlyValidSignatures);
Network.getInstance().broadcast(broadcastPeer -> newTransactionSignatureMessage);
}
} catch (DataException e) {
LOGGER.error("Repository issue while processing incoming transactions", e);
}
@ -254,6 +251,15 @@ public class TransactionImporter extends Thread {
int processedCount = 0;
try (final Repository repository = RepositoryManager.getRepository()) {
// Use a single copy of the unconfirmed transactions list for each cycle, to speed up constant lookups
// when counting unconfirmed transactions by creator.
List<TransactionData> unconfirmedTransactions = repository.getTransactionRepository().getUnconfirmedTransactions();
unconfirmedTransactions.removeIf(t -> t.getType() == Transaction.TransactionType.CHAT);
unconfirmedTransactionsCache = unconfirmedTransactions;
// A list of signatures were imported in this round
List<byte[]> newlyImportedSignatures = new ArrayList<>();
// Import transactions with valid signatures
try {
for (int i = 0; i < sigValidTransactions.size(); ++i) {
@ -286,6 +292,15 @@ public class TransactionImporter extends Thread {
case OK: {
LOGGER.debug(() -> String.format("Imported %s transaction %s", transactionData.getType().name(), Base58.encode(transactionData.getSignature())));
// Add to the unconfirmed transactions cache
if (transactionData.getType() != Transaction.TransactionType.CHAT && unconfirmedTransactionsCache != null) {
unconfirmedTransactionsCache.add(transactionData);
}
// Signature imported in this round
newlyImportedSignatures.add(transactionData.getSignature());
break;
}
@ -314,9 +329,18 @@ public class TransactionImporter extends Thread {
// Transaction has been processed, even if only to reject it
removeIncomingTransaction(transactionData.getSignature());
}
if (!newlyImportedSignatures.isEmpty()) {
LOGGER.debug("Broadcasting {} newly imported signatures", newlyImportedSignatures.size());
Message newTransactionSignatureMessage = new TransactionSignaturesMessage(newlyImportedSignatures);
Network.getInstance().broadcast(broadcastPeer -> newTransactionSignatureMessage);
}
} finally {
LOGGER.debug("Finished importing {} incoming transaction{}", processedCount, (processedCount == 1 ? "" : "s"));
blockchainLock.unlock();
// Clear the unconfirmed transaction cache so new data can be populated in the next cycle
unconfirmedTransactionsCache = null;
}
} catch (DataException e) {
LOGGER.error("Repository issue while importing incoming transactions", e);

17
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java
View File

@ -11,10 +11,7 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.transaction.Transaction;
import org.qortal.transaction.Transaction.TransactionType;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58;
import org.qortal.utils.FilesystemUtils;
import org.qortal.utils.NTP;
import org.qortal.utils.*;
import java.io.File;
import java.io.IOException;
@ -137,7 +134,7 @@ public class ArbitraryDataCleanupManager extends Thread {
// Fetch the transaction data
ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature);
if (arbitraryTransactionData == null) {
if (arbitraryTransactionData == null || arbitraryTransactionData.getService() == null) {
continue;
}
@ -204,7 +201,7 @@ public class ArbitraryDataCleanupManager extends Thread {
if (completeFileExists && !allChunksExist) {
// We have the complete file but not the chunks, so let's convert it
LOGGER.info(String.format("Transaction %s has complete file but no chunks",
LOGGER.debug(String.format("Transaction %s has complete file but no chunks",
Base58.encode(arbitraryTransactionData.getSignature())));
ArbitraryTransactionUtils.convertFileToChunks(arbitraryTransactionData, now, STALE_FILE_TIMEOUT);
@ -239,7 +236,7 @@ public class ArbitraryDataCleanupManager extends Thread {
// Delete random data associated with name if we're over our storage limit for this name
// Use the DELETION_THRESHOLD, for the same reasons as above
for (String followedName : storageManager.followedNames()) {
for (String followedName : ListUtils.followedNames()) {
if (isStopping) {
return;
}
@ -349,6 +346,10 @@ public class ArbitraryDataCleanupManager extends Thread {
/**
* Iteratively walk through given directory and delete a single random file
*
* TODO: public data should be prioritized over private data
* (unless this node is part of a data market contract for that data).
* See: Service.privateServices() for a list of services containing private data.
*
* @param directory - the base directory
* @return boolean - whether a file was deleted
*/
@ -487,7 +488,7 @@ public class ArbitraryDataCleanupManager extends Thread {
// Delete data relating to blocked names
String name = directory.getName();
if (name != null && storageManager.isNameBlocked(name)) {
if (name != null && ListUtils.isNameBlocked(name)) {
this.safeDeleteDirectory(directory, "blocked name");
}

49
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java
View File

@ -20,6 +20,7 @@ import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.utils.Base58;
import org.qortal.utils.ListUtils;
import org.qortal.utils.NTP;
import org.qortal.utils.Triple;
@ -123,29 +124,29 @@ public class ArbitraryDataFileListManager {
}
}
// Then allow another 3 attempts, each 5 minutes apart
if (timeSinceLastAttempt > 5 * 60 * 1000L) {
// We haven't tried for at least 5 minutes
// Then allow another 5 attempts, each 1 minute apart
if (timeSinceLastAttempt > 60 * 1000L) {
// We haven't tried for at least 1 minute
if (networkBroadcastCount < 6) {
// We've made less than 6 total attempts
if (networkBroadcastCount < 8) {
// We've made less than 8 total attempts
return true;
}
}
// Then allow another 4 attempts, each 30 minutes apart
if (timeSinceLastAttempt > 30 * 60 * 1000L) {
// We haven't tried for at least 5 minutes
// Then allow another 8 attempts, each 15 minutes apart
if (timeSinceLastAttempt > 15 * 60 * 1000L) {
// We haven't tried for at least 15 minutes
if (networkBroadcastCount < 10) {
// We've made less than 10 total attempts
if (networkBroadcastCount < 16) {
// We've made less than 16 total attempts
return true;
}
}
// From then on, only try once every 24 hours, to reduce network spam
if (timeSinceLastAttempt > 24 * 60 * 60 * 1000L) {
// We haven't tried for at least 24 hours
// From then on, only try once every 6 hours, to reduce network spam
if (timeSinceLastAttempt > 6 * 60 * 60 * 1000L) {
// We haven't tried for at least 6 hours
return true;
}
@ -258,8 +259,6 @@ public class ArbitraryDataFileListManager {
// Lookup file lists by signature (and optionally hashes)
public boolean fetchArbitraryDataFileList(ArbitraryTransactionData arbitraryTransactionData) {
byte[] digest = arbitraryTransactionData.getData();
byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
byte[] signature = arbitraryTransactionData.getSignature();
String signature58 = Base58.encode(signature);
@ -286,8 +285,7 @@ public class ArbitraryDataFileListManager {
// Find hashes that we are missing
try {
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature);
arbitraryDataFile.setMetadataHash(metadataHash);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
missingHashes = arbitraryDataFile.missingHashes();
} catch (DataException e) {
// Leave missingHashes as null, so that all hashes are requested
@ -460,10 +458,9 @@ public class ArbitraryDataFileListManager {
arbitraryTransactionData = (ArbitraryTransactionData) transactionData;
// Load data file(s)
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature);
arbitraryDataFile.setMetadataHash(arbitraryTransactionData.getMetadataHash());
// // Load data file(s)
// ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
//
// // Check all hashes exist
// for (byte[] hash : hashes) {
// //LOGGER.debug("Received hash {}", Base58.encode(hash));
@ -507,7 +504,7 @@ public class ArbitraryDataFileListManager {
// Forwarding
if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) {
boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName()));
boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
if (!isBlocked) {
Peer requestingPeer = request.getB();
if (requestingPeer != null) {
@ -594,12 +591,8 @@ public class ArbitraryDataFileListManager {
// Check if we're even allowed to serve data for this transaction
if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) {
byte[] hash = transactionData.getData();
byte[] metadataHash = transactionData.getMetadataHash();
// Load file(s) and add any that exist to the list of hashes
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature);
arbitraryDataFile.setMetadataHash(metadataHash);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData);
// If the peer didn't supply a hash list, we need to return all hashes for this transaction
if (requestedHashes == null || requestedHashes.isEmpty()) {
@ -690,7 +683,7 @@ public class ArbitraryDataFileListManager {
}
// We may need to forward this request on
boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName()));
boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it

43
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java
View File

@ -82,7 +82,7 @@ public class ArbitraryDataFileManager extends Thread {
try {
// Use a fixed thread pool to execute the arbitrary data file requests
int threadCount = 10;
int threadCount = 5;
ExecutorService arbitraryDataFileRequestExecutor = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < threadCount; i++) {
arbitraryDataFileRequestExecutor.execute(new ArbitraryDataFileRequestThread());
@ -132,9 +132,7 @@ public class ArbitraryDataFileManager extends Thread {
List<byte[]> hashes) throws DataException {
// Load data file(s)
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature);
byte[] metadataHash = arbitraryTransactionData.getMetadataHash();
arbitraryDataFile.setMetadataHash(metadataHash);
ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData);
boolean receivedAtLeastOneFile = false;
// Now fetch actual data from this peer
@ -148,10 +146,10 @@ public class ArbitraryDataFileManager extends Thread {
if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) {
LOGGER.debug("Requesting data file {} from peer {}", hash58, peer);
Long startTime = NTP.getTime();
ArbitraryDataFileMessage receivedArbitraryDataFileMessage = fetchArbitraryDataFile(peer, null, signature, hash, null);
ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, null, signature, hash, null);
Long endTime = NTP.getTime();
if (receivedArbitraryDataFileMessage != null && receivedArbitraryDataFileMessage.getArbitraryDataFile() != null) {
LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFileMessage.getArbitraryDataFile().getHash58(), peer, (endTime-startTime));
if (receivedArbitraryDataFile != null) {
LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime));
receivedAtLeastOneFile = true;
// Remove this hash from arbitraryDataFileHashResponses now that we have received it
@ -193,11 +191,11 @@ public class ArbitraryDataFileManager extends Thread {
return receivedAtLeastOneFile;
}
private ArbitraryDataFileMessage fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException {
ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature);
boolean fileAlreadyExists = existingFile.exists();
String hash58 = Base58.encode(hash);
ArbitraryDataFileMessage arbitraryDataFileMessage;
ArbitraryDataFile arbitraryDataFile;
// Fetch the file if it doesn't exist locally
if (!fileAlreadyExists) {
@ -227,28 +225,32 @@ public class ArbitraryDataFileManager extends Thread {
}
ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response;
arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, peersArbitraryDataFileMessage.getArbitraryDataFile());
arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile();
} else {
LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58));
arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, existingFile);
arbitraryDataFile = existingFile;
}
if (arbitraryDataFile == null) {
// We don't have a file, so give up here
return null;
}
// We might want to forward the request to the peer that originally requested it
this.handleArbitraryDataFileForwarding(requestingPeer, arbitraryDataFileMessage, originalMessage);
this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage);
boolean isRelayRequest = (requestingPeer != null);
if (isRelayRequest) {
if (!fileAlreadyExists) {
// File didn't exist locally before the request, and it's a forwarding request, so delete it
LOGGER.debug("Deleting file {} because it was needed for forwarding only", Base58.encode(hash));
ArbitraryDataFile dataFile = arbitraryDataFileMessage.getArbitraryDataFile();
// Keep trying to delete the data until it is deleted, or we reach 10 attempts
dataFile.delete(10);
arbitraryDataFile.delete(10);
}
}
return arbitraryDataFileMessage;
return arbitraryDataFile;
}
private void handleFileListRequests(byte[] signature) {
@ -288,7 +290,7 @@ public class ArbitraryDataFileManager extends Thread {
// The ID needs to match that of the original request
message.setId(originalMessage.getId());
if (!requestingPeer.sendMessage(message)) {
if (!requestingPeer.sendMessageWithTimeout(message, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
LOGGER.debug("Failed to forward arbitrary data file to peer {}", requestingPeer);
requestingPeer.disconnect("failed to forward arbitrary data file");
}
@ -564,13 +566,16 @@ public class ArbitraryDataFileManager extends Thread {
LOGGER.trace("Hash {} exists", hash58);
// We can serve the file directly as we already have it
LOGGER.debug("Sending file {}...", arbitraryDataFile);
ArbitraryDataFileMessage arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, arbitraryDataFile);
arbitraryDataFileMessage.setId(message.getId());
if (!peer.sendMessage(arbitraryDataFileMessage)) {
LOGGER.debug("Couldn't sent file");
if (!peer.sendMessageWithTimeout(arbitraryDataFileMessage, (int) ArbitraryDataManager.ARBITRARY_REQUEST_TIMEOUT)) {
LOGGER.debug("Couldn't send file {}", arbitraryDataFile);
peer.disconnect("failed to send file");
}
LOGGER.debug("Sent file {}", arbitraryDataFile);
else {
LOGGER.debug("Sent file {}", arbitraryDataFile);
}
}
else if (relayInfo != null) {
LOGGER.debug("We have relay info for hash {}", Base58.encode(hash));

2
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java
View File

@ -114,7 +114,7 @@ public class ArbitraryDataFileRequestThread implements Runnable {
return;
}
LOGGER.debug("Fetching file {} from peer {} via request thread...", hash58, peer);
LOGGER.trace("Fetching file {} from peer {} via request thread...", hash58, peer);
arbitraryDataFileManager.fetchArbitraryDataFiles(repository, peer, signature, arbitraryTransactionData, Arrays.asList(hash));
} catch (DataException e) {

16
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java
View File

@ -16,7 +16,6 @@ import org.qortal.arbitrary.misc.Service;
import org.qortal.controller.Controller;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.list.ResourceListManager;
import org.qortal.network.Network;
import org.qortal.network.Peer;
import org.qortal.repository.DataException;
@ -27,6 +26,7 @@ import org.qortal.transaction.ArbitraryTransaction;
import org.qortal.transaction.Transaction.TransactionType;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58;
import org.qortal.utils.ListUtils;
import org.qortal.utils.NTP;
public class ArbitraryDataManager extends Thread {
@ -172,7 +172,7 @@ public class ArbitraryDataManager extends Thread {
private void processNames() throws InterruptedException {
// Fetch latest list of followed names
List<String> followedNames = ResourceListManager.getInstance().getStringsInList("followedNames");
List<String> followedNames = ListUtils.followedNames();
if (followedNames == null || followedNames.isEmpty()) {
return;
}
@ -275,7 +275,10 @@ public class ArbitraryDataManager extends Thread {
int offset = 0;
while (!isStopping) {
Thread.sleep(1000L);
final int minSeconds = 3;
final int maxSeconds = 10;
final int randomSleepTime = new Random().nextInt((maxSeconds - minSeconds + 1)) + minSeconds;
Thread.sleep(randomSleepTime * 1000L);
// Any arbitrary transactions we want to fetch data for?
try (final Repository repository = RepositoryManager.getRepository()) {
@ -398,6 +401,11 @@ public class ArbitraryDataManager extends Thread {
// Entrypoint to request new metadata from peers
public ArbitraryDataTransactionMetadata fetchMetadata(ArbitraryTransactionData arbitraryTransactionData) {
if (arbitraryTransactionData.getService() == null) {
// Can't fetch metadata without a valid service
return null;
}
ArbitraryDataResource resource = new ArbitraryDataResource(
arbitraryTransactionData.getName(),
ArbitraryDataFile.ResourceIdType.NAME,
@ -489,7 +497,7 @@ public class ArbitraryDataManager extends Thread {
public void invalidateCache(ArbitraryTransactionData arbitraryTransactionData) {
String signature58 = Base58.encode(arbitraryTransactionData.getSignature());
if (arbitraryTransactionData.getName() != null) {
if (arbitraryTransactionData.getName() != null && arbitraryTransactionData.getService() != null) {
String resourceId = arbitraryTransactionData.getName().toLowerCase();
Service service = arbitraryTransactionData.getService();
String identifier = arbitraryTransactionData.getIdentifier();

58
src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java
View File

@ -5,15 +5,11 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.data.transaction.ArbitraryTransactionData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.list.ResourceListManager;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.settings.Settings;
import org.qortal.transaction.Transaction;
import org.qortal.utils.ArbitraryTransactionUtils;
import org.qortal.utils.Base58;
import org.qortal.utils.FilesystemUtils;
import org.qortal.utils.NTP;
import org.qortal.utils.*;
import java.io.IOException;
import java.io.UncheckedIOException;
@ -48,7 +44,6 @@ public class ArbitraryDataStorageManager extends Thread {
private List<ArbitraryTransactionData> hostedTransactions;
private String searchQuery;
private List<ArbitraryTransactionData> searchResultsTransactions;
private static final long DIRECTORY_SIZE_CHECK_INTERVAL = 10 * 60 * 1000L; // 10 minutes
@ -62,6 +57,8 @@ public class ArbitraryDataStorageManager extends Thread {
* This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */
public static final double DELETION_THRESHOLD = 0.98f; // 98%
private static final long PER_NAME_STORAGE_MULTIPLIER = 4L;
public ArbitraryDataStorageManager() {
}
@ -136,11 +133,11 @@ public class ArbitraryDataStorageManager extends Thread {
case ALL:
case VIEWED:
// If the policy includes viewed data, we can host it as long as it's not blocked
return !this.isNameBlocked(name);
return !ListUtils.isNameBlocked(name);
case FOLLOWED:
// If the policy is for followed data only, we have to be following it
return this.isFollowingName(name);
return ListUtils.isFollowingName(name);
// For NONE or all else, we shouldn't host this data
case NONE:
@ -189,14 +186,14 @@ public class ArbitraryDataStorageManager extends Thread {
}
// Never fetch data from blocked names, even if they are followed
if (this.isNameBlocked(name)) {
if (ListUtils.isNameBlocked(name)) {
return false;
}
switch (Settings.getInstance().getStoragePolicy()) {
case FOLLOWED:
case FOLLOWED_OR_VIEWED:
return this.isFollowingName(name);
return ListUtils.isFollowingName(name);
case ALL:
return true;
@ -236,7 +233,7 @@ public class ArbitraryDataStorageManager extends Thread {
* @return boolean - whether the resource is blocked or not
*/
public boolean isBlocked(ArbitraryTransactionData arbitraryTransactionData) {
return isNameBlocked(arbitraryTransactionData.getName());
return ListUtils.isNameBlocked(arbitraryTransactionData.getName());
}
private boolean isDataTypeAllowed(ArbitraryTransactionData arbitraryTransactionData) {
@ -254,22 +251,6 @@ public class ArbitraryDataStorageManager extends Thread {
return true;
}
public boolean isNameBlocked(String name) {
return ResourceListManager.getInstance().listContains("blockedNames", name, false);
}
private boolean isFollowingName(String name) {
return ResourceListManager.getInstance().listContains("followedNames", name, false);
}
public List<String> followedNames() {
return ResourceListManager.getInstance().getStringsInList("followedNames");
}
private int followedNamesCount() {
return ResourceListManager.getInstance().getItemCountForList("followedNames");
}
public List<ArbitraryTransactionData> loadAllHostedTransactions(Repository repository) {
@ -344,11 +325,6 @@ public class ArbitraryDataStorageManager extends Thread {
*/
public List<ArbitraryTransactionData> searchHostedTransactions(Repository repository, String query, Integer limit, Integer offset) {
// Load from results cache if we can (results that exists for the same query), to avoid disk reads
if (this.searchResultsTransactions != null && this.searchQuery.equals(query.toLowerCase())) {
return ArbitraryTransactionUtils.limitOffsetTransactions(this.searchResultsTransactions, limit, offset);
}
// Using cache if we can, to avoid disk reads
if (this.hostedTransactions == null) {
this.hostedTransactions = this.loadAllHostedTransactions(repository);
@ -376,10 +352,7 @@ public class ArbitraryDataStorageManager extends Thread {
// Sort by newest first
searchResultsList.sort(Comparator.comparingLong(ArbitraryTransactionData::getTimestamp).reversed());
// Update cache
this.searchResultsTransactions = searchResultsList;
return ArbitraryTransactionUtils.limitOffsetTransactions(this.searchResultsTransactions, limit, offset);
return ArbitraryTransactionUtils.limitOffsetTransactions(searchResultsList, limit, offset);
}
/**
@ -517,12 +490,17 @@ public class ArbitraryDataStorageManager extends Thread {
return false;
}
if (Settings.getInstance().getStoragePolicy() == StoragePolicy.ALL) {
// Using storage policy ALL, so don't limit anything per name
return true;
}
if (name == null) {
// This transaction doesn't have a name, so fall back to total space limitations
return true;
}
int followedNamesCount = this.followedNamesCount();
int followedNamesCount = ListUtils.followedNamesCount();
if (followedNamesCount == 0) {
// Not following any names, so we have space
return true;
@ -552,14 +530,16 @@ public class ArbitraryDataStorageManager extends Thread {
}
public long storageCapacityPerName(double threshold) {
int followedNamesCount = this.followedNamesCount();
int followedNamesCount = ListUtils.followedNamesCount();
if (followedNamesCount == 0) {
// Not following any names, so we have the total space available
return this.getStorageCapacityIncludingThreshold(threshold);
}
double maxStorageCapacity = (double)this.storageCapacity * threshold;
long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount);
// Some names won't need/use much space, so give all names a 4x multiplier to compensate
long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount) * PER_NAME_STORAGE_MULTIPLIER;
return maxStoragePerName;
}

14
src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java
View File

@ -16,6 +16,7 @@ import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.settings.Settings;
import org.qortal.utils.Base58;
import org.qortal.utils.ListUtils;
import org.qortal.utils.NTP;
import org.qortal.utils.Triple;
@ -101,7 +102,14 @@ public class ArbitraryMetadataManager {
if (metadataFile.exists()) {
// Use local copy
ArbitraryDataTransactionMetadata transactionMetadata = new ArbitraryDataTransactionMetadata(metadataFile.getFilePath());
transactionMetadata.read();
try {
transactionMetadata.read();
} catch (DataException e) {
// Invalid file, so delete it
LOGGER.info("Deleting invalid metadata file due to exception: {}", e.getMessage());
transactionMetadata.delete();
return null;
}
return transactionMetadata;
}
}
@ -332,7 +340,7 @@ public class ArbitraryMetadataManager {
}
// Check if the name is blocked
boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName()));
boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName()));
if (!isBlocked) {
Peer requestingPeer = request.getB();
if (requestingPeer != null) {
@ -420,7 +428,7 @@ public class ArbitraryMetadataManager {
}
// We may need to forward this request on
boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName()));
boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName()));
if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) {
// In relay mode - so ask our other peers if they have it

7
src/main/java/org/qortal/controller/repository/AtStatesPruner.java
View File

@ -39,9 +39,11 @@ public class AtStatesPruner implements Runnable {
try (final Repository repository = RepositoryManager.getRepository()) {
int pruneStartHeight = repository.getATRepository().getAtPruneHeight();
int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.discardChanges();
repository.getATRepository().rebuildLatestAtStates();
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
while (!Controller.isStopping()) {
repository.discardChanges();
@ -91,7 +93,8 @@ public class AtStatesPruner implements Runnable {
if (upperPrunableHeight > upperBatchHeight) {
pruneStartHeight = upperBatchHeight;
repository.getATRepository().setAtPruneHeight(pruneStartHeight);
repository.getATRepository().rebuildLatestAtStates();
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
final int finalPruneStartHeight = pruneStartHeight;

7
src/main/java/org/qortal/controller/repository/AtStatesTrimmer.java
View File

@ -26,9 +26,11 @@ public class AtStatesTrimmer implements Runnable {
try (final Repository repository = RepositoryManager.getRepository()) {
int trimStartHeight = repository.getATRepository().getAtTrimHeight();
int maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.discardChanges();
repository.getATRepository().rebuildLatestAtStates();
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
while (!Controller.isStopping()) {
repository.discardChanges();
@ -69,7 +71,8 @@ public class AtStatesTrimmer implements Runnable {
if (upperTrimmableHeight > upperBatchHeight) {
trimStartHeight = upperBatchHeight;
repository.getATRepository().setAtTrimHeight(trimStartHeight);
repository.getATRepository().rebuildLatestAtStates();
maxLatestAtStatesHeight = PruneManager.getMaxHeightForLatestAtStates(repository);
repository.getATRepository().rebuildLatestAtStates(maxLatestAtStatesHeight);
repository.saveChanges();
final int finalTrimStartHeight = trimStartHeight;

121
src/main/java/org/qortal/controller/repository/BlockArchiveRebuilder.java Normal file
View File

@ -0,0 +1,121 @@
package org.qortal.controller.repository;
import org.apache.commons.io.FileUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.qortal.controller.Controller;
import org.qortal.controller.Synchronizer;
import org.qortal.repository.*;
import org.qortal.settings.Settings;
import org.qortal.transform.TransformationException;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
public class BlockArchiveRebuilder {
private static final Logger LOGGER = LogManager.getLogger(BlockArchiveRebuilder.class);
private final int serializationVersion;
public BlockArchiveRebuilder(int serializationVersion) {
this.serializationVersion = serializationVersion;
}
public void start() throws DataException, IOException {
if (!Settings.getInstance().isArchiveEnabled() || Settings.getInstance().isLite()) {
return;
}
// New archive path is in a different location from original archive path, to avoid conflicts.
// It will be moved later, once the process is complete.
final Path newArchivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive-rebuild");
final Path originalArchivePath = Paths.get(Settings.getInstance().getRepositoryPath(), "archive");
// Delete archive-rebuild if it exists from a previous attempt
FileUtils.deleteDirectory(newArchivePath.toFile());
try (final Repository repository = RepositoryManager.getRepository()) {
int startHeight = 1; // We need to rebuild the entire archive
LOGGER.info("Rebuilding block archive from height {}...", startHeight);
while (!Controller.isStopping()) {
repository.discardChanges();
Thread.sleep(1000L);
// Don't even attempt if we're mid-sync as our repository requests will be delayed for ages
if (Synchronizer.getInstance().isSynchronizing()) {
continue;
}
// Rebuild archive
try {
final int maximumArchiveHeight = BlockArchiveReader.getInstance().getHeightOfLastArchivedBlock();
if (startHeight >= maximumArchiveHeight) {
// We've finished.
// Delete existing archive and move the newly built one into its place
FileUtils.deleteDirectory(originalArchivePath.toFile());
FileUtils.moveDirectory(newArchivePath.toFile(), originalArchivePath.toFile());
BlockArchiveReader.getInstance().invalidateFileListCache();
LOGGER.info("Block archive successfully rebuilt");
return;
}
BlockArchiveWriter writer = new BlockArchiveWriter(startHeight, maximumArchiveHeight, serializationVersion, newArchivePath, repository);
// Set data source to BLOCK_ARCHIVE as we are rebuilding
writer.setDataSource(BlockArchiveWriter.BlockArchiveDataSource.BLOCK_ARCHIVE);
// We can't enforce the 100MB file size target, as the final file needs to contain all blocks
// that exist in the current archive. Otherwise, the final blocks in the archive will be lost.
writer.setShouldEnforceFileSizeTarget(false);
// We want to log the rebuild progress
writer.setShouldLogProgress(true);
BlockArchiveWriter.BlockArchiveWriteResult result = writer.write();
switch (result) {
case OK:
// Increment block archive height
startHeight += writer.getWrittenCount();
repository.saveChanges();
break;
case STOPPING:
return;
// We've reached the limit of the blocks we can archive
// Sleep for a while to allow more to become available
case NOT_ENOUGH_BLOCKS:
// This shouldn't happen, as we're not enforcing minimum file sizes
repository.discardChanges();
throw new DataException("Unable to rebuild archive due to unexpected NOT_ENOUGH_BLOCKS response.");
case BLOCK_NOT_FOUND:
// We tried to archive a block that didn't exist. This is a major failure and likely means
// that a bootstrap or re-sync is needed. Try again every minute until then.
LOGGER.info("Error: block not found when rebuilding archive. If this error persists, " +
"a bootstrap or re-sync may be needed.");
repository.discardChanges();
throw new DataException("Unable to rebuild archive because a block is missing.");
}
} catch (IOException | TransformationException e) {
LOGGER.info("Caught exception when rebuilding block archive", e);
throw new DataException("Unable to rebuild block archive");
}
}
} catch (InterruptedException e) {
// Do nothing
} finally {
// Delete archive-rebuild if it still exists, as that means something went wrong
FileUtils.deleteDirectory(newArchivePath.toFile());
}
}
}

148
src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java
View File

@ -13,7 +13,9 @@ import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.Transaction.TransactionType;
import org.qortal.utils.Unicode;
import java.math.BigInteger;
import java.util.*;
import java.util.stream.Collectors;
public class NamesDatabaseIntegrityCheck {
@ -23,21 +25,14 @@ public class NamesDatabaseIntegrityCheck {
TransactionType.REGISTER_NAME,
TransactionType.UPDATE_NAME,
TransactionType.BUY_NAME,
TransactionType.SELL_NAME
TransactionType.SELL_NAME,
TransactionType.CANCEL_SELL_NAME
);
private List<TransactionData> nameTransactions = new ArrayList<>();
public int rebuildName(String name, Repository repository) {
return this.rebuildName(name, repository, null);
}
public int rebuildName(String name, Repository repository, List<String> referenceNames) {
// "referenceNames" tracks the linked names that have already been rebuilt, to prevent circular dependencies
if (referenceNames == null) {
referenceNames = new ArrayList<>();
}
int modificationCount = 0;
try {
List<TransactionData> transactions = this.fetchAllTransactionsInvolvingName(name, repository);
@ -46,6 +41,14 @@ public class NamesDatabaseIntegrityCheck {
return modificationCount;
}
// If this name has been updated at any point, we need to add transactions from the other names to the sequence
int added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository);
while (added > 0) {
// Keep going until all have been added
LOGGER.trace("{} added for {}. Looking for more transactions...", added, name);
added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository);
}
// Loop through each past transaction and re-apply it to the Names table
for (TransactionData currentTransaction : transactions) {
@ -61,29 +64,14 @@ public class NamesDatabaseIntegrityCheck {
// Process UPDATE_NAME transactions
if (currentTransaction.getType() == TransactionType.UPDATE_NAME) {
UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) currentTransaction;
if (Objects.equals(updateNameTransactionData.getNewName(), name) &&
!Objects.equals(updateNameTransactionData.getName(), updateNameTransactionData.getNewName())) {
// This renames an existing name, so we need to process that instead
if (!referenceNames.contains(name)) {
referenceNames.add(name);
this.rebuildName(updateNameTransactionData.getName(), repository, referenceNames);
}
else {
// We've already processed this name so there's nothing more to do
}
}
else {
Name nameObj = new Name(repository, name);
if (nameObj != null && nameObj.getNameData() != null) {
nameObj.update(updateNameTransactionData);
modificationCount++;
LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name);
} else {
// Something went wrong
throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName()));
}
Name nameObj = new Name(repository, updateNameTransactionData.getName());
if (nameObj != null && nameObj.getNameData() != null) {
nameObj.update(updateNameTransactionData);
modificationCount++;
LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name);
} else {
// Something went wrong
throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName()));
}
}
@ -102,6 +90,21 @@ public class NamesDatabaseIntegrityCheck {
}
}
// Process CANCEL_SELL_NAME transactions
if (currentTransaction.getType() == TransactionType.CANCEL_SELL_NAME) {
CancelSellNameTransactionData cancelSellNameTransactionData = (CancelSellNameTransactionData) currentTransaction;
Name nameObj = new Name(repository, cancelSellNameTransactionData.getName());
if (nameObj != null && nameObj.getNameData() != null) {
nameObj.cancelSell(cancelSellNameTransactionData);
modificationCount++;
LOGGER.trace("Processed CANCEL_SELL_NAME transaction for name {}", name);
}
else {
// Something went wrong
throw new DataException(String.format("Name data not found for name %s", cancelSellNameTransactionData.getName()));
}
}
// Process BUY_NAME transactions
if (currentTransaction.getType() == TransactionType.BUY_NAME) {
BuyNameTransactionData buyNameTransactionData = (BuyNameTransactionData) currentTransaction;
@ -128,7 +131,7 @@ public class NamesDatabaseIntegrityCheck {
public int rebuildAllNames() {
int modificationCount = 0;
try (final Repository repository = RepositoryManager.getRepository()) {
List<String> names = this.fetchAllNames(repository);
List<String> names = this.fetchAllNames(repository); // TODO: de-duplicate, to speed up this process
for (String name : names) {
modificationCount += this.rebuildName(name, repository);
}
@ -326,6 +329,10 @@ public class NamesDatabaseIntegrityCheck {
TransactionType.BUY_NAME, Arrays.asList("name = ?"), Arrays.asList(name));
signatures.addAll(buyNameTransactions);
List<byte[]> cancelSellNameTransactions = repository.getTransactionRepository().getSignaturesMatchingCustomCriteria(
TransactionType.CANCEL_SELL_NAME, Arrays.asList("name = ?"), Arrays.asList(name));
signatures.addAll(cancelSellNameTransactions);
List<TransactionData> transactions = new ArrayList<>();
for (byte[] signature : signatures) {
TransactionData transactionData = repository.getTransactionRepository().fromSignature(signature);
@ -335,8 +342,8 @@ public class NamesDatabaseIntegrityCheck {
}
}
// Sort by lowest timestamp first
transactions.sort(Comparator.comparingLong(TransactionData::getTimestamp));
// Sort by lowest block height first
sortTransactions(transactions);
return transactions;
}
@ -390,8 +397,77 @@ public class NamesDatabaseIntegrityCheck {
names.add(sellNameTransactionData.getName());
}
}
if ((transactionData instanceof CancelSellNameTransactionData)) {
CancelSellNameTransactionData cancelSellNameTransactionData = (CancelSellNameTransactionData) transactionData;
if (!names.contains(cancelSellNameTransactionData.getName())) {
names.add(cancelSellNameTransactionData.getName());
}
}
}
return names;
}
private int addAdditionalTransactionsRelatingToName(List<TransactionData> transactions, String name, Repository repository) throws DataException {
int added = 0;
// If this name has been updated at any point, we need to add transactions from the other names to the sequence
List<String> otherNames = new ArrayList<>();
List<TransactionData> updateNameTransactions = transactions.stream().filter(t -> t.getType() == TransactionType.UPDATE_NAME).collect(Collectors.toList());
for (TransactionData transactionData : updateNameTransactions) {
UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) transactionData;
// If the newName field isn't empty, and either the "name" or "newName" is different from our reference name,
// we should remember this additional name, in case it has relevant transactions associated with it.
if (updateNameTransactionData.getNewName() != null && !updateNameTransactionData.getNewName().isEmpty()) {
if (!Objects.equals(updateNameTransactionData.getName(), name)) {
otherNames.add(updateNameTransactionData.getName());
}
if (!Objects.equals(updateNameTransactionData.getNewName(), name)) {
otherNames.add(updateNameTransactionData.getNewName());
}
}
}
for (String otherName : otherNames) {
List<TransactionData> otherNameTransactions = this.fetchAllTransactionsInvolvingName(otherName, repository);
for (TransactionData otherNameTransactionData : otherNameTransactions) {
if (!transactions.contains(otherNameTransactionData)) {
// Add new transaction relating to other name
transactions.add(otherNameTransactionData);
added++;
}
}
}
if (added > 0) {
// New transaction(s) added, so re-sort
sortTransactions(transactions);
}
return added;
}
private void sortTransactions(List<TransactionData> transactions) {
Collections.sort(transactions, new Comparator() {
public int compare(Object o1, Object o2) {
TransactionData td1 = (TransactionData) o1;
TransactionData td2 = (TransactionData) o2;
// Sort by block height first
int heightComparison = td1.getBlockHeight().compareTo(td2.getBlockHeight());
if (heightComparison != 0) {
return heightComparison;
}
// Same height so compare timestamps
int timestampComparison = Long.compare(td1.getTimestamp(), td2.getTimestamp());
if (timestampComparison != 0) {
return timestampComparison;
}
// Same timestamp so compare signatures
return new BigInteger(td1.getSignature()).compareTo(new BigInteger(td2.getSignature()));
}});
}
}

14
src/main/java/org/qortal/controller/repository/PruneManager.java
View File

@ -157,4 +157,18 @@ public class PruneManager {
return (height < latestUnprunedHeight);
}
/**
* When rebuilding the latest AT states, we need to specify a maxHeight, so that we aren't tracking
* very recent AT states that could potentially be orphaned. This method ensures that AT states
* are given a sufficient number of blocks to confirm before being tracked as a latest AT state.
*/
public static int getMaxHeightForLatestAtStates(Repository repository) throws DataException {
// Get current chain height, and subtract a certain number of "confirmation" blocks
// This is to ensure we are basing our latest AT states data on confirmed blocks -
// ones that won't be orphaned in any normal circumstances
final int confirmationBlocks = 250;
final int chainHeight = repository.getBlockRepository().getBlockchainHeight();
return chainHeight - confirmationBlocks;
}
}

71
src/main/java/org/qortal/controller/tradebot/BitcoinACCTv3TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class BitcoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/DigibyteACCTv3TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class DigibyteACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/DogecoinACCTv1TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class DogecoinACCTv1TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/DogecoinACCTv3TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class DogecoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/LitecoinACCTv3TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class LitecoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/PirateChainACCTv3TradeBot.java
View File

@ -20,6 +20,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -320,20 +321,36 @@ public class PirateChainACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddressT3));
@ -561,15 +578,25 @@ public class PirateChainACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -681,15 +708,25 @@ public class PirateChainACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

71
src/main/java/org/qortal/controller/tradebot/RavencoinACCTv3TradeBot.java
View File

@ -19,6 +19,7 @@ import org.qortal.data.transaction.MessageTransactionData;
import org.qortal.group.Group;
import org.qortal.repository.DataException;
import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.transaction.DeployAtTransaction;
import org.qortal.transaction.MessageTransaction;
import org.qortal.transaction.Transaction.ValidationResult;
@ -317,20 +318,36 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
boolean isMessageAlreadySent = repository.getMessageRepository().exists(tradeBotData.getTradeNativePublicKey(), messageRecipient, messageData);
if (!isMessageAlreadySent) {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
// Do this in a new thread so caller doesn't have to wait for computeNonce()
// In the unlikely event that the transaction doesn't validate then the buy won't happen and eventually Alice's AT will be refunded
new Thread(() -> {
try (final Repository threadsRepository = RepositoryManager.getRepository()) {
PrivateKeyAccount sender = new PrivateKeyAccount(threadsRepository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(threadsRepository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
messageTransaction.computeNonce();
messageTransaction.sign(sender);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
// reset repository state to prevent deadlock
threadsRepository.discardChanges();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
return ResponseResult.NETWORK_ISSUE;
}
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, result.name()));
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: signature invalid", messageRecipient));
}
} catch (DataException e) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to Bob's trade-bot %s: %s", messageRecipient, e.getMessage()));
}
}, "TradeBot response").start();
}
TradeBot.updateTradeBotState(repository, tradeBotData, () -> String.format("Funding P2SH-A %s. Messaged Bob. Waiting for AT-lock", p2shAddress));
@ -548,15 +565,25 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction outgoingMessageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, outgoingMessageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", outgoingMessageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
outgoingMessageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) outgoingMessageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), outgoingMessageTransaction.getPoWDifficulty());
outgoingMessageTransaction.sign(sender);
// reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (outgoingMessageTransaction.isSignatureValid()) {
ValidationResult result = outgoingMessageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}
@ -668,15 +695,25 @@ public class RavencoinACCTv3TradeBot implements AcctTradeBot {
PrivateKeyAccount sender = new PrivateKeyAccount(repository, tradeBotData.getTradePrivateKey());
MessageTransaction messageTransaction = MessageTransaction.build(repository, sender, Group.NO_GROUP, messageRecipient, messageData, false, false);
LOGGER.info("Computing nonce at difficulty {} for AT {} and recipient {}", messageTransaction.getPoWDifficulty(), tradeBotData.getAtAddress(), messageRecipient);
messageTransaction.computeNonce();
MessageTransactionData newMessageTransactionData = (MessageTransactionData) messageTransaction.getTransactionData();
LOGGER.info("Computed nonce {} at difficulty {}", newMessageTransactionData.getNonce(), messageTransaction.getPoWDifficulty());
messageTransaction.sign(sender);
// Reset repository state to prevent deadlock
repository.discardChanges();
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
if (messageTransaction.isSignatureValid()) {
ValidationResult result = messageTransaction.importAsUnconfirmed();
if (result != ValidationResult.OK) {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: %s", messageRecipient, result.name()));
return;
}
}
else {
LOGGER.warn(() -> String.format("Unable to send MESSAGE to AT %s: signature invalid", messageRecipient));
return;
}
}

80
src/main/java/org/qortal/controller/tradebot/TradeBot.java
View File

@ -10,6 +10,7 @@ import org.apache.logging.log4j.Logger;
import org.bitcoinj.core.ECKey;
import org.qortal.account.PrivateKeyAccount;
import org.qortal.api.model.crosschain.TradeBotCreateRequest;
import org.qortal.api.resource.TransactionsResource;
import org.qortal.controller.Controller;
import org.qortal.controller.Synchronizer;
import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult;
@ -19,6 +20,7 @@ import org.qortal.data.at.ATData;
import org.qortal.data.crosschain.CrossChainTradeData;
import org.qortal.data.crosschain.TradeBotData;
import org.qortal.data.network.TradePresenceData;
import org.qortal.data.transaction.TransactionData;
import org.qortal.event.Event;
import org.qortal.event.EventBus;
import org.qortal.event.Listener;
@ -33,6 +35,7 @@ import org.qortal.repository.Repository;
import org.qortal.repository.RepositoryManager;
import org.qortal.repository.hsqldb.HSQLDBImportExport;
import org.qortal.settings.Settings;
import org.qortal.transaction.Transaction;
import org.qortal.utils.ByteArray;
import org.qortal.utils.NTP;
@ -113,6 +116,9 @@ public class TradeBot implements Listener {
private Map<ByteArray, TradePresenceData> safeAllTradePresencesByPubkey = Collections.emptyMap();
private long nextTradePresenceBroadcastTimestamp = 0L;
private Map<String, Long> failedTrades = new HashMap<>();
private Map<String, Long> validTrades = new HashMap<>();
private TradeBot() {
EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event));
}
@ -327,7 +333,7 @@ public class TradeBot implements Listener {
SysTray.getInstance().showMessage("Trade-Bot", String.format("%s: %s", tradeBotData.getAtAddress(), newState), MessageType.INFO);
if (logMessageSupplier != null)
LOGGER.info(logMessageSupplier);
LOGGER.info(logMessageSupplier.get());
LOGGER.debug(() -> String.format("new state for trade-bot entry based on AT %s: %s", tradeBotData.getAtAddress(), newState));
@ -674,6 +680,78 @@ public class TradeBot implements Listener {
});
}
/** Removes any trades that have had multiple failures */
public List<CrossChainTradeData> removeFailedTrades(Repository repository, List<CrossChainTradeData> crossChainTrades) {
Long now = NTP.getTime();
if (now == null) {
return crossChainTrades;
}
List<CrossChainTradeData> updatedCrossChainTrades = new ArrayList<>(crossChainTrades);
int getMaxTradeOfferAttempts = Settings.getInstance().getMaxTradeOfferAttempts();
for (CrossChainTradeData crossChainTradeData : crossChainTrades) {
// We only care about trades in the OFFERING state
if (crossChainTradeData.mode != AcctMode.OFFERING) {
failedTrades.remove(crossChainTradeData.qortalAtAddress);
validTrades.remove(crossChainTradeData.qortalAtAddress);
continue;
}
// Return recently cached values if they exist
Long failedTimestamp = failedTrades.get(crossChainTradeData.qortalAtAddress);
if (failedTimestamp != null && now - failedTimestamp < 60 * 60 * 1000L) {
updatedCrossChainTrades.remove(crossChainTradeData);
//LOGGER.info("Removing cached failed trade AT {}", crossChainTradeData.qortalAtAddress);
continue;
}
Long validTimestamp = validTrades.get(crossChainTradeData.qortalAtAddress);
if (validTimestamp != null && now - validTimestamp < 60 * 60 * 1000L) {
//LOGGER.info("NOT removing cached valid trade AT {}", crossChainTradeData.qortalAtAddress);
continue;
}
try {
List<byte[]> signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, Arrays.asList(Transaction.TransactionType.MESSAGE), null, null, crossChainTradeData.qortalCreatorTradeAddress, TransactionsResource.ConfirmationStatus.CONFIRMED, null, null, null);
if (signatures.size() < getMaxTradeOfferAttempts) {
// Less than 3 (or user-specified number of) MESSAGE transactions relate to this trade, so assume it is ok
validTrades.put(crossChainTradeData.qortalAtAddress, now);
continue;
}
List<TransactionData> transactions = new ArrayList<>(signatures.size());
for (byte[] signature : signatures) {
transactions.add(repository.getTransactionRepository().fromSignature(signature));
}
transactions.sort(Transaction.getDataComparator());
// Get timestamp of the first MESSAGE transaction
long firstMessageTimestamp = transactions.get(0).getTimestamp();
// Treat as failed if first buy attempt was more than 60 mins ago (as it's still in the OFFERING state)
boolean isFailed = (now - firstMessageTimestamp > 60*60*1000L);
if (isFailed) {
failedTrades.put(crossChainTradeData.qortalAtAddress, now);
updatedCrossChainTrades.remove(crossChainTradeData);
}
else {
validTrades.put(crossChainTradeData.qortalAtAddress, now);
}
} catch (DataException e) {
LOGGER.info("Unable to determine failed state of AT {}", crossChainTradeData.qortalAtAddress);
continue;
}
}
return updatedCrossChainTrades;
}
public boolean isFailedTrade(Repository repository, CrossChainTradeData crossChainTradeData) {
List<CrossChainTradeData> results = removeFailedTrades(repository, Arrays.asList(crossChainTradeData));
return results.isEmpty();
}
private long generateExpiry(long timestamp) {
return ((timestamp - 1) / EXPIRY_ROUNDING) * EXPIRY_ROUNDING + PRESENCE_LIFETIME;
}

54
src/main/java/org/qortal/crosschain/Bitcoin.java
View File

@ -49,6 +49,7 @@ public class Bitcoin extends Bitcoiny {
//CLOSED new Server("bitcoin.grey.pw", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("btc.litepay.ch", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("electrum.pabu.io", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("electrumx.dev", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("electrumx.hodlwallet.com", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("gd42.org", Server.ConnectionType.SSL, 50002),
//CLOSED new Server("korea.electrum-server.com", Server.ConnectionType.SSL, 50002),
@ -56,28 +57,75 @@ public class Bitcoin extends Bitcoiny {
//1.15.0 new Server("alviss.coinjoined.com", Server.ConnectionType.SSL, 50002),
//1.15.0 new Server("electrum.acinq.co", Server.ConnectionType.SSL, 50002),
//1.14.0 new Server("electrum.coinext.com.br", Server.ConnectionType.SSL, 50002),
//F1.7.0 new Server("btc.lastingcoin.net", Server.ConnectionType.SSL, 50002),
new Server("104.248.139.211", Server.ConnectionType.SSL, 50002),
new Server("128.0.190.26", Server.ConnectionType.SSL, 50002),
new Server("142.93.6.38", Server.ConnectionType.SSL, 50002),
new Server("157.245.172.236", Server.ConnectionType.SSL, 50002),
new Server("167.172.226.175", Server.ConnectionType.SSL, 50002),
new Server("167.172.42.31", Server.ConnectionType.SSL, 50002),
new Server("178.62.80.20", Server.ConnectionType.SSL, 50002),
new Server("185.64.116.15", Server.ConnectionType.SSL, 50002),
new Server("188.165.206.215", Server.ConnectionType.SSL, 50002),
new Server("188.165.211.112", Server.ConnectionType.SSL, 50002),
new Server("2azzarita.hopto.org", Server.ConnectionType.SSL, 50002),
new Server("2electrumx.hopto.me", Server.ConnectionType.SSL, 56022),
new Server("2ex.digitaleveryware.com", Server.ConnectionType.SSL, 50002),
new Server("65.39.140.37", Server.ConnectionType.SSL, 50002),
new Server("68.183.188.105", Server.ConnectionType.SSL, 50002),
new Server("71.73.14.254", Server.ConnectionType.SSL, 50002),
new Server("94.23.247.135", Server.ConnectionType.SSL, 50002),
new Server("assuredly.not.fyi", Server.ConnectionType.SSL, 50002),
new Server("ax101.blockeng.ch", Server.ConnectionType.SSL, 50002),
new Server("ax102.blockeng.ch", Server.ConnectionType.SSL, 50002),
new Server("b.1209k.com", Server.ConnectionType.SSL, 50002),
new Server("b6.1209k.com", Server.ConnectionType.SSL, 50002),
new Server("bitcoin.dermichi.com", Server.ConnectionType.SSL, 50002),
new Server("bitcoin.lu.ke", Server.ConnectionType.SSL, 50002),
new Server("bitcoin.lukechilds.co", Server.ConnectionType.SSL, 50002),
new Server("blkhub.net", Server.ConnectionType.SSL, 50002),
new Server("btc.lastingcoin.net", Server.ConnectionType.SSL, 50002),
new Server("btc.electroncash.dk", Server.ConnectionType.SSL, 60002),
new Server("btc.ocf.sh", Server.ConnectionType.SSL, 50002),
new Server("btce.iiiiiii.biz", Server.ConnectionType.SSL, 50002),
new Server("caleb.vegas", Server.ConnectionType.SSL, 50002),
new Server("eai.coincited.net", Server.ConnectionType.SSL, 50002),
new Server("electrum.bhoovd.com", Server.ConnectionType.SSL, 50002),
new Server("electrum.bitaroo.net", Server.ConnectionType.SSL, 50002),
new Server("electrumx.dev", Server.ConnectionType.SSL, 50002),
new Server("electrum.bitcoinlizard.net", Server.ConnectionType.SSL, 50002),
new Server("electrum.blockstream.info", Server.ConnectionType.SSL, 50002),
new Server("electrum.emzy.de", Server.ConnectionType.SSL, 50002),
new Server("electrum.exan.tech", Server.ConnectionType.SSL, 50002),
new Server("electrum.kendigisland.xyz", Server.ConnectionType.SSL, 50002),
new Server("electrum.mmitech.info", Server.ConnectionType.SSL, 50002),
new Server("electrum.petrkr.net", Server.ConnectionType.SSL, 50002),
new Server("electrum.stippy.com", Server.ConnectionType.SSL, 50002),
new Server("electrum.thomasfischbach.de", Server.ConnectionType.SSL, 50002),
new Server("electrum0.snel.it", Server.ConnectionType.SSL, 50002),
new Server("electrum1.cipig.net", Server.ConnectionType.SSL, 50002),
new Server("electrum2.cipig.net", Server.ConnectionType.SSL, 50002),
new Server("electrum3.cipig.net", Server.ConnectionType.SSL, 50002),
new Server("electrumx.alexridevski.net", Server.ConnectionType.SSL, 50002),
new Server("electrumx-core.1209k.com", Server.ConnectionType.SSL, 50002),
new Server("elx.bitske.com", Server.ConnectionType.SSL, 50002),
new Server("ex03.axalgo.com", Server.ConnectionType.SSL, 50002),
new Server("ex05.axalgo.com", Server.ConnectionType.SSL, 50002),
new Server("ex07.axalgo.com", Server.ConnectionType.SSL, 50002),
new Server("fortress.qtornado.com", Server.ConnectionType.SSL, 50002),
new Server("fulcrum.grey.pw", Server.ConnectionType.SSL, 50002),
new Server("fulcrum.sethforprivacy.com", Server.ConnectionType.SSL, 51002),
new Server("guichet.centure.cc", Server.ConnectionType.SSL, 50002),
new Server("kareoke.qoppa.org", Server.ConnectionType.SSL, 50002),
new Server("hodlers.beer", Server.ConnectionType.SSL, 50002),
new Server("kareoke.qoppa.org", Server.ConnectionType.SSL, 50002),
new Server("kirsche.emzy.de", Server.ConnectionType.SSL, 50002),
new Server("node1.btccuracao.com", Server.ConnectionType.SSL, 50002),
new Server("osr1ex1.compumundohipermegared.one", Server.ConnectionType.SSL, 50002),
new Server("smmalis37.ddns.net", Server.ConnectionType.SSL, 50002),
new Server("ulrichard.ch", Server.ConnectionType.SSL, 50002),
new Server("vmd104012.contaboserver.net", Server.ConnectionType.SSL, 50002),
new Server("vmd104014.contaboserver.net", Server.ConnectionType.SSL, 50002),
new Server("vmd63185.contaboserver.net", Server.ConnectionType.SSL, 50002),
new Server("vmd71287.contaboserver.net", Server.ConnectionType.SSL, 50002),
new Server("vmd84592.contaboserver.net", Server.ConnectionType.SSL, 50002),
new Server("xtrum.com", Server.ConnectionType.SSL, 50002));
}

106
src/main/java/org/qortal/crosschain/Bitcoiny.java
View File

@ -167,6 +167,16 @@ public abstract class Bitcoiny implements ForeignBlockchain {
return blockTimestamps.get(5);
}
/**
* Returns height from latest block.
* <p>
* @throws ForeignBlockchainException if error occurs
*/
public int getBlockchainHeight() throws ForeignBlockchainException {
int height = this.blockchainProvider.getCurrentHeight();
return height;
}
/** Returns fee per transaction KB. To be overridden for testnet/regtest. */
public Coin getFeePerKb() {
return this.bitcoinjContext.getFeePerKb();
@ -357,19 +367,33 @@ public abstract class Bitcoiny implements ForeignBlockchain {
* @return unspent BTC balance, or null if unable to determine balance
*/
public Long getWalletBalance(String key58) throws ForeignBlockchainException {
// It's more accurate to calculate the balance from the transactions, rather than asking Bitcoinj
return this.getWalletBalanceFromTransactions(key58);
Long balance = 0L;
// Context.propagate(bitcoinjContext);
//
// Wallet wallet = walletFromDeterministicKey58(key58);
// wallet.setUTXOProvider(new WalletAwareUTXOProvider(this, wallet));
//
// Coin balance = wallet.getBalance();
// if (balance == null)
// return null;
//
// return balance.value;
List<TransactionOutput> allUnspentOutputs = new ArrayList<>();
Set<String> walletAddresses = this.getWalletAddresses(key58);
for (String address : walletAddresses) {
allUnspentOutputs.addAll(this.getUnspentOutputs(address));
}
for (TransactionOutput output : allUnspentOutputs) {
if (!output.isAvailableForSpending()) {
continue;
}
balance += output.getValue().value;
}
return balance;
}
public Long getWalletBalanceFromBitcoinj(String key58) {
Context.propagate(bitcoinjContext);
Wallet wallet = walletFromDeterministicKey58(key58);
wallet.setUTXOProvider(new WalletAwareUTXOProvider(this, wallet));
Coin balance = wallet.getBalance();
if (balance == null)
return null;
return balance.value;
}
public Long getWalletBalanceFromTransactions(String key58) throws ForeignBlockchainException {
@ -464,6 +488,64 @@ public abstract class Bitcoiny implements ForeignBlockchain {
}
}
public Set<String> getWalletAddresses(String key58) throws ForeignBlockchainException {
synchronized (this) {
Context.propagate(bitcoinjContext);
Wallet wallet = walletFromDeterministicKey58(key58);
DeterministicKeyChain keyChain = wallet.getActiveKeyChain();
keyChain.setLookaheadSize(Bitcoiny.WALLET_KEY_LOOKAHEAD_INCREMENT);
keyChain.maybeLookAhead();
List<DeterministicKey> keys = new ArrayList<>(keyChain.getLeafKeys());
Set<String> keySet = new HashSet<>();
int unusedCounter = 0;
int ki = 0;
do {
boolean areAllKeysUnused = true;
for (; ki < keys.size(); ++ki) {
DeterministicKey dKey = keys.get(ki);
// Check for transactions
Address address = Address.fromKey(this.params, dKey, ScriptType.P2PKH);
keySet.add(address.toString());
byte[] script = ScriptBuilder.createOutputScript(address).getProgram();
// Ask for transaction history - if it's empty then key has never been used
List<TransactionHash> historicTransactionHashes = this.getAddressTransactions(script, false);
if (!historicTransactionHashes.isEmpty()) {
areAllKeysUnused = false;
}
}
if (areAllKeysUnused) {
// No transactions
if (unusedCounter >= Settings.getInstance().getGapLimit()) {
// ... and we've hit our search limit
break;
}
// We haven't hit our search limit yet so increment the counter and keep looking
unusedCounter += WALLET_KEY_LOOKAHEAD_INCREMENT;
} else {
// Some keys in this batch were used, so reset the counter
unusedCounter = 0;
}
// Generate some more keys
keys.addAll(generateMoreKeys(keyChain));
// Process new keys
} while (true);
return keySet;
}
}
protected SimpleTransaction convertToSimpleTransaction(BitcoinyTransaction t, Set<String> keySet) {
long amount = 0;
long total = 0L;

Some files were not shown because too many files have changed in this diff Show More