diff --git a/Q-Apps.md b/Q-Apps.md new file mode 100644 index 00000000..ca750e7d --- /dev/null +++ b/Q-Apps.md @@ -0,0 +1,910 @@ +# Qortal Project - Q-Apps Documentation + +## Introduction + +Q-Apps are static web apps written in javascript, HTML, CSS, and other static assets. The key difference between a Q-App and a fully static site is its ability to interact with both the logged-in user and on-chain data. This is achieved using the API described in this document. + + +# Section 0: Basic QDN concepts + +## Introduction to QDN resources +Each published item on QDN (Qortal Data Network) is referred to as a "resource". A resource could contain anything from a few characters of text, to a multi-layered directory structure containing thousands of files. + +Resources are stored on-chain, however the data payload is generally stored off-chain, and verified using an on-chain SHA-256 hash. + +To publish a resource, a user must first register a name, and then include that name when publishing the data. Accounts without a registered name are unable to publish to QDN from a Q-App at this time. + +Owning the name grants update privileges to the data. If that name is later sold or transferred, the permission to update that resource is moved to the new owner. + + +## Name, service & identifier + +Each QDN resource has 3 important fields: +- `name` - the registered name of the account that is publishing the data (which will hold update/edit privileges going forwards).

+- `service` - the type of content (e.g. IMAGE or JSON). Different services have different validation rules. See [list of available services](#services).

+- `identifier` - an optional string to allow more than one resource to exist for a given name/service combination. For example, the name `QortalDemo` may wish to publish multiple images. This can be achieved by using a different identifier string for each. The identifier is only unique to the name in question, and so it doesn't matter if another name is using the same service and identifier string. + + +## Shared identifiers + +Since an identifier can be used by multiple names, this can be used to the advantage of Q-App developers as it allows for data to be stored in a deterministic location. + +An example of this is the user's avatar. This will always be published with service `THUMBNAIL` and identifier `qortal_avatar`, along with the user's name. So, an app can display the avatar of a user just by specifying their name when requesting the data. The same applies when publishing data. + + +## "Default" resources + +A "default" resource refers to one without an identifier. For example, when a website is published via the UI, it will use the user's name and the service `WEBSITE`. These do not have an identifier, and are therefore the "default" website for this name. When requesting or publishing data without an identifier, apps can either omit the `identifier` key entirely, or include `"identifier": "default"` to indicate that the resource(s) being queried or published do not have an identifier. + + + +## Available service types + +Here is a list of currently available services that can be used in Q-Apps: + +### Public services ### +The services below are intended to be used for publicly accessible data. + +IMAGE, +THUMBNAIL, +VIDEO, +AUDIO, +PODCAST, +VOICE, +ARBITRARY_DATA, +JSON, +DOCUMENT, +LIST, +PLAYLIST, +METADATA, +BLOG, +BLOG_POST, +BLOG_COMMENT, +GIF_REPOSITORY, +ATTACHMENT, +FILE, +FILES, +CHAIN_DATA, +STORE, +PRODUCT, +OFFER, +COUPON, +CODE, +PLUGIN, +EXTENSION, +GAME, +ITEM, +NFT, +DATABASE, +SNAPSHOT, +COMMENT, +CHAIN_COMMENT, +WEBSITE, +APP, +QCHAT_ATTACHMENT, +QCHAT_IMAGE, +QCHAT_AUDIO, +QCHAT_VOICE + +### Private services ### +For the services below, data is encrypted for a single recipient, and can only be decrypted using the private key of the recipient's wallet. + +QCHAT_ATTACHMENT_PRIVATE, +ATTACHMENT_PRIVATE, +FILE_PRIVATE, +IMAGE_PRIVATE, +VIDEO_PRIVATE, +AUDIO_PRIVATE, +VOICE_PRIVATE, +DOCUMENT_PRIVATE, +MAIL_PRIVATE, +MESSAGE_PRIVATE + + +## Single vs multi-file resources + +Some resources, such as those published with the `IMAGE` or `JSON` service, consist of a single file or piece of data (the image or the JSON string). This is the most common type of QDN resource, especially in the context of Q-Apps. These can be published by supplying a base64-encoded string containing the data. + +Other resources, such as those published with the `WEBSITE`, `APP`, or `GIF_REPOSITORY` service, can contain multiple files and directories. Publishing these kinds of files is not yet available for Q-Apps, however it is possible to retrieve multi-file resources that are already published. When retrieving this data (via FETCH_QDN_RESOURCE), a `filepath` must be included to indicate the file that you would like to retrieve. There is no need to specify a filepath for single file resources, as these will automatically return the contents of the single file. + + +## App-specific data + +Some apps may want to make all QDN data for a particular service available. However, others may prefer to only deal with data that has been published by their app (if a specific format/schema is being used for instance). + +Identifiers can be used to allow app developers to locate data that has been published by their app. The recommended approach for this is to use the app name as a prefix on all identifiers when publishing data. + +For instance, an app called `MyApp` could allow users to publish JSON data. The app could choose to prefix all identifiers with the string `myapp_`, and then use a random string for each published resource (resulting in identifiers such as `myapp_qR5ndZ8v`). Then, to locate data that has potentially been published by users of MyApp, it can later search the QDN database for items with `"service": "JSON"` and `"identifier": "myapp_"`. The SEARCH_QDN_RESOURCES action has a `prefix` option in order to match identifiers beginning with the supplied string. + +Note that QDN is a permissionless system, and therefore it's not possible to verify that a resource was actually published by the app. It is recommended that apps validate the contents of the resource to ensure it is formatted correctly, instead of making assumptions. + + +## Updating a resource + +To update a resource, it can be overwritten by publishing with the same `name`, `service`, and `identifier` combination. Note that the authenticated account must currently own the name in order to publish an update. + + +## Routing + +If a non-existent `filepath` is accessed, the default behaviour of QDN is to return a `404: File not found` error. This includes anything published using the `WEBSITE` service. + +However, routing is handled differently for anything published using the `APP` service. + +For apps, QDN automatically sends all unhandled requests to the index file (generally index.html). This allows the app to use custom routing, as it is able to listen on any path. If a file exists at a path, the file itself will be served, and so the request won't be sent to the index file. + +It's recommended that all apps return a 404 page if a request isn't able to be routed. + + +# Section 1: Simple links and image loading via HTML + +## Section 1a: Linking to other QDN websites / resources + +The `qortal://` protocol can be used to access QDN data from within Qortal websites and apps. The basic format is as follows: +``` +link text +``` + +However, the system will support the omission of the `identifier` and/or `path` components to allow for simpler URL formats. + +A simple link to another website can be achieved with this HTML code: +``` +link text +``` + +To link to a specific page of another website: +``` +link text +``` + +To link to a standalone resource, such as an avatar +``` +avatar +``` + +For cases where you would prefer to explicitly include an identifier (to remove ambiguity) you can use the keyword `default` to access a resource that doesn't have an identifier. For instance: +``` +link to root of website +link to subpage of website +``` + + +## Section 1b: Linking to other QDN images + +The same applies for images, such as displaying an avatar: +``` + +``` + +...or even an image from an entirely different website: +``` + +``` + + +# Section 2: Integrating a Javascript app + +Javascript apps allow for much more complex integrations with Qortal's blockchain data. + +## Section 2a: Direct API calls + +The standard [Qortal Core API](http://localhost:12391/api-documentation) is available to websites and apps, and can be called directly using a standard AJAX request, such as: +``` +async function getNameInfo(name) { + const response = await fetch("/names/" + name); + const nameData = await response.json(); + console.log("nameData: " + JSON.stringify(nameData)); +} +getNameInfo("QortalDemo"); +``` + +However, this only works for read-only data, such as looking up transactions, names, balances, etc. Also, since the address of the logged in account can't be retrieved from the core, apps can't show personalized data with this approach. + + +## Section 2b: User interaction via qortalRequest() + +To take things a step further, the qortalRequest() function can be used to interact with the user, in order to: + +- Request address and public key of the logged in account +- Publish data to QDN +- Send chat messages +- Join groups +- Deploy ATs (smart contracts) +- Send QORT or any supported foreign coin +- Add/remove items from lists + +In addition to the above, qortalRequest() also supports many read-only functions that are also available via direct core API calls. Using qortalRequest() helps with futureproofing, as the core APIs can be modified without breaking functionality of existing Q-Apps. + + +### Making a request + +Qortal core will automatically inject the `qortalRequest()` javascript function to all websites/apps, which returns a Promise. This can be used to fetch data or publish data to the Qortal blockchain. This functionality supports async/await, as well as try/catch error handling. + +``` +async function myfunction() { + try { + let res = await qortalRequest({ + action: "GET_ACCOUNT_DATA", + address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2" + }); + console.log(JSON.stringify(res)); // Log the response to the console + + } catch(e) { + console.log("Error: " + JSON.stringify(e)); + } +} +myfunction(); +``` + +### Timeouts + +Request timeouts are handled automatically when using qortalRequest(). The timeout value will differ based on the action being used - see `getDefaultTimeout()` in [q-apps.js](src/main/resources/q-apps/q-apps.js) for the current values. + +If a request times out it will throw an error - `The request timed out` - which can be handled by the Q-App. + +It is also possible to specify a custom timeout using `qortalRequestWithTimeout(request, timeout)`, however this is discouraged. It's more reliable and futureproof to let the core handle the timeout values. + + +# Section 3: qortalRequest Documentation + +## Supported actions + +Here is a list of currently supported actions: +- GET_USER_ACCOUNT +- GET_ACCOUNT_DATA +- GET_ACCOUNT_NAMES +- SEARCH_NAMES +- GET_NAME_DATA +- LIST_QDN_RESOURCES +- SEARCH_QDN_RESOURCES +- GET_QDN_RESOURCE_STATUS +- GET_QDN_RESOURCE_PROPERTIES +- GET_QDN_RESOURCE_METADATA +- GET_QDN_RESOURCE_URL +- LINK_TO_QDN_RESOURCE +- FETCH_QDN_RESOURCE +- PUBLISH_QDN_RESOURCE +- PUBLISH_MULTIPLE_QDN_RESOURCES +- DECRYPT_DATA +- SAVE_FILE +- GET_WALLET_BALANCE +- GET_BALANCE +- SEND_COIN +- SEARCH_CHAT_MESSAGES +- SEND_CHAT_MESSAGE +- LIST_GROUPS +- JOIN_GROUP +- DEPLOY_AT +- GET_AT +- GET_AT_DATA +- LIST_ATS +- FETCH_BLOCK +- FETCH_BLOCK_RANGE +- SEARCH_TRANSACTIONS +- GET_PRICE +- GET_LIST_ITEMS +- ADD_LIST_ITEMS +- DELETE_LIST_ITEM + +More functionality will be added in the future. + +## Example Requests + +Here are some example requests for each of the above: + +### Get address of logged in account +_Will likely require user approval_ +``` +let account = await qortalRequest({ + action: "GET_USER_ACCOUNT" +}); +let address = account.address; +``` + +### Get public key of logged in account +_Will likely require user approval_ +``` +let pubkey = await qortalRequest({ + action: "GET_USER_ACCOUNT" +}); +let publicKey = account.publicKey; +``` + +### Get account data +``` +let res = await qortalRequest({ + action: "GET_ACCOUNT_DATA", + address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2" +}); +``` + +### Get names owned by account +``` +let res = await qortalRequest({ + action: "GET_ACCOUNT_NAMES", + address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2" +}); +``` + +### Search names +``` +let res = await qortalRequest({ + action: "SEARCH_NAMES", + query: "search query goes here", + prefix: false, // Optional - if true, only the beginning of the name is matched + limit: 100, + offset: 0, + reverse: false +}); +``` + +### Get name data +``` +let res = await qortalRequest({ + action: "GET_NAME_DATA", + name: "QortalDemo" +}); +``` + + +### List QDN resources +``` +let res = await qortalRequest({ + action: "LIST_QDN_RESOURCES", + service: "THUMBNAIL", + name: "QortalDemo", // Optional (exact match) + identifier: "qortal_avatar", // Optional (exact match) + default: true, // Optional + includeStatus: false, // Optional - will take time to respond, so only request if necessary + includeMetadata: false, // Optional - will take time to respond, so only request if necessary + followedOnly: false, // Optional - include followed names only + excludeBlocked: false, // Optional - exclude blocked content + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Search QDN resources +``` +let res = await qortalRequest({ + action: "SEARCH_QDN_RESOURCES", + service: "THUMBNAIL", + query: "search query goes here", // Optional - searches both "identifier" and "name" fields + identifier: "search query goes here", // Optional - searches only the "identifier" field + name: "search query goes here", // Optional - searches only the "name" field + prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters + exactMatchNames: true, // Optional - if true, partial name matches are excluded + default: false, // Optional - if true, only resources without identifiers are returned + includeStatus: false, // Optional - will take time to respond, so only request if necessary + includeMetadata: false, // Optional - will take time to respond, so only request if necessary + nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list + followedOnly: false, // Optional - include followed names only + excludeBlocked: false, // Optional - exclude blocked content + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Search QDN resources (multiple names) +``` +let res = await qortalRequest({ + action: "SEARCH_QDN_RESOURCES", + service: "THUMBNAIL", + query: "search query goes here", // Optional - searches both "identifier" and "name" fields + identifier: "search query goes here", // Optional - searches only the "identifier" field + names: ["QortalDemo", "crowetic", "AlphaX"], // Optional - searches only the "name" field for any of the supplied names + prefix: false, // Optional - if true, only the beginning of fields are matched in all of the above filters + default: false, // Optional - if true, only resources without identifiers are returned + includeStatus: false, // Optional - will take time to respond, so only request if necessary + includeMetadata: false, // Optional - will take time to respond, so only request if necessary + nameListFilter: "QApp1234Subscriptions", // Optional - will only return results if they are from a name included in supplied list + followedOnly: false, // Optional - include followed names only + excludeBlocked: false, // Optional - exclude blocked content + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Fetch QDN single file resource +``` +let res = await qortalRequest({ + action: "FETCH_QDN_RESOURCE", + name: "QortalDemo", + service: "THUMBNAIL", + identifier: "qortal_avatar", // Optional. If omitted, the default resource is returned, or you can alternatively use the keyword "default" + encoding: "base64", // Optional. If omitted, data is returned in raw form + rebuild: false +}); +``` + +### Fetch file from multi file QDN resource +Data is returned in the base64 format +``` +let res = await qortalRequest({ + action: "FETCH_QDN_RESOURCE", + name: "QortalDemo", + service: "WEBSITE", + identifier: "default", // Optional. If omitted, the default resource is returned, or you can alternatively request that using the keyword "default", as shown here + filepath: "index.html", // Required only for resources containing more than one file + rebuild: false +}); +``` + +### Get QDN resource status +``` +let res = await qortalRequest({ + action: "GET_QDN_RESOURCE_STATUS", + name: "QortalDemo", + service: "THUMBNAIL", + identifier: "qortal_avatar", // Optional + build: true // Optional - request that the resource is fetched & built in the background +}); +``` + +### Get QDN resource properties +``` +let res = await qortalRequest({ + action: "GET_QDN_RESOURCE_PROPERTIES", + name: "QortalDemo", + service: "THUMBNAIL", + identifier: "qortal_avatar" // Optional +}); +// Returns: filename, size, mimeType (where available) +``` + +### Get QDN resource metadata +``` +let res = await qortalRequest({ + action: "GET_QDN_RESOURCE_METADATA", + name: "QortalDemo", + service: "THUMBNAIL", + identifier: "qortal_avatar" // Optional +}); +``` + +### Publish a single file to QDN +_Requires user approval_.
+Note: this publishes a single, base64-encoded file. Multi-file resource publishing (such as a WEBSITE or GIF_REPOSITORY) is not yet supported via a Q-App. It will be added in a future update. +``` +let res = await qortalRequest({ + action: "PUBLISH_QDN_RESOURCE", + name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list + service: "IMAGE", + identifier: "myapp-image1234" // Optional + data64: "base64_encoded_data", + // filename: "image.jpg", // Optional - to help apps determine the file's type + // title: "Title", // Optional + // description: "Description", // Optional + // category: "TECHNOLOGY", // Optional + // tag1: "any", // Optional + // tag2: "strings", // Optional + // tag3: "can", // Optional + // tag4: "go", // Optional + // tag5: "here", // Optional + // encrypt: true, // Optional - to be used with a private service + // recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true +}); +``` + +### Publish multiple resources at once to QDN +_Requires user approval_.
+Note: each resource being published consists of a single, base64-encoded file, each in its own transaction. Useful for publishing two or more related things, such as a video and a video thumbnail. +``` +let res = await qortalRequest({ + action: "PUBLISH_MULTIPLE_QDN_RESOURCES", + resources: [ + name: "Demo", // Publisher must own the registered name - use GET_ACCOUNT_NAMES for a list + service: "IMAGE", + identifier: "myapp-image1234" // Optional + data64: "base64_encoded_data", + // filename: "image.jpg", // Optional - to help apps determine the file's type + // title: "Title", // Optional + // description: "Description", // Optional + // category: "TECHNOLOGY", // Optional + // tag1: "any", // Optional + // tag2: "strings", // Optional + // tag3: "can", // Optional + // tag4: "go", // Optional + // tag5: "here", // Optional + // encrypt: true, // Optional - to be used with a private service + // recipientPublicKey: "publickeygoeshere" // Only required if `encrypt` is set to true + ], + [ + ... more resources here if needed ... + ] +}); +``` + +### Decrypt encrypted/private data +``` +let res = await qortalRequest({ + action: "DECRYPT_DATA", + encryptedData: 'qortalEncryptedDatabMx4fELNTV+ifJxmv4+GcuOIJOTo+3qAvbWKNY2L1r', + publicKey: 'publickeygoeshere' +}); +// Returns base64 encoded string of plaintext data +``` + +### Prompt user to save a file to disk +Note: mimeType not required but recommended. If not specified, saving will fail if the mimeType is unable to be derived from the Blob. +``` +let res = await qortalRequest({ + action: "SAVE_FILE", + blob: dataBlob, + filename: "myfile.pdf", + mimeType: "application/pdf" // Optional but recommended +}); +``` + + +### Get wallet balance (QORT) +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "GET_WALLET_BALANCE", + coin: "QORT" +}); +``` + + +### Get address or asset balance +``` +let res = await qortalRequest({ + action: "GET_BALANCE", + address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2" +}); +``` +``` +let res = await qortalRequest({ + action: "GET_BALANCE", + assetId: 1, + address: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2" +}); +``` + +### Send QORT to address +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "SEND_COIN", + coin: "QORT", + destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2", + amount: 1.00000000 // 1 QORT +}); +``` + +### Send foreign coin to address +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "SEND_COIN", + coin: "LTC", + destinationAddress: "LSdTvMHRm8sScqwCi6x9wzYQae8JeZhx6y", + amount: 1.00000000, // 1 LTC + fee: 0.00000020 // fee per byte +}); +``` + +### Search or list chat messages +``` +let res = await qortalRequest({ + action: "SEARCH_CHAT_MESSAGES", + before: 999999999999999, + after: 0, + txGroupId: 0, // Optional (must specify either txGroupId or two involving addresses) + // involving: ["QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2", "QSefrppsDCsZebcwrqiM1gNbWq7YMDXtG2"], // Optional (must specify either txGroupId or two involving addresses) + // reference: "reference", // Optional + // chatReference: "chatreference", // Optional + // hasChatReference: true, // Optional + encoding: "BASE64", // Optional (defaults to BASE58 if omitted) + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Send a group chat message +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "SEND_CHAT_MESSAGE", + groupId: 0, + message: "Test" +}); +``` + +### Send a private chat message +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "SEND_CHAT_MESSAGE", + destinationAddress: "QZLJV7wbaFyxaoZQsjm6rb9MWMiDzWsqM2", + message: "Test" +}); +``` + +### List groups +``` +let res = await qortalRequest({ + action: "LIST_GROUPS", + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Join a group +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "JOIN_GROUP", + groupId: 100 +}); +``` + + +### Deploy an AT +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "DEPLOY_AT", + creationBytes: "12345", // Must be Base58 encoded + name: "test name", + description: "test description", + type: "test type", + tags: "test tags", + amount: 1.00000000, // 1 QORT + assetId: 0, + // fee: 0.002 // optional - will use default fee if excluded +}); +``` + +### Get AT info +``` +let res = await qortalRequest({ + action: "GET_AT", + atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH" +}); +``` + +### Get AT data bytes (base58 encoded) +``` +let res = await qortalRequest({ + action: "GET_AT_DATA", + atAddress: "ASRUsCjk6fa5bujv3oWYmWaVqNtvxydpPH" +}); +``` + +### List ATs by functionality +``` +let res = await qortalRequest({ + action: "LIST_ATS", + codeHash58: "4KdJETRAdymE7dodDmJbf5d9L1bp4g5Nxky8m47TBkvA", + isExecutable: true, + limit: 100, + offset: 0, + reverse: true +}); +``` + +### Fetch block by signature +``` +let res = await qortalRequest({ + action: "FETCH_BLOCK", + signature: "875yGFUy1zHV2hmxNWzrhtn9S1zkeD7SQppwdXFysvTXrankCHCz4iyAUgCBM3GjvibbnyRQpriuy1cyu953U1u5uQdzuH3QjQivi9UVwz86z1Akn17MGd5Z5STjpDT7248K6vzMamuqDei57Znonr8GGgn8yyyABn35CbZUCeAuXju" +}); +``` + +### Fetch block by height +``` +let res = await qortalRequest({ + action: "FETCH_BLOCK", + height: "1139850" +}); +``` + +### Fetch a range of blocks +``` +let res = await qortalRequest({ + action: "FETCH_BLOCK_RANGE", + height: "1139800", + count: 20, + reverse: false +}); +``` + +### Search transactions +``` +let res = await qortalRequest({ + action: "SEARCH_TRANSACTIONS", + // startBlock: 1139000, + // blockLimit: 1000, + txGroupId: 0, + txType: [ + "PAYMENT", + "REWARD_SHARE" + ], + confirmationStatus: "CONFIRMED", + limit: 10, + offset: 0, + reverse: false +}); +``` + +### Get an estimate of the QORT price +``` +let res = await qortalRequest({ + action: "GET_PRICE", + blockchain: "LITECOIN", + // maxtrades: 10, + inverse: true +}); +``` + +### Get URL to load a QDN resource +Note: this returns a "Resource does not exist" error if a non-existent resource is requested. +``` +let url = await qortalRequest({ + action: "GET_QDN_RESOURCE_URL", + service: "THUMBNAIL", + name: "QortalDemo", + identifier: "qortal_avatar" + // path: "filename.jpg" // optional - not needed if resource contains only one file +}); +``` + +### Get URL to load a QDN website +Note: this returns a "Resource does not exist" error if a non-existent resource is requested. +``` +let url = await qortalRequest({ + action: "GET_QDN_RESOURCE_URL", + service: "WEBSITE", + name: "QortalDemo", +}); +``` + +### Get URL to load a specific file from a QDN website +Note: this returns a "Resource does not exist" error if a non-existent resource is requested. +``` +let url = await qortalRequest({ + action: "GET_QDN_RESOURCE_URL", + service: "WEBSITE", + name: "AlphaX", + path: "/assets/img/logo.png" +}); +``` + +### Link/redirect to another QDN website +Note: an alternate method is to include `link text` within your HTML code. +``` +let res = await qortalRequest({ + action: "LINK_TO_QDN_RESOURCE", + service: "WEBSITE", + name: "QortalDemo", +}); +``` + +### Link/redirect to a specific path of another QDN website +Note: an alternate method is to include `link text` within your HTML code. +``` +let res = await qortalRequest({ + action: "LINK_TO_QDN_RESOURCE", + service: "WEBSITE", + name: "QortalDemo", + path: "/minting-leveling/index.html" +}); +``` + +### Get the contents of a list +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "GET_LIST_ITEMS", + list_name: "followedNames" +}); +``` + +### Add one or more items to a list +_Requires user approval_ +``` +let res = await qortalRequest({ + action: "ADD_LIST_ITEMS", + list_name: "blockedNames", + items: ["QortalDemo"] +}); +``` + +### Delete a single item from a list +_Requires user approval_. +Items must be deleted one at a time. +``` +let res = await qortalRequest({ + action: "DELETE_LIST_ITEM", + list_name: "blockedNames", + item: "QortalDemo" +}); +``` + + +# Section 4: Examples + +Some example projects can be found [here](https://github.com/Qortal/Q-Apps). These can be cloned and modified, or used as a reference when creating a new app. + + +## Sample App + +Here is a sample application to display the logged-in user's avatar: +``` + + + + + + + + +``` + + +# Section 5: Testing and Development + +Publishing an in-development app to mainnet isn't recommended. There are several options for developing and testing a Q-app before publishing to mainnet: + +### Preview mode + +Select "Preview" in the UI after choosing the zip. This allows for full Q-App testing without the need to publish any data. + + +### Testnets + +For an end-to-end test of Q-App publishing, you can use the official testnet, or set up a single node testnet of your own (often referred to as devnet) on your local machine. See [Single Node Testnet Quick Start Guide](testnet/README.md#quick-start). + + +### Debugging + +It is recommended that you develop and test in a web browser, to allow access to the javascript console. To do this: +1. Open the UI app, then minimise it. +2. In a Chromium-based web browser, visit: http://localhost:12388/ +3. Log in to your account and then preview your app/website. +4. Go to `View > Developer > JavaScript Console`. Here you can monitor console logs, errors, and network requests from your app, in the same way as any other web-app. diff --git a/pom.xml b/pom.xml index 35c77bcc..0dfa0cf4 100644 --- a/pom.xml +++ b/pom.xml @@ -3,7 +3,7 @@ 4.0.0 org.qortal qortal - 3.8.9 + 4.0.3 jar true @@ -36,6 +36,7 @@ 4.10 1.45.1 3.19.4 + 1.17 src/main/java @@ -147,6 +148,7 @@ tagsSorter: "alpha", operationsSorter: "alpha", + validatorUrl: false, @@ -728,5 +730,10 @@ protobuf-java ${protobuf.version} + + com.j256.simplemagic + simplemagic + ${simplemagic.version} + diff --git a/src/main/java/org/qortal/api/ApiError.java b/src/main/java/org/qortal/api/ApiError.java index 659104e7..b52332b1 100644 --- a/src/main/java/org/qortal/api/ApiError.java +++ b/src/main/java/org/qortal/api/ApiError.java @@ -79,7 +79,7 @@ public enum ApiError { // BUYER_ALREADY_OWNER(411, 422), // POLLS - // POLL_NO_EXISTS(501, 404), + POLL_NO_EXISTS(501, 404), // POLL_ALREADY_EXISTS(502, 422), // DUPLICATE_OPTION(503, 422), // POLL_OPTION_NO_EXISTS(504, 404), diff --git a/src/main/java/org/qortal/api/ApiRequest.java b/src/main/java/org/qortal/api/ApiRequest.java index 5517ff53..a51a117e 100644 --- a/src/main/java/org/qortal/api/ApiRequest.java +++ b/src/main/java/org/qortal/api/ApiRequest.java @@ -3,6 +3,7 @@ package org.qortal.api; import java.io.IOException; import java.io.InputStream; import java.io.UnsupportedEncodingException; +import java.io.Writer; import java.net.HttpURLConnection; import java.net.InetSocketAddress; import java.net.Socket; @@ -20,14 +21,12 @@ import javax.net.ssl.SNIHostName; import javax.net.ssl.SNIServerName; import javax.net.ssl.SSLParameters; import javax.net.ssl.SSLSocket; -import javax.xml.bind.JAXBContext; -import javax.xml.bind.JAXBException; -import javax.xml.bind.UnmarshalException; -import javax.xml.bind.Unmarshaller; +import javax.xml.bind.*; import javax.xml.transform.stream.StreamSource; import org.eclipse.persistence.exceptions.XMLMarshalException; import org.eclipse.persistence.jaxb.JAXBContextFactory; +import org.eclipse.persistence.jaxb.MarshallerProperties; import org.eclipse.persistence.jaxb.UnmarshallerProperties; public class ApiRequest { @@ -107,6 +106,36 @@ public class ApiRequest { } } + private static Marshaller createMarshaller(Class objectClass) { + try { + // Create JAXB context aware of object's class + JAXBContext jc = JAXBContextFactory.createContext(new Class[] { objectClass }, null); + + // Create marshaller + Marshaller marshaller = jc.createMarshaller(); + + // Set the marshaller media type to JSON + marshaller.setProperty(MarshallerProperties.MEDIA_TYPE, "application/json"); + + // Tell marshaller not to include JSON root element in the output + marshaller.setProperty(MarshallerProperties.JSON_INCLUDE_ROOT, false); + + return marshaller; + } catch (JAXBException e) { + throw new RuntimeException("Unable to create API marshaller", e); + } + } + + public static void marshall(Writer writer, Object object) throws IOException { + Marshaller marshaller = createMarshaller(object.getClass()); + + try { + marshaller.marshal(object, writer); + } catch (JAXBException e) { + throw new IOException("Unable to create marshall object for API", e); + } + } + public static String getParamsString(Map params) { StringBuilder result = new StringBuilder(); diff --git a/src/main/java/org/qortal/api/ApiService.java b/src/main/java/org/qortal/api/ApiService.java index 78c9250c..059b8971 100644 --- a/src/main/java/org/qortal/api/ApiService.java +++ b/src/main/java/org/qortal/api/ApiService.java @@ -13,8 +13,8 @@ import java.security.SecureRandom; import javax.net.ssl.KeyManagerFactory; import javax.net.ssl.SSLContext; +import javax.servlet.http.HttpServletRequest; -import org.checkerframework.checker.units.qual.A; import org.eclipse.jetty.http.HttpVersion; import org.eclipse.jetty.rewrite.handler.RedirectPatternRule; import org.eclipse.jetty.rewrite.handler.RewriteHandler; @@ -41,6 +41,7 @@ import org.glassfish.jersey.servlet.ServletContainer; import org.qortal.api.resource.AnnotationPostProcessor; import org.qortal.api.resource.ApiDefinition; import org.qortal.api.websocket.*; +import org.qortal.network.Network; import org.qortal.settings.Settings; public class ApiService { @@ -51,9 +52,11 @@ public class ApiService { private Server server; private ApiKey apiKey; + public static final String API_VERSION_HEADER = "X-API-VERSION"; + private ApiService() { this.config = new ResourceConfig(); - this.config.packages("org.qortal.api.resource"); + this.config.packages("org.qortal.api.resource", "org.qortal.api.restricted.resource"); this.config.register(OpenApiResource.class); this.config.register(ApiDefinition.class); this.config.register(AnnotationPostProcessor.class); @@ -123,13 +126,13 @@ public class ApiService { ServerConnector portUnifiedConnector = new ServerConnector(this.server, new DetectorConnectionFactory(sslConnectionFactory), httpConnectionFactory); - portUnifiedConnector.setHost(Settings.getInstance().getBindAddress()); + portUnifiedConnector.setHost(Network.getInstance().getBindAddress()); portUnifiedConnector.setPort(Settings.getInstance().getApiPort()); this.server.addConnector(portUnifiedConnector); } else { // Non-SSL - InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress()); + InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress()); InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getApiPort()); this.server = new Server(endpoint); } @@ -230,4 +233,19 @@ public class ApiService { this.server = null; } + public static int getApiVersion(HttpServletRequest request) { + // Get API version + String apiVersionString = request.getHeader(API_VERSION_HEADER); + if (apiVersionString == null) { + // Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141 + apiVersionString = request.getParameter("apiVersion"); + } + + int apiVersion = 1; + if (apiVersionString != null) { + apiVersion = Integer.parseInt(apiVersionString); + } + return apiVersion; + } + } diff --git a/src/main/java/org/qortal/api/DomainMapService.java b/src/main/java/org/qortal/api/DomainMapService.java index ba0fa067..3b81d94c 100644 --- a/src/main/java/org/qortal/api/DomainMapService.java +++ b/src/main/java/org/qortal/api/DomainMapService.java @@ -3,7 +3,6 @@ package org.qortal.api; import io.swagger.v3.jaxrs2.integration.resources.OpenApiResource; import org.eclipse.jetty.http.HttpVersion; import org.eclipse.jetty.rewrite.handler.RewriteHandler; -import org.eclipse.jetty.rewrite.handler.RewritePatternRule; import org.eclipse.jetty.server.*; import org.eclipse.jetty.server.handler.ErrorHandler; import org.eclipse.jetty.server.handler.InetAccessHandler; @@ -16,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig; import org.glassfish.jersey.servlet.ServletContainer; import org.qortal.api.resource.AnnotationPostProcessor; import org.qortal.api.resource.ApiDefinition; +import org.qortal.network.Network; import org.qortal.settings.Settings; import javax.net.ssl.KeyManagerFactory; @@ -38,7 +38,7 @@ public class DomainMapService { private DomainMapService() { this.config = new ResourceConfig(); - this.config.packages("org.qortal.api.domainmap.resource"); + this.config.packages("org.qortal.api.resource", "org.qortal.api.domainmap.resource"); this.config.register(OpenApiResource.class); this.config.register(ApiDefinition.class); this.config.register(AnnotationPostProcessor.class); @@ -99,13 +99,13 @@ public class DomainMapService { ServerConnector portUnifiedConnector = new ServerConnector(this.server, new DetectorConnectionFactory(sslConnectionFactory), httpConnectionFactory); - portUnifiedConnector.setHost(Settings.getInstance().getBindAddress()); + portUnifiedConnector.setHost(Network.getInstance().getBindAddress()); portUnifiedConnector.setPort(Settings.getInstance().getDomainMapPort()); this.server.addConnector(portUnifiedConnector); } else { // Non-SSL - InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress()); + InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress()); InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getDomainMapPort()); this.server = new Server(endpoint); } diff --git a/src/main/java/org/qortal/api/GatewayService.java b/src/main/java/org/qortal/api/GatewayService.java index 030a0f2f..51191af3 100644 --- a/src/main/java/org/qortal/api/GatewayService.java +++ b/src/main/java/org/qortal/api/GatewayService.java @@ -15,6 +15,7 @@ import org.glassfish.jersey.server.ResourceConfig; import org.glassfish.jersey.servlet.ServletContainer; import org.qortal.api.resource.AnnotationPostProcessor; import org.qortal.api.resource.ApiDefinition; +import org.qortal.network.Network; import org.qortal.settings.Settings; import javax.net.ssl.KeyManagerFactory; @@ -37,7 +38,7 @@ public class GatewayService { private GatewayService() { this.config = new ResourceConfig(); - this.config.packages("org.qortal.api.gateway.resource"); + this.config.packages("org.qortal.api.resource", "org.qortal.api.gateway.resource"); this.config.register(OpenApiResource.class); this.config.register(ApiDefinition.class); this.config.register(AnnotationPostProcessor.class); @@ -98,13 +99,13 @@ public class GatewayService { ServerConnector portUnifiedConnector = new ServerConnector(this.server, new DetectorConnectionFactory(sslConnectionFactory), httpConnectionFactory); - portUnifiedConnector.setHost(Settings.getInstance().getBindAddress()); + portUnifiedConnector.setHost(Network.getInstance().getBindAddress()); portUnifiedConnector.setPort(Settings.getInstance().getGatewayPort()); this.server.addConnector(portUnifiedConnector); } else { // Non-SSL - InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress()); + InetAddress bindAddr = InetAddress.getByName(Network.getInstance().getBindAddress()); InetSocketAddress endpoint = new InetSocketAddress(bindAddr, Settings.getInstance().getGatewayPort()); this.server = new Server(endpoint); } diff --git a/src/main/java/org/qortal/api/HTMLParser.java b/src/main/java/org/qortal/api/HTMLParser.java index 026d9210..cc3102e8 100644 --- a/src/main/java/org/qortal/api/HTMLParser.java +++ b/src/main/java/org/qortal/api/HTMLParser.java @@ -5,28 +5,71 @@ import org.apache.logging.log4j.Logger; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; import org.jsoup.select.Elements; +import org.qortal.arbitrary.misc.Service; + +import java.util.Objects; public class HTMLParser { private static final Logger LOGGER = LogManager.getLogger(HTMLParser.class); - private String linkPrefix; + private String qdnBase; + private String qdnBaseWithPath; private byte[] data; + private String qdnContext; + private String resourceId; + private Service service; + private String identifier; + private String path; + private String theme; + private boolean usingCustomRouting; - public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data) { - String inPathWithoutFilename = inPath.substring(0, inPath.lastIndexOf('/')); - this.linkPrefix = usePrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : ""; + public HTMLParser(String resourceId, String inPath, String prefix, boolean usePrefix, byte[] data, + String qdnContext, Service service, String identifier, String theme, boolean usingCustomRouting) { + String inPathWithoutFilename = inPath.contains("/") ? inPath.substring(0, inPath.lastIndexOf('/')) : ""; + this.qdnBase = usePrefix ? String.format("%s/%s", prefix, resourceId) : ""; + this.qdnBaseWithPath = usePrefix ? String.format("%s/%s%s", prefix, resourceId, inPathWithoutFilename) : ""; this.data = data; + this.qdnContext = qdnContext; + this.resourceId = resourceId; + this.service = service; + this.identifier = identifier; + this.path = inPath; + this.theme = theme; + this.usingCustomRouting = usingCustomRouting; } public void addAdditionalHeaderTags() { String fileContents = new String(data); Document document = Jsoup.parse(fileContents); - String baseUrl = this.linkPrefix + "/"; Elements head = document.getElementsByTag("head"); if (!head.isEmpty()) { + // Add q-apps script tag + String qAppsScriptElement = String.format("", qdnContext, theme, service, name, identifier, path, qdnBase, qdnBaseWithPath); + head.get(0).prepend(qdnContextVar); + // Add base href tag - String baseElement = String.format("", baseUrl); + // Exclude the path if this request was routed back to the index automatically + String baseHref = this.usingCustomRouting ? this.qdnBase : this.qdnBaseWithPath; + String baseElement = String.format("", baseHref); head.get(0).prepend(baseElement); // Add meta charset tag diff --git a/src/main/java/org/qortal/api/Security.java b/src/main/java/org/qortal/api/Security.java index 4aca2c49..f009d79f 100644 --- a/src/main/java/org/qortal/api/Security.java +++ b/src/main/java/org/qortal/api/Security.java @@ -15,7 +15,21 @@ public abstract class Security { public static final String API_KEY_HEADER = "X-API-KEY"; + /** + * Check API call is allowed, retrieving the API key from the request header or GET/POST parameters where required + * @param request + */ public static void checkApiCallAllowed(HttpServletRequest request) { + checkApiCallAllowed(request, null); + } + + /** + * Check API call is allowed, retrieving the API key first from the passedApiKey parameter, with a fallback + * to the request header or GET/POST parameters when null. + * @param request + * @param passedApiKey - the API key to test, or null if it should be retrieved from the request headers. + */ + public static void checkApiCallAllowed(HttpServletRequest request, String passedApiKey) { // We may want to allow automatic authentication for local requests, if enabled in settings boolean localAuthBypassEnabled = Settings.getInstance().isLocalAuthBypassEnabled(); if (localAuthBypassEnabled) { @@ -38,7 +52,10 @@ public abstract class Security { } // We require an API key to be passed - String passedApiKey = request.getHeader(API_KEY_HEADER); + if (passedApiKey == null) { + // API call not passed as a parameter, so try the header + passedApiKey = request.getHeader(API_KEY_HEADER); + } if (passedApiKey == null) { // Try query string - this is needed to avoid a CORS preflight. See: https://stackoverflow.com/a/43881141 passedApiKey = request.getParameter("apiKey"); @@ -56,7 +73,7 @@ public abstract class Security { public static void disallowLoopbackRequests(HttpServletRequest request) { try { InetAddress remoteAddr = InetAddress.getByName(request.getRemoteAddr()); - if (remoteAddr.isLoopbackAddress()) { + if (remoteAddr.isLoopbackAddress() && !Settings.getInstance().isGatewayLoopbackEnabled()) { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.UNAUTHORIZED, "Local requests not allowed"); } } catch (UnknownHostException e) { @@ -84,9 +101,9 @@ public abstract class Security { } } - public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier) { + public static void requirePriorAuthorizationOrApiKey(HttpServletRequest request, String resourceId, Service service, String identifier, String apiKey) { try { - Security.checkApiCallAllowed(request); + Security.checkApiCallAllowed(request, apiKey); } catch (ApiException e) { // API call wasn't allowed, but maybe it was pre-authorized diff --git a/src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java b/src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java index cc21587d..4cb9f8e5 100644 --- a/src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java +++ b/src/main/java/org/qortal/api/domainmap/resource/DomainMapResource.java @@ -42,16 +42,16 @@ public class DomainMapResource { // Build synchronously, so that we don't need to make the summary API endpoints available over // the domain map server. This means that there will be no loading screen, but this is potentially // preferred in this situation anyway (e.g. to avoid confusing search engine robots). - return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", false, false); + return this.get(domainMap.get(request.getServerName()), ResourceIdType.NAME, Service.WEBSITE, null, inPath, null, "", false, false); } return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found"); } - private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath, - String secret58, String prefix, boolean usePrefix, boolean async) { + private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier, + String inPath, String secret58, String prefix, boolean usePrefix, boolean async) { - ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath, - secret58, prefix, usePrefix, async, request, response, context); + ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath, + secret58, prefix, usePrefix, async, "domainMap", request, response, context); return renderer.render(); } diff --git a/src/main/java/org/qortal/api/gateway/resource/GatewayResource.java b/src/main/java/org/qortal/api/gateway/resource/GatewayResource.java index a73de1fb..9c77753f 100644 --- a/src/main/java/org/qortal/api/gateway/resource/GatewayResource.java +++ b/src/main/java/org/qortal/api/gateway/resource/GatewayResource.java @@ -2,6 +2,7 @@ package org.qortal.api.gateway.resource; import io.swagger.v3.oas.annotations.security.SecurityRequirement; import io.swagger.v3.oas.annotations.tags.Tag; +import org.apache.commons.lang3.StringUtils; import org.qortal.api.Security; import org.qortal.arbitrary.ArbitraryDataFile; import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType; @@ -16,6 +17,10 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import javax.ws.rs.*; import javax.ws.rs.core.Context; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedList; +import java.util.List; @Path("/") @@ -76,50 +81,83 @@ public class GatewayResource { @GET - @Path("{name}/{path:.*}") + @Path("{path:.*}") @SecurityRequirement(name = "apiKey") - public HttpServletResponse getPathByName(@PathParam("name") String name, - @PathParam("path") String inPath) { + public HttpServletResponse getPath(@PathParam("path") String inPath) { // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data Security.disallowLoopbackRequests(request); - return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "", true, true); - } - - @GET - @Path("{name}") - @SecurityRequirement(name = "apiKey") - public HttpServletResponse getIndexByName(@PathParam("name") String name) { - // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data - Security.disallowLoopbackRequests(request); - return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "", true, true); - } - - - // Optional /site alternative for backwards support - - @GET - @Path("/site/{name}/{path:.*}") - public HttpServletResponse getSitePathByName(@PathParam("name") String name, - @PathParam("path") String inPath) { - // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data - Security.disallowLoopbackRequests(request); - return this.get(name, ResourceIdType.NAME, Service.WEBSITE, inPath, null, "/site", true, true); - } - - @GET - @Path("/site/{name}") - public HttpServletResponse getSiteIndexByName(@PathParam("name") String name) { - // Block requests from localhost, to prevent websites/apps from running javascript that fetches unvetted data - Security.disallowLoopbackRequests(request); - return this.get(name, ResourceIdType.NAME, Service.WEBSITE, "/", null, "/site", true, true); + return this.parsePath(inPath, "gateway", null, true, true); } - private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath, - String secret58, String prefix, boolean usePrefix, boolean async) { + private HttpServletResponse parsePath(String inPath, String qdnContext, String secret58, boolean usePrefix, boolean async) { - ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath, - secret58, prefix, usePrefix, async, request, response, context); + if (inPath == null || inPath.equals("")) { + // Assume not a real file + return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found"); + } + + // Default service is WEBSITE + Service service = Service.WEBSITE; + String name = null; + String identifier = null; + String outPath = ""; + List prefixParts = new ArrayList<>(); + + if (!inPath.contains("/")) { + // Assume entire inPath is a registered name + name = inPath; + } + else { + // Parse the path to determine what we need to load + List parts = new LinkedList<>(Arrays.asList(inPath.split("/"))); + + // Check if the first element is a service + try { + Service parsedService = Service.valueOf(parts.get(0).toUpperCase()); + if (parsedService != null) { + // First element matches a service, so we can assume it is one + service = parsedService; + parts.remove(0); + prefixParts.add(service.name()); + } + } catch (IllegalArgumentException e) { + // Not a service + } + + if (parts.isEmpty()) { + // We need more than just a service + return ArbitraryDataRenderer.getResponse(response, 404, "Error 404: File Not Found"); + } + + // Service is removed, so assume first element is now a registered name + name = parts.get(0); + parts.remove(0); + + if (!parts.isEmpty()) { + // Name is removed, so check if the first element is now an identifier + ArbitraryResourceStatus status = this.getStatus(service, name, parts.get(0), false); + if (status.getTotalChunkCount() > 0) { + // Matched service, name and identifier combination - so assume this is an identifier and can be removed + identifier = parts.get(0); + parts.remove(0); + prefixParts.add(identifier); + } + } + + if (!parts.isEmpty()) { + // outPath can be built by combining any remaining parts + outPath = String.join("/", parts); + } + } + + String prefix = StringUtils.join(prefixParts, "/"); + if (prefix != null && prefix.length() > 0) { + prefix = "/" + prefix; + } + + ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(name, ResourceIdType.NAME, service, identifier, outPath, + secret58, prefix, usePrefix, async, qdnContext, request, response, context); return renderer.render(); } diff --git a/src/main/java/org/qortal/api/model/FileProperties.java b/src/main/java/org/qortal/api/model/FileProperties.java new file mode 100644 index 00000000..c63506dd --- /dev/null +++ b/src/main/java/org/qortal/api/model/FileProperties.java @@ -0,0 +1,16 @@ +package org.qortal.api.model; + +import javax.xml.bind.annotation.XmlAccessType; +import javax.xml.bind.annotation.XmlAccessorType; + +@XmlAccessorType(XmlAccessType.FIELD) +public class FileProperties { + + public String filename; + public String mimeType; + public Long size; + + public FileProperties() { + } + +} diff --git a/src/main/java/org/qortal/api/model/PollVotes.java b/src/main/java/org/qortal/api/model/PollVotes.java new file mode 100644 index 00000000..c57ebc37 --- /dev/null +++ b/src/main/java/org/qortal/api/model/PollVotes.java @@ -0,0 +1,56 @@ +package org.qortal.api.model; + +import java.util.List; +import javax.xml.bind.annotation.XmlAccessType; +import javax.xml.bind.annotation.XmlAccessorType; +import javax.xml.bind.annotation.XmlElement; + +import io.swagger.v3.oas.annotations.media.Schema; + +import org.qortal.data.voting.VoteOnPollData; + +@Schema(description = "Poll vote info, including voters") +// All properties to be converted to JSON via JAX-RS +@XmlAccessorType(XmlAccessType.FIELD) +public class PollVotes { + + @Schema(description = "List of individual votes") + @XmlElement(name = "votes") + public List votes; + + @Schema(description = "Total number of votes") + public Integer totalVotes; + + @Schema(description = "List of vote counts for each option") + public List voteCounts; + + // For JAX-RS + protected PollVotes() { + } + + public PollVotes(List votes, Integer totalVotes, List voteCounts) { + this.votes = votes; + this.totalVotes = totalVotes; + this.voteCounts = voteCounts; + } + + @Schema(description = "Vote info") + // All properties to be converted to JSON via JAX-RS + @XmlAccessorType(XmlAccessType.FIELD) + public static class OptionCount { + @Schema(description = "Option name") + public String optionName; + + @Schema(description = "Vote count") + public Integer voteCount; + + // For JAX-RS + protected OptionCount() { + } + + public OptionCount(String optionName, Integer voteCount) { + this.optionName = optionName; + this.voteCount = voteCount; + } + } +} diff --git a/src/main/java/org/qortal/api/resource/AppsResource.java b/src/main/java/org/qortal/api/resource/AppsResource.java new file mode 100644 index 00000000..19a4a184 --- /dev/null +++ b/src/main/java/org/qortal/api/resource/AppsResource.java @@ -0,0 +1,83 @@ +package org.qortal.api.resource; + +import com.google.common.io.Resources; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.Hidden; +import io.swagger.v3.oas.annotations.tags.Tag; +import org.qortal.api.*; + +import javax.servlet.ServletContext; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import java.io.IOException; +import java.net.URL; +import java.nio.charset.StandardCharsets; + + +@Path("/apps") +@Tag(name = "Apps") +public class AppsResource { + + @Context HttpServletRequest request; + @Context HttpServletResponse response; + @Context ServletContext context; + + @GET + @Path("/q-apps.js") + @Hidden // For internal Q-App API use only + @Operation( + summary = "Javascript interface for Q-Apps", + responses = { + @ApiResponse( + description = "javascript", + content = @Content( + mediaType = MediaType.TEXT_PLAIN, + schema = @Schema( + type = "string" + ) + ) + ) + } + ) + public String getQAppsJs() { + URL url = Resources.getResource("q-apps/q-apps.js"); + try { + return Resources.toString(url, StandardCharsets.UTF_8); + } catch (IOException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND); + } + } + + @GET + @Path("/q-apps-gateway.js") + @Hidden // For internal Q-App API use only + @Operation( + summary = "Gateway-specific interface for Q-Apps", + responses = { + @ApiResponse( + description = "javascript", + content = @Content( + mediaType = MediaType.TEXT_PLAIN, + schema = @Schema( + type = "string" + ) + ) + ) + } + ) + public String getQAppsGatewayJs() { + URL url = Resources.getResource("q-apps/q-apps-gateway.js"); + try { + return Resources.toString(url, StandardCharsets.UTF_8); + } catch (IOException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND); + } + } + +} diff --git a/src/main/java/org/qortal/api/resource/ArbitraryResource.java b/src/main/java/org/qortal/api/resource/ArbitraryResource.java index 235e3edc..c617b517 100644 --- a/src/main/java/org/qortal/api/resource/ArbitraryResource.java +++ b/src/main/java/org/qortal/api/resource/ArbitraryResource.java @@ -1,6 +1,8 @@ package org.qortal.api.resource; import com.google.common.primitives.Bytes; +import com.j256.simplemagic.ContentInfo; +import com.j256.simplemagic.ContentInfoUtil; import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.Parameter; import io.swagger.v3.oas.annotations.media.ArraySchema; @@ -12,11 +14,14 @@ import io.swagger.v3.oas.annotations.security.SecurityRequirement; import io.swagger.v3.oas.annotations.tags.Tag; import java.io.*; +import java.net.FileNameMap; +import java.net.URLConnection; import java.nio.file.Files; import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; import java.util.List; +import java.util.Objects; import javax.servlet.ServletContext; import javax.servlet.http.HttpServletRequest; @@ -25,11 +30,13 @@ import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; +import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.ArrayUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.bouncycastle.util.encoders.Base64; import org.qortal.api.*; +import org.qortal.api.model.FileProperties; import org.qortal.api.resource.TransactionsResource.ConfirmationStatus; import org.qortal.arbitrary.*; import org.qortal.arbitrary.ArbitraryDataFile.ResourceIdType; @@ -38,6 +45,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata; import org.qortal.arbitrary.misc.Category; import org.qortal.arbitrary.misc.Service; import org.qortal.controller.Controller; +import org.qortal.controller.arbitrary.ArbitraryDataRenderManager; import org.qortal.controller.arbitrary.ArbitraryDataStorageManager; import org.qortal.controller.arbitrary.ArbitraryMetadataManager; import org.qortal.data.account.AccountData; @@ -57,10 +65,7 @@ import org.qortal.transaction.Transaction.ValidationResult; import org.qortal.transform.TransformationException; import org.qortal.transform.transaction.ArbitraryTransactionTransformer; import org.qortal.transform.transaction.TransactionTransformer; -import org.qortal.utils.ArbitraryTransactionUtils; -import org.qortal.utils.Base58; -import org.qortal.utils.NTP; -import org.qortal.utils.ZipUtils; +import org.qortal.utils.*; @Path("/arbitrary") @Tag(name = "Arbitrary") @@ -88,12 +93,15 @@ public class ArbitraryResource { @ApiErrors({ApiError.REPOSITORY_ISSUE}) public List getResources( @QueryParam("service") Service service, + @QueryParam("name") String name, @QueryParam("identifier") String identifier, @Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource, @Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset, @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse, - @Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameFilter, + @Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly, + @Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked, + @Parameter(description = "Filter names by list") @QueryParam("namefilter") String nameListFilter, @Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus, @Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) { @@ -110,28 +118,33 @@ public class ArbitraryResource { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource"); } - // Load filter from list if needed + // Set up name filters if supplied List names = null; - if (nameFilter != null) { - names = ResourceListManager.getInstance().getStringsInList(nameFilter); + if (name != null) { + // Filter using single name + names = Arrays.asList(name); + } + else if (nameListFilter != null) { + // Filter using supplied list of names + names = ResourceListManager.getInstance().getStringsInList(nameListFilter); if (names.isEmpty()) { - // List doesn't exist or is empty - so there will be no matches + // If list is empty (or doesn't exist) we can shortcut with empty response return new ArrayList<>(); } } List resources = repository.getArbitraryRepository() - .getArbitraryResources(service, identifier, names, defaultRes, limit, offset, reverse); + .getArbitraryResources(service, identifier, names, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse); if (resources == null) { return new ArrayList<>(); } if (includeStatus != null && includeStatus) { - resources = this.addStatusToResources(resources); + resources = ArbitraryTransactionUtils.addStatusToResources(resources); } if (includeMetadata != null && includeMetadata) { - resources = this.addMetadataToResources(resources); + resources = ArbitraryTransactionUtils.addMetadataToResources(resources); } return resources; @@ -155,30 +168,56 @@ public class ArbitraryResource { @ApiErrors({ApiError.REPOSITORY_ISSUE}) public List searchResources( @QueryParam("service") Service service, - @QueryParam("query") String query, + @Parameter(description = "Query (searches both name and identifier fields)") @QueryParam("query") String query, + @Parameter(description = "Identifier (searches identifier field only)") @QueryParam("identifier") String identifier, + @Parameter(description = "Name (searches name field only)") @QueryParam("name") List names, + @Parameter(description = "Prefix only (if true, only the beginning of fields are matched)") @QueryParam("prefix") Boolean prefixOnly, + @Parameter(description = "Exact match names only (if true, partial name matches are excluded)") @QueryParam("exactmatchnames") Boolean exactMatchNamesOnly, @Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource, + @Parameter(description = "Filter names by list (exact matches only)") @QueryParam("namefilter") String nameListFilter, + @Parameter(description = "Include followed names only") @QueryParam("followedonly") Boolean followedOnly, + @Parameter(description = "Exclude blocked content") @QueryParam("excludeblocked") Boolean excludeBlocked, + @Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus, + @Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata, @Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset, - @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse, - @Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus, - @Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) { + @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) { try (final Repository repository = RepositoryManager.getRepository()) { boolean defaultRes = Boolean.TRUE.equals(defaultResource); + boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly); + + List exactMatchNames = new ArrayList<>(); + + if (nameListFilter != null) { + // Load names from supplied list of names + exactMatchNames.addAll(ResourceListManager.getInstance().getStringsInList(nameListFilter)); + + // If list is empty (or doesn't exist) we can shortcut with empty response + if (exactMatchNames.isEmpty()) { + return new ArrayList<>(); + } + } + + // Move names to exact match list, if requested + if (exactMatchNamesOnly != null && exactMatchNamesOnly && names != null) { + exactMatchNames.addAll(names); + names = null; + } List resources = repository.getArbitraryRepository() - .searchArbitraryResources(service, query, defaultRes, limit, offset, reverse); + .searchArbitraryResources(service, query, identifier, names, usePrefixOnly, exactMatchNames, defaultRes, followedOnly, excludeBlocked, limit, offset, reverse); if (resources == null) { return new ArrayList<>(); } if (includeStatus != null && includeStatus) { - resources = this.addStatusToResources(resources); + resources = ArbitraryTransactionUtils.addStatusToResources(resources); } if (includeMetadata != null && includeMetadata) { - resources = this.addMetadataToResources(resources); + resources = ArbitraryTransactionUtils.addMetadataToResources(resources); } return resources; @@ -188,67 +227,6 @@ public class ArbitraryResource { } } - @GET - @Path("/resources/names") - @Operation( - summary = "List arbitrary resources available on chain, grouped by creator's name", - responses = { - @ApiResponse( - content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = ArbitraryResourceInfo.class)) - ) - } - ) - @ApiErrors({ApiError.REPOSITORY_ISSUE}) - public List getResourcesGroupedByName( - @QueryParam("service") Service service, - @QueryParam("identifier") String identifier, - @Parameter(description = "Default resources (without identifiers) only") @QueryParam("default") Boolean defaultResource, - @Parameter(ref = "limit") @QueryParam("limit") Integer limit, - @Parameter(ref = "offset") @QueryParam("offset") Integer offset, - @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse, - @Parameter(description = "Include status") @QueryParam("includestatus") Boolean includeStatus, - @Parameter(description = "Include metadata") @QueryParam("includemetadata") Boolean includeMetadata) { - - try (final Repository repository = RepositoryManager.getRepository()) { - - // Treat empty identifier as null - if (identifier != null && identifier.isEmpty()) { - identifier = null; - } - - // Ensure that "default" and "identifier" parameters cannot coexist - boolean defaultRes = Boolean.TRUE.equals(defaultResource); - if (defaultRes == true && identifier != null) { - throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "identifier cannot be specified when requesting a default resource"); - } - - List creatorNames = repository.getArbitraryRepository() - .getArbitraryResourceCreatorNames(service, identifier, defaultRes, limit, offset, reverse); - - for (ArbitraryResourceNameInfo creatorName : creatorNames) { - String name = creatorName.name; - if (name != null) { - List resources = repository.getArbitraryRepository() - .getArbitraryResources(service, identifier, Arrays.asList(name), defaultRes, null, null, reverse); - - if (includeStatus != null && includeStatus) { - resources = this.addStatusToResources(resources); - } - if (includeMetadata != null && includeMetadata) { - resources = this.addMetadataToResources(resources); - } - - creatorName.resources = resources; - } - } - - return creatorNames; - - } catch (DataException e) { - throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); - } - } - @GET @Path("/resource/status/{service}/{name}") @Operation( @@ -266,10 +244,35 @@ public class ArbitraryResource { @PathParam("name") String name, @QueryParam("build") Boolean build) { - Security.requirePriorAuthorizationOrApiKey(request, name, service, null); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorizationOrApiKey(request, name, service, null, apiKey); + return ArbitraryTransactionUtils.getStatus(service, name, null, build); } + @GET + @Path("/resource/properties/{service}/{name}/{identifier}") + @Operation( + summary = "Get properties of a QDN resource", + description = "This attempts a download of the data if it's not available locally. A filename will only be returned for single file resources. mimeType is only returned when it can be determined.", + responses = { + @ApiResponse( + content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = FileProperties.class)) + ) + } + ) + @SecurityRequirement(name = "apiKey") + public FileProperties getResourceProperties(@HeaderParam(Security.API_KEY_HEADER) String apiKey, + @PathParam("service") Service service, + @PathParam("name") String name, + @PathParam("identifier") String identifier) { + + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey); + + return this.getFileProperties(service, name, identifier); + } + @GET @Path("/resource/status/{service}/{name}/{identifier}") @Operation( @@ -288,7 +291,9 @@ public class ArbitraryResource { @PathParam("identifier") String identifier, @QueryParam("build") Boolean build) { - Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorizationOrApiKey(request, name, service, identifier, apiKey); + return ArbitraryTransactionUtils.getStatus(service, name, identifier, build); } @@ -501,6 +506,9 @@ public class ArbitraryResource { } for (ArbitraryTransactionData transactionData : transactionDataList) { + if (transactionData.getService() == null) { + continue; + } ArbitraryResourceInfo arbitraryResourceInfo = new ArbitraryResourceInfo(); arbitraryResourceInfo.name = transactionData.getName(); arbitraryResourceInfo.service = transactionData.getService(); @@ -511,10 +519,10 @@ public class ArbitraryResource { } if (includeStatus != null && includeStatus) { - resources = this.addStatusToResources(resources); + resources = ArbitraryTransactionUtils.addStatusToResources(resources); } if (includeMetadata != null && includeMetadata) { - resources = this.addMetadataToResources(resources); + resources = ArbitraryTransactionUtils.addMetadataToResources(resources); } return resources; @@ -544,7 +552,7 @@ public class ArbitraryResource { Security.checkApiCallAllowed(request); ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier); - return resource.delete(); + return resource.delete(false); } @POST @@ -641,6 +649,7 @@ public class ArbitraryResource { @PathParam("service") Service service, @PathParam("name") String name, @QueryParam("filepath") String filepath, + @QueryParam("encoding") String encoding, @QueryParam("rebuild") boolean rebuild, @QueryParam("async") boolean async, @QueryParam("attempts") Integer attempts) { @@ -650,7 +659,7 @@ public class ArbitraryResource { Security.checkApiCallAllowed(request); } - return this.download(service, name, null, filepath, rebuild, async, attempts); + return this.download(service, name, null, filepath, encoding, rebuild, async, attempts); } @GET @@ -676,16 +685,17 @@ public class ArbitraryResource { @PathParam("name") String name, @PathParam("identifier") String identifier, @QueryParam("filepath") String filepath, + @QueryParam("encoding") String encoding, @QueryParam("rebuild") boolean rebuild, @QueryParam("async") boolean async, @QueryParam("attempts") Integer attempts) { // Authentication can be bypassed in the settings, for those running public QDN nodes if (!Settings.getInstance().isQDNAuthBypassEnabled()) { - Security.checkApiCallAllowed(request); + Security.checkApiCallAllowed(request, apiKey); } - return this.download(service, name, identifier, filepath, rebuild, async, attempts); + return this.download(service, name, identifier, filepath, encoding, rebuild, async, attempts); } @@ -708,12 +718,9 @@ public class ArbitraryResource { } ) @SecurityRequirement(name = "apiKey") - public ArbitraryResourceMetadata getMetadata(@HeaderParam(Security.API_KEY_HEADER) String apiKey, - @PathParam("service") Service service, - @PathParam("name") String name, - @PathParam("identifier") String identifier) { - Security.checkApiCallAllowed(request); - + public ArbitraryResourceMetadata getMetadata(@PathParam("service") Service service, + @PathParam("name") String name, + @PathParam("identifier") String identifier) { ArbitraryDataResource resource = new ArbitraryDataResource(name, ResourceIdType.NAME, service, identifier); try { @@ -733,7 +740,7 @@ public class ArbitraryResource { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage()); } - throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA); + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.FILE_NOT_FOUND); } @@ -774,6 +781,7 @@ public class ArbitraryResource { @QueryParam("tags") List tags, @QueryParam("category") Category category, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String path) { Security.checkApiCallAllowed(request); @@ -782,7 +790,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, null, path, null, null, false, - fee, title, description, tags, category); + fee, null, title, description, tags, category, preview); } @POST @@ -820,6 +828,7 @@ public class ArbitraryResource { @QueryParam("tags") List tags, @QueryParam("category") Category category, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String path) { Security.checkApiCallAllowed(request); @@ -828,7 +837,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, identifier, path, null, null, false, - fee, title, description, tags, category); + fee, null, title, description, tags, category, preview); } @@ -866,7 +875,9 @@ public class ArbitraryResource { @QueryParam("description") String description, @QueryParam("tags") List tags, @QueryParam("category") Category category, + @QueryParam("filename") String filename, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String base64) { Security.checkApiCallAllowed(request); @@ -875,7 +886,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, null, null, null, base64, false, - fee, title, description, tags, category); + fee, filename, title, description, tags, category, preview); } @POST @@ -910,7 +921,9 @@ public class ArbitraryResource { @QueryParam("description") String description, @QueryParam("tags") List tags, @QueryParam("category") Category category, + @QueryParam("filename") String filename, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String base64) { Security.checkApiCallAllowed(request); @@ -919,7 +932,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64, false, - fee, title, description, tags, category); + fee, filename, title, description, tags, category, preview); } @@ -957,6 +970,7 @@ public class ArbitraryResource { @QueryParam("tags") List tags, @QueryParam("category") Category category, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String base64Zip) { Security.checkApiCallAllowed(request); @@ -965,7 +979,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, null, null, null, base64Zip, true, - fee, title, description, tags, category); + fee, null, title, description, tags, category, preview); } @POST @@ -1001,6 +1015,7 @@ public class ArbitraryResource { @QueryParam("tags") List tags, @QueryParam("category") Category category, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String base64Zip) { Security.checkApiCallAllowed(request); @@ -1009,7 +1024,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, identifier, null, null, base64Zip, true, - fee, title, description, tags, category); + fee, null, title, description, tags, category, preview); } @@ -1049,7 +1064,9 @@ public class ArbitraryResource { @QueryParam("description") String description, @QueryParam("tags") List tags, @QueryParam("category") Category category, + @QueryParam("filename") String filename, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String string) { Security.checkApiCallAllowed(request); @@ -1058,7 +1075,7 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, null, null, string, null, false, - fee, title, description, tags, category); + fee, filename, title, description, tags, category, preview); } @POST @@ -1095,7 +1112,9 @@ public class ArbitraryResource { @QueryParam("description") String description, @QueryParam("tags") List tags, @QueryParam("category") Category category, + @QueryParam("filename") String filename, @QueryParam("fee") Long fee, + @QueryParam("preview") Boolean preview, String string) { Security.checkApiCallAllowed(request); @@ -1104,15 +1123,48 @@ public class ArbitraryResource { } return this.upload(Service.valueOf(serviceString), name, identifier, null, string, null, false, - fee, title, description, tags, category); + fee, filename, title, description, tags, category, preview); } // Shared methods - private String upload(Service service, String name, String identifier, String path, - String string, String base64, boolean zipped, Long fee, - String title, String description, List tags, Category category) { + private String preview(String directoryPath, Service service) { + Security.checkApiCallAllowed(request); + ArbitraryTransactionData.Method method = ArbitraryTransactionData.Method.PUT; + ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.ZIP; + + ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath), + null, service, null, method, compression, + null, null, null, null); + try { + arbitraryDataWriter.save(); + } catch (IOException | DataException | InterruptedException | MissingDataException e) { + LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage()); + throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage()); + } catch (RuntimeException e) { + LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage()); + throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage()); + } + + ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile(); + if (arbitraryDataFile != null) { + String digest58 = arbitraryDataFile.digest58(); + if (digest58 != null) { + // Pre-authorize resource + ArbitraryDataResource resource = new ArbitraryDataResource(digest58, null, null, null); + ArbitraryDataRenderManager.getInstance().addToAuthorizedResources(resource); + + return "/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret()); + } + } + return "Unable to generate preview URL"; + } + + private String upload(Service service, String name, String identifier, + String path, String string, String base64, boolean zipped, Long fee, String filename, + String title, String description, List tags, Category category, + Boolean preview) { // Fetch public key from registered name try (final Repository repository = RepositoryManager.getRepository()) { NameData nameData = repository.getNameRepository().fromName(name); @@ -1121,7 +1173,11 @@ public class ArbitraryResource { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, error); } - final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L); + final Long now = NTP.getTime(); + if (now == null) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NO_TIME_SYNC); + } + final Long minLatestBlockTimestamp = now - (60 * 60 * 1000L); if (!Controller.getInstance().isUpToDate(minLatestBlockTimestamp)) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.BLOCKCHAIN_NEEDS_SYNC); } @@ -1136,7 +1192,12 @@ public class ArbitraryResource { if (path == null) { // See if we have a string instead if (string != null) { - File tempFile = File.createTempFile("qortal-", ""); + if (filename == null) { + // Use current time as filename + filename = String.format("qortal-%d", NTP.getTime()); + } + java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-"); + File tempFile = Paths.get(tempDirectory.toString(), filename).toFile(); tempFile.deleteOnExit(); BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile.toPath().toString())); writer.write(string); @@ -1146,7 +1207,12 @@ public class ArbitraryResource { } // ... or base64 encoded raw data else if (base64 != null) { - File tempFile = File.createTempFile("qortal-", ""); + if (filename == null) { + // Use current time as filename + filename = String.format("qortal-%d", NTP.getTime()); + } + java.nio.file.Path tempDirectory = Files.createTempDirectory("qortal-"); + File tempFile = Paths.get(tempDirectory.toString(), filename).toFile(); tempFile.deleteOnExit(); Files.write(tempFile.toPath(), Base64.decode(base64)); path = tempFile.toPath().toString(); @@ -1169,12 +1235,17 @@ public class ArbitraryResource { // The actual data will be in a randomly-named subfolder of tempDirectory // Remove hidden folders, i.e. starting with "_", as some systems can add them, e.g. "__MACOSX" String[] files = tempDirectory.toFile().list((parent, child) -> !child.startsWith("_")); - if (files.length == 1) { // Single directory or file only + if (files != null && files.length == 1) { // Single directory or file only path = Paths.get(tempDirectory.toString(), files[0]).toString(); } } } + // Finish here if user has requested a preview + if (preview != null && preview == true) { + return this.preview(path, service); + } + // Default to zero fee if not specified if (fee == null) { fee = 0L; @@ -1196,12 +1267,13 @@ public class ArbitraryResource { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_DATA, e.getMessage()); } - } catch (DataException | IOException e) { + } catch (Exception e) { + LOGGER.info("Exception when publishing data: ", e); throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.REPOSITORY_ISSUE, e.getMessage()); } } - private HttpServletResponse download(Service service, String name, String identifier, String filepath, boolean rebuild, boolean async, Integer maxAttempts) { + private HttpServletResponse download(Service service, String name, String identifier, String filepath, String encoding, boolean rebuild, boolean async, Integer maxAttempts) { ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); try { @@ -1244,7 +1316,7 @@ public class ArbitraryResource { if (filepath == null || filepath.isEmpty()) { // No file path supplied - so check if this is a single file resource String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal"); - if (files.length == 1) { + if (files != null && files.length == 1) { // This is a single file resource filepath = files[0]; } @@ -1254,13 +1326,50 @@ public class ArbitraryResource { } } - // TODO: limit file size that can be read into memory java.nio.file.Path path = Paths.get(outputPath.toString(), filepath); if (!Files.exists(path)) { String message = String.format("No file exists at filepath: %s", filepath); throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, message); } - byte[] data = Files.readAllBytes(path); + + byte[] data; + int fileSize = (int)path.toFile().length(); + int length = fileSize; + + // Parse "Range" header + Integer rangeStart = null; + Integer rangeEnd = null; + String range = request.getHeader("Range"); + if (range != null) { + range = range.replace("bytes=", ""); + String[] parts = range.split("-"); + rangeStart = (parts != null && parts.length > 0) ? Integer.parseInt(parts[0]) : null; + rangeEnd = (parts != null && parts.length > 1) ? Integer.parseInt(parts[1]) : fileSize; + } + + if (rangeStart != null && rangeEnd != null) { + // We have a range, so update the requested length + length = rangeEnd - rangeStart; + } + + if (length < fileSize && encoding == null) { + // Partial content requested, and not encoding the data + response.setStatus(206); + response.addHeader("Content-Range", String.format("bytes %d-%d/%d", rangeStart, rangeEnd-1, fileSize)); + data = FilesystemUtils.readFromFile(path.toString(), rangeStart, length); + } + else { + // Full content requested (or encoded data) + response.setStatus(200); + data = Files.readAllBytes(path); // TODO: limit file size that can be read into memory + } + + // Encode the data if requested + if (encoding != null && Objects.equals(encoding.toLowerCase(), "base64")) { + data = Base64.encode(data); + } + + response.addHeader("Accept-Ranges", "bytes"); response.setContentType(context.getMimeType(path.toString())); response.setContentLength(data.length); response.getOutputStream().write(data); @@ -1272,41 +1381,44 @@ public class ArbitraryResource { } } + private FileProperties getFileProperties(Service service, String name, String identifier) { + ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); + try { + arbitraryDataReader.loadSynchronously(false); + java.nio.file.Path outputPath = arbitraryDataReader.getFilePath(); + if (outputPath == null) { + // Assume the resource doesn't exist + throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, "File not found"); + } - private List addStatusToResources(List resources) { - // Determine and add the status of each resource - List updatedResources = new ArrayList<>(); - for (ArbitraryResourceInfo resourceInfo : resources) { - try { - ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME, - resourceInfo.service, resourceInfo.identifier); - ArbitraryResourceStatus status = resource.getStatus(true); - if (status != null) { - resourceInfo.status = status; + FileProperties fileProperties = new FileProperties(); + fileProperties.size = FileUtils.sizeOfDirectory(outputPath.toFile()); + + String[] files = ArrayUtils.removeElement(outputPath.toFile().list(), ".qortal"); + if (files.length == 1) { + String filename = files[0]; + java.nio.file.Path filePath = Paths.get(outputPath.toString(), files[0]); + ContentInfoUtil util = new ContentInfoUtil(); + ContentInfo info = util.findMatch(filePath.toFile()); + String mimeType; + if (info != null) { + // Attempt to extract MIME type from file contents + mimeType = info.getMimeType(); } - updatedResources.add(resourceInfo); - - } catch (Exception e) { - // Catch and log all exceptions, since some systems are experiencing 500 errors when including statuses - LOGGER.info("Caught exception when adding status to resource %s: %s", resourceInfo, e.toString()); + else { + // Fall back to using the filename + FileNameMap fileNameMap = URLConnection.getFileNameMap(); + mimeType = fileNameMap.getContentTypeFor(filename); + } + fileProperties.filename = filename; + fileProperties.mimeType = mimeType; } - } - return updatedResources; - } - private List addMetadataToResources(List resources) { - // Add metadata fields to each resource if they exist - List updatedResources = new ArrayList<>(); - for (ArbitraryResourceInfo resourceInfo : resources) { - ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ResourceIdType.NAME, - resourceInfo.service, resourceInfo.identifier); - ArbitraryDataTransactionMetadata transactionMetadata = resource.getLatestTransactionMetadata(); - ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata, false); - if (resourceMetadata != null) { - resourceInfo.metadata = resourceMetadata; - } - updatedResources.add(resourceInfo); + return fileProperties; + + } catch (Exception e) { + LOGGER.debug(String.format("Unable to load %s %s: %s", service, name, e.getMessage())); + throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.FILE_NOT_FOUND, e.getMessage()); } - return updatedResources; } } diff --git a/src/main/java/org/qortal/api/resource/ChatResource.java b/src/main/java/org/qortal/api/resource/ChatResource.java index 150b6f63..22e90a43 100644 --- a/src/main/java/org/qortal/api/resource/ChatResource.java +++ b/src/main/java/org/qortal/api/resource/ChatResource.java @@ -40,6 +40,8 @@ import org.qortal.utils.Base58; import com.google.common.primitives.Bytes; +import static org.qortal.data.chat.ChatMessage.Encoding; + @Path("/chat") @Tag(name = "Chat") public class ChatResource { @@ -73,6 +75,7 @@ public class ChatResource { @QueryParam("chatreference") String chatReference, @QueryParam("haschatreference") Boolean hasChatReference, @QueryParam("sender") String sender, + @QueryParam("encoding") Encoding encoding, @Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset, @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) { @@ -109,12 +112,82 @@ public class ChatResource { hasChatReference, involvingAddresses, sender, + encoding, limit, offset, reverse); } catch (DataException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); } } + @GET + @Path("/messages/count") + @Operation( + summary = "Count chat messages", + description = "Returns count of CHAT messages that match criteria. Must provide EITHER 'txGroupId' OR two 'involving' addresses.", + responses = { + @ApiResponse( + description = "count of messages", + content = @Content( + mediaType = MediaType.TEXT_PLAIN, + schema = @Schema( + type = "integer" + ) + ) + ) + } + ) + @ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE}) + public int countChatMessages(@QueryParam("before") Long before, @QueryParam("after") Long after, + @QueryParam("txGroupId") Integer txGroupId, + @QueryParam("involving") List involvingAddresses, + @QueryParam("reference") String reference, + @QueryParam("chatreference") String chatReference, + @QueryParam("haschatreference") Boolean hasChatReference, + @QueryParam("sender") String sender, + @QueryParam("encoding") Encoding encoding, + @Parameter(ref = "limit") @QueryParam("limit") Integer limit, + @Parameter(ref = "offset") @QueryParam("offset") Integer offset, + @Parameter(ref = "reverse") @QueryParam("reverse") Boolean reverse) { + // Check args meet expectations + if ((txGroupId == null && involvingAddresses.size() != 2) + || (txGroupId != null && !involvingAddresses.isEmpty())) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); + + // Check any provided addresses are valid + if (involvingAddresses.stream().anyMatch(address -> !Crypto.isValidAddress(address))) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS); + + if (before != null && before < 1500000000000L) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); + + if (after != null && after < 1500000000000L) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA); + + byte[] referenceBytes = null; + if (reference != null) + referenceBytes = Base58.decode(reference); + + byte[] chatReferenceBytes = null; + if (chatReference != null) + chatReferenceBytes = Base58.decode(chatReference); + + try (final Repository repository = RepositoryManager.getRepository()) { + return repository.getChatRepository().getMessagesMatchingCriteria( + before, + after, + txGroupId, + referenceBytes, + chatReferenceBytes, + hasChatReference, + involvingAddresses, + sender, + encoding, + limit, offset, reverse).size(); + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + @GET @Path("/message/{signature}") @Operation( @@ -131,7 +204,7 @@ public class ChatResource { } ) @ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE}) - public ChatMessage getMessageBySignature(@PathParam("signature") String signature58) { + public ChatMessage getMessageBySignature(@PathParam("signature") String signature58, @QueryParam("encoding") Encoding encoding) { byte[] signature = Base58.decode(signature58); try (final Repository repository = RepositoryManager.getRepository()) { @@ -141,7 +214,7 @@ public class ChatResource { throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Message not found"); } - return repository.getChatRepository().toChatMessage(chatTransactionData); + return repository.getChatRepository().toChatMessage(chatTransactionData, encoding); } catch (DataException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); } @@ -164,12 +237,12 @@ public class ChatResource { } ) @ApiErrors({ApiError.INVALID_CRITERIA, ApiError.INVALID_ADDRESS, ApiError.REPOSITORY_ISSUE}) - public ActiveChats getActiveChats(@PathParam("address") String address) { + public ActiveChats getActiveChats(@PathParam("address") String address, @QueryParam("encoding") Encoding encoding) { if (address == null || !Crypto.isValidAddress(address)) throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_ADDRESS); try (final Repository repository = RepositoryManager.getRepository()) { - return repository.getChatRepository().getActiveChats(address); + return repository.getChatRepository().getActiveChats(address, encoding); } catch (DataException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); } diff --git a/src/main/java/org/qortal/api/resource/CrossChainResource.java b/src/main/java/org/qortal/api/resource/CrossChainResource.java index bb7c70a5..44ef62ad 100644 --- a/src/main/java/org/qortal/api/resource/CrossChainResource.java +++ b/src/main/java/org/qortal/api/resource/CrossChainResource.java @@ -115,6 +115,9 @@ public class CrossChainResource { crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp)); } + // Remove any trades that have had too many failures + crossChainTrades = TradeBot.getInstance().removeFailedTrades(repository, crossChainTrades); + if (limit != null && limit > 0) { // Make sure to not return more than the limit int upperLimit = Math.min(limit, crossChainTrades.size()); @@ -129,6 +132,64 @@ public class CrossChainResource { } } + @GET + @Path("/tradeoffers/hidden") + @Operation( + summary = "Find cross-chain trade offers that have been hidden due to too many failures", + responses = { + @ApiResponse( + content = @Content( + array = @ArraySchema( + schema = @Schema( + implementation = CrossChainTradeData.class + ) + ) + ) + ) + } + ) + @ApiErrors({ApiError.INVALID_CRITERIA, ApiError.REPOSITORY_ISSUE}) + public List getHiddenTradeOffers( + @Parameter( + description = "Limit to specific blockchain", + example = "LITECOIN", + schema = @Schema(implementation = SupportedBlockchain.class) + ) @QueryParam("foreignBlockchain") SupportedBlockchain foreignBlockchain) { + + final boolean isExecutable = true; + List crossChainTrades = new ArrayList<>(); + + try (final Repository repository = RepositoryManager.getRepository()) { + Map> acctsByCodeHash = SupportedBlockchain.getFilteredAcctMap(foreignBlockchain); + + for (Map.Entry> acctInfo : acctsByCodeHash.entrySet()) { + byte[] codeHash = acctInfo.getKey().value; + ACCT acct = acctInfo.getValue().get(); + + List atsData = repository.getATRepository().getATsByFunctionality(codeHash, isExecutable, null, null, null); + + for (ATData atData : atsData) { + CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atData); + if (crossChainTradeData.mode == AcctMode.OFFERING) { + crossChainTrades.add(crossChainTradeData); + } + } + } + + // Sort the trades by timestamp + crossChainTrades.sort((a, b) -> Longs.compare(a.creationTimestamp, b.creationTimestamp)); + + // Remove trades that haven't failed + crossChainTrades.removeIf(t -> !TradeBot.getInstance().isFailedTrade(repository, t)); + + crossChainTrades.stream().forEach(CrossChainResource::decorateTradeDataWithPresence); + + return crossChainTrades; + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + @GET @Path("/trade/{ataddress}") @Operation( diff --git a/src/main/java/org/qortal/api/resource/NamesResource.java b/src/main/java/org/qortal/api/resource/NamesResource.java index a900d6bf..6cde26b3 100644 --- a/src/main/java/org/qortal/api/resource/NamesResource.java +++ b/src/main/java/org/qortal/api/resource/NamesResource.java @@ -47,6 +47,7 @@ import org.qortal.transform.transaction.RegisterNameTransactionTransformer; import org.qortal.transform.transaction.SellNameTransactionTransformer; import org.qortal.transform.transaction.UpdateNameTransactionTransformer; import org.qortal.utils.Base58; +import org.qortal.utils.Unicode; @Path("/names") @Tag(name = "Names") @@ -63,19 +64,19 @@ public class NamesResource { description = "registered name info", content = @Content( mediaType = MediaType.APPLICATION_JSON, - array = @ArraySchema(schema = @Schema(implementation = NameSummary.class)) + array = @ArraySchema(schema = @Schema(implementation = NameData.class)) ) ) } ) @ApiErrors({ApiError.REPOSITORY_ISSUE}) - public List getAllNames(@Parameter(ref = "limit") @QueryParam("limit") Integer limit, @Parameter(ref = "offset") @QueryParam("offset") Integer offset, - @Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) { + public List getAllNames(@Parameter(description = "Return only names registered or updated after timestamp") @QueryParam("after") Long after, + @Parameter(ref = "limit") @QueryParam("limit") Integer limit, + @Parameter(ref = "offset") @QueryParam("offset") Integer offset, + @Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) { try (final Repository repository = RepositoryManager.getRepository()) { - List names = repository.getNameRepository().getAllNames(limit, offset, reverse); - // Convert to summary - return names.stream().map(NameSummary::new).collect(Collectors.toList()); + return repository.getNameRepository().getAllNames(after, limit, offset, reverse); } catch (DataException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); } @@ -135,12 +136,13 @@ public class NamesResource { public NameData getName(@PathParam("name") String name) { try (final Repository repository = RepositoryManager.getRepository()) { NameData nameData; + String reducedName = Unicode.sanitize(name); if (Settings.getInstance().isLite()) { nameData = LiteNode.getInstance().fetchNameData(name); } else { - nameData = repository.getNameRepository().fromName(name); + nameData = repository.getNameRepository().fromReducedName(reducedName); } if (nameData == null) { @@ -155,6 +157,41 @@ public class NamesResource { } } + @GET + @Path("/search") + @Operation( + summary = "Search registered names", + responses = { + @ApiResponse( + description = "registered name info", + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + array = @ArraySchema(schema = @Schema(implementation = NameData.class)) + ) + ) + } + ) + @ApiErrors({ApiError.NAME_UNKNOWN, ApiError.REPOSITORY_ISSUE}) + public List searchNames(@QueryParam("query") String query, + @Parameter(description = "Prefix only (if true, only the beginning of the name is matched)") @QueryParam("prefix") Boolean prefixOnly, + @Parameter(ref = "limit") @QueryParam("limit") Integer limit, + @Parameter(ref = "offset") @QueryParam("offset") Integer offset, + @Parameter(ref="reverse") @QueryParam("reverse") Boolean reverse) { + try (final Repository repository = RepositoryManager.getRepository()) { + if (query == null) { + throw ApiExceptionFactory.INSTANCE.createCustomException(request, ApiError.INVALID_CRITERIA, "Missing query"); + } + + boolean usePrefixOnly = Boolean.TRUE.equals(prefixOnly); + + return repository.getNameRepository().searchNames(query, usePrefixOnly, limit, offset, reverse); + } catch (ApiException e) { + throw e; + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + @POST @Path("/register") @@ -410,4 +447,4 @@ public class NamesResource { } } -} \ No newline at end of file +} diff --git a/src/main/java/org/qortal/api/resource/PollsResource.java b/src/main/java/org/qortal/api/resource/PollsResource.java new file mode 100644 index 00000000..c64a8caf --- /dev/null +++ b/src/main/java/org/qortal/api/resource/PollsResource.java @@ -0,0 +1,258 @@ +package org.qortal.api.resource; + +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.parameters.RequestBody; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.tags.Tag; +import org.qortal.api.ApiError; +import org.qortal.api.ApiErrors; +import org.qortal.api.ApiExceptionFactory; +import org.qortal.data.transaction.CreatePollTransactionData; +import org.qortal.data.transaction.PaymentTransactionData; +import org.qortal.data.transaction.VoteOnPollTransactionData; +import org.qortal.repository.DataException; +import org.qortal.repository.Repository; +import org.qortal.repository.RepositoryManager; +import org.qortal.settings.Settings; +import org.qortal.transaction.Transaction; +import org.qortal.transform.TransformationException; +import org.qortal.transform.transaction.CreatePollTransactionTransformer; +import org.qortal.transform.transaction.PaymentTransactionTransformer; +import org.qortal.transform.transaction.VoteOnPollTransactionTransformer; +import org.qortal.utils.Base58; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; + +import io.swagger.v3.oas.annotations.Parameter; +import io.swagger.v3.oas.annotations.media.ArraySchema; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import javax.ws.rs.GET; +import javax.ws.rs.PathParam; +import javax.ws.rs.QueryParam; +import org.qortal.api.ApiException; +import org.qortal.api.model.PollVotes; +import org.qortal.data.voting.PollData; +import org.qortal.data.voting.PollOptionData; +import org.qortal.data.voting.VoteOnPollData; + +@Path("/polls") +@Tag(name = "Polls") +public class PollsResource { + @Context + HttpServletRequest request; + + @GET + @Operation( + summary = "List all polls", + responses = { + @ApiResponse( + description = "poll info", + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + array = @ArraySchema(schema = @Schema(implementation = PollData.class)) + ) + ) + } + ) + @ApiErrors({ApiError.REPOSITORY_ISSUE}) + public List getAllPolls(@Parameter( + ref = "limit" + ) @QueryParam("limit") Integer limit, @Parameter( + ref = "offset" + ) @QueryParam("offset") Integer offset, @Parameter( + ref = "reverse" + ) @QueryParam("reverse") Boolean reverse) { + try (final Repository repository = RepositoryManager.getRepository()) { + List allPollData = repository.getVotingRepository().getAllPolls(limit, offset, reverse); + return allPollData; + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + + @GET + @Path("/{pollName}") + @Operation( + summary = "Info on poll", + responses = { + @ApiResponse( + description = "poll info", + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = PollData.class) + ) + ) + } + ) + @ApiErrors({ApiError.REPOSITORY_ISSUE}) + public PollData getPollData(@PathParam("pollName") String pollName) { + try (final Repository repository = RepositoryManager.getRepository()) { + PollData pollData = repository.getVotingRepository().fromPollName(pollName); + if (pollData == null) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS); + + return pollData; + } catch (ApiException e) { + throw e; + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + + @GET + @Path("/votes/{pollName}") + @Operation( + summary = "Votes on poll", + responses = { + @ApiResponse( + description = "poll votes", + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + schema = @Schema(implementation = PollVotes.class) + ) + ) + } + ) + @ApiErrors({ApiError.REPOSITORY_ISSUE}) + public PollVotes getPollVotes(@PathParam("pollName") String pollName, @QueryParam("onlyCounts") Boolean onlyCounts) { + try (final Repository repository = RepositoryManager.getRepository()) { + PollData pollData = repository.getVotingRepository().fromPollName(pollName); + if (pollData == null) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.POLL_NO_EXISTS); + + List votes = repository.getVotingRepository().getVotes(pollName); + + // Initialize map for counting votes + Map voteCountMap = new HashMap<>(); + for (PollOptionData optionData : pollData.getPollOptions()) { + voteCountMap.put(optionData.getOptionName(), 0); + } + + int totalVotes = 0; + for (VoteOnPollData vote : votes) { + String selectedOption = pollData.getPollOptions().get(vote.getOptionIndex()).getOptionName(); + if (voteCountMap.containsKey(selectedOption)) { + voteCountMap.put(selectedOption, voteCountMap.get(selectedOption) + 1); + totalVotes++; + } + } + + // Convert map to list of VoteInfo + List voteCounts = voteCountMap.entrySet().stream() + .map(entry -> new PollVotes.OptionCount(entry.getKey(), entry.getValue())) + .collect(Collectors.toList()); + + if (onlyCounts != null && onlyCounts) { + return new PollVotes(null, totalVotes, voteCounts); + } else { + return new PollVotes(votes, totalVotes, voteCounts); + } + } catch (ApiException e) { + throw e; + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + + @POST + @Path("/create") + @Operation( + summary = "Build raw, unsigned, CREATE_POLL transaction", + requestBody = @RequestBody( + required = true, + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + schema = @Schema( + implementation = CreatePollTransactionData.class + ) + ) + ), + responses = { + @ApiResponse( + description = "raw, unsigned, CREATE_POLL transaction encoded in Base58", + content = @Content( + mediaType = MediaType.TEXT_PLAIN, + schema = @Schema( + type = "string" + ) + ) + ) + } + ) + @ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE}) + public String CreatePoll(CreatePollTransactionData transactionData) { + if (Settings.getInstance().isApiRestricted()) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION); + + try (final Repository repository = RepositoryManager.getRepository()) { + Transaction transaction = Transaction.fromData(repository, transactionData); + + Transaction.ValidationResult result = transaction.isValidUnconfirmed(); + if (result != Transaction.ValidationResult.OK) + throw TransactionsResource.createTransactionInvalidException(request, result); + + byte[] bytes = CreatePollTransactionTransformer.toBytes(transactionData); + return Base58.encode(bytes); + } catch (TransformationException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e); + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + + @POST + @Path("/vote") + @Operation( + summary = "Build raw, unsigned, VOTE_ON_POLL transaction", + requestBody = @RequestBody( + required = true, + content = @Content( + mediaType = MediaType.APPLICATION_JSON, + schema = @Schema( + implementation = VoteOnPollTransactionData.class + ) + ) + ), + responses = { + @ApiResponse( + description = "raw, unsigned, VOTE_ON_POLL transaction encoded in Base58", + content = @Content( + mediaType = MediaType.TEXT_PLAIN, + schema = @Schema( + type = "string" + ) + ) + ) + } + ) + @ApiErrors({ApiError.NON_PRODUCTION, ApiError.TRANSACTION_INVALID, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE}) + public String VoteOnPoll(VoteOnPollTransactionData transactionData) { + if (Settings.getInstance().isApiRestricted()) + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.NON_PRODUCTION); + + try (final Repository repository = RepositoryManager.getRepository()) { + Transaction transaction = Transaction.fromData(repository, transactionData); + + Transaction.ValidationResult result = transaction.isValidUnconfirmed(); + if (result != Transaction.ValidationResult.OK) + throw TransactionsResource.createTransactionInvalidException(request, result); + + byte[] bytes = VoteOnPollTransactionTransformer.toBytes(transactionData); + return Base58.encode(bytes); + } catch (TransformationException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e); + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + +} diff --git a/src/main/java/org/qortal/api/resource/StatsResource.java b/src/main/java/org/qortal/api/resource/StatsResource.java new file mode 100644 index 00000000..c1588490 --- /dev/null +++ b/src/main/java/org/qortal/api/resource/StatsResource.java @@ -0,0 +1,70 @@ +package org.qortal.api.resource; + +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.media.Content; +import io.swagger.v3.oas.annotations.media.Schema; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.tags.Tag; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.qortal.api.*; +import org.qortal.block.BlockChain; +import org.qortal.repository.DataException; +import org.qortal.repository.Repository; +import org.qortal.repository.RepositoryManager; +import org.qortal.utils.Amounts; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import java.math.BigDecimal; +import java.util.List; + +@Path("/stats") +@Tag(name = "Stats") +public class StatsResource { + + private static final Logger LOGGER = LogManager.getLogger(StatsResource.class); + + + @Context + HttpServletRequest request; + + @GET + @Path("/supply/circulating") + @Operation( + summary = "Fetch circulating QORT supply", + responses = { + @ApiResponse( + description = "circulating supply of QORT", + content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string", format = "number")) + ) + } + ) + public BigDecimal circulatingSupply() { + long total = 0L; + + try (final Repository repository = RepositoryManager.getRepository()) { + int currentHeight = repository.getBlockRepository().getBlockchainHeight(); + + List rewardsByHeight = BlockChain.getInstance().getBlockRewardsByHeight(); + int rewardIndex = rewardsByHeight.size() - 1; + BlockChain.RewardByHeight rewardInfo = rewardsByHeight.get(rewardIndex); + + for (int height = currentHeight; height > 1; --height) { + if (height < rewardInfo.height) { + --rewardIndex; + rewardInfo = rewardsByHeight.get(rewardIndex); + } + + total += rewardInfo.reward; + } + + return Amounts.toBigDecimal(total); + } catch (DataException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); + } + } + +} diff --git a/src/main/java/org/qortal/api/resource/TransactionsResource.java b/src/main/java/org/qortal/api/resource/TransactionsResource.java index a8962497..61cef867 100644 --- a/src/main/java/org/qortal/api/resource/TransactionsResource.java +++ b/src/main/java/org/qortal/api/resource/TransactionsResource.java @@ -9,6 +9,8 @@ import io.swagger.v3.oas.annotations.parameters.RequestBody; import io.swagger.v3.oas.annotations.responses.ApiResponse; import io.swagger.v3.oas.annotations.tags.Tag; +import java.io.IOException; +import java.io.StringWriter; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; import java.util.ArrayList; @@ -18,19 +20,12 @@ import java.util.concurrent.TimeUnit; import java.util.concurrent.locks.ReentrantLock; import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.GET; -import javax.ws.rs.POST; -import javax.ws.rs.Path; -import javax.ws.rs.PathParam; -import javax.ws.rs.QueryParam; +import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import org.qortal.account.PrivateKeyAccount; -import org.qortal.api.ApiError; -import org.qortal.api.ApiErrors; -import org.qortal.api.ApiException; -import org.qortal.api.ApiExceptionFactory; +import org.qortal.api.*; import org.qortal.api.model.SimpleTransactionSignRequest; import org.qortal.controller.Controller; import org.qortal.controller.LiteNode; @@ -724,7 +719,7 @@ public class TransactionsResource { ), responses = { @ApiResponse( - description = "true if accepted, false otherwise", + description = "For API version 1, this returns true if accepted.\nFor API version 2, the transactionData is returned as a JSON string if accepted.", content = @Content( mediaType = MediaType.TEXT_PLAIN, schema = @Schema( @@ -737,7 +732,9 @@ public class TransactionsResource { @ApiErrors({ ApiError.BLOCKCHAIN_NEEDS_SYNC, ApiError.INVALID_SIGNATURE, ApiError.INVALID_DATA, ApiError.TRANSFORMATION_ERROR, ApiError.REPOSITORY_ISSUE }) - public String processTransaction(String rawBytes58) { + public String processTransaction(String rawBytes58, @HeaderParam(ApiService.API_VERSION_HEADER) String apiVersionHeader) { + int apiVersion = ApiService.getApiVersion(request); + // Only allow a transaction to be processed if our latest block is less than 60 minutes old // If older than this, we should first wait until the blockchain is synced final Long minLatestBlockTimestamp = NTP.getTime() - (60 * 60 * 1000L); @@ -774,13 +771,27 @@ public class TransactionsResource { blockchainLock.unlock(); } - return "true"; + switch (apiVersion) { + case 1: + return "true"; + + case 2: + default: + // Marshall transactionData to string + StringWriter stringWriter = new StringWriter(); + ApiRequest.marshall(stringWriter, transactionData); + return stringWriter.toString(); + } + + } catch (NumberFormatException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA, e); } catch (DataException e) { throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE, e); } catch (InterruptedException e) { throw createTransactionInvalidException(request, ValidationResult.NO_BLOCKCHAIN_LOCK); + } catch (IOException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.TRANSFORMATION_ERROR, e); } } diff --git a/src/main/java/org/qortal/api/resource/AdminResource.java b/src/main/java/org/qortal/api/restricted/resource/AdminResource.java similarity index 93% rename from src/main/java/org/qortal/api/resource/AdminResource.java rename to src/main/java/org/qortal/api/restricted/resource/AdminResource.java index ef2a3f95..ecb8c6c9 100644 --- a/src/main/java/org/qortal/api/resource/AdminResource.java +++ b/src/main/java/org/qortal/api/restricted/resource/AdminResource.java @@ -1,4 +1,4 @@ -package org.qortal.api.resource; +package org.qortal.api.restricted.resource; import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.Parameter; @@ -20,6 +20,7 @@ import java.time.LocalDate; import java.time.LocalTime; import java.time.OffsetDateTime; import java.time.ZoneOffset; +import java.util.Arrays; import java.util.List; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; @@ -31,10 +32,13 @@ import javax.ws.rs.*; import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; +import org.apache.commons.lang3.reflect.FieldUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.core.LoggerContext; import org.apache.logging.log4j.core.appender.RollingFileAppender; +import org.json.JSONArray; +import org.json.JSONObject; import org.qortal.account.Account; import org.qortal.account.PrivateKeyAccount; import org.qortal.api.*; @@ -42,6 +46,7 @@ import org.qortal.api.model.ActivitySummary; import org.qortal.api.model.NodeInfo; import org.qortal.api.model.NodeStatus; import org.qortal.block.BlockChain; +import org.qortal.controller.AutoUpdate; import org.qortal.controller.Controller; import org.qortal.controller.Synchronizer; import org.qortal.controller.Synchronizer.SynchronizationResult; @@ -153,6 +158,53 @@ public class AdminResource { return nodeStatus; } + @GET + @Path("/settings") + @Operation( + summary = "Fetch node settings", + responses = { + @ApiResponse( + content = @Content(mediaType = MediaType.APPLICATION_JSON, schema = @Schema(implementation = Settings.class)) + ) + } + ) + public Settings settings() { + Settings nodeSettings = Settings.getInstance(); + + return nodeSettings; + } + + @GET + @Path("/settings/{setting}") + @Operation( + summary = "Fetch a single node setting", + responses = { + @ApiResponse( + content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string")) + ) + } + ) + public String setting(@PathParam("setting") String setting) { + try { + Object settingValue = FieldUtils.readField(Settings.getInstance(), setting, true); + if (settingValue == null) { + return "null"; + } + else if (settingValue instanceof String[]) { + JSONArray array = new JSONArray(settingValue); + return array.toString(4); + } + else if (settingValue instanceof List) { + JSONArray array = new JSONArray((List) settingValue); + return array.toString(4); + } + + return settingValue.toString(); + } catch (IllegalAccessException e) { + throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_CRITERIA, e); + } + } + @GET @Path("/stop") @Operation( @@ -183,6 +235,37 @@ public class AdminResource { return "true"; } + @GET + @Path("/restart") + @Operation( + summary = "Restart", + description = "Restart", + responses = { + @ApiResponse( + description = "\"true\"", + content = @Content(mediaType = MediaType.TEXT_PLAIN, schema = @Schema(type = "string")) + ) + } + ) + @SecurityRequirement(name = "apiKey") + public String restart(@HeaderParam(Security.API_KEY_HEADER) String apiKey) { + Security.checkApiCallAllowed(request); + + new Thread(() -> { + // Short sleep to allow HTTP response body to be emitted + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + // Not important + } + + AutoUpdate.attemptRestart(); + + }).start(); + + return "true"; + } + @GET @Path("/summary") @Operation( diff --git a/src/main/java/org/qortal/api/resource/BootstrapResource.java b/src/main/java/org/qortal/api/restricted/resource/BootstrapResource.java similarity index 98% rename from src/main/java/org/qortal/api/resource/BootstrapResource.java rename to src/main/java/org/qortal/api/restricted/resource/BootstrapResource.java index 78630dfb..47b7cf42 100644 --- a/src/main/java/org/qortal/api/resource/BootstrapResource.java +++ b/src/main/java/org/qortal/api/restricted/resource/BootstrapResource.java @@ -1,4 +1,4 @@ -package org.qortal.api.resource; +package org.qortal.api.restricted.resource; import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.media.Content; diff --git a/src/main/java/org/qortal/api/resource/RenderResource.java b/src/main/java/org/qortal/api/restricted/resource/RenderResource.java similarity index 62% rename from src/main/java/org/qortal/api/resource/RenderResource.java rename to src/main/java/org/qortal/api/restricted/resource/RenderResource.java index 519e722d..7a772f9f 100644 --- a/src/main/java/org/qortal/api/resource/RenderResource.java +++ b/src/main/java/org/qortal/api/restricted/resource/RenderResource.java @@ -1,4 +1,4 @@ -package org.qortal.api.resource; +package org.qortal.api.restricted.resource; import javax.servlet.ServletContext; import javax.servlet.http.HttpServletRequest; @@ -8,7 +8,6 @@ import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import java.io.*; import java.nio.file.Paths; -import java.util.Map; import io.swagger.v3.oas.annotations.Operation; import io.swagger.v3.oas.annotations.media.Content; @@ -28,8 +27,8 @@ import org.qortal.arbitrary.exception.MissingDataException; import org.qortal.controller.arbitrary.ArbitraryDataRenderManager; import org.qortal.data.transaction.ArbitraryTransactionData.*; import org.qortal.repository.DataException; -import org.qortal.settings.Settings; import org.qortal.arbitrary.ArbitraryDataFile.*; +import org.qortal.settings.Settings; import org.qortal.utils.Base58; @@ -43,60 +42,6 @@ public class RenderResource { @Context HttpServletResponse response; @Context ServletContext context; - @POST - @Path("/preview") - @Operation( - summary = "Generate preview URL based on a user-supplied path and service", - requestBody = @RequestBody( - required = true, - content = @Content( - mediaType = MediaType.TEXT_PLAIN, - schema = @Schema( - type = "string", example = "/Users/user/Documents/MyStaticWebsite" - ) - ) - ), - responses = { - @ApiResponse( - description = "a temporary URL to preview the website", - content = @Content( - mediaType = MediaType.TEXT_PLAIN, - schema = @Schema( - type = "string" - ) - ) - ) - } - ) - @SecurityRequirement(name = "apiKey") - public String preview(@HeaderParam(Security.API_KEY_HEADER) String apiKey, String directoryPath) { - Security.checkApiCallAllowed(request); - Method method = Method.PUT; - Compression compression = Compression.ZIP; - - ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(Paths.get(directoryPath), - null, Service.WEBSITE, null, method, compression, - null, null, null, null); - try { - arbitraryDataWriter.save(); - } catch (IOException | DataException | InterruptedException | MissingDataException e) { - LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage()); - throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.REPOSITORY_ISSUE); - } catch (RuntimeException e) { - LOGGER.info("Unable to create arbitrary data file: {}", e.getMessage()); - throw ApiExceptionFactory.INSTANCE.createException(request, ApiError.INVALID_DATA); - } - - ArbitraryDataFile arbitraryDataFile = arbitraryDataWriter.getArbitraryDataFile(); - if (arbitraryDataFile != null) { - String digest58 = arbitraryDataFile.digest58(); - if (digest58 != null) { - return "http://localhost:12393/render/hash/" + digest58 + "?secret=" + Base58.encode(arbitraryDataFile.getSecret()); - } - } - return "Unable to generate preview URL"; - } - @POST @Path("/authorize/{resourceId}") @SecurityRequirement(name = "apiKey") @@ -140,8 +85,10 @@ public class RenderResource { @SecurityRequirement(name = "apiKey") public HttpServletResponse getIndexBySignature(@PathParam("signature") String signature, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); - return this.get(signature, ResourceIdType.SIGNATURE, null, "/", null, "/render/signature", true, true, theme); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); + + return this.get(signature, ResourceIdType.SIGNATURE, null, null, "/", null, "/render/signature", true, true, theme); } @GET @@ -149,8 +96,10 @@ public class RenderResource { @SecurityRequirement(name = "apiKey") public HttpServletResponse getPathBySignature(@PathParam("signature") String signature, @PathParam("path") String inPath, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); - return this.get(signature, ResourceIdType.SIGNATURE, null, inPath,null, "/render/signature", true, true, theme); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, signature, Service.WEBSITE, null); + + return this.get(signature, ResourceIdType.SIGNATURE, null, null, inPath,null, "/render/signature", true, true, theme); } @GET @@ -158,8 +107,10 @@ public class RenderResource { @SecurityRequirement(name = "apiKey") public HttpServletResponse getIndexByHash(@PathParam("hash") String hash58, @QueryParam("secret") String secret58, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); - return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, "/", secret58, "/render/hash", true, false, theme); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); + + return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, "/", secret58, "/render/hash", true, false, theme); } @GET @@ -168,8 +119,10 @@ public class RenderResource { public HttpServletResponse getPathByHash(@PathParam("hash") String hash58, @PathParam("path") String inPath, @QueryParam("secret") String secret58, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); - return this.get(hash58, ResourceIdType.FILE_HASH, Service.WEBSITE, inPath, secret58, "/render/hash", true, false, theme); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, hash58, Service.WEBSITE, null); + + return this.get(hash58, ResourceIdType.FILE_HASH, Service.ARBITRARY_DATA, null, inPath, secret58, "/render/hash", true, false, theme); } @GET @@ -178,10 +131,13 @@ public class RenderResource { public HttpServletResponse getPathByName(@PathParam("service") Service service, @PathParam("name") String name, @PathParam("path") String inPath, + @QueryParam("identifier") String identifier, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, name, service, null); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, name, service, null); + String prefix = String.format("/render/%s", service); - return this.get(name, ResourceIdType.NAME, service, inPath, null, prefix, true, true, theme); + return this.get(name, ResourceIdType.NAME, service, identifier, inPath, null, prefix, true, true, theme); } @GET @@ -189,19 +145,22 @@ public class RenderResource { @SecurityRequirement(name = "apiKey") public HttpServletResponse getIndexByName(@PathParam("service") Service service, @PathParam("name") String name, + @QueryParam("identifier") String identifier, @QueryParam("theme") String theme) { - Security.requirePriorAuthorization(request, name, service, null); + if (!Settings.getInstance().isQDNAuthBypassEnabled()) + Security.requirePriorAuthorization(request, name, service, null); + String prefix = String.format("/render/%s", service); - return this.get(name, ResourceIdType.NAME, service, "/", null, prefix, true, true, theme); + return this.get(name, ResourceIdType.NAME, service, identifier, "/", null, prefix, true, true, theme); } - private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String inPath, - String secret58, String prefix, boolean usePrefix, boolean async, String theme) { + private HttpServletResponse get(String resourceId, ResourceIdType resourceIdType, Service service, String identifier, + String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String theme) { - ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, inPath, - secret58, prefix, usePrefix, async, request, response, context); + ArbitraryDataRenderer renderer = new ArbitraryDataRenderer(resourceId, resourceIdType, service, identifier, inPath, + secret58, prefix, usePrefix, async, "render", request, response, context); if (theme != null) { renderer.setTheme(theme); diff --git a/src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java b/src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java index 405fe7e5..9ac9f87d 100644 --- a/src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java +++ b/src/main/java/org/qortal/api/websocket/ActiveChatsWebSocket.java @@ -2,7 +2,9 @@ package org.qortal.api.websocket; import java.io.IOException; import java.io.StringWriter; +import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.concurrent.atomic.AtomicReference; import org.eclipse.jetty.websocket.api.Session; @@ -21,6 +23,8 @@ import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; +import static org.qortal.data.chat.ChatMessage.Encoding; + @WebSocket @SuppressWarnings("serial") public class ActiveChatsWebSocket extends ApiWebSocket { @@ -62,7 +66,9 @@ public class ActiveChatsWebSocket extends ApiWebSocket { @OnWebSocketMessage public void onWebSocketMessage(Session session, String message) { - /* ignored */ + if (Objects.equals(message, "ping")) { + session.getRemote().sendStringByFuture("pong"); + } } private void onNotify(Session session, ChatTransactionData chatTransactionData, String ourAddress, AtomicReference previousOutput) { @@ -75,7 +81,7 @@ public class ActiveChatsWebSocket extends ApiWebSocket { } try (final Repository repository = RepositoryManager.getRepository()) { - ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress); + ActiveChats activeChats = repository.getChatRepository().getActiveChats(ourAddress, getTargetEncoding(session)); StringWriter stringWriter = new StringWriter(); @@ -93,4 +99,12 @@ public class ActiveChatsWebSocket extends ApiWebSocket { } } + private Encoding getTargetEncoding(Session session) { + // Default to Base58 if not specified, for backwards support + Map> queryParams = session.getUpgradeRequest().getParameterMap(); + List encodingList = queryParams.get("encoding"); + String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58"; + return Encoding.valueOf(encoding); + } + } diff --git a/src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java b/src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java index c6d7aaed..3046c1c1 100644 --- a/src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java +++ b/src/main/java/org/qortal/api/websocket/ChatMessagesWebSocket.java @@ -2,10 +2,7 @@ package org.qortal.api.websocket; import java.io.IOException; import java.io.StringWriter; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.Map; +import java.util.*; import org.eclipse.jetty.websocket.api.Session; import org.eclipse.jetty.websocket.api.WebSocketException; @@ -22,6 +19,8 @@ import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; +import static org.qortal.data.chat.ChatMessage.Encoding; + @WebSocket @SuppressWarnings("serial") public class ChatMessagesWebSocket extends ApiWebSocket { @@ -35,6 +34,16 @@ public class ChatMessagesWebSocket extends ApiWebSocket { @Override public void onWebSocketConnect(Session session) { Map> queryParams = session.getUpgradeRequest().getParameterMap(); + Encoding encoding = getTargetEncoding(session); + + List limitList = queryParams.get("limit"); + Integer limit = (limitList != null && limitList.size() == 1) ? Integer.parseInt(limitList.get(0)) : null; + + List offsetList = queryParams.get("offset"); + Integer offset = (offsetList != null && offsetList.size() == 1) ? Integer.parseInt(offsetList.get(0)) : null; + + List reverseList = queryParams.get("offset"); + Boolean reverse = (reverseList != null && reverseList.size() == 1) ? Boolean.getBoolean(reverseList.get(0)) : null; List txGroupIds = queryParams.get("txGroupId"); if (txGroupIds != null && txGroupIds.size() == 1) { @@ -50,7 +59,8 @@ public class ChatMessagesWebSocket extends ApiWebSocket { null, null, null, - null, null, null); + encoding, + limit, offset, reverse); sendMessages(session, chatMessages); } catch (DataException e) { @@ -81,7 +91,8 @@ public class ChatMessagesWebSocket extends ApiWebSocket { null, involvingAddresses, null, - null, null, null); + encoding, + limit, offset, reverse); sendMessages(session, chatMessages); } catch (DataException e) { @@ -107,7 +118,9 @@ public class ChatMessagesWebSocket extends ApiWebSocket { @OnWebSocketMessage public void onWebSocketMessage(Session session, String message) { - /* ignored */ + if (Objects.equals(message, "ping")) { + session.getRemote().sendStringByFuture("pong"); + } } private void onNotify(Session session, ChatTransactionData chatTransactionData, int txGroupId) { @@ -155,7 +168,7 @@ public class ChatMessagesWebSocket extends ApiWebSocket { // Convert ChatTransactionData to ChatMessage ChatMessage chatMessage; try (final Repository repository = RepositoryManager.getRepository()) { - chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData); + chatMessage = repository.getChatRepository().toChatMessage(chatTransactionData, getTargetEncoding(session)); } catch (DataException e) { // No output this time? return; @@ -164,4 +177,12 @@ public class ChatMessagesWebSocket extends ApiWebSocket { sendMessages(session, Collections.singletonList(chatMessage)); } + private Encoding getTargetEncoding(Session session) { + // Default to Base58 if not specified, for backwards support + Map> queryParams = session.getUpgradeRequest().getParameterMap(); + List encodingList = queryParams.get("encoding"); + String encoding = (encodingList != null && encodingList.size() == 1) ? encodingList.get(0) : "BASE58"; + return Encoding.valueOf(encoding); + } + } diff --git a/src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java b/src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java index 78c53dc3..9c48b018 100644 --- a/src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java +++ b/src/main/java/org/qortal/api/websocket/TradeOffersWebSocket.java @@ -24,6 +24,7 @@ import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory; import org.qortal.api.model.CrossChainOfferSummary; import org.qortal.controller.Controller; import org.qortal.controller.Synchronizer; +import org.qortal.controller.tradebot.TradeBot; import org.qortal.crosschain.SupportedBlockchain; import org.qortal.crosschain.ACCT; import org.qortal.crosschain.AcctMode; @@ -315,7 +316,7 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener { throw new DataException("Couldn't fetch historic trades from repository"); for (ATStateData historicAtState : historicAtStates) { - CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null); + CrossChainOfferSummary historicOfferSummary = produceSummary(repository, acct, historicAtState, null, null); if (!isHistoric.test(historicOfferSummary)) continue; @@ -330,8 +331,10 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener { } } - private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, Long timestamp) throws DataException { - CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState); + private static CrossChainOfferSummary produceSummary(Repository repository, ACCT acct, ATStateData atState, CrossChainTradeData crossChainTradeData, Long timestamp) throws DataException { + if (crossChainTradeData == null) { + crossChainTradeData = acct.populateTradeData(repository, atState); + } long atStateTimestamp; @@ -346,9 +349,16 @@ public class TradeOffersWebSocket extends ApiWebSocket implements Listener { private static List produceSummaries(Repository repository, ACCT acct, List atStates, Long timestamp) throws DataException { List offerSummaries = new ArrayList<>(); + for (ATStateData atState : atStates) { + CrossChainTradeData crossChainTradeData = acct.populateTradeData(repository, atState); - for (ATStateData atState : atStates) - offerSummaries.add(produceSummary(repository, acct, atState, timestamp)); + // Ignore trade if it has failed + if (TradeBot.getInstance().isFailedTrade(repository, crossChainTradeData)) { + continue; + } + + offerSummaries.add(produceSummary(repository, acct, atState, crossChainTradeData, timestamp)); + } return offerSummaries; } diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java index b6b17ea5..fba6a32b 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataBuilder.java @@ -54,10 +54,6 @@ public class ArbitraryDataBuilder { /** * Process transactions, but do not build anything * This is useful for checking the status of a given resource - * - * @throws DataException - * @throws IOException - * @throws MissingDataException */ public void process() throws DataException, IOException, MissingDataException { this.fetchTransactions(); @@ -69,10 +65,6 @@ public class ArbitraryDataBuilder { /** * Build the latest state of a given resource - * - * @throws DataException - * @throws IOException - * @throws MissingDataException */ public void build() throws DataException, IOException, MissingDataException { this.process(); diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java index 1e86ee98..71378461 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataFile.java @@ -4,6 +4,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata; import org.qortal.crypto.Crypto; +import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.repository.DataException; import org.qortal.settings.Settings; import org.qortal.utils.Base58; @@ -15,7 +16,6 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; import java.util.*; -import java.util.stream.Stream; import static java.util.Arrays.stream; import static java.util.stream.Collectors.toMap; @@ -79,17 +79,31 @@ public class ArbitraryDataFile { this.signature = signature; } - public ArbitraryDataFile(byte[] fileContent, byte[] signature) throws DataException { + public ArbitraryDataFile(byte[] fileContent, byte[] signature, boolean useTemporaryFile) throws DataException { if (fileContent == null) { LOGGER.error("fileContent is null"); return; } + this.chunks = new ArrayList<>(); this.hash58 = Base58.encode(Crypto.digest(fileContent)); this.signature = signature; LOGGER.trace(String.format("File digest: %s, size: %d bytes", this.hash58, fileContent.length)); - Path outputFilePath = getOutputFilePath(this.hash58, signature, true); + Path outputFilePath; + if (useTemporaryFile) { + try { + outputFilePath = Files.createTempFile("qortalRawData", null); + outputFilePath.toFile().deleteOnExit(); + } + catch (IOException e) { + throw new DataException(String.format("Unable to write data with hash %s to temporary file: %s", this.hash58, e.getMessage())); + } + } + else { + outputFilePath = getOutputFilePath(this.hash58, signature, true); + } + File outputFile = outputFilePath.toFile(); try (FileOutputStream outputStream = new FileOutputStream(outputFile)) { outputStream.write(fileContent); @@ -111,6 +125,41 @@ public class ArbitraryDataFile { return ArbitraryDataFile.fromHash58(Base58.encode(hash), signature); } + public static ArbitraryDataFile fromRawData(byte[] data, byte[] signature) throws DataException { + if (data == null) { + return null; + } + return new ArbitraryDataFile(data, signature, true); + } + + public static ArbitraryDataFile fromTransactionData(ArbitraryTransactionData transactionData) throws DataException { + ArbitraryDataFile arbitraryDataFile = null; + byte[] signature = transactionData.getSignature(); + byte[] data = transactionData.getData(); + + if (data == null) { + return null; + } + + // Create data file + switch (transactionData.getDataType()) { + case DATA_HASH: + arbitraryDataFile = ArbitraryDataFile.fromHash(data, signature); + break; + + case RAW_DATA: + arbitraryDataFile = ArbitraryDataFile.fromRawData(data, signature); + break; + } + + // Set metadata hash + if (arbitraryDataFile != null) { + arbitraryDataFile.setMetadataHash(transactionData.getMetadataHash()); + } + + return arbitraryDataFile; + } + public static ArbitraryDataFile fromPath(Path path, byte[] signature) { if (path == null) { return null; @@ -260,6 +309,11 @@ public class ArbitraryDataFile { this.chunks = new ArrayList<>(); if (file != null) { + if (file.exists() && file.length() <= chunkSize) { + // No need to split into chunks if we're already below the chunk size + return 0; + } + try (FileInputStream fileInputStream = new FileInputStream(file); BufferedInputStream bis = new BufferedInputStream(fileInputStream)) { @@ -388,12 +442,15 @@ public class ArbitraryDataFile { return false; } - public boolean deleteAll() { + public boolean deleteAll(boolean deleteMetadata) { // Delete the complete file boolean fileDeleted = this.delete(); - // Delete the metadata file - boolean metadataDeleted = this.deleteMetadata(); + // Delete the metadata file if requested + boolean metadataDeleted = false; + if (deleteMetadata) { + metadataDeleted = this.deleteMetadata(); + } // Delete the individual chunks boolean chunksDeleted = this.deleteAllChunks(); @@ -612,6 +669,22 @@ public class ArbitraryDataFile { return this.chunks.size(); } + public int fileCount() { + int fileCount = this.chunkCount(); + + if (fileCount == 0) { + // Transactions without any chunks can already be treated as a complete file + fileCount++; + } + + if (this.getMetadataHash() != null) { + // Add the metadata file + fileCount++; + } + + return fileCount; + } + public List getChunks() { return this.chunks; } diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java index 5f6695df..1fd388da 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataFileChunk.java @@ -18,7 +18,7 @@ public class ArbitraryDataFileChunk extends ArbitraryDataFile { } public ArbitraryDataFileChunk(byte[] fileContent, byte[] signature) throws DataException { - super(fileContent, signature); + super(fileContent, signature, false); } public static ArbitraryDataFileChunk fromHash58(String hash58, byte[] signature) throws DataException { diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java index 78723958..b9e62e56 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataReader.java @@ -9,7 +9,6 @@ import org.qortal.arbitrary.exception.MissingDataException; import org.qortal.arbitrary.misc.Service; import org.qortal.controller.arbitrary.ArbitraryDataBuildManager; import org.qortal.controller.arbitrary.ArbitraryDataManager; -import org.qortal.controller.arbitrary.ArbitraryDataStorageManager; import org.qortal.crypto.AES; import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData.*; @@ -19,10 +18,7 @@ import org.qortal.repository.RepositoryManager; import org.qortal.arbitrary.ArbitraryDataFile.*; import org.qortal.settings.Settings; import org.qortal.transform.Transformer; -import org.qortal.utils.ArbitraryTransactionUtils; -import org.qortal.utils.Base58; -import org.qortal.utils.FilesystemUtils; -import org.qortal.utils.ZipUtils; +import org.qortal.utils.*; import javax.crypto.BadPaddingException; import javax.crypto.IllegalBlockSizeException; @@ -38,6 +34,9 @@ import java.security.InvalidAlgorithmParameterException; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; public class ArbitraryDataReader { @@ -63,6 +62,10 @@ public class ArbitraryDataReader { // The resource being read ArbitraryDataResource arbitraryDataResource = null; + // Track resources that are currently being loaded, to avoid duplicate concurrent builds + // TODO: all builds could be handled by the build queue (even synchronous ones), to avoid the need for this + private static Map inProgress = Collections.synchronizedMap(new HashMap<>()); + public ArbitraryDataReader(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) { // Ensure names are always lowercase if (resourceIdType == ResourceIdType.NAME) { @@ -157,9 +160,6 @@ public class ArbitraryDataReader { * If no exception is thrown, you can then use getFilePath() to access the data immediately after returning * * @param overwrite - set to true to force rebuild an existing cache - * @throws IOException - * @throws DataException - * @throws MissingDataException */ public void loadSynchronously(boolean overwrite) throws DataException, IOException, MissingDataException { try { @@ -173,6 +173,12 @@ public class ArbitraryDataReader { this.arbitraryDataResource = this.createArbitraryDataResource(); + // Don't allow duplicate loads + if (!this.canStartLoading()) { + LOGGER.debug("Skipping duplicate load of {}", this.arbitraryDataResource); + return; + } + this.preExecute(); this.deleteExistingFiles(); this.fetch(); @@ -200,6 +206,7 @@ public class ArbitraryDataReader { private void preExecute() throws DataException { ArbitraryDataBuildManager.getInstance().setBuildInProgress(true); + this.checkEnabled(); this.createWorkingDirectory(); this.createUncompressedDirectory(); @@ -207,6 +214,9 @@ public class ArbitraryDataReader { private void postExecute() { ArbitraryDataBuildManager.getInstance().setBuildInProgress(false); + + this.arbitraryDataResource = this.createArbitraryDataResource(); + ArbitraryDataReader.inProgress.remove(this.arbitraryDataResource.getUniqueKey()); } private void checkEnabled() throws DataException { @@ -215,6 +225,17 @@ public class ArbitraryDataReader { } } + private boolean canStartLoading() { + // Avoid duplicate builds if we're already loading this resource + String uniqueKey = this.arbitraryDataResource.getUniqueKey(); + if (ArbitraryDataReader.inProgress.containsKey(uniqueKey)) { + return false; + } + ArbitraryDataReader.inProgress.put(uniqueKey, NTP.getTime()); + + return true; + } + private void createWorkingDirectory() throws DataException { try { Files.createDirectories(this.workingPath); @@ -226,7 +247,6 @@ public class ArbitraryDataReader { /** * Working directory should only be deleted on failure, since it is currently used to * serve a cached version of the resource for subsequent requests. - * @throws IOException */ private void deleteWorkingDirectory() { try { @@ -306,7 +326,7 @@ public class ArbitraryDataReader { break; default: - throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType.toString())); + throw new DataException(String.format("Unknown resource ID type specified: %s", resourceIdType)); } } @@ -362,11 +382,6 @@ public class ArbitraryDataReader { throw new DataException(String.format("Transaction data not found for signature %s", this.resourceId)); } - // Load hashes - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - // Load secret byte[] secret = transactionData.getSecret(); if (secret != null) { @@ -374,16 +389,17 @@ public class ArbitraryDataReader { } // Load data file(s) - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); ArbitraryTransactionUtils.checkAndRelocateMiscFiles(transactionData); - arbitraryDataFile.setMetadataHash(metadataHash); + if (arbitraryDataFile == null) { + throw new DataException(String.format("arbitraryDataFile is null")); + } if (!arbitraryDataFile.allFilesExist()) { - if (ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())) { + if (ListUtils.isNameBlocked(transactionData.getName())) { throw new DataException( String.format("Unable to request missing data for file %s because the name is blocked", arbitraryDataFile)); - } - else { + } else { // Ask the arbitrary data manager to fetch data for this transaction String message; if (this.canRequestMissingFiles) { @@ -394,8 +410,7 @@ public class ArbitraryDataReader { } else { message = String.format("Unable to reissue request for missing file %s for signature %s due to rate limit. Please try again later.", arbitraryDataFile, Base58.encode(transactionData.getSignature())); } - } - else { + } else { message = String.format("Missing data for file %s", arbitraryDataFile); } @@ -405,21 +420,25 @@ public class ArbitraryDataReader { } } - if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) { - // We have all the chunks but not the complete file, so join them - arbitraryDataFile.join(); + // Data hashes need some extra processing + if (transactionData.getDataType() == DataType.DATA_HASH) { + if (arbitraryDataFile.allChunksExist() && !arbitraryDataFile.exists()) { + // We have all the chunks but not the complete file, so join them + arbitraryDataFile.join(); + } + + // If the complete file still doesn't exist then something went wrong + if (!arbitraryDataFile.exists()) { + throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile)); + } + // Ensure the complete hash matches the joined chunks + if (!Arrays.equals(arbitraryDataFile.digest(), transactionData.getData())) { + // Delete the invalid file + arbitraryDataFile.delete(); + throw new DataException("Unable to validate complete file hash"); + } } - // If the complete file still doesn't exist then something went wrong - if (!arbitraryDataFile.exists()) { - throw new IOException(String.format("File doesn't exist: %s", arbitraryDataFile)); - } - // Ensure the complete hash matches the joined chunks - if (!Arrays.equals(arbitraryDataFile.digest(), digest)) { - // Delete the invalid file - arbitraryDataFile.delete(); - throw new DataException("Unable to validate complete file hash"); - } // Ensure the file's size matches the size reported by the transaction (throws a DataException if not) arbitraryDataFile.validateFileSize(transactionData.getSize()); @@ -450,6 +469,7 @@ public class ArbitraryDataReader { Path unencryptedPath = Paths.get(this.workingPath.toString(), "zipped.zip"); SecretKey aesKey = new SecretKeySpec(secret, 0, secret.length, "AES"); AES.decryptFile(algorithm, aesKey, this.filePath.toString(), unencryptedPath.toString()); + LOGGER.debug("Finished decrypting {} using algorithm {}", this.arbitraryDataResource, algorithm); // Replace filePath pointer with the encrypted file path // Don't delete the original ArbitraryDataFile, as this is handled in the cleanup phase @@ -484,7 +504,9 @@ public class ArbitraryDataReader { // Handle each type of compression if (compression == Compression.ZIP) { + LOGGER.debug("Unzipping {}...", this.arbitraryDataResource); ZipUtils.unzip(this.filePath.toString(), this.uncompressedPath.getParent().toString()); + LOGGER.debug("Finished unzipping {}", this.arbitraryDataResource); } else if (compression == Compression.NONE) { Files.createDirectories(this.uncompressedPath); @@ -520,10 +542,12 @@ public class ArbitraryDataReader { private void validate() throws IOException, DataException { if (this.service.isValidationRequired()) { + LOGGER.debug("Validating {}...", this.arbitraryDataResource); Service.ValidationResult result = this.service.validate(this.filePath); if (result != Service.ValidationResult.OK) { throw new DataException(String.format("Validation of %s failed: %s", this.service, result.toString())); } + LOGGER.debug("Finished validating {}", this.arbitraryDataResource); } } diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java index 847f2aa8..089a99ca 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataRenderer.java @@ -2,6 +2,7 @@ package org.qortal.arbitrary; import com.google.common.io.Resources; import org.apache.commons.io.FileUtils; +import org.apache.commons.lang3.ArrayUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.qortal.api.HTMLParser; @@ -34,36 +35,40 @@ public class ArbitraryDataRenderer { private final String resourceId; private final ResourceIdType resourceIdType; private final Service service; + private final String identifier; private String theme = "light"; private String inPath; private final String secret58; private final String prefix; private final boolean usePrefix; private final boolean async; + private final String qdnContext; private final HttpServletRequest request; private final HttpServletResponse response; private final ServletContext context; - public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String inPath, - String secret58, String prefix, boolean usePrefix, boolean async, + public ArbitraryDataRenderer(String resourceId, ResourceIdType resourceIdType, Service service, String identifier, + String inPath, String secret58, String prefix, boolean usePrefix, boolean async, String qdnContext, HttpServletRequest request, HttpServletResponse response, ServletContext context) { this.resourceId = resourceId; this.resourceIdType = resourceIdType; this.service = service; + this.identifier = identifier != null ? identifier : "default"; this.inPath = inPath; this.secret58 = secret58; this.prefix = prefix; this.usePrefix = usePrefix; this.async = async; + this.qdnContext = qdnContext; this.request = request; this.response = response; this.context = context; } public HttpServletResponse render() { - if (!inPath.startsWith(File.separator)) { - inPath = File.separator + inPath; + if (!inPath.startsWith("/")) { + inPath = "/" + inPath; } // Don't render data if QDN is disabled @@ -71,14 +76,14 @@ public class ArbitraryDataRenderer { return ArbitraryDataRenderer.getResponse(response, 500, "QDN is disabled in settings"); } - ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, null); + ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(resourceId, resourceIdType, service, identifier); arbitraryDataReader.setSecret58(secret58); // Optional, used for loading encrypted file hashes only try { if (!arbitraryDataReader.isCachedDataAvailable()) { // If async is requested, show a loading screen whilst build is in progress if (async) { arbitraryDataReader.loadAsynchronously(false, 10); - return this.getLoadingResponse(service, resourceId, theme); + return this.getLoadingResponse(service, resourceId, identifier, theme); } // Otherwise, loop until we have data @@ -111,23 +116,59 @@ public class ArbitraryDataRenderer { } String unzippedPath = path.toString(); + // Set path automatically for single file resources (except for apps, which handle routing differently) + String[] files = ArrayUtils.removeElement(new File(unzippedPath).list(), ".qortal"); + if (files.length == 1 && this.service != Service.APP) { + // This is a single file resource + inPath = files[0]; + } + try { String filename = this.getFilename(unzippedPath, inPath); - String filePath = Paths.get(unzippedPath, filename).toString(); + Path filePath = Paths.get(unzippedPath, filename); + boolean usingCustomRouting = false; + + // If the file doesn't exist, we may need to route the request elsewhere, or cleanup + if (!Files.exists(filePath)) { + if (inPath.equals("/")) { + // Delete the unzipped folder if no index file was found + try { + FileUtils.deleteDirectory(new File(unzippedPath)); + } catch (IOException e) { + LOGGER.debug("Unable to delete directory: {}", unzippedPath, e); + } + } + + // If this is an app, then forward all unhandled requests to the index, to give the app the option to route it + if (this.service == Service.APP) { + // Locate index file + List indexFiles = ArbitraryDataRenderer.indexFiles(); + for (String indexFile : indexFiles) { + Path indexPath = Paths.get(unzippedPath, indexFile); + if (Files.exists(indexPath)) { + // Forward request to index file + filePath = indexPath; + filename = indexFile; + usingCustomRouting = true; + break; + } + } + } + } if (HTMLParser.isHtmlFile(filename)) { // HTML file - needs to be parsed - byte[] data = Files.readAllBytes(Paths.get(filePath)); // TODO: limit file size that can be read into memory - HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data); + byte[] data = Files.readAllBytes(filePath); // TODO: limit file size that can be read into memory + HTMLParser htmlParser = new HTMLParser(resourceId, inPath, prefix, usePrefix, data, qdnContext, service, identifier, theme, usingCustomRouting); htmlParser.addAdditionalHeaderTags(); - response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' blob:; img-src 'self' data: blob:;"); + response.addHeader("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval'; media-src 'self' data: blob:; img-src 'self' data: blob:;"); response.setContentType(context.getMimeType(filename)); response.setContentLength(htmlParser.getData().length); response.getOutputStream().write(htmlParser.getData()); } else { // Regular file - can be streamed directly - File file = new File(filePath); + File file = filePath.toFile(); FileInputStream inputStream = new FileInputStream(file); response.addHeader("Content-Security-Policy", "default-src 'self'"); response.setContentType(context.getMimeType(filename)); @@ -143,14 +184,6 @@ public class ArbitraryDataRenderer { return response; } catch (FileNotFoundException | NoSuchFileException e) { LOGGER.info("Unable to serve file: {}", e.getMessage()); - if (inPath.equals("/")) { - // Delete the unzipped folder if no index file was found - try { - FileUtils.deleteDirectory(new File(unzippedPath)); - } catch (IOException ioException) { - LOGGER.debug("Unable to delete directory: {}", unzippedPath, e); - } - } } catch (IOException e) { LOGGER.info("Unable to serve file at path {}: {}", inPath, e.getMessage()); } @@ -172,7 +205,7 @@ public class ArbitraryDataRenderer { return userPath; } - private HttpServletResponse getLoadingResponse(Service service, String name, String theme) { + private HttpServletResponse getLoadingResponse(Service service, String name, String identifier, String theme) { String responseString = ""; URL url = Resources.getResource("loading/index.html"); try { @@ -181,6 +214,7 @@ public class ArbitraryDataRenderer { // Replace vars responseString = responseString.replace("%%SERVICE%%", service.toString()); responseString = responseString.replace("%%NAME%%", name); + responseString = responseString.replace("%%IDENTIFIER%%", identifier); responseString = responseString.replace("%%THEME%%", theme); } catch (IOException e) { diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java index 2720e4b2..a4650dfc 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataResource.java @@ -11,13 +11,13 @@ import org.qortal.controller.arbitrary.ArbitraryDataManager; import org.qortal.controller.arbitrary.ArbitraryDataStorageManager; import org.qortal.data.arbitrary.ArbitraryResourceStatus; import org.qortal.data.transaction.ArbitraryTransactionData; -import org.qortal.list.ResourceListManager; import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; import org.qortal.settings.Settings; import org.qortal.utils.ArbitraryTransactionUtils; import org.qortal.utils.FilesystemUtils; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; import java.io.IOException; @@ -43,6 +43,7 @@ public class ArbitraryDataResource { private int layerCount; private Integer localChunkCount = null; private Integer totalChunkCount = null; + private boolean exists = false; public ArbitraryDataResource(String resourceId, ResourceIdType resourceIdType, Service service, String identifier) { this.resourceId = resourceId.toLowerCase(); @@ -61,6 +62,10 @@ public class ArbitraryDataResource { // Avoid this for "quick" statuses, to speed things up if (!quick) { this.calculateChunkCounts(); + + if (!this.exists) { + return new ArbitraryResourceStatus(Status.NOT_PUBLISHED, this.localChunkCount, this.totalChunkCount); + } } if (resourceIdType != ResourceIdType.NAME) { @@ -69,8 +74,7 @@ public class ArbitraryDataResource { } // Check if the name is blocked - if (ResourceListManager.getInstance() - .listContains("blockedNames", this.resourceId, false)) { + if (ListUtils.isNameBlocked(this.resourceId)) { return new ArbitraryResourceStatus(Status.BLOCKED, this.localChunkCount, this.totalChunkCount); } @@ -135,21 +139,23 @@ public class ArbitraryDataResource { return null; } - public boolean delete() { + public boolean delete(boolean deleteMetadata) { try { this.fetchTransactions(); + if (this.transactions == null) { + return false; + } List transactionDataList = new ArrayList<>(this.transactions); for (ArbitraryTransactionData transactionData : transactionDataList) { - byte[] hash = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); + if (arbitraryDataFile == null) { + continue; + } // Delete any chunks or complete files from each transaction - arbitraryDataFile.deleteAll(); + arbitraryDataFile.deleteAll(deleteMetadata); } // Also delete cached data for the entire resource @@ -193,6 +199,9 @@ public class ArbitraryDataResource { try { this.fetchTransactions(); + if (this.transactions == null) { + return false; + } List transactionDataList = new ArrayList<>(this.transactions); @@ -212,6 +221,14 @@ public class ArbitraryDataResource { private void calculateChunkCounts() { try { this.fetchTransactions(); + if (this.transactions == null) { + this.exists = false; + this.localChunkCount = 0; + this.totalChunkCount = 0; + return; + } + + this.exists = true; List transactionDataList = new ArrayList<>(this.transactions); int localChunkCount = 0; @@ -231,6 +248,9 @@ public class ArbitraryDataResource { private boolean isRateLimited() { try { this.fetchTransactions(); + if (this.transactions == null) { + return true; + } List transactionDataList = new ArrayList<>(this.transactions); @@ -254,6 +274,10 @@ public class ArbitraryDataResource { private boolean isDataPotentiallyAvailable() { try { this.fetchTransactions(); + if (this.transactions == null) { + return false; + } + Long now = NTP.getTime(); if (now == null) { return false; @@ -285,6 +309,10 @@ public class ArbitraryDataResource { private boolean isDownloading() { try { this.fetchTransactions(); + if (this.transactions == null) { + return false; + } + Long now = NTP.getTime(); if (now == null) { return false; @@ -337,7 +365,10 @@ public class ArbitraryDataResource { this.transactions = transactionDataList; this.layerCount = transactionDataList.size(); - } catch (DataException e) { + } catch (DataNotPublishedException e) { + // Ignore without logging + } + catch (DataException e) { LOGGER.info(String.format("Repository error when fetching transactions for resource %s: %s", this, e.getMessage())); } } diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java index b27e511c..a9dd4fcf 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataTransactionBuilder.java @@ -9,6 +9,7 @@ import org.qortal.arbitrary.metadata.ArbitraryDataMetadataPatch; import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata; import org.qortal.arbitrary.misc.Category; import org.qortal.arbitrary.misc.Service; +import org.qortal.crypto.AES; import org.qortal.crypto.Crypto; import org.qortal.data.PaymentData; import org.qortal.data.transaction.ArbitraryTransactionData; @@ -181,6 +182,7 @@ public class ArbitraryDataTransactionBuilder { for (ModifiedPath path : metadata.getModifiedPaths()) { if (path.getDiffType() != DiffType.COMPLETE_FILE) { atLeastOnePatch = true; + break; } } } @@ -189,6 +191,14 @@ public class ArbitraryDataTransactionBuilder { return Method.PUT; } + // We can't use PATCH for on-chain data because this requires the .qortal directory, which can't be put on chain + final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(this.path, false); + final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE); + if (shouldUseOnChainData) { + LOGGER.info("Data size is small enough to go on chain - using PUT"); + return Method.PUT; + } + // State is appropriate for a PATCH transaction return Method.PATCH; } @@ -229,10 +239,12 @@ public class ArbitraryDataTransactionBuilder { random.nextBytes(lastReference); } - Compression compression = Compression.ZIP; + // Single file resources are handled differently, especially for very small data payloads, as these go on chain + final boolean isSingleFileResource = FilesystemUtils.isSingleFileResource(path, false); + final boolean shouldUseOnChainData = (isSingleFileResource && AES.getEncryptedFileSize(FilesystemUtils.getSingleFileContents(path).length) <= ArbitraryTransaction.MAX_DATA_SIZE); - // FUTURE? Use zip compression for directories, or no compression for single files - // Compression compression = (path.toFile().isDirectory()) ? Compression.ZIP : Compression.NONE; + // Use zip compression if data isn't going on chain + Compression compression = shouldUseOnChainData ? Compression.NONE : Compression.ZIP; ArbitraryDataWriter arbitraryDataWriter = new ArbitraryDataWriter(path, name, service, identifier, method, compression, title, description, tags, category); @@ -250,16 +262,21 @@ public class ArbitraryDataTransactionBuilder { throw new DataException("Arbitrary data file is null"); } - // Get chunks metadata file + // Get metadata file ArbitraryDataFile metadataFile = arbitraryDataFile.getMetadataFile(); if (metadataFile == null && arbitraryDataFile.chunkCount() > 1) { throw new DataException(String.format("Chunks metadata data file is null but there are %d chunks", arbitraryDataFile.chunkCount())); } - String digest58 = arbitraryDataFile.digest58(); - if (digest58 == null) { - LOGGER.error("Unable to calculate file digest"); - throw new DataException("Unable to calculate file digest"); + // Default to using a data hash, with data held off-chain + ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH; + byte[] data = arbitraryDataFile.digest(); + + // For small, single-chunk resources, we can store the data directly on chain + if (shouldUseOnChainData && arbitraryDataFile.getBytes().length <= ArbitraryTransaction.MAX_DATA_SIZE && arbitraryDataFile.chunkCount() == 0) { + // Within allowed on-chain data size + dataType = DataType.RAW_DATA; + data = arbitraryDataFile.getBytes(); } final BaseTransactionData baseTransactionData = new BaseTransactionData(now, Group.NO_GROUP, @@ -268,27 +285,29 @@ public class ArbitraryDataTransactionBuilder { final int version = 5; final int nonce = 0; byte[] secret = arbitraryDataFile.getSecret(); - final ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH; - final byte[] digest = arbitraryDataFile.digest(); + final byte[] metadataHash = (metadataFile != null) ? metadataFile.getHash() : null; final List payments = new ArrayList<>(); ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData, - version, service, nonce, size, name, identifier, method, - secret, compression, digest, dataType, metadataHash, payments); + version, service.value, nonce, size, name, identifier, method, + secret, compression, data, dataType, metadataHash, payments); this.arbitraryTransactionData = transactionData; - } catch (DataException e) { + } catch (DataException | IOException e) { if (arbitraryDataFile != null) { - arbitraryDataFile.deleteAll(); + arbitraryDataFile.deleteAll(true); } - throw(e); + throw new DataException(e); } } private boolean isMetadataEqual(ArbitraryDataTransactionMetadata existingMetadata) { + if (existingMetadata == null) { + return !this.hasMetadata(); + } if (!Objects.equals(existingMetadata.getTitle(), this.title)) { return false; } @@ -304,6 +323,10 @@ public class ArbitraryDataTransactionBuilder { return true; } + private boolean hasMetadata() { + return (this.title != null || this.description != null || this.category != null || this.tags != null); + } + public void computeNonce() throws DataException { if (this.arbitraryTransactionData == null) { throw new DataException("Arbitrary transaction data is required to compute nonce"); @@ -315,7 +338,7 @@ public class ArbitraryDataTransactionBuilder { Transaction.ValidationResult result = transaction.isValidUnconfirmed(); if (result != Transaction.ValidationResult.OK) { - arbitraryDataFile.deleteAll(); + arbitraryDataFile.deleteAll(true); throw new DataException(String.format("Arbitrary transaction invalid: %s", result)); } LOGGER.info("Transaction is valid"); diff --git a/src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java b/src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java index 8b1d00c3..db29ee20 100644 --- a/src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java +++ b/src/main/java/org/qortal/arbitrary/ArbitraryDataWriter.java @@ -1,5 +1,7 @@ package org.qortal.arbitrary; +import com.j256.simplemagic.ContentInfo; +import com.j256.simplemagic.ContentInfoUtil; import org.apache.commons.io.FileUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -23,6 +25,8 @@ import javax.crypto.NoSuchPaddingException; import javax.crypto.SecretKey; import java.io.File; import java.io.IOException; +import java.net.FileNameMap; +import java.net.URLConnection; import java.nio.file.*; import java.security.InvalidAlgorithmParameterException; import java.security.InvalidKeyException; @@ -48,6 +52,7 @@ public class ArbitraryDataWriter { private final List tags; private final Category category; private List files; + private String mimeType; private int chunkSize = ArbitraryDataFile.CHUNK_SIZE; @@ -79,6 +84,7 @@ public class ArbitraryDataWriter { this.tags = ArbitraryDataTransactionMetadata.limitTags(tags); this.category = category; this.files = new ArrayList<>(); // Populated in buildFileList() + this.mimeType = null; // Populated in buildFileList() } public void save() throws IOException, DataException, InterruptedException, MissingDataException { @@ -101,10 +107,9 @@ public class ArbitraryDataWriter { private void preExecute() throws DataException { this.checkEnabled(); - // Enforce compression when uploading a directory - File file = new File(this.filePath.toString()); - if (file.isDirectory() && compression == Compression.NONE) { - throw new DataException("Unable to upload a directory without compression"); + // Enforce compression when uploading multiple files + if (!FilesystemUtils.isSingleFileResource(this.filePath, false) && compression == Compression.NONE) { + throw new DataException("Unable to publish multiple files without compression"); } // Create temporary working directory @@ -144,20 +149,44 @@ public class ArbitraryDataWriter { } private void buildFileList() throws IOException { - // Single file resources consist of a single element in the file list + // Check if the path already points to a single file boolean isSingleFile = this.filePath.toFile().isFile(); + Path singleFilePath = null; if (isSingleFile) { this.files.add(this.filePath.getFileName().toString()); - return; + singleFilePath = this.filePath; + } + else { + // Multi file resources (or a single file in a directory) require a walk through the directory tree + try (Stream stream = Files.walk(this.filePath)) { + this.files = stream + .filter(Files::isRegularFile) + .map(p -> this.filePath.relativize(p).toString()) + .filter(s -> !s.isEmpty()) + .collect(Collectors.toList()); + + if (this.files.size() == 1) { + singleFilePath = Paths.get(this.filePath.toString(), this.files.get(0)); + + // Update filePath to point to the single file (instead of the directory containing the file) + this.filePath = singleFilePath; + } + } } - // Multi file resources require a walk through the directory tree - try (Stream stream = Files.walk(this.filePath)) { - this.files = stream - .filter(Files::isRegularFile) - .map(p -> this.filePath.relativize(p).toString()) - .filter(s -> !s.isEmpty()) - .collect(Collectors.toList()); + if (singleFilePath != null) { + // Single file resource, so try and determine the MIME type + ContentInfoUtil util = new ContentInfoUtil(); + ContentInfo info = util.findMatch(singleFilePath.toFile()); + if (info != null) { + // Attempt to extract MIME type from file contents + this.mimeType = info.getMimeType(); + } + else { + // Fall back to using the filename + FileNameMap fileNameMap = URLConnection.getFileNameMap(); + this.mimeType = fileNameMap.getContentTypeFor(singleFilePath.toFile().getName()); + } } } @@ -287,9 +316,6 @@ public class ArbitraryDataWriter { if (chunkCount > 0) { LOGGER.info(String.format("Successfully split into %d chunk%s", chunkCount, (chunkCount == 1 ? "" : "s"))); } - else { - throw new DataException("Unable to split file into chunks"); - } } private void createMetadataFile() throws IOException, DataException { @@ -304,6 +330,7 @@ public class ArbitraryDataWriter { metadata.setCategory(this.category); metadata.setChunks(this.arbitraryDataFile.chunkHashList()); metadata.setFiles(this.files); + metadata.setMimeType(this.mimeType); metadata.write(); // Create an ArbitraryDataFile from the JSON file (we don't have a signature yet) diff --git a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java index 127fefb5..498f3296 100644 --- a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java +++ b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadata.java @@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.json.JSONException; import org.qortal.repository.DataException; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; @@ -34,7 +36,7 @@ public class ArbitraryDataMetadata { this.filePath = filePath; } - protected void readJson() throws DataException { + protected void readJson() throws DataException, JSONException { // To be overridden } @@ -44,8 +46,13 @@ public class ArbitraryDataMetadata { public void read() throws IOException, DataException { - this.loadJson(); - this.readJson(); + try { + this.loadJson(); + this.readJson(); + + } catch (JSONException e) { + throw new DataException(String.format("Unable to read JSON at path %s: %s", this.filePath, e.getMessage())); + } } public void write() throws IOException, DataException { @@ -58,6 +65,10 @@ public class ArbitraryDataMetadata { writer.close(); } + public void delete() throws IOException { + Files.delete(this.filePath); + } + protected void loadJson() throws IOException { File metadataFile = new File(this.filePath.toString()); @@ -65,7 +76,7 @@ public class ArbitraryDataMetadata { throw new IOException(String.format("Metadata file doesn't exist: %s", this.filePath.toString())); } - this.jsonString = new String(Files.readAllBytes(this.filePath)); + this.jsonString = new String(Files.readAllBytes(this.filePath), StandardCharsets.UTF_8); } diff --git a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java index bd6bb219..e9b49298 100644 --- a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java +++ b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataCache.java @@ -1,5 +1,6 @@ package org.qortal.arbitrary.metadata; +import org.json.JSONException; import org.json.JSONObject; import org.qortal.repository.DataException; import org.qortal.utils.Base58; @@ -22,7 +23,7 @@ public class ArbitraryDataMetadataCache extends ArbitraryDataQortalMetadata { } @Override - protected void readJson() throws DataException { + protected void readJson() throws DataException, JSONException { if (this.jsonString == null) { throw new DataException("Patch JSON string is null"); } diff --git a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java index 954dcb03..46a1f57e 100644 --- a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java +++ b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataMetadataPatch.java @@ -3,6 +3,7 @@ package org.qortal.arbitrary.metadata; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.json.JSONArray; +import org.json.JSONException; import org.json.JSONObject; import org.qortal.arbitrary.ArbitraryDataDiff.*; import org.qortal.repository.DataException; @@ -40,7 +41,7 @@ public class ArbitraryDataMetadataPatch extends ArbitraryDataQortalMetadata { } @Override - protected void readJson() throws DataException { + protected void readJson() throws DataException, JSONException { if (this.jsonString == null) { throw new DataException("Patch JSON string is null"); } diff --git a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java index 4c188843..eb3d6cc9 100644 --- a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java +++ b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataQortalMetadata.java @@ -2,12 +2,14 @@ package org.qortal.arbitrary.metadata; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.json.JSONException; import org.qortal.repository.DataException; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; @@ -46,20 +48,6 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata { return null; } - protected void readJson() throws DataException { - // To be overridden - } - - protected void buildJson() { - // To be overridden - } - - - @Override - public void read() throws IOException, DataException { - this.loadJson(); - this.readJson(); - } @Override public void write() throws IOException, DataException { @@ -82,7 +70,7 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata { throw new IOException(String.format("Patch file doesn't exist: %s", path.toString())); } - this.jsonString = new String(Files.readAllBytes(path)); + this.jsonString = new String(Files.readAllBytes(path), StandardCharsets.UTF_8); } @@ -94,9 +82,4 @@ public class ArbitraryDataQortalMetadata extends ArbitraryDataMetadata { } } - - public String getJsonString() { - return this.jsonString; - } - } diff --git a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java index 33da343c..d9dba037 100644 --- a/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java +++ b/src/main/java/org/qortal/arbitrary/metadata/ArbitraryDataTransactionMetadata.java @@ -1,11 +1,13 @@ package org.qortal.arbitrary.metadata; import org.json.JSONArray; +import org.json.JSONException; import org.json.JSONObject; import org.qortal.arbitrary.misc.Category; import org.qortal.repository.DataException; import org.qortal.utils.Base58; +import java.nio.charset.StandardCharsets; import java.nio.file.Path; import java.util.ArrayList; import java.util.Arrays; @@ -20,9 +22,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { private List tags; private Category category; private List files; + private String mimeType; private static int MAX_TITLE_LENGTH = 80; - private static int MAX_DESCRIPTION_LENGTH = 500; + private static int MAX_DESCRIPTION_LENGTH = 240; private static int MAX_TAG_LENGTH = 20; private static int MAX_TAGS_COUNT = 5; @@ -32,7 +35,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { } @Override - protected void readJson() throws DataException { + protected void readJson() throws DataException, JSONException { if (this.jsonString == null) { throw new DataException("Transaction metadata JSON string is null"); } @@ -92,6 +95,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { } this.files = filesList; } + + if (metadata.has("mimeType")) { + this.mimeType = metadata.getString("mimeType"); + } } @Override @@ -134,6 +141,10 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { } outer.put("files", files); + if (this.mimeType != null && !this.mimeType.isEmpty()) { + outer.put("mimeType", this.mimeType); + } + this.jsonString = outer.toString(2); LOGGER.trace("Transaction metadata: {}", this.jsonString); } @@ -187,6 +198,14 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { return this.files; } + public void setMimeType(String mimeType) { + this.mimeType = mimeType; + } + + public String getMimeType() { + return this.mimeType; + } + public boolean containsChunk(byte[] chunk) { for (byte[] c : this.chunks) { if (Arrays.equals(c, chunk)) { @@ -199,6 +218,25 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { // Static helper methods + public static String trimUTF8String(String string, int maxLength) { + byte[] inputBytes = string.getBytes(StandardCharsets.UTF_8); + int length = Math.min(inputBytes.length, maxLength); + byte[] outputBytes = new byte[length]; + + System.arraycopy(inputBytes, 0, outputBytes, 0, length); + String result = new String(outputBytes, StandardCharsets.UTF_8); + + // check if last character is truncated + int lastIndex = result.length() - 1; + + if (lastIndex > 0 && result.charAt(lastIndex) != string.charAt(lastIndex)) { + // last character is truncated so remove the last character + return result.substring(0, lastIndex); + } + + return result; + } + public static String limitTitle(String title) { if (title == null) { return null; @@ -207,7 +245,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { return null; } - return title.substring(0, Math.min(title.length(), MAX_TITLE_LENGTH)); + return trimUTF8String(title, MAX_TITLE_LENGTH); } public static String limitDescription(String description) { @@ -218,7 +256,7 @@ public class ArbitraryDataTransactionMetadata extends ArbitraryDataMetadata { return null; } - return description.substring(0, Math.min(description.length(), MAX_DESCRIPTION_LENGTH)); + return trimUTF8String(description, MAX_DESCRIPTION_LENGTH); } public static List limitTags(List tags) { diff --git a/src/main/java/org/qortal/arbitrary/misc/Service.java b/src/main/java/org/qortal/arbitrary/misc/Service.java index 5ea1b7aa..94ca9252 100644 --- a/src/main/java/org/qortal/arbitrary/misc/Service.java +++ b/src/main/java/org/qortal/arbitrary/misc/Service.java @@ -8,17 +8,20 @@ import org.qortal.utils.FilesystemUtils; import java.io.File; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Path; import java.nio.file.Paths; import java.util.*; +import com.fasterxml.jackson.databind.ObjectMapper; + import static java.util.Arrays.stream; import static java.util.stream.Collectors.toMap; public enum Service { - AUTO_UPDATE(1, false, null, null), - ARBITRARY_DATA(100, false, null, null), - QCHAT_ATTACHMENT(120, true, 1024*1024L, null) { + AUTO_UPDATE(1, false, null, false, false, null), + ARBITRARY_DATA(100, false, null, false, false, null), + QCHAT_ATTACHMENT(120, true, 1024*1024L, true, false, null) { @Override public ValidationResult validate(Path path) throws IOException { ValidationResult superclassResult = super.validate(path); @@ -26,37 +29,31 @@ public enum Service { return superclassResult; } - // Custom validation function to require a single file, with a whitelisted extension - int fileCount = 0; File[] files = path.toFile().listFiles(); // If already a single file, replace the list with one that contains that file only if (files == null && path.toFile().isFile()) { files = new File[] { path.toFile() }; } - if (files != null) { - for (File file : files) { - if (file.getName().equals(".qortal")) { - continue; - } - if (file.isDirectory()) { - return ValidationResult.DIRECTORIES_NOT_ALLOWED; - } - final String extension = FilenameUtils.getExtension(file.getName()).toLowerCase(); - // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string - final List allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", ""); - if (extension == null || !allowedExtensions.contains(extension)) { - return ValidationResult.INVALID_FILE_EXTENSION; - } - fileCount++; + // Now validate the file's extension + if (files != null && files[0] != null) { + final String extension = FilenameUtils.getExtension(files[0].getName()).toLowerCase(); + // We must allow blank file extensions because these are used by data published from a plaintext or base64-encoded string + final List allowedExtensions = Arrays.asList("zip", "pdf", "txt", "odt", "ods", "doc", "docx", "xls", "xlsx", "ppt", "pptx", ""); + if (extension == null || !allowedExtensions.contains(extension)) { + return ValidationResult.INVALID_FILE_EXTENSION; } } - if (fileCount != 1) { - return ValidationResult.INVALID_FILE_COUNT; - } return ValidationResult.OK; } }, - WEBSITE(200, true, null, null) { + QCHAT_ATTACHMENT_PRIVATE(121, true, 1024*1024L, true, true, null), + ATTACHMENT(130, false, 50*1024*1024L, true, false, null), + ATTACHMENT_PRIVATE(131, true, 50*1024*1024L, true, true, null), + FILE(140, false, null, true, false, null), + FILE_PRIVATE(141, true, null, true, true, null), + FILES(150, false, null, false, false, null), + CHAIN_DATA(160, true, 239L, true, false, null), + WEBSITE(200, true, null, false, false, null) { @Override public ValidationResult validate(Path path) throws IOException { ValidationResult superclassResult = super.validate(path); @@ -78,23 +75,49 @@ public enum Service { return ValidationResult.MISSING_INDEX_FILE; } }, - GIT_REPOSITORY(300, false, null, null), - IMAGE(400, true, 10*1024*1024L, null), - THUMBNAIL(410, true, 500*1024L, null), - QCHAT_IMAGE(420, true, 500*1024L, null), - VIDEO(500, false, null, null), - AUDIO(600, false, null, null), - QCHAT_AUDIO(610, true, 10*1024*1024L, null), - QCHAT_VOICE(620, true, 10*1024*1024L, null), - BLOG(700, false, null, null), - BLOG_POST(777, false, null, null), - BLOG_COMMENT(778, false, null, null), - DOCUMENT(800, false, null, null), - LIST(900, true, null, null), - PLAYLIST(910, true, null, null), - APP(1000, false, null, null), - METADATA(1100, false, null, null), - GIF_REPOSITORY(1200, true, 25*1024*1024L, null) { + GIT_REPOSITORY(300, false, null, false, false, null), + IMAGE(400, true, 10*1024*1024L, true, false, null), + IMAGE_PRIVATE(401, true, 10*1024*1024L, true, true, null), + THUMBNAIL(410, true, 500*1024L, true, false, null), + QCHAT_IMAGE(420, true, 500*1024L, true, false, null), + VIDEO(500, false, null, true, false, null), + VIDEO_PRIVATE(501, true, null, true, true, null), + AUDIO(600, false, null, true, false, null), + AUDIO_PRIVATE(601, true, null, true, true, null), + QCHAT_AUDIO(610, true, 10*1024*1024L, true, false, null), + QCHAT_VOICE(620, true, 10*1024*1024L, true, false, null), + VOICE(630, true, 10*1024*1024L, true, false, null), + VOICE_PRIVATE(631, true, 10*1024*1024L, true, true, null), + PODCAST(640, false, null, true, false, null), + BLOG(700, false, null, false, false, null), + BLOG_POST(777, false, null, true, false, null), + BLOG_COMMENT(778, true, 500*1024L, true, false, null), + DOCUMENT(800, false, null, true, false, null), + DOCUMENT_PRIVATE(801, true, null, true, true, null), + LIST(900, true, null, true, false, null), + PLAYLIST(910, true, null, true, false, null), + APP(1000, true, 50*1024*1024L, false, false, null), + METADATA(1100, false, null, true, false, null), + JSON(1110, true, 25*1024L, true, false, null) { + @Override + public ValidationResult validate(Path path) throws IOException { + ValidationResult superclassResult = super.validate(path); + if (superclassResult != ValidationResult.OK) { + return superclassResult; + } + + // Require valid JSON + byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024); + String json = new String(data, StandardCharsets.UTF_8); + try { + objectMapper.readTree(json); + return ValidationResult.OK; + } catch (IOException e) { + return ValidationResult.INVALID_CONTENT; + } + } + }, + GIF_REPOSITORY(1200, true, 25*1024*1024L, false, false, null) { @Override public ValidationResult validate(Path path) throws IOException { ValidationResult superclassResult = super.validate(path); @@ -129,20 +152,47 @@ public enum Service { } return ValidationResult.OK; } - }; + }, + STORE(1300, false, null, true, false, null), + PRODUCT(1310, false, null, true, false, null), + OFFER(1330, false, null, true, false, null), + COUPON(1340, false, null, true, false, null), + CODE(1400, false, null, true, false, null), + PLUGIN(1410, false, null, true, false, null), + EXTENSION(1420, false, null, true, false, null), + GAME(1500, false, null, false, false, null), + ITEM(1510, false, null, true, false, null), + NFT(1600, false, null, true, false, null), + DATABASE(1700, false, null, false, false, null), + SNAPSHOT(1710, false, null, false, false, null), + COMMENT(1800, true, 500*1024L, true, false, null), + CHAIN_COMMENT(1810, true, 239L, true, false, null), + MAIL(1900, true, 1024*1024L, true, false, null), + MAIL_PRIVATE(1901, true, 1024*1024L, true, true, null), + MESSAGE(1910, true, 1024*1024L, true, false, null), + MESSAGE_PRIVATE(1911, true, 1024*1024L, true, true, null); public final int value; private final boolean requiresValidation; private final Long maxSize; + private final boolean single; + private final boolean isPrivate; private final List requiredKeys; private static final Map map = stream(Service.values()) .collect(toMap(service -> service.value, service -> service)); - Service(int value, boolean requiresValidation, Long maxSize, List requiredKeys) { + // For JSON validation + private static final ObjectMapper objectMapper = new ObjectMapper(); + + private static final String encryptedDataPrefix = "qortalEncryptedData"; + + Service(int value, boolean requiresValidation, Long maxSize, boolean single, boolean isPrivate, List requiredKeys) { this.value = value; this.requiresValidation = requiresValidation; this.maxSize = maxSize; + this.single = single; + this.isPrivate = isPrivate; this.requiredKeys = requiredKeys; } @@ -151,7 +201,9 @@ public enum Service { return ValidationResult.OK; } - byte[] data = FilesystemUtils.getSingleFileContents(path); + // Load the first 25KB of data. This only needs to be long enough to check the prefix + // and also to allow for possible additional future validation of smaller files. + byte[] data = FilesystemUtils.getSingleFileContents(path, 25*1024); long size = FilesystemUtils.getDirectorySize(path); // Validate max size if needed @@ -161,6 +213,22 @@ public enum Service { } } + // Validate file count if needed + if (this.single && data == null) { + return ValidationResult.INVALID_FILE_COUNT; + } + + // Validate private data for single file resources + if (this.single) { + String dataString = new String(data, StandardCharsets.UTF_8); + if (this.isPrivate && !dataString.startsWith(encryptedDataPrefix)) { + return ValidationResult.DATA_NOT_ENCRYPTED; + } + if (!this.isPrivate && dataString.startsWith(encryptedDataPrefix)) { + return ValidationResult.DATA_ENCRYPTED; + } + } + // Validate required keys if needed if (this.requiredKeys != null) { if (data == null) { @@ -179,7 +247,12 @@ public enum Service { } public boolean isValidationRequired() { - return this.requiresValidation; + // We must always validate single file resources, to ensure they are actually a single file + return this.requiresValidation || this.single; + } + + public boolean isPrivate() { + return this.isPrivate; } public static Service valueOf(int value) { @@ -187,10 +260,41 @@ public enum Service { } public static JSONObject toJsonObject(byte[] data) { - String dataString = new String(data); + String dataString = new String(data, StandardCharsets.UTF_8); return new JSONObject(dataString); } + public static List publicServices() { + List privateServices = new ArrayList<>(); + for (Service service : Service.values()) { + if (!service.isPrivate) { + privateServices.add(service); + } + } + return privateServices; + } + + /** + * Fetch a list of Service objects that require encrypted data. + * + * These can ultimately be used to help inform the cleanup manager + * on the best order to delete files when the node runs out of space. + * Public data should be given priority over private data (unless + * this node is part of a data market contract for that data - this + * isn't developed yet). + * + * @return a list of Service objects that require encrypted data. + */ + public static List privateServices() { + List privateServices = new ArrayList<>(); + for (Service service : Service.values()) { + if (service.isPrivate) { + privateServices.add(service); + } + } + return privateServices; + } + public enum ValidationResult { OK(1), MISSING_KEYS(2), @@ -199,7 +303,10 @@ public enum Service { DIRECTORIES_NOT_ALLOWED(5), INVALID_FILE_EXTENSION(6), MISSING_DATA(7), - INVALID_FILE_COUNT(8); + INVALID_FILE_COUNT(8), + INVALID_CONTENT(9), + DATA_NOT_ENCRYPTED(10), + DATA_ENCRYPTED(10); public final int value; diff --git a/src/main/java/org/qortal/controller/AutoUpdate.java b/src/main/java/org/qortal/controller/AutoUpdate.java index 2ec7c94a..fde52fb1 100644 --- a/src/main/java/org/qortal/controller/AutoUpdate.java +++ b/src/main/java/org/qortal/controller/AutoUpdate.java @@ -293,4 +293,77 @@ public class AutoUpdate extends Thread { } } + public static boolean attemptRestart() { + LOGGER.info(String.format("Restarting node...")); + + // Give repository a chance to backup in case things go badly wrong (if enabled) + if (Settings.getInstance().getRepositoryBackupInterval() > 0) { + try { + // Timeout if the database isn't ready for backing up after 60 seconds + long timeout = 60 * 1000L; + RepositoryManager.backup(true, "backup", timeout); + + } catch (TimeoutException e) { + LOGGER.info("Attempt to backup repository failed due to timeout: {}", e.getMessage()); + // Continue with the node restart anyway... + } + } + + // Call ApplyUpdate to end this process (unlocking current JAR so it can be replaced) + String javaHome = System.getProperty("java.home"); + LOGGER.debug(String.format("Java home: %s", javaHome)); + + Path javaBinary = Paths.get(javaHome, "bin", "java"); + LOGGER.debug(String.format("Java binary: %s", javaBinary)); + + try { + List javaCmd = new ArrayList<>(); + // Java runtime binary itself + javaCmd.add(javaBinary.toString()); + + // JVM arguments + javaCmd.addAll(ManagementFactory.getRuntimeMXBean().getInputArguments()); + + // Disable, but retain, any -agentlib JVM arg as sub-process might fail if it tries to reuse same port + javaCmd = javaCmd.stream() + .map(arg -> arg.replace("-agentlib", AGENTLIB_JVM_HOLDER_ARG)) + .collect(Collectors.toList()); + + // Remove JNI options as they won't be supported by command-line 'java' + // These are typically added by the AdvancedInstaller Java launcher EXE + javaCmd.removeAll(Arrays.asList("abort", "exit", "vfprintf")); + + // Call ApplyUpdate using JAR + javaCmd.addAll(Arrays.asList("-cp", JAR_FILENAME, ApplyUpdate.class.getCanonicalName())); + + // Add command-line args saved from start-up + String[] savedArgs = Controller.getInstance().getSavedArgs(); + if (savedArgs != null) + javaCmd.addAll(Arrays.asList(savedArgs)); + + LOGGER.info(String.format("Restarting node with: %s", String.join(" ", javaCmd))); + + SysTray.getInstance().showMessage(Translator.INSTANCE.translate("SysTray", "AUTO_UPDATE"), //TODO + Translator.INSTANCE.translate("SysTray", "APPLYING_UPDATE_AND_RESTARTING"), //TODO + MessageType.INFO); + + ProcessBuilder processBuilder = new ProcessBuilder(javaCmd); + + // New process will inherit our stdout and stderr + processBuilder.redirectOutput(ProcessBuilder.Redirect.INHERIT); + processBuilder.redirectError(ProcessBuilder.Redirect.INHERIT); + + Process process = processBuilder.start(); + + // Nothing to pipe to new process, so close output stream (process's stdin) + process.getOutputStream().close(); + + return true; // restarting node OK + } catch (Exception e) { + LOGGER.error(String.format("Failed to restart node: %s", e.getMessage())); + + return true; // repo was okay, even if applying update failed + } + } + } diff --git a/src/main/java/org/qortal/controller/BlockMinter.java b/src/main/java/org/qortal/controller/BlockMinter.java index 185dd7cd..bc879f23 100644 --- a/src/main/java/org/qortal/controller/BlockMinter.java +++ b/src/main/java/org/qortal/controller/BlockMinter.java @@ -432,6 +432,10 @@ public class BlockMinter extends Thread { // Unable to process block - report and discard LOGGER.error("Unable to process newly minted block?", e); newBlocks.clear(); + } catch (ArithmeticException e) { + // Unable to process block - report and discard + LOGGER.error("Unable to process newly minted block?", e); + newBlocks.clear(); } } finally { blockchainLock.unlock(); diff --git a/src/main/java/org/qortal/controller/OnlineAccountsManager.java b/src/main/java/org/qortal/controller/OnlineAccountsManager.java index fd2c38df..224228b8 100644 --- a/src/main/java/org/qortal/controller/OnlineAccountsManager.java +++ b/src/main/java/org/qortal/controller/OnlineAccountsManager.java @@ -504,110 +504,118 @@ public class OnlineAccountsManager { computeOurAccountsForTimestamp(onlineAccountsTimestamp); } - private boolean computeOurAccountsForTimestamp(long onlineAccountsTimestamp) { - List mintingAccounts; - try (final Repository repository = RepositoryManager.getRepository()) { - mintingAccounts = repository.getAccountRepository().getMintingAccounts(); + private boolean computeOurAccountsForTimestamp(Long onlineAccountsTimestamp) { + if (onlineAccountsTimestamp != null) { + List mintingAccounts; + try (final Repository repository = RepositoryManager.getRepository()) { + mintingAccounts = repository.getAccountRepository().getMintingAccounts(); - // We have no accounts to send - if (mintingAccounts.isEmpty()) + // We have no accounts to send + if (mintingAccounts.isEmpty()) + return false; + + // Only active reward-shares allowed + Iterator iterator = mintingAccounts.iterator(); + int i = 0; + while (iterator.hasNext()) { + MintingAccountData mintingAccountData = iterator.next(); + + RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey()); + if (rewardShareData == null) { + // Reward-share doesn't even exist - probably not a good sign + iterator.remove(); + continue; + } + + Account mintingAccount = new Account(repository, rewardShareData.getMinter()); + if (!mintingAccount.canMint()) { + // Minting-account component of reward-share can no longer mint - disregard + iterator.remove(); + continue; + } + + if (++i > 1 + 1) { + iterator.remove(); + continue; + } + } + } catch (DataException e) { + LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage())); return false; + } - // Only active reward-shares allowed - Iterator iterator = mintingAccounts.iterator(); - while (iterator.hasNext()) { - MintingAccountData mintingAccountData = iterator.next(); + byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp); + List ourOnlineAccounts = new ArrayList<>(); - RewardShareData rewardShareData = repository.getAccountRepository().getRewardShare(mintingAccountData.getPublicKey()); - if (rewardShareData == null) { - // Reward-share doesn't even exist - probably not a good sign - iterator.remove(); + int remaining = mintingAccounts.size(); + for (MintingAccountData mintingAccountData : mintingAccounts) { + remaining--; + byte[] privateKey = mintingAccountData.getPrivateKey(); + byte[] publicKey = Crypto.toPublicKey(privateKey); + + // We don't want to compute the online account nonce and signature again if it already exists + Set onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet()); + boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey)); + if (alreadyExists) { + this.hasOurOnlineAccounts = true; + + if (remaining > 0) { + // Move on to next account + continue; + } else { + // Everything exists, so return true + return true; + } + } + + // Generate bytes for mempow + byte[] mempowBytes; + try { + mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp); + } catch (IOException e) { + LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account..."); continue; } - Account mintingAccount = new Account(repository, rewardShareData.getMinter()); - if (!mintingAccount.canMint()) { - // Minting-account component of reward-share can no longer mint - disregard - iterator.remove(); - continue; - } - } - } catch (DataException e) { - LOGGER.warn(String.format("Repository issue trying to fetch minting accounts: %s", e.getMessage())); - return false; - } - - byte[] timestampBytes = Longs.toByteArray(onlineAccountsTimestamp); - List ourOnlineAccounts = new ArrayList<>(); - - int remaining = mintingAccounts.size(); - for (MintingAccountData mintingAccountData : mintingAccounts) { - remaining--; - byte[] privateKey = mintingAccountData.getPrivateKey(); - byte[] publicKey = Crypto.toPublicKey(privateKey); - - // We don't want to compute the online account nonce and signature again if it already exists - Set onlineAccounts = this.currentOnlineAccounts.computeIfAbsent(onlineAccountsTimestamp, k -> ConcurrentHashMap.newKeySet()); - boolean alreadyExists = onlineAccounts.stream().anyMatch(a -> Arrays.equals(a.getPublicKey(), publicKey)); - if (alreadyExists) { - this.hasOurOnlineAccounts = true; - - if (remaining > 0) { - // Move on to next account - continue; - } - else { - // Everything exists, so return true - return true; - } - } - - // Generate bytes for mempow - byte[] mempowBytes; - try { - mempowBytes = this.getMemoryPoWBytes(publicKey, onlineAccountsTimestamp); - } - catch (IOException e) { - LOGGER.info("Unable to create bytes for MemoryPoW. Moving on to next account..."); - continue; - } - - // Compute nonce - Integer nonce; - try { - nonce = this.computeMemoryPoW(mempowBytes, publicKey, onlineAccountsTimestamp); - if (nonce == null) { - // A nonce is required + // Compute nonce + Integer nonce; + try { + nonce = this.computeMemoryPoW(mempowBytes, publicKey, onlineAccountsTimestamp); + if (nonce == null) { + // A nonce is required + return false; + } + } catch (TimeoutException e) { + LOGGER.info(String.format("Timed out computing nonce for account %.8s", Base58.encode(publicKey))); return false; } - } catch (TimeoutException e) { - LOGGER.info(String.format("Timed out computing nonce for account %.8s", Base58.encode(publicKey))); + + byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes); + + // Our account is online + OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce); + + // Make sure to verify before adding + if (verifyMemoryPoW(ourOnlineAccountData, null)) { + ourOnlineAccounts.add(ourOnlineAccountData); + } + } + + this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty(); + + boolean hasInfoChanged = addAccounts(ourOnlineAccounts); + + if (!hasInfoChanged) return false; - } - byte[] signature = Qortal25519Extras.signForAggregation(privateKey, timestampBytes); + Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts)); - // Our account is online - OnlineAccountData ourOnlineAccountData = new OnlineAccountData(onlineAccountsTimestamp, signature, publicKey, nonce); + LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp); - // Make sure to verify before adding - if (verifyMemoryPoW(ourOnlineAccountData, null)) { - ourOnlineAccounts.add(ourOnlineAccountData); - } + return true; } - this.hasOurOnlineAccounts = !ourOnlineAccounts.isEmpty(); - - boolean hasInfoChanged = addAccounts(ourOnlineAccounts); - - if (!hasInfoChanged) - return false; - - Network.getInstance().broadcast(peer -> new OnlineAccountsV3Message(ourOnlineAccounts)); - - LOGGER.debug("Broadcasted {} online account{} with timestamp {}", ourOnlineAccounts.size(), (ourOnlineAccounts.size() != 1 ? "s" : ""), onlineAccountsTimestamp); - - return true; + return false; } diff --git a/src/main/java/org/qortal/controller/PirateChainWalletController.java b/src/main/java/org/qortal/controller/PirateChainWalletController.java index 333c2cda..90e65329 100644 --- a/src/main/java/org/qortal/controller/PirateChainWalletController.java +++ b/src/main/java/org/qortal/controller/PirateChainWalletController.java @@ -163,7 +163,7 @@ public class PirateChainWalletController extends Thread { // Library not found, so check if we've fetched the resource from QDN ArbitraryTransactionData t = this.getTransactionData(repository); - if (t == null) { + if (t == null || t.getService() == null) { // Can't find the transaction - maybe on a different chain? return; } diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java index 34acf0cb..e0c62acb 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataCleanupManager.java @@ -11,10 +11,7 @@ import org.qortal.repository.RepositoryManager; import org.qortal.settings.Settings; import org.qortal.transaction.Transaction; import org.qortal.transaction.Transaction.TransactionType; -import org.qortal.utils.ArbitraryTransactionUtils; -import org.qortal.utils.Base58; -import org.qortal.utils.FilesystemUtils; -import org.qortal.utils.NTP; +import org.qortal.utils.*; import java.io.File; import java.io.IOException; @@ -137,7 +134,7 @@ public class ArbitraryDataCleanupManager extends Thread { // Fetch the transaction data ArbitraryTransactionData arbitraryTransactionData = ArbitraryTransactionUtils.fetchTransactionData(repository, signature); - if (arbitraryTransactionData == null) { + if (arbitraryTransactionData == null || arbitraryTransactionData.getService() == null) { continue; } @@ -239,7 +236,7 @@ public class ArbitraryDataCleanupManager extends Thread { // Delete random data associated with name if we're over our storage limit for this name // Use the DELETION_THRESHOLD, for the same reasons as above - for (String followedName : storageManager.followedNames()) { + for (String followedName : ListUtils.followedNames()) { if (isStopping) { return; } @@ -349,6 +346,10 @@ public class ArbitraryDataCleanupManager extends Thread { /** * Iteratively walk through given directory and delete a single random file * + * TODO: public data should be prioritized over private data + * (unless this node is part of a data market contract for that data). + * See: Service.privateServices() for a list of services containing private data. + * * @param directory - the base directory * @return boolean - whether a file was deleted */ @@ -487,7 +488,7 @@ public class ArbitraryDataCleanupManager extends Thread { // Delete data relating to blocked names String name = directory.getName(); - if (name != null && storageManager.isNameBlocked(name)) { + if (name != null && ListUtils.isNameBlocked(name)) { this.safeDeleteDirectory(directory, "blocked name"); } diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java index 63a6df80..5ed8df21 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java @@ -20,6 +20,7 @@ import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; import org.qortal.settings.Settings; import org.qortal.utils.Base58; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; import org.qortal.utils.Triple; @@ -123,29 +124,29 @@ public class ArbitraryDataFileListManager { } } - // Then allow another 3 attempts, each 5 minutes apart - if (timeSinceLastAttempt > 5 * 60 * 1000L) { - // We haven't tried for at least 5 minutes + // Then allow another 5 attempts, each 1 minute apart + if (timeSinceLastAttempt > 60 * 1000L) { + // We haven't tried for at least 1 minute - if (networkBroadcastCount < 6) { - // We've made less than 6 total attempts + if (networkBroadcastCount < 8) { + // We've made less than 8 total attempts return true; } } - // Then allow another 4 attempts, each 30 minutes apart - if (timeSinceLastAttempt > 30 * 60 * 1000L) { - // We haven't tried for at least 5 minutes + // Then allow another 8 attempts, each 15 minutes apart + if (timeSinceLastAttempt > 15 * 60 * 1000L) { + // We haven't tried for at least 15 minutes - if (networkBroadcastCount < 10) { - // We've made less than 10 total attempts + if (networkBroadcastCount < 16) { + // We've made less than 16 total attempts return true; } } - // From then on, only try once every 24 hours, to reduce network spam - if (timeSinceLastAttempt > 24 * 60 * 60 * 1000L) { - // We haven't tried for at least 24 hours + // From then on, only try once every 6 hours, to reduce network spam + if (timeSinceLastAttempt > 6 * 60 * 60 * 1000L) { + // We haven't tried for at least 6 hours return true; } @@ -258,8 +259,6 @@ public class ArbitraryDataFileListManager { // Lookup file lists by signature (and optionally hashes) public boolean fetchArbitraryDataFileList(ArbitraryTransactionData arbitraryTransactionData) { - byte[] digest = arbitraryTransactionData.getData(); - byte[] metadataHash = arbitraryTransactionData.getMetadataHash(); byte[] signature = arbitraryTransactionData.getSignature(); String signature58 = Base58.encode(signature); @@ -286,8 +285,7 @@ public class ArbitraryDataFileListManager { // Find hashes that we are missing try { - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); missingHashes = arbitraryDataFile.missingHashes(); } catch (DataException e) { // Leave missingHashes as null, so that all hashes are requested @@ -460,10 +458,9 @@ public class ArbitraryDataFileListManager { arbitraryTransactionData = (ArbitraryTransactionData) transactionData; - // Load data file(s) - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature); - arbitraryDataFile.setMetadataHash(arbitraryTransactionData.getMetadataHash()); - +// // Load data file(s) +// ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); +// // // Check all hashes exist // for (byte[] hash : hashes) { // //LOGGER.debug("Received hash {}", Base58.encode(hash)); @@ -507,7 +504,7 @@ public class ArbitraryDataFileListManager { // Forwarding if (isRelayRequest && Settings.getInstance().isRelayModeEnabled()) { - boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName())); + boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName())); if (!isBlocked) { Peer requestingPeer = request.getB(); if (requestingPeer != null) { @@ -594,12 +591,8 @@ public class ArbitraryDataFileListManager { // Check if we're even allowed to serve data for this transaction if (ArbitraryDataStorageManager.getInstance().canStoreData(transactionData)) { - byte[] hash = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - // Load file(s) and add any that exist to the list of hashes - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); // If the peer didn't supply a hash list, we need to return all hashes for this transaction if (requestedHashes == null || requestedHashes.isEmpty()) { @@ -690,7 +683,7 @@ public class ArbitraryDataFileListManager { } // We may need to forward this request on - boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())); + boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName())); if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) { // In relay mode - so ask our other peers if they have it diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java index e2de1ae0..48c41496 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileManager.java @@ -132,9 +132,7 @@ public class ArbitraryDataFileManager extends Thread { List hashes) throws DataException { // Load data file(s) - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(arbitraryTransactionData.getData(), signature); - byte[] metadataHash = arbitraryTransactionData.getMetadataHash(); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); boolean receivedAtLeastOneFile = false; // Now fetch actual data from this peer @@ -148,10 +146,10 @@ public class ArbitraryDataFileManager extends Thread { if (!arbitraryDataFileRequests.containsKey(Base58.encode(hash))) { LOGGER.debug("Requesting data file {} from peer {}", hash58, peer); Long startTime = NTP.getTime(); - ArbitraryDataFileMessage receivedArbitraryDataFileMessage = fetchArbitraryDataFile(peer, null, signature, hash, null); + ArbitraryDataFile receivedArbitraryDataFile = fetchArbitraryDataFile(peer, null, signature, hash, null); Long endTime = NTP.getTime(); - if (receivedArbitraryDataFileMessage != null && receivedArbitraryDataFileMessage.getArbitraryDataFile() != null) { - LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFileMessage.getArbitraryDataFile().getHash58(), peer, (endTime-startTime)); + if (receivedArbitraryDataFile != null) { + LOGGER.debug("Received data file {} from peer {}. Time taken: {} ms", receivedArbitraryDataFile.getHash58(), peer, (endTime-startTime)); receivedAtLeastOneFile = true; // Remove this hash from arbitraryDataFileHashResponses now that we have received it @@ -193,11 +191,11 @@ public class ArbitraryDataFileManager extends Thread { return receivedAtLeastOneFile; } - private ArbitraryDataFileMessage fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException { + private ArbitraryDataFile fetchArbitraryDataFile(Peer peer, Peer requestingPeer, byte[] signature, byte[] hash, Message originalMessage) throws DataException { ArbitraryDataFile existingFile = ArbitraryDataFile.fromHash(hash, signature); boolean fileAlreadyExists = existingFile.exists(); String hash58 = Base58.encode(hash); - ArbitraryDataFileMessage arbitraryDataFileMessage; + ArbitraryDataFile arbitraryDataFile; // Fetch the file if it doesn't exist locally if (!fileAlreadyExists) { @@ -227,28 +225,32 @@ public class ArbitraryDataFileManager extends Thread { } ArbitraryDataFileMessage peersArbitraryDataFileMessage = (ArbitraryDataFileMessage) response; - arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, peersArbitraryDataFileMessage.getArbitraryDataFile()); + arbitraryDataFile = peersArbitraryDataFileMessage.getArbitraryDataFile(); } else { LOGGER.debug(String.format("File hash %s already exists, so skipping the request", hash58)); - arbitraryDataFileMessage = new ArbitraryDataFileMessage(signature, existingFile); + arbitraryDataFile = existingFile; + } + + if (arbitraryDataFile == null) { + // We don't have a file, so give up here + return null; } // We might want to forward the request to the peer that originally requested it - this.handleArbitraryDataFileForwarding(requestingPeer, arbitraryDataFileMessage, originalMessage); + this.handleArbitraryDataFileForwarding(requestingPeer, new ArbitraryDataFileMessage(signature, arbitraryDataFile), originalMessage); boolean isRelayRequest = (requestingPeer != null); if (isRelayRequest) { if (!fileAlreadyExists) { // File didn't exist locally before the request, and it's a forwarding request, so delete it LOGGER.debug("Deleting file {} because it was needed for forwarding only", Base58.encode(hash)); - ArbitraryDataFile dataFile = arbitraryDataFileMessage.getArbitraryDataFile(); // Keep trying to delete the data until it is deleted, or we reach 10 attempts - dataFile.delete(10); + arbitraryDataFile.delete(10); } } - return arbitraryDataFileMessage; + return arbitraryDataFile; } private void handleFileListRequests(byte[] signature) { diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java index 2d1beadc..654c6844 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java @@ -114,7 +114,7 @@ public class ArbitraryDataFileRequestThread implements Runnable { return; } - LOGGER.debug("Fetching file {} from peer {} via request thread...", hash58, peer); + LOGGER.trace("Fetching file {} from peer {} via request thread...", hash58, peer); arbitraryDataFileManager.fetchArbitraryDataFiles(repository, peer, signature, arbitraryTransactionData, Arrays.asList(hash)); } catch (DataException e) { diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java index 6b3f0160..9284e672 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataManager.java @@ -16,7 +16,6 @@ import org.qortal.arbitrary.misc.Service; import org.qortal.controller.Controller; import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.TransactionData; -import org.qortal.list.ResourceListManager; import org.qortal.network.Network; import org.qortal.network.Peer; import org.qortal.repository.DataException; @@ -27,6 +26,7 @@ import org.qortal.transaction.ArbitraryTransaction; import org.qortal.transaction.Transaction.TransactionType; import org.qortal.utils.ArbitraryTransactionUtils; import org.qortal.utils.Base58; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; public class ArbitraryDataManager extends Thread { @@ -172,7 +172,7 @@ public class ArbitraryDataManager extends Thread { private void processNames() throws InterruptedException { // Fetch latest list of followed names - List followedNames = ResourceListManager.getInstance().getStringsInList("followedNames"); + List followedNames = ListUtils.followedNames(); if (followedNames == null || followedNames.isEmpty()) { return; } @@ -398,6 +398,11 @@ public class ArbitraryDataManager extends Thread { // Entrypoint to request new metadata from peers public ArbitraryDataTransactionMetadata fetchMetadata(ArbitraryTransactionData arbitraryTransactionData) { + if (arbitraryTransactionData.getService() == null) { + // Can't fetch metadata without a valid service + return null; + } + ArbitraryDataResource resource = new ArbitraryDataResource( arbitraryTransactionData.getName(), ArbitraryDataFile.ResourceIdType.NAME, @@ -489,7 +494,7 @@ public class ArbitraryDataManager extends Thread { public void invalidateCache(ArbitraryTransactionData arbitraryTransactionData) { String signature58 = Base58.encode(arbitraryTransactionData.getSignature()); - if (arbitraryTransactionData.getName() != null) { + if (arbitraryTransactionData.getName() != null && arbitraryTransactionData.getService() != null) { String resourceId = arbitraryTransactionData.getName().toLowerCase(); Service service = arbitraryTransactionData.getService(); String identifier = arbitraryTransactionData.getIdentifier(); diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java index ededbfa6..f6b2dc0a 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java @@ -5,15 +5,11 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.TransactionData; -import org.qortal.list.ResourceListManager; import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.settings.Settings; import org.qortal.transaction.Transaction; -import org.qortal.utils.ArbitraryTransactionUtils; -import org.qortal.utils.Base58; -import org.qortal.utils.FilesystemUtils; -import org.qortal.utils.NTP; +import org.qortal.utils.*; import java.io.IOException; import java.io.UncheckedIOException; @@ -61,6 +57,8 @@ public class ArbitraryDataStorageManager extends Thread { * This must be higher than STORAGE_FULL_THRESHOLD in order to avoid a fetch/delete loop. */ public static final double DELETION_THRESHOLD = 0.98f; // 98% + private static final long PER_NAME_STORAGE_MULTIPLIER = 4L; + public ArbitraryDataStorageManager() { } @@ -135,11 +133,11 @@ public class ArbitraryDataStorageManager extends Thread { case ALL: case VIEWED: // If the policy includes viewed data, we can host it as long as it's not blocked - return !this.isNameBlocked(name); + return !ListUtils.isNameBlocked(name); case FOLLOWED: // If the policy is for followed data only, we have to be following it - return this.isFollowingName(name); + return ListUtils.isFollowingName(name); // For NONE or all else, we shouldn't host this data case NONE: @@ -188,14 +186,14 @@ public class ArbitraryDataStorageManager extends Thread { } // Never fetch data from blocked names, even if they are followed - if (this.isNameBlocked(name)) { + if (ListUtils.isNameBlocked(name)) { return false; } switch (Settings.getInstance().getStoragePolicy()) { case FOLLOWED: case FOLLOWED_OR_VIEWED: - return this.isFollowingName(name); + return ListUtils.isFollowingName(name); case ALL: return true; @@ -235,7 +233,7 @@ public class ArbitraryDataStorageManager extends Thread { * @return boolean - whether the resource is blocked or not */ public boolean isBlocked(ArbitraryTransactionData arbitraryTransactionData) { - return isNameBlocked(arbitraryTransactionData.getName()); + return ListUtils.isNameBlocked(arbitraryTransactionData.getName()); } private boolean isDataTypeAllowed(ArbitraryTransactionData arbitraryTransactionData) { @@ -253,22 +251,6 @@ public class ArbitraryDataStorageManager extends Thread { return true; } - public boolean isNameBlocked(String name) { - return ResourceListManager.getInstance().listContains("blockedNames", name, false); - } - - private boolean isFollowingName(String name) { - return ResourceListManager.getInstance().listContains("followedNames", name, false); - } - - public List followedNames() { - return ResourceListManager.getInstance().getStringsInList("followedNames"); - } - - private int followedNamesCount() { - return ResourceListManager.getInstance().getItemCountForList("followedNames"); - } - public List loadAllHostedTransactions(Repository repository) { @@ -508,12 +490,17 @@ public class ArbitraryDataStorageManager extends Thread { return false; } + if (Settings.getInstance().getStoragePolicy() == StoragePolicy.ALL) { + // Using storage policy ALL, so don't limit anything per name + return true; + } + if (name == null) { // This transaction doesn't have a name, so fall back to total space limitations return true; } - int followedNamesCount = this.followedNamesCount(); + int followedNamesCount = ListUtils.followedNamesCount(); if (followedNamesCount == 0) { // Not following any names, so we have space return true; @@ -543,14 +530,16 @@ public class ArbitraryDataStorageManager extends Thread { } public long storageCapacityPerName(double threshold) { - int followedNamesCount = this.followedNamesCount(); + int followedNamesCount = ListUtils.followedNamesCount(); if (followedNamesCount == 0) { // Not following any names, so we have the total space available return this.getStorageCapacityIncludingThreshold(threshold); } double maxStorageCapacity = (double)this.storageCapacity * threshold; - long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount); + + // Some names won't need/use much space, so give all names a 4x multiplier to compensate + long maxStoragePerName = (long)(maxStorageCapacity / (double)followedNamesCount) * PER_NAME_STORAGE_MULTIPLIER; return maxStoragePerName; } diff --git a/src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java b/src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java index eec0d935..663bc22a 100644 --- a/src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java +++ b/src/main/java/org/qortal/controller/arbitrary/ArbitraryMetadataManager.java @@ -16,6 +16,7 @@ import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; import org.qortal.settings.Settings; import org.qortal.utils.Base58; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; import org.qortal.utils.Triple; @@ -101,7 +102,14 @@ public class ArbitraryMetadataManager { if (metadataFile.exists()) { // Use local copy ArbitraryDataTransactionMetadata transactionMetadata = new ArbitraryDataTransactionMetadata(metadataFile.getFilePath()); - transactionMetadata.read(); + try { + transactionMetadata.read(); + } catch (DataException e) { + // Invalid file, so delete it + LOGGER.info("Deleting invalid metadata file due to exception: {}", e.getMessage()); + transactionMetadata.delete(); + return null; + } return transactionMetadata; } } @@ -332,7 +340,7 @@ public class ArbitraryMetadataManager { } // Check if the name is blocked - boolean isBlocked = (arbitraryTransactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(arbitraryTransactionData.getName())); + boolean isBlocked = (arbitraryTransactionData == null || ListUtils.isNameBlocked(arbitraryTransactionData.getName())); if (!isBlocked) { Peer requestingPeer = request.getB(); if (requestingPeer != null) { @@ -420,7 +428,7 @@ public class ArbitraryMetadataManager { } // We may need to forward this request on - boolean isBlocked = (transactionData == null || ArbitraryDataStorageManager.getInstance().isNameBlocked(transactionData.getName())); + boolean isBlocked = (transactionData == null || ListUtils.isNameBlocked(transactionData.getName())); if (Settings.getInstance().isRelayModeEnabled() && !isBlocked) { // In relay mode - so ask our other peers if they have it diff --git a/src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java b/src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java index 004fa692..99eaf105 100644 --- a/src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java +++ b/src/main/java/org/qortal/controller/repository/NamesDatabaseIntegrityCheck.java @@ -13,7 +13,9 @@ import org.qortal.repository.RepositoryManager; import org.qortal.transaction.Transaction.TransactionType; import org.qortal.utils.Unicode; +import java.math.BigInteger; import java.util.*; +import java.util.stream.Collectors; public class NamesDatabaseIntegrityCheck { @@ -28,16 +30,8 @@ public class NamesDatabaseIntegrityCheck { private List nameTransactions = new ArrayList<>(); + public int rebuildName(String name, Repository repository) { - return this.rebuildName(name, repository, null); - } - - public int rebuildName(String name, Repository repository, List referenceNames) { - // "referenceNames" tracks the linked names that have already been rebuilt, to prevent circular dependencies - if (referenceNames == null) { - referenceNames = new ArrayList<>(); - } - int modificationCount = 0; try { List transactions = this.fetchAllTransactionsInvolvingName(name, repository); @@ -46,6 +40,14 @@ public class NamesDatabaseIntegrityCheck { return modificationCount; } + // If this name has been updated at any point, we need to add transactions from the other names to the sequence + int added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository); + while (added > 0) { + // Keep going until all have been added + LOGGER.trace("{} added for {}. Looking for more transactions...", added, name); + added = this.addAdditionalTransactionsRelatingToName(transactions, name, repository); + } + // Loop through each past transaction and re-apply it to the Names table for (TransactionData currentTransaction : transactions) { @@ -61,29 +63,14 @@ public class NamesDatabaseIntegrityCheck { // Process UPDATE_NAME transactions if (currentTransaction.getType() == TransactionType.UPDATE_NAME) { UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) currentTransaction; - - if (Objects.equals(updateNameTransactionData.getNewName(), name) && - !Objects.equals(updateNameTransactionData.getName(), updateNameTransactionData.getNewName())) { - // This renames an existing name, so we need to process that instead - - if (!referenceNames.contains(name)) { - referenceNames.add(name); - this.rebuildName(updateNameTransactionData.getName(), repository, referenceNames); - } - else { - // We've already processed this name so there's nothing more to do - } - } - else { - Name nameObj = new Name(repository, name); - if (nameObj != null && nameObj.getNameData() != null) { - nameObj.update(updateNameTransactionData); - modificationCount++; - LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name); - } else { - // Something went wrong - throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName())); - } + Name nameObj = new Name(repository, updateNameTransactionData.getName()); + if (nameObj != null && nameObj.getNameData() != null) { + nameObj.update(updateNameTransactionData); + modificationCount++; + LOGGER.trace("Processed UPDATE_NAME transaction for name {}", name); + } else { + // Something went wrong + throw new DataException(String.format("Name data not found for name %s", updateNameTransactionData.getName())); } } @@ -354,8 +341,8 @@ public class NamesDatabaseIntegrityCheck { } } - // Sort by lowest timestamp first - transactions.sort(Comparator.comparingLong(TransactionData::getTimestamp)); + // Sort by lowest block height first + sortTransactions(transactions); return transactions; } @@ -419,4 +406,67 @@ public class NamesDatabaseIntegrityCheck { return names; } + private int addAdditionalTransactionsRelatingToName(List transactions, String name, Repository repository) throws DataException { + int added = 0; + + // If this name has been updated at any point, we need to add transactions from the other names to the sequence + List otherNames = new ArrayList<>(); + List updateNameTransactions = transactions.stream().filter(t -> t.getType() == TransactionType.UPDATE_NAME).collect(Collectors.toList()); + for (TransactionData transactionData : updateNameTransactions) { + UpdateNameTransactionData updateNameTransactionData = (UpdateNameTransactionData) transactionData; + // If the newName field isn't empty, and either the "name" or "newName" is different from our reference name, + // we should remember this additional name, in case it has relevant transactions associated with it. + if (updateNameTransactionData.getNewName() != null && !updateNameTransactionData.getNewName().isEmpty()) { + if (!Objects.equals(updateNameTransactionData.getName(), name)) { + otherNames.add(updateNameTransactionData.getName()); + } + if (!Objects.equals(updateNameTransactionData.getNewName(), name)) { + otherNames.add(updateNameTransactionData.getNewName()); + } + } + } + + + for (String otherName : otherNames) { + List otherNameTransactions = this.fetchAllTransactionsInvolvingName(otherName, repository); + for (TransactionData otherNameTransactionData : otherNameTransactions) { + if (!transactions.contains(otherNameTransactionData)) { + // Add new transaction relating to other name + transactions.add(otherNameTransactionData); + added++; + } + } + } + + if (added > 0) { + // New transaction(s) added, so re-sort + sortTransactions(transactions); + } + + return added; + } + + private void sortTransactions(List transactions) { + Collections.sort(transactions, new Comparator() { + public int compare(Object o1, Object o2) { + TransactionData td1 = (TransactionData) o1; + TransactionData td2 = (TransactionData) o2; + + // Sort by block height first + int heightComparison = td1.getBlockHeight().compareTo(td2.getBlockHeight()); + if (heightComparison != 0) { + return heightComparison; + } + + // Same height so compare timestamps + int timestampComparison = Long.compare(td1.getTimestamp(), td2.getTimestamp()); + if (timestampComparison != 0) { + return timestampComparison; + } + + // Same timestamp so compare signatures + return new BigInteger(td1.getSignature()).compareTo(new BigInteger(td2.getSignature())); + }}); + } + } diff --git a/src/main/java/org/qortal/controller/tradebot/TradeBot.java b/src/main/java/org/qortal/controller/tradebot/TradeBot.java index 5880f561..96eeaf36 100644 --- a/src/main/java/org/qortal/controller/tradebot/TradeBot.java +++ b/src/main/java/org/qortal/controller/tradebot/TradeBot.java @@ -10,6 +10,7 @@ import org.apache.logging.log4j.Logger; import org.bitcoinj.core.ECKey; import org.qortal.account.PrivateKeyAccount; import org.qortal.api.model.crosschain.TradeBotCreateRequest; +import org.qortal.api.resource.TransactionsResource; import org.qortal.controller.Controller; import org.qortal.controller.Synchronizer; import org.qortal.controller.tradebot.AcctTradeBot.ResponseResult; @@ -19,6 +20,7 @@ import org.qortal.data.at.ATData; import org.qortal.data.crosschain.CrossChainTradeData; import org.qortal.data.crosschain.TradeBotData; import org.qortal.data.network.TradePresenceData; +import org.qortal.data.transaction.TransactionData; import org.qortal.event.Event; import org.qortal.event.EventBus; import org.qortal.event.Listener; @@ -33,6 +35,7 @@ import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; import org.qortal.repository.hsqldb.HSQLDBImportExport; import org.qortal.settings.Settings; +import org.qortal.transaction.Transaction; import org.qortal.utils.ByteArray; import org.qortal.utils.NTP; @@ -113,6 +116,9 @@ public class TradeBot implements Listener { private Map safeAllTradePresencesByPubkey = Collections.emptyMap(); private long nextTradePresenceBroadcastTimestamp = 0L; + private Map failedTrades = new HashMap<>(); + private Map validTrades = new HashMap<>(); + private TradeBot() { EventBus.INSTANCE.addListener(event -> TradeBot.getInstance().listen(event)); } @@ -674,6 +680,78 @@ public class TradeBot implements Listener { }); } + /** Removes any trades that have had multiple failures */ + public List removeFailedTrades(Repository repository, List crossChainTrades) { + Long now = NTP.getTime(); + if (now == null) { + return crossChainTrades; + } + + List updatedCrossChainTrades = new ArrayList<>(crossChainTrades); + int getMaxTradeOfferAttempts = Settings.getInstance().getMaxTradeOfferAttempts(); + + for (CrossChainTradeData crossChainTradeData : crossChainTrades) { + // We only care about trades in the OFFERING state + if (crossChainTradeData.mode != AcctMode.OFFERING) { + failedTrades.remove(crossChainTradeData.qortalAtAddress); + validTrades.remove(crossChainTradeData.qortalAtAddress); + continue; + } + + // Return recently cached values if they exist + Long failedTimestamp = failedTrades.get(crossChainTradeData.qortalAtAddress); + if (failedTimestamp != null && now - failedTimestamp < 60 * 60 * 1000L) { + updatedCrossChainTrades.remove(crossChainTradeData); + //LOGGER.info("Removing cached failed trade AT {}", crossChainTradeData.qortalAtAddress); + continue; + } + Long validTimestamp = validTrades.get(crossChainTradeData.qortalAtAddress); + if (validTimestamp != null && now - validTimestamp < 60 * 60 * 1000L) { + //LOGGER.info("NOT removing cached valid trade AT {}", crossChainTradeData.qortalAtAddress); + continue; + } + + try { + List signatures = repository.getTransactionRepository().getSignaturesMatchingCriteria(null, null, null, Arrays.asList(Transaction.TransactionType.MESSAGE), null, null, crossChainTradeData.qortalCreatorTradeAddress, TransactionsResource.ConfirmationStatus.CONFIRMED, null, null, null); + if (signatures.size() < getMaxTradeOfferAttempts) { + // Less than 3 (or user-specified number of) MESSAGE transactions relate to this trade, so assume it is ok + validTrades.put(crossChainTradeData.qortalAtAddress, now); + continue; + } + + List transactions = new ArrayList<>(signatures.size()); + for (byte[] signature : signatures) { + transactions.add(repository.getTransactionRepository().fromSignature(signature)); + } + transactions.sort(Transaction.getDataComparator()); + + // Get timestamp of the first MESSAGE transaction + long firstMessageTimestamp = transactions.get(0).getTimestamp(); + + // Treat as failed if first buy attempt was more than 60 mins ago (as it's still in the OFFERING state) + boolean isFailed = (now - firstMessageTimestamp > 60*60*1000L); + if (isFailed) { + failedTrades.put(crossChainTradeData.qortalAtAddress, now); + updatedCrossChainTrades.remove(crossChainTradeData); + } + else { + validTrades.put(crossChainTradeData.qortalAtAddress, now); + } + + } catch (DataException e) { + LOGGER.info("Unable to determine failed state of AT {}", crossChainTradeData.qortalAtAddress); + continue; + } + } + + return updatedCrossChainTrades; + } + + public boolean isFailedTrade(Repository repository, CrossChainTradeData crossChainTradeData) { + List results = removeFailedTrades(repository, Arrays.asList(crossChainTradeData)); + return results.isEmpty(); + } + private long generateExpiry(long timestamp) { return ((timestamp - 1) / EXPIRY_ROUNDING) * EXPIRY_ROUNDING + PRESENCE_LIFETIME; } diff --git a/src/main/java/org/qortal/crypto/AES.java b/src/main/java/org/qortal/crypto/AES.java index 0e8018f5..1286fb81 100644 --- a/src/main/java/org/qortal/crypto/AES.java +++ b/src/main/java/org/qortal/crypto/AES.java @@ -202,4 +202,12 @@ public class AES { .decode(cipherText))); } + public static long getEncryptedFileSize(long inFileSize) { + // To calculate the resulting file size, add 16 (for the IV), then round up to the nearest multiple of 16 + final int ivSize = 16; + final int chunkSize = 16; + final int expectedSize = Math.round((inFileSize + ivSize) / chunkSize) * chunkSize + chunkSize; + return expectedSize; + } + } diff --git a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceInfo.java b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceInfo.java index 135065aa..a09fc5ff 100644 --- a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceInfo.java +++ b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceInfo.java @@ -16,10 +16,17 @@ public class ArbitraryResourceInfo { public ArbitraryResourceMetadata metadata; public Long size; + public Long created; + public Long updated; public ArbitraryResourceInfo() { } + @Override + public String toString() { + return String.format("%s %s %s", name, service, identifier); + } + @Override public boolean equals(Object o) { if (o == this) diff --git a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceMetadata.java b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceMetadata.java index 497e214f..a6aa6e26 100644 --- a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceMetadata.java +++ b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceMetadata.java @@ -16,16 +16,18 @@ public class ArbitraryResourceMetadata { private Category category; private String categoryName; private List files; + private String mimeType; public ArbitraryResourceMetadata() { } - public ArbitraryResourceMetadata(String title, String description, List tags, Category category, List files) { + public ArbitraryResourceMetadata(String title, String description, List tags, Category category, List files, String mimeType) { this.title = title; this.description = description; this.tags = tags; this.category = category; this.files = files; + this.mimeType = mimeType; if (category != null) { this.categoryName = category.getName(); @@ -40,6 +42,7 @@ public class ArbitraryResourceMetadata { String description = transactionMetadata.getDescription(); List tags = transactionMetadata.getTags(); Category category = transactionMetadata.getCategory(); + String mimeType = transactionMetadata.getMimeType(); // We don't always want to include the file list as it can be too verbose List files = null; @@ -47,11 +50,11 @@ public class ArbitraryResourceMetadata { files = transactionMetadata.getFiles(); } - if (title == null && description == null && tags == null && category == null && files == null) { + if (title == null && description == null && tags == null && category == null && files == null && mimeType == null) { return null; } - return new ArbitraryResourceMetadata(title, description, tags, category, files); + return new ArbitraryResourceMetadata(title, description, tags, category, files, mimeType); } public List getFiles() { diff --git a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java index b1fbbd3c..54dd2af6 100644 --- a/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java +++ b/src/main/java/org/qortal/data/arbitrary/ArbitraryResourceStatus.java @@ -8,6 +8,7 @@ public class ArbitraryResourceStatus { public enum Status { PUBLISHED("Published", "Published but not yet downloaded"), + NOT_PUBLISHED("Not published", "Resource does not exist"), DOWNLOADING("Downloading", "Locating and downloading files..."), DOWNLOADED("Downloaded", "Files downloaded"), BUILDING("Building", "Building..."), @@ -33,6 +34,7 @@ public class ArbitraryResourceStatus { private Integer localChunkCount; private Integer totalChunkCount; + private Float percentLoaded; public ArbitraryResourceStatus() { } @@ -44,6 +46,7 @@ public class ArbitraryResourceStatus { this.description = status.description; this.localChunkCount = localChunkCount; this.totalChunkCount = totalChunkCount; + this.percentLoaded = (this.localChunkCount != null && this.totalChunkCount != null && this.totalChunkCount > 0) ? this.localChunkCount / (float)this.totalChunkCount * 100.0f : null; } public ArbitraryResourceStatus(Status status) { diff --git a/src/main/java/org/qortal/data/chat/ActiveChats.java b/src/main/java/org/qortal/data/chat/ActiveChats.java index d5ebcf3f..248af82e 100644 --- a/src/main/java/org/qortal/data/chat/ActiveChats.java +++ b/src/main/java/org/qortal/data/chat/ActiveChats.java @@ -1,10 +1,15 @@ package org.qortal.data.chat; +import org.bouncycastle.util.encoders.Base64; +import org.qortal.utils.Base58; + import java.util.List; import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; +import static org.qortal.data.chat.ChatMessage.Encoding; + @XmlAccessorType(XmlAccessType.FIELD) public class ActiveChats { @@ -18,20 +23,38 @@ public class ActiveChats { private String sender; private String senderName; private byte[] signature; - private byte[] data; + private Encoding encoding; + private String data; protected GroupChat() { /* JAXB */ } - public GroupChat(int groupId, String groupName, Long timestamp, String sender, String senderName, byte[] signature, byte[] data) { + public GroupChat(int groupId, String groupName, Long timestamp, String sender, String senderName, + byte[] signature, Encoding encoding, byte[] data) { this.groupId = groupId; this.groupName = groupName; this.timestamp = timestamp; this.sender = sender; this.senderName = senderName; this.signature = signature; - this.data = data; + this.encoding = encoding != null ? encoding : Encoding.BASE58; + + if (data != null) { + switch (this.encoding) { + case BASE64: + this.data = Base64.toBase64String(data); + break; + + case BASE58: + default: + this.data = Base58.encode(data); + break; + } + } + else { + this.data = null; + } } public int getGroupId() { @@ -58,7 +81,7 @@ public class ActiveChats { return this.signature; } - public byte[] getData() { + public String getData() { return this.data; } } diff --git a/src/main/java/org/qortal/data/chat/ChatMessage.java b/src/main/java/org/qortal/data/chat/ChatMessage.java index 5d16bb7c..5d9ecb4e 100644 --- a/src/main/java/org/qortal/data/chat/ChatMessage.java +++ b/src/main/java/org/qortal/data/chat/ChatMessage.java @@ -1,11 +1,19 @@ package org.qortal.data.chat; +import org.bouncycastle.util.encoders.Base64; +import org.qortal.utils.Base58; + import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; @XmlAccessorType(XmlAccessType.FIELD) public class ChatMessage { + public enum Encoding { + BASE58, + BASE64 + } + // Properties private long timestamp; @@ -29,7 +37,9 @@ public class ChatMessage { private byte[] chatReference; - private byte[] data; + private Encoding encoding; + + private String data; private boolean isText; private boolean isEncrypted; @@ -44,8 +54,8 @@ public class ChatMessage { // For repository use public ChatMessage(long timestamp, int txGroupId, byte[] reference, byte[] senderPublicKey, String sender, - String senderName, String recipient, String recipientName, byte[] chatReference, byte[] data, - boolean isText, boolean isEncrypted, byte[] signature) { + String senderName, String recipient, String recipientName, byte[] chatReference, + Encoding encoding, byte[] data, boolean isText, boolean isEncrypted, byte[] signature) { this.timestamp = timestamp; this.txGroupId = txGroupId; this.reference = reference; @@ -55,7 +65,24 @@ public class ChatMessage { this.recipient = recipient; this.recipientName = recipientName; this.chatReference = chatReference; - this.data = data; + this.encoding = encoding != null ? encoding : Encoding.BASE58; + + if (data != null) { + switch (this.encoding) { + case BASE64: + this.data = Base64.toBase64String(data); + break; + + case BASE58: + default: + this.data = Base58.encode(data); + break; + } + } + else { + this.data = null; + } + this.isText = isText; this.isEncrypted = isEncrypted; this.signature = signature; @@ -97,7 +124,7 @@ public class ChatMessage { return this.chatReference; } - public byte[] getData() { + public String getData() { return this.data; } diff --git a/src/main/java/org/qortal/data/transaction/ArbitraryTransactionData.java b/src/main/java/org/qortal/data/transaction/ArbitraryTransactionData.java index acd5c3a6..3ab06ecc 100644 --- a/src/main/java/org/qortal/data/transaction/ArbitraryTransactionData.java +++ b/src/main/java/org/qortal/data/transaction/ArbitraryTransactionData.java @@ -73,7 +73,7 @@ public class ArbitraryTransactionData extends TransactionData { @Schema(example = "sender_public_key") private byte[] senderPublicKey; - private Service service; + private int service; private int nonce; private int size; @@ -103,7 +103,7 @@ public class ArbitraryTransactionData extends TransactionData { } public ArbitraryTransactionData(BaseTransactionData baseTransactionData, - int version, Service service, int nonce, int size, + int version, int service, int nonce, int size, String name, String identifier, Method method, byte[] secret, Compression compression, byte[] data, DataType dataType, byte[] metadataHash, List payments) { super(TransactionType.ARBITRARY, baseTransactionData); @@ -135,6 +135,10 @@ public class ArbitraryTransactionData extends TransactionData { } public Service getService() { + return Service.valueOf(this.service); + } + + public int getServiceInt() { return this.service; } diff --git a/src/main/java/org/qortal/data/transaction/CreatePollTransactionData.java b/src/main/java/org/qortal/data/transaction/CreatePollTransactionData.java index 4df7d79d..8b904aa0 100644 --- a/src/main/java/org/qortal/data/transaction/CreatePollTransactionData.java +++ b/src/main/java/org/qortal/data/transaction/CreatePollTransactionData.java @@ -2,9 +2,11 @@ package org.qortal.data.transaction; import java.util.List; +import javax.xml.bind.Unmarshaller; import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; +import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorValue; import org.qortal.data.voting.PollOptionData; import org.qortal.transaction.Transaction; import org.qortal.transaction.Transaction.TransactionType; @@ -14,8 +16,13 @@ import io.swagger.v3.oas.annotations.media.Schema; // All properties to be converted to JSON via JAXB @XmlAccessorType(XmlAccessType.FIELD) @Schema(allOf = { TransactionData.class }) +@XmlDiscriminatorValue("CREATE_POLL") public class CreatePollTransactionData extends TransactionData { + + @Schema(description = "Poll creator's public key", example = "2tiMr5LTpaWCgbRvkPK8TFd7k63DyHJMMFFsz9uBf1ZP") + private byte[] pollCreatorPublicKey; + // Properties private String owner; private String pollName; @@ -29,10 +36,15 @@ public class CreatePollTransactionData extends TransactionData { super(TransactionType.CREATE_POLL); } + public void afterUnmarshal(Unmarshaller u, Object parent) { + this.creatorPublicKey = this.pollCreatorPublicKey; + } + public CreatePollTransactionData(BaseTransactionData baseTransactionData, String owner, String pollName, String description, List pollOptions) { super(Transaction.TransactionType.CREATE_POLL, baseTransactionData); + this.creatorPublicKey = baseTransactionData.creatorPublicKey; this.owner = owner; this.pollName = pollName; this.description = description; @@ -41,6 +53,7 @@ public class CreatePollTransactionData extends TransactionData { // Getters/setters + public byte[] getPollCreatorPublicKey() { return this.creatorPublicKey; } public String getOwner() { return this.owner; } diff --git a/src/main/java/org/qortal/data/transaction/TransactionData.java b/src/main/java/org/qortal/data/transaction/TransactionData.java index 713f98b5..c4a115df 100644 --- a/src/main/java/org/qortal/data/transaction/TransactionData.java +++ b/src/main/java/org/qortal/data/transaction/TransactionData.java @@ -12,6 +12,8 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter; import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorNode; import org.qortal.crypto.Crypto; +import org.qortal.data.voting.PollData; +import org.qortal.data.voting.VoteOnPollData; import org.qortal.transaction.Transaction.ApprovalStatus; import org.qortal.transaction.Transaction.TransactionType; @@ -29,6 +31,7 @@ import io.swagger.v3.oas.annotations.media.Schema.AccessMode; @XmlSeeAlso({GenesisTransactionData.class, PaymentTransactionData.class, RegisterNameTransactionData.class, UpdateNameTransactionData.class, SellNameTransactionData.class, CancelSellNameTransactionData.class, BuyNameTransactionData.class, CreatePollTransactionData.class, VoteOnPollTransactionData.class, ArbitraryTransactionData.class, + PollData.class, VoteOnPollData.class, IssueAssetTransactionData.class, TransferAssetTransactionData.class, CreateAssetOrderTransactionData.class, CancelAssetOrderTransactionData.class, MultiPaymentTransactionData.class, DeployAtTransactionData.class, MessageTransactionData.class, ATTransactionData.class, diff --git a/src/main/java/org/qortal/data/transaction/VoteOnPollTransactionData.java b/src/main/java/org/qortal/data/transaction/VoteOnPollTransactionData.java index 6145d741..a23d5e2b 100644 --- a/src/main/java/org/qortal/data/transaction/VoteOnPollTransactionData.java +++ b/src/main/java/org/qortal/data/transaction/VoteOnPollTransactionData.java @@ -3,7 +3,9 @@ package org.qortal.data.transaction; import javax.xml.bind.Unmarshaller; import javax.xml.bind.annotation.XmlAccessType; import javax.xml.bind.annotation.XmlAccessorType; +import javax.xml.bind.annotation.XmlTransient; +import org.eclipse.persistence.oxm.annotations.XmlDiscriminatorValue; import org.qortal.transaction.Transaction.TransactionType; import io.swagger.v3.oas.annotations.media.Schema; @@ -11,12 +13,17 @@ import io.swagger.v3.oas.annotations.media.Schema; // All properties to be converted to JSON via JAXB @XmlAccessorType(XmlAccessType.FIELD) @Schema(allOf = { TransactionData.class }) +@XmlDiscriminatorValue("VOTE_ON_POLL") public class VoteOnPollTransactionData extends TransactionData { // Properties + @Schema(description = "Vote creator's public key", example = "2tiMr5LTpaWCgbRvkPK8TFd7k63DyHJMMFFsz9uBf1ZP") private byte[] voterPublicKey; private String pollName; private int optionIndex; + // For internal use when orphaning + @XmlTransient + @Schema(hidden = true) private Integer previousOptionIndex; // Constructors diff --git a/src/main/java/org/qortal/data/voting/PollData.java b/src/main/java/org/qortal/data/voting/PollData.java index 4af62087..1850ddc7 100644 --- a/src/main/java/org/qortal/data/voting/PollData.java +++ b/src/main/java/org/qortal/data/voting/PollData.java @@ -14,6 +14,11 @@ public class PollData { // Constructors + // For JAXB + protected PollData() { + super(); + } + public PollData(byte[] creatorPublicKey, String owner, String pollName, String description, List pollOptions, long published) { this.creatorPublicKey = creatorPublicKey; this.owner = owner; @@ -29,22 +34,42 @@ public class PollData { return this.creatorPublicKey; } + public void setCreatorPublicKey(byte[] creatorPublicKey) { + this.creatorPublicKey = creatorPublicKey; + } + public String getOwner() { return this.owner; } + public void setOwner(String owner) { + this.owner = owner; + } + public String getPollName() { return this.pollName; } + public void setPollName(String pollName) { + this.pollName = pollName; + } + public String getDescription() { return this.description; } + public void setDescription(String description) { + this.description = description; + } + public List getPollOptions() { return this.pollOptions; } + public void setPollOptions(List pollOptions) { + this.pollOptions = pollOptions; + } + public long getPublished() { return this.published; } diff --git a/src/main/java/org/qortal/data/voting/VoteOnPollData.java b/src/main/java/org/qortal/data/voting/VoteOnPollData.java index 47c06a54..531ed286 100644 --- a/src/main/java/org/qortal/data/voting/VoteOnPollData.java +++ b/src/main/java/org/qortal/data/voting/VoteOnPollData.java @@ -9,6 +9,11 @@ public class VoteOnPollData { // Constructors + // For JAXB + protected VoteOnPollData() { + super(); + } + public VoteOnPollData(String pollName, byte[] voterPublicKey, int optionIndex) { this.pollName = pollName; this.voterPublicKey = voterPublicKey; @@ -21,12 +26,24 @@ public class VoteOnPollData { return this.pollName; } + public void setPollName(String pollName) { + this.pollName = pollName; + } + public byte[] getVoterPublicKey() { return this.voterPublicKey; } + public void setVoterPublicKey(byte[] voterPublicKey) { + this.voterPublicKey = voterPublicKey; + } + public int getOptionIndex() { return this.optionIndex; } + public void setOptionIndex(int optionIndex) { + this.optionIndex = optionIndex; + } + } diff --git a/src/main/java/org/qortal/list/ResourceList.java b/src/main/java/org/qortal/list/ResourceList.java index 099aa168..855c9068 100644 --- a/src/main/java/org/qortal/list/ResourceList.java +++ b/src/main/java/org/qortal/list/ResourceList.java @@ -9,6 +9,7 @@ import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; @@ -52,6 +53,15 @@ public class ResourceList { String jsonString = ResourceList.listToJSONString(this.list); Path filePath = this.getFilePath(); + // Don't create list if it's empty + if (this.list != null && this.list.isEmpty()) { + if (filePath != null && Files.exists(filePath)) { + // Delete empty list + Files.delete(filePath); + } + return; + } + // Create parent directory if needed try { Files.createDirectories(filePath.getParent()); @@ -72,7 +82,7 @@ public class ResourceList { } try { - String jsonString = new String(Files.readAllBytes(path)); + String jsonString = new String(Files.readAllBytes(path), StandardCharsets.UTF_8); this.list = ResourceList.listFromJSONString(jsonString); } catch (IOException e) { throw new IOException(String.format("Couldn't read contents from file %s", path.toString())); @@ -109,6 +119,13 @@ public class ResourceList { this.list.remove(resource); } + public void clear() { + if (this.list == null) { + return; + } + this.list.clear(); + } + public boolean contains(String resource, boolean caseSensitive) { if (resource == null || this.list == null) { return false; diff --git a/src/main/java/org/qortal/list/ResourceListManager.java b/src/main/java/org/qortal/list/ResourceListManager.java index 4182f87c..f694ebd5 100644 --- a/src/main/java/org/qortal/list/ResourceListManager.java +++ b/src/main/java/org/qortal/list/ResourceListManager.java @@ -2,8 +2,11 @@ package org.qortal.list; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import org.qortal.settings.Settings; import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; import java.util.ArrayList; import java.util.Collections; import java.util.List; @@ -18,6 +21,7 @@ public class ResourceListManager { public ResourceListManager() { + this.lists = this.fetchLists(); } public static synchronized ResourceListManager getInstance() { @@ -27,6 +31,38 @@ public class ResourceListManager { return instance; } + public static synchronized void reset() { + if (instance != null) { + instance = null; + } + } + + private List fetchLists() { + List lists = new ArrayList<>(); + Path listsPath = Paths.get(Settings.getInstance().getListsPath()); + + if (listsPath.toFile().isDirectory()) { + String[] files = listsPath.toFile().list(); + + for (String fileName : files) { + try { + // Remove .json extension + if (fileName.endsWith(".json")) { + fileName = fileName.substring(0, fileName.length() - 5); + } + + ResourceList list = new ResourceList(fileName); + if (list != null) { + lists.add(list); + } + } catch (IOException e) { + // Ignore this list + } + } + } + return lists; + } + private ResourceList getList(String listName) { for (ResourceList list : this.lists) { if (Objects.equals(list.getName(), listName)) { @@ -48,6 +84,18 @@ public class ResourceListManager { } + private List getListsByPrefix(String listNamePrefix) { + List lists = new ArrayList<>(); + + for (ResourceList list : this.lists) { + if (list != null && list.getName() != null && list.getName().startsWith(listNamePrefix)) { + lists.add(list); + } + } + + return lists; + } + public boolean addToList(String listName, String item, boolean save) { ResourceList list = this.getList(listName); if (list == null) { @@ -95,6 +143,16 @@ public class ResourceListManager { return list.contains(item, caseSensitive); } + public boolean listWithPrefixContains(String listNamePrefix, String item, boolean caseSensitive) { + List lists = getListsByPrefix(listNamePrefix); + for (ResourceList list : lists) { + if (list.contains(item, caseSensitive)) { + return true; + } + } + return false; + } + public void saveList(String listName) { ResourceList list = this.getList(listName); if (list == null) { @@ -133,6 +191,15 @@ public class ResourceListManager { return list.getList(); } + public List getStringsInListsWithPrefix(String listNamePrefix) { + List items = new ArrayList<>(); + List lists = getListsByPrefix(listNamePrefix); + for (ResourceList list : lists) { + items.addAll(list.getList()); + } + return items; + } + public int getItemCountForList(String listName) { ResourceList list = this.getList(listName); if (list == null) { diff --git a/src/main/java/org/qortal/network/Network.java b/src/main/java/org/qortal/network/Network.java index f8f73c2a..ca79f367 100644 --- a/src/main/java/org/qortal/network/Network.java +++ b/src/main/java/org/qortal/network/Network.java @@ -124,6 +124,8 @@ public class Network { private final List selfPeers = new ArrayList<>(); + private String bindAddress = null; + private final ExecuteProduceConsume networkEPC; private Selector channelSelector; private ServerSocketChannel serverChannel; @@ -159,25 +161,43 @@ public class Network { // Grab P2P port from settings int listenPort = Settings.getInstance().getListenPort(); - // Grab P2P bind address from settings - try { - InetAddress bindAddr = InetAddress.getByName(Settings.getInstance().getBindAddress()); - InetSocketAddress endpoint = new InetSocketAddress(bindAddr, listenPort); + // Grab P2P bind addresses from settings + List bindAddresses = new ArrayList<>(); + if (Settings.getInstance().getBindAddress() != null) { + bindAddresses.add(Settings.getInstance().getBindAddress()); + } + if (Settings.getInstance().getBindAddressFallback() != null) { + bindAddresses.add(Settings.getInstance().getBindAddressFallback()); + } - channelSelector = Selector.open(); + for (int i=0; i getArbitraryResources(Service service, String identifier, List names, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException; + public List getArbitraryResources(Service service, String identifier, List names, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, Integer limit, Integer offset, Boolean reverse) throws DataException; - public List searchArbitraryResources(Service service, String query, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException; + public List searchArbitraryResources(Service service, String query, String identifier, List names, boolean prefixOnly, List namesFilter, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, Integer limit, Integer offset, Boolean reverse) throws DataException; public List getArbitraryResourceCreatorNames(Service service, String identifier, boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException; diff --git a/src/main/java/org/qortal/repository/ChatRepository.java b/src/main/java/org/qortal/repository/ChatRepository.java index 34ad77dd..7443fb51 100644 --- a/src/main/java/org/qortal/repository/ChatRepository.java +++ b/src/main/java/org/qortal/repository/ChatRepository.java @@ -6,6 +6,8 @@ import org.qortal.data.chat.ActiveChats; import org.qortal.data.chat.ChatMessage; import org.qortal.data.transaction.ChatTransactionData; +import static org.qortal.data.chat.ChatMessage.Encoding; + public interface ChatRepository { /** @@ -15,10 +17,11 @@ public interface ChatRepository { */ public List getMessagesMatchingCriteria(Long before, Long after, Integer txGroupId, byte[] reference, byte[] chatReferenceBytes, Boolean hasChatReference, - List involving, String senderAddress, Integer limit, Integer offset, Boolean reverse) throws DataException; + List involving, String senderAddress, Encoding encoding, + Integer limit, Integer offset, Boolean reverse) throws DataException; - public ChatMessage toChatMessage(ChatTransactionData chatTransactionData) throws DataException; + public ChatMessage toChatMessage(ChatTransactionData chatTransactionData, Encoding encoding) throws DataException; - public ActiveChats getActiveChats(String address) throws DataException; + public ActiveChats getActiveChats(String address, Encoding encoding) throws DataException; } diff --git a/src/main/java/org/qortal/repository/NameRepository.java b/src/main/java/org/qortal/repository/NameRepository.java index d6c0f33e..52a43a18 100644 --- a/src/main/java/org/qortal/repository/NameRepository.java +++ b/src/main/java/org/qortal/repository/NameRepository.java @@ -14,10 +14,12 @@ public interface NameRepository { public boolean reducedNameExists(String reducedName) throws DataException; - public List getAllNames(Integer limit, Integer offset, Boolean reverse) throws DataException; + public List searchNames(String query, boolean prefixOnly, Integer limit, Integer offset, Boolean reverse) throws DataException; + + public List getAllNames(Long after, Integer limit, Integer offset, Boolean reverse) throws DataException; public default List getAllNames() throws DataException { - return getAllNames(null, null, null); + return getAllNames(null, null, null, null); } public List getNamesForSale(Integer limit, Integer offset, Boolean reverse) throws DataException; diff --git a/src/main/java/org/qortal/repository/VotingRepository.java b/src/main/java/org/qortal/repository/VotingRepository.java index 28a9f6c7..b0e2954c 100644 --- a/src/main/java/org/qortal/repository/VotingRepository.java +++ b/src/main/java/org/qortal/repository/VotingRepository.java @@ -9,6 +9,8 @@ public interface VotingRepository { // Polls + public List getAllPolls(Integer limit, Integer offset, Boolean reverse) throws DataException; + public PollData fromPollName(String pollName) throws DataException; public boolean pollExists(String pollName) throws DataException; diff --git a/src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java b/src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java index c21dd038..87841ca9 100644 --- a/src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java +++ b/src/main/java/org/qortal/repository/hsqldb/HSQLDBArbitraryRepository.java @@ -5,9 +5,7 @@ import org.apache.logging.log4j.Logger; import org.bouncycastle.util.Longs; import org.qortal.arbitrary.misc.Service; import org.qortal.data.arbitrary.ArbitraryResourceInfo; -import org.qortal.crypto.Crypto; import org.qortal.data.arbitrary.ArbitraryResourceNameInfo; -import org.qortal.data.network.ArbitraryPeerData; import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.ArbitraryTransactionData.*; import org.qortal.data.transaction.BaseTransactionData; @@ -15,8 +13,10 @@ import org.qortal.data.transaction.TransactionData; import org.qortal.repository.ArbitraryRepository; import org.qortal.repository.DataException; import org.qortal.arbitrary.ArbitraryDataFile; +import org.qortal.transaction.ArbitraryTransaction; import org.qortal.transaction.Transaction.ApprovalStatus; import org.qortal.utils.Base58; +import org.qortal.utils.ListUtils; import java.sql.ResultSet; import java.sql.SQLException; @@ -27,8 +27,6 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { private static final Logger LOGGER = LogManager.getLogger(HSQLDBArbitraryRepository.class); - private static final int MAX_RAW_DATA_SIZE = 255; // size of VARBINARY - protected HSQLDBRepository repository; public HSQLDBArbitraryRepository(HSQLDBRepository repository) { @@ -55,13 +53,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { return true; } - // Load hashes - byte[] hash = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - // Load data file(s) - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); // Check if we already have the complete data file or all chunks if (arbitraryDataFile.allFilesExist()) { @@ -84,13 +77,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { return transactionData.getData(); } - // Load hashes - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - // Load data file(s) - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); // If we have the complete data file, return it if (arbitraryDataFile.exists()) { @@ -105,6 +93,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { arbitraryDataFile.join(); // Verify that the combined hash matches the expected hash + byte[] digest = transactionData.getData(); if (!digest.equals(arbitraryDataFile.digest())) { LOGGER.info(String.format("Hash mismatch for transaction: %s", Base58.encode(signature))); return null; @@ -132,11 +121,11 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { } // Trivial-sized payloads can remain in raw form - if (arbitraryTransactionData.getDataType() == DataType.RAW_DATA && arbitraryTransactionData.getData().length <= MAX_RAW_DATA_SIZE) { + if (arbitraryTransactionData.getDataType() == DataType.RAW_DATA && arbitraryTransactionData.getData().length <= ArbitraryTransaction.MAX_DATA_SIZE) { return; } - throw new IllegalStateException(String.format("Supplied data is larger than maximum size (%d bytes). Please use ArbitraryDataWriter.", MAX_RAW_DATA_SIZE)); + throw new IllegalStateException(String.format("Supplied data is larger than maximum size (%d bytes). Please use ArbitraryDataWriter.", ArbitraryTransaction.MAX_DATA_SIZE)); } @Override @@ -146,17 +135,11 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { return; } - // Load hashes - byte[] hash = arbitraryTransactionData.getData(); - byte[] metadataHash = arbitraryTransactionData.getMetadataHash(); - // Load data file(s) - byte[] signature = arbitraryTransactionData.getSignature(); - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); - // Delete file and chunks - arbitraryDataFile.deleteAll(); + // Delete file, chunks, and metadata + arbitraryDataFile.deleteAll(true); } @Override @@ -202,7 +185,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { int version = resultSet.getInt(11); int nonce = resultSet.getInt(12); - Service serviceResult = Service.valueOf(resultSet.getInt(13)); + int serviceInt = resultSet.getInt(13); int size = resultSet.getInt(14); boolean isDataRaw = resultSet.getBoolean(15); // NOT NULL, so no null to false DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH; @@ -216,7 +199,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { // FUTURE: get payments from signature if needed. Avoiding for now to reduce database calls. ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData, - version, serviceResult, nonce, size, nameResult, identifierResult, method, secret, + version, serviceInt, nonce, size, nameResult, identifierResult, method, secret, compression, data, dataType, metadataHash, null); arbitraryTransactionData.add(transactionData); @@ -277,7 +260,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { int version = resultSet.getInt(11); int nonce = resultSet.getInt(12); - Service serviceResult = Service.valueOf(resultSet.getInt(13)); + int serviceInt = resultSet.getInt(13); int size = resultSet.getInt(14); boolean isDataRaw = resultSet.getBoolean(15); // NOT NULL, so no null to false DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH; @@ -291,7 +274,7 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { // FUTURE: get payments from signature if needed. Avoiding for now to reduce database calls. ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData, - version, serviceResult, nonce, size, nameResult, identifierResult, methodResult, secret, + version, serviceInt, nonce, size, nameResult, identifierResult, methodResult, secret, compression, data, dataType, metadataHash, null); return transactionData; @@ -302,7 +285,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { @Override public List getArbitraryResources(Service service, String identifier, List names, - boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException { + boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, + Integer limit, Integer offset, Boolean reverse) throws DataException { StringBuilder sql = new StringBuilder(512); List bindParams = new ArrayList<>(); @@ -337,6 +321,36 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { sql.append(")"); } + // Handle "followed only" + if (followedOnly != null && followedOnly) { + List followedNames = ListUtils.followedNames(); + if (followedNames != null && !followedNames.isEmpty()) { + sql.append(" AND name IN (?"); + bindParams.add(followedNames.get(0)); + + for (int i = 1; i < followedNames.size(); ++i) { + sql.append(", ?"); + bindParams.add(followedNames.get(i)); + } + sql.append(")"); + } + } + + // Handle "exclude blocked" + if (excludeBlocked != null && excludeBlocked) { + List blockedNames = ListUtils.blockedNames(); + if (blockedNames != null && !blockedNames.isEmpty()) { + sql.append(" AND name NOT IN (?"); + bindParams.add(blockedNames.get(0)); + + for (int i = 1; i < blockedNames.size(); ++i) { + sql.append(", ?"); + bindParams.add(blockedNames.get(i)); + } + sql.append(")"); + } + } + sql.append(" GROUP BY name, service, identifier ORDER BY name COLLATE SQL_TEXT_UCC_NO_PAD"); if (reverse != null && reverse) { @@ -378,37 +392,107 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { } @Override - public List searchArbitraryResources(Service service, String query, - boolean defaultResource, Integer limit, Integer offset, Boolean reverse) throws DataException { + public List searchArbitraryResources(Service service, String query, String identifier, List names, boolean prefixOnly, + List exactMatchNames, boolean defaultResource, Boolean followedOnly, Boolean excludeBlocked, + Integer limit, Integer offset, Boolean reverse) throws DataException { StringBuilder sql = new StringBuilder(512); List bindParams = new ArrayList<>(); - // For now we are searching anywhere in the fields - // Note that this will bypass any indexes so may not scale well - // Longer term we probably want to copy resources to their own table anyway - String queryWildcard = String.format("%%%s%%", query.toLowerCase()); - - sql.append("SELECT name, service, identifier, MAX(size) AS max_size FROM ArbitraryTransactions WHERE 1=1"); + sql.append("SELECT name, service, identifier, MAX(size) AS max_size, MIN(created_when) AS date_created, MAX(created_when) AS date_updated " + + "FROM ArbitraryTransactions " + + "JOIN Transactions USING (signature) " + + "WHERE 1=1"); if (service != null) { sql.append(" AND service = "); sql.append(service.value); } - if (defaultResource) { - // Default resource requested - use NULL identifier and search name only - sql.append(" AND LCASE(name) LIKE ? AND identifier IS NULL"); - bindParams.add(queryWildcard); + // Handle general query matches + if (query != null) { + // Search anywhere in the fields, unless "prefixOnly" has been requested + // Note that without prefixOnly it will bypass any indexes so may not scale well + // Longer term we probably want to copy resources to their own table anyway + String queryWildcard = prefixOnly ? String.format("%s%%", query.toLowerCase()) : String.format("%%%s%%", query.toLowerCase()); + + if (defaultResource) { + // Default resource requested - use NULL identifier and search name only + sql.append(" AND LCASE(name) LIKE ? AND identifier IS NULL"); + bindParams.add(queryWildcard); + } else { + // Non-default resource requested + // In this case we search the identifier as well as the name + sql.append(" AND (LCASE(name) LIKE ? OR LCASE(identifier) LIKE ?)"); + bindParams.add(queryWildcard); + bindParams.add(queryWildcard); + } } - else { - // Non-default resource requested - // In this case we search the identifier as well as the name - sql.append(" AND (LCASE(name) LIKE ? OR LCASE(identifier) LIKE ?)"); - bindParams.add(queryWildcard); + + // Handle identifier matches + if (identifier != null) { + // Search anywhere in the identifier, unless "prefixOnly" has been requested + String queryWildcard = prefixOnly ? String.format("%s%%", identifier.toLowerCase()) : String.format("%%%s%%", identifier.toLowerCase()); + sql.append(" AND LCASE(identifier) LIKE ?"); bindParams.add(queryWildcard); } - sql.append(" GROUP BY name, service, identifier ORDER BY name COLLATE SQL_TEXT_UCC_NO_PAD"); + // Handle name searches + if (names != null && !names.isEmpty()) { + sql.append(" AND ("); + + for (int i = 0; i < names.size(); ++i) { + // Search anywhere in the name, unless "prefixOnly" has been requested + String queryWildcard = prefixOnly ? String.format("%s%%", names.get(i).toLowerCase()) : String.format("%%%s%%", names.get(i).toLowerCase()); + if (i > 0) sql.append(" OR "); + sql.append("LCASE(name) LIKE ?"); + bindParams.add(queryWildcard); + } + sql.append(")"); + } + + // Handle name exact matches + if (exactMatchNames != null && !exactMatchNames.isEmpty()) { + sql.append(" AND LCASE(name) IN (?"); + bindParams.add(exactMatchNames.get(0).toLowerCase()); + + for (int i = 1; i < exactMatchNames.size(); ++i) { + sql.append(", ?"); + bindParams.add(exactMatchNames.get(i).toLowerCase()); + } + sql.append(")"); + } + + // Handle "followed only" + if (followedOnly != null && followedOnly) { + List followedNames = ListUtils.followedNames(); + if (followedNames != null && !followedNames.isEmpty()) { + sql.append(" AND LCASE(name) IN (?"); + bindParams.add(followedNames.get(0).toLowerCase()); + + for (int i = 1; i < followedNames.size(); ++i) { + sql.append(", ?"); + bindParams.add(followedNames.get(i).toLowerCase()); + } + sql.append(")"); + } + } + + // Handle "exclude blocked" + if (excludeBlocked != null && excludeBlocked) { + List blockedNames = ListUtils.blockedNames(); + if (blockedNames != null && !blockedNames.isEmpty()) { + sql.append(" AND LCASE(name) NOT IN (?"); + bindParams.add(blockedNames.get(0).toLowerCase()); + + for (int i = 1; i < blockedNames.size(); ++i) { + sql.append(", ?"); + bindParams.add(blockedNames.get(i).toLowerCase()); + } + sql.append(")"); + } + } + + sql.append(" GROUP BY name, service, identifier ORDER BY date_created"); if (reverse != null && reverse) { sql.append(" DESC"); @@ -427,6 +511,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { Service serviceResult = Service.valueOf(resultSet.getInt(2)); String identifierResult = resultSet.getString(3); Integer sizeResult = resultSet.getInt(4); + long dateCreated = resultSet.getLong(5); + long dateUpdated = resultSet.getLong(6); // We should filter out resources without names if (nameResult == null) { @@ -438,6 +524,8 @@ public class HSQLDBArbitraryRepository implements ArbitraryRepository { arbitraryResourceInfo.service = serviceResult; arbitraryResourceInfo.identifier = identifierResult; arbitraryResourceInfo.size = Longs.valueOf(sizeResult); + arbitraryResourceInfo.created = dateCreated; + arbitraryResourceInfo.updated = dateUpdated; arbitraryResources.add(arbitraryResourceInfo); } while (resultSet.next()); diff --git a/src/main/java/org/qortal/repository/hsqldb/HSQLDBChatRepository.java b/src/main/java/org/qortal/repository/hsqldb/HSQLDBChatRepository.java index 55467d87..9e310e78 100644 --- a/src/main/java/org/qortal/repository/hsqldb/HSQLDBChatRepository.java +++ b/src/main/java/org/qortal/repository/hsqldb/HSQLDBChatRepository.java @@ -14,6 +14,8 @@ import org.qortal.repository.ChatRepository; import org.qortal.repository.DataException; import org.qortal.transaction.Transaction.TransactionType; +import static org.qortal.data.chat.ChatMessage.Encoding; + public class HSQLDBChatRepository implements ChatRepository { protected HSQLDBRepository repository; @@ -24,8 +26,8 @@ public class HSQLDBChatRepository implements ChatRepository { @Override public List getMessagesMatchingCriteria(Long before, Long after, Integer txGroupId, byte[] referenceBytes, - byte[] chatReferenceBytes, Boolean hasChatReference, List involving, String senderAddress, - Integer limit, Integer offset, Boolean reverse) throws DataException { + byte[] chatReferenceBytes, Boolean hasChatReference, List involving, String senderAddress, + Encoding encoding, Integer limit, Integer offset, Boolean reverse) throws DataException { // Check args meet expectations if ((txGroupId != null && involving != null && !involving.isEmpty()) || (txGroupId == null && (involving == null || involving.size() != 2))) @@ -127,7 +129,7 @@ public class HSQLDBChatRepository implements ChatRepository { byte[] signature = resultSet.getBytes(13); ChatMessage chatMessage = new ChatMessage(timestamp, groupId, reference, senderPublicKey, sender, - senderName, recipient, recipientName, chatReference, data, isText, isEncrypted, signature); + senderName, recipient, recipientName, chatReference, encoding, data, isText, isEncrypted, signature); chatMessages.add(chatMessage); } while (resultSet.next()); @@ -139,7 +141,7 @@ public class HSQLDBChatRepository implements ChatRepository { } @Override - public ChatMessage toChatMessage(ChatTransactionData chatTransactionData) throws DataException { + public ChatMessage toChatMessage(ChatTransactionData chatTransactionData, Encoding encoding) throws DataException { String sql = "SELECT SenderNames.name, RecipientNames.name " + "FROM ChatTransactions " + "LEFT OUTER JOIN Names AS SenderNames ON SenderNames.owner = sender " @@ -166,21 +168,22 @@ public class HSQLDBChatRepository implements ChatRepository { byte[] signature = chatTransactionData.getSignature(); return new ChatMessage(timestamp, groupId, reference, senderPublicKey, sender, - senderName, recipient, recipientName, chatReference, data, isText, isEncrypted, signature); + senderName, recipient, recipientName, chatReference, encoding, data, + isText, isEncrypted, signature); } catch (SQLException e) { throw new DataException("Unable to fetch convert chat transaction from repository", e); } } @Override - public ActiveChats getActiveChats(String address) throws DataException { - List groupChats = getActiveGroupChats(address); + public ActiveChats getActiveChats(String address, Encoding encoding) throws DataException { + List groupChats = getActiveGroupChats(address, encoding); List directChats = getActiveDirectChats(address); return new ActiveChats(groupChats, directChats); } - private List getActiveGroupChats(String address) throws DataException { + private List getActiveGroupChats(String address, Encoding encoding) throws DataException { // Find groups where address is a member and potential latest message details String groupsSql = "SELECT group_id, group_name, latest_timestamp, sender, sender_name, signature, data " + "FROM GroupMembers " @@ -213,7 +216,7 @@ public class HSQLDBChatRepository implements ChatRepository { byte[] signature = resultSet.getBytes(6); byte[] data = resultSet.getBytes(7); - GroupChat groupChat = new GroupChat(groupId, groupName, timestamp, sender, senderName, signature, data); + GroupChat groupChat = new GroupChat(groupId, groupName, timestamp, sender, senderName, signature, encoding, data); groupChats.add(groupChat); } while (resultSet.next()); } @@ -247,7 +250,7 @@ public class HSQLDBChatRepository implements ChatRepository { data = resultSet.getBytes(5); } - GroupChat groupChat = new GroupChat(0, null, timestamp, sender, senderName, signature, data); + GroupChat groupChat = new GroupChat(0, null, timestamp, sender, senderName, signature, encoding, data); groupChats.add(groupChat); } catch (SQLException e) { throw new DataException("Unable to fetch active group chats from repository", e); diff --git a/src/main/java/org/qortal/repository/hsqldb/HSQLDBNameRepository.java b/src/main/java/org/qortal/repository/hsqldb/HSQLDBNameRepository.java index 3a3574ef..2fefcf8b 100644 --- a/src/main/java/org/qortal/repository/hsqldb/HSQLDBNameRepository.java +++ b/src/main/java/org/qortal/repository/hsqldb/HSQLDBNameRepository.java @@ -103,12 +103,18 @@ public class HSQLDBNameRepository implements NameRepository { } } - @Override - public List getAllNames(Integer limit, Integer offset, Boolean reverse) throws DataException { - StringBuilder sql = new StringBuilder(256); + public List searchNames(String query, boolean prefixOnly, Integer limit, Integer offset, Boolean reverse) throws DataException { + StringBuilder sql = new StringBuilder(512); + List bindParams = new ArrayList<>(); sql.append("SELECT name, reduced_name, owner, data, registered_when, updated_when, " - + "is_for_sale, sale_price, reference, creation_group_id FROM Names ORDER BY name"); + + "is_for_sale, sale_price, reference, creation_group_id FROM Names " + + "WHERE LCASE(name) LIKE ? ORDER BY name"); + + // Search anywhere in the name, unless "prefixOnly" has been requested + // Note that without prefixOnly it will bypass any indexes + String queryWildcard = prefixOnly ? String.format("%s%%", query.toLowerCase()) : String.format("%%%s%%", query.toLowerCase()); + bindParams.add(queryWildcard); if (reverse != null && reverse) sql.append(" DESC"); @@ -117,7 +123,64 @@ public class HSQLDBNameRepository implements NameRepository { List names = new ArrayList<>(); - try (ResultSet resultSet = this.repository.checkedExecute(sql.toString())) { + try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) { + if (resultSet == null) + return names; + + do { + String name = resultSet.getString(1); + String reducedName = resultSet.getString(2); + String owner = resultSet.getString(3); + String data = resultSet.getString(4); + long registered = resultSet.getLong(5); + + // Special handling for possibly-NULL "updated" column + Long updated = resultSet.getLong(6); + if (updated == 0 && resultSet.wasNull()) + updated = null; + + boolean isForSale = resultSet.getBoolean(7); + + Long salePrice = resultSet.getLong(8); + if (salePrice == 0 && resultSet.wasNull()) + salePrice = null; + + byte[] reference = resultSet.getBytes(9); + int creationGroupId = resultSet.getInt(10); + + names.add(new NameData(name, reducedName, owner, data, registered, updated, isForSale, salePrice, reference, creationGroupId)); + } while (resultSet.next()); + + return names; + } catch (SQLException e) { + throw new DataException("Unable to search names in repository", e); + } + } + + @Override + public List getAllNames(Long after, Integer limit, Integer offset, Boolean reverse) throws DataException { + StringBuilder sql = new StringBuilder(256); + List bindParams = new ArrayList<>(); + + sql.append("SELECT name, reduced_name, owner, data, registered_when, updated_when, " + + "is_for_sale, sale_price, reference, creation_group_id FROM Names"); + + if (after != null) { + sql.append(" WHERE registered_when > ? OR updated_when > ?"); + bindParams.add(after); + bindParams.add(after); + } + + sql.append(" ORDER BY name"); + + if (reverse != null && reverse) + sql.append(" DESC"); + + HSQLDBRepository.limitOffsetSql(sql, limit, offset); + + List names = new ArrayList<>(); + + try (ResultSet resultSet = this.repository.checkedExecute(sql.toString(), bindParams.toArray())) { if (resultSet == null) return names; diff --git a/src/main/java/org/qortal/repository/hsqldb/HSQLDBVotingRepository.java b/src/main/java/org/qortal/repository/hsqldb/HSQLDBVotingRepository.java index 447fbe4c..cc33426b 100644 --- a/src/main/java/org/qortal/repository/hsqldb/HSQLDBVotingRepository.java +++ b/src/main/java/org/qortal/repository/hsqldb/HSQLDBVotingRepository.java @@ -21,6 +21,55 @@ public class HSQLDBVotingRepository implements VotingRepository { // Polls + @Override + public List getAllPolls(Integer limit, Integer offset, Boolean reverse) throws DataException { + StringBuilder sql = new StringBuilder(512); + + sql.append("SELECT poll_name, description, creator, owner, published_when FROM Polls ORDER BY poll_name"); + + if (reverse != null && reverse) + sql.append(" DESC"); + + HSQLDBRepository.limitOffsetSql(sql, limit, offset); + + List polls = new ArrayList<>(); + + try (ResultSet resultSet = this.repository.checkedExecute(sql.toString())) { + if (resultSet == null) + return polls; + + do { + String pollName = resultSet.getString(1); + String description = resultSet.getString(2); + byte[] creatorPublicKey = resultSet.getBytes(3); + String owner = resultSet.getString(4); + long published = resultSet.getLong(5); + + String optionsSql = "SELECT option_name FROM PollOptions WHERE poll_name = ? ORDER BY option_index ASC"; + try (ResultSet optionsResultSet = this.repository.checkedExecute(optionsSql, pollName)) { + if (optionsResultSet == null) + return null; + + List pollOptions = new ArrayList<>(); + + // NOTE: do-while because checkedExecute() above has already called rs.next() for us + do { + String optionName = optionsResultSet.getString(1); + + pollOptions.add(new PollOptionData(optionName)); + } while (optionsResultSet.next()); + + polls.add(new PollData(creatorPublicKey, owner, pollName, description, pollOptions, published)); + } + + } while (resultSet.next()); + + return polls; + } catch (SQLException e) { + throw new DataException("Unable to fetch polls from repository", e); + } + } + @Override public PollData fromPollName(String pollName) throws DataException { String sql = "SELECT description, creator, owner, published_when FROM Polls WHERE poll_name = ?"; diff --git a/src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBArbitraryTransactionRepository.java b/src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBArbitraryTransactionRepository.java index c7f4c958..57b75a29 100644 --- a/src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBArbitraryTransactionRepository.java +++ b/src/main/java/org/qortal/repository/hsqldb/transaction/HSQLDBArbitraryTransactionRepository.java @@ -31,7 +31,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos int version = resultSet.getInt(1); int nonce = resultSet.getInt(2); - Service service = Service.valueOf(resultSet.getInt(3)); + int serviceInt = resultSet.getInt(3); int size = resultSet.getInt(4); boolean isDataRaw = resultSet.getBoolean(5); // NOT NULL, so no null to false DataType dataType = isDataRaw ? DataType.RAW_DATA : DataType.DATA_HASH; @@ -44,7 +44,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.valueOf(resultSet.getInt(12)); List payments = this.getPaymentsFromSignature(baseTransactionData.getSignature()); - return new ArbitraryTransactionData(baseTransactionData, version, service, nonce, size, name, + return new ArbitraryTransactionData(baseTransactionData, version, serviceInt, nonce, size, name, identifier, method, secret, compression, data, dataType, metadataHash, payments); } catch (SQLException e) { throw new DataException("Unable to fetch arbitrary transaction from repository", e); @@ -66,7 +66,7 @@ public class HSQLDBArbitraryTransactionRepository extends HSQLDBTransactionRepos HSQLDBSaver saveHelper = new HSQLDBSaver("ArbitraryTransactions"); saveHelper.bind("signature", arbitraryTransactionData.getSignature()).bind("sender", arbitraryTransactionData.getSenderPublicKey()) - .bind("version", arbitraryTransactionData.getVersion()).bind("service", arbitraryTransactionData.getService().value) + .bind("version", arbitraryTransactionData.getVersion()).bind("service", arbitraryTransactionData.getServiceInt()) .bind("nonce", arbitraryTransactionData.getNonce()).bind("size", arbitraryTransactionData.getSize()) .bind("is_data_raw", arbitraryTransactionData.getDataType() == DataType.RAW_DATA).bind("data", arbitraryTransactionData.getData()) .bind("metadata_hash", arbitraryTransactionData.getMetadataHash()).bind("name", arbitraryTransactionData.getName()) diff --git a/src/main/java/org/qortal/settings/Settings.java b/src/main/java/org/qortal/settings/Settings.java index c6c49c1f..a87a72f4 100644 --- a/src/main/java/org/qortal/settings/Settings.java +++ b/src/main/java/org/qortal/settings/Settings.java @@ -61,6 +61,7 @@ public class Settings { // Common to all networking (API/P2P) private String bindAddress = "::"; // Use IPv6 wildcard to listen on all local addresses + private String bindAddressFallback = "0.0.0.0"; // Some systems are unable to bind using IPv6 // UI servers private int uiPort = 12388; @@ -104,6 +105,7 @@ public class Settings { private Integer gatewayPort; private boolean gatewayEnabled = false; private boolean gatewayLoggingEnabled = false; + private boolean gatewayLoopbackEnabled = false; // Specific to this node private boolean wipeUnconfirmedOnStart = false; @@ -251,6 +253,9 @@ public class Settings { /** Whether to show SysTray pop-up notifications when trade-bot entries change state */ private boolean tradebotSystrayEnabled = false; + /** Maximum buy attempts for each trade offer before it is considered failed, and hidden from the list */ + private int maxTradeOfferAttempts = 3; + /** Wallets path - used for storing encrypted wallet caches for coins that require them */ private String walletsPath = "wallets"; @@ -353,7 +358,7 @@ public class Settings { private Long maxStorageCapacity = null; /** Whether to serve QDN data without authentication */ - private boolean qdnAuthBypassEnabled = false; + private boolean qdnAuthBypassEnabled = true; // Domain mapping public static class DomainMap { @@ -636,6 +641,10 @@ public class Settings { return this.gatewayLoggingEnabled; } + public boolean isGatewayLoopbackEnabled() { + return this.gatewayLoopbackEnabled; + } + public boolean getWipeUnconfirmedOnStart() { return this.wipeUnconfirmedOnStart; @@ -684,6 +693,10 @@ public class Settings { return this.bindAddress; } + public String getBindAddressFallback() { + return this.bindAddressFallback; + } + public boolean isUPnPEnabled() { return this.uPnPEnabled; } @@ -761,6 +774,10 @@ public class Settings { return this.pirateChainNet; } + public int getMaxTradeOfferAttempts() { + return this.maxTradeOfferAttempts; + } + public String getWalletsPath() { return this.walletsPath; } @@ -1001,6 +1018,10 @@ public class Settings { } public boolean isQDNAuthBypassEnabled() { + if (this.gatewayEnabled) { + // We must always bypass QDN authentication in gateway mode, in order for it to function properly + return true; + } return this.qdnAuthBypassEnabled; } } diff --git a/src/main/java/org/qortal/transaction/ArbitraryTransaction.java b/src/main/java/org/qortal/transaction/ArbitraryTransaction.java index 3452f916..7034d7b8 100644 --- a/src/main/java/org/qortal/transaction/ArbitraryTransaction.java +++ b/src/main/java/org/qortal/transaction/ArbitraryTransaction.java @@ -33,7 +33,7 @@ public class ArbitraryTransaction extends Transaction { private ArbitraryTransactionData arbitraryTransactionData; // Other useful constants - public static final int MAX_DATA_SIZE = 4000; + public static final int MAX_DATA_SIZE = 256; public static final int MAX_METADATA_LENGTH = 32; public static final int HASH_LENGTH = TransactionTransformer.SHA256_LENGTH; public static final int MAX_IDENTIFIER_LENGTH = 64; diff --git a/src/main/java/org/qortal/transaction/ChatTransaction.java b/src/main/java/org/qortal/transaction/ChatTransaction.java index 5ed96494..f6e26802 100644 --- a/src/main/java/org/qortal/transaction/ChatTransaction.java +++ b/src/main/java/org/qortal/transaction/ChatTransaction.java @@ -8,6 +8,7 @@ import java.util.function.Predicate; import org.qortal.account.Account; import org.qortal.account.PublicKeyAccount; import org.qortal.asset.Asset; +import org.qortal.controller.arbitrary.ArbitraryDataStorageManager; import org.qortal.crypto.Crypto; import org.qortal.crypto.MemoryPoW; import org.qortal.data.naming.NameData; @@ -22,6 +23,7 @@ import org.qortal.settings.Settings; import org.qortal.transform.TransformationException; import org.qortal.transform.transaction.ChatTransactionTransformer; import org.qortal.transform.transaction.TransactionTransformer; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; public class ChatTransaction extends Transaction { @@ -156,8 +158,7 @@ public class ChatTransaction extends Transaction { } // Check for blocked author by address - ResourceListManager listManager = ResourceListManager.getInstance(); - if (listManager.listContains("blockedAddresses", this.chatTransactionData.getSender(), true)) { + if (ListUtils.isAddressBlocked(this.chatTransactionData.getSender())) { return ValidationResult.ADDRESS_BLOCKED; } @@ -166,7 +167,7 @@ public class ChatTransaction extends Transaction { if (names != null && names.size() > 0) { for (NameData nameData : names) { if (nameData != null && nameData.getName() != null) { - if (listManager.listContains("blockedNames", nameData.getName(), false)) { + if (ListUtils.isNameBlocked(nameData.getName())) { return ValidationResult.NAME_BLOCKED; } } diff --git a/src/main/java/org/qortal/transform/transaction/ArbitraryTransactionTransformer.java b/src/main/java/org/qortal/transform/transaction/ArbitraryTransactionTransformer.java index b1554e8d..1ae80e1f 100644 --- a/src/main/java/org/qortal/transform/transaction/ArbitraryTransactionTransformer.java +++ b/src/main/java/org/qortal/transform/transaction/ArbitraryTransactionTransformer.java @@ -7,7 +7,6 @@ import java.util.ArrayList; import java.util.List; import com.google.common.base.Utf8; -import org.qortal.arbitrary.misc.Service; import org.qortal.crypto.Crypto; import org.qortal.data.PaymentData; import org.qortal.data.transaction.ArbitraryTransactionData; @@ -131,7 +130,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer { payments.add(PaymentTransformer.fromByteBuffer(byteBuffer)); } - Service service = Service.valueOf(byteBuffer.getInt()); + int service = byteBuffer.getInt(); // We might be receiving hash of data instead of actual raw data boolean isRaw = byteBuffer.get() != 0; @@ -226,7 +225,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer { for (PaymentData paymentData : payments) bytes.write(PaymentTransformer.toBytes(paymentData)); - bytes.write(Ints.toByteArray(arbitraryTransactionData.getService().value)); + bytes.write(Ints.toByteArray(arbitraryTransactionData.getServiceInt())); bytes.write((byte) (arbitraryTransactionData.getDataType() == DataType.RAW_DATA ? 1 : 0)); @@ -299,7 +298,7 @@ public class ArbitraryTransactionTransformer extends TransactionTransformer { bytes.write(PaymentTransformer.toBytes(paymentData)); } - bytes.write(Ints.toByteArray(arbitraryTransactionData.getService().value)); + bytes.write(Ints.toByteArray(arbitraryTransactionData.getServiceInt())); bytes.write(Ints.toByteArray(arbitraryTransactionData.getData().length)); diff --git a/src/main/java/org/qortal/utils/ArbitraryTransactionUtils.java b/src/main/java/org/qortal/utils/ArbitraryTransactionUtils.java index 4c464bee..efd84110 100644 --- a/src/main/java/org/qortal/utils/ArbitraryTransactionUtils.java +++ b/src/main/java/org/qortal/utils/ArbitraryTransactionUtils.java @@ -3,11 +3,11 @@ package org.qortal.utils; import org.apache.commons.lang3.ArrayUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; -import org.qortal.arbitrary.ArbitraryDataFile; -import org.qortal.arbitrary.ArbitraryDataFileChunk; -import org.qortal.arbitrary.ArbitraryDataReader; -import org.qortal.arbitrary.ArbitraryDataResource; +import org.qortal.arbitrary.*; +import org.qortal.arbitrary.metadata.ArbitraryDataTransactionMetadata; import org.qortal.arbitrary.misc.Service; +import org.qortal.data.arbitrary.ArbitraryResourceInfo; +import org.qortal.data.arbitrary.ArbitraryResourceMetadata; import org.qortal.data.arbitrary.ArbitraryResourceStatus; import org.qortal.data.transaction.ArbitraryTransactionData; import org.qortal.data.transaction.TransactionData; @@ -110,13 +110,8 @@ public class ArbitraryTransactionUtils { return false; } - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - // Load complete file and chunks - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); return arbitraryDataFile.allChunksExist(); } @@ -126,18 +121,13 @@ public class ArbitraryTransactionUtils { return false; } - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - - if (metadataHash == null) { + if (transactionData.getMetadataHash() == null) { // This file doesn't have any metadata/chunks, which means none exist return false; } // Load complete file and chunks - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); return arbitraryDataFile.anyChunksExist(); } @@ -147,12 +137,7 @@ public class ArbitraryTransactionUtils { return 0; } - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); // Find the folder containing the files Path parentPath = arbitraryDataFile.getFilePath().getParent(); @@ -180,20 +165,15 @@ public class ArbitraryTransactionUtils { return 0; } - byte[] digest = transactionData.getData(); - byte[] metadataHash = transactionData.getMetadataHash(); - byte[] signature = transactionData.getSignature(); - - if (metadataHash == null) { + if (transactionData.getMetadataHash() == null) { // This file doesn't have any metadata, therefore it has a single (complete) chunk return 1; } // Load complete file and chunks - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(digest, signature); - arbitraryDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(transactionData); - return arbitraryDataFile.chunkCount() + 1; // +1 for the metadata file + return arbitraryDataFile.fileCount(); } public static boolean isFileRecent(Path filePath, long now, long cleanupAfter) { @@ -243,31 +223,24 @@ public class ArbitraryTransactionUtils { } public static void deleteCompleteFileAndChunks(ArbitraryTransactionData arbitraryTransactionData) throws DataException { - byte[] completeHash = arbitraryTransactionData.getData(); - byte[] metadataHash = arbitraryTransactionData.getMetadataHash(); - byte[] signature = arbitraryTransactionData.getSignature(); - - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(completeHash, signature); - arbitraryDataFile.setMetadataHash(metadataHash); - arbitraryDataFile.deleteAll(); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); + arbitraryDataFile.deleteAll(true); } public static void convertFileToChunks(ArbitraryTransactionData arbitraryTransactionData, long now, long cleanupAfter) throws DataException { - byte[] completeHash = arbitraryTransactionData.getData(); - byte[] metadataHash = arbitraryTransactionData.getMetadataHash(); - byte[] signature = arbitraryTransactionData.getSignature(); - // Find the expected chunk hashes - ArbitraryDataFile expectedDataFile = ArbitraryDataFile.fromHash(completeHash, signature); - expectedDataFile.setMetadataHash(metadataHash); + ArbitraryDataFile expectedDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); - if (metadataHash == null || !expectedDataFile.getMetadataFile().exists()) { + if (arbitraryTransactionData.getMetadataHash() == null || !expectedDataFile.getMetadataFile().exists()) { // We don't have the metadata file, or this transaction doesn't have one - nothing to do return; } + byte[] completeHash = arbitraryTransactionData.getData(); + byte[] signature = arbitraryTransactionData.getSignature(); + // Split the file into chunks - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(completeHash, signature); + ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromTransactionData(arbitraryTransactionData); int chunkCount = arbitraryDataFile.split(ArbitraryDataFile.CHUNK_SIZE); if (chunkCount > 1) { LOGGER.info(String.format("Successfully split %s into %d chunk%s", @@ -426,7 +399,7 @@ public class ArbitraryTransactionUtils { // If "build" has been specified, build the resource before returning its status if (build != null && build == true) { - ArbitraryDataReader reader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, null); + ArbitraryDataReader reader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); try { if (!reader.isBuilding()) { reader.loadSynchronously(false); @@ -440,4 +413,41 @@ public class ArbitraryTransactionUtils { return resource.getStatus(false); } + public static List addStatusToResources(List resources) { + // Determine and add the status of each resource + List updatedResources = new ArrayList<>(); + for (ArbitraryResourceInfo resourceInfo : resources) { + try { + ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ArbitraryDataFile.ResourceIdType.NAME, + resourceInfo.service, resourceInfo.identifier); + ArbitraryResourceStatus status = resource.getStatus(true); + if (status != null) { + resourceInfo.status = status; + } + updatedResources.add(resourceInfo); + + } catch (Exception e) { + // Catch and log all exceptions, since some systems are experiencing 500 errors when including statuses + LOGGER.info("Caught exception when adding status to resource {}: {}", resourceInfo, e.toString()); + } + } + return updatedResources; + } + + public static List addMetadataToResources(List resources) { + // Add metadata fields to each resource if they exist + List updatedResources = new ArrayList<>(); + for (ArbitraryResourceInfo resourceInfo : resources) { + ArbitraryDataResource resource = new ArbitraryDataResource(resourceInfo.name, ArbitraryDataFile.ResourceIdType.NAME, + resourceInfo.service, resourceInfo.identifier); + ArbitraryDataTransactionMetadata transactionMetadata = resource.getLatestTransactionMetadata(); + ArbitraryResourceMetadata resourceMetadata = ArbitraryResourceMetadata.fromTransactionMetadata(transactionMetadata, false); + if (resourceMetadata != null) { + resourceInfo.metadata = resourceMetadata; + } + updatedResources.add(resourceInfo); + } + return updatedResources; + } + } diff --git a/src/main/java/org/qortal/utils/FilesystemUtils.java b/src/main/java/org/qortal/utils/FilesystemUtils.java index 1b3de544..e9921561 100644 --- a/src/main/java/org/qortal/utils/FilesystemUtils.java +++ b/src/main/java/org/qortal/utils/FilesystemUtils.java @@ -228,12 +228,18 @@ public class FilesystemUtils { * @throws IOException */ public static byte[] getSingleFileContents(Path path) throws IOException { + return getSingleFileContents(path, null); + } + + public static byte[] getSingleFileContents(Path path, Integer maxLength) throws IOException { byte[] data = null; // TODO: limit the file size that can be loaded into memory // If the path is a file, read the contents directly if (path.toFile().isFile()) { - data = Files.readAllBytes(path); + int fileSize = (int)path.toFile().length(); + maxLength = maxLength != null ? Math.min(maxLength, fileSize) : fileSize; + data = FilesystemUtils.readFromFile(path.toString(), 0, maxLength); } // Or if it's a directory, only load file contents if there is a single file inside it @@ -241,13 +247,50 @@ public class FilesystemUtils { String[] files = ArrayUtils.removeElement(path.toFile().list(), ".qortal"); if (files.length == 1) { Path filePath = Paths.get(path.toString(), files[0]); - data = Files.readAllBytes(filePath); + if (filePath.toFile().isFile()) { + int fileSize = (int)filePath.toFile().length(); + maxLength = maxLength != null ? Math.min(maxLength, fileSize) : fileSize; + data = FilesystemUtils.readFromFile(filePath.toString(), 0, maxLength); + } } } return data; } + /** + * isSingleFileResource + * Returns true if the path points to a file, or a + * directory containing a single file only. + * + * @param path to file or directory + * @param excludeQortalDirectory - if true, a directory containing a single file and a .qortal directory is considered a single file resource + * @return + * @throws IOException + */ + public static boolean isSingleFileResource(Path path, boolean excludeQortalDirectory) { + // If the path is a file, read the contents directly + if (path.toFile().isFile()) { + return true; + } + + // Or if it's a directory, only load file contents if there is a single file inside it + else if (path.toFile().isDirectory()) { + String[] files = path.toFile().list(); + if (excludeQortalDirectory) { + files = ArrayUtils.removeElement(files, ".qortal"); + } + if (files.length == 1) { + Path filePath = Paths.get(path.toString(), files[0]); + if (filePath.toFile().isFile()) { + return true; + } + } + } + + return false; + } + public static byte[] readFromFile(String filePath, long position, int size) throws IOException { RandomAccessFile file = new RandomAccessFile(filePath, "r"); file.seek(position); diff --git a/src/main/java/org/qortal/utils/ListUtils.java b/src/main/java/org/qortal/utils/ListUtils.java new file mode 100644 index 00000000..73c847d2 --- /dev/null +++ b/src/main/java/org/qortal/utils/ListUtils.java @@ -0,0 +1,38 @@ +package org.qortal.utils; + +import org.qortal.list.ResourceListManager; + +import java.util.List; + +public class ListUtils { + + /* Blocking */ + + public static List blockedNames() { + return ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames"); + } + + public static boolean isNameBlocked(String name) { + return ResourceListManager.getInstance().listWithPrefixContains("blockedNames", name, false); + } + + public static boolean isAddressBlocked(String address) { + return ResourceListManager.getInstance().listWithPrefixContains("blockedAddresses", address, true); + } + + + /* Following */ + + public static List followedNames() { + return ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames"); + } + + public static boolean isFollowingName(String name) { + return ResourceListManager.getInstance().listWithPrefixContains("followedNames", name, false); + } + + public static int followedNamesCount() { + return ListUtils.followedNames().size(); + } + +} diff --git a/src/main/resources/blockchain.json b/src/main/resources/blockchain.json index 7ce93a28..c6151204 100644 --- a/src/main/resources/blockchain.json +++ b/src/main/resources/blockchain.json @@ -86,7 +86,7 @@ "selfSponsorshipAlgoV1Height": 1092400, "feeValidationFixTimestamp": 1671918000000, "chatReferenceTimestamp": 1674316800000, - "arbitraryOptionalFeeTimestamp": 9999999999999 + "arbitraryOptionalFeeTimestamp": 1680278400000 }, "checkpoints": [ { "height": 1136300, "signature": "3BbwawEF2uN8Ni5ofpJXkukoU8ctAPxYoFB7whq9pKfBnjfZcpfEJT4R95NvBDoTP8WDyWvsUvbfHbcr9qSZuYpSKZjUQTvdFf6eqznHGEwhZApWfvXu6zjGCxYCp65F4jsVYYJjkzbjmkCg5WAwN5voudngA23kMK6PpTNygapCzXt" } diff --git a/src/main/resources/loading/index.html b/src/main/resources/loading/index.html index a828e04e..574645cc 100644 --- a/src/main/resources/loading/index.html +++ b/src/main/resources/loading/index.html @@ -43,8 +43,9 @@ var host = location.protocol + '//' + location.host; var service = "%%SERVICE%%" var name = "%%NAME%%" + var identifier = "%%IDENTIFIER%%" - var url = host + '/arbitrary/resource/status/' + service + '/' + name + '?build=true'; + var url = host + '/arbitrary/resource/status/' + service + '/' + name + '/' + identifier + '?build=true'; var textStatus = "Loading..."; var textProgress = ""; var retryInterval = 2500; @@ -74,18 +75,18 @@ } else if (status.id == "BUILDING") { textStatus = status.description; - retryInterval = 1000; + retryInterval = 2000; } else if (status.id == "BUILD_FAILED") { textStatus = status.description; } else if (status.id == "NOT_STARTED") { textStatus = status.description; - retryInterval = 1000; + retryInterval = 2000; } else if (status.id == "DOWNLOADING") { textStatus = status.description; - retryInterval = 1000; + retryInterval = 2000; } else if (status.id == "MISSING_DATA") { textStatus = status.description; @@ -96,8 +97,14 @@ else if (status.id == "DOWNLOADED") { textStatus = status.description; } + else if (status.id == "NOT_PUBLISHED") { + document.getElementById("title").innerHTML = "File not found"; + document.getElementById("description").innerHTML = ""; + document.getElementById("c").style.opacity = "0.5"; + textStatus = status.description; + } - if (status.localChunkCount != null && status.totalChunkCount != null) { + if (status.localChunkCount != null && status.totalChunkCount != null && status.totalChunkCount > 0) { textProgress = "Files downloaded: " + status.localChunkCount + " / " + status.totalChunkCount; } @@ -275,8 +282,8 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
-

Loading

-

+

Loading

+

Files are being retrieved from the Qortal Data Network. This page will refresh automatically when the content becomes available.

diff --git a/src/main/resources/q-apps/q-apps-gateway.js b/src/main/resources/q-apps/q-apps-gateway.js new file mode 100644 index 00000000..d5028dca --- /dev/null +++ b/src/main/resources/q-apps/q-apps-gateway.js @@ -0,0 +1,76 @@ +console.log("Gateway mode"); + +function qdnGatewayShowModal(message) { + const modalElementId = "qdnGatewayModal"; + + if (document.getElementById(modalElementId) != null) { + document.body.removeChild(document.getElementById(modalElementId)); + } + + var modalElement = document.createElement('div'); + modalElement.style.cssText = 'position:fixed; z-index:99999; background:#fff; padding:20px; border-radius:5px; font-family:sans-serif; bottom:20px; right:20px; color:#000; max-width:400px; box-shadow:0 3px 10px rgb(0 0 0 / 0.2); font-family:arial; font-weight:normal; font-size:16px;'; + modalElement.innerHTML = message + "

"; + modalElement.id = modalElementId; + + var closeButton = document.createElement('button'); + closeButton.style.cssText = 'background-color:#008CBA; border:none; color:white; cursor:pointer; float: right; margin: 10px; padding:15px; border-radius:5px; display:inline-block; text-align:center; text-decoration:none; font-family:arial; font-weight:normal; font-size:16px;'; + closeButton.innerText = "Close"; + closeButton.addEventListener ("click", function() { + document.body.removeChild(document.getElementById(modalElementId)); + }); + modalElement.appendChild(closeButton); + + var qortalButton = document.createElement('button'); + qortalButton.style.cssText = 'background-color:#4CAF50; border:none; color:white; cursor:pointer; float: right; margin: 10px; padding:15px; border-radius:5px; text-align:center; text-decoration:none; display:inline-block; font-family:arial; font-weight:normal; font-size:16px;'; + qortalButton.innerText = "Learn more"; + qortalButton.addEventListener ("click", function() { + document.body.removeChild(document.getElementById(modalElementId)); + window.open("https://qortal.org"); + }); + modalElement.appendChild(qortalButton); + + document.body.appendChild(modalElement); +} + +window.addEventListener("message", (event) => { + if (event == null || event.data == null || event.data.length == 0) { + return; + } + if (event.data.action == null || event.data.requestedHandler == null) { + return; + } + if (event.data.requestedHandler != "UI") { + // Gateway mode only cares about requests that were intended for the UI + return; + } + + let response; + let data = event.data; + + switch (data.action) { + case "GET_USER_ACCOUNT": + case "PUBLISH_QDN_RESOURCE": + case "PUBLISH_MULTIPLE_QDN_RESOURCES": + case "SEND_CHAT_MESSAGE": + case "JOIN_GROUP": + case "DEPLOY_AT": + case "GET_WALLET_BALANCE": + case "SEND_COIN": + case "GET_LIST_ITEMS": + case "ADD_LIST_ITEMS": + case "DELETE_LIST_ITEM": + const errorString = "Interactive features were requested, but these are not yet supported when viewing via a gateway. To use interactive features, please access using the Qortal UI desktop app. More info at: https://qortal.org"; + response = "{\"error\": \"" + errorString + "\"}" + + const modalText = "This app is powered by the Qortal blockchain. You are viewing in read-only mode. To use interactive features, please access using the Qortal UI desktop app."; + qdnGatewayShowModal(modalText); + break; + + default: + console.log('Unhandled gateway message: ' + JSON.stringify(data)); + return; + } + + handleResponse(event, response); + +}, false); \ No newline at end of file diff --git a/src/main/resources/q-apps/q-apps.js b/src/main/resources/q-apps/q-apps.js new file mode 100644 index 00000000..dae20e5d --- /dev/null +++ b/src/main/resources/q-apps/q-apps.js @@ -0,0 +1,535 @@ +function httpGet(url) { + var request = new XMLHttpRequest(); + request.open("GET", url, false); + request.send(null); + return request.responseText; +} + +function httpGetAsyncWithEvent(event, url) { + fetch(url) + .then((response) => response.text()) + .then((responseText) => { + + if (responseText == null) { + // Pass to parent (UI), in case they can fulfil this request + event.data.requestedHandler = "UI"; + parent.postMessage(event.data, '*', [event.ports[0]]); + return; + } + + handleResponse(event, responseText); + + }) + .catch((error) => { + let res = {}; + res.error = error; + handleResponse(event, JSON.stringify(res)); + }) +} + +function handleResponse(event, response) { + if (event == null) { + return; + } + + // Handle empty or missing responses + if (response == null || response.length == 0) { + response = "{\"error\": \"Empty response\"}" + } + + // Parse response + let responseObj; + try { + responseObj = JSON.parse(response); + } catch (e) { + // Not all responses will be JSON + responseObj = response; + } + + // GET_QDN_RESOURCE_URL has custom handling + const data = event.data; + if (data.action == "GET_QDN_RESOURCE_URL") { + if (responseObj == null || responseObj.status == null || responseObj.status == "NOT_PUBLISHED") { + responseObj = {}; + responseObj.error = "Resource does not exist"; + } + else { + responseObj = buildResourceUrl(data.service, data.name, data.identifier, data.path, false); + } + } + + // Respond to app + if (responseObj.error != null) { + event.ports[0].postMessage({ + result: null, + error: responseObj + }); + } + else { + event.ports[0].postMessage({ + result: responseObj, + error: null + }); + } +} + +function buildResourceUrl(service, name, identifier, path, isLink) { + if (isLink == false) { + // If this URL isn't being used as a link, then we need to fetch the data + // synchronously, instead of showing the loading screen. + url = "/arbitrary/" + service + "/" + name; + if (identifier != null) url = url.concat("/" + identifier); + if (path != null) url = url.concat("?filepath=" + path); + } + else if (_qdnContext == "render") { + url = "/render/" + service + "/" + name; + if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path); + if (identifier != null) url = url.concat("?identifier=" + identifier); + } + else if (_qdnContext == "gateway") { + url = "/" + service + "/" + name; + if (identifier != null) url = url.concat("/" + identifier); + if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path); + } + else { + // domainMap only serves websites right now + url = "/" + name; + if (path != null) url = url.concat((path.startsWith("/") ? "" : "/") + path); + } + + if (isLink) url = url.concat((url.includes("?") ? "" : "?") + "&theme=" + _qdnTheme); + + return url; +} + +function extractComponents(url) { + if (!url.startsWith("qortal://")) { + return null; + } + + url = url.replace(/^(qortal\:\/\/)/,""); + if (url.includes("/")) { + let parts = url.split("/"); + const service = parts[0].toUpperCase(); + parts.shift(); + const name = parts[0]; + parts.shift(); + let identifier; + + if (parts.length > 0) { + identifier = parts[0]; // Do not shift yet + // Check if a resource exists with this service, name and identifier combination + const url = "/arbitrary/resource/status/" + service + "/" + name + "/" + identifier; + const response = httpGet(url); + const responseObj = JSON.parse(response); + if (responseObj.totalChunkCount > 0) { + // Identifier exists, so don't include it in the path + parts.shift(); + } + else { + identifier = null; + } + } + + const path = parts.join("/"); + + const components = {}; + components["service"] = service; + components["name"] = name; + components["identifier"] = identifier; + components["path"] = path; + return components; + } + + return null; +} + +function convertToResourceUrl(url, isLink) { + if (!url.startsWith("qortal://")) { + return null; + } + const c = extractComponents(url); + if (c == null) { + return null; + } + + return buildResourceUrl(c.service, c.name, c.identifier, c.path, isLink); +} + +window.addEventListener("message", (event) => { + if (event == null || event.data == null || event.data.length == 0) { + return; + } + if (event.data.action == null) { + // This could be a response from the UI + handleResponse(event, event.data); + } + if (event.data.requestedHandler != null && event.data.requestedHandler === "UI") { + // This request was destined for the UI, so ignore it + return; + } + + console.log("Core received action: " + JSON.stringify(event.data.action)); + + let url; + let data = event.data; + + switch (data.action) { + case "GET_ACCOUNT_DATA": + return httpGetAsyncWithEvent(event, "/addresses/" + data.address); + + case "GET_ACCOUNT_NAMES": + return httpGetAsyncWithEvent(event, "/names/address/" + data.address); + + case "SEARCH_NAMES": + url = "/names/search?"; + if (data.query != null) url = url.concat("&query=" + data.query); + if (data.prefix != null) url = url.concat("&prefix=" + new Boolean(data.prefix).toString()); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "GET_NAME_DATA": + return httpGetAsyncWithEvent(event, "/names/" + data.name); + + case "GET_QDN_RESOURCE_URL": + // Check status first; URL is built and returned automatically after status check + url = "/arbitrary/resource/status/" + data.service + "/" + data.name; + if (data.identifier != null) url = url.concat("/" + data.identifier); + return httpGetAsyncWithEvent(event, url); + + case "LINK_TO_QDN_RESOURCE": + if (data.service == null) data.service = "WEBSITE"; // Default to WEBSITE + window.location = buildResourceUrl(data.service, data.name, data.identifier, data.path, true); + return; + + case "LIST_QDN_RESOURCES": + url = "/arbitrary/resources?"; + if (data.service != null) url = url.concat("&service=" + data.service); + if (data.name != null) url = url.concat("&name=" + data.name); + if (data.identifier != null) url = url.concat("&identifier=" + data.identifier); + if (data.default != null) url = url.concat("&default=" + new Boolean(data.default).toString()); + if (data.includeStatus != null) url = url.concat("&includestatus=" + new Boolean(data.includeStatus).toString()); + if (data.includeMetadata != null) url = url.concat("&includemetadata=" + new Boolean(data.includeMetadata).toString()); + if (data.nameListFilter != null) url = url.concat("&namefilter=" + data.nameListFilter); + if (data.followedOnly != null) url = url.concat("&followedonly=" + new Boolean(data.followedOnly).toString()); + if (data.excludeBlocked != null) url = url.concat("&excludeblocked=" + new Boolean(data.excludeBlocked).toString()); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "SEARCH_QDN_RESOURCES": + url = "/arbitrary/resources/search?"; + if (data.service != null) url = url.concat("&service=" + data.service); + if (data.query != null) url = url.concat("&query=" + data.query); + if (data.identifier != null) url = url.concat("&identifier=" + data.identifier); + if (data.name != null) url = url.concat("&name=" + data.name); + if (data.names != null) data.names.forEach((x, i) => url = url.concat("&name=" + x)); + if (data.prefix != null) url = url.concat("&prefix=" + new Boolean(data.prefix).toString()); + if (data.exactMatchNames != null) url = url.concat("&exactmatchnames=" + new Boolean(data.exactMatchNames).toString()); + if (data.default != null) url = url.concat("&default=" + new Boolean(data.default).toString()); + if (data.includeStatus != null) url = url.concat("&includestatus=" + new Boolean(data.includeStatus).toString()); + if (data.includeMetadata != null) url = url.concat("&includemetadata=" + new Boolean(data.includeMetadata).toString()); + if (data.nameListFilter != null) url = url.concat("&namefilter=" + data.nameListFilter); + if (data.followedOnly != null) url = url.concat("&followedonly=" + new Boolean(data.followedOnly).toString()); + if (data.excludeBlocked != null) url = url.concat("&excludeblocked=" + new Boolean(data.excludeBlocked).toString()); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "FETCH_QDN_RESOURCE": + url = "/arbitrary/" + data.service + "/" + data.name; + if (data.identifier != null) url = url.concat("/" + data.identifier); + url = url.concat("?"); + if (data.filepath != null) url = url.concat("&filepath=" + data.filepath); + if (data.rebuild != null) url = url.concat("&rebuild=" + new Boolean(data.rebuild).toString()); + if (data.encoding != null) url = url.concat("&encoding=" + data.encoding); + return httpGetAsyncWithEvent(event, url); + + case "GET_QDN_RESOURCE_STATUS": + url = "/arbitrary/resource/status/" + data.service + "/" + data.name; + if (data.identifier != null) url = url.concat("/" + data.identifier); + url = url.concat("?"); + if (data.build != null) url = url.concat("&build=" + new Boolean(data.build).toString()); + return httpGetAsyncWithEvent(event, url); + + case "GET_QDN_RESOURCE_PROPERTIES": + let identifier = (data.identifier != null) ? data.identifier : "default"; + url = "/arbitrary/resource/properties/" + data.service + "/" + data.name + "/" + identifier; + return httpGetAsyncWithEvent(event, url); + + case "GET_QDN_RESOURCE_METADATA": + identifier = (data.identifier != null) ? data.identifier : "default"; + url = "/arbitrary/metadata/" + data.service + "/" + data.name + "/" + identifier; + return httpGetAsyncWithEvent(event, url); + + case "SEARCH_CHAT_MESSAGES": + url = "/chat/messages?"; + if (data.before != null) url = url.concat("&before=" + data.before); + if (data.after != null) url = url.concat("&after=" + data.after); + if (data.txGroupId != null) url = url.concat("&txGroupId=" + data.txGroupId); + if (data.involving != null) data.involving.forEach((x, i) => url = url.concat("&involving=" + x)); + if (data.reference != null) url = url.concat("&reference=" + data.reference); + if (data.chatReference != null) url = url.concat("&chatreference=" + data.chatReference); + if (data.hasChatReference != null) url = url.concat("&haschatreference=" + new Boolean(data.hasChatReference).toString()); + if (data.encoding != null) url = url.concat("&encoding=" + data.encoding); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "LIST_GROUPS": + url = "/groups?"; + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "GET_BALANCE": + url = "/addresses/balance/" + data.address; + if (data.assetId != null) url = url.concat("&assetId=" + data.assetId); + return httpGetAsyncWithEvent(event, url); + + case "GET_AT": + url = "/at" + data.atAddress; + return httpGetAsyncWithEvent(event, url); + + case "GET_AT_DATA": + url = "/at/" + data.atAddress + "/data"; + return httpGetAsyncWithEvent(event, url); + + case "LIST_ATS": + url = "/at/byfunction/" + data.codeHash58 + "?"; + if (data.isExecutable != null) url = url.concat("&isExecutable=" + data.isExecutable); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "FETCH_BLOCK": + if (data.signature != null) { + url = "/blocks/" + data.signature; + } else if (data.height != null) { + url = "/blocks/byheight/" + data.height; + } + url = url.concat("?"); + if (data.includeOnlineSignatures != null) url = url.concat("&includeOnlineSignatures=" + data.includeOnlineSignatures); + return httpGetAsyncWithEvent(event, url); + + case "FETCH_BLOCK_RANGE": + url = "/blocks/range/" + data.height + "?"; + if (data.count != null) url = url.concat("&count=" + data.count); + if (data.reverse != null) url = url.concat("&reverse=" + data.reverse); + if (data.includeOnlineSignatures != null) url = url.concat("&includeOnlineSignatures=" + data.includeOnlineSignatures); + return httpGetAsyncWithEvent(event, url); + + case "SEARCH_TRANSACTIONS": + url = "/transactions/search?"; + if (data.startBlock != null) url = url.concat("&startBlock=" + data.startBlock); + if (data.blockLimit != null) url = url.concat("&blockLimit=" + data.blockLimit); + if (data.txGroupId != null) url = url.concat("&txGroupId=" + data.txGroupId); + if (data.txType != null) data.txType.forEach((x, i) => url = url.concat("&txType=" + x)); + if (data.address != null) url = url.concat("&address=" + data.address); + if (data.confirmationStatus != null) url = url.concat("&confirmationStatus=" + data.confirmationStatus); + if (data.limit != null) url = url.concat("&limit=" + data.limit); + if (data.offset != null) url = url.concat("&offset=" + data.offset); + if (data.reverse != null) url = url.concat("&reverse=" + new Boolean(data.reverse).toString()); + return httpGetAsyncWithEvent(event, url); + + case "GET_PRICE": + url = "/crosschain/price/" + data.blockchain + "?"; + if (data.maxtrades != null) url = url.concat("&maxtrades=" + data.maxtrades); + if (data.inverse != null) url = url.concat("&inverse=" + data.inverse); + return httpGetAsyncWithEvent(event, url); + + default: + // Pass to parent (UI), in case they can fulfil this request + event.data.requestedHandler = "UI"; + parent.postMessage(event.data, '*', [event.ports[0]]); + return; + } + +}, false); + + +/** + * Listen for and intercept all link click events + */ +function interceptClickEvent(e) { + var target = e.target || e.srcElement; + if (target.tagName !== 'A') { + target = target.closest('A'); + } + if (target == null || target.getAttribute('href') == null) { + return; + } + let href = target.getAttribute('href'); + if (href.startsWith("qortal://")) { + const c = extractComponents(href); + if (c != null) { + qortalRequest({ + action: "LINK_TO_QDN_RESOURCE", + service: c.service, + name: c.name, + identifier: c.identifier, + path: c.path + }); + } + e.preventDefault(); + } + else if (href.startsWith("http://") || href.startsWith("https://") || href.startsWith("//")) { + // Block external links + e.preventDefault(); + } +} +if (document.addEventListener) { + document.addEventListener('click', interceptClickEvent); +} +else if (document.attachEvent) { + document.attachEvent('onclick', interceptClickEvent); +} + + + +/** + * Intercept image loads from the DOM + */ +document.addEventListener('DOMContentLoaded', () => { + const imgElements = document.querySelectorAll('img'); + imgElements.forEach((img) => { + let url = img.src; + const newUrl = convertToResourceUrl(url, false); + if (newUrl != null) { + document.querySelector('img').src = newUrl; + } + }); +}); + +/** + * Intercept img src updates + */ +document.addEventListener('DOMContentLoaded', () => { + const imgElements = document.querySelectorAll('img'); + imgElements.forEach((img) => { + let observer = new MutationObserver((changes) => { + changes.forEach(change => { + if (change.attributeName.includes('src')) { + const newUrl = convertToResourceUrl(img.src, false); + if (newUrl != null) { + document.querySelector('img').src = newUrl; + } + } + }); + }); + observer.observe(img, {attributes: true}); + }); +}); + + + +const awaitTimeout = (timeout, reason) => + new Promise((resolve, reject) => + setTimeout( + () => (reason === undefined ? resolve() : reject(reason)), + timeout + ) + ); + +function getDefaultTimeout(action) { + if (action != null) { + // Some actions need longer default timeouts, especially those that create transactions + switch (action) { + case "GET_USER_ACCOUNT": + case "SAVE_FILE": + case "DECRYPT_DATA": + // User may take a long time to accept/deny the popup + return 60 * 60 * 1000; + + case "FETCH_QDN_RESOURCE": + // Fetching data can take a while, especially if the status hasn't been checked first + return 60 * 1000; + + case "PUBLISH_QDN_RESOURCE": + case "PUBLISH_MULTIPLE_QDN_RESOURCES": + // Publishing could take a very long time on slow system, due to the proof-of-work computation + return 60 * 60 * 1000; + + case "SEND_CHAT_MESSAGE": + // Chat messages rely on PoW computations, so allow extra time + return 60 * 1000; + + case "JOIN_GROUP": + case "DEPLOY_AT": + case "SEND_COIN": + // Allow extra time for other actions that create transactions, even if there is no PoW + return 5 * 60 * 1000; + + default: + break; + } + } + return 10 * 1000; +} + +/** + * Make a Qortal (Q-Apps) request with no timeout + */ +const qortalRequestWithNoTimeout = (request) => new Promise((res, rej) => { + const channel = new MessageChannel(); + + channel.port1.onmessage = ({data}) => { + channel.port1.close(); + + if (data.error) { + rej(data.error); + } else { + res(data.result); + } + }; + + window.postMessage(request, '*', [channel.port2]); +}); + +/** + * Make a Qortal (Q-Apps) request with the default timeout (10 seconds) + */ +const qortalRequest = (request) => + Promise.race([qortalRequestWithNoTimeout(request), awaitTimeout(getDefaultTimeout(request.action), "The request timed out")]); + +/** + * Make a Qortal (Q-Apps) request with a custom timeout, specified in milliseconds + */ +const qortalRequestWithTimeout = (request, timeout) => + Promise.race([qortalRequestWithNoTimeout(request), awaitTimeout(timeout, "The request timed out")]); + + +/** + * Send current page details to UI + */ +document.addEventListener('DOMContentLoaded', () => { + qortalRequest({ + action: "QDN_RESOURCE_DISPLAYED", + service: _qdnService, + name: _qdnName, + identifier: _qdnIdentifier, + path: _qdnPath + }); +}); + +/** + * Handle app navigation + */ +navigation.addEventListener('navigate', (event) => { + const url = new URL(event.destination.url); + let fullpath = url.pathname + url.hash; + qortalRequest({ + action: "QDN_RESOURCE_DISPLAYED", + service: _qdnService, + name: _qdnName, + identifier: _qdnIdentifier, + path: (fullpath.startsWith(_qdnBase)) ? fullpath.slice(_qdnBase.length) : fullpath + }); +}); diff --git a/src/test/java/org/qortal/test/ListTests.java b/src/test/java/org/qortal/test/ListTests.java new file mode 100644 index 00000000..75ff8d36 --- /dev/null +++ b/src/test/java/org/qortal/test/ListTests.java @@ -0,0 +1,129 @@ +package org.qortal.test; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.qortal.list.ResourceList; +import org.qortal.list.ResourceListManager; +import org.qortal.repository.DataException; +import org.qortal.settings.Settings; +import org.qortal.test.common.Common; +import org.qortal.utils.ListUtils; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.List; + +import static org.junit.Assert.*; + +public class ListTests { + + @Before + public void beforeTest() throws DataException, IOException { + Common.useDefaultSettings(); + this.cleanup(); + } + + @After + public void afterTest() throws DataException, IOException { + this.cleanup(); + } + + private void cleanup() throws IOException { + // Delete custom lists created by test methods + ResourceList followedNamesTestList = new ResourceList("followedNames_test"); + followedNamesTestList.clear(); + followedNamesTestList.save(); + + ResourceList blockedNamesTestList = new ResourceList("blockedNames_test"); + blockedNamesTestList.clear(); + blockedNamesTestList.save(); + + // Clear resource list manager instance + ResourceListManager.reset(); + } + + @Test + public void testSingleList() { + ResourceListManager resourceListManager = ResourceListManager.getInstance(); + String listName = "followedNames_test"; + String name = "testName"; + + resourceListManager.addToList(listName, name, false); + + List followedNames = resourceListManager.getStringsInList(listName); + assertEquals(1, followedNames.size()); + assertEquals(followedNames.size(), ListUtils.followedNamesCount()); + assertEquals(name, followedNames.get(0)); + } + + @Test + public void testListPrefix() { + ResourceListManager resourceListManager = ResourceListManager.getInstance(); + + List initialFollowedNames = resourceListManager.getStringsInListsWithPrefix("followedNames"); + assertEquals(0, initialFollowedNames.size()); + + List initialBlockedNames = resourceListManager.getStringsInListsWithPrefix("blockedNames"); + assertEquals(0, initialBlockedNames.size()); + + // Add to multiple lists + resourceListManager.addToList("followedNames_CustomList1", "testName1", false); + resourceListManager.addToList("followedNames_CustomList1", "testName2", false); + resourceListManager.addToList("followedNames_CustomList2", "testName3", false); + resourceListManager.addToList("followedNames_CustomList3", "testName4", false); + resourceListManager.addToList("blockedNames_CustomList1", "testName5", false); + + // Check followedNames + List followedNames = resourceListManager.getStringsInListsWithPrefix("followedNames"); + assertEquals(4, followedNames.size()); + assertEquals(followedNames.size(), ListUtils.followedNamesCount()); + assertTrue(followedNames.contains("testName1")); + assertTrue(followedNames.contains("testName2")); + assertTrue(followedNames.contains("testName3")); + assertTrue(followedNames.contains("testName4")); + assertFalse(followedNames.contains("testName5")); + + // Check blockedNames + List blockedNames = resourceListManager.getStringsInListsWithPrefix("blockedNames"); + assertEquals(1, blockedNames.size()); + assertEquals(blockedNames.size(), ListUtils.blockedNames().size()); + assertTrue(blockedNames.contains("testName5")); + } + + @Test + public void testDataPersistence() { + // Ensure lists are empty to begin with + assertEquals(0, ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames").size()); + assertEquals(0, ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames").size()); + + // Add some items to multiple lists + ResourceListManager.getInstance().addToList("followedNames_test", "testName1", true); + ResourceListManager.getInstance().addToList("followedNames_test", "testName2", true); + ResourceListManager.getInstance().addToList("blockedNames_test", "testName3", true); + + // Ensure they are added + assertEquals(2, ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames").size()); + assertEquals(1, ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames").size()); + + // Clear local state + ResourceListManager.reset(); + + // Ensure items are automatically loaded back in from disk + assertEquals(2, ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames").size()); + assertEquals(1, ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames").size()); + + // Delete followedNames file + File followedNamesFile = Paths.get(Settings.getInstance().getListsPath(), "followedNames_test.json").toFile(); + followedNamesFile.delete(); + + // Clear local state again + ResourceListManager.reset(); + + // Ensure only the blocked names are loaded back in + assertEquals(0, ResourceListManager.getInstance().getStringsInListsWithPrefix("followedNames").size()); + assertEquals(1, ResourceListManager.getInstance().getStringsInListsWithPrefix("blockedNames").size()); + } + +} diff --git a/src/test/java/org/qortal/test/RepositoryTests.java b/src/test/java/org/qortal/test/RepositoryTests.java index bb6510d5..30cbaea5 100644 --- a/src/test/java/org/qortal/test/RepositoryTests.java +++ b/src/test/java/org/qortal/test/RepositoryTests.java @@ -9,6 +9,7 @@ import org.qortal.crosschain.BitcoinACCTv1; import org.qortal.crypto.Crypto; import org.qortal.data.account.AccountBalanceData; import org.qortal.data.account.AccountData; +import org.qortal.data.chat.ChatMessage; import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; @@ -417,7 +418,7 @@ public class RepositoryTests extends Common { try (final HSQLDBRepository hsqldb = (HSQLDBRepository) RepositoryManager.getRepository()) { String address = Crypto.toAddress(new byte[32]); - hsqldb.getChatRepository().getActiveChats(address); + hsqldb.getChatRepository().getActiveChats(address, ChatMessage.Encoding.BASE58); } catch (DataException e) { fail("HSQLDB bug #1580"); } diff --git a/src/test/java/org/qortal/test/api/AdminApiTests.java b/src/test/java/org/qortal/test/api/AdminApiTests.java index b3e6da03..01f2ebc9 100644 --- a/src/test/java/org/qortal/test/api/AdminApiTests.java +++ b/src/test/java/org/qortal/test/api/AdminApiTests.java @@ -5,7 +5,7 @@ import static org.junit.Assert.*; import org.apache.commons.lang3.reflect.FieldUtils; import org.junit.Before; import org.junit.Test; -import org.qortal.api.resource.AdminResource; +import org.qortal.api.restricted.resource.AdminResource; import org.qortal.repository.DataException; import org.qortal.settings.Settings; import org.qortal.test.common.ApiCommon; diff --git a/src/test/java/org/qortal/test/api/NamesApiTests.java b/src/test/java/org/qortal/test/api/NamesApiTests.java index 0e03b6a6..effdfea4 100644 --- a/src/test/java/org/qortal/test/api/NamesApiTests.java +++ b/src/test/java/org/qortal/test/api/NamesApiTests.java @@ -37,8 +37,8 @@ public class NamesApiTests extends ApiCommon { @Test public void testGetAllNames() { - assertNotNull(this.namesResource.getAllNames(null, null, null)); - assertNotNull(this.namesResource.getAllNames(1, 1, true)); + assertNotNull(this.namesResource.getAllNames(null, null, null, null)); + assertNotNull(this.namesResource.getAllNames(1L, 1, 1, true)); } @Test diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataFileTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataFileTests.java index aabbe502..d2ee61c6 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataFileTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataFileTests.java @@ -20,7 +20,7 @@ public class ArbitraryDataFileTests extends Common { @Test public void testSplitAndJoin() throws DataException { String dummyDataString = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890"; - ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(dummyDataString.getBytes(), null); + ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(dummyDataString.getBytes(), null, false); assertTrue(arbitraryDataFile.exists()); assertEquals(62, arbitraryDataFile.size()); assertEquals("3eyjYjturyVe61grRX42bprGr3Cvw6ehTy4iknVnosDj", arbitraryDataFile.digest58()); @@ -50,7 +50,7 @@ public class ArbitraryDataFileTests extends Common { byte[] randomData = new byte[fileSize]; new Random().nextBytes(randomData); // No need for SecureRandom here - ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(randomData, null); + ArbitraryDataFile arbitraryDataFile = new ArbitraryDataFile(randomData, null, false); assertTrue(arbitraryDataFile.exists()); assertEquals(fileSize, arbitraryDataFile.size()); String originalFileDigest = arbitraryDataFile.digest58(); diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataStorageCapacityTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataStorageCapacityTests.java index 2042a30c..c05ceabf 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataStorageCapacityTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataStorageCapacityTests.java @@ -24,6 +24,7 @@ import org.qortal.test.common.TransactionUtils; import org.qortal.test.common.transaction.TestTransaction; import org.qortal.transaction.RegisterNameTransaction; import org.qortal.utils.Base58; +import org.qortal.utils.ListUtils; import org.qortal.utils.NTP; import java.io.IOException; @@ -103,6 +104,7 @@ public class ArbitraryDataStorageCapacityTests extends Common { // Storage capacity should initially equal the total assertEquals(0, resourceListManager.getItemCountForList("followedNames")); + assertEquals(0, ListUtils.followedNamesCount()); long totalStorageCapacity = storageManager.getStorageCapacityIncludingThreshold(storageFullThreshold); assertEquals(totalStorageCapacity, storageManager.storageCapacityPerName(storageFullThreshold)); @@ -111,12 +113,16 @@ public class ArbitraryDataStorageCapacityTests extends Common { assertTrue(resourceListManager.addToList("followedNames", "Test2", false)); assertTrue(resourceListManager.addToList("followedNames", "Test3", false)); assertTrue(resourceListManager.addToList("followedNames", "Test4", false)); + assertTrue(resourceListManager.addToList("followedNames", "Test5", false)); + assertTrue(resourceListManager.addToList("followedNames", "Test6", false)); // Ensure the followed name count is correct - assertEquals(4, resourceListManager.getItemCountForList("followedNames")); + assertEquals(6, resourceListManager.getItemCountForList("followedNames")); + assertEquals(6, ListUtils.followedNamesCount()); // Storage space per name should be the total storage capacity divided by the number of names - long expectedStorageCapacityPerName = (long)(totalStorageCapacity / 4.0f); + // then multiplied by 4, to allow for names that don't use much space + long expectedStorageCapacityPerName = (long)(totalStorageCapacity / 6.0f) * 4L; assertEquals(expectedStorageCapacityPerName, storageManager.storageCapacityPerName(storageFullThreshold)); } diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataTests.java index 573c2ae2..cfc656e1 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryDataTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryDataTests.java @@ -111,8 +111,8 @@ public class ArbitraryDataTests extends Common { fail("Creating transaction should fail due to nonexistent PUT transaction"); } catch (DataException expectedException) { - assertEquals(String.format("Couldn't find PUT transaction for " + - "name %s, service %s and identifier ", name, service), expectedException.getMessage()); + assertTrue(expectedException.getMessage().contains(String.format("Couldn't find PUT transaction for " + + "name %s, service %s and identifier ", name, service))); } } @@ -358,7 +358,7 @@ public class ArbitraryDataTests extends Common { byte[] path1FileDigest = Crypto.digest(path1.toFile()); ArbitraryDataDigest path1DirectoryDigest = new ArbitraryDataDigest(path1.getParent()); path1DirectoryDigest.compute(); - ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, Method.PUT, service, alice); + ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, identifier, Method.PUT, service, alice); // Now build the latest data state for this name ArbitraryDataReader arbitraryDataReader1 = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier); diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryEncryptionTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryEncryptionTests.java new file mode 100644 index 00000000..2e4dc133 --- /dev/null +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryEncryptionTests.java @@ -0,0 +1,135 @@ +package org.qortal.test.arbitrary; + +import org.junit.Before; +import org.junit.Test; +import org.qortal.arbitrary.ArbitraryDataDigest; +import org.qortal.crypto.AES; +import org.qortal.crypto.Crypto; +import org.qortal.repository.DataException; +import org.qortal.test.common.Common; +import org.qortal.utils.ZipUtils; + +import javax.crypto.BadPaddingException; +import javax.crypto.IllegalBlockSizeException; +import javax.crypto.NoSuchPaddingException; +import javax.crypto.SecretKey; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardOpenOption; +import java.security.InvalidAlgorithmParameterException; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; +import java.util.Arrays; +import java.util.Random; + +import static org.junit.Assert.*; + +public class ArbitraryEncryptionTests extends Common { + + @Before + public void beforeTest() throws DataException { + Common.useDefaultSettings(); + } + + @Test + public void testEncryption() throws IOException, NoSuchAlgorithmException, InvalidAlgorithmParameterException, NoSuchPaddingException, IllegalBlockSizeException, BadPaddingException, InvalidKeyException { + String enclosingFolderName = "data"; + Path inputFilePath = Files.createTempFile("inputFile", null); + Path outputDirectory = Files.createTempDirectory("outputDirectory"); + Path outputFilePath = Paths.get(outputDirectory.toString(), enclosingFolderName); + inputFilePath.toFile().deleteOnExit(); + outputDirectory.toFile().deleteOnExit(); + + // Write random data to the input file + byte[] data = new byte[10]; + new Random().nextBytes(data); + Files.write(inputFilePath, data, StandardOpenOption.CREATE); + + assertTrue(Files.exists(inputFilePath)); + assertFalse(Files.exists(outputFilePath)); + + // Encrypt... + String algorithm = "AES/CBC/PKCS5Padding"; + SecretKey aesKey = AES.generateKey(256); + AES.encryptFile(algorithm, aesKey, inputFilePath.toString(), outputFilePath.toString()); + + assertTrue(Files.exists(inputFilePath)); + assertTrue(Files.exists(outputFilePath)); + + // Ensure encrypted file's hash differs from the original + assertFalse(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(outputFilePath.toFile()))); + + // Create paths for decrypting + Path decryptedDirectory = Files.createTempDirectory("decryptedDirectory"); + Path decryptedFile = Paths.get(decryptedDirectory.toString(), enclosingFolderName, inputFilePath.getFileName().toString()); + decryptedDirectory.toFile().deleteOnExit(); + assertFalse(Files.exists(decryptedFile)); + + // Now decrypt... + AES.decryptFile(algorithm, aesKey, outputFilePath.toString(), decryptedFile.toString()); + + // Ensure resulting file exists + assertTrue(Files.exists(decryptedFile)); + + // And make sure it matches the original input file + assertTrue(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(decryptedFile.toFile()))); + } + + @Test + public void testEncryptionSizeOverhead() throws IOException, NoSuchAlgorithmException, InvalidAlgorithmParameterException, NoSuchPaddingException, IllegalBlockSizeException, BadPaddingException, InvalidKeyException { + for (int size = 1; size < 256; size++) { + String enclosingFolderName = "data"; + Path inputFilePath = Files.createTempFile("inputFile", null); + Path outputDirectory = Files.createTempDirectory("outputDirectory"); + Path outputFilePath = Paths.get(outputDirectory.toString(), enclosingFolderName); + inputFilePath.toFile().deleteOnExit(); + outputDirectory.toFile().deleteOnExit(); + + // Write random data to the input file + byte[] data = new byte[size]; + new Random().nextBytes(data); + Files.write(inputFilePath, data, StandardOpenOption.CREATE); + + assertTrue(Files.exists(inputFilePath)); + assertFalse(Files.exists(outputFilePath)); + + // Ensure input file is the same size as the data + assertEquals(size, inputFilePath.toFile().length()); + + // Encrypt... + String algorithm = "AES/CBC/PKCS5Padding"; + SecretKey aesKey = AES.generateKey(256); + AES.encryptFile(algorithm, aesKey, inputFilePath.toString(), outputFilePath.toString()); + + assertTrue(Files.exists(inputFilePath)); + assertTrue(Files.exists(outputFilePath)); + + final long expectedSize = AES.getEncryptedFileSize(inputFilePath.toFile().length()); + System.out.println(String.format("Plaintext size: %d bytes, Ciphertext size: %d bytes", inputFilePath.toFile().length(), outputFilePath.toFile().length())); + + // Ensure encryption added a fixed amount of space to the output file + assertEquals(expectedSize, outputFilePath.toFile().length()); + + // Ensure encrypted file's hash differs from the original + assertFalse(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(outputFilePath.toFile()))); + + // Create paths for decrypting + Path decryptedDirectory = Files.createTempDirectory("decryptedDirectory"); + Path decryptedFile = Paths.get(decryptedDirectory.toString(), enclosingFolderName, inputFilePath.getFileName().toString()); + decryptedDirectory.toFile().deleteOnExit(); + assertFalse(Files.exists(decryptedFile)); + + // Now decrypt... + AES.decryptFile(algorithm, aesKey, outputFilePath.toString(), decryptedFile.toString()); + + // Ensure resulting file exists + assertTrue(Files.exists(decryptedFile)); + + // And make sure it matches the original input file + assertTrue(Arrays.equals(Crypto.digest(inputFilePath.toFile()), Crypto.digest(decryptedFile.toFile()))); + } + } + +} diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryServiceTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryServiceTests.java index 96843876..33632b4a 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryServiceTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryServiceTests.java @@ -22,11 +22,14 @@ import org.qortal.test.common.transaction.TestTransaction; import org.qortal.transaction.RegisterNameTransaction; import org.qortal.utils.Base58; +import java.io.BufferedWriter; +import java.io.FileWriter; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardOpenOption; +import java.util.List; import java.util.Random; import static org.junit.Assert.*; @@ -317,17 +320,15 @@ public class ArbitraryServiceTests extends Common { // Write the data to several files in a temp path Path path = Files.createTempDirectory("testValidateMultiLayerQChatAttachment"); path.toFile().deleteOnExit(); - Files.write(Paths.get(path.toString(), "file1.txt"), data, StandardOpenOption.CREATE); Path subdirectory = Paths.get(path.toString(), "subdirectory"); Files.createDirectories(subdirectory); - Files.write(Paths.get(subdirectory.toString(), "file2.txt"), data, StandardOpenOption.CREATE); - Files.write(Paths.get(subdirectory.toString(), "file3.txt"), data, StandardOpenOption.CREATE); + Files.write(Paths.get(subdirectory.toString(), "file.txt"), data, StandardOpenOption.CREATE); Service service = Service.QCHAT_ATTACHMENT; assertTrue(service.isValidationRequired()); - assertEquals(ValidationResult.DIRECTORIES_NOT_ALLOWED, service.validate(path)); + assertEquals(ValidationResult.INVALID_FILE_COUNT, service.validate(path)); } @Test @@ -389,4 +390,134 @@ public class ArbitraryServiceTests extends Common { } } + @Test + public void testValidateValidJson() throws IOException { + String invalidJsonString = "{\"test\": true, \"test2\": \"valid\"}"; + + // Write the data a single file in a temp path + Path path = Files.createTempDirectory("testValidateValidJson"); + Path filePath = Paths.get(path.toString(), "test.json"); + filePath.toFile().deleteOnExit(); + + BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile())); + writer.write(invalidJsonString); + writer.close(); + + Service service = Service.JSON; + assertTrue(service.isValidationRequired()); + + assertEquals(ValidationResult.OK, service.validate(filePath)); + } + @Test + public void testValidateInvalidJson() throws IOException { + String invalidJsonString = "{\"test\": true, \"test2\": invalid}"; + + // Write the data a single file in a temp path + Path path = Files.createTempDirectory("testValidateInvalidJson"); + Path filePath = Paths.get(path.toString(), "test.json"); + filePath.toFile().deleteOnExit(); + + BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile())); + writer.write(invalidJsonString); + writer.close(); + + Service service = Service.JSON; + assertTrue(service.isValidationRequired()); + + assertEquals(ValidationResult.INVALID_CONTENT, service.validate(filePath)); + } + + @Test + public void testValidateEmptyJson() throws IOException { + Path path = Files.createTempDirectory("testValidateEmptyJson"); + + Service service = Service.JSON; + assertTrue(service.isValidationRequired()); + + assertEquals(ValidationResult.INVALID_FILE_COUNT, service.validate(path)); + } + + @Test + public void testValidPrivateData() throws IOException { + String dataString = "qortalEncryptedDatabMx4fELNTV+ifJxmv4+GcuOIJOTo+3qAvbWKNY2L1rfla5UBoEcoxbtjgZ9G7FLPb8V/Qfr0bfKWfvMmN06U/pgUdLuv2mGL2V0D3qYd1011MUzGdNG1qERjaCDz8GAi63+KnHHjfMtPgYt6bcqjs4CNV+ZZ4dIt3xxHYyVEBNc="; + + // Write the data a single file in a temp path + Path path = Files.createTempDirectory("testValidPrivateData"); + Path filePath = Paths.get(path.toString(), "test"); + filePath.toFile().deleteOnExit(); + + BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile())); + writer.write(dataString); + writer.close(); + + Service service = Service.FILE_PRIVATE; + assertTrue(service.isValidationRequired()); + + assertEquals(ValidationResult.OK, service.validate(filePath)); + } + + @Test + public void testEncryptedData() throws IOException { + String dataString = "qortalEncryptedDatabMx4fELNTV+ifJxmv4+GcuOIJOTo+3qAvbWKNY2L1rfla5UBoEcoxbtjgZ9G7FLPb8V/Qfr0bfKWfvMmN06U/pgUdLuv2mGL2V0D3qYd1011MUzGdNG1qERjaCDz8GAi63+KnHHjfMtPgYt6bcqjs4CNV+ZZ4dIt3xxHYyVEBNc="; + + // Write the data a single file in a temp path + Path path = Files.createTempDirectory("testValidPrivateData"); + Path filePath = Paths.get(path.toString(), "test"); + filePath.toFile().deleteOnExit(); + + BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile())); + writer.write(dataString); + writer.close(); + + // Validate a private service + Service service = Service.FILE_PRIVATE; + assertTrue(service.isValidationRequired()); + assertEquals(ValidationResult.OK, service.validate(filePath)); + + // Validate a regular service + service = Service.FILE; + assertTrue(service.isValidationRequired()); + assertEquals(ValidationResult.DATA_ENCRYPTED, service.validate(filePath)); + } + + @Test + public void testPlainTextData() throws IOException { + String dataString = "plaintext"; + + // Write the data a single file in a temp path + Path path = Files.createTempDirectory("testInvalidPrivateData"); + Path filePath = Paths.get(path.toString(), "test"); + filePath.toFile().deleteOnExit(); + + BufferedWriter writer = new BufferedWriter(new FileWriter(filePath.toFile())); + writer.write(dataString); + writer.close(); + + // Validate a private service + Service service = Service.FILE_PRIVATE; + assertTrue(service.isValidationRequired()); + assertEquals(ValidationResult.DATA_NOT_ENCRYPTED, service.validate(filePath)); + + // Validate a regular service + service = Service.FILE; + assertTrue(service.isValidationRequired()); + assertEquals(ValidationResult.OK, service.validate(filePath)); + } + + @Test + public void testGetPrivateServices() { + List privateServices = Service.privateServices(); + for (Service service : privateServices) { + assertTrue(service.isPrivate()); + } + } + + @Test + public void testGetPublicServices() { + List publicServices = Service.publicServices(); + for (Service service : publicServices) { + assertFalse(service.isPrivate()); + } + } + } \ No newline at end of file diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionMetadataTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionMetadataTests.java index bf4f0a70..47c68b25 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionMetadataTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionMetadataTests.java @@ -91,8 +91,8 @@ public class ArbitraryTransactionMetadataTests extends Common { String name = "TEST"; // Can be anything for this test String identifier = null; // Not used for this test Service service = Service.ARBITRARY_DATA; - int chunkSize = 1000; - int dataLength = 10; // Actual data length will be longer due to encryption + int chunkSize = 10000; + int dataLength = 1000; // Actual data length will be longer due to encryption String title = "Test title"; String description = "Test description"; @@ -118,6 +118,7 @@ public class ArbitraryTransactionMetadataTests extends Common { assertEquals(description, arbitraryDataFile.getMetadata().getDescription()); assertEquals(tags, arbitraryDataFile.getMetadata().getTags()); assertEquals(category, arbitraryDataFile.getMetadata().getCategory()); + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); // Now build the latest data state for this name ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier); @@ -141,8 +142,8 @@ public class ArbitraryTransactionMetadataTests extends Common { String name = "TEST"; // Can be anything for this test String identifier = null; // Not used for this test Service service = Service.ARBITRARY_DATA; - int chunkSize = 1000; - int dataLength = 10; // Actual data length will be longer due to encryption + int chunkSize = 10000; + int dataLength = 1000; // Actual data length will be longer due to encryption String title = "Test title"; String description = "Test description"; @@ -168,6 +169,7 @@ public class ArbitraryTransactionMetadataTests extends Common { assertEquals(description, arbitraryDataFile.getMetadata().getDescription()); assertEquals(tags, arbitraryDataFile.getMetadata().getTags()); assertEquals(category, arbitraryDataFile.getMetadata().getCategory()); + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); // Delete the file, to simulate that it hasn't been fetched from the network yet arbitraryDataFile.delete(); @@ -230,6 +232,7 @@ public class ArbitraryTransactionMetadataTests extends Common { assertEquals(description, arbitraryDataFile.getMetadata().getDescription()); assertEquals(tags, arbitraryDataFile.getMetadata().getTags()); assertEquals(category, arbitraryDataFile.getMetadata().getCategory()); + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); // Now build the latest data state for this name ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ResourceIdType.NAME, service, identifier); @@ -245,6 +248,47 @@ public class ArbitraryTransactionMetadataTests extends Common { } } + @Test + public void testUTF8Metadata() throws DataException, IOException, MissingDataException { + try (final Repository repository = RepositoryManager.getRepository()) { + PrivateKeyAccount alice = Common.getTestAccount(repository, "alice"); + String publicKey58 = Base58.encode(alice.getPublicKey()); + String name = "TEST"; // Can be anything for this test + String identifier = null; // Not used for this test + Service service = Service.ARBITRARY_DATA; + int chunkSize = 100; + int dataLength = 900; // Actual data length will be longer due to encryption + + // Example (modified) strings from real world content + String title = "Доля юаня в трансграничных Доля юаня в трансграничных"; + String description = "Когда рыночек порешал"; + List tags = Arrays.asList("Доля", "юаня", "трансграничных"); + Category category = Category.OTHER; + + // Register the name to Alice + RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, ""); + transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp())); + TransactionUtils.signAndMint(repository, transactionData, alice); + + // Create PUT transaction + Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength); + ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, + identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true, + title, description, tags, category); + + // Check the chunk count is correct + assertEquals(10, arbitraryDataFile.chunkCount()); + + // Check the metadata is correct + String expectedTrimmedTitle = "Доля юаня в трансграничных Доля юаня в тран"; + assertEquals(expectedTrimmedTitle, arbitraryDataFile.getMetadata().getTitle()); + assertEquals(description, arbitraryDataFile.getMetadata().getDescription()); + assertEquals(tags, arbitraryDataFile.getMetadata().getTags()); + assertEquals(category, arbitraryDataFile.getMetadata().getCategory()); + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); + } + } + @Test public void testMetadataLengths() throws DataException, IOException, MissingDataException { try (final Repository repository = RepositoryManager.getRepository()) { @@ -262,7 +306,7 @@ public class ArbitraryTransactionMetadataTests extends Common { Category category = Category.CRYPTOCURRENCY; String expectedTitle = "title Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat "; // 80 chars - String expectedDescription = "description Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat pretium massa, non pulvinar mi pretium id. Ut gravida sapien vitae dui posuere tincidunt. Quisque in nibh est. Curabitur at blandit nunc, id aliquet neque. Nulla condimentum eget dolor a egestas. Vestibulum vel tincidunt ex. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Cras congue lacus in risus mattis suscipit. Quisque nisl eros, facilisis a lorem quis, vehicula biben"; // 500 chars + String expectedDescription = "description Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent feugiat pretium massa, non pulvinar mi pretium id. Ut gravida sapien vitae dui posuere tincidunt. Quisque in nibh est. Curabitur at blandit nunc, id aliquet neque"; // 240 chars List expectedTags = Arrays.asList("tag 1", "tag 2", "tag 4", "tag 5", "tag 6"); // Register the name to Alice @@ -318,9 +362,11 @@ public class ArbitraryTransactionMetadataTests extends Common { assertTrue(resourceMetadata.getFiles().contains("file.txt")); // Ensure it's not returned when specified to be excluded - // The entire object will be null because there is no metadata ArbitraryResourceMetadata resourceMetadataSimple = ArbitraryResourceMetadata.fromTransactionMetadata(arbitraryDataFile.getMetadata(), false); - assertNull(resourceMetadataSimple); + assertNull(resourceMetadataSimple.getFiles()); + + // Single-file resources should have a MIME type + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); } } @@ -369,6 +415,9 @@ public class ArbitraryTransactionMetadataTests extends Common { // The entire object will be null because there is no metadata ArbitraryResourceMetadata resourceMetadataSimple = ArbitraryResourceMetadata.fromTransactionMetadata(arbitraryDataFile.getMetadata(), false); assertNull(resourceMetadataSimple); + + // Multi-file resources won't have a MIME type + assertEquals(null, arbitraryDataFile.getMetadata().getMimeType()); } } diff --git a/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionTests.java b/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionTests.java index 2c2d52b2..855aeafd 100644 --- a/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionTests.java +++ b/src/test/java/org/qortal/test/arbitrary/ArbitraryTransactionTests.java @@ -5,13 +5,19 @@ import org.junit.Before; import org.junit.Test; import org.qortal.account.PrivateKeyAccount; import org.qortal.arbitrary.ArbitraryDataFile; +import org.qortal.arbitrary.ArbitraryDataReader; import org.qortal.arbitrary.ArbitraryDataTransactionBuilder; import org.qortal.arbitrary.exception.MissingDataException; +import org.qortal.arbitrary.misc.Category; import org.qortal.arbitrary.misc.Service; import org.qortal.controller.arbitrary.ArbitraryDataManager; +import org.qortal.crypto.Crypto; +import org.qortal.data.PaymentData; import org.qortal.data.transaction.ArbitraryTransactionData; +import org.qortal.data.transaction.BaseTransactionData; import org.qortal.data.transaction.RegisterNameTransactionData; import org.qortal.data.transaction.TransactionData; +import org.qortal.group.Group; import org.qortal.repository.DataException; import org.qortal.repository.Repository; import org.qortal.repository.RepositoryManager; @@ -22,12 +28,18 @@ import org.qortal.test.common.transaction.TestTransaction; import org.qortal.transaction.ArbitraryTransaction; import org.qortal.transaction.RegisterNameTransaction; import org.qortal.transaction.Transaction; +import org.qortal.transform.transaction.TransactionTransformer; import org.qortal.utils.Base58; import org.qortal.utils.NTP; -import javax.xml.crypto.Data; +import java.io.File; import java.io.IOException; import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Random; import static org.junit.Assert.*; @@ -423,4 +435,213 @@ public class ArbitraryTransactionTests extends Common { assertTrue(transaction.isSignatureValid()); } } + + @Test + public void testInvalidService() { + byte[] randomHash = new byte[32]; + new Random().nextBytes(randomHash); + + byte[] lastReference = new byte[64]; + new Random().nextBytes(lastReference); + + Long now = NTP.getTime(); + + final BaseTransactionData baseTransactionData = new BaseTransactionData(now, Group.NO_GROUP, + lastReference, randomHash, 0L, null); + final String name = "test"; + final String identifier = "test"; + final ArbitraryTransactionData.Method method = ArbitraryTransactionData.Method.PUT; + final ArbitraryTransactionData.Compression compression = ArbitraryTransactionData.Compression.ZIP; + final int size = 999; + final int version = 5; + final int nonce = 0; + final byte[] secret = randomHash; + final ArbitraryTransactionData.DataType dataType = ArbitraryTransactionData.DataType.DATA_HASH; + final byte[] digest = randomHash; + final byte[] metadataHash = null; + final List payments = new ArrayList<>(); + final int validService = Service.IMAGE.value; + final int invalidService = 99999999; + + // Try with valid service + ArbitraryTransactionData transactionData = new ArbitraryTransactionData(baseTransactionData, + version, validService, nonce, size, name, identifier, method, + secret, compression, digest, dataType, metadataHash, payments); + assertEquals(Service.IMAGE, transactionData.getService()); + + // Try with invalid service + transactionData = new ArbitraryTransactionData(baseTransactionData, + version, invalidService, nonce, size, name, identifier, method, + secret, compression, digest, dataType, metadataHash, payments); + assertNull(transactionData.getService()); + } + + @Test + public void testOnChainData() throws DataException, IOException, MissingDataException, IllegalAccessException { + try (final Repository repository = RepositoryManager.getRepository()) { + // Set difficulty to 1 to speed up the tests + FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true); + + PrivateKeyAccount alice = Common.getTestAccount(repository, "alice"); + String publicKey58 = Base58.encode(alice.getPublicKey()); + String name = "TEST"; // Can be anything for this test + String identifier = null; // Not used for this test + Service service = Service.ARBITRARY_DATA; + int chunkSize = 1000; + int dataLength = 239; // Max possible size. Becomes 256 bytes after encryption. + + // Register the name to Alice + RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, ""); + transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp())); + TransactionUtils.signAndMint(repository, transactionData, alice); + + // Create PUT transaction + Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true); + ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, + identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true, + null, null, null, null); + + byte[] signature = arbitraryDataFile.getSignature(); + ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature); + + // Check that the data is published on chain + assertEquals(ArbitraryTransactionData.DataType.RAW_DATA, arbitraryTransactionData.getDataType()); + assertEquals(arbitraryDataFile.getBytes().length, arbitraryTransactionData.getData().length); + assertArrayEquals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData()); + + // Check that we have no chunks because the complete file is already less than the chunk size + assertEquals(0, arbitraryDataFile.chunkCount()); + + // Check that we have one file total - just the complete file (no chunks or metadata) + assertEquals(1, arbitraryDataFile.fileCount()); + + // Check the metadata isn't present + assertNull(arbitraryDataFile.getMetadata()); + + // Now build the latest data state for this name + ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); + arbitraryDataReader.loadSynchronously(true); + + // Filename will be "data" because it's been held as raw bytes in the transaction, + // so there is nowhere to store the original filename + File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "data").toFile(); + + assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile())); + } + } + + @Test + public void testOnChainDataWithMetadata() throws DataException, IOException, MissingDataException, IllegalAccessException { + try (final Repository repository = RepositoryManager.getRepository()) { + // Set difficulty to 1 to speed up the tests + FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true); + + PrivateKeyAccount alice = Common.getTestAccount(repository, "alice"); + String publicKey58 = Base58.encode(alice.getPublicKey()); + String name = "TEST"; // Can be anything for this test + String identifier = null; // Not used for this test + Service service = Service.ARBITRARY_DATA; + int chunkSize = 1000; + int dataLength = 239; // Max possible size. Becomes 256 bytes after encryption. + + String title = "Test title"; + String description = "Test description"; + List tags = Arrays.asList("Test", "tag", "another tag"); + Category category = Category.QORTAL; + + // Register the name to Alice + RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, ""); + transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp())); + TransactionUtils.signAndMint(repository, transactionData, alice); + + // Create PUT transaction + Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true); + ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, + identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true, + title, description, tags, category); + + byte[] signature = arbitraryDataFile.getSignature(); + ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature); + + // Check that the data is published on chain + assertEquals(ArbitraryTransactionData.DataType.RAW_DATA, arbitraryTransactionData.getDataType()); + assertEquals(arbitraryDataFile.getBytes().length, arbitraryTransactionData.getData().length); + assertArrayEquals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData()); + + // Check that we have no chunks because the complete file is already less than the chunk size + assertEquals(0, arbitraryDataFile.chunkCount()); + + // Check that we have two files total - one for the complete file, and the other for the metadata + assertEquals(2, arbitraryDataFile.fileCount()); + + // Check the metadata is correct + assertEquals(title, arbitraryDataFile.getMetadata().getTitle()); + assertEquals(description, arbitraryDataFile.getMetadata().getDescription()); + assertEquals(tags, arbitraryDataFile.getMetadata().getTags()); + assertEquals(category, arbitraryDataFile.getMetadata().getCategory()); + assertEquals("text/plain", arbitraryDataFile.getMetadata().getMimeType()); + + // Now build the latest data state for this name + ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); + arbitraryDataReader.loadSynchronously(true); + + // Filename will be "data" because it's been held as raw bytes in the transaction, + // so there is nowhere to store the original filename + File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "data").toFile(); + + assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile())); + } + } + + @Test + public void testOffChainData() throws DataException, IOException, MissingDataException, IllegalAccessException { + try (final Repository repository = RepositoryManager.getRepository()) { + // Set difficulty to 1 to speed up the tests + FieldUtils.writeField(ArbitraryDataManager.getInstance(), "powDifficulty", 1, true); + + PrivateKeyAccount alice = Common.getTestAccount(repository, "alice"); + String publicKey58 = Base58.encode(alice.getPublicKey()); + String name = "TEST"; // Can be anything for this test + String identifier = null; // Not used for this test + Service service = Service.ARBITRARY_DATA; + int chunkSize = 1000; + int dataLength = 240; // Min possible size. Becomes 257 bytes after encryption. + + // Register the name to Alice + RegisterNameTransactionData transactionData = new RegisterNameTransactionData(TestTransaction.generateBase(alice), name, ""); + transactionData.setFee(new RegisterNameTransaction(null, null).getUnitFee(transactionData.getTimestamp())); + TransactionUtils.signAndMint(repository, transactionData, alice); + + // Create PUT transaction + Path path1 = ArbitraryUtils.generateRandomDataPath(dataLength, true); + ArbitraryDataFile arbitraryDataFile = ArbitraryUtils.createAndMintTxn(repository, publicKey58, path1, name, + identifier, ArbitraryTransactionData.Method.PUT, service, alice, chunkSize, 0L, true, + null, null, null, null); + + byte[] signature = arbitraryDataFile.getSignature(); + ArbitraryTransactionData arbitraryTransactionData = (ArbitraryTransactionData) repository.getTransactionRepository().fromSignature(signature); + + // Check that the data is published on chain + assertEquals(ArbitraryTransactionData.DataType.DATA_HASH, arbitraryTransactionData.getDataType()); + assertEquals(TransactionTransformer.SHA256_LENGTH, arbitraryTransactionData.getData().length); + assertFalse(Arrays.equals(arbitraryDataFile.getBytes(), arbitraryTransactionData.getData())); + + // Check that we have no chunks because the complete file is already less than the chunk size + assertEquals(0, arbitraryDataFile.chunkCount()); + + // Check that we have one file total - just the complete file (no chunks or metadata) + assertEquals(1, arbitraryDataFile.fileCount()); + + // Check the metadata isn't present + assertNull(arbitraryDataFile.getMetadata()); + + // Now build the latest data state for this name + ArbitraryDataReader arbitraryDataReader = new ArbitraryDataReader(name, ArbitraryDataFile.ResourceIdType.NAME, service, identifier); + arbitraryDataReader.loadSynchronously(true); + + // File content should match original file + File outputFile = Paths.get(arbitraryDataReader.getFilePath().toString(), "file.txt").toFile(); + assertArrayEquals(Crypto.digest(outputFile), Crypto.digest(path1.toFile())); + } + } } diff --git a/src/test/java/org/qortal/test/common/ArbitraryUtils.java b/src/test/java/org/qortal/test/common/ArbitraryUtils.java index 73dc8097..1741d22c 100644 --- a/src/test/java/org/qortal/test/common/ArbitraryUtils.java +++ b/src/test/java/org/qortal/test/common/ArbitraryUtils.java @@ -51,12 +51,8 @@ public class ArbitraryUtils { BlockUtils.mintBlock(repository); // We need a new ArbitraryDataFile instance because the files will have been moved to the signature's folder - byte[] hash = txnBuilder.getArbitraryDataFile().getHash(); - byte[] signature = transactionData.getSignature(); - ArbitraryDataFile arbitraryDataFile = ArbitraryDataFile.fromHash(hash, signature); - arbitraryDataFile.setMetadataHash(transactionData.getMetadataHash()); - - return arbitraryDataFile; + // Or, it may now be using RAW_DATA instead of a hash + return ArbitraryDataFile.fromTransactionData(transactionData); } public static ArbitraryDataFile createAndMintTxn(Repository repository, String publicKey58, Path path, String name, String identifier, @@ -68,6 +64,17 @@ public class ArbitraryUtils { } public static Path generateRandomDataPath(int length) throws IOException { + return generateRandomDataPath(length, false); + } + + /** + * Generate random data, held in a single file within a directory + * @param length - size of file to create + * @param returnFilePath - if true, the file's path is returned. If false, the outer directory's path is returned. + * @return - path to file or directory, depending on the "returnFilePath" boolean + * @throws IOException + */ + public static Path generateRandomDataPath(int length, boolean returnFilePath) throws IOException { // Create a file in a random temp directory Path tempDir = Files.createTempDirectory("generateRandomDataPath"); File file = new File(Paths.get(tempDir.toString(), "file.txt").toString()); @@ -84,6 +91,10 @@ public class ArbitraryUtils { file1Writer.newLine(); file1Writer.close(); + if (returnFilePath) { + return file.toPath(); + } + return tempDir; } diff --git a/src/test/java/org/qortal/test/common/transaction/ArbitraryTestTransaction.java b/src/test/java/org/qortal/test/common/transaction/ArbitraryTestTransaction.java index d831eaf1..8688ed73 100644 --- a/src/test/java/org/qortal/test/common/transaction/ArbitraryTestTransaction.java +++ b/src/test/java/org/qortal/test/common/transaction/ArbitraryTestTransaction.java @@ -33,7 +33,7 @@ public class ArbitraryTestTransaction extends TestTransaction { final byte[] metadataHash = new byte[32]; random.nextBytes(metadataHash); - byte[] data = new byte[1024]; + byte[] data = new byte[256]; random.nextBytes(data); DataType dataType = DataType.RAW_DATA; @@ -45,7 +45,7 @@ public class ArbitraryTestTransaction extends TestTransaction { List payments = new ArrayList<>(); payments.add(new PaymentData(recipient, assetId, amount)); - return new ArbitraryTransactionData(generateBase(account), version, service, nonce, size,name, identifier, + return new ArbitraryTransactionData(generateBase(account), version, service.value, nonce, size,name, identifier, method, secret, compression, data, dataType, metadataHash, payments); } diff --git a/src/test/java/org/qortal/test/naming/IntegrityTests.java b/src/test/java/org/qortal/test/naming/IntegrityTests.java index d52d4983..767ea388 100644 --- a/src/test/java/org/qortal/test/naming/IntegrityTests.java +++ b/src/test/java/org/qortal/test/naming/IntegrityTests.java @@ -128,7 +128,7 @@ public class IntegrityTests extends Common { // Run the database integrity check for the initial name, to ensure it doesn't get into a loop NamesDatabaseIntegrityCheck integrityCheck = new NamesDatabaseIntegrityCheck(); - assertEquals(2, integrityCheck.rebuildName(initialName, repository)); + assertEquals(4, integrityCheck.rebuildName(initialName, repository)); // 4 transactions total // Ensure the new name still exists and the data is still correct assertTrue(repository.getNameRepository().nameExists(initialName)); diff --git a/testnet/README.md b/testnet/README.md index 2973f2e2..42e153d8 100644 --- a/testnet/README.md +++ b/testnet/README.md @@ -26,6 +26,7 @@ - Make sure to reference testnet blockchain config file: `"blockchainConfig": "testchain.json",` - It is a good idea to use a separate database: `"repositoryPath": "db-testnet",` - You might also need to add `"bitcoinNet": "TEST3",` and `"litecoinNet": "TEST3",` +- Also make sure to use a custom `listenPort` (not 62391 or 12391) to ensure that transactions remain isolated to your testnet. ## Other nodes @@ -59,7 +60,7 @@ Just add this setting: "singleNodeTestnet": true ``` This will automatically allow multiple consecutive blocks to be minted, as well as setting minBlockchainPeers to 0. -Remember to put these values back after introducing other nodes +Remember to put these values back after introducing other nodes. ## Fixed network @@ -115,14 +116,16 @@ Your options are: } ``` + ## Quick start Here are some steps to quickly get a single node testnet up and running with a generic minting account: 1. Start with template `settings-test.json`, and `testchain.json` which can be found in this folder. Copy/move them to the same directory as the jar. -2. Make sure feature triggers and other timestamp/height activations are correctly set. Generally these would be `0` so that they are enabled from the start. -3. Set a recent genesis `timestamp` in testchain.json, and add this reward share entry: +2. Set a custom `listenPort` in settings-test.json (not 62391 or 12391) to ensure that transactions remain isolated to your testnet. +3. Make sure feature triggers and other timestamp/height activations are correctly set. Generally these would be `0` so that they are enabled from the start. +4. Set a recent genesis `timestamp` in testchain.json, and add this reward share entry: `{ "type": "REWARD_SHARE", "minterPublicKey": "DwcUnhxjamqppgfXCLgbYRx8H9XFPUc2qYRy3CEvQWEw", "recipient": "QbTDMss7NtRxxQaSqBZtSLSNdSYgvGaqFf", "rewardSharePublicKey": "CRvQXxFfUMfr4q3o1PcUZPA4aPCiubBsXkk47GzRo754", "sharePercent": 0 },` -4. Start the node, passing in settings-test.json, e.g: `java -jar qortal.jar settings-test.json` -5. Once started, add the corresponding minting key to the node: +5. Start the node, passing in settings-test.json, e.g: `java -jar qortal.jar settings-test.json` +6. Once started, add the corresponding minting key to the node: `curl -X POST "http://localhost:62391/admin/mintingaccounts" -d "F48mYJycFgRdqtc58kiovwbcJgVukjzRE4qRRtRsK9ix"` -6. Alternatively you can use your own minting account instead of the generic one above. -7. After a short while, blocks should be minted from the genesis timestamp until the current time. \ No newline at end of file +7. Alternatively you can use your own minting account instead of the generic one above. +8. After a short while, blocks should be minted from the genesis timestamp until the current time. \ No newline at end of file diff --git a/tools/qdn b/tools/qdn index 869bf5c4..ea52e3c9 100755 --- a/tools/qdn +++ b/tools/qdn @@ -8,11 +8,11 @@ if [ -z "$*" ]; then echo "Usage:" echo echo "Host/update data:" - echo "qdata POST [service] [name] PATH [dirpath] " - echo "qdata POST [service] [name] STRING [data-string] " + echo "qdn POST [service] [name] PATH [dirpath] " + echo "qdn POST [service] [name] STRING [data-string] " echo echo "Fetch data:" - echo "qdata GET [service] [name] " + echo "qdn GET [service] [name] " echo echo "Notes:" echo "- When requesting a resource, please use 'default' to indicate a file with no identifier." diff --git a/tools/tx.pl b/tools/tx.pl index fe3cd872..1cb3dd5b 100755 --- a/tools/tx.pl +++ b/tools/tx.pl @@ -1,16 +1,23 @@ #!/usr/bin/env perl +# v4.0.2 + use JSON; use warnings; use strict; use Getopt::Std; use File::Basename; +use Digest::SHA qw( sha256 sha256_hex ); +use Crypt::RIPEMD160; our %opt; getopts('dpst', \%opt); my $proc = basename($0); +my $dirname = dirname($0); +my $OPENSSL_SIGN = "${dirname}/openssl-sign.sh"; +my $OPENSSL_PRIV_TO_PUB = index(`$ENV{SHELL} -i -c 'openssl version;exit' 2>/dev/null`, 'OpenSSL 3.') != -1; if (@ARGV < 1) { print STDERR "usage: $proc [-d] [-p] [-s] [-t] []\n"; @@ -24,7 +31,15 @@ if (@ARGV < 1) { exit 2; } -our $BASE_URL = $ENV{BASE_URL} || $opt{t} ? 'http://localhost:62391' : 'http://localhost:12391'; +our @b58 = qw{ + 1 2 3 4 5 6 7 8 9 + A B C D E F G H J K L M N P Q R S T U V W X Y Z + a b c d e f g h i j k m n o p q r s t u v w x y z +}; +our %b58 = map { $b58[$_] => $_ } 0 .. 57; +our %reverseb58 = reverse %b58; + +our $BASE_URL = $ENV{BASE_URL} || ($opt{t} ? 'http://localhost:62391' : 'http://localhost:12391'); our $DEFAULT_FEE = 0.001; our %TRANSACTION_TYPES = ( @@ -42,6 +57,7 @@ our %TRANSACTION_TYPES = ( create_group => { url => 'groups/create', required => [qw(groupName description isOpen approvalThreshold)], + defaults => { minimumBlockDelay => 10, maximumBlockDelay => 30 }, key_name => 'creatorPublicKey', }, update_group => { @@ -75,10 +91,10 @@ our %TRANSACTION_TYPES = ( key_name => 'ownerPublicKey', }, remove_group_admin => { - url => 'groups/removeadmin', - required => [qw(groupId txGroupId admin)], - key_name => 'ownerPublicKey', - }, + url => 'groups/removeadmin', + required => [qw(groupId txGroupId member)], + key_name => 'ownerPublicKey', + }, group_approval => { url => 'groups/approval', required => [qw(pendingSignature approval)], @@ -113,7 +129,7 @@ our %TRANSACTION_TYPES = ( }, update_name => { url => 'names/update', - required => [qw(newName newData)], + required => [qw(name newName newData)], key_name => 'ownerPublicKey', }, # reward-shares @@ -144,13 +160,21 @@ our %TRANSACTION_TYPES = ( key_name => 'senderPublicKey', pow_url => 'addresses/publicize/compute', }, - # Cross-chain trading - build_trade => { - url => 'crosschain/build', - required => [qw(initialQortAmount finalQortAmount fundingQortAmount secretHash bitcoinAmount)], - optional => [qw(tradeTimeout)], + # AT + deploy_at => { + url => 'at', + required => [qw(name description aTType tags creationBytes amount)], + optional => [qw(assetId)], key_name => 'creatorPublicKey', - defaults => { tradeTimeout => 10800 }, + defaults => { assetId => 0 }, + }, + # Cross-chain trading + create_trade => { + url => 'crosschain/tradebot/create', + required => [qw(qortAmount fundingQortAmount foreignAmount receivingAddress)], + optional => [qw(tradeTimeout foreignBlockchain)], + key_name => 'creatorPublicKey', + defaults => { tradeTimeout => 1440, foreignBlockchain => 'LITECOIN' }, }, trade_recipient => { url => 'crosschain/tradeoffer/recipient', @@ -196,7 +220,7 @@ if (@ARGV < @required + 1) { my $priv_key = shift @ARGV; -my $account = account($priv_key); +my $account; my $raw; if ($tx_type ne 'sign') { @@ -215,6 +239,8 @@ if ($tx_type ne 'sign') { %extras = (%extras, @ARGV); + $account = account($priv_key, %extras); + $raw = build_raw($tx_type, $account, %extras); printf "Raw: %s\n", $raw if $opt{d} || (!$opt{s} && !$opt{p}); @@ -229,7 +255,7 @@ if ($tx_type ne 'sign') { } if ($opt{s}) { - my $signed = sign($account->{private}, $raw); + my $signed = sign($priv_key, $raw); printf "Signed: %s\n", $signed if $opt{d} || $tx_type eq 'sign'; if ($opt{p}) { @@ -246,15 +272,25 @@ if ($opt{s}) { } sub account { - my ($creator) = @_; + my ($privkey, %extras) = @_; - my $account = { private => $creator }; - $account->{public} = api('utils/publickey', $creator); - $account->{address} = api('addresses/convert/{publickey}', '', '{publickey}', $account->{public}); + my $account = { private => $privkey }; + $account->{public} = $extras{publickey} || priv_to_pub($privkey); + $account->{address} = $extras{address} || pubkey_to_address($account->{public}); # api('addresses/convert/{publickey}', '', '{publickey}', $account->{public}); return $account; } +sub priv_to_pub { + my ($privkey) = @_; + + if ($OPENSSL_PRIV_TO_PUB) { + return openssl_priv_to_pub($privkey); + } else { + return api('utils/publickey', $privkey); + } +} + sub build_raw { my ($type, $account, %extras) = @_; @@ -306,6 +342,21 @@ sub build_raw { sub sign { my ($private, $raw) = @_; + if (-x "$OPENSSL_SIGN") { + my $private_hex = decode_base58($private); + chomp $private_hex; + + my $raw_hex = decode_base58($raw); + chomp $raw_hex; + + my $sig = `${OPENSSL_SIGN} ${private_hex} ${raw_hex}`; + chomp $sig; + + my $sig58 = encode_base58(${raw_hex} . ${sig}); + chomp $sig58; + return $sig58; + } + my $json = <<" __JSON__"; { "privateKey": "$private", @@ -344,7 +395,14 @@ sub api { my $curl = "curl --silent --output - --url '$BASE_URL/$url'"; if (defined $postdata && $postdata ne '') { $postdata =~ tr|\n| |s; - $curl .= " --header 'Content-Type: application/json' --data-binary '$postdata'"; + + if ($postdata =~ /^\s*\{/so) { + $curl .= " --header 'Content-Type: application/json'"; + } else { + $curl .= " --header 'Content-Type: text/plain'"; + } + + $curl .= " --data-binary '$postdata'"; $method = 'POST'; } my $response = `$curl 2>/dev/null`; @@ -356,3 +414,87 @@ sub api { return $response; } + +sub encode_base58 { + use integer; + my @in = map { hex($_) } ($_[0] =~ /(..)/g); + my $bzeros = length($1) if join('', @in) =~ /^(0*)/; + my @out; + my $size = 2 * scalar @in; + for my $c (@in) { + for (my $j = $size; $j--; ) { + $c += 256 * ($out[$j] // 0); + $out[$j] = $c % 58; + $c /= 58; + } + } + my $out = join('', map { $reverseb58{$_} } @out); + return $1 if $out =~ /(1{$bzeros}[^1].*)/; + return $1 if $out =~ /(1{$bzeros})/; + die "Invalid base58!\n"; +} + + +sub decode_base58 { + use integer; + my @out; + my $azeros = length($1) if $_[0] =~ /^(1*)/; + for my $c ( map { $b58{$_} } $_[0] =~ /./g ) { + die("Invalid character!\n") unless defined $c; + for (my $j = length($_[0]); $j--; ) { + $c += 58 * ($out[$j] // 0); + $out[$j] = $c % 256; + $c /= 256; + } + } + shift @out while @out && $out[0] == 0; + unshift(@out, (0) x $azeros); + return sprintf('%02x' x @out, @out); +} + +sub openssl_priv_to_pub { + my ($privkey) = @_; + + my $privkey_hex = decode_base58($privkey); + + my $key_type = "04"; # hex + my $length = "20"; # hex + + my $asn1 = <<"__ASN1__"; +asn1=SEQUENCE:private_key + +[private_key] +version=INTEGER:0 +included=SEQUENCE:key_info +raw=FORMAT:HEX,OCTETSTRING:${key_type}${length}${privkey_hex} + +[key_info] +type=OBJECT:ED25519 + +__ASN1__ + + my $output = `echo "${asn1}" | openssl asn1parse -i -genconf - -out - | openssl pkey -in - -inform der -noout -text_pub`; + + # remove colons + my $pubkey = ''; + $pubkey .= $1 while $output =~ m/([0-9a-f]{2})(?::|$)/g; + + return encode_base58($pubkey); +} + +sub pubkey_to_address { + my ($pubkey) = @_; + + my $pubkey_hex = decode_base58($pubkey); + my $pubkey_raw = pack('H*', $pubkey_hex); + + my $pkh_hex = Crypt::RIPEMD160->hexhash(sha256($pubkey_raw)); + $pkh_hex =~ tr/ //ds; + + my $version = '3a'; # hex + + my $raw = pack('H*', $version . $pkh_hex); + my $chksum = substr(sha256_hex(sha256($raw)), 0, 8); + + return encode_base58($version . $pkh_hex . $chksum); +}