Commit Graph

1303 Commits

Author SHA1 Message Date
CalDescent
944e396823 Added AES utility class from baeldung and updated copyright notice for ZipUtils which was based on code from the same author. This code still needs reviewing and modifying but it's a good starting point for AES encryption and decryption. 2021-07-15 19:50:42 +01:00
CalDescent
8a654834ac Small reorganization. 2021-07-15 19:47:53 +01:00
CalDescent
bb76fa80cd Another significant upgrade of arbitrary transactions
Adds "name", "method", "secret", and "compression" properties. These are the foundations needed in order to handle updates, encryption, and name registration. Compression has been added so that we have the option of switching to different algorithms whilst maintaining support for existing transactions.
2021-07-15 09:27:49 +01:00
CalDescent
53f44a4029 Added support for subdirectories in the HTML parser. 2021-07-14 18:03:51 +01:00
CalDescent
182dcc7e5f MAX_FILE_SIZE reduced to 500MiB to match the difficulty calculation. 2021-07-14 18:03:35 +01:00
CalDescent
2d272e0207 Added some service constants.
These combine some Qora services (SERVICE_NAME_STORAGE, SERVICE_BLOG_POST, and SERVICE_BLOG_COMMENT) with existing Qortal services (SERVICE_AUTO_UPDATE), and some new additions (SERVICE_ARBITRARY_DATA, SERVICE_WEBSITE, and SERVICE_GIT_REPOSITORY)
2021-07-14 17:50:55 +01:00
CalDescent
9384a50879 Derive PoW difficulty from the file size. Exact values TBC. 2021-07-13 22:18:21 +01:00
CalDescent
00d4f35f2c Track the request time in arbitraryDataFileRequests and automatically remove those that have timed out. 2021-07-11 10:28:14 +01:00
CalDescent
483557163e Refactor: moved arbitrary data code from Controller to ArbitraryDataManager 2021-07-11 10:17:38 +01:00
CalDescent
2679252b04 Rework of arbitrary data requests
Previously we would ask all connected peers for the file itself, but this caused the network to be swamped when multiple peers responded with the same file.

This new approach instead asks all connected peers to send back a list of hashes for all files they have relating to a transaction signature. The requesting node then uses these lists to make separate requests for each missing file.
2021-07-11 10:00:09 +01:00
CalDescent
5a95c827b4 When serving a website, delete the unzipped directory if the index file is not found.
This is a quick solution to rebuild directory structures with missing files. This whole area of the code needs some reworking, as serving the site from a temporary folder is not a robust long term solution.
2021-07-10 15:51:46 +01:00
CalDescent
f938d8c878 More refactoring 2021-07-10 14:47:51 +01:00
CalDescent
bfc0122c1b Added warning to README 2021-07-10 13:53:07 +01:00
CalDescent
cdc5348a06 Added "domain map" server
Domain names can be mapped to arbitrary transaction signatures via the node's settings, and then served over port 80 or 443. This allows Qortal hosted sites to be accessible via a traditional domain name.

Example configuration to map two domains:

"domainMapServiceEnabled": true,
"domainMapServicePort": 80,
"domainMap": [
  {
    "domain": "example.com",
    "signature": "tEsw4kUn4ZJfPha7CotUL6BHkFPs79BwKXdY6yrf28YTpDn4KSY6ZKX3nwZCkqDF9RyXbgaVnB1rTEExY3h9CQA"
  },
  {
    "domain": "demo.qortal.org",
    "signature": "ZdBWWPMhR7AZwSu5xZm89mQEacekqkNfAimSCqFP6rQGKaGnXR9G4SWYpY5awFGfhmNBWzvRnXkWZKCsj6EMgc8"
  }
]

Each domain needs to be pointed to the Qortal data node via an A record or CNAME. You can add redundant nodes by adding multiple A records for the same domain (this is known as DNS Failover).

Note that running a webserver on port 80 (or anything less than 1024) requires running the data node as root. There are workarounds to this, such as disabling privileged ports, or using a reverse proxy. I will investigate this more as time goes on, but this is okay for a proof of concept.
2021-07-06 18:50:40 +01:00
CalDescent
e64a3978e6 Moved HTML parsing to new class. 2021-07-06 08:23:28 +01:00
CalDescent
f2feb12708 /site API endpoint now tales a signature rather than a file hash
This allows it to verify that the data in a transaction, after which it will then build the complete file from its chunks if needed.
2021-07-05 09:07:06 +01:00
CalDescent
5319c5f832 Reworked existing unused ArbitraryDataManager.
It's now capable of syncing chunks as well as complete files. This isn't production ready as it currently requests/receives the same file from multiple peers at once, which slows down the sync and wastes lots of bandwidth. Ideally we would find an appropriate peer first and then sync the file from them.
2021-07-05 09:05:45 +01:00
CalDescent
7531fe14fe Fixed major performance issue in DataFile.toString() 2021-07-05 08:23:29 +01:00
CalDescent
0086c6373b Significant refactor of DataFile and DataFileChunk
This introduces the hash58 property, which stores the base58 hash of the file passed in at initialization. It leaves digest() and digest58() for when we need to compute a new hash from the file itself.
2021-07-05 07:26:20 +01:00
CalDescent
10dc19652e Use "qortaldata-" version prefix 2021-07-04 16:52:20 +01:00
CalDescent
2f2c4964c5 Transaction version temporarily bumped to 5. TODO: We will need to set a hard fork timestamp if this is ever merged back into the main Qortal core repo. 2021-07-04 16:43:12 +01:00
CalDescent
cb4203b6db Use public key as parameter instead of address, since we can obtain the address from the public key in all cases. 2021-07-04 14:53:54 +01:00
CalDescent
bb5b62466e Fixed bug introduced in recent commit. 2021-07-04 13:41:37 +01:00
CalDescent
6407b5452b Delete our copies of data if any exception is thrown. 2021-07-04 13:39:00 +01:00
CalDescent
a742fecf9c API refactors to avoid generic unhandled states. 2021-07-04 13:38:20 +01:00
CalDescent
60415b9222 /arbitrary/upload/creator/{address} now returns an unsigned ARBITRARY transaction, currently with pre-computed nonce (same as the /site equivalent) 2021-07-04 10:25:15 +01:00
CalDescent
ffb39ef074 /data API endpoints moved to /arbitrary 2021-07-04 09:49:01 +01:00
CalDescent
d73f5ed2b5 /site/upload/creator/{address} now returns an unsigned ARBITRARY transaction, currently with pre-computed nonce 2021-07-04 09:45:52 +01:00
CalDescent
7af973b60d Removed TODO comments that are now done. 2021-07-04 09:41:53 +01:00
CalDescent
49eddc9da5 Allow zero fee transactions if the fee is zero in blockchain.json
Until now it wasn't possible to set up a chain with zero transaction fees due to a hardcoded zero check in Payment.isValid(), and a divide by zero error in Transaction.hasMinimumFeePerByte()
2021-07-04 09:31:51 +01:00
CalDescent
4b1de108d1 Fixed bug in expected chunk count. 2021-07-03 18:42:42 +01:00
CalDescent
e46c735efa Fixed recently introduced bugs with file management. 2021-07-03 18:05:17 +01:00
CalDescent
56da7deb4c DataFile updates to simplify integration with arbitrary transactions. 2021-07-03 17:41:52 +01:00
CalDescent
5f4649ee2b Major upgrade of arbitrary transactions
- Adds support for files up 500MiB per transaction (at 2MiB chunk sizes). Previously, the max data size was 4000 bytes.
- Adds a nonce, giving us the option to remove the transaction fees altogether on the data chain.

These features become enabled in version 5 of arbitrary transactions.
2021-07-03 17:40:02 +01:00
CalDescent
7cc2c4f621 Progress on website API endpoints. 2021-07-02 08:57:55 +01:00
CalDescent
cc449f9304 Added DataFile.chunkHashes() method which appends all hashes into a single byte array (32 bytes / 256 bits each). 2021-07-02 08:54:05 +01:00
CalDescent
28425efbe7 Added jsoup dependency - this was missing from an earlier commit. 2021-07-02 08:52:38 +01:00
CalDescent
8c3a22aa5c Improved link replacement criteria. 2021-07-01 11:21:40 +01:00
CalDescent
f3e5933599 Fixed naming error in joinFiles API. 2021-07-01 11:18:46 +01:00
CalDescent
aac4fe37e8 Fixed API response description. 2021-06-26 10:28:22 +01:00
CalDescent
ebfa941a4f Fixed some file separators. 2021-06-26 10:27:32 +01:00
CalDescent
47c70eea9e Use system temp directory instead of making a "temp" subfolder when zipping files. 2021-06-26 10:11:34 +01:00
CalDescent
fe7c40cb7c Reduced log spam. 2021-06-25 19:31:28 +01:00
CalDescent
8973626a4b Fixed issue with temp directory on Linux. 2021-06-25 19:31:13 +01:00
CalDescent
ace5d999e2 Log a comma separated list of hashes after splitting a file into chunks, so they can easily be requested from another node using the //data/files/frompeer/{peer} API endpoint. Again temporary until the sync happens automatically. 2021-06-25 19:30:45 +01:00
CalDescent
52829a244b More work on APIs to request files from peers. I won't spend too long making these perfect as they are mostly a temporary measure. 2021-06-25 19:28:39 +01:00
CalDescent
71c247fe56 Added POST /data/file/{hash}/build API, used to join multiple chunks together. 2021-06-25 19:28:01 +01:00
CalDescent
b34066f579 More work on HTML parsing.
The style tag parsing ideally needs rewriting using an actual CSS parser, but we can get away with this hacky approach in the short term.
2021-06-25 08:34:44 +01:00
CalDescent
b286c15c51 Optimized website serving, and added code to return the correct content types.
This is probably the most efficient way to process the data on the fly, but it's still not very scalable. A better approach would be to pre-process the HTML when building the file structure, and then serve them completely statically (i.e. using a standard webserver rather than via application memory). But it makes sense to keep it this way for development and maybe early beta testing.
2021-06-25 08:32:22 +01:00
CalDescent
ea5e2f5580 Added POST /site/preview API
This can be used to preview a site before signing a transaction and announcing it to the network. The response will need reworking to return JSON (along with most of the other new APIs)
2021-06-23 09:28:38 +01:00