Commit Graph

1751 Commits

Author SHA1 Message Date
CalDescent
b771544c5d Added test to check website/data updates. 2021-11-02 09:09:54 +00:00
CalDescent
8c7f09c454 Fixed yet another case sensitivity issue. 2021-11-01 20:02:38 +00:00
CalDescent
618cffefb1 Made names case insensitive when using them as a search filter. 2021-11-01 19:44:43 +00:00
CalDescent
8fd37e857e Fixed case sensitivity issue when building from past transactions. 2021-11-01 19:41:50 +00:00
CalDescent
8218bfd24b Initial implementation of storage policies.
- Don't attempt to fetch data for transactions which fall outside of the storage policy
- Delete files relating to transactions that are no longer within the scope of the storage policy

Note: some additional work needs to be done to ensure that viewed files are deleted when using a storage policy that excludes "VIEWED" content.
2021-10-31 22:38:30 +00:00
CalDescent
cbb2dbffb9 The /arbitrary/search API endpoint now uses a string instead of an int for the "service", and shows a dropdown of possible values in the API documentation page. 2021-10-31 21:22:51 +00:00
CalDescent
528a838643 Fixed bug in previous commit 2021-10-31 21:20:59 +00:00
CalDescent
cbed6418e7 Added ability to filter arbitrary transactions by name when searching. 2021-10-31 21:07:14 +00:00
CalDescent
4882cc92a8 StoragePolicy enum moved to new ArbitraryDataStorageManager class 2021-10-31 20:32:32 +00:00
CalDescent
28fb11068e Added "storagePolicy" setting, including startup validation.
Possible values are:

FOLLOWED_AND_VIEWED (default)
FOLLOWED
VIEWED
ALL
NONE

Nothing is affected by this setting yet, but this will be added shortly.
2021-10-31 19:15:26 +00:00
CalDescent
90465149e6 Added API endpoints to allow registered names to be followed/unfollowed.
These will ultimately be used by the UI to control which data the node should host.
2021-10-31 18:42:18 +00:00
CalDescent
c6d868d981 "Lists" feature is now generic.
This means that no additional structural code is required to add new lists. The only non-generic aspect are the API endpoints - it's best to keep these specific until we have a need for user-created lists.
2021-10-31 18:39:58 +00:00
CalDescent
bada4fd140 Use a buffered digest where possible instead of reading the whole file contents into memory. 2021-10-30 18:21:06 +01:00
CalDescent
60f96d15bd When specifying a domain without a subdomain, add a www. version automatically.
Longer term we will probably need a 301 redirect in these cases for SEO purposes, but this is a nice convenience feature for now.
2021-10-30 17:47:57 +01:00
CalDescent
0328007345 Domain mapping now uses registered name instead of transaction signature.
There's no real need to maintain support for signature mapping anymore. Using this new method means that the latest version of a site is always served via the traditional domain name, whereas using transaction signatures caused older versions to be shown.

Example settings.json configuration:

"domainMapServiceEnabled": true,
"domainMapServicePort": 80,
"domainMap": [
  {
    "domain": "webdemo.qortal.uk",
    "name": "QortalDemo"
  },
 {
    "domain": "www.reqorder.org",
    "name": "ReQorder"
  }
]
2021-10-30 17:29:23 +01:00
CalDescent
3ad0e92a0f Log empty responses in qdata 2021-10-29 18:58:59 +01:00
CalDescent
3934120541 Return the data directly in GET /arbitrary/{service}/{name}, instead of a path to the data 2021-10-29 18:52:05 +01:00
CalDescent
24ca126f5a Various qdata improvements 2021-10-29 17:58:47 +01:00
CalDescent
651ca71126 Added transaction validity log. 2021-10-29 17:50:03 +01:00
CalDescent
e7cb33d8e2 Synchronize peer data lookups.
Without this we could broadcast the same data multiple times, due to more than one thread processing the same message from different peers.
2021-10-29 17:46:58 +01:00
CalDescent
c63d238316 Log the error response in the qdata utility if broadcasting a transaction fails. 2021-10-29 17:26:40 +01:00
CalDescent
dcdc48d917 Private key moved from command line argument to QORTAL_PRIVKEY env variable, to improve security. 2021-10-29 17:24:22 +01:00
CalDescent
f4c1671079 Removed the need to include public key in recently added API endpoints, as it can be derived from the registered name's owner. 2021-10-29 17:08:26 +01:00
CalDescent
7aa2fbee1c Added "qdata" command line tool to host and retrieve data 2021-10-29 16:51:41 +01:00
CalDescent
f1939fdc2b Added generic PUT, PATCH, and GET /arbitrary API endpoints
These will likely avoid the need for some of the /site APIs.
2021-10-29 16:45:08 +01:00
CalDescent
c9356d0ff5 Re-broadcast the arbitrary signatures message if it contains new data, so that the message finds its way to all online peers. 2021-10-29 16:16:58 +01:00
CalDescent
6b5d938a40 Added saveChanges() missing from previous commit, and a discardChanges() just in case. 2021-10-29 16:16:05 +01:00
CalDescent
d82da160f3 Added DHT-style lookup table to track file locations
This maps ARBITRARY transactions to peer addresses, but also includes additional metadata/stats to track the success rate and reachability.

Once a node receives files for a transaction, it broadcasts this info to its peers so they can update their records.

TLDR: this allows us to locate peers that are hosting a copy of the file we need.
2021-10-29 13:35:17 +01:00
CalDescent
54c8aac20d Commented out unused forwarding code 2021-10-29 13:27:37 +01:00
CalDescent
05be5c1199 Version set to 2.0.0, given that this will ultimately be merged into the main qortal core. It makes sense to keep the version numbers in sync. 2021-10-25 19:06:48 +01:00
CalDescent
f19a65148a Revert "Default minPeerVersion set to 0.1.0. TODO: revert this if ever merged into the main repo."
This reverts commit 9b4d832d17.
2021-10-25 19:04:31 +01:00
CalDescent
a55fc4fff9 When validating an ARBITRARY transaction, ensure that the supplied name exists and is registered to the account that is signing the transaction.
This ensures that only the owner of a name is able to update data associated with that name.

Note that this doesn't take into account the ability for group members to update a resource, so this will need modifying when that feature is ultimately introduced (likely after v3.0)
2021-10-25 18:58:33 +01:00
CalDescent
35a7a70b93 Merge remote-tracking branch 'qortal/master' 2021-10-25 18:26:06 +01:00
CalDescent
3e0574e563 Added another missing block archive lookup relating to trade timestamps.
Note that this is unlikely to be the cause of some of the zero timestamps issue seen on a subset of nodes - there is still likely to be another problem that needs fixing.
2021-10-25 18:21:40 +01:00
CalDescent
69e557e70d Delete .sha256 file which was left lying around after running the bootstrap unit tests. 2021-10-25 18:20:58 +01:00
CalDescent
1b846be5fc Fixed bug in arbitrary transaction builder 2021-10-24 22:43:58 +01:00
CalDescent
707eb58068 Added testPatchBeforePut() unit test 2021-10-24 22:43:28 +01:00
CalDescent
8630f3be96 Added first end-to-end test of data storage 2021-10-24 19:08:09 +01:00
CalDescent
c90aeba286 Return ArbitraryTransactionData instead of a byte array, as it is more useful if the bytes are transformed separately. 2021-10-24 19:06:30 +01:00
CalDescent
5055cfc6cb Removed unused imports 2021-10-24 17:43:28 +01:00
CalDescent
c222c4eb29 Updated expected hash of demo data as it has been updated. 2021-10-24 17:41:51 +01:00
CalDescent
6c01955561 Refactor to move arbitrary transaction building to its own class. 2021-10-24 17:41:28 +01:00
CalDescent
305e0f1772 Disable validation of previous hash unless validateAllDataLayers is true
We may not need to validate this at all now that we have the ability to validate the current layer, but I'll leave it as it could be useful for debugging. It is disabled by default so not an issue.
2021-10-24 15:35:51 +01:00
CalDescent
52a94e3256 Added "validateAllDataLayers" setting (default false)
When true, the hashes of every layer are validated when building a data resource. When false, only the final layer's hash is validated.
2021-10-24 14:37:29 +01:00
CalDescent
a418fb18b6 Hash the current state when creating a patch
This is included in the patch metadata and then validated every time it is rebuilt.
2021-10-24 13:00:21 +01:00
CalDescent
9cd579d3db Another typo 2021-10-24 12:20:49 +01:00
CalDescent
e1a6ba7377 Fixed incorrect comment. 2021-10-24 12:03:22 +01:00
CalDescent
04aabe0921 Include the original file instead of a patch if the patch is larger than the original file.
This saves processing and disk space, as there is no point in applying a patch when the original file is smaller and can be included in its entirety.
2021-10-24 12:02:09 +01:00
CalDescent
8dd4d71d75 Significant rework of patches
- The "diff type" is now specified per file, allowing for different diff methods in each modified file.
- Patches will only be created when both the before and after files are less than 100kiB in size.
- Patches are validated after creation, and if invalid it will fall back to including the entire file.

This has identified a bug where patching fails for files without trailing newline characters, which still needs to be fixed. Until then, it will fall back to including the entire file in these cases.
2021-10-24 10:47:47 +01:00
49dd63af1e
updated BTC electrum servers 2021-10-23 00:46:02 -04:00