crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
2,171 Commits 85 Branches 118 Tags 530 MiB
6c9600cda0
Commit Graph

1930 Commits

Author SHA1 Message Date
CalDescent
6423d5e474 Optimized onNetworkGetArbitraryDataFileListMessage() to remove duplicate calls to Files.exists() 2022-01-21 14:12:59 +00:00
CalDescent
6e91157dcf Improved cache clearing process and logging. 2022-01-21 13:32:59 +00:00
CalDescent
85c61c1bc1 Added GET /arbitrary/resources/search API 
Example usage

List all websites with a name containing the word "crow" (using default identifier):
http://localhost:12391/arbitrary/resources/search?service=WEBSITE&query=crow&default=true&limit=20

List all resources with name or identifier containing the word "crow":
http://localhost:12391/arbitrary/resources/search?query=crow&default=false&limit=20
 2022-01-21 10:04:18 +00:00
CalDescent
54af36fb85 Remove duplicates in GET /arbitrary/hosted/resources response 2022-01-20 22:34:41 +00:00
CalDescent
fcdcc939e6 Sort hosted data in reverse order (newest first) 2022-01-20 22:29:19 +00:00
CalDescent
13450d5afa Added limit/offset to GET /arbitrary/hosted API endpoints 2022-01-20 22:28:28 +00:00
CalDescent
5e1e653095 Removed unnecessary database lookups in GET /hosted/resources API 2022-01-20 20:39:46 +00:00
CalDescent
e8fabcb449 Removed extra isDataLocal() check in GET /hosted/resources which was evading the cache. 2022-01-20 20:38:57 +00:00
CalDescent
1b42062d57 Default minPeerVersion set to 3.0.1 2022-01-20 20:25:20 +00:00
CalDescent
c2a4b01a9c Allow local playback of media files 2022-01-20 20:25:06 +00:00
CalDescent
47e763b0cf Take a copy of the IP address history so it can be safely iterated. Without this, another thread could remove an element, resulting in an exception. 2022-01-20 20:24:51 +00:00
CalDescent
0278f6c9f2 Reduced more log spam 2022-01-20 20:24:26 +00:00
CalDescent
d96bc14516 Allow execution of inline scripts, at least for now. 2022-01-17 20:25:25 +00:00
CalDescent
318f433f22 Reduced log spam when checking for avatars. 2022-01-17 20:04:54 +00:00
CalDescent
cfc80cb9b0 Use a header instead of a meta tag for Content-Security-Policy, because we can't guarantee that we are parsing all HTML files. 
Also use default-src instead of connect-src, as we want to block all external requests.
 2022-01-17 20:04:35 +00:00
CalDescent
01c6149422 Restrict websites to same origin requests only, using a Content-Security-Policy meta tag. 2022-01-16 20:52:30 +00:00
CalDescent
6f80a6c08a Rework of file list requests and relays, allowing it to handle multiple chunk resources in a much more sensible way. 
This could create a lot of additional relay traffic as a result, so needs lots of testing and possibly optimizing.
 2022-01-16 20:39:37 +00:00
CalDescent
5018d27c25 "Not started" renamed to "Published" 2022-01-15 20:21:52 +00:00
CalDescent
1d77101253 Use AES/CBC/PKCS5Padding for encryption, and fall back to just AES for legacy resource support. 
Should fix "ECB mode cannot use IV" error due to mode and padding not being stated.
 2022-01-15 20:14:32 +00:00
CalDescent
119c1b43be Use default values for method and compression if not specified. 
Should fix issue with v4 transactions where these aren't used. Matches with the NOT NULL DEFAULT 0 which automatically transitions existing v4 ARBITRARY transactions to use the same defaults.
 2022-01-13 19:09:00 +00:00
CalDescent
6761b91400 QDN go-live timestamp set to Fri, 14 Jan 2022 16:00:00 UTC 2022-01-12 20:53:57 +00:00
CalDescent
e2b13791bb Don't reload the log4j2.properties file as this seems to be buggy in a lot of cases. 2022-01-12 20:26:41 +00:00
CalDescent
f44c21ce59 Disallow any kind of website/app/data rendering when localAuthBypassEnabled is enabled. 
This allows node operators to return their authentication to the legacy rules (local requests allowed), without introducing javascript vulnerabilities. The websites, apps, etc are just prevented from loading, to avoid the risk of any API calls from javascript.
 2022-01-12 19:32:52 +00:00
CalDescent
ade977e416 Don't broadcast any arbitrary signatures if the list is empty (i.e. the node isn't yet hosting anything) 2022-01-12 19:23:36 +00:00
CalDescent
f09a131bd6 Added requestHops to log entry. 2022-01-12 19:21:11 +00:00
CalDescent
4815587de1 Use V2 of string serialization methods in ArbitrarySignaturesMessage, as it is designed to allow null values. 2022-01-12 19:20:49 +00:00
CalDescent
e0ebfb9b53 Reduced log spam. 2022-01-12 19:20:14 +00:00
CalDescent
90836afd91 External IP address updates now require 10 consecutive readings. 2022-01-12 19:19:56 +00:00
CalDescent
4e1b0a25bb Fixed arbitrary peer tests, which used a local address (but we now treat them as invalid). 2022-01-11 20:06:14 +00:00
CalDescent
89c3236bf5 Updated bundled log4j2.properties files 2022-01-11 20:02:42 +00:00
CalDescent
7658bc2025 Added X-API-KEY header field to API documentation endpoints that require it. 2022-01-11 19:13:56 +00:00
CalDescent
ccde725d3b Check for an empty string as well as null in a couple of places, so that deserializeSizedString() can be safely used. 2022-01-10 19:42:21 +00:00
CalDescent
e3b45cac0a Use an alternative version of Serialization.serializeSizedString() and Serialization.deserializeSizedString() for the new ARBITRARY transaction additions. 
The modifications made to these methods were causing issues with other transaction types that were expecting blank strings instead of null. To keep risk to a minimum, I have split into two different sets of functions until there is more time to unify them.
 2022-01-10 19:41:02 +00:00
CalDescent
8f8a500dcd Fixed some issues left over from the qortaldata project 2022-01-10 19:26:25 +00:00
CalDescent
f9749cd82c Merge remote-tracking branch 'qortal-data/master' into qdn 2022-01-09 21:10:48 +00:00
CalDescent
051052fdd2 Removed authentication for GET /peers/summary endpoint 2022-01-09 21:09:20 +00:00
CalDescent
940304b4c2 Added GET /admin/apikey/test endpoint, so that we have a dedicated place to test if authentication works. 2022-01-09 20:08:45 +00:00
CalDescent
b4d2fae27f Fixed a couple of FOLLOWED_AND_VIEWED references that were missed 2022-01-09 16:26:52 +00:00
CalDescent
11e194292c Removed API key requirement from GET /admin/status and GET /admin/mintingaccounts 2022-01-09 16:26:23 +00:00
CalDescent
5ba6f6f53e FOLLOWED_AND_VIEWED renamed to FOLLOWED_OR_VIEWED, since it's technically an OR not an AND. 2022-01-09 13:25:49 +00:00
CalDescent
f58a16905f Removed unused setting. 2022-01-09 13:19:07 +00:00
CalDescent
33e82b336b Limit arbitrary signature requests to 3 hops, just in case a bug caused any kind of circular broadcasting. 2022-01-09 11:22:27 +00:00
CalDescent
0ced712974 Merge remote-tracking branch 'qortal-data/master' into qdn 2022-01-08 12:29:48 +00:00
CalDescent
db8e35cc13 Allow a new API key to be generated if the existing apikey.txt file has been deleted 2022-01-08 12:27:24 +00:00
CalDescent
b6db5aa2d3 Use "apikey.txt" instead of "apikey" as the filename to store the api key, to make it easier for users to open. 2022-01-08 10:22:14 +00:00
CalDescent
396dc5c9b0 Always log "Synchronizing with peer..." as it may help give more clarity to those with sync issues. 2022-01-08 10:12:54 +00:00
CalDescent
67e424a32a Added GET /arbitrary/relaymode API endpoint, which returns whether relay mode is enabled in the settings or not. 2022-01-07 14:38:05 +00:00
CalDescent
d8cbec41d2 Various logging improvements and fixes. 2022-01-07 14:08:11 +00:00
CalDescent
374f6b8d52 Added restrictions when relaying file list requests 
1) Each relay request expires after 5 seconds, after which nodes will stop relaying it, preventing any kind of infinite loop. So it has to reach the destination peer within 5 seconds. This should be fine, because the original peer's request would timeout anyway, so there's nothing to be gained by continuing to relay it.

2) Each relay request stops being forwarded after 3 "hops" - i.e. once it has been relayed through 3 different peers, it will no longer be transmitted any further. If we assume that each node has 16 connections, that allows it to reach a theoretical maximum of 4096 peers in 3 hops. In practice it will be less, and may not reach everyone due to peer "islands". But it will automatically retry a few times on a timer, so should hopefully find what it needs eventually. Plus, it still has the ability to make a direct connection to anyone hosting the data, as long as they are port forwarded.
 2022-01-07 14:01:57 +00:00
CalDescent
20ec4cbd14 ARBITRARY_REQUEST_TIMEOUT increased from 6 to 10 seconds 
This is likely longer than needed, but it's best to allow extra for now and then optimize the timeouts once we've had some experience with real world data.
 2022-01-07 12:50:34 +00:00
CalDescent
1c80835f49 Default relayModeEnabled to true. 
Even though a final decision is yet to be made, it makes sense to test with this scenario to ensure that everything works correctly.
 2022-01-07 12:31:16 +00:00
CalDescent
5e0af26c27 Keep track of successes or failures for a particular arbitrary peer / signature combination. 
This can help to inform decisions on data retention (although there is no deletion yet).
 2022-01-05 21:23:29 +00:00
CalDescent
b42674ac06 Small code reorganization to improve logic when adding arbitrary peer data 2022-01-05 19:26:06 +00:00
CalDescent
3394543705 Don't save arbitrary peer data if it's a local address 2022-01-05 19:22:24 +00:00
CalDescent
75c51aa61b If a direct connection can't be made to a peer over the original or default port, match the peer's host with any entries in knownPeers and try connecting to each of those until one succeeds. 2022-01-05 19:03:41 +00:00
CalDescent
6041722250 Added missing import. 2022-01-05 18:35:03 +00:00
CalDescent
60d038b367 Return result of Network.connectPeer() back to caller. 2022-01-05 18:34:54 +00:00
CalDescent
b2c4bf96af Rebroadcast the entire list of hosted signatures each time our external IP address changes. 2022-01-03 19:52:42 +00:00
CalDescent
f007f9a86d Added optional "senderPeerAddress" string to HELLO messages, to allow external IP changes to be detected without using a centralized service. 2022-01-03 19:05:10 +00:00
CalDescent
b1c1634950 Updated log4j to 2.17.1 
This involves modifying the log4j2.properties file on node startup to fix an incompatibility with ${dirname:-}. Thanks to AlphaX Projects for tracking down this incompatibility.
 2022-01-02 20:50:38 +00:00
CalDescent
5157ccf7c0 Expire each authorization after 60 minutes. 2022-01-02 20:11:38 +00:00
CalDescent
c4a782301d Delete reader cache directories if a resource is unable to be built. 2022-01-02 18:31:17 +00:00
CalDescent
17fe94fa46 Added POST /render/authorize/{resourceId} endpoint 
This allows an entire registered name to be preauthorized, therefore allowing for instance a website to automatically request other resources from the same author, such as videos.
 2022-01-02 18:23:27 +00:00
CalDescent
75d9347d23 Show blank root page for the gateway. 
This can ultimately be replaced with a website list / search engine.
 2022-01-02 17:07:48 +00:00
CalDescent
ef784124f3 Fixed status bugs in loading screen. 2022-01-02 16:11:25 +00:00
CalDescent
bd1b631914 Use <base href="..."> in HTMLParser, rather than attempting to swap out every relative link 
This delegates the task to the browser rather than doing it in java. It should also catch a few remaining types of links that we had missed - e.g. ones that originate from within js files.
 2022-01-02 15:22:53 +00:00
CalDescent
edfc8cfdc4 Fixed bug which saved an incorrect peer address for a forwarded arbitrary signatures message. 2022-01-01 22:34:02 +00:00
CalDescent
fbe34015d4 Validate peer addresses before saving anything to the db. 2022-01-01 22:33:01 +00:00
CalDescent
391fa008d0 When making a direct connection to a peer, try using the default listen port along with the one specified in the ArbitraryPeers table. 
This is a workaround to account for any ephemeral ports that may have made it into the dataset.
 2022-01-01 21:37:23 +00:00
CalDescent
7df8381b8f Don't allow a row to be added to ArbitraryPeers (or the message to be rebroadcast) if an entry already exists with the same hash and host/ip. 
This avoids duplicate entries from the same host/ip with differing ports. This can occur due to some requests using ephemeral port numbers. Ideally we would filter these out altogether, but this at least acts as a safety net to prevent a very cluttered db and associated "broadcast storm". The main tradeoff here is that multiple nodes on the same IP address will be recorded as a single entry. This doesn't seem like it will be a major limitation, because one of them will remain available.
 2022-01-01 21:09:48 +00:00
CalDescent
c0234ae328 Added unit test to make sure the mempow nonce is being validated in ARBITRARY transactions. 2022-01-01 18:12:18 +00:00
CalDescent
5c64a85d7c Test a registered name with a space in it 2022-01-01 15:43:00 +00:00
CalDescent
7aa8f115ce Added "original copy indicator file", which prevents the node from deleting its own published content when storage space runs out. 
Since some files won't have any mirrors, this prevents the cleanup manager from deleting the only copy in existence when freeing up space. This feature can be disabled by setting "originalCopyIndicatorFileEnabled": false in settings.json (or by deleting the ".original" files). The trade off is that the only copy in existence could be deleted if space gets low.

This will also allow for better reporting of own vs third party files in the local UI (not yet implemented).
 2022-01-01 14:52:09 +00:00
CalDescent
cf2c8d6c67 Switched some IOExceptions to DataExceptions 2022-01-01 14:02:54 +00:00
CalDescent
37edebcad9 Fixed cleanup bug which could cause problems for ArbitraryDataWriter 2021-12-30 15:50:14 +00:00
CalDescent
4d4f661548 Renamed ArbitraryResourceSummary to ArbitraryResourceStatus, and added status titles & descriptions 
This allows for consistent messaging about each status to be shown in different parts of the system. Previously these strings were hardcoded in the loading screen html so were inaccessible elsewhere.
 2021-12-29 19:41:49 +00:00
CalDescent
34e622cf0c Removed unused line. 2021-12-28 17:00:54 +00:00
CalDescent
7ccb99aa2c Improved clarity of file deletion message. 2021-12-28 17:00:41 +00:00
CalDescent
9e3847e56f Moved some of the less important arbitrary transaction related logs from INFO to DEBUG/TRACE. 
Logs can be reinstated by adding these lines to log4j2.properties:

logger.arbitrary.name = org.qortal.arbitrary
logger.arbitrary.level = debug
logger.arbitrarycontroller.name = org.qortal.controller.arbitrary
logger.arbitrarycontroller.level = debug
 2021-12-28 17:00:09 +00:00
CalDescent
90ced351f4 ARBITRARY transaction difficulty increased to 14, bringing it in line with MESSAGE transactions. 
14 is currently the lowest accepted difficulty for on-chain transactions, so it's best that we don't go lower than that.
 2021-12-28 14:25:17 +00:00
CalDescent
04295ea8c5 Set arbitrary transaction difficulty to 1 during unit tests, as they were taking too long. 2021-12-28 13:25:03 +00:00
CalDescent
2452d3c24b Moved proof of work difficulty definition from ArbitraryTransaction to ArbitraryDataManager 2021-12-28 13:01:26 +00:00
CalDescent
302428f1d1 Exclude RAW_DATA arbitrary transactions from various arbitrary data controller methods. 2021-12-28 12:01:03 +00:00
QuickMythril
cf603aa80e
disable Open UI menu item 
this doesn't work and should either be fixed or removed completely.
 2021-12-25 14:23:49 -05:00
CalDescent
1f9f949a8c Added GET /peers/summary API which returns counts of the number of inbound and outbound connections that currently exist. 2021-12-24 15:25:58 +00:00
CalDescent
0bde1e97dc Added DELETE /resource/{service}/{name}/{identifier} API endpoint, to delete local data 2021-12-24 13:04:16 +00:00
CalDescent
42aca2e40f arbitraryDataCachedResources is now keyed by all ArbitraryDataResource elements, not just the resourceId 2021-12-24 12:12:16 +00:00
CalDescent
e1e44d35bb ArbitraryDataBuildQueueItem now extends ArbitraryDataResource 
This is the start of a refactor to use ArbitraryDataResource objects rather than passing around separate resourceId, service and identifier, or other duplicate objects. Most of this will need to be done after the initial release due to time constraints.
 2021-12-24 12:05:03 +00:00
QuickMythril
a790b2e529
reduce DOGE fees 
https://github.com/dogecoin/dogecoin/blob/master/doc/fee-recommendation.md
 2021-12-23 19:10:17 -05:00
CalDescent
357946388c New manager classes are no longer subclasses of Thread, as this part wasn't being used for anything. 2021-12-23 22:23:07 +00:00
CalDescent
b774583f28 Disabled all arbitrary data manager threads if QDN is disabled in the settings. 2021-12-23 22:19:14 +00:00
CalDescent
6436daca08 Split up ArbitraryDataManager into three separate manager classes, to make the code more readable and to keep each class within 500 lines. 
This is still the least readable area of the data hosting code, but it should be a bit more maintainable now.
 2021-12-23 22:15:28 +00:00
CalDescent
f153c7bb80 Added "qdnEnabled" setting (default: true) to allow users to opt out of QDN functionality. 2021-12-23 21:38:30 +00:00
CalDescent
1f8a618dcc Removed recently added logs. 2021-12-23 17:30:34 +00:00
CalDescent
2d853e5a2f Added support of patch creation from files without a newline at the end 
The simplest solution was to only include a newline at the end of the patch file if the source file ended with a newline. This is used to inform the merge code as to whether to add the newline to the end of the resulting file. Without this, the checksums do not match (and therefore previously the complete file would have been included as a result).
 2021-12-23 16:59:49 +00:00
CalDescent
361dc79ede Fixed bug caused by multiple concurrent builds of the same resource. 
Several parts of the code request resources to be loaded/built, and these separate threads were tripping over each other and causing build failures. This has been avoided by making sure the resource isn't already building before requesting it.
 2021-12-22 19:31:44 +00:00
CalDescent
19173321ea Small refactor to reduce code duplication. 2021-12-22 17:03:54 +00:00
CalDescent
87b724ec72 Updated various places that used File.separator when they could have used Paths.get() 2021-12-22 15:57:40 +00:00
CalDescent
67db0f950b Builds are now keyed by service, resourceId and identifier. 
This allows for concurrent builds of multiple resources from the same registered name.
 2021-12-22 15:44:00 +00:00
CalDescent
f85bbf12ca Applied reserved "default" keyword when publishing data too - it now maps to a blank identifier. 2021-12-22 13:59:08 +00:00