crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
2,523 Commits 85 Branches 118 Tags 530 MiB
a4bcd4451c
Commit Graph

2257 Commits

Author SHA1 Message Date
CalDescent
becb0b37e6
Merge branch 'master' into network-online-accounts-v2 2022-02-18 17:12:44 +00:00
CalDescent
67ca876567 Log if we're unable to process the received file. 2022-02-18 15:35:37 +00:00
CalDescent
464ce66fd5 Moved deletion retry code into ArbitraryDataFile 2022-02-18 15:09:50 +00:00
CalDescent
3e505481fe Default minPeerVersion also increased to 3.1.0 2022-02-18 14:50:46 +00:00
CalDescent
c90c3a183e Block peers below 3.1.0 2022-02-18 14:50:29 +00:00
CalDescent
85b3278c8a Don't throttle the arbitrary data file request threads when there are items to process. 2022-02-15 19:39:26 +00:00
CalDescent
c90c287601 Increased ARBITRARY_REQUEST_TIMEOUT from 10 to 12 seconds, as some were coming back around 9-10 seconds later. 2022-02-15 19:38:30 +00:00
CalDescent
6ee395ed12 Stop bulk arbitrary signature broadcasts, as they were creating a lot of network traffic, and are in the process of being replaced with a better method. 2022-02-15 19:10:20 +00:00
CalDescent
6275ac2b81 Increased numberOfAdditionalBatchesToSearch from 5 to 7. 
This is the equivalent of increasing the max address gap from 15 to 21. The electrum standalone wallet uses 20, so this should be the most we will ever need.
 2022-02-14 22:58:37 +00:00
CalDescent
fd0a6ec71f Fix for invalid balance (and transaction amount) when there are no outputs relating to this wallet. 2022-02-14 22:53:30 +00:00
CalDescent
43791f00aa Wait 2 minutes on node startup before trying to fetch followed QDN data. 2022-02-14 19:33:58 +00:00
CalDescent
538ac30b4e Request only the missing hashes, not all of them. 2022-02-14 19:33:36 +00:00
CalDescent
acddf36467 Handle missing includeMetadata parameter. 2022-02-13 19:27:12 +00:00
CalDescent
166d32032a Fixed message IDs. 2022-02-13 19:22:20 +00:00
CalDescent
e4238a62c9 Exclude metadata-only transactions in the data management page (but added an API parameter to allow them to optionally be included). 
This ensures that the list will only show resources where there is at least 1 chunk.
 2022-02-13 19:21:16 +00:00
CalDescent
ad9c466712 Fall back to UNCATEGORIZED if the parsed category doesn't match any available categories. 
This allows for deletion of categories, as the resources will just move into UNCATEGORIZED until they are next updated.
 2022-02-13 18:10:56 +00:00
CalDescent
a3d31bbaf1 Category updates based on feedback so far. 2022-02-13 17:56:47 +00:00
CalDescent
4821139501 Merge branch 'master' into qdn-metadata 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java
 2022-02-13 15:50:12 +00:00
CalDescent
83213800b9 Use the timestamp from the registerNameTransactionData in unit tests, rather than the current time. 2022-02-13 15:05:28 +00:00
CalDescent
265ae19591 Fixed other failing tests due to increased REGISTER_NAME transaction fee. At some point we should determine the correct fee inside of generateBase(), but setting it explicitly adds confidence in testing for now. 2022-02-13 14:31:21 +00:00
CalDescent
c1598d20b5 Name registration fee increase timestamp set to Sunday, 20 February 2022 16:00:00 UTC 2022-02-13 13:47:00 +00:00
CalDescent
0712259057 Implemented REGISTER_NAME transaction fee increase from 0.001 to 5 QORT (average value based on community vote). 2022-02-13 13:45:48 +00:00
CalDescent
ea42a5617f Fixed ElectrumX log spam and errors 2022-02-13 10:58:45 +00:00
CalDescent
58a690e2c3 Route through new getAddressTransactions() wrapper. 2022-02-11 18:15:27 +00:00
CalDescent
3ae2f0086e Removed unusably slow electrum peer 2022-02-11 18:13:45 +00:00
CalDescent
19c83cc54d MAX_AVG_RESPONSE_TIME reduced to 500, as one peer regularly takes around 600ms to reply. 2022-02-11 18:12:34 +00:00
CalDescent
8ac298e07d Allow 3 retries for getTransaction() and getAddressTransactions() requests 2022-02-11 18:11:00 +00:00
CalDescent
9b43e4ea3d Time electrum requests, and move on to another server if one takes more than 1000ms on average to respond (measured over the last 5 requests). 2022-02-11 18:02:56 +00:00
CalDescent
dbacfb964b Increased TX_CACHE_SIZE from 200 to 1000, to speed up loading times on large wallets. 2022-02-11 16:55:29 +00:00
CalDescent
a664a6a790 Added more LTC Electrum peers from https://1209k.com/bitcoin-eye/ele.php?chain=ltc 2022-02-11 16:44:34 +00:00
CalDescent
ee1f072056 Improvement to last commit, so that caller class names are preserved. 2022-02-11 15:34:31 +00:00
CalDescent
a6aabaa7f0 Reduce build queue log spam by only logging high priority items (5 and above). 2022-02-11 15:28:41 +00:00
CalDescent
49b307db60 Treat a null priority as 0 2022-02-11 15:17:02 +00:00
CalDescent
f7341cd9ab Increased /arbitrary priority to 1 2022-02-11 15:13:53 +00:00
CalDescent
6932fb9935 Added "priority" property to build queue items. 
/render APIs use priority 10, whereas /arbitrary use priority 0, to prevent thumbnail downloads from holding up website loading. The priorities can be adjusted later, with maybe some service types being given higher priority than others.
 2022-02-11 15:08:12 +00:00
CalDescent
2343e739d1 Handle case where a file cannot be unzipped. 2022-02-11 14:35:46 +00:00
CalDescent
fc82f0b622 Use 5 builder threads, so that one slow resource (e.g. a thumbnail) doesn't hold up the other queued build items. 
This can be replaced with a task-based approach longer term.
 2022-02-11 13:58:45 +00:00
CalDescent
c0c50f2e18 Updated bootstrap hosts 2022-02-11 13:33:25 +00:00
CalDescent
9332d7207e Fixed bug in cache clearing logic, which was often preventing resource updates from being detected. 2022-02-10 09:22:54 +00:00
CalDescent
a8c79b807b Discard any uncommitted changes as a result of the higher weight chain detection 2022-02-10 08:16:30 +00:00
CalDescent
2637311ef5 Prevent potential ConcurrentModificationException in the build queue 2022-02-09 20:20:30 +00:00
CalDescent
06b5b8f793 Reduced time between processing build tasks, to prevent builds with invalid criteria from holding up legitimate builds too much. 2022-02-09 20:17:56 +00:00
CalDescent
61f58173cb Revert "Removed transaction caching. Can be reintroduced later." 
This reverts commit 9804eccbf0.
 2022-02-09 19:46:20 +00:00
CalDescent
b7b66f6cba Revert "Removed getWalletTransactions() synchronization. Again, can be re-added later." 
This reverts commit 70c864bc2f.
 2022-02-09 19:46:16 +00:00
CalDescent
dda2316884 Revert "Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance." 
This reverts commit d7658ee9f9.
 2022-02-09 19:46:10 +00:00
CalDescent
b782679d1f Revert "Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj."" 
This reverts commit 214f49e356.
 2022-02-09 19:46:06 +00:00
CalDescent
b0f19f8f70 Merge branch 'block-minter-updates' 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java
 2022-02-09 19:42:39 +00:00
CalDescent
de5f31ac58 Don't process file hashes if we're stopping 2022-02-09 19:40:20 +00:00
CalDescent
214f49e356 Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj." 
This reverts commit 892612c084.

# Conflicts:
#	src/main/java/org/qortal/crosschain/Bitcoiny.java
 2022-02-08 18:29:32 +00:00
CalDescent
d7658ee9f9 Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-08 18:27:44 +00:00
CalDescent
70c864bc2f Removed getWalletTransactions() synchronization. Again, can be re-added later. 2022-02-08 18:27:08 +00:00
CalDescent
9804eccbf0 Removed transaction caching. Can be reintroduced later. 2022-02-08 18:26:15 +00:00
CalDescent
d1f24d45da Added defensiveness in convertToSimpleTransaction() 2022-02-08 18:24:42 +00:00
CalDescent
9630625449 Rework of processIncomingTransactionsQueue() so that it no longer holds the lock while processing. 
This should fix an issue where network threads could be blocked when new transactions arrived, due to waiting for the incomingTransactions lock to free up.
 2022-02-08 09:18:14 +00:00
CalDescent
b72153f62b Renamed main thread from "Controller" to "Qortal" 2022-02-08 09:02:20 +00:00
CalDescent
0a88a0c95e Perform the base58 decoding outside of the arbitraryDataFileHashResponses lock, to reduce the amount of waiting around by other threads. 2022-02-08 08:45:58 +00:00
CalDescent
ab4ba9bb17 Don't re-fetch unconfirmed transactions that are already in the queue 2022-02-08 08:36:45 +00:00
CalDescent
a49218a840 Optimized ArbitraryDataFileRequestThread - only start a database transaction when there's something to process. 2022-02-07 22:06:45 +00:00
CalDescent
b6d633ab24 Break out of incoming transactions processing loop if we need to sync. 2022-02-07 22:05:13 +00:00
CalDescent
133943cd4e Reduce log spam 2022-02-07 22:03:41 +00:00
CalDescent
f8ffb1a179 Updated thread names 2022-02-07 22:03:26 +00:00
CalDescent
41c4e0c83e Merge branch 'master' into block-minter-updates 2022-02-06 18:40:32 +00:00
CalDescent
99f6bb5ac6 Reorganized some controller methods. 2022-02-06 18:32:43 +00:00
CalDescent
3e0306f646 Increased minPeerConnectionTime and maxPeerConnectionTime to reduce the chances of forced connections during relays. 
An alternate option would be to avoid force disconnecting while relays are in progress, but some nodes could have active relays 100% of the time and therefore would never recycle their peers. So it is simpler to just increase the average peer connection time for everyone.
 2022-02-06 17:29:00 +00:00
CalDescent
84e4f9a1c1 Rework of arbitraryRelayMap to keep track of multiple responses. 
Previously, only one peer's response for a hash would be remembered, even if multiple others reported back too. This would cause useful mapping to be lost.
 2022-02-06 17:20:01 +00:00
CalDescent
cd5ce6dd5e Don't remove from the relay map after a file is requested, as it may be needed by other peers. 
It will be cleaned up automatically after 60 seconds, so it is best to keep the data intact until then.
 2022-02-06 16:04:54 +00:00
CalDescent
9ec4e24ef6 Slightly optimized logic in fetchArbitraryDataFiles() 2022-02-06 15:45:40 +00:00
CalDescent
fa447ccded Builder thread updates. 2022-02-06 15:37:21 +00:00
CalDescent
ef838627c4 Stop asking for hashes from a peer if one fails. 
This fixes the request looping that occurs on when a peer is unable to serve files.
 2022-02-06 15:37:08 +00:00
CalDescent
b8aaf14cdc Introduced ArbitraryDataFileRequestThread to allow for multiple concurrent file requests. 
This is likely a short term solution (to allow existing code to be repurposed) until replaced with a task-based approach, as this will allow for a much greater number of threads.
 2022-02-06 15:34:06 +00:00
CalDescent
2740543abf Added "async" and "attempts" parameters to GET /arbitrary/{service}/{name}* endpoints. 
async = fail immediately with 404 if missing, and request in the background
attempts = the number of times to request the data (synchronous mode only for now)
 2022-02-06 13:04:58 +00:00
CalDescent
3c526db52e Fixed bug in build manager which would prevent future builds until the core was restarted. 2022-02-06 13:03:01 +00:00
CalDescent
cfe0414d96 Small rework of invalidUnconfirmedTransactions to specify the expiry time instead of the time added. 
This allows TIMESTAMP_TOO_OLD transactions to be tracked for a shorter time (10 minutes) than the other invalid transactions (60 minutes). Should reduce network traffic and db load around the time that transactions are expiring, as there is a lag before they are noticed and removed from each node. Due to the variance, it could cause other peers to request them again after deleting. They are now ignored for 10 minutes to avoid request spam.
 2022-02-06 12:35:41 +00:00
CalDescent
08e06ba11a Fixed bugs preventing invalidUnconfirmedTransactions from working as intended. 2022-02-06 12:09:44 +00:00
CalDescent
8c03164ea5 Don't add expired transactions to invalidUnconfirmedTransactions, as there is no need to keep track of these. 2022-02-06 11:55:07 +00:00
CalDescent
0fe2f226bc Added invalidUnconfirmedTransactions map 
An incoming invalid unconfirmed transaction will be added to this map if its timestamp is more than 30 minutes old. This should allow enough time and opportunities for it to be imported and included in a block (allowing for re-orgs which could switch its status from invalid to valid).

Once added, it will be removed after an hour to allow for another chance to be requested from any peers that still have it. If invalid again, it's added back to the map for another hour.

This fixes a 24 hour long loop, where invalid transactions are requested over and over from peers that have already imported them. It could be improved further by periodically removing invalid unconfirmed transactions from the database, but this will be a higher risk.

The results of this feature should be less network traffic, and less blockchain locks (which should ultimately increase the responsiveness of the synchronizer).
 2022-02-06 11:23:28 +00:00
CalDescent
55b5702158 Invalidate last low weight block signature whenever the previous block data changes. 2022-02-05 17:54:36 +00:00
CalDescent
a4cbbb3868 Moved block minter sleep to later in the process, otherwise it can remain there for longer than expected. 2022-02-05 17:54:36 +00:00
CalDescent
816b01c1fc Fixed issue in rebase 2022-02-05 17:54:36 +00:00
CalDescent
483e7549f8 Revert "Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before." 
This reverts commit e2e87766fa.
 2022-02-05 17:54:35 +00:00
CalDescent
60d71863dc Allow 3 seconds for the synchronizer to obtain a blockchain lock, to reduce missed attempts. 2022-02-05 17:54:35 +00:00
CalDescent
170244e679 More work on higher weight chain detection in the block minter. 
Added 30 second timeout, so that any errors should self correct.
 2022-02-05 17:54:35 +00:00
CalDescent
472e1da792 Added debug level logging in higherWeightChainExists() for better visibility. 2022-02-05 17:54:35 +00:00
CalDescent
cbf03d58c8 Made synchronizer method public as it is now also used by the block minter. 2022-02-05 17:54:35 +00:00
CalDescent
ba41d84af9 Initial attempt to avoid an unnecessary block submission if one of our peers already has a higher weight chain. 2022-02-05 17:54:35 +00:00
CalDescent
98831a9449 Break out of the various loops in the cleanup manager if the thread is stopping. 2022-02-05 17:53:49 +00:00
CalDescent
9692539a3f Don't include fee in balance calculation (it looks like it could be double counting at the moment). 2022-02-05 17:24:33 +00:00
CalDescent
76df332b57 Check for null IP address before notifying of an external IP update. 2022-02-05 11:51:54 +00:00
CalDescent
c6405340bc minAccountLevelForBlockSubmissions reduced from 6 to 5 2022-02-05 11:33:28 +00:00
CalDescent
775e3c065e Invalidate ElectrumX transactions cache when switching accounts. 2022-02-05 10:23:25 +00:00
CalDescent
8937b3ec86 Don't allow duplicate transaction in the incoming transactions queue. 
This should reduce database load slightly, as it won't have to check the same transaction multiple times in each batch.
 2022-02-05 10:19:26 +00:00
CalDescent
3fbb86fded Added indexes, to make looking up name transactions by name around 5x faster. 2022-02-04 16:27:31 +00:00
CalDescent
0cf2f7f254 Missing import from last commit 2022-02-04 16:18:00 +00:00
CalDescent
9e571b87e8 Yet another rewrite of fetchAllTransactionsInvolvingName() - this time over 1000x faster since it doesn't involve joining the Transactions table. 2022-02-04 16:17:31 +00:00
CalDescent
23bafb6233 Removed unused methods 2022-02-04 14:00:44 +00:00
CalDescent
6dec65c5d9 Rewrite of fetchAllTransactionsInvolvingName() to avoid having to load all name transactions into memory. 2022-02-04 13:58:05 +00:00
CalDescent
4e59eb8958 Added unit test to simulate the false association between previous a UPDATE_NAME transaction, and the emoji name with a blank reducedName. 2022-02-04 12:51:22 +00:00
CalDescent
756d5e685a Added naming tests for blank new names 2022-02-04 12:50:05 +00:00
CalDescent
f52530b848 More thorough approach to fetchAllTransactionsInvolvingName(), to fix an issue found in unit testing. 2022-02-04 11:58:43 +00:00
CalDescent
c2bf37b878 Added transactionV5Timestamp featureTrigger to unit tests 2022-02-04 11:37:41 +00:00
CalDescent
98a2dd04b8 Fixed bug caused by improper handling of UPDATE_NAME transactions, similar to commit d16663f. 2022-02-04 10:28:26 +00:00
CalDescent
694ea689c8 Synchronize the loop and break out of it before fetching arbitrary data files. Hopeful fix for ConcurrentModificationException, and maybe a potential deadlock. 2022-02-03 21:14:41 +00:00
CalDescent
618aaaf243 Removed logs used for debugging only 2022-02-03 21:06:36 +00:00
CalDescent
9224ffbf73 Cache transaction list for 2 minutes, and synchronize, to prevent the balance and transactions APIs both requesting at once. 
This ensures that only a single round of requests (per coin) is used for the wallettransactions and balance APIs. It also speeds up loading on subsequent requests. The 2 minute cache isn't much longer than the foreign block times, so shouldn't cause values to be too out of date.
 2022-02-03 21:04:49 +00:00
CalDescent
892612c084 Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj. 
Hopeful fix for incorrect balances in wallets with large numbers of transactions. At the very least, this gives us control of the code that calculates the balance.
 2022-02-03 20:22:25 +00:00
CalDescent
077165b807 Modified fetchArbitraryDataFileList() to support requesting only the missing hashes, but it is not yet used. 
Once GetArbitraryDataFileListMessage is rolled out to the network we can uncomment this code. Needs testnet testing prior to that.
 2022-02-03 20:01:44 +00:00
CalDescent
7994fc6407 Rework of onNetworkGetArbitraryDataFileListMessage() to support custom hashes to be optionally supplied. 
Also simplified the existing logic, to make the code more readable.
 2022-02-03 19:58:50 +00:00
CalDescent
d98df3e47d Added support for file hashes to optionally be included in GetArbitraryDataFileListMessage. 
Needs testing on testnet, as the majority of nodes need to update before the hashes can be used.
 2022-02-03 19:55:22 +00:00
CalDescent
1064b1a08b Removed duplicate metadata file checks. 2022-02-03 19:53:29 +00:00
CalDescent
0b7a7ed0f1 Added some more debug logging relating to file responses. 2022-02-03 19:52:28 +00:00
CalDescent
114b1aac76 Added arbitrary data file manager thread, which will ensure that all file list responses are tried until we receive the files. 
Previously we would only try the first response and then discard the others due to being duplicates. They are now added to a queue and retried by the dedicated thread (up to the 60 second timeout).
 2022-02-03 19:51:02 +00:00
CalDescent
6d06953a0e Increased RELAY_REQUEST_MAX_HOPS from 3 to 4, in an attempt to reach more peers. 2022-02-02 22:31:40 +00:00
CalDescent
0430fc8a47 Give up immediately after a synchronization if we are shutting down the synchronizer 2022-02-02 09:18:46 +00:00
CalDescent
7338f5f985 Attempt to acquire a blockchain lock (for up to 5 seconds) before shutting down the repository. 
This should fix conflicts caused by the synchronizer and controller now being on separate threads. It may also reduce the chances of the database corrupting on shutdown, but this remains to be seen.
 2022-02-02 09:17:24 +00:00
CalDescent
640bcdd504 Shutdown/interrupt the synchronizer as early as possible 2022-02-02 09:13:49 +00:00
CalDescent
c9d5d996e5 Increased accuracy of the block weights in the synchronizer logs, as the extra precision is now needed for debugging. 2022-02-01 22:06:01 +00:00
CalDescent
710befec0c Increased ARBITRARY_RELAY_TIMEOUT from 30 to 60 seconds, so that relay peers remember their mappings for longer. 2022-02-01 22:04:53 +00:00
CalDescent
8ccb158241 Reduce log spam when in DEBUG mode. 2022-02-01 22:04:12 +00:00
CalDescent
97199d9b91 Display the local and total chunk counts on the loading screen. 2022-02-01 22:03:52 +00:00
CalDescent
5a8b895475 Fixed bug in loading screen, which prevented the DOWNLOADING status from showing. 2022-02-01 22:03:17 +00:00
CalDescent
6c9600cda0 Added API key header to GET /arbitrary/resource/status/* endpoints 2022-02-01 22:02:41 +00:00
CalDescent
82fa6a4fd8 Include "localChunkCount" and "totalChunkCount" in the GET /arbitrary/resource/status/* API responses. 
These values are left out of other API endpoints where multiple resources are returned, because calculating the chunk counts is too time consuming.
 2022-02-01 22:02:14 +00:00
CalDescent
45f2d7ab70 Revert "We (currently) can't filter unconfirmed transactions by address, because only the public key is stored in the database until it is confirmed (at which point there is an entry in the TransactionParticipants table which contains the address). Given that this isn't a simple problem to solve, for now it makes sense to reject this combination if requested via the /transactions/search API." 
This reverts commit 7aed0354f1.
 2022-02-01 21:55:41 +00:00
CalDescent
33731b969a Direct peer connections now send a file list request to the peer, rather than individually requesting every chunk for a transaction. 2022-02-01 09:04:31 +00:00
CalDescent
40a8cdc71f Improved logging when fetching data files 2022-01-31 22:35:12 +00:00
CalDescent
cbe83987d8 Fixed occasional ConcurrentModificationException in the block archive reader. 2022-01-31 21:39:49 +00:00
CalDescent
01e4bf3a77 Try to speed up the shutdown process of the cleanup manager. 2022-01-31 21:39:20 +00:00
CalDescent
b198a8ea07 Fixed issue in ArbitraryDataFile.chunkExists() due to it not checking for the metadata file. 2022-01-31 21:38:51 +00:00
CalDescent
e2e87766fa Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before. 2022-01-30 20:15:43 +00:00
CalDescent
f005a0975d Shut down the synchronizer as soon as the controller is stopping, if we are able to. 2022-01-30 20:14:11 +00:00
CalDescent
5700369935 Prioritize syncing over transaction importing. 2022-01-30 20:09:35 +00:00
CalDescent
8a1fb6fe4e If a lock can't be obtained when synchronizing, automatically request the sync again. 2022-01-30 20:09:09 +00:00
CalDescent
5b788dad2f Relocated all synchronizer code from the controller to the synchronizer, and also moved synchonization onto its own thread. 2022-01-30 20:01:22 +00:00
CalDescent
fa2bd40d5f Reduce log spam 2022-01-30 19:51:49 +00:00
CalDescent
074bfadb28 Don't bother locking if there are no new transactions to process 2022-01-30 19:50:09 +00:00
CalDescent
bd60c793be Incoming transactions are now added to a queue, and then processed soon after. 
This solves a problem where incoming transactions could rarely obtain a blockchain lock (due to multiple transactions arriving at once) and therefore most messages were thrown away. It was also causing constant blockchain locks to be acquired, which would often prevent the synchronizer from running.
 2022-01-30 19:03:31 +00:00
CalDescent
90f3d2568a Log whenever the synchronizer can't obtain a blockchain lock, so that blockchain lock issues are more easily noticed. 2022-01-30 19:00:28 +00:00
CalDescent
c73cdefe6f transactionV5Timestamp moved to blockchain.json 2022-01-30 13:47:20 +00:00
CalDescent
c5093168b1 minAccountLevelToMint value moved to blockchain.json 2022-01-29 22:59:04 +00:00
CalDescent
a35e309a2f minAccountLevelForBlockSubmissions moved to blockchain.json 2022-01-29 22:57:22 +00:00
CalDescent
bb35030112
Merge pull request #65 from QuickMythril/reduce-doge-fee 
reduce DOGE fees
 2022-01-29 20:14:04 +00:00
CalDescent
7aed0354f1 We (currently) can't filter unconfirmed transactions by address, because only the public key is stored in the database until it is confirmed (at which point there is an entry in the TransactionParticipants table which contains the address). Given that this isn't a simple problem to solve, for now it makes sense to reject this combination if requested via the /transactions/search API. 2022-01-29 19:24:28 +00:00
CalDescent
c4f763960c Don't delete a resource's cache if a build is in progress. 
Hopeful fix for "Unable to delete cache for resource: Unable to delete directory" error, and possibly some other file conflicts.
 2022-01-29 19:18:06 +00:00
CalDescent
c5182a4589 Increased MAX_ACCOUNT_COUNT in GetOnlineAccountsMessage from 1000 to 5000 2022-01-29 12:39:43 +00:00
CalDescent
fc1a376fbd Added POST /transaction/fee API endpoint, to return the recommended fee for the supplied transaction data. 2022-01-29 10:38:15 +00:00
CalDescent
27387a134f Fixed typo 2022-01-28 19:58:32 +00:00
CalDescent
be7bb2df9e Added GET /transaction/unitfee API endpoint, to obtain the unit fee for a transaction type 
Additional params:
- timestamp: to allow for hard forks. Default: the current time
- level: the account level, to allow for the future possibility of different fees per level. Not currently used.
 2022-01-28 19:00:15 +00:00
CalDescent
72a291a54a Added initial support of different unit fees per transaction type. 
Included a timestamp property, as this will be needed for each update (hard fork).
 2022-01-28 18:50:03 +00:00
CalDescent
b1342d84fb Updated potentially misleading log message. 2022-01-28 11:02:37 +00:00
CalDescent
cdd57190ce Use getEffectiveMintingLevel() rather than getLevel() 2022-01-28 10:16:42 +00:00
CalDescent
7808a1553e Fixed case sensitive ordering issues in other aspects of QDN. 2022-01-27 18:46:59 +00:00
CalDescent
a0ba016171 Fixed case sensitive ordering issue in websites list. 2022-01-27 18:42:52 +00:00
CalDescent
344704b6bf MIN_LEVEL_FOR_BLOCK_SUBMISSION temporarily increased to 6. 
This is to hopefully improve network stability whilst a more advanced solution is being worked on. It also allows us to collect some data on how well the network behaves when there are less block candidates. It should have no effect on minting rewards (other than any side effects as a result of improved network stability).
 2022-01-27 18:15:00 +00:00
CalDescent
3303e41a39 Fixed unhandled case in GET /arbitrary/{service}/{name}* endpoints 2022-01-27 18:12:21 +00:00
CalDescent
4e71ae0e59 Allow QDN data to be served without authentication by setting "qdnAuthBypassEnabled":true 
This allows the GET /arbitrary/{service}/{name} and GET /{service}/{name}/{identifier} endpoints to operate without any authentication. Useful for those who are running public QDN nodes and need to serve data over http(s).
 2022-01-27 18:10:24 +00:00
CalDescent
9daf7a6668 Synchronize lists, to prevent an occasional ConcurrentModificationException 2022-01-26 22:40:34 +00:00
CalDescent
af06774ba6 Clear the cache when deleting data, so that it disappears from the data management screen. 2022-01-23 22:53:35 +00:00
catbref
244d4f78e2 New network messages ONLINE_ACCOUNTS_V2 and GET_ONLINE_ACCOUNTS_V2. 
Increased GetOnlineAccountsMessage.MAX_ACCOUNT_COUNT from 1000 to 5000.

The V2 versions are more efficiently encoded and also cache the payload bytes
which reduces CPU when sending to multiple peers.

Serialization / deserialization unit tests included.

Tentative V2 message activation set at core version 3.1.2
see Controller.ONLINE_ACCOUNTS_V2_PEER_VERSION
 2022-01-23 16:24:10 +00:00
CalDescent
6f7c8d96b9 Removed ApiService instance creation in ApplyUpdate as it wasn't really needed, and probably not sensible to instantiate it here. 2022-01-23 12:57:28 +00:00
CalDescent
ff6ec83b1c Removed localAuthBypassEnabled override in unit tests. 
Hopefully this will allow us to proactively catch any missing API keys in the future.
 2022-01-23 12:48:37 +00:00
CalDescent
ea10eec926 Fixed bug in auto update process - use the API key when stopping the node. 
Luckily this code is included in the new JAR, not the old one, so we should be able to regain auto update ability by issuing a new update.
 2022-01-23 12:38:19 +00:00
CalDescent
6f724f648d Fixed testDirectoryDigest() which has been failing for a couple of versions (due to gitignore removing the cache file) 2022-01-22 20:48:58 +00:00
CalDescent
048776e090 Ignore failing test due to recent API update, which makes the test incompatible. To be fixed later. 2022-01-22 20:43:28 +00:00
CalDescent
dedf65bd4b Added initial protocol methods for metadata requests and forwarding. Not tested yet. 2022-01-22 20:24:37 +00:00
CalDescent
25efee55b8 Added networking optimization, to avoid wasted processing on every read. 
Thanks to @catbref for finding this.
 2022-01-22 18:43:32 +00:00
CalDescent
a79ed02ccf Added initial (unfinished) category list, as well as the GET /arbitrary/categories API, and converted the category field from a string to an enum 2022-01-22 12:11:16 +00:00
CalDescent
79f87babdf Limit the metadata string lengths 2022-01-21 22:52:31 +00:00
CalDescent
f296d5138b Allow metadata to optionally be included with any arbitrary resource. 2022-01-21 21:14:28 +00:00
CalDescent
b30445c5f8 Increase MAX_ACCOUNT_COUNT to 5000 (around 0.5MB of data) to see if it helps with minting efficiency. 2022-01-21 16:11:21 +00:00
CalDescent
d105613e51 Improvements to some /blocks API endpoints 2022-01-21 16:07:18 +00:00
CalDescent
ef43e78d54 Reduced log spam 2022-01-21 15:20:16 +00:00
CalDescent
6f61fbb127 Reduced strictness of rate limiter, to allow two additional file list requests (15 and 30 seconds after initial attempt) 
This should help catch any remaining chunks that were unable to be fetched in the first attempt due to network failures, bad peers, etc.
 2022-01-21 15:20:04 +00:00
CalDescent
9f9b7cab99 Include "size" in /arbitrary/resource APIs 
Note that this is not always accurate - it relates to the largest transaction size for this name, not necessarily the latest or the combined size of multiple transactions. This can be made accurate as soon as we have a "Resources" table to store this info. Trying to do it before then will be too inefficient in terms of queries.
 2022-01-21 14:42:37 +00:00
CalDescent
f129e16878 Added retry mechanism to relay file deletions, just in case it fails on the first try. 
A longer term solution to this hypothetical problem is to store relays in RAM or a temp folder only. Or maybe add an indicator file to instruct the cleanup manager to delete it. But this will require more development. 10 deletion attempts (each 1 second apart) should be enough for now.
 2022-01-21 14:32:15 +00:00
CalDescent
8a42dce763 Use the first responding peer in the relay map. 
This encourages shorter relays, since longer ones will take more time to respond, and also prevents a peer from intentionally taking a long time to respond so that it overwrites an existing entry.

Longer term we could consider keeping track of all respondents for each hash, if there are still issues with data retrieval. I suspect this won't be needed though, as the requesting peer has 16+ different peers connected, and therefore potentially 16 different mappings already.
 2022-01-21 14:23:55 +00:00
CalDescent
6423d5e474 Optimized onNetworkGetArbitraryDataFileListMessage() to remove duplicate calls to Files.exists() 2022-01-21 14:12:59 +00:00
CalDescent
6e91157dcf Improved cache clearing process and logging. 2022-01-21 13:32:59 +00:00
CalDescent
85c61c1bc1 Added GET /arbitrary/resources/search API 
Example usage

List all websites with a name containing the word "crow" (using default identifier):
http://localhost:12391/arbitrary/resources/search?service=WEBSITE&query=crow&default=true&limit=20

List all resources with name or identifier containing the word "crow":
http://localhost:12391/arbitrary/resources/search?query=crow&default=false&limit=20
 2022-01-21 10:04:18 +00:00
CalDescent
54af36fb85 Remove duplicates in GET /arbitrary/hosted/resources response 2022-01-20 22:34:41 +00:00
CalDescent
fcdcc939e6 Sort hosted data in reverse order (newest first) 2022-01-20 22:29:19 +00:00
CalDescent
13450d5afa Added limit/offset to GET /arbitrary/hosted API endpoints 2022-01-20 22:28:28 +00:00
CalDescent
5e1e653095 Removed unnecessary database lookups in GET /hosted/resources API 2022-01-20 20:39:46 +00:00
CalDescent
e8fabcb449 Removed extra isDataLocal() check in GET /hosted/resources which was evading the cache. 2022-01-20 20:38:57 +00:00
CalDescent
1b42062d57 Default minPeerVersion set to 3.0.1 2022-01-20 20:25:20 +00:00
CalDescent
c2a4b01a9c Allow local playback of media files 2022-01-20 20:25:06 +00:00
CalDescent
47e763b0cf Take a copy of the IP address history so it can be safely iterated. Without this, another thread could remove an element, resulting in an exception. 2022-01-20 20:24:51 +00:00
CalDescent
0278f6c9f2 Reduced more log spam 2022-01-20 20:24:26 +00:00
CalDescent
d96bc14516 Allow execution of inline scripts, at least for now. 2022-01-17 20:25:25 +00:00
CalDescent
318f433f22 Reduced log spam when checking for avatars. 2022-01-17 20:04:54 +00:00
CalDescent
cfc80cb9b0 Use a header instead of a meta tag for Content-Security-Policy, because we can't guarantee that we are parsing all HTML files. 
Also use default-src instead of connect-src, as we want to block all external requests.
 2022-01-17 20:04:35 +00:00
CalDescent
01c6149422 Restrict websites to same origin requests only, using a Content-Security-Policy meta tag. 2022-01-16 20:52:30 +00:00
CalDescent
6f80a6c08a Rework of file list requests and relays, allowing it to handle multiple chunk resources in a much more sensible way. 
This could create a lot of additional relay traffic as a result, so needs lots of testing and possibly optimizing.
 2022-01-16 20:39:37 +00:00
CalDescent
5018d27c25 "Not started" renamed to "Published" 2022-01-15 20:21:52 +00:00
CalDescent
1d77101253 Use AES/CBC/PKCS5Padding for encryption, and fall back to just AES for legacy resource support. 
Should fix "ECB mode cannot use IV" error due to mode and padding not being stated.
 2022-01-15 20:14:32 +00:00
CalDescent
119c1b43be Use default values for method and compression if not specified. 
Should fix issue with v4 transactions where these aren't used. Matches with the NOT NULL DEFAULT 0 which automatically transitions existing v4 ARBITRARY transactions to use the same defaults.
 2022-01-13 19:09:00 +00:00
CalDescent
6761b91400 QDN go-live timestamp set to Fri, 14 Jan 2022 16:00:00 UTC 2022-01-12 20:53:57 +00:00
CalDescent
e2b13791bb Don't reload the log4j2.properties file as this seems to be buggy in a lot of cases. 2022-01-12 20:26:41 +00:00
CalDescent
f44c21ce59 Disallow any kind of website/app/data rendering when localAuthBypassEnabled is enabled. 
This allows node operators to return their authentication to the legacy rules (local requests allowed), without introducing javascript vulnerabilities. The websites, apps, etc are just prevented from loading, to avoid the risk of any API calls from javascript.
 2022-01-12 19:32:52 +00:00
CalDescent
ade977e416 Don't broadcast any arbitrary signatures if the list is empty (i.e. the node isn't yet hosting anything) 2022-01-12 19:23:36 +00:00
CalDescent
f09a131bd6 Added requestHops to log entry. 2022-01-12 19:21:11 +00:00
CalDescent
4815587de1 Use V2 of string serialization methods in ArbitrarySignaturesMessage, as it is designed to allow null values. 2022-01-12 19:20:49 +00:00
CalDescent
e0ebfb9b53 Reduced log spam. 2022-01-12 19:20:14 +00:00
CalDescent
90836afd91 External IP address updates now require 10 consecutive readings. 2022-01-12 19:19:56 +00:00
CalDescent
4e1b0a25bb Fixed arbitrary peer tests, which used a local address (but we now treat them as invalid). 2022-01-11 20:06:14 +00:00
CalDescent
89c3236bf5 Updated bundled log4j2.properties files 2022-01-11 20:02:42 +00:00
CalDescent
7658bc2025 Added X-API-KEY header field to API documentation endpoints that require it. 2022-01-11 19:13:56 +00:00
CalDescent
ccde725d3b Check for an empty string as well as null in a couple of places, so that deserializeSizedString() can be safely used. 2022-01-10 19:42:21 +00:00
CalDescent
e3b45cac0a Use an alternative version of Serialization.serializeSizedString() and Serialization.deserializeSizedString() for the new ARBITRARY transaction additions. 
The modifications made to these methods were causing issues with other transaction types that were expecting blank strings instead of null. To keep risk to a minimum, I have split into two different sets of functions until there is more time to unify them.
 2022-01-10 19:41:02 +00:00
CalDescent
8f8a500dcd Fixed some issues left over from the qortaldata project 2022-01-10 19:26:25 +00:00
CalDescent
f9749cd82c Merge remote-tracking branch 'qortal-data/master' into qdn 2022-01-09 21:10:48 +00:00
CalDescent
051052fdd2 Removed authentication for GET /peers/summary endpoint 2022-01-09 21:09:20 +00:00
CalDescent
940304b4c2 Added GET /admin/apikey/test endpoint, so that we have a dedicated place to test if authentication works. 2022-01-09 20:08:45 +00:00
CalDescent
b4d2fae27f Fixed a couple of FOLLOWED_AND_VIEWED references that were missed 2022-01-09 16:26:52 +00:00
CalDescent
11e194292c Removed API key requirement from GET /admin/status and GET /admin/mintingaccounts 2022-01-09 16:26:23 +00:00
CalDescent
5ba6f6f53e FOLLOWED_AND_VIEWED renamed to FOLLOWED_OR_VIEWED, since it's technically an OR not an AND. 2022-01-09 13:25:49 +00:00
CalDescent
f58a16905f Removed unused setting. 2022-01-09 13:19:07 +00:00
CalDescent
33e82b336b Limit arbitrary signature requests to 3 hops, just in case a bug caused any kind of circular broadcasting. 2022-01-09 11:22:27 +00:00
CalDescent
0ced712974 Merge remote-tracking branch 'qortal-data/master' into qdn 2022-01-08 12:29:48 +00:00
CalDescent
db8e35cc13 Allow a new API key to be generated if the existing apikey.txt file has been deleted 2022-01-08 12:27:24 +00:00
CalDescent
b6db5aa2d3 Use "apikey.txt" instead of "apikey" as the filename to store the api key, to make it easier for users to open. 2022-01-08 10:22:14 +00:00
CalDescent
396dc5c9b0 Always log "Synchronizing with peer..." as it may help give more clarity to those with sync issues. 2022-01-08 10:12:54 +00:00
CalDescent
67e424a32a Added GET /arbitrary/relaymode API endpoint, which returns whether relay mode is enabled in the settings or not. 2022-01-07 14:38:05 +00:00
CalDescent
d8cbec41d2 Various logging improvements and fixes. 2022-01-07 14:08:11 +00:00
CalDescent
374f6b8d52 Added restrictions when relaying file list requests 
1) Each relay request expires after 5 seconds, after which nodes will stop relaying it, preventing any kind of infinite loop. So it has to reach the destination peer within 5 seconds. This should be fine, because the original peer's request would timeout anyway, so there's nothing to be gained by continuing to relay it.

2) Each relay request stops being forwarded after 3 "hops" - i.e. once it has been relayed through 3 different peers, it will no longer be transmitted any further. If we assume that each node has 16 connections, that allows it to reach a theoretical maximum of 4096 peers in 3 hops. In practice it will be less, and may not reach everyone due to peer "islands". But it will automatically retry a few times on a timer, so should hopefully find what it needs eventually. Plus, it still has the ability to make a direct connection to anyone hosting the data, as long as they are port forwarded.
 2022-01-07 14:01:57 +00:00
CalDescent
20ec4cbd14 ARBITRARY_REQUEST_TIMEOUT increased from 6 to 10 seconds 
This is likely longer than needed, but it's best to allow extra for now and then optimize the timeouts once we've had some experience with real world data.
 2022-01-07 12:50:34 +00:00
CalDescent
1c80835f49 Default relayModeEnabled to true. 
Even though a final decision is yet to be made, it makes sense to test with this scenario to ensure that everything works correctly.
 2022-01-07 12:31:16 +00:00
CalDescent
5e0af26c27 Keep track of successes or failures for a particular arbitrary peer / signature combination. 
This can help to inform decisions on data retention (although there is no deletion yet).
 2022-01-05 21:23:29 +00:00
CalDescent
b42674ac06 Small code reorganization to improve logic when adding arbitrary peer data 2022-01-05 19:26:06 +00:00
CalDescent
3394543705 Don't save arbitrary peer data if it's a local address 2022-01-05 19:22:24 +00:00
CalDescent
75c51aa61b If a direct connection can't be made to a peer over the original or default port, match the peer's host with any entries in knownPeers and try connecting to each of those until one succeeds. 2022-01-05 19:03:41 +00:00
CalDescent
6041722250 Added missing import. 2022-01-05 18:35:03 +00:00
CalDescent
60d038b367 Return result of Network.connectPeer() back to caller. 2022-01-05 18:34:54 +00:00
CalDescent
b2c4bf96af Rebroadcast the entire list of hosted signatures each time our external IP address changes. 2022-01-03 19:52:42 +00:00
CalDescent
f007f9a86d Added optional "senderPeerAddress" string to HELLO messages, to allow external IP changes to be detected without using a centralized service. 2022-01-03 19:05:10 +00:00
CalDescent
b1c1634950 Updated log4j to 2.17.1 
This involves modifying the log4j2.properties file on node startup to fix an incompatibility with ${dirname:-}. Thanks to AlphaX Projects for tracking down this incompatibility.
 2022-01-02 20:50:38 +00:00
CalDescent
5157ccf7c0 Expire each authorization after 60 minutes. 2022-01-02 20:11:38 +00:00
CalDescent
c4a782301d Delete reader cache directories if a resource is unable to be built. 2022-01-02 18:31:17 +00:00
CalDescent
17fe94fa46 Added POST /render/authorize/{resourceId} endpoint 
This allows an entire registered name to be preauthorized, therefore allowing for instance a website to automatically request other resources from the same author, such as videos.
 2022-01-02 18:23:27 +00:00
CalDescent
75d9347d23 Show blank root page for the gateway. 
This can ultimately be replaced with a website list / search engine.
 2022-01-02 17:07:48 +00:00
CalDescent
ef784124f3 Fixed status bugs in loading screen. 2022-01-02 16:11:25 +00:00
CalDescent
bd1b631914 Use <base href="..."> in HTMLParser, rather than attempting to swap out every relative link 
This delegates the task to the browser rather than doing it in java. It should also catch a few remaining types of links that we had missed - e.g. ones that originate from within js files.
 2022-01-02 15:22:53 +00:00
CalDescent
edfc8cfdc4 Fixed bug which saved an incorrect peer address for a forwarded arbitrary signatures message. 2022-01-01 22:34:02 +00:00
CalDescent
fbe34015d4 Validate peer addresses before saving anything to the db. 2022-01-01 22:33:01 +00:00
CalDescent
391fa008d0 When making a direct connection to a peer, try using the default listen port along with the one specified in the ArbitraryPeers table. 
This is a workaround to account for any ephemeral ports that may have made it into the dataset.
 2022-01-01 21:37:23 +00:00
CalDescent
7df8381b8f Don't allow a row to be added to ArbitraryPeers (or the message to be rebroadcast) if an entry already exists with the same hash and host/ip. 
This avoids duplicate entries from the same host/ip with differing ports. This can occur due to some requests using ephemeral port numbers. Ideally we would filter these out altogether, but this at least acts as a safety net to prevent a very cluttered db and associated "broadcast storm". The main tradeoff here is that multiple nodes on the same IP address will be recorded as a single entry. This doesn't seem like it will be a major limitation, because one of them will remain available.
 2022-01-01 21:09:48 +00:00
CalDescent
c0234ae328 Added unit test to make sure the mempow nonce is being validated in ARBITRARY transactions. 2022-01-01 18:12:18 +00:00
CalDescent
5c64a85d7c Test a registered name with a space in it 2022-01-01 15:43:00 +00:00
CalDescent
7aa8f115ce Added "original copy indicator file", which prevents the node from deleting its own published content when storage space runs out. 
Since some files won't have any mirrors, this prevents the cleanup manager from deleting the only copy in existence when freeing up space. This feature can be disabled by setting "originalCopyIndicatorFileEnabled": false in settings.json (or by deleting the ".original" files). The trade off is that the only copy in existence could be deleted if space gets low.

This will also allow for better reporting of own vs third party files in the local UI (not yet implemented).
 2022-01-01 14:52:09 +00:00
CalDescent
cf2c8d6c67 Switched some IOExceptions to DataExceptions 2022-01-01 14:02:54 +00:00
CalDescent
37edebcad9 Fixed cleanup bug which could cause problems for ArbitraryDataWriter 2021-12-30 15:50:14 +00:00