AlphaX/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Issues Pull Requests Projects Releases Wiki Activity
2,312 Commits 77 Branches 112 Tags 430 MiB
c65a63fc7e
Commit Graph

2065 Commits

Author SHA1 Message Date
CalDescent
fc82f0b622 Use 5 builder threads, so that one slow resource (e.g. a thumbnail) doesn't hold up the other queued build items. 
This can be replaced with a task-based approach longer term.
 2022-02-11 13:58:45 +00:00
CalDescent
c0c50f2e18 Updated bootstrap hosts 2022-02-11 13:33:25 +00:00
CalDescent
9332d7207e Fixed bug in cache clearing logic, which was often preventing resource updates from being detected. 2022-02-10 09:22:54 +00:00
CalDescent
a8c79b807b Discard any uncommitted changes as a result of the higher weight chain detection 2022-02-10 08:16:30 +00:00
CalDescent
2637311ef5 Prevent potential ConcurrentModificationException in the build queue 2022-02-09 20:20:30 +00:00
CalDescent
06b5b8f793 Reduced time between processing build tasks, to prevent builds with invalid criteria from holding up legitimate builds too much. 2022-02-09 20:17:56 +00:00
CalDescent
61f58173cb Revert "Removed transaction caching. Can be reintroduced later." 
This reverts commit 9804eccbf0.
 2022-02-09 19:46:20 +00:00
CalDescent
b7b66f6cba Revert "Removed getWalletTransactions() synchronization. Again, can be re-added later." 
This reverts commit 70c864bc2f.
 2022-02-09 19:46:16 +00:00
CalDescent
dda2316884 Revert "Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance." 
This reverts commit d7658ee9f9.
 2022-02-09 19:46:10 +00:00
CalDescent
b782679d1f Revert "Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj."" 
This reverts commit 214f49e356.
 2022-02-09 19:46:06 +00:00
CalDescent
b0f19f8f70 Merge branch 'block-minter-updates' 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java
 2022-02-09 19:42:39 +00:00
CalDescent
de5f31ac58 Don't process file hashes if we're stopping 2022-02-09 19:40:20 +00:00
CalDescent
214f49e356 Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj." 
This reverts commit 892612c084.

# Conflicts:
#	src/main/java/org/qortal/crosschain/Bitcoiny.java
 2022-02-08 18:29:32 +00:00
CalDescent
d7658ee9f9 Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-08 18:27:44 +00:00
CalDescent
70c864bc2f Removed getWalletTransactions() synchronization. Again, can be re-added later. 2022-02-08 18:27:08 +00:00
CalDescent
9804eccbf0 Removed transaction caching. Can be reintroduced later. 2022-02-08 18:26:15 +00:00
CalDescent
d1f24d45da Added defensiveness in convertToSimpleTransaction() 2022-02-08 18:24:42 +00:00
CalDescent
9630625449 Rework of processIncomingTransactionsQueue() so that it no longer holds the lock while processing. 
This should fix an issue where network threads could be blocked when new transactions arrived, due to waiting for the incomingTransactions lock to free up.
 2022-02-08 09:18:14 +00:00
CalDescent
b72153f62b Renamed main thread from "Controller" to "Qortal" 2022-02-08 09:02:20 +00:00
CalDescent
0a88a0c95e Perform the base58 decoding outside of the arbitraryDataFileHashResponses lock, to reduce the amount of waiting around by other threads. 2022-02-08 08:45:58 +00:00
CalDescent
ab4ba9bb17 Don't re-fetch unconfirmed transactions that are already in the queue 2022-02-08 08:36:45 +00:00
CalDescent
a49218a840 Optimized ArbitraryDataFileRequestThread - only start a database transaction when there's something to process. 2022-02-07 22:06:45 +00:00
CalDescent
b6d633ab24 Break out of incoming transactions processing loop if we need to sync. 2022-02-07 22:05:13 +00:00
CalDescent
133943cd4e Reduce log spam 2022-02-07 22:03:41 +00:00
CalDescent
f8ffb1a179 Updated thread names 2022-02-07 22:03:26 +00:00
CalDescent
41c4e0c83e Merge branch 'master' into block-minter-updates 2022-02-06 18:40:32 +00:00
CalDescent
99f6bb5ac6 Reorganized some controller methods. 2022-02-06 18:32:43 +00:00
CalDescent
3e0306f646 Increased minPeerConnectionTime and maxPeerConnectionTime to reduce the chances of forced connections during relays. 
An alternate option would be to avoid force disconnecting while relays are in progress, but some nodes could have active relays 100% of the time and therefore would never recycle their peers. So it is simpler to just increase the average peer connection time for everyone.
 2022-02-06 17:29:00 +00:00
CalDescent
84e4f9a1c1 Rework of arbitraryRelayMap to keep track of multiple responses. 
Previously, only one peer's response for a hash would be remembered, even if multiple others reported back too. This would cause useful mapping to be lost.
 2022-02-06 17:20:01 +00:00
CalDescent
cd5ce6dd5e Don't remove from the relay map after a file is requested, as it may be needed by other peers. 
It will be cleaned up automatically after 60 seconds, so it is best to keep the data intact until then.
 2022-02-06 16:04:54 +00:00
CalDescent
9ec4e24ef6 Slightly optimized logic in fetchArbitraryDataFiles() 2022-02-06 15:45:40 +00:00
CalDescent
fa447ccded Builder thread updates. 2022-02-06 15:37:21 +00:00
CalDescent
ef838627c4 Stop asking for hashes from a peer if one fails. 
This fixes the request looping that occurs on when a peer is unable to serve files.
 2022-02-06 15:37:08 +00:00
CalDescent
b8aaf14cdc Introduced ArbitraryDataFileRequestThread to allow for multiple concurrent file requests. 
This is likely a short term solution (to allow existing code to be repurposed) until replaced with a task-based approach, as this will allow for a much greater number of threads.
 2022-02-06 15:34:06 +00:00
CalDescent
2740543abf Added "async" and "attempts" parameters to GET /arbitrary/{service}/{name}* endpoints. 
async = fail immediately with 404 if missing, and request in the background
attempts = the number of times to request the data (synchronous mode only for now)
 2022-02-06 13:04:58 +00:00
CalDescent
3c526db52e Fixed bug in build manager which would prevent future builds until the core was restarted. 2022-02-06 13:03:01 +00:00
CalDescent
cfe0414d96 Small rework of invalidUnconfirmedTransactions to specify the expiry time instead of the time added. 
This allows TIMESTAMP_TOO_OLD transactions to be tracked for a shorter time (10 minutes) than the other invalid transactions (60 minutes). Should reduce network traffic and db load around the time that transactions are expiring, as there is a lag before they are noticed and removed from each node. Due to the variance, it could cause other peers to request them again after deleting. They are now ignored for 10 minutes to avoid request spam.
 2022-02-06 12:35:41 +00:00
CalDescent
08e06ba11a Fixed bugs preventing invalidUnconfirmedTransactions from working as intended. 2022-02-06 12:09:44 +00:00
CalDescent
8c03164ea5 Don't add expired transactions to invalidUnconfirmedTransactions, as there is no need to keep track of these. 2022-02-06 11:55:07 +00:00
CalDescent
0fe2f226bc Added invalidUnconfirmedTransactions map 
An incoming invalid unconfirmed transaction will be added to this map if its timestamp is more than 30 minutes old. This should allow enough time and opportunities for it to be imported and included in a block (allowing for re-orgs which could switch its status from invalid to valid).

Once added, it will be removed after an hour to allow for another chance to be requested from any peers that still have it. If invalid again, it's added back to the map for another hour.

This fixes a 24 hour long loop, where invalid transactions are requested over and over from peers that have already imported them. It could be improved further by periodically removing invalid unconfirmed transactions from the database, but this will be a higher risk.

The results of this feature should be less network traffic, and less blockchain locks (which should ultimately increase the responsiveness of the synchronizer).
 2022-02-06 11:23:28 +00:00
CalDescent
55b5702158 Invalidate last low weight block signature whenever the previous block data changes. 2022-02-05 17:54:36 +00:00
CalDescent
a4cbbb3868 Moved block minter sleep to later in the process, otherwise it can remain there for longer than expected. 2022-02-05 17:54:36 +00:00
CalDescent
816b01c1fc Fixed issue in rebase 2022-02-05 17:54:36 +00:00
CalDescent
483e7549f8 Revert "Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before." 
This reverts commit e2e87766fa.
 2022-02-05 17:54:35 +00:00
CalDescent
60d71863dc Allow 3 seconds for the synchronizer to obtain a blockchain lock, to reduce missed attempts. 2022-02-05 17:54:35 +00:00
CalDescent
170244e679 More work on higher weight chain detection in the block minter. 
Added 30 second timeout, so that any errors should self correct.
 2022-02-05 17:54:35 +00:00
CalDescent
472e1da792 Added debug level logging in higherWeightChainExists() for better visibility. 2022-02-05 17:54:35 +00:00
CalDescent
cbf03d58c8 Made synchronizer method public as it is now also used by the block minter. 2022-02-05 17:54:35 +00:00
CalDescent
ba41d84af9 Initial attempt to avoid an unnecessary block submission if one of our peers already has a higher weight chain. 2022-02-05 17:54:35 +00:00
CalDescent
98831a9449 Break out of the various loops in the cleanup manager if the thread is stopping. 2022-02-05 17:53:49 +00:00
CalDescent
9692539a3f Don't include fee in balance calculation (it looks like it could be double counting at the moment). 2022-02-05 17:24:33 +00:00
CalDescent
76df332b57 Check for null IP address before notifying of an external IP update. 2022-02-05 11:51:54 +00:00
CalDescent
c6405340bc minAccountLevelForBlockSubmissions reduced from 6 to 5 2022-02-05 11:33:28 +00:00
CalDescent
775e3c065e Invalidate ElectrumX transactions cache when switching accounts. 2022-02-05 10:23:25 +00:00
CalDescent
8937b3ec86 Don't allow duplicate transaction in the incoming transactions queue. 
This should reduce database load slightly, as it won't have to check the same transaction multiple times in each batch.
 2022-02-05 10:19:26 +00:00
CalDescent
3fbb86fded Added indexes, to make looking up name transactions by name around 5x faster. 2022-02-04 16:27:31 +00:00
CalDescent
0cf2f7f254 Missing import from last commit 2022-02-04 16:18:00 +00:00
CalDescent
9e571b87e8 Yet another rewrite of fetchAllTransactionsInvolvingName() - this time over 1000x faster since it doesn't involve joining the Transactions table. 2022-02-04 16:17:31 +00:00
CalDescent
23bafb6233 Removed unused methods 2022-02-04 14:00:44 +00:00
CalDescent
6dec65c5d9 Rewrite of fetchAllTransactionsInvolvingName() to avoid having to load all name transactions into memory. 2022-02-04 13:58:05 +00:00
CalDescent
4e59eb8958 Added unit test to simulate the false association between previous a UPDATE_NAME transaction, and the emoji name with a blank reducedName. 2022-02-04 12:51:22 +00:00
CalDescent
756d5e685a Added naming tests for blank new names 2022-02-04 12:50:05 +00:00
CalDescent
f52530b848 More thorough approach to fetchAllTransactionsInvolvingName(), to fix an issue found in unit testing. 2022-02-04 11:58:43 +00:00
CalDescent
c2bf37b878 Added transactionV5Timestamp featureTrigger to unit tests 2022-02-04 11:37:41 +00:00
CalDescent
98a2dd04b8 Fixed bug caused by improper handling of UPDATE_NAME transactions, similar to commit d16663f. 2022-02-04 10:28:26 +00:00
CalDescent
694ea689c8 Synchronize the loop and break out of it before fetching arbitrary data files. Hopeful fix for ConcurrentModificationException, and maybe a potential deadlock. 2022-02-03 21:14:41 +00:00
CalDescent
618aaaf243 Removed logs used for debugging only 2022-02-03 21:06:36 +00:00
CalDescent
9224ffbf73 Cache transaction list for 2 minutes, and synchronize, to prevent the balance and transactions APIs both requesting at once. 
This ensures that only a single round of requests (per coin) is used for the wallettransactions and balance APIs. It also speeds up loading on subsequent requests. The 2 minute cache isn't much longer than the foreign block times, so shouldn't cause values to be too out of date.
 2022-02-03 21:04:49 +00:00
CalDescent
892612c084 Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj. 
Hopeful fix for incorrect balances in wallets with large numbers of transactions. At the very least, this gives us control of the code that calculates the balance.
 2022-02-03 20:22:25 +00:00
CalDescent
077165b807 Modified fetchArbitraryDataFileList() to support requesting only the missing hashes, but it is not yet used. 
Once GetArbitraryDataFileListMessage is rolled out to the network we can uncomment this code. Needs testnet testing prior to that.
 2022-02-03 20:01:44 +00:00
CalDescent
7994fc6407 Rework of onNetworkGetArbitraryDataFileListMessage() to support custom hashes to be optionally supplied. 
Also simplified the existing logic, to make the code more readable.
 2022-02-03 19:58:50 +00:00
CalDescent
d98df3e47d Added support for file hashes to optionally be included in GetArbitraryDataFileListMessage. 
Needs testing on testnet, as the majority of nodes need to update before the hashes can be used.
 2022-02-03 19:55:22 +00:00
CalDescent
1064b1a08b Removed duplicate metadata file checks. 2022-02-03 19:53:29 +00:00
CalDescent
0b7a7ed0f1 Added some more debug logging relating to file responses. 2022-02-03 19:52:28 +00:00
CalDescent
114b1aac76 Added arbitrary data file manager thread, which will ensure that all file list responses are tried until we receive the files. 
Previously we would only try the first response and then discard the others due to being duplicates. They are now added to a queue and retried by the dedicated thread (up to the 60 second timeout).
 2022-02-03 19:51:02 +00:00
CalDescent
6d06953a0e Increased RELAY_REQUEST_MAX_HOPS from 3 to 4, in an attempt to reach more peers. 2022-02-02 22:31:40 +00:00
CalDescent
0430fc8a47 Give up immediately after a synchronization if we are shutting down the synchronizer 2022-02-02 09:18:46 +00:00
CalDescent
7338f5f985 Attempt to acquire a blockchain lock (for up to 5 seconds) before shutting down the repository. 
This should fix conflicts caused by the synchronizer and controller now being on separate threads. It may also reduce the chances of the database corrupting on shutdown, but this remains to be seen.
 2022-02-02 09:17:24 +00:00
CalDescent
640bcdd504 Shutdown/interrupt the synchronizer as early as possible 2022-02-02 09:13:49 +00:00
CalDescent
c9d5d996e5 Increased accuracy of the block weights in the synchronizer logs, as the extra precision is now needed for debugging. 2022-02-01 22:06:01 +00:00
CalDescent
710befec0c Increased ARBITRARY_RELAY_TIMEOUT from 30 to 60 seconds, so that relay peers remember their mappings for longer. 2022-02-01 22:04:53 +00:00
CalDescent
8ccb158241 Reduce log spam when in DEBUG mode. 2022-02-01 22:04:12 +00:00
CalDescent
97199d9b91 Display the local and total chunk counts on the loading screen. 2022-02-01 22:03:52 +00:00
CalDescent
5a8b895475 Fixed bug in loading screen, which prevented the DOWNLOADING status from showing. 2022-02-01 22:03:17 +00:00
CalDescent
6c9600cda0 Added API key header to GET /arbitrary/resource/status/* endpoints 2022-02-01 22:02:41 +00:00
CalDescent
82fa6a4fd8 Include "localChunkCount" and "totalChunkCount" in the GET /arbitrary/resource/status/* API responses. 
These values are left out of other API endpoints where multiple resources are returned, because calculating the chunk counts is too time consuming.
 2022-02-01 22:02:14 +00:00
CalDescent
45f2d7ab70 Revert "We (currently) can't filter unconfirmed transactions by address, because only the public key is stored in the database until it is confirmed (at which point there is an entry in the TransactionParticipants table which contains the address). Given that this isn't a simple problem to solve, for now it makes sense to reject this combination if requested via the /transactions/search API." 
This reverts commit 7aed0354f1.
 2022-02-01 21:55:41 +00:00
CalDescent
33731b969a Direct peer connections now send a file list request to the peer, rather than individually requesting every chunk for a transaction. 2022-02-01 09:04:31 +00:00
CalDescent
40a8cdc71f Improved logging when fetching data files 2022-01-31 22:35:12 +00:00
CalDescent
cbe83987d8 Fixed occasional ConcurrentModificationException in the block archive reader. 2022-01-31 21:39:49 +00:00
CalDescent
01e4bf3a77 Try to speed up the shutdown process of the cleanup manager. 2022-01-31 21:39:20 +00:00
CalDescent
b198a8ea07 Fixed issue in ArbitraryDataFile.chunkExists() due to it not checking for the metadata file. 2022-01-31 21:38:51 +00:00
CalDescent
e2e87766fa Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before. 2022-01-30 20:15:43 +00:00
CalDescent
f005a0975d Shut down the synchronizer as soon as the controller is stopping, if we are able to. 2022-01-30 20:14:11 +00:00
CalDescent
5700369935 Prioritize syncing over transaction importing. 2022-01-30 20:09:35 +00:00
CalDescent
8a1fb6fe4e If a lock can't be obtained when synchronizing, automatically request the sync again. 2022-01-30 20:09:09 +00:00
CalDescent
5b788dad2f Relocated all synchronizer code from the controller to the synchronizer, and also moved synchonization onto its own thread. 2022-01-30 20:01:22 +00:00
CalDescent
fa2bd40d5f Reduce log spam 2022-01-30 19:51:49 +00:00
CalDescent
074bfadb28 Don't bother locking if there are no new transactions to process 2022-01-30 19:50:09 +00:00
CalDescent
bd60c793be Incoming transactions are now added to a queue, and then processed soon after. 
This solves a problem where incoming transactions could rarely obtain a blockchain lock (due to multiple transactions arriving at once) and therefore most messages were thrown away. It was also causing constant blockchain locks to be acquired, which would often prevent the synchronizer from running.
 2022-01-30 19:03:31 +00:00