crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
2,611 Commits 85 Branches 118 Tags 530 MiB
9f9a74809e
Commit Graph

2338 Commits

Author SHA1 Message Date
CalDescent
8545a8bf0d Automatically fetch metadata for all resources that have it. 2022-03-05 13:00:49 +00:00
CalDescent
f0136a5018 Include the external port when responding ArbitraryDataFileListRequests 2022-03-05 13:00:17 +00:00
CalDescent
6697b3376b Direct peer connections now use the on-demand data retrieved from file list requests, rather than the stale and incomplete ArbitraryPeerData. 2022-03-05 12:59:13 +00:00
CalDescent
ea785f79b8 Removed unnecessary synchronization 2022-03-04 19:02:30 +00:00
CalDescent
0352a09de7 New online accounts are now verified on the OnlineAccountsManager thread rather than on network threads. This is an attempt to reduce the amount of blocked network threads due to signature verification, and is necessary for the upcoming mempow addition. 2022-03-04 17:58:06 +00:00
CalDescent
5b4f15ab2e Transaction importing code moved to TransactionImporter controller class 
As with online accounts, no logic changes other than moving transaction queue processing from the controller thread to its own dedicated thread.
 2022-03-04 16:47:21 +00:00
CalDescent
fd37c2b76b Moved all online accounts code to a new OnlineAccountsManager controller class 
There are no logic changes here other than moving performOnlineAccountsTasks() onto its own thread, so that it's not subject to anything that might be slowing down the main controller thread.
 2022-03-04 16:24:04 +00:00
CalDescent
924aa05681 Optimized peer lists 
- Removed synchronization from connectedPeers, and replaced it with an unmodifiableList.
- Added additional immuatable caches: handshakedPeers and outboundHandshakedPeers

This should greatly reduce the amount of time spent waiting around for access to the connectedPeers array, since it is now immediately accessible without needing to obtain a lock. It also removes calls to stream() which were consuming large amounts of CPU to constantly filter the connected peers down to a list of handshaked peers.

Thanks to @catbref for these great suggestions.
 2022-03-04 15:14:12 +00:00
CalDescent
84b42210f1 Use ArbitraryDataFileRequestThreads only - instead of reusing file list response threads. 2022-03-04 13:34:16 +00:00
CalDescent
941080c395 Rework of arbitraryDataFileHashResponses to use a list rather than a map (limited to 1000) items. Sort the list by routes with the lowest number of peer hops first, to try and prioritize those which are easiest and quickest to reach. 2022-03-04 13:33:17 +00:00
CalDescent
35d9a10cf4 Avoid logging if there are no remaining transaction signatures to validate. There was too much log spam, none of which was particularly useful. 2022-03-04 12:03:58 +00:00
CalDescent
7c181379b4 Added more granularity to logging, to differentiate between signature validation and general processing/importing, as well as showing counts of the transactions being processed in each round. 2022-03-04 11:12:23 +00:00
CalDescent
f9576d8afb Further optimizations to Controller.processIncomingTransactionsQueue() 
- Signature validation is now able to run concurrently with synchronization, to reduce the chances of the queue building up, and to speed up the propagation of new transactions. There's no need to break out of the loop - or avoid looping in the first place - since signatures can be validated without holding the blockchain lock.
- A blockchain lock isn't even attempted if a sync request is pending.
 2022-03-04 11:05:58 +00:00
CalDescent
6a8a113fa1
Merge pull request #74 from catbref/presence-txns-removal 
PRESENCE transactions changed to always fail signature validation
 2022-03-04 10:33:11 +00:00
CalDescent
ef59c34165 Added missing "break" which was causing additional unnecessary debug logging. Originally introduced due to a merge conflict with the metadata branch. 2022-03-04 10:28:44 +00:00
catbref
a9371f0a90 In Controller.processIncomingTransactionsQueue(), don't bother with 2nd-phase of locking blockchain and importing if there are no valid signature transactions to actually import 2022-03-03 20:32:27 +00:00
catbref
a7a94e49e8 PRESENCE transactions changed to always fail signature validation 2022-03-03 20:25:58 +00:00
catbref
affd100298 Reworking of Controller.processIncomingTransactionsQueue() 
Main changes are:
* Check transaction signature validity in initial round, without blockchain lock
* Convert List of incoming transactions to Map so we can record whether we have validated transaction signature before to save rechecking effort
* Add invalid signature transactions to invalidUnconfirmedTransactions map with INVALID_TRANSACTION_RECHECK_INTERVAL expiry (~60min)
* Other minor changes related to List->Map change and Java object synchronization
 2022-03-03 20:21:04 +00:00
CalDescent
69309c437e Tightened up the content security policy for non HTML files. 2022-03-01 20:36:34 +00:00
CalDescent
e392e4d344 Allow eval(), setTimeout(), etc, to enable various QDN sites to function correctly. The existing sandboxing should be locking this down enough already. Limited to .html and .htm files only. 2022-03-01 20:35:56 +00:00
CalDescent
bd53856927 Disabled auto fetching of metadata. To be re-enabled at a later date. 2022-03-01 20:26:09 +00:00
CalDescent
cbd1018ecf Allow identical data to be published if the metadata differs. 2022-03-01 20:22:47 +00:00
CalDescent
46606152eb /arbitrary/metadata/* endpoint now returns ArbitraryResourceMetadata rather than a raw JSON string. 2022-03-01 20:22:20 +00:00
CalDescent
e6f93e0a08 Added categoryName to ArbitraryResourceMetadata, along with the existing category ID 2022-03-01 20:19:08 +00:00
CalDescent
8d81f1822f Merge branch 'master' into qdn-metadata 
# Conflicts:
#	src/main/java/org/qortal/controller/Controller.java
#	src/main/java/org/qortal/network/message/Message.java
 2022-02-28 20:10:39 +00:00
catbref
590a8f52db Remove future work comment from Controller 2022-02-27 16:57:26 +00:00
catbref
ecac47d1bc Also notify TradePresenceWebsocket (using TradePresenceEvent) when bridging old PRESENCE txns 2022-02-27 16:56:17 +00:00
catbref
3b477ef637 Fix JAXB marshalling error (duplicate tradeAddress) in TradePresenceWebSocket. No need to send signature. Make sure publicKey is sent in Base58, not Base64. 2022-02-27 16:56:17 +00:00
catbref
e2ef5b2ef3 Missed change from last commit: incorrect logic in TradePresenceWebSocket 2022-02-27 16:56:17 +00:00
catbref
1d59feeb72 Created /websockets/crosschain/tradepresence to replace /websockets/presence 2022-02-27 16:55:30 +00:00
catbref
c53dd31765 Tidy up of trade presence timestamp generation & checking. Added tests. Renamed "online trades" to "trade presences" 2022-02-27 16:54:42 +00:00
catbref
4c02081992 Tidy up TradeBot presence logging. Decorate API endpoints /crosschain/tradeoffers and /crosschain/trade with presence expiry timestamps 2022-02-27 16:54:42 +00:00
catbref
cb57af3c53 Bugfixes to online trade sigs + bridging from PRESENCE transactions 2022-02-27 16:54:42 +00:00
catbref
01d810fc00 Initial effort at migrating PRESENCE transactions to dedicated network messages 2022-02-27 16:54:42 +00:00
CalDescent
8c2a9279ee Return metadata in various /arbitrary APIs if the "includemetadata" parameter is included. 
This is very inefficient and will soon be replaced with dedicated ArbitraryResources / ArbitraryMetadata tables. But this is acceptable in the short term, especially if limit and offset are used.
 2022-02-27 09:09:18 +00:00
CalDescent
0d65448f3d Request all metadata automatically. 2022-02-27 08:20:39 +00:00
CalDescent
9da2b3c11a Don't respond to file list requests with just the metadata file. 
We have the separate metadata protocol for this now.
 2022-02-27 07:28:11 +00:00
CalDescent
95400da977 Fixed typo in various tests (copy and paste error) 2022-02-26 22:10:55 +00:00
CalDescent
dc41dc4c69 Tags now use an array of strings, rather than a single string. 2022-02-26 22:09:07 +00:00
CalDescent
a5c11d4c23 Reduced "Ignoring hash list request" logs from DEBUG to TRACE 2022-02-26 16:10:44 +00:00
CalDescent
878394535e Improvements relating to fetching metadata 
- Rate limiter is disabled when using the API
- fetchArbitraryMetadata() returns the actual metadata content rather than a boolean
- Exceptions are thrown on certain errors, rather than returning null
 2022-02-26 16:10:26 +00:00
CalDescent
35dba27a55 Fixed issue due to not updating arbitraryMetadataRequests when receiving the metadata file. 2022-02-26 16:07:06 +00:00
CalDescent
f22ad13fa9 Merge branch 'master' into qdn-metadata 
This involved a slight rewrite to remove the "includeMetadataOnly" boolean. Metadata is now always excluded, otherwise it complicates the caching too much.

# Conflicts:
#	src/main/java/org/qortal/api/resource/ArbitraryResource.java
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java
 2022-02-26 14:39:20 +00:00
CalDescent
aa2e5cb87b Merge branch 'hosted-resources-search' 2022-02-26 14:05:52 +00:00
CalDescent
7740f3da7e Small formatting tweaks, for consistency with existing code. 2022-02-26 14:05:28 +00:00
CalDescent
badb576991 Fixed exception when identifier is null. Also handling null names as this may be a future scenario. 2022-02-26 14:04:35 +00:00
CalDescent
c65a63fc7e Fixed "query" parameter error in swagger documentation 2022-02-26 13:59:53 +00:00
CalDescent
0111747016 Added debug logging of new file list stats. 2022-02-25 13:30:07 +00:00
CalDescent
eac4b0d87b Maintain backwards support for pre-3.2.0 peers by only including new file list message params when sending to newer peers. 
These params are optional and the process will function without them, just less efficiently.
 2022-02-25 12:24:02 +00:00
CalDescent
3dadce4da4 Renamed a reference 2022-02-25 12:24:02 +00:00
CalDescent
1864468818 Prefer the route with the least number of hops when relaying. 2022-02-25 12:24:02 +00:00
CalDescent
1a59379162 Optionally include requestTime, requestHops, peerAddress, and isRelayPossible flag in ArbitraryDataFileListMessage 2022-02-25 12:24:02 +00:00
CalDescent
3cc394f02d
Merge pull request #70 from catbref/synchronizer-newchaintipevent 
Modify TradeBot to trigger when chain tip changes instead of with every new block
 2022-02-24 20:04:26 +00:00
CalDescent
53c4fe9e80 Fixed another ElectrumX issue found in unit tests. 
Peers that were thought to be missing output address data may actually have just been using a different key - "address" instead of "addresses". Now reading the addresses from both keys, which may remove the need for the previously added checks.
 2022-02-24 20:01:56 +00:00
CalDescent
d5521068b0 Fixed issue in earlier commit, found in unit tests. 2022-02-24 19:45:37 +00:00
CalDescent
a63ef4010d Disabled expired transaction data deletion code for now, as this was often causing data to be incorrectly deleted. 
This will need to be re-enabled at some point, but only after it's modified to be much less aggressive.
 2022-02-24 19:05:29 +00:00
catbref
8950bb7af9 Very slightly relax validity checks for TRANSFER_PRIVS to allow for skeletal account records, e.g. due to CHAT transactions, but account last reference still needs to be null. Example at block height 736196 / 7 2022-02-24 09:13:51 +00:00
catbref
9e6fe7ceb9 Modify TradeBot, some related websockets, to trigger when chain tip changes instead of with every new block 2022-02-24 09:06:21 +00:00
CalDescent
c333d18cd0 Merge branch 'segwit' 2022-02-23 20:07:07 +00:00
CalDescent
0271ef69c9 When submitting a new transaction, treat the chain as "synced" if the latest block is less than 30 minutes old. Increased from around 7.5 minutes. 2022-02-23 20:06:55 +00:00
CalDescent
2d493a4ea2 Added logging when no addresses are returned for a bitcoiny transaction output. 2022-02-23 09:29:16 +00:00
CalDescent
e339ab856f Skip over Electrum servers that don't return any output addresses. Hopeful fix for BTC transactions that report a zero value due to incomplete data being returned from certain ElectrumX peers. 2022-02-23 08:34:38 +00:00
proto
782904a971 improvement to the search on hosted resources 
1) use the cached version instead of rescanning all the files
2) separating the loading (which include files scanning) and listing logic
 2022-02-22 17:54:08 +01:00
proto
a3753c01bc Add search functionality to hosted resources 2022-02-22 15:50:46 +01:00
CalDescent
d5c3921846 Only show the red "synchronizing" systray icon if the latest block isn't recent. 
This should fix issue where the icon unnecessarily jumps between synced and synchronizing.
 2022-02-21 22:34:13 +00:00
CalDescent
a2c462b3da Add <meta charset="UTF-8"> tag to websites. Fixed issue rendering emojis and other special characters. 2022-02-21 22:28:59 +00:00
CalDescent
8673c7ef6e Fixed bug in GET /peers/summary API 2022-02-21 22:28:18 +00:00
CalDescent
8d7be7757f Fixed incorrectly named tag. 2022-02-21 22:27:44 +00:00
proto
6b83927048 Persist MintingAccounts.json on minting accounts add/remove 
this fix the behavior of the node, After adding or removing a minting account, allowing it to persist it to the backup folder
 2022-02-21 16:17:17 +01:00
proto
e07adbd60e online accounts api call, fix level zero accounts 
Added online zero level accounts to the response of /addresses/online/levels api endpoints
 2022-02-21 15:40:10 +01:00
CalDescent
7798b8dcdc Keep items in arbitraryDataFileHashResponses if they are currently being requested by another thread. This should help to locate the higher numbered chunks from larger resources. 2022-02-20 11:33:09 +00:00
CalDescent
146e7970bf Synchronize this.allKnownPeers and this.connectedPeers in Network.requestDataFromPeer(), to make the method thread-safe. 
This could be further improved by taking an immutable copy, but I'll leave this until the same approach is applied to other Network methods.
 2022-02-20 11:04:33 +00:00
CalDescent
f4f7cc58e3 Removed unused import. 2022-02-20 10:44:59 +00:00
CalDescent
21b4b494e7 Renamed method. 2022-02-20 10:44:38 +00:00
CalDescent
7307844bee If UPnP is disabled in settings, close the existing external listenPort if a UPnP rule exists. 2022-02-20 10:44:20 +00:00
CalDescent
5d419dd4ec Handle case where funds are sent to and from the same bitcoiny deterministic wallet. 2022-02-19 17:45:24 +00:00
CalDescent
6d0db7cc5e Catch UncheckedIOException in findAllHostedPaths() which was seen when a file was deleted by another thread. 2022-02-19 17:18:51 +00:00
CalDescent
8de606588c Attempt to open the listen port (default 12392) using UPnP, if the local network supports it. 2022-02-18 20:11:00 +00:00
CalDescent
35b0a85818 Increased WALLET_KEY_LOOKAHEAD_INCREMENT_BITCOINJ to 50 2022-02-18 18:42:06 +00:00
CalDescent
fcdd85af6c Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-18 18:39:47 +00:00
CalDescent
5aac2dc9df ONLINE_ACCOUNTS_V2_PEER_VERSION set to 3.2.0 2022-02-18 18:34:00 +00:00
CalDescent
becb0b37e6
Merge branch 'master' into network-online-accounts-v2 2022-02-18 17:12:44 +00:00
CalDescent
67ca876567 Log if we're unable to process the received file. 2022-02-18 15:35:37 +00:00
CalDescent
464ce66fd5 Moved deletion retry code into ArbitraryDataFile 2022-02-18 15:09:50 +00:00
CalDescent
3e505481fe Default minPeerVersion also increased to 3.1.0 2022-02-18 14:50:46 +00:00
CalDescent
c90c3a183e Block peers below 3.1.0 2022-02-18 14:50:29 +00:00
CalDescent
85b3278c8a Don't throttle the arbitrary data file request threads when there are items to process. 2022-02-15 19:39:26 +00:00
CalDescent
c90c287601 Increased ARBITRARY_REQUEST_TIMEOUT from 10 to 12 seconds, as some were coming back around 9-10 seconds later. 2022-02-15 19:38:30 +00:00
CalDescent
6ee395ed12 Stop bulk arbitrary signature broadcasts, as they were creating a lot of network traffic, and are in the process of being replaced with a better method. 2022-02-15 19:10:20 +00:00
CalDescent
6275ac2b81 Increased numberOfAdditionalBatchesToSearch from 5 to 7. 
This is the equivalent of increasing the max address gap from 15 to 21. The electrum standalone wallet uses 20, so this should be the most we will ever need.
 2022-02-14 22:58:37 +00:00
CalDescent
fd0a6ec71f Fix for invalid balance (and transaction amount) when there are no outputs relating to this wallet. 2022-02-14 22:53:30 +00:00
CalDescent
43791f00aa Wait 2 minutes on node startup before trying to fetch followed QDN data. 2022-02-14 19:33:58 +00:00
CalDescent
538ac30b4e Request only the missing hashes, not all of them. 2022-02-14 19:33:36 +00:00
CalDescent
acddf36467 Handle missing includeMetadata parameter. 2022-02-13 19:27:12 +00:00
CalDescent
166d32032a Fixed message IDs. 2022-02-13 19:22:20 +00:00
CalDescent
e4238a62c9 Exclude metadata-only transactions in the data management page (but added an API parameter to allow them to optionally be included). 
This ensures that the list will only show resources where there is at least 1 chunk.
 2022-02-13 19:21:16 +00:00
CalDescent
ad9c466712 Fall back to UNCATEGORIZED if the parsed category doesn't match any available categories. 
This allows for deletion of categories, as the resources will just move into UNCATEGORIZED until they are next updated.
 2022-02-13 18:10:56 +00:00
CalDescent
a3d31bbaf1 Category updates based on feedback so far. 2022-02-13 17:56:47 +00:00
CalDescent
4821139501 Merge branch 'master' into qdn-metadata 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java
 2022-02-13 15:50:12 +00:00
CalDescent
83213800b9 Use the timestamp from the registerNameTransactionData in unit tests, rather than the current time. 2022-02-13 15:05:28 +00:00
CalDescent
265ae19591 Fixed other failing tests due to increased REGISTER_NAME transaction fee. At some point we should determine the correct fee inside of generateBase(), but setting it explicitly adds confidence in testing for now. 2022-02-13 14:31:21 +00:00
CalDescent
c1598d20b5 Name registration fee increase timestamp set to Sunday, 20 February 2022 16:00:00 UTC 2022-02-13 13:47:00 +00:00
CalDescent
0712259057 Implemented REGISTER_NAME transaction fee increase from 0.001 to 5 QORT (average value based on community vote). 2022-02-13 13:45:48 +00:00
CalDescent
ea42a5617f Fixed ElectrumX log spam and errors 2022-02-13 10:58:45 +00:00
CalDescent
58a690e2c3 Route through new getAddressTransactions() wrapper. 2022-02-11 18:15:27 +00:00
CalDescent
3ae2f0086e Removed unusably slow electrum peer 2022-02-11 18:13:45 +00:00
CalDescent
19c83cc54d MAX_AVG_RESPONSE_TIME reduced to 500, as one peer regularly takes around 600ms to reply. 2022-02-11 18:12:34 +00:00
CalDescent
8ac298e07d Allow 3 retries for getTransaction() and getAddressTransactions() requests 2022-02-11 18:11:00 +00:00
CalDescent
9b43e4ea3d Time electrum requests, and move on to another server if one takes more than 1000ms on average to respond (measured over the last 5 requests). 2022-02-11 18:02:56 +00:00
CalDescent
dbacfb964b Increased TX_CACHE_SIZE from 200 to 1000, to speed up loading times on large wallets. 2022-02-11 16:55:29 +00:00
CalDescent
a664a6a790 Added more LTC Electrum peers from https://1209k.com/bitcoin-eye/ele.php?chain=ltc 2022-02-11 16:44:34 +00:00
CalDescent
ee1f072056 Improvement to last commit, so that caller class names are preserved. 2022-02-11 15:34:31 +00:00
CalDescent
a6aabaa7f0 Reduce build queue log spam by only logging high priority items (5 and above). 2022-02-11 15:28:41 +00:00
CalDescent
49b307db60 Treat a null priority as 0 2022-02-11 15:17:02 +00:00
CalDescent
f7341cd9ab Increased /arbitrary priority to 1 2022-02-11 15:13:53 +00:00
CalDescent
6932fb9935 Added "priority" property to build queue items. 
/render APIs use priority 10, whereas /arbitrary use priority 0, to prevent thumbnail downloads from holding up website loading. The priorities can be adjusted later, with maybe some service types being given higher priority than others.
 2022-02-11 15:08:12 +00:00
CalDescent
2343e739d1 Handle case where a file cannot be unzipped. 2022-02-11 14:35:46 +00:00
CalDescent
fc82f0b622 Use 5 builder threads, so that one slow resource (e.g. a thumbnail) doesn't hold up the other queued build items. 
This can be replaced with a task-based approach longer term.
 2022-02-11 13:58:45 +00:00
CalDescent
c0c50f2e18 Updated bootstrap hosts 2022-02-11 13:33:25 +00:00
CalDescent
9332d7207e Fixed bug in cache clearing logic, which was often preventing resource updates from being detected. 2022-02-10 09:22:54 +00:00
CalDescent
a8c79b807b Discard any uncommitted changes as a result of the higher weight chain detection 2022-02-10 08:16:30 +00:00
CalDescent
2637311ef5 Prevent potential ConcurrentModificationException in the build queue 2022-02-09 20:20:30 +00:00
CalDescent
06b5b8f793 Reduced time between processing build tasks, to prevent builds with invalid criteria from holding up legitimate builds too much. 2022-02-09 20:17:56 +00:00
CalDescent
61f58173cb Revert "Removed transaction caching. Can be reintroduced later." 
This reverts commit 9804eccbf0.
 2022-02-09 19:46:20 +00:00
CalDescent
b7b66f6cba Revert "Removed getWalletTransactions() synchronization. Again, can be re-added later." 
This reverts commit 70c864bc2f.
 2022-02-09 19:46:16 +00:00
CalDescent
dda2316884 Revert "Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance." 
This reverts commit d7658ee9f9.
 2022-02-09 19:46:10 +00:00
CalDescent
b782679d1f Revert "Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj."" 
This reverts commit 214f49e356.
 2022-02-09 19:46:06 +00:00
CalDescent
b0f19f8f70 Merge branch 'block-minter-updates' 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java
 2022-02-09 19:42:39 +00:00
CalDescent
de5f31ac58 Don't process file hashes if we're stopping 2022-02-09 19:40:20 +00:00
CalDescent
214f49e356 Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj." 
This reverts commit 892612c084.

# Conflicts:
#	src/main/java/org/qortal/crosschain/Bitcoiny.java
 2022-02-08 18:29:32 +00:00
CalDescent
d7658ee9f9 Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-08 18:27:44 +00:00
CalDescent
70c864bc2f Removed getWalletTransactions() synchronization. Again, can be re-added later. 2022-02-08 18:27:08 +00:00
CalDescent
9804eccbf0 Removed transaction caching. Can be reintroduced later. 2022-02-08 18:26:15 +00:00
CalDescent
d1f24d45da Added defensiveness in convertToSimpleTransaction() 2022-02-08 18:24:42 +00:00
CalDescent
9630625449 Rework of processIncomingTransactionsQueue() so that it no longer holds the lock while processing. 
This should fix an issue where network threads could be blocked when new transactions arrived, due to waiting for the incomingTransactions lock to free up.
 2022-02-08 09:18:14 +00:00
CalDescent
b72153f62b Renamed main thread from "Controller" to "Qortal" 2022-02-08 09:02:20 +00:00
CalDescent
0a88a0c95e Perform the base58 decoding outside of the arbitraryDataFileHashResponses lock, to reduce the amount of waiting around by other threads. 2022-02-08 08:45:58 +00:00
CalDescent
ab4ba9bb17 Don't re-fetch unconfirmed transactions that are already in the queue 2022-02-08 08:36:45 +00:00
CalDescent
a49218a840 Optimized ArbitraryDataFileRequestThread - only start a database transaction when there's something to process. 2022-02-07 22:06:45 +00:00
CalDescent
b6d633ab24 Break out of incoming transactions processing loop if we need to sync. 2022-02-07 22:05:13 +00:00
CalDescent
133943cd4e Reduce log spam 2022-02-07 22:03:41 +00:00
CalDescent
f8ffb1a179 Updated thread names 2022-02-07 22:03:26 +00:00
CalDescent
41c4e0c83e Merge branch 'master' into block-minter-updates 2022-02-06 18:40:32 +00:00
CalDescent
99f6bb5ac6 Reorganized some controller methods. 2022-02-06 18:32:43 +00:00
CalDescent
3e0306f646 Increased minPeerConnectionTime and maxPeerConnectionTime to reduce the chances of forced connections during relays. 
An alternate option would be to avoid force disconnecting while relays are in progress, but some nodes could have active relays 100% of the time and therefore would never recycle their peers. So it is simpler to just increase the average peer connection time for everyone.
 2022-02-06 17:29:00 +00:00
CalDescent
84e4f9a1c1 Rework of arbitraryRelayMap to keep track of multiple responses. 
Previously, only one peer's response for a hash would be remembered, even if multiple others reported back too. This would cause useful mapping to be lost.
 2022-02-06 17:20:01 +00:00
CalDescent
cd5ce6dd5e Don't remove from the relay map after a file is requested, as it may be needed by other peers. 
It will be cleaned up automatically after 60 seconds, so it is best to keep the data intact until then.
 2022-02-06 16:04:54 +00:00
CalDescent
9ec4e24ef6 Slightly optimized logic in fetchArbitraryDataFiles() 2022-02-06 15:45:40 +00:00
CalDescent
fa447ccded Builder thread updates. 2022-02-06 15:37:21 +00:00
CalDescent
ef838627c4 Stop asking for hashes from a peer if one fails. 
This fixes the request looping that occurs on when a peer is unable to serve files.
 2022-02-06 15:37:08 +00:00
CalDescent
b8aaf14cdc Introduced ArbitraryDataFileRequestThread to allow for multiple concurrent file requests. 
This is likely a short term solution (to allow existing code to be repurposed) until replaced with a task-based approach, as this will allow for a much greater number of threads.
 2022-02-06 15:34:06 +00:00
CalDescent
2740543abf Added "async" and "attempts" parameters to GET /arbitrary/{service}/{name}* endpoints. 
async = fail immediately with 404 if missing, and request in the background
attempts = the number of times to request the data (synchronous mode only for now)
 2022-02-06 13:04:58 +00:00
CalDescent
3c526db52e Fixed bug in build manager which would prevent future builds until the core was restarted. 2022-02-06 13:03:01 +00:00
CalDescent
cfe0414d96 Small rework of invalidUnconfirmedTransactions to specify the expiry time instead of the time added. 
This allows TIMESTAMP_TOO_OLD transactions to be tracked for a shorter time (10 minutes) than the other invalid transactions (60 minutes). Should reduce network traffic and db load around the time that transactions are expiring, as there is a lag before they are noticed and removed from each node. Due to the variance, it could cause other peers to request them again after deleting. They are now ignored for 10 minutes to avoid request spam.
 2022-02-06 12:35:41 +00:00
CalDescent
08e06ba11a Fixed bugs preventing invalidUnconfirmedTransactions from working as intended. 2022-02-06 12:09:44 +00:00
CalDescent
8c03164ea5 Don't add expired transactions to invalidUnconfirmedTransactions, as there is no need to keep track of these. 2022-02-06 11:55:07 +00:00
CalDescent
0fe2f226bc Added invalidUnconfirmedTransactions map 
An incoming invalid unconfirmed transaction will be added to this map if its timestamp is more than 30 minutes old. This should allow enough time and opportunities for it to be imported and included in a block (allowing for re-orgs which could switch its status from invalid to valid).

Once added, it will be removed after an hour to allow for another chance to be requested from any peers that still have it. If invalid again, it's added back to the map for another hour.

This fixes a 24 hour long loop, where invalid transactions are requested over and over from peers that have already imported them. It could be improved further by periodically removing invalid unconfirmed transactions from the database, but this will be a higher risk.

The results of this feature should be less network traffic, and less blockchain locks (which should ultimately increase the responsiveness of the synchronizer).
 2022-02-06 11:23:28 +00:00
CalDescent
55b5702158 Invalidate last low weight block signature whenever the previous block data changes. 2022-02-05 17:54:36 +00:00
CalDescent
a4cbbb3868 Moved block minter sleep to later in the process, otherwise it can remain there for longer than expected. 2022-02-05 17:54:36 +00:00
CalDescent
816b01c1fc Fixed issue in rebase 2022-02-05 17:54:36 +00:00
CalDescent
483e7549f8 Revert "Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before." 
This reverts commit e2e87766fa.
 2022-02-05 17:54:35 +00:00
CalDescent
60d71863dc Allow 3 seconds for the synchronizer to obtain a blockchain lock, to reduce missed attempts. 2022-02-05 17:54:35 +00:00
CalDescent
170244e679 More work on higher weight chain detection in the block minter. 
Added 30 second timeout, so that any errors should self correct.
 2022-02-05 17:54:35 +00:00
CalDescent
472e1da792 Added debug level logging in higherWeightChainExists() for better visibility. 2022-02-05 17:54:35 +00:00
CalDescent
cbf03d58c8 Made synchronizer method public as it is now also used by the block minter. 2022-02-05 17:54:35 +00:00
CalDescent
ba41d84af9 Initial attempt to avoid an unnecessary block submission if one of our peers already has a higher weight chain. 2022-02-05 17:54:35 +00:00
CalDescent
98831a9449 Break out of the various loops in the cleanup manager if the thread is stopping. 2022-02-05 17:53:49 +00:00
CalDescent
9692539a3f Don't include fee in balance calculation (it looks like it could be double counting at the moment). 2022-02-05 17:24:33 +00:00
CalDescent
76df332b57 Check for null IP address before notifying of an external IP update. 2022-02-05 11:51:54 +00:00
CalDescent
c6405340bc minAccountLevelForBlockSubmissions reduced from 6 to 5 2022-02-05 11:33:28 +00:00
CalDescent
775e3c065e Invalidate ElectrumX transactions cache when switching accounts. 2022-02-05 10:23:25 +00:00
CalDescent
8937b3ec86 Don't allow duplicate transaction in the incoming transactions queue. 
This should reduce database load slightly, as it won't have to check the same transaction multiple times in each batch.
 2022-02-05 10:19:26 +00:00
CalDescent
3fbb86fded Added indexes, to make looking up name transactions by name around 5x faster. 2022-02-04 16:27:31 +00:00
CalDescent
0cf2f7f254 Missing import from last commit 2022-02-04 16:18:00 +00:00
CalDescent
9e571b87e8 Yet another rewrite of fetchAllTransactionsInvolvingName() - this time over 1000x faster since it doesn't involve joining the Transactions table. 2022-02-04 16:17:31 +00:00
CalDescent
23bafb6233 Removed unused methods 2022-02-04 14:00:44 +00:00
CalDescent
6dec65c5d9 Rewrite of fetchAllTransactionsInvolvingName() to avoid having to load all name transactions into memory. 2022-02-04 13:58:05 +00:00
CalDescent
4e59eb8958 Added unit test to simulate the false association between previous a UPDATE_NAME transaction, and the emoji name with a blank reducedName. 2022-02-04 12:51:22 +00:00
CalDescent
756d5e685a Added naming tests for blank new names 2022-02-04 12:50:05 +00:00
CalDescent
f52530b848 More thorough approach to fetchAllTransactionsInvolvingName(), to fix an issue found in unit testing. 2022-02-04 11:58:43 +00:00
CalDescent
c2bf37b878 Added transactionV5Timestamp featureTrigger to unit tests 2022-02-04 11:37:41 +00:00
CalDescent
98a2dd04b8 Fixed bug caused by improper handling of UPDATE_NAME transactions, similar to commit d16663f. 2022-02-04 10:28:26 +00:00
CalDescent
694ea689c8 Synchronize the loop and break out of it before fetching arbitrary data files. Hopeful fix for ConcurrentModificationException, and maybe a potential deadlock. 2022-02-03 21:14:41 +00:00
CalDescent
618aaaf243 Removed logs used for debugging only 2022-02-03 21:06:36 +00:00
CalDescent
9224ffbf73 Cache transaction list for 2 minutes, and synchronize, to prevent the balance and transactions APIs both requesting at once. 
This ensures that only a single round of requests (per coin) is used for the wallettransactions and balance APIs. It also speeds up loading on subsequent requests. The 2 minute cache isn't much longer than the foreign block times, so shouldn't cause values to be too out of date.
 2022-02-03 21:04:49 +00:00
CalDescent
892612c084 Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj. 
Hopeful fix for incorrect balances in wallets with large numbers of transactions. At the very least, this gives us control of the code that calculates the balance.
 2022-02-03 20:22:25 +00:00
CalDescent
077165b807 Modified fetchArbitraryDataFileList() to support requesting only the missing hashes, but it is not yet used. 
Once GetArbitraryDataFileListMessage is rolled out to the network we can uncomment this code. Needs testnet testing prior to that.
 2022-02-03 20:01:44 +00:00
CalDescent
7994fc6407 Rework of onNetworkGetArbitraryDataFileListMessage() to support custom hashes to be optionally supplied. 
Also simplified the existing logic, to make the code more readable.
 2022-02-03 19:58:50 +00:00
CalDescent
d98df3e47d Added support for file hashes to optionally be included in GetArbitraryDataFileListMessage. 
Needs testing on testnet, as the majority of nodes need to update before the hashes can be used.
 2022-02-03 19:55:22 +00:00
CalDescent
1064b1a08b Removed duplicate metadata file checks. 2022-02-03 19:53:29 +00:00
CalDescent
0b7a7ed0f1 Added some more debug logging relating to file responses. 2022-02-03 19:52:28 +00:00
CalDescent
114b1aac76 Added arbitrary data file manager thread, which will ensure that all file list responses are tried until we receive the files. 
Previously we would only try the first response and then discard the others due to being duplicates. They are now added to a queue and retried by the dedicated thread (up to the 60 second timeout).
 2022-02-03 19:51:02 +00:00
CalDescent
6d06953a0e Increased RELAY_REQUEST_MAX_HOPS from 3 to 4, in an attempt to reach more peers. 2022-02-02 22:31:40 +00:00
CalDescent
0430fc8a47 Give up immediately after a synchronization if we are shutting down the synchronizer 2022-02-02 09:18:46 +00:00
CalDescent
7338f5f985 Attempt to acquire a blockchain lock (for up to 5 seconds) before shutting down the repository. 
This should fix conflicts caused by the synchronizer and controller now being on separate threads. It may also reduce the chances of the database corrupting on shutdown, but this remains to be seen.
 2022-02-02 09:17:24 +00:00
CalDescent
640bcdd504 Shutdown/interrupt the synchronizer as early as possible 2022-02-02 09:13:49 +00:00
CalDescent
c9d5d996e5 Increased accuracy of the block weights in the synchronizer logs, as the extra precision is now needed for debugging. 2022-02-01 22:06:01 +00:00
CalDescent
710befec0c Increased ARBITRARY_RELAY_TIMEOUT from 30 to 60 seconds, so that relay peers remember their mappings for longer. 2022-02-01 22:04:53 +00:00
CalDescent
8ccb158241 Reduce log spam when in DEBUG mode. 2022-02-01 22:04:12 +00:00
CalDescent
97199d9b91 Display the local and total chunk counts on the loading screen. 2022-02-01 22:03:52 +00:00
CalDescent
5a8b895475 Fixed bug in loading screen, which prevented the DOWNLOADING status from showing. 2022-02-01 22:03:17 +00:00
CalDescent
6c9600cda0 Added API key header to GET /arbitrary/resource/status/* endpoints 2022-02-01 22:02:41 +00:00
CalDescent
82fa6a4fd8 Include "localChunkCount" and "totalChunkCount" in the GET /arbitrary/resource/status/* API responses. 
These values are left out of other API endpoints where multiple resources are returned, because calculating the chunk counts is too time consuming.
 2022-02-01 22:02:14 +00:00
CalDescent
45f2d7ab70 Revert "We (currently) can't filter unconfirmed transactions by address, because only the public key is stored in the database until it is confirmed (at which point there is an entry in the TransactionParticipants table which contains the address). Given that this isn't a simple problem to solve, for now it makes sense to reject this combination if requested via the /transactions/search API." 
This reverts commit 7aed0354f1.
 2022-02-01 21:55:41 +00:00
CalDescent
33731b969a Direct peer connections now send a file list request to the peer, rather than individually requesting every chunk for a transaction. 2022-02-01 09:04:31 +00:00
CalDescent
40a8cdc71f Improved logging when fetching data files 2022-01-31 22:35:12 +00:00
CalDescent
cbe83987d8 Fixed occasional ConcurrentModificationException in the block archive reader. 2022-01-31 21:39:49 +00:00
CalDescent
01e4bf3a77 Try to speed up the shutdown process of the cleanup manager. 2022-01-31 21:39:20 +00:00
CalDescent
b198a8ea07 Fixed issue in ArbitraryDataFile.chunkExists() due to it not checking for the metadata file. 2022-01-31 21:38:51 +00:00
CalDescent
e2e87766fa Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before. 2022-01-30 20:15:43 +00:00
CalDescent
f005a0975d Shut down the synchronizer as soon as the controller is stopping, if we are able to. 2022-01-30 20:14:11 +00:00
CalDescent
5700369935 Prioritize syncing over transaction importing. 2022-01-30 20:09:35 +00:00
CalDescent
8a1fb6fe4e If a lock can't be obtained when synchronizing, automatically request the sync again. 2022-01-30 20:09:09 +00:00
CalDescent
5b788dad2f Relocated all synchronizer code from the controller to the synchronizer, and also moved synchonization onto its own thread. 2022-01-30 20:01:22 +00:00
CalDescent
fa2bd40d5f Reduce log spam 2022-01-30 19:51:49 +00:00
CalDescent
074bfadb28 Don't bother locking if there are no new transactions to process 2022-01-30 19:50:09 +00:00
CalDescent
bd60c793be Incoming transactions are now added to a queue, and then processed soon after. 
This solves a problem where incoming transactions could rarely obtain a blockchain lock (due to multiple transactions arriving at once) and therefore most messages were thrown away. It was also causing constant blockchain locks to be acquired, which would often prevent the synchronizer from running.
 2022-01-30 19:03:31 +00:00
CalDescent
90f3d2568a Log whenever the synchronizer can't obtain a blockchain lock, so that blockchain lock issues are more easily noticed. 2022-01-30 19:00:28 +00:00
CalDescent
c73cdefe6f transactionV5Timestamp moved to blockchain.json 2022-01-30 13:47:20 +00:00
CalDescent
c5093168b1 minAccountLevelToMint value moved to blockchain.json 2022-01-29 22:59:04 +00:00
CalDescent
a35e309a2f minAccountLevelForBlockSubmissions moved to blockchain.json 2022-01-29 22:57:22 +00:00
CalDescent
bb35030112
Merge pull request #65 from QuickMythril/reduce-doge-fee 
reduce DOGE fees
 2022-01-29 20:14:04 +00:00
CalDescent
7aed0354f1 We (currently) can't filter unconfirmed transactions by address, because only the public key is stored in the database until it is confirmed (at which point there is an entry in the TransactionParticipants table which contains the address). Given that this isn't a simple problem to solve, for now it makes sense to reject this combination if requested via the /transactions/search API. 2022-01-29 19:24:28 +00:00
CalDescent
c4f763960c Don't delete a resource's cache if a build is in progress. 
Hopeful fix for "Unable to delete cache for resource: Unable to delete directory" error, and possibly some other file conflicts.
 2022-01-29 19:18:06 +00:00
CalDescent
c5182a4589 Increased MAX_ACCOUNT_COUNT in GetOnlineAccountsMessage from 1000 to 5000 2022-01-29 12:39:43 +00:00
CalDescent
fc1a376fbd Added POST /transaction/fee API endpoint, to return the recommended fee for the supplied transaction data. 2022-01-29 10:38:15 +00:00
CalDescent
27387a134f Fixed typo 2022-01-28 19:58:32 +00:00
CalDescent
be7bb2df9e Added GET /transaction/unitfee API endpoint, to obtain the unit fee for a transaction type 
Additional params:
- timestamp: to allow for hard forks. Default: the current time
- level: the account level, to allow for the future possibility of different fees per level. Not currently used.
 2022-01-28 19:00:15 +00:00
CalDescent
72a291a54a Added initial support of different unit fees per transaction type. 
Included a timestamp property, as this will be needed for each update (hard fork).
 2022-01-28 18:50:03 +00:00
CalDescent
b1342d84fb Updated potentially misleading log message. 2022-01-28 11:02:37 +00:00
CalDescent
cdd57190ce Use getEffectiveMintingLevel() rather than getLevel() 2022-01-28 10:16:42 +00:00
CalDescent
7808a1553e Fixed case sensitive ordering issues in other aspects of QDN. 2022-01-27 18:46:59 +00:00
CalDescent
a0ba016171 Fixed case sensitive ordering issue in websites list. 2022-01-27 18:42:52 +00:00
CalDescent
344704b6bf MIN_LEVEL_FOR_BLOCK_SUBMISSION temporarily increased to 6. 
This is to hopefully improve network stability whilst a more advanced solution is being worked on. It also allows us to collect some data on how well the network behaves when there are less block candidates. It should have no effect on minting rewards (other than any side effects as a result of improved network stability).
 2022-01-27 18:15:00 +00:00
CalDescent
3303e41a39 Fixed unhandled case in GET /arbitrary/{service}/{name}* endpoints 2022-01-27 18:12:21 +00:00
CalDescent
4e71ae0e59 Allow QDN data to be served without authentication by setting "qdnAuthBypassEnabled":true 
This allows the GET /arbitrary/{service}/{name} and GET /{service}/{name}/{identifier} endpoints to operate without any authentication. Useful for those who are running public QDN nodes and need to serve data over http(s).
 2022-01-27 18:10:24 +00:00
CalDescent
9daf7a6668 Synchronize lists, to prevent an occasional ConcurrentModificationException 2022-01-26 22:40:34 +00:00
CalDescent
af06774ba6 Clear the cache when deleting data, so that it disappears from the data management screen. 2022-01-23 22:53:35 +00:00
catbref
244d4f78e2 New network messages ONLINE_ACCOUNTS_V2 and GET_ONLINE_ACCOUNTS_V2. 
Increased GetOnlineAccountsMessage.MAX_ACCOUNT_COUNT from 1000 to 5000.

The V2 versions are more efficiently encoded and also cache the payload bytes
which reduces CPU when sending to multiple peers.

Serialization / deserialization unit tests included.

Tentative V2 message activation set at core version 3.1.2
see Controller.ONLINE_ACCOUNTS_V2_PEER_VERSION
 2022-01-23 16:24:10 +00:00
CalDescent
6f7c8d96b9 Removed ApiService instance creation in ApplyUpdate as it wasn't really needed, and probably not sensible to instantiate it here. 2022-01-23 12:57:28 +00:00
CalDescent
ff6ec83b1c Removed localAuthBypassEnabled override in unit tests. 
Hopefully this will allow us to proactively catch any missing API keys in the future.
 2022-01-23 12:48:37 +00:00
CalDescent
ea10eec926 Fixed bug in auto update process - use the API key when stopping the node. 
Luckily this code is included in the new JAR, not the old one, so we should be able to regain auto update ability by issuing a new update.
 2022-01-23 12:38:19 +00:00
CalDescent
6f724f648d Fixed testDirectoryDigest() which has been failing for a couple of versions (due to gitignore removing the cache file) 2022-01-22 20:48:58 +00:00
CalDescent
048776e090 Ignore failing test due to recent API update, which makes the test incompatible. To be fixed later. 2022-01-22 20:43:28 +00:00
CalDescent
dedf65bd4b Added initial protocol methods for metadata requests and forwarding. Not tested yet. 2022-01-22 20:24:37 +00:00
CalDescent
25efee55b8 Added networking optimization, to avoid wasted processing on every read. 
Thanks to @catbref for finding this.
 2022-01-22 18:43:32 +00:00
CalDescent
a79ed02ccf Added initial (unfinished) category list, as well as the GET /arbitrary/categories API, and converted the category field from a string to an enum 2022-01-22 12:11:16 +00:00
CalDescent
79f87babdf Limit the metadata string lengths 2022-01-21 22:52:31 +00:00
CalDescent
f296d5138b Allow metadata to optionally be included with any arbitrary resource. 2022-01-21 21:14:28 +00:00
CalDescent
b30445c5f8 Increase MAX_ACCOUNT_COUNT to 5000 (around 0.5MB of data) to see if it helps with minting efficiency. 2022-01-21 16:11:21 +00:00