crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
2,699 Commits 85 Branches 118 Tags 530 MiB
a01652b816
Commit Graph

2414 Commits

Author SHA1 Message Date
QuickMythril
e61a24ee7b
removed electrum-ltc.bysh.me 
this server often gives a false positive for phishing by some antivirus software.
 2022-04-03 22:32:57 -04:00
JaymenChou
55ed342b59
Create TransactionValidity_zh_CN.properties 
Add Simple Chinese For better understanding of logs
 2022-04-03 13:27:52 +08:00
JaymenChou
3c6f79eec0
Create TransactionValidity_zh_TW.properties 
Add Traditional Chinese For TransactionValidity Logs.
 2022-04-03 13:25:32 +08:00
JaymenChou
590800ac1d
Create ApiError_zh_CN.properties 
Add Simple Chinese Support For API Error Message 
Hope it helps in understanding the API !
 2022-04-03 12:43:18 +08:00
JaymenChou
95c412b946
Create ApiError_zh_TW.properties 
Add Traditional Chinese support to API Responses
 2022-04-03 12:40:27 +08:00
CalDescent
a232395750 Merge branch 'master' of github.com:Qortal/qortal 2022-04-01 11:24:56 +01:00
QuickMythril
6edbc8b6a5
add decimal precision to download progress 2022-03-31 13:46:40 -04:00
QuickMythril
f8ffeed302
updated BTC electrumx servers 
added new and removed TCP, closed servers, and versions older than 1.16.0
 2022-03-31 11:32:55 -04:00
QuickMythril
e2ee68427c
removed TCP electrumx servers 2022-03-31 11:29:54 -04:00
QuickMythril
74ff23239d
removed TCP electrumx servers 2022-03-31 11:27:56 -04:00
QuickMythril
f1fa2ba2f6
added SSL electrumx servers 2022-03-31 10:02:31 -04:00
QuickMythril
e1522cec94
updated LTC electrumx servers 2022-03-31 09:58:53 -04:00
QuickMythril
8841b3cbb1
add spanish translations 2022-03-31 08:44:33 -04:00
CalDescent
94260bd93f Decreased the number of retries for missing metadata, to reduce broadcast spam. 2022-03-30 08:23:22 +01:00
CalDescent
15ff8af7ac Don't process trade bots or broadcast presence timestamps if our chain is more than 30 minutes old 2022-03-30 08:11:02 +01:00
CalDescent
d420033b36 Revert "Revert "Add Qortal AT FunctionCodes for getting account level / blocks minted + tests"" 
This reverts commit 59025b8f47.
 2022-03-30 08:07:07 +01:00
CalDescent
bda63f0310 Removed hardcoded "qortal-backup/TradeBotStates.json" from POST /admin/repository/data API, as it's no longer needed now that API keys are required. 2022-03-30 08:06:09 +01:00
QuickMythril
54add26ccb
fixed typo 2022-03-25 23:39:41 -04:00
CalDescent
bbe15b563c Added unit test to simulate recent issue. 
This fails with the 3.2.2 code but now passes when using the latest fixes.
 2022-03-19 20:41:38 +00:00
CalDescent
59025b8f47 Revert "Add Qortal AT FunctionCodes for getting account level / blocks minted + tests" 
This reverts commit eb9b94b9c6.
 2022-03-19 19:52:14 +00:00
CalDescent
1b42c5edb1 Fixed NPE in runIntegrityCheck() 
This feature is disabled by default so can be tidied up later. For now, the unhandled scenario is logged and the checking continues on.

One name's transactions are too complex for the current integrity check code to verify (MangoSalsa), but it has been verified manually. All other names pass the automated test.
 2022-03-19 19:22:16 +00:00
CalDescent
362335913d Fixed infinite loop in name rebuilding. 
If an account is renamed and then at some point renamed back to one of the original names, it confused the names rebuilding code. The current solution is to track the linked names that have already been rebuilt, and then break out of the loop once a name is encountered a second time.
 2022-03-19 18:55:19 +00:00
CalDescent
4340dac595 Fixed recently introduced issue in name rebuilding code causing transactions to be unordered. 
This is the likely cause of inconsistent name entries across different nodes, as we can't guarantee that every environment will return the same transaction order from the SQL queries.
 2022-03-19 18:44:16 +00:00
CalDescent
f3e1fc884c
Merge pull request #63 from catbref/master 
Add Qortal AT FunctionCodes for getting account level / blocks minted
 2022-03-19 11:32:39 +00:00
CalDescent
39c06d8817
Merge pull request #75 from catbref/name-unicode 
Unicode / NAME updates.
 2022-03-19 11:32:22 +00:00
CalDescent
91cee36c21 Catch and log all exceptions in addStatusToResources() 
Some users are seeing 500 errors deriving from this code. This should hopefully allow more info to be obtained, as well as causing it to omit the status for resources that encounter problems.
 2022-03-19 11:08:42 +00:00
CalDescent
12a4a260c8 Handle new sync result case. 2022-03-14 22:04:11 +00:00
CalDescent
268f02b5c3 Added automated test to ensure that the core's default bootstrap hosts are functional. 
Whilst not strictly a unit test, this should allow issues with the core's bootstrap servers to be caught quickly.
 2022-03-14 21:52:54 +00:00
CalDescent
13eff43b87 Fixed synchronizer issues which caused large re-orgs 
Peers without a recent block are removed at the start of the sync process, however, due to the time lag involved in fetching block summaries and comparing the list of peers, some of these could subsequently drop back to a non-recent block and still be chosen as the next peer to sync with. The end result being that nodes could unnecessarily orphan as many as 20 blocks due to syncing with a peer that doesn't have a recent block (but has a couple of high weight blocks after the common block).

This commit adds some additional filtering to avoid this situation.

1) Peers without a recent block are removed as candidates in comparePeers(), allowing for alternate peers to be chosen.
2) After comparePeers() completes, the list is filtered a second time to make sure that all are still recent.
3) Finally, the peer's state is checked one last time in syncToPeerChain(), just before any orphaning takes place.

Whilst just one of the above would probably have been sufficient, the consequences of this bug are so severe that it makes sense to be very thorough.

The only exception to the above is when the node is in "recovery mode", in which case peers without recent blocks are allowed to be included. Items 1 and 3 above do not apply in recovery mode. Item 2 does apply, since the entire comparePeers() functionality is already skipped in a recovery situation due to our chain being out of date.
 2022-03-14 21:47:37 +00:00
catbref
e604a19bce Unicode / NAME updates. 
Fix UPDATE_NAME not processing empty 'newName' transactions correctly.
Fix some emoji code-points not being processed correctly.
Updated tests.
Now included ICU4J v70.1 - WARNING: this could add around 10MB to JAR size!
Bumped homoglyph to v1.2.1.
 2022-03-14 08:45:32 +00:00
CalDescent
f0d9982ee4 Made arbitraryDataFileHashResponses final, and use .sort() rather than .stream() to avoid new instance creation. 2022-03-12 09:43:56 +00:00
CalDescent
c65de74d13 Revert "Synchronize arbitrary data list removals, as it seems that SynchronizedList and SynchronizedMap aren't overriding removeIf() with a thread-safe version." 
This reverts commit e5f88fe2f4.
 2022-03-12 09:40:13 +00:00
CalDescent
df0a9701ba Improved logging in onNetworkGetArbitraryDataFileListMessage() 2022-03-11 16:51:19 +00:00
CalDescent
4ec7b1ff1e Removed time estimations that are no longer correct or relevant. 2022-03-11 16:50:34 +00:00
CalDescent
7d3a465386 Including the number of hashes (even if zero) is now required in GetArbitraryDataFileListMessage, to allow for additional fields. Enough peers should have updated by now for this to be ok. 2022-03-11 16:50:11 +00:00
CalDescent
30347900d9 Tidied up one last place that was accessing immutableConnectedPeers directly. This makes no difference, but helps with code consistency. 2022-03-11 15:28:54 +00:00
CalDescent
e5f88fe2f4 Synchronize arbitrary data list removals, as it seems that SynchronizedList and SynchronizedMap aren't overriding removeIf() with a thread-safe version. 2022-03-11 15:22:34 +00:00
CalDescent
0d0ccfd0ac Small refactor for code simplicity. 2022-03-11 15:11:07 +00:00
CalDescent
9013d11d24 Report as 100% synced if the latest block is within the last 30 mins 
This should hopefully reduce confusion due to APIs reporting 99% synced even though up to date. The systray should never show this since it already treats blocks in the last 30 mins as synced.
 2022-03-11 14:53:10 +00:00
CalDescent
fc5672a161 Use a more tolerant latest block timestamp in the isUpToDate() calls below to reduce misleading systray statuses. 
Any block in the last 30 minutes is considered "up to date" for the purposes of displaying statuses.
 2022-03-11 14:49:02 +00:00
CalDescent
221c3629e4 Don't refetch the file list if the fileListCache is empty, since an empty list now means that there are likely to be no files available on disk. 2022-03-11 13:08:37 +00:00
CalDescent
76fc56f1c9 Fetch the file list in getFilenameForHeight() if needed. 2022-03-11 13:07:16 +00:00
CalDescent
8e59aa2885 Peer getter methods renamed to include "immutable", for consistency with underlying lists and also to make it clearer to the callers. 2022-03-11 13:00:47 +00:00
CalDescent
0738dbd613 Avoid direct access to this.connectedPeers, as we need to use the immutable copy. 2022-03-11 12:58:11 +00:00
CalDescent
196ecffaf3 Skip calls to this.logger.trace() in ExecuteProduceConsume.run() if trace logging isn't enabled. 
This could very slightly reduce load due to skipping the internal filtering inside log4j. Given that this method is causing major problems with CPU at times, I'm trying to make it as optimized as possible.
 2022-03-11 11:59:18 +00:00
CalDescent
a0fedbd4b0 Implemented suggestions from catbref to avoid potential thread safety issue in peer arrays. 2022-03-11 11:27:13 +00:00
CalDescent
7c47e22000 Set fileListCache to null when invalidating. 2022-03-11 11:01:29 +00:00
CalDescent
6aad6a1618 fileListCache is now an immutable Map, which is thread safe. Thanks to catbref for this idea. 2022-03-11 10:59:07 +00:00
CalDescent
b764172500 Revert "Hopeful fix for ConcurrentModificationException in BlockArchiveReader.getFilenameForHeight()" 
This reverts commit a12ae8ad24.
 2022-03-11 10:55:22 +00:00
CalDescent
c185d79672 Loop through all available direct peer connections and try each one in turn. 
Also added some extra conditionals to avoid repeated attempts with the same port.
 2022-03-09 20:55:27 +00:00
CalDescent
76b8ba91dd Only add an entry to directConnectionInfo if one with this peer-signature combination doesn't already exist. 2022-03-09 20:50:03 +00:00
CalDescent
0418c831e6 Direct connections with peers now prefer those with the highest number of chunks for a resource. Once a connection has been attempted with a peer, remove it from the list so that it isn't attempted again in the same round. 2022-03-09 20:15:26 +00:00
CalDescent
4078f94caa Modified GetArbitraryDataFileListMessage to allow requesting peer's address to be optionally included. 
This can ultimately be used to notify the serving peer to expect a direct connection from the requesting peer (to allow it to temporarily bypass maxConnections for long enough for the files to be retrieved). Or it could even possibly be used to trigger a reverse connection (from the serving peer to the requesting peer).
 2022-03-09 19:58:02 +00:00
CalDescent
a12ae8ad24 Hopeful fix for ConcurrentModificationException in BlockArchiveReader.getFilenameForHeight() 2022-03-09 19:46:50 +00:00
CalDescent
498ca29aab Wait until a successful connection with a peer before tracking the direct request. 2022-03-08 23:07:08 +00:00
CalDescent
ba70e457b6 Default chunk size reduced from 1MB to 0.5MB 2022-03-08 22:44:43 +00:00
CalDescent
d62808fe1d Don't attempt to create the data directory every time an ArbitraryDataFile instance is instantiated. This was using excessive amounts of CPU and disk I/O. 2022-03-08 22:42:07 +00:00
CalDescent
6c14b79dfb Removed bootstrap host that is no longer functional. 2022-03-08 22:30:01 +00:00
CalDescent
631a253bcc Added support for dark theme in loading screen. 2022-03-08 22:29:37 +00:00
CalDescent
4cb63100d3 Drop the ArbitraryPeers table as it's no longer needed 2022-03-06 13:01:09 +00:00
CalDescent
42fcee0cfd Removed all code that interfaced with the ArbitraryPeers table 2022-03-06 13:00:11 +00:00
CalDescent
829a2e937b Removed all arbitrary signature broadcasts 2022-03-06 12:58:01 +00:00
CalDescent
5d7e5e8e59 Dropped support of ARBITRARY_SIGNATURES message handling, as this feature has been superseded by the peerAddress in file list requests. 2022-03-06 12:46:06 +00:00
CalDescent
6f0a0ef324 Small refactor 2022-03-06 12:42:19 +00:00
CalDescent
f7fe91abeb sendOurOnlineAccountsInfo() moved to its own thread, in preparation for mempow 2022-03-06 12:41:54 +00:00
CalDescent
7252e8d160 Deleted presence tests, as they are no longer relevant, and aren't easily adaptable to the new approach. 2022-03-06 12:03:18 +00:00
CalDescent
2630c35f8c Chunk validation now uses MAX_CHUNK_SIZE rather than CHUNK_SIZE, to allow for a smaller CHUNK_SIZE value to be optionally used, without failing the validation of existing resources. 2022-03-06 11:43:28 +00:00
CalDescent
49f466c073 Added missing break; 2022-03-06 11:21:55 +00:00
CalDescent
c198f785e6 Added significant CPU optimizations to ArbitraryDataManager 
- Slow down loops that query the db
- Check for new metadata every 5 minutes instead of constantly
- Check for new data every 1 minute instead of constantly

This could be further improved in the future by having block.process() notify the ArbitraryDataManager that there is new data to process. This would avoid the need for the frequent checks/loops, and only a single complete sweep would be needed on node startup (as long as failures are then retried). But I will avoid this additional complexity for now.
 2022-03-06 11:21:39 +00:00
CalDescent
5be093dafc Fix for "Synchronizing null%" systray bug introduced in 3.2.0 2022-03-06 11:00:53 +00:00
CalDescent
2c33d5256c Added code accidentally missed out of commit 1b036b7 2022-03-05 20:44:01 +00:00
CalDescent
4448e2b5df Handle case when metadata isn't returned. 2022-03-05 17:39:13 +00:00
CalDescent
146d234dec Additional defensiveness in ArbitraryDataFile.fromHash() to avoid similar future bugs. 2022-03-05 17:25:48 +00:00
CalDescent
18d5c924e6 Fixed bug cased by fetchAllMetadata() 2022-03-05 17:25:14 +00:00
CalDescent
b520838195 Increased default maxNetworkThreadPoolSize from 20 to 32 
This will hopefully offset some of the additional network demands from arbitrary data requests.
 2022-03-05 17:24:55 +00:00
CalDescent
1b036b763c Major CPU optimization to block minter 
Load sorted list of reward share public keys into memory, so that the indexes can be obtained. This is around 100x faster than querying each index separately (and the savings will increase as more keys are added).

For 4150 reward share keys, it was taking around 5000ms to query individually, vs 50ms using this approach.

The main trade off is that these 4150 keys require around 130kB of additional memory when minting (and this will increase proportionally with more minters). However, this one query was often accounting for 50% of the entire core's CPU usage, so the additional memory usage seems insignificant by comparison.

To gain confidence, I ran both old and new approaches side by side, and confirmed that the indexes matched exactly.
 2022-03-05 16:10:43 +00:00
CalDescent
8545a8bf0d Automatically fetch metadata for all resources that have it. 2022-03-05 13:00:49 +00:00
CalDescent
f0136a5018 Include the external port when responding ArbitraryDataFileListRequests 2022-03-05 13:00:17 +00:00
CalDescent
6697b3376b Direct peer connections now use the on-demand data retrieved from file list requests, rather than the stale and incomplete ArbitraryPeerData. 2022-03-05 12:59:13 +00:00
CalDescent
ea785f79b8 Removed unnecessary synchronization 2022-03-04 19:02:30 +00:00
CalDescent
0352a09de7 New online accounts are now verified on the OnlineAccountsManager thread rather than on network threads. This is an attempt to reduce the amount of blocked network threads due to signature verification, and is necessary for the upcoming mempow addition. 2022-03-04 17:58:06 +00:00
CalDescent
5b4f15ab2e Transaction importing code moved to TransactionImporter controller class 
As with online accounts, no logic changes other than moving transaction queue processing from the controller thread to its own dedicated thread.
 2022-03-04 16:47:21 +00:00
CalDescent
fd37c2b76b Moved all online accounts code to a new OnlineAccountsManager controller class 
There are no logic changes here other than moving performOnlineAccountsTasks() onto its own thread, so that it's not subject to anything that might be slowing down the main controller thread.
 2022-03-04 16:24:04 +00:00
CalDescent
924aa05681 Optimized peer lists 
- Removed synchronization from connectedPeers, and replaced it with an unmodifiableList.
- Added additional immuatable caches: handshakedPeers and outboundHandshakedPeers

This should greatly reduce the amount of time spent waiting around for access to the connectedPeers array, since it is now immediately accessible without needing to obtain a lock. It also removes calls to stream() which were consuming large amounts of CPU to constantly filter the connected peers down to a list of handshaked peers.

Thanks to @catbref for these great suggestions.
 2022-03-04 15:14:12 +00:00
CalDescent
84b42210f1 Use ArbitraryDataFileRequestThreads only - instead of reusing file list response threads. 2022-03-04 13:34:16 +00:00
CalDescent
941080c395 Rework of arbitraryDataFileHashResponses to use a list rather than a map (limited to 1000) items. Sort the list by routes with the lowest number of peer hops first, to try and prioritize those which are easiest and quickest to reach. 2022-03-04 13:33:17 +00:00
CalDescent
35d9a10cf4 Avoid logging if there are no remaining transaction signatures to validate. There was too much log spam, none of which was particularly useful. 2022-03-04 12:03:58 +00:00
CalDescent
7c181379b4 Added more granularity to logging, to differentiate between signature validation and general processing/importing, as well as showing counts of the transactions being processed in each round. 2022-03-04 11:12:23 +00:00
CalDescent
f9576d8afb Further optimizations to Controller.processIncomingTransactionsQueue() 
- Signature validation is now able to run concurrently with synchronization, to reduce the chances of the queue building up, and to speed up the propagation of new transactions. There's no need to break out of the loop - or avoid looping in the first place - since signatures can be validated without holding the blockchain lock.
- A blockchain lock isn't even attempted if a sync request is pending.
 2022-03-04 11:05:58 +00:00
CalDescent
6a8a113fa1
Merge pull request #74 from catbref/presence-txns-removal 
PRESENCE transactions changed to always fail signature validation
 2022-03-04 10:33:11 +00:00
CalDescent
ef59c34165 Added missing "break" which was causing additional unnecessary debug logging. Originally introduced due to a merge conflict with the metadata branch. 2022-03-04 10:28:44 +00:00
catbref
a9371f0a90 In Controller.processIncomingTransactionsQueue(), don't bother with 2nd-phase of locking blockchain and importing if there are no valid signature transactions to actually import 2022-03-03 20:32:27 +00:00
catbref
a7a94e49e8 PRESENCE transactions changed to always fail signature validation 2022-03-03 20:25:58 +00:00
catbref
affd100298 Reworking of Controller.processIncomingTransactionsQueue() 
Main changes are:
* Check transaction signature validity in initial round, without blockchain lock
* Convert List of incoming transactions to Map so we can record whether we have validated transaction signature before to save rechecking effort
* Add invalid signature transactions to invalidUnconfirmedTransactions map with INVALID_TRANSACTION_RECHECK_INTERVAL expiry (~60min)
* Other minor changes related to List->Map change and Java object synchronization
 2022-03-03 20:21:04 +00:00
CalDescent
69309c437e Tightened up the content security policy for non HTML files. 2022-03-01 20:36:34 +00:00
CalDescent
e392e4d344 Allow eval(), setTimeout(), etc, to enable various QDN sites to function correctly. The existing sandboxing should be locking this down enough already. Limited to .html and .htm files only. 2022-03-01 20:35:56 +00:00
CalDescent
bd53856927 Disabled auto fetching of metadata. To be re-enabled at a later date. 2022-03-01 20:26:09 +00:00
CalDescent
cbd1018ecf Allow identical data to be published if the metadata differs. 2022-03-01 20:22:47 +00:00
CalDescent
46606152eb /arbitrary/metadata/* endpoint now returns ArbitraryResourceMetadata rather than a raw JSON string. 2022-03-01 20:22:20 +00:00
CalDescent
e6f93e0a08 Added categoryName to ArbitraryResourceMetadata, along with the existing category ID 2022-03-01 20:19:08 +00:00
CalDescent
8d81f1822f Merge branch 'master' into qdn-metadata 
# Conflicts:
#	src/main/java/org/qortal/controller/Controller.java
#	src/main/java/org/qortal/network/message/Message.java
 2022-02-28 20:10:39 +00:00
catbref
590a8f52db Remove future work comment from Controller 2022-02-27 16:57:26 +00:00
catbref
ecac47d1bc Also notify TradePresenceWebsocket (using TradePresenceEvent) when bridging old PRESENCE txns 2022-02-27 16:56:17 +00:00
catbref
3b477ef637 Fix JAXB marshalling error (duplicate tradeAddress) in TradePresenceWebSocket. No need to send signature. Make sure publicKey is sent in Base58, not Base64. 2022-02-27 16:56:17 +00:00
catbref
e2ef5b2ef3 Missed change from last commit: incorrect logic in TradePresenceWebSocket 2022-02-27 16:56:17 +00:00
catbref
1d59feeb72 Created /websockets/crosschain/tradepresence to replace /websockets/presence 2022-02-27 16:55:30 +00:00
catbref
c53dd31765 Tidy up of trade presence timestamp generation & checking. Added tests. Renamed "online trades" to "trade presences" 2022-02-27 16:54:42 +00:00
catbref
4c02081992 Tidy up TradeBot presence logging. Decorate API endpoints /crosschain/tradeoffers and /crosschain/trade with presence expiry timestamps 2022-02-27 16:54:42 +00:00
catbref
cb57af3c53 Bugfixes to online trade sigs + bridging from PRESENCE transactions 2022-02-27 16:54:42 +00:00
catbref
01d810fc00 Initial effort at migrating PRESENCE transactions to dedicated network messages 2022-02-27 16:54:42 +00:00
CalDescent
8c2a9279ee Return metadata in various /arbitrary APIs if the "includemetadata" parameter is included. 
This is very inefficient and will soon be replaced with dedicated ArbitraryResources / ArbitraryMetadata tables. But this is acceptable in the short term, especially if limit and offset are used.
 2022-02-27 09:09:18 +00:00
CalDescent
0d65448f3d Request all metadata automatically. 2022-02-27 08:20:39 +00:00
CalDescent
9da2b3c11a Don't respond to file list requests with just the metadata file. 
We have the separate metadata protocol for this now.
 2022-02-27 07:28:11 +00:00
CalDescent
95400da977 Fixed typo in various tests (copy and paste error) 2022-02-26 22:10:55 +00:00
CalDescent
dc41dc4c69 Tags now use an array of strings, rather than a single string. 2022-02-26 22:09:07 +00:00
CalDescent
a5c11d4c23 Reduced "Ignoring hash list request" logs from DEBUG to TRACE 2022-02-26 16:10:44 +00:00
CalDescent
878394535e Improvements relating to fetching metadata 
- Rate limiter is disabled when using the API
- fetchArbitraryMetadata() returns the actual metadata content rather than a boolean
- Exceptions are thrown on certain errors, rather than returning null
 2022-02-26 16:10:26 +00:00
CalDescent
35dba27a55 Fixed issue due to not updating arbitraryMetadataRequests when receiving the metadata file. 2022-02-26 16:07:06 +00:00
CalDescent
f22ad13fa9 Merge branch 'master' into qdn-metadata 
This involved a slight rewrite to remove the "includeMetadataOnly" boolean. Metadata is now always excluded, otherwise it complicates the caching too much.

# Conflicts:
#	src/main/java/org/qortal/api/resource/ArbitraryResource.java
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataStorageManager.java
 2022-02-26 14:39:20 +00:00
CalDescent
aa2e5cb87b Merge branch 'hosted-resources-search' 2022-02-26 14:05:52 +00:00
CalDescent
7740f3da7e Small formatting tweaks, for consistency with existing code. 2022-02-26 14:05:28 +00:00
CalDescent
badb576991 Fixed exception when identifier is null. Also handling null names as this may be a future scenario. 2022-02-26 14:04:35 +00:00
CalDescent
c65a63fc7e Fixed "query" parameter error in swagger documentation 2022-02-26 13:59:53 +00:00
CalDescent
0111747016 Added debug logging of new file list stats. 2022-02-25 13:30:07 +00:00
CalDescent
eac4b0d87b Maintain backwards support for pre-3.2.0 peers by only including new file list message params when sending to newer peers. 
These params are optional and the process will function without them, just less efficiently.
 2022-02-25 12:24:02 +00:00
CalDescent
3dadce4da4 Renamed a reference 2022-02-25 12:24:02 +00:00
CalDescent
1864468818 Prefer the route with the least number of hops when relaying. 2022-02-25 12:24:02 +00:00
CalDescent
1a59379162 Optionally include requestTime, requestHops, peerAddress, and isRelayPossible flag in ArbitraryDataFileListMessage 2022-02-25 12:24:02 +00:00
CalDescent
3cc394f02d
Merge pull request #70 from catbref/synchronizer-newchaintipevent 
Modify TradeBot to trigger when chain tip changes instead of with every new block
 2022-02-24 20:04:26 +00:00
CalDescent
53c4fe9e80 Fixed another ElectrumX issue found in unit tests. 
Peers that were thought to be missing output address data may actually have just been using a different key - "address" instead of "addresses". Now reading the addresses from both keys, which may remove the need for the previously added checks.
 2022-02-24 20:01:56 +00:00
CalDescent
d5521068b0 Fixed issue in earlier commit, found in unit tests. 2022-02-24 19:45:37 +00:00
CalDescent
a63ef4010d Disabled expired transaction data deletion code for now, as this was often causing data to be incorrectly deleted. 
This will need to be re-enabled at some point, but only after it's modified to be much less aggressive.
 2022-02-24 19:05:29 +00:00
catbref
8950bb7af9 Very slightly relax validity checks for TRANSFER_PRIVS to allow for skeletal account records, e.g. due to CHAT transactions, but account last reference still needs to be null. Example at block height 736196 / 7 2022-02-24 09:13:51 +00:00
catbref
9e6fe7ceb9 Modify TradeBot, some related websockets, to trigger when chain tip changes instead of with every new block 2022-02-24 09:06:21 +00:00
CalDescent
c333d18cd0 Merge branch 'segwit' 2022-02-23 20:07:07 +00:00
CalDescent
0271ef69c9 When submitting a new transaction, treat the chain as "synced" if the latest block is less than 30 minutes old. Increased from around 7.5 minutes. 2022-02-23 20:06:55 +00:00
CalDescent
2d493a4ea2 Added logging when no addresses are returned for a bitcoiny transaction output. 2022-02-23 09:29:16 +00:00
CalDescent
e339ab856f Skip over Electrum servers that don't return any output addresses. Hopeful fix for BTC transactions that report a zero value due to incomplete data being returned from certain ElectrumX peers. 2022-02-23 08:34:38 +00:00
proto
782904a971 improvement to the search on hosted resources 
1) use the cached version instead of rescanning all the files
2) separating the loading (which include files scanning) and listing logic
 2022-02-22 17:54:08 +01:00
proto
a3753c01bc Add search functionality to hosted resources 2022-02-22 15:50:46 +01:00
CalDescent
d5c3921846 Only show the red "synchronizing" systray icon if the latest block isn't recent. 
This should fix issue where the icon unnecessarily jumps between synced and synchronizing.
 2022-02-21 22:34:13 +00:00
CalDescent
a2c462b3da Add <meta charset="UTF-8"> tag to websites. Fixed issue rendering emojis and other special characters. 2022-02-21 22:28:59 +00:00
CalDescent
8673c7ef6e Fixed bug in GET /peers/summary API 2022-02-21 22:28:18 +00:00
CalDescent
8d7be7757f Fixed incorrectly named tag. 2022-02-21 22:27:44 +00:00
proto
6b83927048 Persist MintingAccounts.json on minting accounts add/remove 
this fix the behavior of the node, After adding or removing a minting account, allowing it to persist it to the backup folder
 2022-02-21 16:17:17 +01:00
proto
e07adbd60e online accounts api call, fix level zero accounts 
Added online zero level accounts to the response of /addresses/online/levels api endpoints
 2022-02-21 15:40:10 +01:00
CalDescent
7798b8dcdc Keep items in arbitraryDataFileHashResponses if they are currently being requested by another thread. This should help to locate the higher numbered chunks from larger resources. 2022-02-20 11:33:09 +00:00
CalDescent
146e7970bf Synchronize this.allKnownPeers and this.connectedPeers in Network.requestDataFromPeer(), to make the method thread-safe. 
This could be further improved by taking an immutable copy, but I'll leave this until the same approach is applied to other Network methods.
 2022-02-20 11:04:33 +00:00
CalDescent
f4f7cc58e3 Removed unused import. 2022-02-20 10:44:59 +00:00
CalDescent
21b4b494e7 Renamed method. 2022-02-20 10:44:38 +00:00
CalDescent
7307844bee If UPnP is disabled in settings, close the existing external listenPort if a UPnP rule exists. 2022-02-20 10:44:20 +00:00
CalDescent
5d419dd4ec Handle case where funds are sent to and from the same bitcoiny deterministic wallet. 2022-02-19 17:45:24 +00:00
CalDescent
6d0db7cc5e Catch UncheckedIOException in findAllHostedPaths() which was seen when a file was deleted by another thread. 2022-02-19 17:18:51 +00:00
CalDescent
8de606588c Attempt to open the listen port (default 12392) using UPnP, if the local network supports it. 2022-02-18 20:11:00 +00:00
CalDescent
35b0a85818 Increased WALLET_KEY_LOOKAHEAD_INCREMENT_BITCOINJ to 50 2022-02-18 18:42:06 +00:00
CalDescent
fcdd85af6c Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-18 18:39:47 +00:00
CalDescent
5aac2dc9df ONLINE_ACCOUNTS_V2_PEER_VERSION set to 3.2.0 2022-02-18 18:34:00 +00:00
CalDescent
becb0b37e6
Merge branch 'master' into network-online-accounts-v2 2022-02-18 17:12:44 +00:00
CalDescent
67ca876567 Log if we're unable to process the received file. 2022-02-18 15:35:37 +00:00
CalDescent
464ce66fd5 Moved deletion retry code into ArbitraryDataFile 2022-02-18 15:09:50 +00:00
CalDescent
3e505481fe Default minPeerVersion also increased to 3.1.0 2022-02-18 14:50:46 +00:00
CalDescent
c90c3a183e Block peers below 3.1.0 2022-02-18 14:50:29 +00:00
CalDescent
85b3278c8a Don't throttle the arbitrary data file request threads when there are items to process. 2022-02-15 19:39:26 +00:00
CalDescent
c90c287601 Increased ARBITRARY_REQUEST_TIMEOUT from 10 to 12 seconds, as some were coming back around 9-10 seconds later. 2022-02-15 19:38:30 +00:00
CalDescent
6ee395ed12 Stop bulk arbitrary signature broadcasts, as they were creating a lot of network traffic, and are in the process of being replaced with a better method. 2022-02-15 19:10:20 +00:00
CalDescent
6275ac2b81 Increased numberOfAdditionalBatchesToSearch from 5 to 7. 
This is the equivalent of increasing the max address gap from 15 to 21. The electrum standalone wallet uses 20, so this should be the most we will ever need.
 2022-02-14 22:58:37 +00:00
CalDescent
fd0a6ec71f Fix for invalid balance (and transaction amount) when there are no outputs relating to this wallet. 2022-02-14 22:53:30 +00:00
CalDescent
43791f00aa Wait 2 minutes on node startup before trying to fetch followed QDN data. 2022-02-14 19:33:58 +00:00
CalDescent
538ac30b4e Request only the missing hashes, not all of them. 2022-02-14 19:33:36 +00:00
CalDescent
acddf36467 Handle missing includeMetadata parameter. 2022-02-13 19:27:12 +00:00
CalDescent
166d32032a Fixed message IDs. 2022-02-13 19:22:20 +00:00
CalDescent
e4238a62c9 Exclude metadata-only transactions in the data management page (but added an API parameter to allow them to optionally be included). 
This ensures that the list will only show resources where there is at least 1 chunk.
 2022-02-13 19:21:16 +00:00
CalDescent
ad9c466712 Fall back to UNCATEGORIZED if the parsed category doesn't match any available categories. 
This allows for deletion of categories, as the resources will just move into UNCATEGORIZED until they are next updated.
 2022-02-13 18:10:56 +00:00
CalDescent
a3d31bbaf1 Category updates based on feedback so far. 2022-02-13 17:56:47 +00:00
CalDescent
4821139501 Merge branch 'master' into qdn-metadata 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileListManager.java
 2022-02-13 15:50:12 +00:00
CalDescent
83213800b9 Use the timestamp from the registerNameTransactionData in unit tests, rather than the current time. 2022-02-13 15:05:28 +00:00
CalDescent
265ae19591 Fixed other failing tests due to increased REGISTER_NAME transaction fee. At some point we should determine the correct fee inside of generateBase(), but setting it explicitly adds confidence in testing for now. 2022-02-13 14:31:21 +00:00
CalDescent
c1598d20b5 Name registration fee increase timestamp set to Sunday, 20 February 2022 16:00:00 UTC 2022-02-13 13:47:00 +00:00
CalDescent
0712259057 Implemented REGISTER_NAME transaction fee increase from 0.001 to 5 QORT (average value based on community vote). 2022-02-13 13:45:48 +00:00
CalDescent
ea42a5617f Fixed ElectrumX log spam and errors 2022-02-13 10:58:45 +00:00
CalDescent
58a690e2c3 Route through new getAddressTransactions() wrapper. 2022-02-11 18:15:27 +00:00
CalDescent
3ae2f0086e Removed unusably slow electrum peer 2022-02-11 18:13:45 +00:00
CalDescent
19c83cc54d MAX_AVG_RESPONSE_TIME reduced to 500, as one peer regularly takes around 600ms to reply. 2022-02-11 18:12:34 +00:00
CalDescent
8ac298e07d Allow 3 retries for getTransaction() and getAddressTransactions() requests 2022-02-11 18:11:00 +00:00
CalDescent
9b43e4ea3d Time electrum requests, and move on to another server if one takes more than 1000ms on average to respond (measured over the last 5 requests). 2022-02-11 18:02:56 +00:00
CalDescent
dbacfb964b Increased TX_CACHE_SIZE from 200 to 1000, to speed up loading times on large wallets. 2022-02-11 16:55:29 +00:00
CalDescent
a664a6a790 Added more LTC Electrum peers from https://1209k.com/bitcoin-eye/ele.php?chain=ltc 2022-02-11 16:44:34 +00:00
CalDescent
ee1f072056 Improvement to last commit, so that caller class names are preserved. 2022-02-11 15:34:31 +00:00
CalDescent
a6aabaa7f0 Reduce build queue log spam by only logging high priority items (5 and above). 2022-02-11 15:28:41 +00:00
CalDescent
49b307db60 Treat a null priority as 0 2022-02-11 15:17:02 +00:00
CalDescent
f7341cd9ab Increased /arbitrary priority to 1 2022-02-11 15:13:53 +00:00
CalDescent
6932fb9935 Added "priority" property to build queue items. 
/render APIs use priority 10, whereas /arbitrary use priority 0, to prevent thumbnail downloads from holding up website loading. The priorities can be adjusted later, with maybe some service types being given higher priority than others.
 2022-02-11 15:08:12 +00:00
CalDescent
2343e739d1 Handle case where a file cannot be unzipped. 2022-02-11 14:35:46 +00:00
CalDescent
fc82f0b622 Use 5 builder threads, so that one slow resource (e.g. a thumbnail) doesn't hold up the other queued build items. 
This can be replaced with a task-based approach longer term.
 2022-02-11 13:58:45 +00:00
CalDescent
c0c50f2e18 Updated bootstrap hosts 2022-02-11 13:33:25 +00:00
CalDescent
9332d7207e Fixed bug in cache clearing logic, which was often preventing resource updates from being detected. 2022-02-10 09:22:54 +00:00
CalDescent
a8c79b807b Discard any uncommitted changes as a result of the higher weight chain detection 2022-02-10 08:16:30 +00:00
CalDescent
2637311ef5 Prevent potential ConcurrentModificationException in the build queue 2022-02-09 20:20:30 +00:00
CalDescent
06b5b8f793 Reduced time between processing build tasks, to prevent builds with invalid criteria from holding up legitimate builds too much. 2022-02-09 20:17:56 +00:00
CalDescent
61f58173cb Revert "Removed transaction caching. Can be reintroduced later." 
This reverts commit 9804eccbf0.
 2022-02-09 19:46:20 +00:00
CalDescent
b7b66f6cba Revert "Removed getWalletTransactions() synchronization. Again, can be re-added later." 
This reverts commit 70c864bc2f.
 2022-02-09 19:46:16 +00:00
CalDescent
dda2316884 Revert "Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance." 
This reverts commit d7658ee9f9.
 2022-02-09 19:46:10 +00:00
CalDescent
b782679d1f Revert "Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj."" 
This reverts commit 214f49e356.
 2022-02-09 19:46:06 +00:00
CalDescent
b0f19f8f70 Merge branch 'block-minter-updates' 
# Conflicts:
#	src/main/java/org/qortal/controller/arbitrary/ArbitraryDataFileRequestThread.java
 2022-02-09 19:42:39 +00:00
CalDescent
de5f31ac58 Don't process file hashes if we're stopping 2022-02-09 19:40:20 +00:00
CalDescent
214f49e356 Revert "Calculate wallet balances from the transactions (ElectrumX) rather than using bitcoinj." 
This reverts commit 892612c084.

# Conflicts:
#	src/main/java/org/qortal/crosschain/Bitcoiny.java
 2022-02-08 18:29:32 +00:00
CalDescent
d7658ee9f9 Try a lookahead size of 20 (instead of 3) when asking Bitcoinj for the balance. 2022-02-08 18:27:44 +00:00
CalDescent
70c864bc2f Removed getWalletTransactions() synchronization. Again, can be re-added later. 2022-02-08 18:27:08 +00:00
CalDescent
9804eccbf0 Removed transaction caching. Can be reintroduced later. 2022-02-08 18:26:15 +00:00
CalDescent
d1f24d45da Added defensiveness in convertToSimpleTransaction() 2022-02-08 18:24:42 +00:00
CalDescent
9630625449 Rework of processIncomingTransactionsQueue() so that it no longer holds the lock while processing. 
This should fix an issue where network threads could be blocked when new transactions arrived, due to waiting for the incomingTransactions lock to free up.
 2022-02-08 09:18:14 +00:00
CalDescent
b72153f62b Renamed main thread from "Controller" to "Qortal" 2022-02-08 09:02:20 +00:00
CalDescent
0a88a0c95e Perform the base58 decoding outside of the arbitraryDataFileHashResponses lock, to reduce the amount of waiting around by other threads. 2022-02-08 08:45:58 +00:00
CalDescent
ab4ba9bb17 Don't re-fetch unconfirmed transactions that are already in the queue 2022-02-08 08:36:45 +00:00
CalDescent
a49218a840 Optimized ArbitraryDataFileRequestThread - only start a database transaction when there's something to process. 2022-02-07 22:06:45 +00:00
CalDescent
b6d633ab24 Break out of incoming transactions processing loop if we need to sync. 2022-02-07 22:05:13 +00:00
CalDescent
133943cd4e Reduce log spam 2022-02-07 22:03:41 +00:00
CalDescent
f8ffb1a179 Updated thread names 2022-02-07 22:03:26 +00:00
CalDescent
41c4e0c83e Merge branch 'master' into block-minter-updates 2022-02-06 18:40:32 +00:00
CalDescent
99f6bb5ac6 Reorganized some controller methods. 2022-02-06 18:32:43 +00:00
CalDescent
3e0306f646 Increased minPeerConnectionTime and maxPeerConnectionTime to reduce the chances of forced connections during relays. 
An alternate option would be to avoid force disconnecting while relays are in progress, but some nodes could have active relays 100% of the time and therefore would never recycle their peers. So it is simpler to just increase the average peer connection time for everyone.
 2022-02-06 17:29:00 +00:00
CalDescent
84e4f9a1c1 Rework of arbitraryRelayMap to keep track of multiple responses. 
Previously, only one peer's response for a hash would be remembered, even if multiple others reported back too. This would cause useful mapping to be lost.
 2022-02-06 17:20:01 +00:00
CalDescent
cd5ce6dd5e Don't remove from the relay map after a file is requested, as it may be needed by other peers. 
It will be cleaned up automatically after 60 seconds, so it is best to keep the data intact until then.
 2022-02-06 16:04:54 +00:00
CalDescent
9ec4e24ef6 Slightly optimized logic in fetchArbitraryDataFiles() 2022-02-06 15:45:40 +00:00
CalDescent
fa447ccded Builder thread updates. 2022-02-06 15:37:21 +00:00
CalDescent
ef838627c4 Stop asking for hashes from a peer if one fails. 
This fixes the request looping that occurs on when a peer is unable to serve files.
 2022-02-06 15:37:08 +00:00
CalDescent
b8aaf14cdc Introduced ArbitraryDataFileRequestThread to allow for multiple concurrent file requests. 
This is likely a short term solution (to allow existing code to be repurposed) until replaced with a task-based approach, as this will allow for a much greater number of threads.
 2022-02-06 15:34:06 +00:00
CalDescent
2740543abf Added "async" and "attempts" parameters to GET /arbitrary/{service}/{name}* endpoints. 
async = fail immediately with 404 if missing, and request in the background
attempts = the number of times to request the data (synchronous mode only for now)
 2022-02-06 13:04:58 +00:00
CalDescent
3c526db52e Fixed bug in build manager which would prevent future builds until the core was restarted. 2022-02-06 13:03:01 +00:00
CalDescent
cfe0414d96 Small rework of invalidUnconfirmedTransactions to specify the expiry time instead of the time added. 
This allows TIMESTAMP_TOO_OLD transactions to be tracked for a shorter time (10 minutes) than the other invalid transactions (60 minutes). Should reduce network traffic and db load around the time that transactions are expiring, as there is a lag before they are noticed and removed from each node. Due to the variance, it could cause other peers to request them again after deleting. They are now ignored for 10 minutes to avoid request spam.
 2022-02-06 12:35:41 +00:00
CalDescent
08e06ba11a Fixed bugs preventing invalidUnconfirmedTransactions from working as intended. 2022-02-06 12:09:44 +00:00
CalDescent
8c03164ea5 Don't add expired transactions to invalidUnconfirmedTransactions, as there is no need to keep track of these. 2022-02-06 11:55:07 +00:00
CalDescent
0fe2f226bc Added invalidUnconfirmedTransactions map 
An incoming invalid unconfirmed transaction will be added to this map if its timestamp is more than 30 minutes old. This should allow enough time and opportunities for it to be imported and included in a block (allowing for re-orgs which could switch its status from invalid to valid).

Once added, it will be removed after an hour to allow for another chance to be requested from any peers that still have it. If invalid again, it's added back to the map for another hour.

This fixes a 24 hour long loop, where invalid transactions are requested over and over from peers that have already imported them. It could be improved further by periodically removing invalid unconfirmed transactions from the database, but this will be a higher risk.

The results of this feature should be less network traffic, and less blockchain locks (which should ultimately increase the responsiveness of the synchronizer).
 2022-02-06 11:23:28 +00:00
CalDescent
55b5702158 Invalidate last low weight block signature whenever the previous block data changes. 2022-02-05 17:54:36 +00:00
CalDescent
a4cbbb3868 Moved block minter sleep to later in the process, otherwise it can remain there for longer than expected. 2022-02-05 17:54:36 +00:00
CalDescent
816b01c1fc Fixed issue in rebase 2022-02-05 17:54:36 +00:00
CalDescent
483e7549f8 Revert "Moved log from INFO to DEBUG, as now the synchronizer is on its own thread it can occur more often than before." 
This reverts commit e2e87766fa.
 2022-02-05 17:54:35 +00:00
CalDescent
60d71863dc Allow 3 seconds for the synchronizer to obtain a blockchain lock, to reduce missed attempts. 2022-02-05 17:54:35 +00:00
CalDescent
170244e679 More work on higher weight chain detection in the block minter. 
Added 30 second timeout, so that any errors should self correct.
 2022-02-05 17:54:35 +00:00
CalDescent
472e1da792 Added debug level logging in higherWeightChainExists() for better visibility. 2022-02-05 17:54:35 +00:00
CalDescent
cbf03d58c8 Made synchronizer method public as it is now also used by the block minter. 2022-02-05 17:54:35 +00:00
CalDescent
ba41d84af9 Initial attempt to avoid an unnecessary block submission if one of our peers already has a higher weight chain. 2022-02-05 17:54:35 +00:00
CalDescent
98831a9449 Break out of the various loops in the cleanup manager if the thread is stopping. 2022-02-05 17:53:49 +00:00
CalDescent
9692539a3f Don't include fee in balance calculation (it looks like it could be double counting at the moment). 2022-02-05 17:24:33 +00:00
CalDescent
76df332b57 Check for null IP address before notifying of an external IP update. 2022-02-05 11:51:54 +00:00
CalDescent
c6405340bc minAccountLevelForBlockSubmissions reduced from 6 to 5 2022-02-05 11:33:28 +00:00
CalDescent
775e3c065e Invalidate ElectrumX transactions cache when switching accounts. 2022-02-05 10:23:25 +00:00
CalDescent
8937b3ec86 Don't allow duplicate transaction in the incoming transactions queue. 
This should reduce database load slightly, as it won't have to check the same transaction multiple times in each batch.
 2022-02-05 10:19:26 +00:00
CalDescent
3fbb86fded Added indexes, to make looking up name transactions by name around 5x faster. 2022-02-04 16:27:31 +00:00
CalDescent
0cf2f7f254 Missing import from last commit 2022-02-04 16:18:00 +00:00