1246 Commits

Author SHA1 Message Date
CalDescent
45b0d9e19b Added more initial peers, submitted by CWDSYSTEMS 2021-06-19 16:19:53 +01:00
CalDescent
026a4b896c Added BTC and LTC electrum nodes submitted by CWDSYSTEMS 2021-06-19 14:50:18 +01:00
CalDescent
78237fcd11 Don't intentionally disconnect peers if we are currently syncing with them. 2021-06-19 13:12:16 +01:00
CalDescent
73cc3dcb92 Force a disconnect of each peer when the connection age reaches the maximum allowed time.
Connection limits are defined in settings (denominated in seconds):
"minPeerConnectionTime": 120,
"maxPeerConnectionTime": 3600

Peers will disconnect after a randomly chosen amount of time between the min and the max. The default range is 2 minutes to 1 hour, as above.

This encourages nodes to connect to a wider range of peers across the course of each day, rather than staying connected to an "island" of peers for an extended period of time. Hopefully this will reduce the amount of parallel chains that can form due to permanently connected clusters of peers.

We may find that we need to reduce the defaults to get optimal results, however it is best to do this incrementally, with the option for reducing further via each node's settings. Being too aggressive here could cause some of the earlier problems (e.g. 20% missing blocks minted) to reappear. We can re-evaluate this in the next version. Note that if defaults are reduced significantly, we may need to add code to prevent this from happening mid-sync. With higher defaults, this is less of an issue.

Thanks to @szisti for supplying some base code for this commit, and also to @CWDSYSTEMS for diagnosing the original problem.
2021-06-19 13:03:46 +01:00
CalDescent
4cff03e7fe Include "size" value in the "Synchronized with peer" logs.
This indicates the size of the re-org/rollback that was required in order to perform this sync operation. It is only included if it's greater than 0 blocks.
2021-06-19 09:04:14 +01:00
CalDescent
8e35f131d5 Removed debugging log that wasn't intended to be committed. 2021-06-18 08:33:51 +01:00
CalDescent
aafb9d7e4f Include running total in "Sent X bytes" log entry. 2021-06-18 08:05:35 +01:00
CalDescent
652f30bdbd Added DATA_FILE and GET_DATA_FILE message types. 2021-06-18 08:02:57 +01:00
CalDescent
f4ba7b2a0c Added onNetworkGetDataFileMessage() handler 2021-06-18 08:02:13 +01:00
CalDescent
592490d709 Added GET /data/file/frompeer API endpoint
This requests a file from the supplied peer address, and stores a copy locally if successful. Still a work in progress.
2021-06-18 07:59:46 +01:00
CalDescent
5ac676d201 Added DataFileMessage and GetDataFileMessage, used for requesting and sending files between peers. 2021-06-17 19:05:00 +01:00
CalDescent
abfe0a925a More DataFile methods and improvements 2021-06-17 18:20:42 +01:00
CalDescent
fa11f4f45b Moved DataFileChunk(byte[] fileContent) constructor to the superclass so it can be used by regular data files too. 2021-06-17 18:20:11 +01:00
CalDescent
1e8dbfe4b7 Delete chunk if it fails the hash validation in the constructor. 2021-06-17 18:18:19 +01:00
CalDescent
f82f2bd287 Added DELETE /data/file API endpoint
This deletes a file referenced by a user supplied SHA256 digest string (which we will use as the file's "ID" in the Qortal data system). In the future this could be extended to delete all associated chunks, but first we need to build out the data chain so we have a way to look up chunks associated with a file hash.
2021-06-16 20:03:15 +01:00
CalDescent
76742c3869 Removed .dat extension, and use an extra level in the directory structure (data/aB/cD/aBcDeF... etc) 2021-06-16 19:49:12 +01:00
CalDescent
9407e7e418 Data storage location moved to settings ("dataPath") 2021-06-16 19:22:50 +01:00
CalDescent
120552b36e Added resource file missing from last commit. 2021-06-16 19:13:24 +01:00
CalDescent
9fb58c7ae3 Added the beginnings of an upload API, with some basic data file management. 2021-06-16 19:10:35 +01:00
CalDescent
b917da765c Removed block 212937 2021-06-15 09:29:07 +01:00
CalDescent
cc59510cd0 Removed all cross-chain code. It's not needed for data nodes. 2021-06-15 09:27:05 +01:00
CalDescent
86aab7023c Initial settings for data node development. Most to be decided later. 2021-06-14 19:40:37 +01:00
CalDescent
904be3005f Enable fast sync by default. 2021-06-12 13:07:25 +01:00
CalDescent
95eaf4c887 Merge branch 'master' into sync-multiple-blocks 2021-06-12 11:15:28 +01:00
CalDescent
e3923b7b22 Fixed issue causing frequent disconnects (found by szisti)
When sending or requesting more than 1000 online accounts, peers would be disconnected with an EOF or connection reset error due to an intentional null response. This response has been removed and it will instead now only send the first 1000 accounts, which prevents the disconnections from occurring.

In theory, these accounts should be in a different order on each node, so the 1000 limit should still result in a fairly even propagation of accounts. However, we may want to consider increasing this limit, to maximise the propagation speed.

Thanks to szisti for tracking this one down.
2021-06-11 19:10:04 +01:00
CalDescent
a43993e3ec Use placeholder build timestamp and build version when building without mvn package (e.g. from within an IDE)
This fixes an exception thrown when running directly in IntelliJ, as previously we relied on mvn package to parse the commit hash and timestamp.
2021-06-11 19:01:35 +01:00
CalDescent
df47f5d47b Merge branch 'master' into sync-multiple-blocks 2021-06-06 10:35:47 +01:00
CalDescent
319e64bacc Defend against an edge case NPE in the chat messages websocket. 2021-06-06 10:34:20 +01:00
CalDescent
76e1de38e8 Workaround for issue where sometimes an AT stays in "TRADING" mode even after it is marked as finished. This caused Bob's tradebot to enter BOB_REFUNDED mode instead of redeeming the LTC. This workaround treats "TRADING" as "REDEEMED" as long as the AT is finished. It will still enter the BOB_REFUNDED state if the AT's trade state is "REFUNDED" or "CANCELLED", to prevent it trying to redeem LTC without the secret. Longer term we need to prevent the AT itself from getting in this state to begin with, but this should at least solve the LTC redemption problem that occurs as a result. 2021-06-05 12:31:49 +01:00
CalDescent
1648a74ed7 Removed code which auto deletes trade bot data if it can't locate the AT after 24 hours. It's not a good idea to ever delete trade bot data, since it can contain private keys necessary to redeem or refund LTC. We have seen at least one instance of this where the trade bot data was deleted for an active trade. We still have the auto backup in these cases, so the keys are recoverable, but it's safest to avoid any auto deletions. 2021-06-05 11:25:05 +01:00
CalDescent
c63a7884cb Limit to 10 untrimmed blocks per response, as they are larger than the trimmed ones. 2021-06-02 09:10:25 +01:00
CalDescent
cffbd41f26 Reduce memory allocations in onNetworkGetBlocksMessage 2021-06-02 09:09:06 +01:00
CalDescent
c443187d0b Reduce log spam by logging the total number of expired unconfirmed transactions that are deleted, rather than each one individually. The individual deletion logs have been moved from INFO to DEBUG. 2021-06-01 20:06:37 +01:00
CalDescent
8c305d8390 Reduced log levels of recent synchronizer / controller log additions from INFO to DEBUG.
These are no longer necessary now that we have achieved good stability in the network.
2021-06-01 08:39:38 +01:00
CalDescent
2ceba45782 Fast sync default blocks per request increased to 100. 2021-05-30 14:57:58 +01:00
CalDescent
ed423ed041 Increased MAX_DATA_SIZE and SYNC_BATCH_SIZE, to increase the effectiveness of the batch sync. 2021-05-30 14:54:13 +01:00
CalDescent
f58a52eaa4 Further work to increase the response timeout when requesting multiple blocks. 2021-05-30 13:10:38 +01:00
CalDescent
688404011b Relocate FETCH_BLOCKS_TIMEOUT to Peer.java and use a static import. 2021-05-30 10:08:50 +01:00
CalDescent
8881e0fb75 Merge branch 'master' into sync-multiple-blocks 2021-05-30 09:57:56 +01:00
CalDescent
61de7e144e Merge branch 'networking' into sync-multiple-blocks
# Conflicts:
#	src/main/java/org/qortal/network/Peer.java
2021-05-30 09:57:35 +01:00
CalDescent
815934ff5c Added GET /crosschain/htlc/redeemAll/LITECOIN API
This loops through all sell orders and attempts to redeem the LTC from each one. It will return true if at least one was redeemed, or false if none are available to be redeemed. Details are logged to the log.txt file rather than returned in the API response.
2021-05-29 19:43:08 +01:00
Istvan Szabo
d52875aa8f Added logs to intentional disconnects 2021-05-28 16:06:27 +01:00
Istvan Szabo
9027cd290c Filter out on demand connections when using fixed network 2021-05-28 14:47:30 +01:00
Istvan Szabo
58a7203ede Support for configuration based fixed network 2021-05-28 14:47:30 +01:00
Istvan Szabo
bb0269f484 Converted time format 2021-05-28 08:53:01 +01:00
Istvan Szabo
1adc9349fc Added connection age to connected peers dto 2021-05-28 08:04:57 +01:00
Istvan Szabo
06215c83f2 Reduced log levels 2021-05-27 10:48:17 +01:00
Istvan Szabo
de4b1c8f09 Removed missed functional change 2021-05-27 09:15:32 +01:00
Istvan Szabo
265d40f04a Code formatting and logging changes for networking 2021-05-27 09:03:18 +01:00
szisti
b64e52c0c0
Automated testing (#38)
* added basic workflow

* Testing workflow

* renamed workflow file

* Disabled extremely slow test

* Disabled currently failing tests

* Added jacoco and updated workflow

* We cannot run gui tests headless

* Fixed jacoco configuration

* Updated job name in the workflow

* Adjusting workflow

* Testing maven caching

* Added logging for one of the jacoco related issues

* Updated coverage logging

Co-authored-by: Istvan Szabo <istvan.szabo@betvictor.com>
2021-05-26 11:27:46 +01:00