Commit Graph

391 Commits

Author SHA1 Message Date
catbref
482947dbf4 Share leftover block reward to founders, regardless whether online
Added support in AccountRepository to fetch account using flags.

+ renamed some local variables in some Block methods to avoid clashes
2019-11-12 14:55:46 +00:00
catbref
49ac7921a1 Fix new account level not taking effect in block rewards
Added encoded online accounts list decoder.
2019-11-11 17:55:58 +00:00
catbref
30df320e7f Redoing account balances with block height.
*** WARNING ***
Possible block reward bug in this commit. Further investigation needed.

Reverted AccountBalances back to height-less form.
Added HistoricAccountBalances table that is populated via trigger on AccountBalances.
This saves work when performing common requests for latest/confirmed balances,
shunting the extra work to when requesting height-related account balances.

Unified API call GET /addresses/balance/{address} by having address/assetId/height as
query params.

Simpler call for fetching legacy QORA holders during block rewarding.

Improved SQL for fetching asset balances, in all conditions,
e.g. with/without filtering addresses, with/without filtering assetIds,
etc.

Unit test for above to make sure query execution is fast enough.
(At one point, some SQL query was taking 6 seconds!)

Added optional 'height' Integer to AccountBalanceData, but this
is not populated/used very often.

HSQLDBAccountRepository.save(AccountBalanceData) now checks zero balance saves
to see if the row can be deleted instead. This fixes a lot of unhappy tests
that complain that there are residual account balance rows left after
post-test orphaning back to genesis block.

Yet more tests.
Removed very old 'TransactionTests' which are mostly covered in more specific tests elsewhere.
Added cancel-sell-name test from above.

Fixed AssetsApiTests to check for QORT not QORA!

Changed hard-coded assetIDs in test.common.AssetUtils in light of new LEGACY_QORA & QORT_FROM_QORA genesis assets.

Some test blockchain config changes.
2019-11-08 17:30:09 +00:00
catbref
31cbc1f15b Account assets balances now height-dependant. QORT-from-QORA block reward fixes. 2019-11-07 15:53:37 +00:00
catbref
f5918bd9bf Added range-check on online account timestamps 2019-11-06 10:11:10 +00:00
catbref
00aee1458e Higher account levels more likely to win blocks
Also:

RewardShareKeys app now supports only one arg (minter private key)
in self-reward-share mode, where recipient public key is derived
from minter private key.

Added methods to Account for returning 'effective' minting level
where minting level for founders is read from blockchain config.
(Or returns zero if unable to mint).

Changed two Block constructors into static methods that return
a new Block as there was way too much work being done to really
be called a constructor, especially with all the opportunities
to throw an exception too.

Main blockchain config updated to reflect near-launch version.

Added/changed blockchain weight tests to check block winning
based on higher account levels.
2019-11-06 09:47:10 +00:00
catbref
ebc2ee6ea9 Rework distributing block reward to legacy qora holders
Fix related unit tests.

Fix assetID support in GENESIS transactions.
2019-11-01 10:01:31 +00:00
catbref
62e2fd759c Fixing unit tests 2019-10-30 13:19:56 +00:00
catbref
fef16e7620 Self-reward-share tests and fixes.
Added Transaction.isFeeValid() to allow transaction subclasses to override and allow zero fees, etc.

Added tests to cover self-reward-shares, including zero fee scenario.

Set 'dilbert' test account to level 8 in test genesis block.

Removed leftover mention of "previous_level" from HSQLDBAccountLevelTransactions,
and AccountLevelTransactionData. (Previous level makes no sense as ACCOUNT_LEVEL
transactions are genesis-block only).

Fixed some incorrect uses of PrivateKeyAccount.getSharedSecret() to
PrivateKeyAccount.getRewardSharePrivateKey() in tests.
2019-10-30 10:23:59 +00:00
catbref
491e79b8e6 ProxyForging->RewardShare massive refactor & more...
Unified terminology from block "generator", "forger", etc. to "minter"

Unified terminology "proxy forger" to "reward share" as it was
incorrect, or at least highly ambiguous, which account had which role.

AccountRepository.findRewardShares() has different arg order!

Account.canMint() now returns true if account has 'founder' flag set.
Added Account.canRewardShare() which returns whether acocunt can create
a reward-share (e.g. level 5+ or founder).

Fixed HSQLDBAssetRepository.getAllAssets() which had incorrect
resultSet column indexes.

Removed old traces of EnableForging transaction.

ACCOUNT_LEVEL and ACCOUNT_FLAGS (genesis-block-only transaction types)
now set target account's last-reference. This is allow later REWARD_SHARE
transactions in genesis block and post-genesis transactions by that account.

REWARD_SHARE transactions are now FREE, but only if minter is also recipient.
If a self-reward-share already exists, then unless share-percent is zero (to
terminate reward-share), any subsequent self-reward-share is invalid.

Updated SysTray i18n properties file.

BlockChain config file requires 'minAccountLevelToRewardShare' and optional
'minAccountLevelToMint'.

Added potential, but currently unused, memory-hard PoW algorithm.

Fixed/removed/disabled some unit tests.
BlockMinter.generateTestingBlock asks Controller to pretend mintingAccount is 'online'.
More testing needed!
2019-10-29 17:46:55 +00:00
catbref
843aad4930 Update BouncyCastle Ed25519 to X25519 key conversion shim 2019-10-25 13:22:08 +01:00
catbref
5798c69449 Code clean-up thanks to SonarLint & bugfix for ByteArray::compareTo! 2019-10-15 17:10:13 +01:00
catbref
a3751823eb Improve code dealing with increase account level due to generated blocks & add orphan equivalent. 2019-10-14 16:37:48 +01:00
catbref
6ba24e1820 Minor code tidying 2019-10-14 11:05:07 +01:00
catbref
319bfc8d75 Add account level change due to blocks generated.
Removed ENABLE_FORGING and related.

Still to do: 'orphan' version of Block.increaseAccountLevels
2019-10-10 13:52:00 +01:00
catbref
54d49e0f1d More work on block rewards in relation to account level/legacy qora held.
Rename Asset.QORA to Asset.QORT so we can also have Asset.LEGACY_QORA
as another hard-coded asset.

Add "is unspendable" aspect to assets where only the asset owner can
transfer/pay asset to other people. Asset trading is barred regardless,
as is use of asset for ATs.

Added "initial level" to account data in preparation for accounts levelling
up from generating blocks.

Added distribution/removal of block reward based on legacy-QORA held.

Removed "previous level" from ACCOUNT_LEVEL transactions as they're
only ever valid in genesis block and so previous level is never needed.
2019-10-08 14:58:00 +01:00
catbref
a1e83109a8 Qortal-style block rewards 2019-10-03 14:20:46 +01:00
catbref
a0be1273c7 Remove max coin supply aspects
Maximum amount/price for selling a name limited to 10 billion,
with 8 decimal prices, so encoding fits within 8 bytes.
2019-10-02 13:24:17 +01:00
catbref
4062bdb7bb Workaround for Handshaking sometimes not correctly dealing with connect-to-self. 2019-10-01 13:15:21 +01:00
catbref
1016d0ca16 Fix Synchronizer bugs
Synchronizer incorrectly tried to compare chain weights in the case
where we had no extra blocks after common block, and thus no blocks
to actually compare.

During chain compare, when more block summaries are needed from peer,
the block signature sent to peer wasn't updated from the last batch
and so the same common block signature was sent. This caused the
code to incorrectly bail out with
"Peer respond with invalid block summary" error as the block heights
didn't match.

Post chain-compare/orphan (if any), and when fetching block signatures,
the code referenced peerBlockSummaries where it should have been
referencing peerBlockSignatures instead. This caused to code to incorrectly
bail out with the "Peer failed to respond with more block signatures..."
error.
2019-10-01 13:02:23 +01:00
catbref
381c032cec Fixed Network's broken choice of next connectable peer.
Slight improvement to comment in Peer.
2019-10-01 12:59:54 +01:00
catbref
568a1f8a30 Migration to Java 11
Updated pom.xml.
Updated dependencies, including various minor code mods, esp. bitcoin-related.

Starts up and talks to Java 8 nodes!

Dev environment should be at least Eclipse 4.11 with m2e 1.9.1+
2019-09-30 18:06:00 +01:00
catbref
305bb38446 Remove "generating balance", and all related aspects/code.
Block forging is now allowed by level 1+ accounts,
instead of accounts with "minting flag" set.
2019-09-27 15:16:12 +01:00
catbref
4c6656dd17 New synchronizer and other improvements
API call GET /addresses/online reports online accounts,
including both addresses relating to the proxy-forge public key.

New PeerChainTipData class to replace the broken "peer data lock"
that was supposed to make sure peer's last height/blockSig/timestamp
were all in sync. Now peer's chain tip data is a single object
reference that can be replaced in one go.

Removed pointless API calls /blocks/time and /blocks/{generatingbalance}.

Various changes, mostly in Block class, to do with switching to BlockTimingByHeight
from old min/max block time.

New block 'weight' based on number of online accounts
and 'distance' of perturbed generator public key from 'ideal' public key
(for that particular block's height).

New sub-chain 'weight' based on accumulating block weights,
currently by shifting previous accumulator left by 8 bits then
adding next block's weight.

More validation of BlockChain config. Helpful for debugging, probably
not very useful to end-users.

BlockGenerator now uses unified Peer predicates from Controller, like:
Controller.hasMisbehaved, Controller.hasNoRecentBlock, etc.

Controller now keeps a list of chain-tip signatures that are for inferior
chains, so it doesn't try to synchronize with peers with inferior chains.
(This list is wiped when node's blockchain changes/block is generated).

Controller now asks Gui to display error box if it can't parse Settings.

Controller.potentiallySynchronize() does more filtering of potential peers
before calling actuallySynchronize(). (Mostly moved from Synchronizer,
so now we expect actuallySynchronize() to do something rather than bail
out because it doesn't like the peer after all).

If synchronization discovers that peer has an inferior chain,
then Controller notifies that peer of our superior chain, to help keep
the network in sync.

Renamed OnlineAccount to OnlineAccountData, as it is in package org.qora.data
after all...

Synchronizer reworked to request block summaries so it can judge which chain
is better, and hence whether to sync with peer or abort.

Slight optimization of Peer.readChannel() to exit earlier if no more network
messages can be extracted from buffer.

More tests.
Improved documentation and logging.
2019-09-26 17:43:50 +01:00
catbref
14e9ae8887 ExecuteProduceConsume no longer spawns a pointless, new thread if we have an excess of non-consuming threads 2019-09-26 17:25:05 +01:00
catbref
344b9436ca Much improved ByteArray, with unsigned compareTo() & "raw" renamed to "value" 2019-09-26 17:23:46 +01:00
catbref
40d879813d Added exists() test to BlockRepository & HSQLDB implementation 2019-09-26 17:22:03 +01:00
catbref
c00481a750 Added fatal error pop-up and exit to Gui 2019-09-26 17:20:57 +01:00
catbref
a1cfe31574 Block size and transaction expiry period now in BlockChain config
Block.MAX_BLOCK_BYTES now BlockChain.getMaxBlockSize()

Network.MAXIMUM_MESSAGE_SIZE now Network.getMaxMessageSize() as
it depends on block size (above).
2019-09-23 17:01:25 +01:00
catbref
e009147956 Additional checks on byte lengths in BlockTransformer, especially before buffer allocation 2019-09-23 16:46:16 +01:00
catbref
aa54ec212f Convert TransactionTransformer LOGGER.trace to use lambda for speed 2019-09-23 16:45:26 +01:00
catbref
4b5ed79c5a Added testNtpOffset settings field (+ minor settings docs changes) 2019-09-23 16:44:47 +01:00
catbref
0dd5b1e65a More work on online accounts / blocks
Block data now includes number of online accounts, as encoded online account indexes can't be
validated by ConciseSet it seems.
Corresponding changes to repository, transformer, block validation, data object, block summaries...

Block timestamps are now calculated using parent block data and generator's public key,
instead of old qora1 generating balance code.

Generators are valid to forge if they have forging flag enabled. This will probably change
to an account-level check in the near future.

Added trimming of old online accounts signatures from blocks.

Tidied up SysTray/BlockGenerator generation enabled/possible flag.

Although we perform online accounts tasks (currently) every 10 seconds,
only broadcast our online accounts every 60 seconds.

In Controller.main(), if args are present then use first as a filename to
settings JSON file (overriding the default filename).

Still to do: change Block/BlockChain/Synchronizer to prefer blocks with more online accounts,
failing that use generator nearest 'ideal', etc.
2019-09-18 14:49:47 +01:00
catbref
aa81c86cf1 Rename timestampSignatures to onlineAccountsSignatures 2019-09-16 11:11:42 +01:00
catbref
504cfc6a74 Interim commit: online accounts and account levels
Safety commit in case of data loss!

Lots of changes to do with "online accounts", including:

* networking
	+ GET_ONLINE_ACCOUNTS * ONLINE_ACCOUNTS messages & handling
	+ Changes to serialization of block data to include online accounts info
* block-related
	+ Adding online accounts info when generating blocks
	+ Validating online accounts info in block data
	+ Repository changes to store online accounts info
* Controller
	+ Managing in-memory cache of online accounts
	+ Updating/broadcasting our online accounts
* BlockChain config

Added "account levels", so new code/changes required in the usual places, like:

* transaction data object
* repository
* transaction transformer
* transaction processing
2019-09-13 14:32:32 +01:00
catbref
2cc926666b Qortal-specific Settings changes
Mostly changes to default port numbers
2019-09-13 14:32:32 +01:00
catbref
b9a96019a4 Qortal-specific Network changes
Changed message magics
Changed initial peers
Corrected filtering of potential peers for new connection.
2019-09-13 14:32:32 +01:00
catbref
cfc86d567c Qortal splash image and icons 2019-09-13 14:32:32 +01:00
catbref
978a22cab3 Added ProxyKeys standalone for generating proxy keys 2019-09-13 14:32:32 +01:00
catbref
158631e68a Fix forging bit mask for account flags 2019-09-13 14:32:32 +01:00
catbref
8149e18f49 Fix incorrect tx type values in transaction layouts 2019-09-13 14:32:32 +01:00
catbref
a9122cc6cb Improve choosing which peer to connect to 2019-08-21 09:45:41 +01:00
catbref
f7c6978151 Add threading to VanityGen 2019-08-21 09:44:52 +01:00
catbref
2889d04633 Fix incorrect orphaning of BUY_NAME transactions.
Orphaning a BUY_NAME transaction would not reinstate the
sale price. Sale price could be nullified by (e.g.) orphaning
a SELL_NAME transaction.

Also added test case to cover above and other test-related support,
e.g. test-mode in NTP.
2019-08-16 11:48:30 +01:00
catbref
f83dc26ae0 ExecuteProduceConsume and networking improvements
Added "volatile" to more fields, for thread-safety on reads.
Changes to field values are done inside synchronized blocks
so no need for AtomicInteger/AtomicBoolean. (Could be changed
in the future to show intention/readability though).

Added more statistics (tasks produced/consumed).

Limited Network's EPC executor to 10 threads max.
2019-08-16 08:28:26 +01:00
catbref
c597f11c37 Fix fetching asset-related transactions from DB where asset might not exist.
Some asset-related transactions (CREATE_ASSET_ORDER and TRANSFER_ASSET)
also try to fetch asset names at the same time.

If one of these transactions, and a corresponding ISSUE_ASSET transaction,
have been orphaned then it's possible that the asset no longer exists.

Thus the SQL "JOIN" fails during transaction retrieval, causing an error.

Changing the table-join to "LEFT OUTER JOIN" makes the asset name aspect
optional. Repercussions might be nameless assets when fetching transaction
info via API.
2019-08-16 08:19:12 +01:00
catbref
3b3888ae0d Fix asset ordering for old-pricing orders 2019-08-16 08:18:34 +01:00
catbref
6abc3f4d39 Networking improvements
Fix issue where sometimes the channelSelector.select(1000) would
block processing of queued messages.

Improve support with older v1 peers.
2019-08-14 15:15:20 +01:00
catbref
ea3528015a Add support for v1-protocol BLOCK message 2019-08-14 15:14:47 +01:00
catbref
84e812484b Fix GenesisBlock ISSUE_ASSET transactions for v1 chains 2019-08-14 15:14:28 +01:00
catbref
e631e69fa1 Use bindAddress from Settings for UI, API and P2P. 2019-08-14 15:11:20 +01:00
catbref
8a0d93f304 Fix up after epic git history rebuild due to LFS issue 2019-08-13 10:27:28 +01:00
catbref
fa0b7615a6 Fixing peer disconnections due to slow processing & Transaction expiry
Transaction expiry wasn't happening. Use NTP.getTime to check whether
transactions have expired. Also reject expired transactions when trying
to add them to unconfirmed pool.

Sometimes producing a task took way too long, causing massive
spikes in the number of threads and peer disconnections.

This is down to a repository pool exhaustion, so
RepositoryManager.getRepository() would block (for up to 5 minutes).

The key method at fault was Network.getConnectablePeer().

Various fixes:

NetworkProcessor's executor now reaps old threads after only 10 seconds
instead of the usual 60 seconds.

Change logging in Network to help diagnose disconnection and repository
issues.

RepositoryManager now has a tryRepository() call that is non-blocking
and returns null if repository pool is exhausted.

Repository pool size increased from default (10) to 100.

Pruning peers is now opportunistic, using tryRepository(), and returns
early if repository pool is exhausted.

getConnectablePeer() is now opportunistic, using tryRepository(), and
returns null (no peer candidate for connection) if repository pool
is exhausted.

Merging peers is not opportunistic, using tryRepository().

Peer ping interval increased from 8s to 20s.

HSQLDBRepositoryFactory now logs when getConnection() takes over 1000ms.

Added more trace-level logging to ExecuteProduceConsume to
highlight slow produceTask() calls.
2019-08-13 09:54:35 +01:00
catbref
1094db288e Catch SystemTray.isSupported errors (and act as if no support).
Symptoms were on Ubuntu with 8u222:

Exception in thread "Controller" java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
        at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:807)
        at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
        at java.awt.SystemTray.isSupported(SystemTray.java:219)
        at org.qora.gui.SysTray.<init>(SysTray.java:50)
        at org.qora.gui.SysTray.getInstance(SysTray.java:249)
        at org.qora.controller.Controller.updateSysTray(Controller.java:480)
        at org.qora.controller.Controller.run(Controller.java:368)

(Underlying cause not known)

This would cause Controller thread to silently exit, and so no
synchronization would occur. Node would still have connections
and thus happily generate its own blocks on its on fork.

Maybe other peers would try to sync with this node but would
likely reject this node's chain after a short while.
2019-08-13 09:54:25 +01:00
catbref
63b262a76e NTP and performance changes + fixes.
New NTP class now runs as a simplistic NTP client, repeatedly polling
several NTP servers and maintaining a more accurate time independent
of operating system.

Several occurrences of System.currentTimeMillis() replaced with NTP.getTime()
particularly where block/transaction/networking is involved.

GET /admin/info now includes "currentTimestamp" as reported from NTP.

Added support for block timestamps determined by generator, instead of
supplied by clock. (BlockChain.newBlockTimestampHeight - not yet activated).
Incorrect timestamps will produce a TIMESTAMP_INCORRECT Block.ValidationResult.

Block.calcMinimumTimestamp repurposed as Block.calcTimestamp for above.

Block timestamps are now allowed to be max 2000ms in the future,
was previously max 500ms.

Block generation prohibited until initial NTP sync.

Instead of deleting INVALID unconfirmed transactions in BlockGenerator,
Controller now deletes EXPIRED unconfirmed transactions every so often.
This also fixes persistent expired unconfirmed transactions on nodes
that do not generate blocks, as BlockGenerator.deleteInvalidTransactions()
was never reached.

Abbreviated block sigs added to log entries declaring a new block is generated
in BlockGenerator.

Controller checks for NTP sync much faster during start-up and SysTray's
tooltip text starts as "Synchronizing clock" until NTP sync occurs.
After NTP sync, Controller logs NTP offset every so often (currently every 5 mins).

When considering synchronizing, Controller skips peers that have the same block sig
as last time when synchronization resulted in no action, e.g. INFERIOR_CHAIN,
NOTHING_TO_DO and also OK. OK is included as another sync attempt would result in
NOTHING_TO_DO.
Previously this skipping check only happened after prior INFERIOR_CHAIN.

During inbound peer handshaking, if we receive a peer ID that matches an existing inbound
peer then send peer ID of all zeros, then close connection.
Remote end should detect this and cleanly close connection instead of waiting for handshake timeout.
Randomly generated peer IDs have lowest bit set to avoid all zeros.
Might need further work.

Networking doesn't connect, or accept, until NTP has synced.

Transaction validation can fail with CLOCK_NOT_SYNCED if NTP not synced.
2019-08-13 09:47:44 +01:00
catbref
05e491f65b Relax max clock offsets for block gen and peer connections.
It seems unachievable for nodes to keep their clocks accurate to
within 500ms. It is unclear whether this is due to Windows'
implementation of client NTP or because of terrible network
conditions in China.

So increasing max NTP offset to allow block generation from
500ms to 30s. Correspondingly increasing max peer timestamp
delta from 5s to 30s.

The block consensus algorithm will need to change in the near
future to address clock issues.
2019-08-13 09:18:30 +01:00
catbref
63036f3592 Increased NTP check interval from 5 minutes to 10 minutes.
Controller batches SysTray updates into one per second.

Block generation only allowed by Controller is clock
known to be accurate. ('not sure' stops generation).

NTP MAX_STDDEV increased from 25ms to 125ms to cater
for poorly connected nodes.

Controller sends peers list over outbound peer connections,
requests peers list from inbound peer connections.

When peer handshake completes, Network & Controller only send
initial messages over outbound peer connections.
This is to fix HEIGHT_V2 messages being processed out-of-order
breaking handshaking, as indicated by log entries like:

2019-07-26 16:16:35 DEBUG Network:702 - Unexpected HEIGHT_V2 message from xx.xxx.xx.xx:pppp, expected PROOF
2019-07-26 16:16:35 DEBUG Network:840 - Handshake completed with peer xx.xxx.xx.xx:pppp

Increased connection failure backoff from 1 minute to 5 minutes,
as handshake timeout is 1 minute and then nodes would immediately reconnect.

Changed default NTP servers from asia to cn.
2019-08-13 09:17:38 +01:00
catbref
33f3d35784 Fix DB backups not happening when no previous backup exists. 2019-08-13 09:15:30 +01:00
catbref
671dc5995a Don't allow block generation unless system clock is accurate.
Controller performs NTP check on startup (and every 5 minutes)
which determines whether block generation is allowed.

System Tray tooltip updated to reflect generating status.
Plus new translations.

Improved GuiTests.

BlockGenerator fetches forging accounts first, and sleeps
if none configured, which is less work than processing peer lists.
2019-08-13 09:15:21 +01:00
catbref
73e53120a9 Improved detection of inaccurate system clock & nagging.
Now uses several NTP servers to determine mean offset from
system clock to internet time.

If abs(offset) > 500ms or NTP service not running then
user is 'nagged' via system tray pop-up notification
with instructions on how to fix.

Also improved system tray translations!
2019-08-13 09:14:07 +01:00
catbref
0c17f9cff6 More useful Synchronizer logging + sync report tool.
Synchronizer logging now includes abbreviated block signature.
2019-08-13 09:08:57 +01:00
catbref
7042dd819f Include NTP checking/reconfigure tools + bump version to 1.3.1
SysTray pop-up menu now includes entry for launching https://time.is
so node owners can check their system clocks against internet time.

Windows installs also have additional systray menu entry which
runs ntpcfg.bat script, included in resources.
Also available as download via node-UI servlet,
e.g. http://localhost:9880/downloads/ntpcfg.bat

ntpcfg.bat reconfigures Windows Time Service with many NTP servers,
restarts the service, and also makes sure it auto-starts on boot.

Added DEBUG-level logging when rejecting nodes due to excessive
time difference (during PROOF handshake stage).

Bumped default settings values for minOutboundPeers from 10 to 20.
Bumped default settings values for maxPeers from 30 to 50.
2019-08-13 09:03:29 +01:00
catbref
9ee12f3e45 Reduce execute-produce-consume excessive thread spawning.
Defer the clearing of hasThreadPending flag until about to produce a task,
inside synchronized block.

This gives a new thread a chance to produce at least once before other threads
decide to spawn new threads.

Previously there could be an excessive number of unncessary threads,
all waiting for their initial attempt to produce a task.
2019-08-13 08:41:17 +01:00
catbref
b038e10ee7 Prevent multiple system tray icons
Added "synchronized" to SysTray.getInstance.

Also log launching of system tray icon.
2019-08-13 08:41:05 +01:00
catbref
964e0a02ca Fixes/improvements to networking
Reworked networking execute-produce-consume threading.
Some networking task were wrongly performed during 'produce' phase,
and some producing was happening in 'consume' phase (also corrected).

Peer connection tasks are rate-limited to 1 per second to reduce CPU thrashing.

Show P2P listen port in logs on startup.

Tests for general purpose ExecuteProduceConsume class to cover both
random task scenario and mass-ping scenario.
2019-08-13 08:40:50 +01:00
catbref
f8b496ff3c Convert SysTray to use JPopupMenu for Unicode support
Correct Systray_zh.properties to ISO 8859-1 instead of UTF-8.

Added SysTray test to GuiTests
2019-08-13 08:30:40 +01:00
catbref
6942c02700 Add ZH translations for SysTray pop-up menu.
Reduce log spam when SysTray can't open Node UI in browser,
e.g. no browser installed, or no association for URLs.
2019-08-13 08:29:16 +01:00
catbref
0d85a60c54 New network threading model
Instead of 3 threads per peer:

1. peer main thread
2. peer's unsolicited messages processor
3. peer pinger

We now use a Jetty-style Execute-Produce-Consume server threading model.

For 60 connected peers, we no longer have 180 threads but typically only
the usual ~6 threads.

Also in this commit:

* merging peers locking changed from lock() to tryLock()

* PROOF handshake maximum time difference increased from 2000ms to 5000ms

* Peers still handshaking after 60s are considered stuck and hence disconnected

* We now use NIO SocketChannels instead of raw sockets
2019-08-13 08:28:46 +01:00
catbref
67c245bb9d Don't attempt to synchronize with peers that we know have inferior chain 2019-08-13 08:28:08 +01:00
catbref
82910b6524 Add periodic system-tray pop-up if Windows Time service not running 2019-08-13 08:27:43 +01:00
catbref
9c1ca8de04 Fix HSQLDB backups.
Change AutoUpdate to use 'quick' backup.

For HSQLDBRepository.backup, don't perform CHECKPOINT DEFRAG while repository in use
as it never completes.

Similarly, use BACKUP DATABASE ... NOT BLOCKING.
2019-08-13 08:27:07 +01:00
catbref
8109214087 Reduce misbehaviour cooloff from 60min to 10min 2019-08-13 08:25:06 +01:00
catbref
b9737372d9 Use default testnet/mainnet listen port as appropriate when peer addresses are given without port 2019-08-13 08:22:46 +01:00
catbref
a154a7c073 Change default testnet ports to 9989 (P2P), 9988 (API) and 9980 (node UI) 2019-08-06 11:20:08 +01:00
catbref
21e64d0c8b Increase logging for ApplyUpdate to help debug issues 2019-08-06 11:18:15 +01:00
catbref
0da21356c7 Repository backup/recovery
Controller requests 'quick' repository backup every 123 minutes.

On start-up, if repository fails to load, recovery is attempted using
backup (if present).

AutoUpdate also requests 'slow' repository backup just before
calling ApplyUpdate. ('Slow' means perform "CHECKPOINT DEFRAG" first).
2019-08-06 11:17:49 +01:00
catbref
7eb5cd55ff Faster shutdown for Peers doing PROOF part of handshake 2019-08-06 11:16:45 +01:00
catbref
2f2d9a664d Replaced all NTP.getTime with System.currentTimeMillis. Clocks handled by O/S 2019-08-06 11:12:29 +01:00
catbref
4d265b8acb Don't log API exceptions or errors if Settings.isApiLoggingEnabled is false 2019-08-06 11:08:15 +01:00
catbref
a3e6c24a89 Minor performance improvement with lambda-based logging after initial profiling run 2019-08-06 11:07:55 +01:00
catbref
7b51b1e88d Improve Network.shutdown() and logging in Peer.
Network.shutdown() called Peer.shutdown() on each Peer
while holding synchronization lock on this.connectedPeers.

This would cause a problem during Peer.shutdown() as some
other reachable code would also want synchronized access
to this.connectedPeers. Typical symptoms would be log
entries like:

2019-07-02 11:13:05 DEBUG Peer:512 - Message processor for peer 192.144.182.73:9889 failed to terminate

Eventually Network.shutdown() would exit, releasing synchronization
lock and awaking stuck Peer threads, which could then try to access
repository (now closed) causing further log spam.

Now it uses Network.getConnectedPeers to return duplicated
List<Peer>, minimizing lock time on this.connectedPeers.

Also made Peer main thread logging more informative when a IOException
occurs, as most situations are harmless EOF or connection reset by peer.
2019-08-06 11:07:21 +01:00
catbref
192a072796 Unify setting Transaction's initial group-approval status.
Refactored duplicate code into Transaction.setInitialApprovalStatus().

This is make sure transactions HAVE a group-approval status
in Synchronizer before Block.isValid is called.

This wasn't a problem for new, unconfirmed, individual transactions
arriving over the wire due to Transaction.importAsUnconfirmed()
doing the right thing.

Also added a groupApprovalTimestamp to BlockChain feature-triggers
to support legacy chains.

Tested by syncing mainnet from scratch.
2019-08-06 11:05:36 +01:00
catbref
ad827ae01d Fix Controller's processing of HEIGHT_V2 to update all peers with same ID 2019-08-06 10:59:13 +01:00
catbref
976ea97af1 Improve pruning of old peers 2019-08-06 10:57:02 +01:00
catbref
9435e9576a Move getRewardByHeight from Block to BlockChain 2019-08-06 10:53:13 +01:00
catbref
f45cedb6ff Save lastConnected for outbound peers on completed handshake 2019-08-06 09:49:44 +01:00
catbref
840f52ff90 Added API call GET /blocks/timestamp/{timestamp} and increase timeout on POST /admin/forcesync from 5s to 30s 2019-08-06 09:47:38 +01:00
catbref
2621e04025 Change Peer ping interval to below that of SOCKET_TIMEOUT, used by blocking read()s 2019-08-06 09:47:12 +01:00
catbref
e47b4dceb2 StringBuilder, and other, optimizations for repository-related classes.
Add optional "excludeZero" to API call GET /assets/balances

Added tests to call most API calls to check no exceptions are thrown.
2019-08-06 09:46:31 +01:00
catbref
99ffd62a6e Change to how "best block" is determined. 2019-08-02 15:29:48 +01:00
catbref
c559c16a4a Add extra isInterrupted() quick-exits to Controller during TRANSACTION_SIGNATURES processing. 2019-08-02 15:11:04 +01:00
catbref
fc82bcaf49 Tidying up peer info.
lastHeight/blockSig/blockTimestamp, etc. moved from PeerData/repository
to Peer as it's transient so no need to store in repository.

Repository now keeps track of when/who added peer, e.g. API/INIT/another peer.

API calls DELETE /peers and DELETE /peers/known also disconnect peer
as well as deleting from repository.

Connection timestamp now reported by API call GET /peers

Some repository-updating code removed from Network/Controller as no
longer needed.

Removed obsolete Controller.hasShorterBlockchain predicate.
2019-08-02 14:56:19 +01:00
catbref
8b135eb447 Improve startup & shutdown. Improve API call POST /admin/forcesync.
Previous version was prone to throwing exceptions during shutdown
sequence, especially in Peer/Network-related threads.

Shutdown now tries to wait for Peer/Network threads to cleanly exit.

API call POST /admin/forcesync now tries to grab blockchain lock,
with timeout, and then repeatedly sync with requested peer until
either fully synced or something unexpected happens.
2019-08-02 14:54:22 +01:00
catbref
84f5935d38 Improve shutdown of Network.
Detect interrupt during peer connect so we can return without trying
to set up new peer.

Do join() after interrupt() in Network.shutdown.
2019-08-02 14:52:46 +01:00
catbref
c68e0eb6ea change auto-update download locations & logging message 2019-08-02 14:50:17 +01:00
catbref
5b70f0004d Disregard misbehaved peers before counting to see if we can synchronize.
Also added simple system-tray tooltip update to show number of peers and current height.
2019-08-02 14:47:40 +01:00
catbref
915eebb8e5 BlockChain.isTestNet now BlockChain.isTestChain.
Added Settings.isTestNet.

Disabled ArbitraryDataManager for now.
2019-08-02 14:42:10 +01:00
catbref
1d81c4db6b BlockGen / Synchronizer improvements
BlockGenerator will now attempt to generate a new block if none of its
peers have a recent block either (in case of network stall). BlockGenerator
still needs a minimum number of peers before generating though.

Reduce BlockGenerator workload and use of blockchain lock if it
can't generate a block.

Reduce Synchronizer logging output.

Unify calculating timestamp threshold for 'recent' block into Controller.
2019-08-02 14:33:41 +01:00
catbref
5acc92ef26 Improve TRANSACTION_SIGNATURES handling in Controller.
Don't disconnect peers that fail to send a requested transaction,
as they may no longer have it. e.g. transaction might have expired
or become invalid.

For some other cases, e.g. we have transaction already, move on to
requesting the next transaction instead of giving up on the list.
2019-08-02 14:16:18 +01:00
catbref
b48f671774 In AutoUpdate, pass download buffer to SHA256 digester BEFORE deXORing. 2019-08-02 14:15:56 +01:00
catbref
8727780b77 Added XorUpdate utility to help prepare auto-updates. 2019-08-02 14:15:38 +01:00
catbref
877c126ba3 Don't allow incorrect/null ARBITRARY dataType when building transactions via
API's POST /arbitrary call.
2019-08-02 14:14:56 +01:00
catbref
4cdb5e6b95 Auto-Update improvements
NOTE: Downloaded update JARs are now expected to have been XORed with 0x5A!

This is to help prevent Windows Firewall from blocking update downloads
based on deep packet inspection.

Download read timeout reduced from 5s to 3s.

Download locations reordered so github entries are at the top as they have
better CDNs.

ApplyUpdate now assumes null response from GET /admin/stop means node
is not running.

ApplyUpdate now checks replacement JAR actually exists before attempting
to overwrite previous version.

ApplyUpdate now tries to use Windows EXE launcher in preference to raw
java command line. (This should improve Windows installer behaviour
in detecting running process and possibly firewall implications too).
2019-08-02 14:13:04 +01:00
catbref
2e6d33659c Fix CANCEL_ASSET_ORDER check for already closed, partially/fully matched, order.
Additional unit tests to cover above case.
2019-08-02 14:07:47 +01:00
catbref
9af18aad34 Setting proxy forging reward share to 0% removes proxy forging relationship.
Included unit test to cover change.
Modified test blockchain config "test-chain-v2.json" to add maxProxyRelationships.

Added comments to some proxy-related methods in AccountRepository class.
2019-08-02 13:05:56 +01:00
catbref
ae6cf9fc32 Fix ordering when requesting summary of block forgers 2019-08-02 13:04:48 +01:00
catbref
4f25fffbe9 Change API transaction subsmission from single-shot to 500ms timeout for obtaining blockchain lock 2019-08-02 13:04:21 +01:00
catbref
03f60ef898 Fix incorrect declared transaction length for SET_GROUP and VOTE_ON_POLL transactions.
Added transaction [de]serialization test, along with corresponding random transaction generators.

Minor typo fix in Transaction.

Minor clarification in MessageTransactionTransformer.

Added debugging to Account.
2019-08-02 13:03:54 +01:00
catbref
02e8bdb034 Add support for fetching updates using a combination of hostname and IP address.
IP address used to create socket, hostname used for SNI, HTTPS, etc.

Added hostname+IP auto-update locations to Settings.
2019-08-02 12:53:49 +01:00
catbref
8dd4745c5c Work on auto-update
Arbitrary transactions now [de]serialize data-type (raw/hash) for v4+ transactions.
Data type also stored in repository. Very small (<=255 byte) data payloads are also stored directly in HSQLDB.

Added ArbitraryDataManager which looks for hash-only data payloads and possibly requests raw data over network
depending on 'policy' (which currently is "fetch everything").

Added networking support for finding, and transferring, arbitrary data payloads.

Minor optimization to message ID generation in Peer.

Minor optimization in Serialization.serializeSizedString()
2019-08-02 12:48:48 +01:00
catbref
4ced9cc3e2 More flexible auto-update locations 2019-08-02 12:44:52 +01:00
catbref
1d7bda5d46 Don't propagate new transactions back to sending peer. Added small inter-peer delay when doing network broadcast. 2019-08-02 12:42:30 +01:00
catbref
c2e8392f05 Synchronization improvements (again!)
Bumped version

Controller no longer uses block height to determine whether to sync
but now uses peer's latest block's timestamp and signature.

Also BlockGenerator checks whether it's generating in isolation using
the same peer info (latest block timestamp and signature).

Added API call POST /admin/forcesync peer-address to help get wayward
nodes back on track.

Unified code around, and calling, Transaction.importAsUnconfirmed().

Tidied code around somelock.tryLock() to be more readable.

Controller (post-sync) now broadcasts new chaintip info if
our latest block's signature has changed, not simply the height.

Network.broadcast() only sends out via outbound peer if node has
more than one connection from the same peer. So Controller would
only update one of the peer records with chaintip info.
Controller now updates all connected peers with the ID when it
receives a HEIGHT or HEIGHT_V2 message.

Added node1 thru node7.mcfamily.io to default peers in Network.

Network ignores first "listen port" entry when receiving peers
list from an outbound-connection peer as it already knows
by virtue of having connected to it!

More network message debug logging (hopefully never to be seen).

[some old code left in, but commented out, for a while]
2019-07-23 17:50:07 +01:00
catbref
d910cce807 Improve API peer info output 2019-07-23 17:38:56 +01:00
catbref
20200b844e Networking and synchronization improvements
Controller now sets (volatile) requestSync flag when a peer sends new height info.
This allows much quicker response to new blocks which might hopefully improve synchronization
compared with the old periodic sync method.

"Unsolicited" network messages are now added to a BlockingQueue,
and a separate unsolicited message processing thread (one per peer)
deals with this messages in turn.
This allows "reply" network messages to propagate up to the
threads that are waiting for them, preventing deadlocks and
peer disconnections due to lost pings.

Controller tries to do as much new transaction processing
outside of the blockchain lock as possible, and only
broadcasts new transaction's signature if we successfully
import transaction to our unconfirmed pile.

Synchronizer.findSignaturesFromCommonBlock now returns null
to indicate some sort of connection issue (no cool-off)
and an empty list to indicate NO COMMON BLOCK.
That method also tries to work back to genesis block
instead of giving up too early if test block height
becomes negative.

Network.createConnection additionally filters out
candidates if their addresses resolve to the same
IP+port as an existing connection. So now it won't
connect to localhost:1234 if it has an existing
connection with 127.0.0.1:1234.

Network.broadcast only considers unique peers,
i.e. prefers outbound connection if a peer has
corresponding inbound connection.

Added Thread.currentThread().setName() where possible.
2019-07-23 17:32:21 +01:00
catbref
ffffb50884 Networking/Controller changes to aid broadcast of unconfirmed transactions.
Notably: network messages passed up to Controller are now processed in their
own thread, as opposed to peer's thread.
So each message processor in Controller needs to thread-safe.

V2 network protocol asks for unconfirmed transactions, can send round lists
of transaction signatures and ask for individual transactions, to save
bandwidth/processing.
2019-07-23 17:28:20 +01:00
catbref
680361daa6 Temporary fix to Controller to only try to acquire blockchain for incoming new unconfirmed transactions 2019-07-23 16:19:01 +01:00
catbref
c03a5c3d0e Fix incorrect count of forged blocks 2019-07-23 16:18:24 +01:00
catbref
b7f53afe68 Improve API output for GET /peers 2019-07-23 16:18:01 +01:00
catbref
3a3cc5a81b Improve API output of GET /blocks/forgers 2019-07-23 16:17:42 +01:00
catbref
97a5b3ef9b Increase max proxy relationships in blockchain config 2019-07-23 16:17:19 +01:00
catbref
5429cb12dd Possible fix for single instance check by AI Windows launcher 2019-07-23 16:13:51 +01:00
catbref
2f3123a315 Add lock around some Peer.peerData actions to help sync 2019-07-23 16:06:04 +01:00
catbref
6a969f9473 Add limit to number of proxy forging relationships per generator account.
For testing, current limit is 4 as set in blockchain.json
2019-07-23 16:05:21 +01:00
catbref
de96e8347d Move handling of GET_PEERS from Controller to Network. On sync fail, only update outbound peer info in repository to help prevent useless 'known peer' entries. 2019-07-23 16:02:32 +01:00
catbref
dcb8d67236 Don't allow new transactions to be submitted if not up to date 2019-07-23 15:59:01 +01:00
catbref
491a6ecd67 Fix incorrect declared transaction length for create-group transactions 2019-07-23 15:46:42 +01:00
catbref
9ac52f6a86 Fix missing network-related repository.saveChanges() 2019-07-23 15:45:04 +01:00
catbref
7993c3e594 Change node UI start URL to index.html 2019-07-23 15:43:08 +01:00
catbref
921953d304 Clean up inbound peer entries from repository & fix sending peer lists 2019-07-23 15:36:14 +01:00
catbref
912ec22417 Minor warning fix 2019-07-23 15:26:52 +01:00
catbref
a2cc4983a0 Add "involved" optional query param to API call GET /address/proxying, for convenience 2019-07-23 15:26:26 +01:00
catbref
ab9fd681f3 Fix synchronizer so incoming block's transactions have approval status set 2019-07-23 15:13:59 +01:00
catbref
4667b768df Added tests for each group-approval outcome + fixes 2019-07-23 15:13:28 +01:00
catbref
c9f226cf88 group approval tests and fixes 2019-07-23 15:12:35 +01:00
catbref
8af761c1c3 Updated orphaning code of Block and Transaction subclasses 2019-07-23 15:10:43 +01:00
catbref
4b3f877dc0 Interim commit - refactored transaction transformers and fixed unit test compiler errors 2019-07-23 15:06:28 +01:00
catbref
06ba004238 Interim commit - refactored HSQLDBTransactionRepository, and subclasses. Also added approval_height to transactions, renaming height to block_height. 2019-07-23 14:50:56 +01:00
catbref
431e15c7ae interim commit after refactoring TransactionData, and subclasses 2019-07-23 14:49:41 +01:00
catbref
da1bd82c19 Minor integration progress
Remove fetching unconfirmed from Synchronizer

Add extra validity/reference/processable checks to
Transaction.isValidUnconfirmed

Update TransactionUtils to use Transaction.importAsUnconfirmed
for unit tests.
2019-07-23 14:48:49 +01:00
catbref
9bb673ba82 Interim commit of *Transaction refactoring
Transaction subclass isValid methods have last reference checks removed.
(Now handled by Transaction.hasValidReference)

Some checks in subclass' isValid moved to isProcessable, typically
for transaction types that require group-approval.

Removed fee extraction and last-reference update code from
subclass' process() method to Transaction.processReferencesAndFees.

Other changes to transaction/block processing.
2019-07-23 14:46:44 +01:00
catbref
c9968b3dd2 Interim commit of *TransactionData classes for safety 2019-07-23 14:42:02 +01:00
catbref
041773cf41 Synchronizer asks for approval-pending transaction from peer if needed 2019-07-23 12:49:38 +01:00
catbref
3f2453e404 ApiError zh locale (but untranslated) 2019-07-23 12:47:52 +01:00
catbref
414f9679c8 Synchronizer logging improvements 2019-07-23 12:47:20 +01:00
catbref
cd5f9a1e6c Convert ClassLoader.getSystemResource* calls to class.getResource* variant for OSGi, etc. safety 2019-07-23 12:46:27 +01:00
catbref
94fceeb34a Move default blockchain.json into resources 2019-07-23 12:46:05 +01:00
catbref
a4f2cf50b0 Updated node management UI (was previously "bundled ui")
Fix root path redirects for node management UI and API documentation servlets.
2019-07-23 12:44:10 +01:00
catbref
f4022dd243 initial work on adding bundled node-management UI 2019-07-23 12:37:42 +01:00
catbref
99024ee2ef Synchronization improvements
Don't attempt to sync, or generate blocks, if we think we're not up to date.

Notify Controller of newly generated block AFTER releasing blockchain lock.

Loads of changes to synchronizer.

Added missing GET_PEERS handling to Controller.onNetworkMessage.

More detailed peer information (last block sig, last generator sig, last block timestamp, ...)
New HEIGHT_V2 network message to help support above.

More, and improved, logging.

Fix for HSQLDB serialization failure caused by trying to save the same new transaction
received by more than one peer/thread simultaneously.
2019-07-23 12:22:17 +01:00