Streamlined BTC class and switched to memory block store.
Split BTCACCTTests into BTCACCT utility class and (so far)
three stand-alone apps: Initiate1, Refund2 and Respond2
Moved some Qortal-specific CIYAM AT constants into blockchain config.
Removed redundant BTCTests
Bump bitcoinj to 0.15.5 for fixes.
lockTime is int (seconds since epoch), not long (ms since epoch).
Improve output of Initiate1.
Added (most of) Respond2.
Added a setting "showBackupNotification", which is false by default,
that shows a tray notification when a repository backup occurs.
Above notification, and the auto-update notification, now refer to
the SysTray i18n translation lookup resources.
Added Ed25519 private key to public key function accessible from SQL.
Added Ed25519 public key to Qortal address function accessible from SQL.
Used above functions to store minting account public key in SQL to
reduce the number of unnecessarily repeated Ed25519 conversions.
Used above functions to store reward-share minting's accounts address
to reduce the number of unneccessarily repeated PK-to-address conversions.
Reduced the usage of PublicKeyAccount to simply Account where possible,
to reduce the number of Ed25519 conversions.
Account.canMint(), Account.canRewardShare() and Account.getEffectiveMintingLevel()
now only perform 1 repository fetch instead of potentially 2 or more.
Cleaned up NTP main thread to reduce CPU load.
A fixed offset can be applied to NTP.getTime() responses, for both
scenarios when NTP is running or not. Useful for testing or simulating
distant remote peers.
Controller.onNetworkMessage() and Network.onMessage() have both had their
complexity simplified by extracting per-case code to separate methods.
Network's EPC engine's thread pool size no longer hard-coded, but comes
from Settings.maxNetworkThreadPoolSize, which is still 10 by default,
but can be increased for high-availability nodes.
Network's EPC task-producing code streamlined to reduce CPU load.
Generally reduced calls to System.currentTimeMillis(), especially
where the value would only be used in verbose logging situations,
and especially in high-call-volume methods, like within repository.
Keep track of when EPC engine can't spawn a new thread as this
might indicate thread-pool exhaustion and cause some network
messages to be lost.
If logging level is NOT 'trace' (or 'all') then don't call
System.currentTimeMillis() as we'll never use the value.
Similarly, don't set thread names if not logging at 'trace' either.
Update EPC tests, particularly unified per-second/end-of-test stats
reporting.
Added API call GET /peers/enginestats to allow external monitoring.
Extract all engine stats in one synchronized block, instead of
separate calls, for better consistency.
No more "node UI". UI provided by 3rd party.
"Open UI" tray icon menu item now attempts to open UI at various
local servers (see Settings.uilocalServers) or some random
remote server (Settings.uiRemoteServers).
Default UI port now 12388 (Settings.uiPort).
Translator class no longer logs warnings for every failed translation.
Commented out unused ApiError enum entries.
Renamed some ApiError names like "_NO_EXISTS" to "_UNKNOWN".
Removed old src/main/resources/globalization/* files.
Added CheckTranslations test app.
Fixed some extraneous/missing ApiError aspects in some API-related classes.
e.g. added NAME_UNKNOWN to GET /names/{name}
We've never needed this before but now it's fixed.
Added corresponding +ve & -ve tests just to make sure.
Only actual use-case that comes to mind is cancelling reward-share.
Before:
0-fee self-share REWARD_SHARE when there is an existing self-share would result in
INSUFFICIENT_FEE from isFeeValid()
After:
isFeeValid() returns OK for above, but isValid() returns SELF_SHARE_EXISTS.
In addition, a transaction that tries to modify existing self-share, even with fee,
also returns SELF_SHARE_EXISTS.
Improved tests to double check.
NOTE: first startup after this commit can take a while due to building index!
SQL statement "DELETE FROM HistoricAccountBalances WHERE height >= ?" required
a full table scan and so was very slow on VMs/routers. This statement used by
HSQLDBAccountRepository.deleteBalancesFromHeight(), itself called during
Block.orphan().
Symptoms particularly evident during shutdown where above statement could take
upwards of 15 minutes on single-CPU, small-memory VMs!
Statement wasn't noticed before as slow-query checking wasn't involved.
Slow-query checking now applies to HSQLDBRepository.delete() and
HSQLDBRepository.exists() calls.
Added test for correct HSQLDB interrupt handling.
Fixed some typos.
Blockchain configs will need "v2Timestamp" feature trigger renaming to "qortalTimestamp"!
Due to Controller.VERSION_PREFIX changing, peer-to-peer protocol version detection has
been changed. Was previous a substring match, now we test peers's buildTimestamp is at
least Peer.V2_PROTOCOL_TIMESTAMP_THRESHOLD. Changes in Peer.java.
Also added comment and throw() to QortalATAPI.getNextTransactionTimestamp()
as this needs given the change to Qortal's block timestamps from legacy Qora.
Changes to HSQLDB data types, e.g. QoraAddress to QortalAddress, means existing
database will need to be thrown away after this commit!
Converted AccountData's initialLevel to blocksMintedAdjustment.
Corresponding changes to AccountLevelTransaction so that level
set in genesis block is converted to blocksMintedAdjustment,
via cumulativeBlocksByLevel.
Ditto changes to HSQLDBAccountRepository, HSQLDBDatabaseUpdates,
[HSQLDB]TransactionRepository, etc.
Changes to API call POST /admin/mintingaccounts to check passed
reward-share private key maps to a reward-share with minting
account that still has privilege to mint. It's possible for
a TRANSFER_PRIVS transaction to transfer away minting privileges
from a minting account referenced by in a reward-share.
Change to RewardShareTransaction to allow users to cancel a
reward-share even if minting-account component no longer has
minting privs. This should allow users to clean up more after
a privs transfer.
Re-order processing/orphaning in Block.process()/Block.orphan()
to be more consistent and also to take in account changes that
might have been caused by TRANSFER_PRIVS transactions which affect
who might actually receive block rewards/tx fees.
Founders now gain blocksMinted & levels as part of minting blocks.
(Needed to make TRANSFER_PRIVS from a founder account to work).
BlockMinter now has added checks to make sure that the reward-shares
it might use to mint blocks still have valid minting-accounts.
i.e. that the minting-account component of reward-share hasn't had
minting privs transferred away by TRANSFER_PRIVS tx.
Controller now rejects online-accounts from peers that no longer
have minting privs (e.g. transferred away by TRANSFER_PRIVS)
Corresponding, Controller no longer SENDS online-accounts that no
longer have minting privs to other peers.
Added some tests - more tests needed, e.g. for multiple transfers
into the same account, or a test for minting post transfer for both
sender & recipient.
Now reward-shares with zero percent are valid, to allow the 'recipient' party to gain
"number of minted blocks" but no actual block reward.
Correspondly, the special zero share used to cancel reward-shares has been changed to
be any negative value.
Block rewards, founder 'leftovers': if founder is minter account in any online
reward shares, then the per-founder-share is spread across their online reward-shares,
otherwise it's simply/wholy given to that founder.
Created a new DB table to hold "next block height", updated via triggers on Blocks.
This is so various sub-queries can simply read the next-block-height value instead
of complex IFNULL(MAX(height),0)+1 or SELECT height FROM Blocks ORDER BY height DESC.
Prior code was also broken in edge cases, e.g. no genesis block, or ran slow.
Added tests to cover above.
Deleted BTC tests as they're obsolete.
Added/improved other tests.
*** WARNING ***
Possible block reward bug in this commit. Further investigation needed.
Reverted AccountBalances back to height-less form.
Added HistoricAccountBalances table that is populated via trigger on AccountBalances.
This saves work when performing common requests for latest/confirmed balances,
shunting the extra work to when requesting height-related account balances.
Unified API call GET /addresses/balance/{address} by having address/assetId/height as
query params.
Simpler call for fetching legacy QORA holders during block rewarding.
Improved SQL for fetching asset balances, in all conditions,
e.g. with/without filtering addresses, with/without filtering assetIds,
etc.
Unit test for above to make sure query execution is fast enough.
(At one point, some SQL query was taking 6 seconds!)
Added optional 'height' Integer to AccountBalanceData, but this
is not populated/used very often.
HSQLDBAccountRepository.save(AccountBalanceData) now checks zero balance saves
to see if the row can be deleted instead. This fixes a lot of unhappy tests
that complain that there are residual account balance rows left after
post-test orphaning back to genesis block.
Yet more tests.
Removed very old 'TransactionTests' which are mostly covered in more specific tests elsewhere.
Added cancel-sell-name test from above.
Fixed AssetsApiTests to check for QORT not QORA!
Changed hard-coded assetIDs in test.common.AssetUtils in light of new LEGACY_QORA & QORT_FROM_QORA genesis assets.
Some test blockchain config changes.
Also:
RewardShareKeys app now supports only one arg (minter private key)
in self-reward-share mode, where recipient public key is derived
from minter private key.
Added methods to Account for returning 'effective' minting level
where minting level for founders is read from blockchain config.
(Or returns zero if unable to mint).
Changed two Block constructors into static methods that return
a new Block as there was way too much work being done to really
be called a constructor, especially with all the opportunities
to throw an exception too.
Main blockchain config updated to reflect near-launch version.
Added/changed blockchain weight tests to check block winning
based on higher account levels.
Added Transaction.isFeeValid() to allow transaction subclasses to override and allow zero fees, etc.
Added tests to cover self-reward-shares, including zero fee scenario.
Set 'dilbert' test account to level 8 in test genesis block.
Removed leftover mention of "previous_level" from HSQLDBAccountLevelTransactions,
and AccountLevelTransactionData. (Previous level makes no sense as ACCOUNT_LEVEL
transactions are genesis-block only).
Fixed some incorrect uses of PrivateKeyAccount.getSharedSecret() to
PrivateKeyAccount.getRewardSharePrivateKey() in tests.
Unified terminology from block "generator", "forger", etc. to "minter"
Unified terminology "proxy forger" to "reward share" as it was
incorrect, or at least highly ambiguous, which account had which role.
AccountRepository.findRewardShares() has different arg order!
Account.canMint() now returns true if account has 'founder' flag set.
Added Account.canRewardShare() which returns whether acocunt can create
a reward-share (e.g. level 5+ or founder).
Fixed HSQLDBAssetRepository.getAllAssets() which had incorrect
resultSet column indexes.
Removed old traces of EnableForging transaction.
ACCOUNT_LEVEL and ACCOUNT_FLAGS (genesis-block-only transaction types)
now set target account's last-reference. This is allow later REWARD_SHARE
transactions in genesis block and post-genesis transactions by that account.
REWARD_SHARE transactions are now FREE, but only if minter is also recipient.
If a self-reward-share already exists, then unless share-percent is zero (to
terminate reward-share), any subsequent self-reward-share is invalid.
Updated SysTray i18n properties file.
BlockChain config file requires 'minAccountLevelToRewardShare' and optional
'minAccountLevelToMint'.
Added potential, but currently unused, memory-hard PoW algorithm.
Fixed/removed/disabled some unit tests.
BlockMinter.generateTestingBlock asks Controller to pretend mintingAccount is 'online'.
More testing needed!
Rename Asset.QORA to Asset.QORT so we can also have Asset.LEGACY_QORA
as another hard-coded asset.
Add "is unspendable" aspect to assets where only the asset owner can
transfer/pay asset to other people. Asset trading is barred regardless,
as is use of asset for ATs.
Added "initial level" to account data in preparation for accounts levelling
up from generating blocks.
Added distribution/removal of block reward based on legacy-QORA held.
Removed "previous level" from ACCOUNT_LEVEL transactions as they're
only ever valid in genesis block and so previous level is never needed.
Updated pom.xml.
Updated dependencies, including various minor code mods, esp. bitcoin-related.
Starts up and talks to Java 8 nodes!
Dev environment should be at least Eclipse 4.11 with m2e 1.9.1+
API call GET /addresses/online reports online accounts,
including both addresses relating to the proxy-forge public key.
New PeerChainTipData class to replace the broken "peer data lock"
that was supposed to make sure peer's last height/blockSig/timestamp
were all in sync. Now peer's chain tip data is a single object
reference that can be replaced in one go.
Removed pointless API calls /blocks/time and /blocks/{generatingbalance}.
Various changes, mostly in Block class, to do with switching to BlockTimingByHeight
from old min/max block time.
New block 'weight' based on number of online accounts
and 'distance' of perturbed generator public key from 'ideal' public key
(for that particular block's height).
New sub-chain 'weight' based on accumulating block weights,
currently by shifting previous accumulator left by 8 bits then
adding next block's weight.
More validation of BlockChain config. Helpful for debugging, probably
not very useful to end-users.
BlockGenerator now uses unified Peer predicates from Controller, like:
Controller.hasMisbehaved, Controller.hasNoRecentBlock, etc.
Controller now keeps a list of chain-tip signatures that are for inferior
chains, so it doesn't try to synchronize with peers with inferior chains.
(This list is wiped when node's blockchain changes/block is generated).
Controller now asks Gui to display error box if it can't parse Settings.
Controller.potentiallySynchronize() does more filtering of potential peers
before calling actuallySynchronize(). (Mostly moved from Synchronizer,
so now we expect actuallySynchronize() to do something rather than bail
out because it doesn't like the peer after all).
If synchronization discovers that peer has an inferior chain,
then Controller notifies that peer of our superior chain, to help keep
the network in sync.
Renamed OnlineAccount to OnlineAccountData, as it is in package org.qora.data
after all...
Synchronizer reworked to request block summaries so it can judge which chain
is better, and hence whether to sync with peer or abort.
Slight optimization of Peer.readChannel() to exit earlier if no more network
messages can be extracted from buffer.
More tests.
Improved documentation and logging.
Safety commit in case of data loss!
Lots of changes to do with "online accounts", including:
* networking
+ GET_ONLINE_ACCOUNTS * ONLINE_ACCOUNTS messages & handling
+ Changes to serialization of block data to include online accounts info
* block-related
+ Adding online accounts info when generating blocks
+ Validating online accounts info in block data
+ Repository changes to store online accounts info
* Controller
+ Managing in-memory cache of online accounts
+ Updating/broadcasting our online accounts
* BlockChain config
Added "account levels", so new code/changes required in the usual places, like:
* transaction data object
* repository
* transaction transformer
* transaction processing
Orphaning a BUY_NAME transaction would not reinstate the
sale price. Sale price could be nullified by (e.g.) orphaning
a SELL_NAME transaction.
Also added test case to cover above and other test-related support,
e.g. test-mode in NTP.
Added "volatile" to more fields, for thread-safety on reads.
Changes to field values are done inside synchronized blocks
so no need for AtomicInteger/AtomicBoolean. (Could be changed
in the future to show intention/readability though).
Added more statistics (tasks produced/consumed).
Limited Network's EPC executor to 10 threads max.
New NTP class now runs as a simplistic NTP client, repeatedly polling
several NTP servers and maintaining a more accurate time independent
of operating system.
Several occurrences of System.currentTimeMillis() replaced with NTP.getTime()
particularly where block/transaction/networking is involved.
GET /admin/info now includes "currentTimestamp" as reported from NTP.
Added support for block timestamps determined by generator, instead of
supplied by clock. (BlockChain.newBlockTimestampHeight - not yet activated).
Incorrect timestamps will produce a TIMESTAMP_INCORRECT Block.ValidationResult.
Block.calcMinimumTimestamp repurposed as Block.calcTimestamp for above.
Block timestamps are now allowed to be max 2000ms in the future,
was previously max 500ms.
Block generation prohibited until initial NTP sync.
Instead of deleting INVALID unconfirmed transactions in BlockGenerator,
Controller now deletes EXPIRED unconfirmed transactions every so often.
This also fixes persistent expired unconfirmed transactions on nodes
that do not generate blocks, as BlockGenerator.deleteInvalidTransactions()
was never reached.
Abbreviated block sigs added to log entries declaring a new block is generated
in BlockGenerator.
Controller checks for NTP sync much faster during start-up and SysTray's
tooltip text starts as "Synchronizing clock" until NTP sync occurs.
After NTP sync, Controller logs NTP offset every so often (currently every 5 mins).
When considering synchronizing, Controller skips peers that have the same block sig
as last time when synchronization resulted in no action, e.g. INFERIOR_CHAIN,
NOTHING_TO_DO and also OK. OK is included as another sync attempt would result in
NOTHING_TO_DO.
Previously this skipping check only happened after prior INFERIOR_CHAIN.
During inbound peer handshaking, if we receive a peer ID that matches an existing inbound
peer then send peer ID of all zeros, then close connection.
Remote end should detect this and cleanly close connection instead of waiting for handshake timeout.
Randomly generated peer IDs have lowest bit set to avoid all zeros.
Might need further work.
Networking doesn't connect, or accept, until NTP has synced.
Transaction validation can fail with CLOCK_NOT_SYNCED if NTP not synced.
Controller performs NTP check on startup (and every 5 minutes)
which determines whether block generation is allowed.
System Tray tooltip updated to reflect generating status.
Plus new translations.
Improved GuiTests.
BlockGenerator fetches forging accounts first, and sleeps
if none configured, which is less work than processing peer lists.
Now uses several NTP servers to determine mean offset from
system clock to internet time.
If abs(offset) > 500ms or NTP service not running then
user is 'nagged' via system tray pop-up notification
with instructions on how to fix.
Also improved system tray translations!
Reworked networking execute-produce-consume threading.
Some networking task were wrongly performed during 'produce' phase,
and some producing was happening in 'consume' phase (also corrected).
Peer connection tasks are rate-limited to 1 per second to reduce CPU thrashing.
Show P2P listen port in logs on startup.
Tests for general purpose ExecuteProduceConsume class to cover both
random task scenario and mass-ping scenario.
