Also improved BTC.WalletAwareUTXOProvider to derive more keys itself
instead of throwing and relying on caller to do the work.
Added benefit of cleaning up caller code and being more efficient.
Needed because not all receiving/change addresses were being picked up.
Also: renamed trade bot field/column "receiving_public_key_hash"
to "receiving_account_info" as Alice's trade bot uses it to
store Alice's Qortal address, not PKH.
Added some extra simplistic repository calls to support above,
like BlockRepository.getTimestampFromHeight,
ATRepository.getCreatorPublicKey(atAddress)
Bitcoin receive address no longer stored in AT but dealt with by trade-bot.
This allows 'Bob' to have his BTC sent anywhere he likes when redeeming P2SH-A
thus saving a step, typically incurred by UI. DB shape change due to this.
Similarly, AT code has been updated to expect a Qortal receiving address when
Alice sends MESSAGE to redeem AT.
This means both trade-bot entries (Alice/Bob) can be safely wiped once trade completes.
Some terms were confusing like "trade recipient" which actually referred to
Alice and so have been unified as "trade partner" as to not be confused with
(say) "recipient address"
The MESSAGEs sent from Alice to Bob, from Bob to AT and from Alice to AT have been
given more useful names: 'offer', 'trade' and 'redeem'. There is also a cancel
MESSAGE sent from Bob to AT to cancel AT before trading occurs.
Some API calls have been renamed in light of above.
AT's 'mode' has been expanded from simply OFFER/TRADE to:
OFFERING, TRADING, REFUNDED, REDEEMED, CANCELLED
Tests updated, but MORE TESTING REQUIRED BEFORE RELEASE
Various issues in Jetty v9.4.22 (and some later versions too)
cause websockets to use up all available threads.
Bumped Jetty to v9.4.29 to resolve some of these issues.
Changed some Qortal-side websocket code to minimize
locking on websocket notifiers. Websocket messages now
sent async, although the returned Futures are discarded,
as it's up to the remote end to consume fast enough.
Changed Controller to only request a SysTray update before
synchronization if there's a chance node might change height.
Similarly, Controller only requests SysTray update after
synchronization if chain tip has actually changed.
Both of the above together should reduce the number of
messages sent out via the admin status websockets.
Previous version fetched all the blocks from previous 'timestamp'
to current height, checking each transaction. (very slow)
New implementation leverages repository to do the heavy lifting.
Could potentially benefit from some DB indexes in the future?
Added unit test to cover.
Bitcoin main-net ElectrumX server list added to ElectrumX class,
albeit commented out at this point until it is decided that trade-bot
is ready for production use. (Simply remove the leading //s)
More comments and documentation has been added to TradeBot class
to further describe the actions taken.
It is important to note that:
Bitcoin wallet access is required by trade-bot
and so:
A Bitcoin WALLET PRIVATE KEY is stored in the database by trade-bot
and hence, if you use trade-bot:
DO NOT DISTRIBUTE YOUR DB FILES TO ANYONE ELSE!
Furthermore it should be obvious that this functionality is provided on
a 'best effort", not guaranteed, basis, therefore:
YOUR FUNDS ARE AT RISK!
If you are unsure about any aspect, or cannot afford to lose your funds,
or it's possible that unexpected outcomes occur, then DO NOT USE.
To use trade-bot on Bitcoin TESTNET then this to your settings JSON file:
"bitcoinNet": "TEST3",
See Settings.java line 100, and BTC class for more info.
bitcoinj now uses ElectrumX as an UTXO provider in order to keep track
of coins in BIP32 deterministic wallet.
Trade responder (Alice) needs to pass a BIP32 extended private key to API
so trade-bot can create unattended spends.
Both Alice and Bob can find their final funds in accounts using the
ephemeral 'tradePrivateKey' from trade-bot state data.
Most cross-chain API calls are now only allowed from localhost.
Most Bitcoin fees pegged at 0.00001000 BTC.
More work needed to handle refunds in case of trade failures.
(See XXX comment tags in TradeBot.java)
Qortal AT now includes suggested tradeTimeout again as a constant so trade partner/recipient can use that to calculate a suitable lockTimeA. CODE_HASH changed!
Renamed some secret_hash to hash_of_secret.
Changed TradeBotStates.trade_state back to TINYINT and adjusted values in TradeBotData.State enum to suit.
Added lockTimeA to TradeBotData & repository.
Added JAXB-only extra representations of Bitcoin PKHs as addresses.
Fixed incorrect expected length in BTCACCT.extractOfferMessageData().
CrossChainTradeData.refundTimeout now only present in TRADE mode.
Added BTC.pkhToAddress().
Added initial TradeBot.handleAliceWaitingForP2shA().
Enforce only one TradeBot thread running using 'activeFlag' atomic boolean.
Replace incorrect SHA256 with HASH160 for hashOfSecretA in TradeBot.startResponse().
Controller now calls TradeBot.onChainTipChange() inside thread
started by Controller.onNewBlock(), instead of blocking
Controller.setChainTip().
DB TradeBotStates has trade_foreign_public_key changed to VARBINARY(33)
as Bitcoin pubkeys aren't uniformly 32 bytes!
Also, trade_state changed from TINYINT to SMALLINT to cover enum value range.
TradeBot.createTrade() incorrectly used Crypto.digest() to create hash-of-secret
instead of Crypto.hash160(). Also corrected tradeState to
BOB_WAITING_FOR_AT_CONFIRM. Also added missing fee calculation.
Added missing repository.saveChanges() to TradeBot methods.
Added balance check to API POST /crosschain/tradebot before passing
request to TradeBot.createTrade(), which also ensures there's a
usable account last-reference too.
BTC.getBalance() now returns Long instead of Coin.
BTC.FORMAT.format(Coin) changed to BTC.format(Coin or long).
Added BTC.deriveP2shAddress(byte[] redeemScriptBytes).
Added GET_MESSAGE_LENGTH_FROM_TX_IN_A
and PUT_PARTIAL_MESSAGE_FROM_TX_IN_A_INTO_B.
Replaced AT-1.3.4 with version including bug-fix for off-by-one
data address bounds checking.
Moved long-from-bytes method to BitTwiddling class.
Renamed some methods to make it more obvious they work with
little/big endian data.
Also added Named/DaemonThreadFactory classes.
Network EPC now uses NamedThreadFactory for easier debugging.
Added settings field "networkPoWComputePoolSize", default 2, which
seems to work with both low-power ARM boards and high-power desktops.
If a node accepts a connection from an inbound peer
then remote peer will send RESPONSE first
and local node would previously change handshaking state
to COMPLETED while computing their own RESPONSE.
This meant that the local node would sometimes also start
sending post-handshake messages to the remote peer,
e.g. TRANSACTION_SIGNATURES.
Remote peer is only expecting a RESPONSE message, so would
close connection.
So we introduce an extra handshaking state "RESPONDING" for use
by local node while they compute RESPONSE in a separate thread.
Once the RESPONSE has been sent, local node moves to COMPLETED
state and called onHandshakeCompleted() as per usual.
Note that the code path when connecting outbound to a remote peer
is not changed, and the RESPONDING state is not used.
Also in this commit:
Network.onPeerReady now bypasses call to onMessage and instead
calls onHandshakingMessage() directly to avoid race condition
where peer's handshake status could change between
onPeerReady's caller and onMessage() calling peer.getHandshakeStatus()
NodeStatus contructor now fills in fields, which themselves are now 'final'.
NodeStatus also includes numberOfConnections and height as per systray.
AdminResource.status() unified with websocket version.
Incorrect column names when saving a group ban.
Missing column in LeaveGroupTransactions.
More stringent validity checks in group-kick, group-ban and remove-group-admin.
Added loads more tests to cover group actions.
Requires entries 'sslKeystorePathname' and 'sslKeystorePassword'
in settings.json.
With SSL enabled, API will auto-detect HTTP or HTTPs on the same port.
Included tools/build-keystore.sh to help build keystore from
Let's Encrypt certificates.
Renamed GET /blocks/minters to /blocks/signers
Renamed GET /blocks/minter/{address} to /blocks/signer/{address}
Changed corresponding repository methods and data classes.
Controller.onBlockMinted() now .onNewBlock(BlockData)
which saves having to fetch from repository.
Controller.onNewBlock also takes care of updating Controller's
cached chain tip, requesting SysTray refresh, broadcasting
new tip info to peers and notifying websockets.
BlockMinter and Controller.actuallySynchronize updated
to use unified .onNewBlock.
BlocksWebsocket also returns blocks on demand, given either
integer block height or base58 block signature.
Added support to return ApiError via websockets.
Unified Transaction.importAsUnconfirmed() and Controller.onNetworkTransactionMessage()
to both call Controller.onNewTransaction().
Modified Controller.onNewTransaction() to only send transaction signature to
other peers, instead of full transaction. Peers can request full transaction if they
don't have it.
Controller.onNewTransaction() also calls ChatNotifier, which in turn
notifies websocket handlers about new CHAT transactions.
Added jetty websocket dependency to pom.xml
Any reward leftover from ditributing to legacy QORA holders is reallocated to either:
founders if any online
or
account-level-based reward candidates, if no founders online
We should get pretty close to 100% block reward distribution, barring rounding artifacts.
More documentation and tests.
Removed BlockChain's founderShare as it is calculated in Block on a per-block basis instead.
Now we sum generic block reward + transaction fees before performing
distribution only once.
Added Map to collate account-balance changes during block reward
distribution so the final changes can be applied in one batch,
reducing DB load.
Some other optimizations like a faster ExpandedAccount.getShareBin().
Passes test EXCEPT RewardTests.testLegacyQoraReward(), pending decision
on how to reallocate 'unspent' block reward.
No more bitcoinj peer-group stalls, or slow startups,
or downloading tons of block headers, or checkpoint files.
Now we use ElectrumX protocol to query info from random servers.
Also:
BTC.hash160 callers now use Crypto.hash160 instead.
Added BitTwiddling.fromLEBytes() returns int.
Unit tests seem OK, but needs complete testnet ACCT walkthrough.
Old Qora v1 message types removed.
Message type values changed.
Network handshaking reworked to fix multiple-connections issue.
Instead of using some random peerID, we now use proper keypairs and a challenge-response handshake to prevent doppelgangers/ID-theft.
This results in simpler handshaking code as we don't have to perform some arcane doppelganger resolution.
Handshaking still uses proof-of-work for challenge-response, but switched to newer MemoryPoW.
API call GET /peers no longer has 'buildTimestamp' field, but does now have 'nodeId' field.
Network no longer has a whole raft of getXXXpeers() due to simplified handshaking.
Quite a few method calls changed to simply Network.getHandshakedPeers(), which is also faster.
Previously GET /chats/active/{address} would only return an active group chat
entry where 'address' was a member AND there was an existing CHAT
transaction with the same tx_group_id (and no recipient).
Now the response contains entries for ALL groups where 'address' is a member,
regardless of an existing CHAT transactions, omitting the 'timestamp' entry
if there are none.
CREATE_GROUP, ISSUE_ASSET, REGISTER_NAME and UPDATE_NAME transactions affected.
The code to actually generate 'reduced' name was called inside isValid() and
relied on setting the corresponding transaction data object field so that it would
be saved by isValid()'s caller. Although this worked, it wasn't a very clean
solution.
Now the 'reduced' name is generated by transaction data object's constructors so
it is always present.
Also removed name/group/asset reduceName(String) methods as they were all the
same single-line call to Unicode.sanitize().
Group owner now derived from CREATE_GROUP transaction creator's public key.
Added 'reduced' group name to GroupData, with corresponding change to DB.
Renamed GroupData.getIsOpen() to simply isOpen().
Tidied up CreateGroupTransactionData, adding 'reduced' group name.
Renamed getIsOpen() to simply isOpen().
Added code to generated reduced group name when building genesis block.
Added Group.MIN_NAME_SIZE of 3.
DB tables changed to add reduced_group_name where appropriate,
removing owner where necessary.
Added GroupRepository.reducedGroupNameExists(String).
Fixed up test blockchain configs in src/test/resources/test-chain-v2*.json.
This allows on-chain messages to a group, including NO_GROUP / groupID zero.
No-recipient messages cannot have an amount - where would it go?
Changed MESSAGE serialization layout to add boolean indicating
whether recipient is present.
Changed MESSAGE serialization layout so assetID is after amount,
and only present if amount is non-zero.
Changed DB table structures to cover above.
Added unit tests to cover above.
Owner now derived from issuer's public key.
Maximum asset name length reduced to 40 characters.
Repository table changes.
"owner" removed from test blockchain configs and "issuerPublicKey" used instead
where applicable.
Some getters in the form of "getIs___()" renamed to simply "is____()".
Changes include:
* Allowing renaming
* Tracking last-updated timestamps
* More stringent Unicode processing
* Way more unit tests
* Max name length reduction to 40 chars
Note: HSQLDB repository table changes
Controller no longer starts up BTC support during main startup.
This does mean that BTC startup is deferred until first BTC-related
action, and that the first BTC-related action will take much longer
to complete.
Added tests to cover startup/shutdown.
This also fixes splash logo stuck on-screen and broken Controller
shutdown when using REGTEST bitcoin network AND there is no
local regtest bitcoin server running.
REGISTER_NAME has an "owner" field which can be different from the actual
registrant (transaction creator's public key, used for signing transaction).
This allowed people to register names to be owned by someone else, thus breaking
the whole "one name per account" aspect.
So now "owner" is removed from REGISTER_NAME, and the actual owner address is
derived from transaction creator's public key, as you would expect.
Similarly, UPDATE_NAME has a corresponding "newOwner" field which has been removed.
In addition, UPDATE_NAME now allows users to change their registered name using a new
"newName" field.
Various changes made to DB, Name class, etc. to accomodate above, along with some minor
bug-fixes and comment improvements/corrections.
Needs new unit tests to cover both new functionality and old!
Always add group 0 info to output of API call GET /chats/active/{address}.
No groupName entry as it's "no group" or "group-less" or "not group related".
Timestamp also might be omitted if no message found.
Fix output of POST /chats/compute so it doesn't include zeroed 64-byte signature.
Renamed GET /chats/search to /chats/messages.
Added GET /chats/active/{address} to return lists of group chats
and direct chats involving {address}, where a chat message exists.
Change CHAT API call GET /chat/search to better support the two
main scenarios of:
group-based chatting: supply txGroupId only
private chatting: supply 2 'involving' addresses only
Added some DB indexes to cater for above.
GET /chat/search now returns specialized ChatMessage objects
instead of ChatTransactions. This is to reduce unnecessary fetching
of data from repository, and onward sending to API client.
Previously Controller would loop through the transaction signatures,
discard those already known, and then requesting the full transaction
via peer.getResponse(). This would tie up a networking thread for some
time and also potentially cause repository deadlocks, although the latter
could have been fixed another way.
However, the code after peer.getResponse() was identical to the code
processing an incoming TRANSACTION message. Now instead of requesting
and waiting for then processing each transaction, Controller simply
sends the peer a GET_TRANSACTION for each unknown transaction signature.
As the peer responds with corresponding TRANSACTION messages, these can
be processed individually with shorter period of locking.
When using fixed NTP offset, e.g. via "testNtpoffset" in settings.json,
Controller calls NTP.shutdownNow() which throws a NPE because
NTP.instanceExecutor is null.
Collated all development changes to DB so now we build
initial DB structure directly with final layout.
i.e. no ALTER TABLE, etc.
Reordered HSQLDB 'CREATE TYPE' statements into alphabetical order
for easier maintainability.
Replaced TIMESTAMP WITH TIME ZONE with simple BIGINT ("EpochMillis").
Timezone conversion is now a presentation task, rather than having
pretty values in database.
Removed associated conversion methods, like toOffsetDateTime(),
fromOffsetDateTime() and getZonedTimestampMilli().
Renamed some DB columns to make them more obviously timestamps, like:
Names.registered is now Names.registered_when.
Removed IFNULL(balance, 0) from HSQLDBAccountRepository as balances
are never null, or actually never 0 either.
Added more tests to increase API call, and hence repository, coverage.
Removed unused "milestone block" from Transactions.
In some cases, a freshly cancelled reward-share could still have
an associated signed timestamp. Block.mint() failed to spot this
and used an incorrect "online account" index when building the
to-be-minted block.
Block.mint() now checks that AccountRepository.getRewardShareIndex()
doesn't return null, i.e. indicating that the associated reward-share
for that "online account" no longer exists.
In turn, AccountRepository.getRewardShareIndex() didn't fulfill its
contract of returning null when the passed public key wasn't present
in the repository. So this method has been corrected also.
AccountRepository.rewardShareExists(byte[] publicKey) : boolean added.
BlockMinter had another bug where it didn't check the return from
Block.remint() for null properly. This has been fixed.
BlockMinter now has additional logging, with cool-off to prevent log
spam, for situations where minting could not happen.
Unit test (DisagreementTests) added to cover cancelled reward-share
case above. BlockMinter testing support slightly modified to help.
Brought more into line with isValidUnconfirmed().
No need to update creator's lastReference under new last-ref scheme.
Correspondingly, no need to acquire blockchain lock or repository
shenanigans in getUnconfirmedTransactions() and getInvalidTransactions()
for the same reason.
getInvalidTransactions() seems to be unused and may well be cleaned up
in a future commit.
Change code of the form (assetId aspect not shown):
account.setConfirmedBalance( account.getConfirmedBalance(), amount )
to:
account.modifyAssetBalance( amount )
Also tidied "0 - value" to use unary negate: "- value"
Moved Asset.MULTIPLIER, etc. to Amounts class.
Had to reintroduce BigInteger for asset trading code.
Various helper methods added to Amounts class.
Payment.process/orphan no longer needs unused transaction
signature or reference.
Added post block process/orphan tidying, which currently deletes zero account balances to satisfy post-orphan checks in unit tests.
Fix for possible bug when orphaning TRANSFER_PRIVS.
Added RewardSharePercentTypeAdapter like AmountTypeAdapter.
Replaced a whole load of JAXB-special getters with type-adapters.
Tests looking good!
Now possible thanks to removing Qora v1 support.
Maximum asset quantities now unified to 10_000_000_000,
to 8 decimal places, removing prior 10 billion billion
indivisible maximum.
All values can now fit into a 64bit long.
(Except maybe when processing asset trades).
Added a general-use JAXB AmountTypeAdapter for converting
amounts to/from String/long.
Asset trading engine split into more methods for easier
readability.
Switched to using FIXED founder block reward distribution code,
ready for launch.
In HSQLDBDatabaseUpdates,
QortalAmount changed from DECIMAL(27, 0) to BIGINT
RewardSharePercent added to replace DECIMAL(5,2) with INT
Ripped out unused Transaction.isInvolved and Transaction.getAmount
in all subclasses.
Changed
Transaction.getRecipientAccounts() : List<Account>
to
Transaction.getRecipientAddresses() : List<String>
as only addresses are ever used.
Corrected returned values for above getRecipientAddresses() for
some transaction subclasses.
Added some account caching to some transactions to reduce repeated
loads during validation and then processing.
Transaction transformers:
Changed serialization of asset amounts from using 12 bytes to
now standard 8 byte long.
Updated transaction 'layouts' to reflect new sizes.
RewardShareTransactionTransformer still uses 8byte long to represent
reward share percent.
Updated some unit tests - more work needed!
CHAT transactions don't ever get included into a block.
They use a memory-intensive proof-of-work instead of a fee.
Reference field isn't checked but must be present.
Recipient is optional.
isText/isEncrypted as per MESSAGE, basically indicative flags only.
Some API support.
Memory PoW takes roughly 800ms on Ryzen 3600, maybe 2400ms on QORTector?
As this changes how lastReferences are checked and updated,
this is not suitable for rolling into current chain without a
"feature trigger", or chain restart!
Added unit tests.
NullAccount has 'empty' public key (32 bytes of zeros) compared
with GenesisAccount's vague sometimes 8 bytes, sometimes 32 bytes
public key.
NullAccount has static public key and address, plus overridden
methods to speed up pointless calls like verify().
Genesis Block also tidied up, dropping old Qora v1 compatibility
and using proper block signature and public key to generate
minter's block signature.
Genesis Block transaction processing also simplified, with no need
to access repository to handle fake references, due to new
last-reference code (which will need to be merged).
Dropped support for old, broken RMD160 code.
Qortal is never going to continue off the old Qora blockchain,
so removed all code regarding compatibility.
Removals include:
* various blockchain "feature triggers"
* special Qora-only broken code for various transaction signatures
* "old" asset pricing / trading
* pre-group txGroupId field in transactions
* compatibility unit tests
Possibly safe for roll-out on pre-genesis blockchain?
Tidied up duplicated cross-chain API code that
fetched Qortal AT info.
Added Bitcoin-related cross-chain API calls
for building, checking, refunding and redeeming
P2SH.
Added new Bitcoin-related API error codes.
Controller now starts up, and shuts down, bitcoinj.
Speed-up in BTC class so bitcoinj doesn't have
to throw away all peers and rediscover & reconnect
to them with every chain-related call.
Added API calls to aid Qortal-side of cross-chain trading.
POST /crosschain/build - for building Qortal AT
POST /crosschain/tradeoffer/recipient - for sending trade partner/recipient to AT
POST /crosschain/tradeoffer/secret - for sending secret to AT
DELETE /crosschain/tradeoffer - for cancelling AT
More fixes regarding Blocks processing/orphaning ATs.
More fixes regarding sending/receiving blocks containing AT data.
AT-related fix to genesis block.
Improved cross-chain trading AT code, removing offer-mode timeout
and replacing that with allowing AT creator to cancel offer/end AT
by sending AT the creator's own address as trade partner/recipient.
After all, they're not going to trade with themselves.
Added assertion to check BTCACCT.CODE_BYTES_HASH matches compiled code hash.
Added cross-chain AT's 'mode' for easier diagnosis, either OFFER or TRADE.
We can't use AT's signature to generate AT address because address is needed
before DEPLOY_AT transaction is signed. So we use a hash of signature-less
transaction bytes.
Corresponding changes to tests.
Reworked the cross-chain trading AT so it is now 2-stage:
stage 1: 'offer' mode
waiting for message from creator containing trade partner's address
stage 2: 'trade' mode
waiting for message from trade partner containing secret
Adjusted unit tests to cover above.
Changed QortalATAPI.putCreatorAddressIntoB from storing
creator's public key to actually storing creator's address.
Refactored BTCACCT.AtConstants to CrossChainTradeData.
Now we also store hash of AT's code bytes in DB so we can look up
ATs by what they do. Affects ATData class, ATRepository, etc.
Added "Automated Transactions" and "Cross-Chain" API sections.
New API call GET /at/byfunction/{codehash} for looking up ATs
by what they do, based on hash of their code bytes.
New API call GET /at/{ataddress} for fetching info for specific AT.
New API call GET /at/{ataddress}/data for fetch an AT's data segment.
Mostly for diagnosis of AT's current state.
New API call POST /at for building a raw, unsigned DEPLOY_AT transaction.
New API call GET /crosschain/tradeoffers for finding open BTC-QORT trading ATs.
We require AT v1.3.4 now!
Updated AT-related logging.
Added "isInitial" flag to AT state data so that state data created at
deployment is not added to serialized block data.
Updated BTC-QORT AT code and tests to cover various scenarios.
Added missing 'testNtpOffset' to various test versions of 'settings.json'.
Added missing 'ciyamAtSettings' to various test blockchain configs.
Loads of AT-related additions/fixes/etc. to core code, e.g Block
Requires fix in CIYAM AT v1.3.2
New version of Qortal cross-trade AT code.
Change how Qortal addresses are managed in QortalATAPI from using
base58 strings (that are too long) to using hex form (25 bytes)
as they need to fix into 32 byte A/B register.
Generate AT addresses using DeployAtTransaction's signature instead
of convoluted hash of AT data like name, description, etc.
Add startTime as arg to GetTransaction test app.
Add missing fields (name, description, ATType, tags) to DeployAT test app.
Bump CIYAM AT requirement to v1.3
Remove multi-blockchain AT aspect for now (BlockchainAPI).
For PUT_PREVIOUS_BLOCK_HASH_INTO_A we no longer use SHA256 to condense 64-byte block signature into 32 bytes.
Now we put block height into A1 and SHA192 of signature into A2 through A4.
This allows possible future lookup of block data using "block hash", with verification that it is the same block.
Some AT functions use "address in B" but sometimes we populate B with account's public key instead.
So the method "getAccountFromB" is smart and checks for an actual, textual address in B starting with 'Q', otherwise assumes B contains public key.
The Settings field "useBitcoinTestNet" (boolean) now replaced with "bitcoinNet" (String) with possible values MAIN (default), TEST3, REGTEST.
This allows for more varied development/testing scenarios.
Use correct Bitcoin nSequence value 0xFFFFFFFE for P2SH, i.e. enable locktime, disable RBF.
Roll REGTEST checkpoints file generator into main BTC class.
Yet another rewrite of Bitcoin P2SH scripts for BTC-QORT cross-chain trading.
Added associated test classes BuildP2SH, CheckP2SH, DeployAT (unfinished).
Streamlined BTC class and switched to memory block store.
Split BTCACCTTests into BTCACCT utility class and (so far)
three stand-alone apps: Initiate1, Refund2 and Respond2
Moved some Qortal-specific CIYAM AT constants into blockchain config.
Removed redundant BTCTests
Bump bitcoinj to 0.15.5 for fixes.
lockTime is int (seconds since epoch), not long (ms since epoch).
Improve output of Initiate1.
Added (most of) Respond2.
If the timestamp-pubkey-sig is still 'current' then it'll be in
Controller's list of current online accounts, so we can quickly scan
that list before falling back to the more expensive Ed25519 verify.
Added equals() and hashCode() to OnlineAccountData to support above.
When synchronizing is forced via API call, the SysTray doesn't update
to reflect this.
We fix this by moving the SysTray updating code from
Controller.potentiallySynchronize() to the inner method
Controller.actuallySynchronize(), which is also the method called
directly by the API.
Previously BlockMinter & Synchronizer would both try opportunistic
locking, with no wait/timeout or fairness.
This could lead to a situation where a majority of nodes are
synchronizing, albeit only the top 1 or 2 blocks, but no node
manages to mint within the 'recent' period, so the chain stalls.
However, if a node is at/near the top of the chain then synchronization
shouldn't take very long so we let BlockMinter wait until to 30s
(approx. half typical block time) to obtain lock.
This makes minting blocks more likely in a BlockMinter/Sync fight
which helps keep the chain going.
Detecting chain stalls, and allowing minting if we have plenty of peers,
also produces blockchain 'islands' so isn't a simple fix at this point.
Bumped TCP timeouts for fetching auto-update from 5s (connect) and
3s (read) to 30s (connect) and 10s (read) to allow for nodes with
slower internet connections.
Increased interval between checking for auto-updates from 5 minutes
to 20 minutes to reduce load on update sources and also to reduce
the number of nodes that restart at any one time.
Obviously this new checking interval will only apply after the NEXT
auto-update...
Used when checking that node has shutdown and when replacing old JAR with new update.
ApplyUpdate previously waited 5 seconds between checks/retries, for up to 5 times: 25 seconds.
Now waits 10 seconds, for up to 12 times: 120 seconds.
Hopefully this will give slower nodes enough time to shut down and prevent errors like these on Windows installs:
2020-03-24 12:05:50 INFO ApplyUpdate:114 - Unable to replace JAR: qortal.jar: The process cannot access the file because it is being used by another process.
Added a setting "showBackupNotification", which is false by default,
that shows a tray notification when a repository backup occurs.
Above notification, and the auto-update notification, now refer to
the SysTray i18n translation lookup resources.
Typical users don't need quite so many connections, so minOutboundPeers and
maxPeers reduced accordingly.
maxNetworkThreadPoolSize increased from 10 to 20.
5s is way too long, and even 2s might still be considered excessive.
However, reducing the timeout might also reduce the number of
network engine "spawn failures" due to too many threads tied up
waiting for ping responses from overloaded peers.
Does not affect peer handshaking: that has a separate timeout.
Added Ed25519 private key to public key function accessible from SQL.
Added Ed25519 public key to Qortal address function accessible from SQL.
Used above functions to store minting account public key in SQL to
reduce the number of unnecessarily repeated Ed25519 conversions.
Used above functions to store reward-share minting's accounts address
to reduce the number of unneccessarily repeated PK-to-address conversions.
Reduced the usage of PublicKeyAccount to simply Account where possible,
to reduce the number of Ed25519 conversions.
Account.canMint(), Account.canRewardShare() and Account.getEffectiveMintingLevel()
now only perform 1 repository fetch instead of potentially 2 or more.
Cleaned up NTP main thread to reduce CPU load.
A fixed offset can be applied to NTP.getTime() responses, for both
scenarios when NTP is running or not. Useful for testing or simulating
distant remote peers.
Controller.onNetworkMessage() and Network.onMessage() have both had their
complexity simplified by extracting per-case code to separate methods.
Network's EPC engine's thread pool size no longer hard-coded, but comes
from Settings.maxNetworkThreadPoolSize, which is still 10 by default,
but can be increased for high-availability nodes.
Network's EPC task-producing code streamlined to reduce CPU load.
Generally reduced calls to System.currentTimeMillis(), especially
where the value would only be used in verbose logging situations,
and especially in high-call-volume methods, like within repository.
Keep track of when EPC engine can't spawn a new thread as this
might indicate thread-pool exhaustion and cause some network
messages to be lost.
If logging level is NOT 'trace' (or 'all') then don't call
System.currentTimeMillis() as we'll never use the value.
Similarly, don't set thread names if not logging at 'trace' either.
Update EPC tests, particularly unified per-second/end-of-test stats
reporting.
Added API call GET /peers/enginestats to allow external monitoring.
Extract all engine stats in one synchronized block, instead of
separate calls, for better consistency.
Synchronizer now bails out early when trying to find common block with
a peer. There's no need to keep searching if common block is too far
behind that a TOO_DIVERGENT result would be returned.
fetchSummariesFromCommonBlock() reworked to return a useful
SynchronizationResult directly instead of caller trying to infer
what happened based on null/empty returned list!
Added API call GET /admin/status which reports whether minting
is possible (have minting keys & up-to-date) and whether node is
currently attempting to sync.
Corresponding change to system tray mouseover text.
Corresponding text added to SysTray transaction resources.
Previously BlockMinter would attempt to mint if there were at least
'minBlockchainPeers' connected peers and none of them had an
up-to-date block and we did. This was maybe useful for minting block 2
but possibly causes minting chain islands where a badly connected
node mints by itself, even though connected to not up-to-date peers.
Now BlockMinter requires 'minBlockchainPeers' up-to-date peers, not
simply just connected. This should let synchronization bring the
node up-to-date but does require the node to have better peers.
Currently, the default for minBlockchainPeers is 10. So a node
requires 10 up-to-date peers before it will consider minting. It
might be possible to reduce this in the future to lessen network load.
Previous images had a small hole in the icon (probably a result of a background removal), I just filled it back in with white like it's supposed to be. Also, the previous square icons were streched into a square aspect ratio, these are unstreched.
Although HSQLDB is happy being given unix-style path separator '/'
and converting as necessary on other platforms (e.g. Windows),
manipulation of repository pathnames in Java, outside of HSQLDB,
needs to use platform-specific path separators.
Thus, changes made to replace '/' with File.separator where
necessary.
This should fix repository rebuild errors, which then lead to odd
start-up errors like:
2020-03-11 13:55:19 INFO Controller:270 - Starting repository
2020-03-11 13:55:20 INFO Controller:287 - Validating blockchain
2020-03-11 13:55:20 INFO HSQLDBRepository:227 - Rebuilding repository from scratch
2020-03-11 13:55:20 INFO GenesisBlock:296 - Using genesis block timestamp of 1583870000000
2020-03-11 13:55:21 WARN HSQLDBRepository:720 - Uncommitted changes (882) after connection close, session [3]
java.lang.NullPointerException
at org.qortal.transform.block.BlockTransformer.decodeOnlineAccounts(BlockTransformer.java:422)
at org.qortal.block.Block.getExpandedAccounts(Block.java:546)
at org.qortal.block.Block.increaseAccountLevels(Block.java:1245)
at org.qortal.block.Block.increaseAccountLevels(Block.java:1239)
at org.qortal.block.Block.process(Block.java:1206)
at org.qortal.block.GenesisBlock.process(GenesisBlock.java:345)
at org.qortal.block.BlockChain.rebuildBlockchain(BlockChain.java:526)
at org.qortal.block.BlockChain.validate(BlockChain.java:481)
at org.qortal.controller.Controller.main(Controller.java:289)
The above happens because the old blockchain still exists when trying to process
the genesis block.
AdvancedInstaller's Java launcher EXE seems to use JNI to launch
the JAR, instead of using the command-line 'java' binary directly.
When AI's launcher does this, it adds options like "abort" and "exit",
along with corresponding hook addresses.
These options are returned by the call to
ManagementFactory.getRuntimeMXBean().getInputArguments() which is
done in AutoUpdate while building the command line for launching
ApplyUpdate.
Because command-line 'java' binary doesn't support these options,
they are now stripped out.
No more "node UI". UI provided by 3rd party.
"Open UI" tray icon menu item now attempts to open UI at various
local servers (see Settings.uilocalServers) or some random
remote server (Settings.uiRemoteServers).
Default UI port now 12388 (Settings.uiPort).
Translator class no longer logs warnings for every failed translation.
Commented out unused ApiError enum entries.
Renamed some ApiError names like "_NO_EXISTS" to "_UNKNOWN".
Removed old src/main/resources/globalization/* files.
Added CheckTranslations test app.
Fixed some extraneous/missing ApiError aspects in some API-related classes.
e.g. added NAME_UNKNOWN to GET /names/{name}
We've never needed this before but now it's fixed.
Added corresponding +ve & -ve tests just to make sure.
Only actual use-case that comes to mind is cancelling reward-share.
Before:
0-fee self-share REWARD_SHARE when there is an existing self-share would result in
INSUFFICIENT_FEE from isFeeValid()
After:
isFeeValid() returns OK for above, but isValid() returns SELF_SHARE_EXISTS.
In addition, a transaction that tries to modify existing self-share, even with fee,
also returns SELF_SHARE_EXISTS.
Improved tests to double check.
NOTE: first startup after this commit can take a while due to building index!
SQL statement "DELETE FROM HistoricAccountBalances WHERE height >= ?" required
a full table scan and so was very slow on VMs/routers. This statement used by
HSQLDBAccountRepository.deleteBalancesFromHeight(), itself called during
Block.orphan().
Symptoms particularly evident during shutdown where above statement could take
upwards of 15 minutes on single-CPU, small-memory VMs!
Statement wasn't noticed before as slow-query checking wasn't involved.
Slow-query checking now applies to HSQLDBRepository.delete() and
HSQLDBRepository.exists() calls.
Added test for correct HSQLDB interrupt handling.
Fixed some typos.
Blockchain configs will need "v2Timestamp" feature trigger renaming to "qortalTimestamp"!
Due to Controller.VERSION_PREFIX changing, peer-to-peer protocol version detection has
been changed. Was previous a substring match, now we test peers's buildTimestamp is at
least Peer.V2_PROTOCOL_TIMESTAMP_THRESHOLD. Changes in Peer.java.
Also added comment and throw() to QortalATAPI.getNextTransactionTimestamp()
as this needs given the change to Qortal's block timestamps from legacy Qora.
Changes to HSQLDB data types, e.g. QoraAddress to QortalAddress, means existing
database will need to be thrown away after this commit!
Seems to produce fat JARs with the same MD5 after repeated runs of:
mvn clean package
Needs testing on other platforms, hence this commit.
Also trimmed some unneeded included JAR libraries from pom.xml,
e.g. netty, async-http-client,
and the *.js.map files from Swagger-UI, which seems to reduce
output JAR size from about 51MB to about 44MB.
Converted AccountData's initialLevel to blocksMintedAdjustment.
Corresponding changes to AccountLevelTransaction so that level
set in genesis block is converted to blocksMintedAdjustment,
via cumulativeBlocksByLevel.
Ditto changes to HSQLDBAccountRepository, HSQLDBDatabaseUpdates,
[HSQLDB]TransactionRepository, etc.
Changes to API call POST /admin/mintingaccounts to check passed
reward-share private key maps to a reward-share with minting
account that still has privilege to mint. It's possible for
a TRANSFER_PRIVS transaction to transfer away minting privileges
from a minting account referenced by in a reward-share.
Change to RewardShareTransaction to allow users to cancel a
reward-share even if minting-account component no longer has
minting privs. This should allow users to clean up more after
a privs transfer.
Re-order processing/orphaning in Block.process()/Block.orphan()
to be more consistent and also to take in account changes that
might have been caused by TRANSFER_PRIVS transactions which affect
who might actually receive block rewards/tx fees.
Founders now gain blocksMinted & levels as part of minting blocks.
(Needed to make TRANSFER_PRIVS from a founder account to work).
BlockMinter now has added checks to make sure that the reward-shares
it might use to mint blocks still have valid minting-accounts.
i.e. that the minting-account component of reward-share hasn't had
minting privs transferred away by TRANSFER_PRIVS tx.
Controller now rejects online-accounts from peers that no longer
have minting privs (e.g. transferred away by TRANSFER_PRIVS)
Corresponding, Controller no longer SENDS online-accounts that no
longer have minting privs to other peers.
Added some tests - more tests needed, e.g. for multiple transfers
into the same account, or a test for minting post transfer for both
sender & recipient.
Previously it was possible broadcast an auto-update tx that was
created by a 'dev' group admin.
Now the auto-update tx must be created by a 'dev' group non-admin/owner
and hence require group approval before it takes effect.
To this end, a new TransactionRepository.getLatestAutoUpdateTransaction()
method has been added to simplify finding the latest matching, approved
auto-update transaction.
Corresponding changes also made to AutoUpdate.run()
These changes are for allowing devices to be shipped with an interim blockchain config & genesis block, running with "main-net" settings, prior to launch.
At launch, an "auto-update" will be broadcast to replace blockchain config & genesis block with the final, launch version.
When updated nodes restart (immediately after auto-update) they should notice their existing genesis blocks are invalid and hence wipe their blockchains,
effectively starting the final, launch blockchain.
To this end, genesis block signatures have been changed to better incorporate values that are likely to change at launch, e.g. block timestamp and transaction data.
Added Crypto.dupDigest(), which is effectively Bytes.concat(digest(msg), digest(msg)).
No need for HSQLDB repository to backup, or wipe, non-existent files for "in-memory" databases!
New interim blockchain config/genesis block.
**NOTE** These changes are being committed but require testing. However the auto-update suite requires a pushed commit in order to build an update file!
Now reward-shares with zero percent are valid, to allow the 'recipient' party to gain
"number of minted blocks" but no actual block reward.
Correspondly, the special zero share used to cancel reward-shares has been changed to
be any negative value.
Block rewards, founder 'leftovers': if founder is minter account in any online
reward shares, then the per-founder-share is spread across their online reward-shares,
otherwise it's simply/wholy given to that founder.
Created a new DB table to hold "next block height", updated via triggers on Blocks.
This is so various sub-queries can simply read the next-block-height value instead
of complex IFNULL(MAX(height),0)+1 or SELECT height FROM Blocks ORDER BY height DESC.
Prior code was also broken in edge cases, e.g. no genesis block, or ran slow.
Added tests to cover above.
Deleted BTC tests as they're obsolete.
Added/improved other tests.
*** WARNING ***
Possible block reward bug in this commit. Further investigation needed.
Reverted AccountBalances back to height-less form.
Added HistoricAccountBalances table that is populated via trigger on AccountBalances.
This saves work when performing common requests for latest/confirmed balances,
shunting the extra work to when requesting height-related account balances.
Unified API call GET /addresses/balance/{address} by having address/assetId/height as
query params.
Simpler call for fetching legacy QORA holders during block rewarding.
Improved SQL for fetching asset balances, in all conditions,
e.g. with/without filtering addresses, with/without filtering assetIds,
etc.
Unit test for above to make sure query execution is fast enough.
(At one point, some SQL query was taking 6 seconds!)
Added optional 'height' Integer to AccountBalanceData, but this
is not populated/used very often.
HSQLDBAccountRepository.save(AccountBalanceData) now checks zero balance saves
to see if the row can be deleted instead. This fixes a lot of unhappy tests
that complain that there are residual account balance rows left after
post-test orphaning back to genesis block.
Yet more tests.
Removed very old 'TransactionTests' which are mostly covered in more specific tests elsewhere.
Added cancel-sell-name test from above.
Fixed AssetsApiTests to check for QORT not QORA!
Changed hard-coded assetIDs in test.common.AssetUtils in light of new LEGACY_QORA & QORT_FROM_QORA genesis assets.
Some test blockchain config changes.
Also:
RewardShareKeys app now supports only one arg (minter private key)
in self-reward-share mode, where recipient public key is derived
from minter private key.
Added methods to Account for returning 'effective' minting level
where minting level for founders is read from blockchain config.
(Or returns zero if unable to mint).
Changed two Block constructors into static methods that return
a new Block as there was way too much work being done to really
be called a constructor, especially with all the opportunities
to throw an exception too.
Main blockchain config updated to reflect near-launch version.
Added/changed blockchain weight tests to check block winning
based on higher account levels.
Added Transaction.isFeeValid() to allow transaction subclasses to override and allow zero fees, etc.
Added tests to cover self-reward-shares, including zero fee scenario.
Set 'dilbert' test account to level 8 in test genesis block.
Removed leftover mention of "previous_level" from HSQLDBAccountLevelTransactions,
and AccountLevelTransactionData. (Previous level makes no sense as ACCOUNT_LEVEL
transactions are genesis-block only).
Fixed some incorrect uses of PrivateKeyAccount.getSharedSecret() to
PrivateKeyAccount.getRewardSharePrivateKey() in tests.
Unified terminology from block "generator", "forger", etc. to "minter"
Unified terminology "proxy forger" to "reward share" as it was
incorrect, or at least highly ambiguous, which account had which role.
AccountRepository.findRewardShares() has different arg order!
Account.canMint() now returns true if account has 'founder' flag set.
Added Account.canRewardShare() which returns whether acocunt can create
a reward-share (e.g. level 5+ or founder).
Fixed HSQLDBAssetRepository.getAllAssets() which had incorrect
resultSet column indexes.
Removed old traces of EnableForging transaction.
ACCOUNT_LEVEL and ACCOUNT_FLAGS (genesis-block-only transaction types)
now set target account's last-reference. This is allow later REWARD_SHARE
transactions in genesis block and post-genesis transactions by that account.
REWARD_SHARE transactions are now FREE, but only if minter is also recipient.
If a self-reward-share already exists, then unless share-percent is zero (to
terminate reward-share), any subsequent self-reward-share is invalid.
Updated SysTray i18n properties file.
BlockChain config file requires 'minAccountLevelToRewardShare' and optional
'minAccountLevelToMint'.
Added potential, but currently unused, memory-hard PoW algorithm.
Fixed/removed/disabled some unit tests.
BlockMinter.generateTestingBlock asks Controller to pretend mintingAccount is 'online'.
More testing needed!
Rename Asset.QORA to Asset.QORT so we can also have Asset.LEGACY_QORA
as another hard-coded asset.
Add "is unspendable" aspect to assets where only the asset owner can
transfer/pay asset to other people. Asset trading is barred regardless,
as is use of asset for ATs.
Added "initial level" to account data in preparation for accounts levelling
up from generating blocks.
Added distribution/removal of block reward based on legacy-QORA held.
Removed "previous level" from ACCOUNT_LEVEL transactions as they're
only ever valid in genesis block and so previous level is never needed.
Synchronizer incorrectly tried to compare chain weights in the case
where we had no extra blocks after common block, and thus no blocks
to actually compare.
During chain compare, when more block summaries are needed from peer,
the block signature sent to peer wasn't updated from the last batch
and so the same common block signature was sent. This caused the
code to incorrectly bail out with
"Peer respond with invalid block summary" error as the block heights
didn't match.
Post chain-compare/orphan (if any), and when fetching block signatures,
the code referenced peerBlockSummaries where it should have been
referencing peerBlockSignatures instead. This caused to code to incorrectly
bail out with the "Peer failed to respond with more block signatures..."
error.
Updated pom.xml.
Updated dependencies, including various minor code mods, esp. bitcoin-related.
Starts up and talks to Java 8 nodes!
Dev environment should be at least Eclipse 4.11 with m2e 1.9.1+
API call GET /addresses/online reports online accounts,
including both addresses relating to the proxy-forge public key.
New PeerChainTipData class to replace the broken "peer data lock"
that was supposed to make sure peer's last height/blockSig/timestamp
were all in sync. Now peer's chain tip data is a single object
reference that can be replaced in one go.
Removed pointless API calls /blocks/time and /blocks/{generatingbalance}.
Various changes, mostly in Block class, to do with switching to BlockTimingByHeight
from old min/max block time.
New block 'weight' based on number of online accounts
and 'distance' of perturbed generator public key from 'ideal' public key
(for that particular block's height).
New sub-chain 'weight' based on accumulating block weights,
currently by shifting previous accumulator left by 8 bits then
adding next block's weight.
More validation of BlockChain config. Helpful for debugging, probably
not very useful to end-users.
BlockGenerator now uses unified Peer predicates from Controller, like:
Controller.hasMisbehaved, Controller.hasNoRecentBlock, etc.
Controller now keeps a list of chain-tip signatures that are for inferior
chains, so it doesn't try to synchronize with peers with inferior chains.
(This list is wiped when node's blockchain changes/block is generated).
Controller now asks Gui to display error box if it can't parse Settings.
Controller.potentiallySynchronize() does more filtering of potential peers
before calling actuallySynchronize(). (Mostly moved from Synchronizer,
so now we expect actuallySynchronize() to do something rather than bail
out because it doesn't like the peer after all).
If synchronization discovers that peer has an inferior chain,
then Controller notifies that peer of our superior chain, to help keep
the network in sync.
Renamed OnlineAccount to OnlineAccountData, as it is in package org.qora.data
after all...
Synchronizer reworked to request block summaries so it can judge which chain
is better, and hence whether to sync with peer or abort.
Slight optimization of Peer.readChannel() to exit earlier if no more network
messages can be extracted from buffer.
More tests.
Improved documentation and logging.
Block.MAX_BLOCK_BYTES now BlockChain.getMaxBlockSize()
Network.MAXIMUM_MESSAGE_SIZE now Network.getMaxMessageSize() as
it depends on block size (above).
Block data now includes number of online accounts, as encoded online account indexes can't be
validated by ConciseSet it seems.
Corresponding changes to repository, transformer, block validation, data object, block summaries...
Block timestamps are now calculated using parent block data and generator's public key,
instead of old qora1 generating balance code.
Generators are valid to forge if they have forging flag enabled. This will probably change
to an account-level check in the near future.
Added trimming of old online accounts signatures from blocks.
Tidied up SysTray/BlockGenerator generation enabled/possible flag.
Although we perform online accounts tasks (currently) every 10 seconds,
only broadcast our online accounts every 60 seconds.
In Controller.main(), if args are present then use first as a filename to
settings JSON file (overriding the default filename).
Still to do: change Block/BlockChain/Synchronizer to prefer blocks with more online accounts,
failing that use generator nearest 'ideal', etc.
Safety commit in case of data loss!
Lots of changes to do with "online accounts", including:
* networking
+ GET_ONLINE_ACCOUNTS * ONLINE_ACCOUNTS messages & handling
+ Changes to serialization of block data to include online accounts info
* block-related
+ Adding online accounts info when generating blocks
+ Validating online accounts info in block data
+ Repository changes to store online accounts info
* Controller
+ Managing in-memory cache of online accounts
+ Updating/broadcasting our online accounts
* BlockChain config
Added "account levels", so new code/changes required in the usual places, like:
* transaction data object
* repository
* transaction transformer
* transaction processing
Orphaning a BUY_NAME transaction would not reinstate the
sale price. Sale price could be nullified by (e.g.) orphaning
a SELL_NAME transaction.
Also added test case to cover above and other test-related support,
e.g. test-mode in NTP.
Added "volatile" to more fields, for thread-safety on reads.
Changes to field values are done inside synchronized blocks
so no need for AtomicInteger/AtomicBoolean. (Could be changed
in the future to show intention/readability though).
Added more statistics (tasks produced/consumed).
Limited Network's EPC executor to 10 threads max.
Some asset-related transactions (CREATE_ASSET_ORDER and TRANSFER_ASSET)
also try to fetch asset names at the same time.
If one of these transactions, and a corresponding ISSUE_ASSET transaction,
have been orphaned then it's possible that the asset no longer exists.
Thus the SQL "JOIN" fails during transaction retrieval, causing an error.
Changing the table-join to "LEFT OUTER JOIN" makes the asset name aspect
optional. Repercussions might be nameless assets when fetching transaction
info via API.
Transaction expiry wasn't happening. Use NTP.getTime to check whether
transactions have expired. Also reject expired transactions when trying
to add them to unconfirmed pool.
Sometimes producing a task took way too long, causing massive
spikes in the number of threads and peer disconnections.
This is down to a repository pool exhaustion, so
RepositoryManager.getRepository() would block (for up to 5 minutes).
The key method at fault was Network.getConnectablePeer().
Various fixes:
NetworkProcessor's executor now reaps old threads after only 10 seconds
instead of the usual 60 seconds.
Change logging in Network to help diagnose disconnection and repository
issues.
RepositoryManager now has a tryRepository() call that is non-blocking
and returns null if repository pool is exhausted.
Repository pool size increased from default (10) to 100.
Pruning peers is now opportunistic, using tryRepository(), and returns
early if repository pool is exhausted.
getConnectablePeer() is now opportunistic, using tryRepository(), and
returns null (no peer candidate for connection) if repository pool
is exhausted.
Merging peers is not opportunistic, using tryRepository().
Peer ping interval increased from 8s to 20s.
HSQLDBRepositoryFactory now logs when getConnection() takes over 1000ms.
Added more trace-level logging to ExecuteProduceConsume to
highlight slow produceTask() calls.
Symptoms were on Ubuntu with 8u222:
Exception in thread "Controller" java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:807)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
at java.awt.SystemTray.isSupported(SystemTray.java:219)
at org.qora.gui.SysTray.<init>(SysTray.java:50)
at org.qora.gui.SysTray.getInstance(SysTray.java:249)
at org.qora.controller.Controller.updateSysTray(Controller.java:480)
at org.qora.controller.Controller.run(Controller.java:368)
(Underlying cause not known)
This would cause Controller thread to silently exit, and so no
synchronization would occur. Node would still have connections
and thus happily generate its own blocks on its on fork.
Maybe other peers would try to sync with this node but would
likely reject this node's chain after a short while.
New NTP class now runs as a simplistic NTP client, repeatedly polling
several NTP servers and maintaining a more accurate time independent
of operating system.
Several occurrences of System.currentTimeMillis() replaced with NTP.getTime()
particularly where block/transaction/networking is involved.
GET /admin/info now includes "currentTimestamp" as reported from NTP.
Added support for block timestamps determined by generator, instead of
supplied by clock. (BlockChain.newBlockTimestampHeight - not yet activated).
Incorrect timestamps will produce a TIMESTAMP_INCORRECT Block.ValidationResult.
Block.calcMinimumTimestamp repurposed as Block.calcTimestamp for above.
Block timestamps are now allowed to be max 2000ms in the future,
was previously max 500ms.
Block generation prohibited until initial NTP sync.
Instead of deleting INVALID unconfirmed transactions in BlockGenerator,
Controller now deletes EXPIRED unconfirmed transactions every so often.
This also fixes persistent expired unconfirmed transactions on nodes
that do not generate blocks, as BlockGenerator.deleteInvalidTransactions()
was never reached.
Abbreviated block sigs added to log entries declaring a new block is generated
in BlockGenerator.
Controller checks for NTP sync much faster during start-up and SysTray's
tooltip text starts as "Synchronizing clock" until NTP sync occurs.
After NTP sync, Controller logs NTP offset every so often (currently every 5 mins).
When considering synchronizing, Controller skips peers that have the same block sig
as last time when synchronization resulted in no action, e.g. INFERIOR_CHAIN,
NOTHING_TO_DO and also OK. OK is included as another sync attempt would result in
NOTHING_TO_DO.
Previously this skipping check only happened after prior INFERIOR_CHAIN.
During inbound peer handshaking, if we receive a peer ID that matches an existing inbound
peer then send peer ID of all zeros, then close connection.
Remote end should detect this and cleanly close connection instead of waiting for handshake timeout.
Randomly generated peer IDs have lowest bit set to avoid all zeros.
Might need further work.
Networking doesn't connect, or accept, until NTP has synced.
Transaction validation can fail with CLOCK_NOT_SYNCED if NTP not synced.
It seems unachievable for nodes to keep their clocks accurate to
within 500ms. It is unclear whether this is due to Windows'
implementation of client NTP or because of terrible network
conditions in China.
So increasing max NTP offset to allow block generation from
500ms to 30s. Correspondingly increasing max peer timestamp
delta from 5s to 30s.
The block consensus algorithm will need to change in the near
future to address clock issues.
Controller batches SysTray updates into one per second.
Block generation only allowed by Controller is clock
known to be accurate. ('not sure' stops generation).
NTP MAX_STDDEV increased from 25ms to 125ms to cater
for poorly connected nodes.
Controller sends peers list over outbound peer connections,
requests peers list from inbound peer connections.
When peer handshake completes, Network & Controller only send
initial messages over outbound peer connections.
This is to fix HEIGHT_V2 messages being processed out-of-order
breaking handshaking, as indicated by log entries like:
2019-07-26 16:16:35 DEBUG Network:702 - Unexpected HEIGHT_V2 message from xx.xxx.xx.xx:pppp, expected PROOF
2019-07-26 16:16:35 DEBUG Network:840 - Handshake completed with peer xx.xxx.xx.xx:pppp
Increased connection failure backoff from 1 minute to 5 minutes,
as handshake timeout is 1 minute and then nodes would immediately reconnect.
Changed default NTP servers from asia to cn.
Controller performs NTP check on startup (and every 5 minutes)
which determines whether block generation is allowed.
System Tray tooltip updated to reflect generating status.
Plus new translations.
Improved GuiTests.
BlockGenerator fetches forging accounts first, and sleeps
if none configured, which is less work than processing peer lists.
Now uses several NTP servers to determine mean offset from
system clock to internet time.
If abs(offset) > 500ms or NTP service not running then
user is 'nagged' via system tray pop-up notification
with instructions on how to fix.
Also improved system tray translations!
SysTray pop-up menu now includes entry for launching https://time.is
so node owners can check their system clocks against internet time.
Windows installs also have additional systray menu entry which
runs ntpcfg.bat script, included in resources.
Also available as download via node-UI servlet,
e.g. http://localhost:9880/downloads/ntpcfg.bat
ntpcfg.bat reconfigures Windows Time Service with many NTP servers,
restarts the service, and also makes sure it auto-starts on boot.
Added DEBUG-level logging when rejecting nodes due to excessive
time difference (during PROOF handshake stage).
Bumped default settings values for minOutboundPeers from 10 to 20.
Bumped default settings values for maxPeers from 30 to 50.
Defer the clearing of hasThreadPending flag until about to produce a task,
inside synchronized block.
This gives a new thread a chance to produce at least once before other threads
decide to spawn new threads.
Previously there could be an excessive number of unncessary threads,
all waiting for their initial attempt to produce a task.
Reworked networking execute-produce-consume threading.
Some networking task were wrongly performed during 'produce' phase,
and some producing was happening in 'consume' phase (also corrected).
Peer connection tasks are rate-limited to 1 per second to reduce CPU thrashing.
Show P2P listen port in logs on startup.
Tests for general purpose ExecuteProduceConsume class to cover both
random task scenario and mass-ping scenario.
Instead of 3 threads per peer:
1. peer main thread
2. peer's unsolicited messages processor
3. peer pinger
We now use a Jetty-style Execute-Produce-Consume server threading model.
For 60 connected peers, we no longer have 180 threads but typically only
the usual ~6 threads.
Also in this commit:
* merging peers locking changed from lock() to tryLock()
* PROOF handshake maximum time difference increased from 2000ms to 5000ms
* Peers still handshaking after 60s are considered stuck and hence disconnected
* We now use NIO SocketChannels instead of raw sockets
Change AutoUpdate to use 'quick' backup.
For HSQLDBRepository.backup, don't perform CHECKPOINT DEFRAG while repository in use
as it never completes.
Similarly, use BACKUP DATABASE ... NOT BLOCKING.
Controller requests 'quick' repository backup every 123 minutes.
On start-up, if repository fails to load, recovery is attempted using
backup (if present).
AutoUpdate also requests 'slow' repository backup just before
calling ApplyUpdate. ('Slow' means perform "CHECKPOINT DEFRAG" first).
Network.shutdown() called Peer.shutdown() on each Peer
while holding synchronization lock on this.connectedPeers.
This would cause a problem during Peer.shutdown() as some
other reachable code would also want synchronized access
to this.connectedPeers. Typical symptoms would be log
entries like:
2019-07-02 11:13:05 DEBUG Peer:512 - Message processor for peer 192.144.182.73:9889 failed to terminate
Eventually Network.shutdown() would exit, releasing synchronization
lock and awaking stuck Peer threads, which could then try to access
repository (now closed) causing further log spam.
Now it uses Network.getConnectedPeers to return duplicated
List<Peer>, minimizing lock time on this.connectedPeers.
Also made Peer main thread logging more informative when a IOException
occurs, as most situations are harmless EOF or connection reset by peer.
Refactored duplicate code into Transaction.setInitialApprovalStatus().
This is make sure transactions HAVE a group-approval status
in Synchronizer before Block.isValid is called.
This wasn't a problem for new, unconfirmed, individual transactions
arriving over the wire due to Transaction.importAsUnconfirmed()
doing the right thing.
Also added a groupApprovalTimestamp to BlockChain feature-triggers
to support legacy chains.
Tested by syncing mainnet from scratch.
