Previously it was possible broadcast an auto-update tx that was
created by a 'dev' group admin.
Now the auto-update tx must be created by a 'dev' group non-admin/owner
and hence require group approval before it takes effect.
To this end, a new TransactionRepository.getLatestAutoUpdateTransaction()
method has been added to simplify finding the latest matching, approved
auto-update transaction.
Corresponding changes also made to AutoUpdate.run()
These changes are for allowing devices to be shipped with an interim blockchain config & genesis block, running with "main-net" settings, prior to launch.
At launch, an "auto-update" will be broadcast to replace blockchain config & genesis block with the final, launch version.
When updated nodes restart (immediately after auto-update) they should notice their existing genesis blocks are invalid and hence wipe their blockchains,
effectively starting the final, launch blockchain.
To this end, genesis block signatures have been changed to better incorporate values that are likely to change at launch, e.g. block timestamp and transaction data.
Added Crypto.dupDigest(), which is effectively Bytes.concat(digest(msg), digest(msg)).
No need for HSQLDB repository to backup, or wipe, non-existent files for "in-memory" databases!
New interim blockchain config/genesis block.
**NOTE** These changes are being committed but require testing. However the auto-update suite requires a pushed commit in order to build an update file!
Now reward-shares with zero percent are valid, to allow the 'recipient' party to gain
"number of minted blocks" but no actual block reward.
Correspondly, the special zero share used to cancel reward-shares has been changed to
be any negative value.
Block rewards, founder 'leftovers': if founder is minter account in any online
reward shares, then the per-founder-share is spread across their online reward-shares,
otherwise it's simply/wholy given to that founder.
Created a new DB table to hold "next block height", updated via triggers on Blocks.
This is so various sub-queries can simply read the next-block-height value instead
of complex IFNULL(MAX(height),0)+1 or SELECT height FROM Blocks ORDER BY height DESC.
Prior code was also broken in edge cases, e.g. no genesis block, or ran slow.
Added tests to cover above.
Deleted BTC tests as they're obsolete.
Added/improved other tests.
*** WARNING ***
Possible block reward bug in this commit. Further investigation needed.
Reverted AccountBalances back to height-less form.
Added HistoricAccountBalances table that is populated via trigger on AccountBalances.
This saves work when performing common requests for latest/confirmed balances,
shunting the extra work to when requesting height-related account balances.
Unified API call GET /addresses/balance/{address} by having address/assetId/height as
query params.
Simpler call for fetching legacy QORA holders during block rewarding.
Improved SQL for fetching asset balances, in all conditions,
e.g. with/without filtering addresses, with/without filtering assetIds,
etc.
Unit test for above to make sure query execution is fast enough.
(At one point, some SQL query was taking 6 seconds!)
Added optional 'height' Integer to AccountBalanceData, but this
is not populated/used very often.
HSQLDBAccountRepository.save(AccountBalanceData) now checks zero balance saves
to see if the row can be deleted instead. This fixes a lot of unhappy tests
that complain that there are residual account balance rows left after
post-test orphaning back to genesis block.
Yet more tests.
Removed very old 'TransactionTests' which are mostly covered in more specific tests elsewhere.
Added cancel-sell-name test from above.
Fixed AssetsApiTests to check for QORT not QORA!
Changed hard-coded assetIDs in test.common.AssetUtils in light of new LEGACY_QORA & QORT_FROM_QORA genesis assets.
Some test blockchain config changes.
Also:
RewardShareKeys app now supports only one arg (minter private key)
in self-reward-share mode, where recipient public key is derived
from minter private key.
Added methods to Account for returning 'effective' minting level
where minting level for founders is read from blockchain config.
(Or returns zero if unable to mint).
Changed two Block constructors into static methods that return
a new Block as there was way too much work being done to really
be called a constructor, especially with all the opportunities
to throw an exception too.
Main blockchain config updated to reflect near-launch version.
Added/changed blockchain weight tests to check block winning
based on higher account levels.
Added Transaction.isFeeValid() to allow transaction subclasses to override and allow zero fees, etc.
Added tests to cover self-reward-shares, including zero fee scenario.
Set 'dilbert' test account to level 8 in test genesis block.
Removed leftover mention of "previous_level" from HSQLDBAccountLevelTransactions,
and AccountLevelTransactionData. (Previous level makes no sense as ACCOUNT_LEVEL
transactions are genesis-block only).
Fixed some incorrect uses of PrivateKeyAccount.getSharedSecret() to
PrivateKeyAccount.getRewardSharePrivateKey() in tests.
Unified terminology from block "generator", "forger", etc. to "minter"
Unified terminology "proxy forger" to "reward share" as it was
incorrect, or at least highly ambiguous, which account had which role.
AccountRepository.findRewardShares() has different arg order!
Account.canMint() now returns true if account has 'founder' flag set.
Added Account.canRewardShare() which returns whether acocunt can create
a reward-share (e.g. level 5+ or founder).
Fixed HSQLDBAssetRepository.getAllAssets() which had incorrect
resultSet column indexes.
Removed old traces of EnableForging transaction.
ACCOUNT_LEVEL and ACCOUNT_FLAGS (genesis-block-only transaction types)
now set target account's last-reference. This is allow later REWARD_SHARE
transactions in genesis block and post-genesis transactions by that account.
REWARD_SHARE transactions are now FREE, but only if minter is also recipient.
If a self-reward-share already exists, then unless share-percent is zero (to
terminate reward-share), any subsequent self-reward-share is invalid.
Updated SysTray i18n properties file.
BlockChain config file requires 'minAccountLevelToRewardShare' and optional
'minAccountLevelToMint'.
Added potential, but currently unused, memory-hard PoW algorithm.
Fixed/removed/disabled some unit tests.
BlockMinter.generateTestingBlock asks Controller to pretend mintingAccount is 'online'.
More testing needed!
Rename Asset.QORA to Asset.QORT so we can also have Asset.LEGACY_QORA
as another hard-coded asset.
Add "is unspendable" aspect to assets where only the asset owner can
transfer/pay asset to other people. Asset trading is barred regardless,
as is use of asset for ATs.
Added "initial level" to account data in preparation for accounts levelling
up from generating blocks.
Added distribution/removal of block reward based on legacy-QORA held.
Removed "previous level" from ACCOUNT_LEVEL transactions as they're
only ever valid in genesis block and so previous level is never needed.
Synchronizer incorrectly tried to compare chain weights in the case
where we had no extra blocks after common block, and thus no blocks
to actually compare.
During chain compare, when more block summaries are needed from peer,
the block signature sent to peer wasn't updated from the last batch
and so the same common block signature was sent. This caused the
code to incorrectly bail out with
"Peer respond with invalid block summary" error as the block heights
didn't match.
Post chain-compare/orphan (if any), and when fetching block signatures,
the code referenced peerBlockSummaries where it should have been
referencing peerBlockSignatures instead. This caused to code to incorrectly
bail out with the "Peer failed to respond with more block signatures..."
error.
Updated pom.xml.
Updated dependencies, including various minor code mods, esp. bitcoin-related.
Starts up and talks to Java 8 nodes!
Dev environment should be at least Eclipse 4.11 with m2e 1.9.1+
API call GET /addresses/online reports online accounts,
including both addresses relating to the proxy-forge public key.
New PeerChainTipData class to replace the broken "peer data lock"
that was supposed to make sure peer's last height/blockSig/timestamp
were all in sync. Now peer's chain tip data is a single object
reference that can be replaced in one go.
Removed pointless API calls /blocks/time and /blocks/{generatingbalance}.
Various changes, mostly in Block class, to do with switching to BlockTimingByHeight
from old min/max block time.
New block 'weight' based on number of online accounts
and 'distance' of perturbed generator public key from 'ideal' public key
(for that particular block's height).
New sub-chain 'weight' based on accumulating block weights,
currently by shifting previous accumulator left by 8 bits then
adding next block's weight.
More validation of BlockChain config. Helpful for debugging, probably
not very useful to end-users.
BlockGenerator now uses unified Peer predicates from Controller, like:
Controller.hasMisbehaved, Controller.hasNoRecentBlock, etc.
Controller now keeps a list of chain-tip signatures that are for inferior
chains, so it doesn't try to synchronize with peers with inferior chains.
(This list is wiped when node's blockchain changes/block is generated).
Controller now asks Gui to display error box if it can't parse Settings.
Controller.potentiallySynchronize() does more filtering of potential peers
before calling actuallySynchronize(). (Mostly moved from Synchronizer,
so now we expect actuallySynchronize() to do something rather than bail
out because it doesn't like the peer after all).
If synchronization discovers that peer has an inferior chain,
then Controller notifies that peer of our superior chain, to help keep
the network in sync.
Renamed OnlineAccount to OnlineAccountData, as it is in package org.qora.data
after all...
Synchronizer reworked to request block summaries so it can judge which chain
is better, and hence whether to sync with peer or abort.
Slight optimization of Peer.readChannel() to exit earlier if no more network
messages can be extracted from buffer.
More tests.
Improved documentation and logging.
Block.MAX_BLOCK_BYTES now BlockChain.getMaxBlockSize()
Network.MAXIMUM_MESSAGE_SIZE now Network.getMaxMessageSize() as
it depends on block size (above).
Block data now includes number of online accounts, as encoded online account indexes can't be
validated by ConciseSet it seems.
Corresponding changes to repository, transformer, block validation, data object, block summaries...
Block timestamps are now calculated using parent block data and generator's public key,
instead of old qora1 generating balance code.
Generators are valid to forge if they have forging flag enabled. This will probably change
to an account-level check in the near future.
Added trimming of old online accounts signatures from blocks.
Tidied up SysTray/BlockGenerator generation enabled/possible flag.
Although we perform online accounts tasks (currently) every 10 seconds,
only broadcast our online accounts every 60 seconds.
In Controller.main(), if args are present then use first as a filename to
settings JSON file (overriding the default filename).
Still to do: change Block/BlockChain/Synchronizer to prefer blocks with more online accounts,
failing that use generator nearest 'ideal', etc.
Safety commit in case of data loss!
Lots of changes to do with "online accounts", including:
* networking
+ GET_ONLINE_ACCOUNTS * ONLINE_ACCOUNTS messages & handling
+ Changes to serialization of block data to include online accounts info
* block-related
+ Adding online accounts info when generating blocks
+ Validating online accounts info in block data
+ Repository changes to store online accounts info
* Controller
+ Managing in-memory cache of online accounts
+ Updating/broadcasting our online accounts
* BlockChain config
Added "account levels", so new code/changes required in the usual places, like:
* transaction data object
* repository
* transaction transformer
* transaction processing
Orphaning a BUY_NAME transaction would not reinstate the
sale price. Sale price could be nullified by (e.g.) orphaning
a SELL_NAME transaction.
Also added test case to cover above and other test-related support,
e.g. test-mode in NTP.
Added "volatile" to more fields, for thread-safety on reads.
Changes to field values are done inside synchronized blocks
so no need for AtomicInteger/AtomicBoolean. (Could be changed
in the future to show intention/readability though).
Added more statistics (tasks produced/consumed).
Limited Network's EPC executor to 10 threads max.
Some asset-related transactions (CREATE_ASSET_ORDER and TRANSFER_ASSET)
also try to fetch asset names at the same time.
If one of these transactions, and a corresponding ISSUE_ASSET transaction,
have been orphaned then it's possible that the asset no longer exists.
Thus the SQL "JOIN" fails during transaction retrieval, causing an error.
Changing the table-join to "LEFT OUTER JOIN" makes the asset name aspect
optional. Repercussions might be nameless assets when fetching transaction
info via API.
Transaction expiry wasn't happening. Use NTP.getTime to check whether
transactions have expired. Also reject expired transactions when trying
to add them to unconfirmed pool.
Sometimes producing a task took way too long, causing massive
spikes in the number of threads and peer disconnections.
This is down to a repository pool exhaustion, so
RepositoryManager.getRepository() would block (for up to 5 minutes).
The key method at fault was Network.getConnectablePeer().
Various fixes:
NetworkProcessor's executor now reaps old threads after only 10 seconds
instead of the usual 60 seconds.
Change logging in Network to help diagnose disconnection and repository
issues.
RepositoryManager now has a tryRepository() call that is non-blocking
and returns null if repository pool is exhausted.
Repository pool size increased from default (10) to 100.
Pruning peers is now opportunistic, using tryRepository(), and returns
early if repository pool is exhausted.
getConnectablePeer() is now opportunistic, using tryRepository(), and
returns null (no peer candidate for connection) if repository pool
is exhausted.
Merging peers is not opportunistic, using tryRepository().
Peer ping interval increased from 8s to 20s.
HSQLDBRepositoryFactory now logs when getConnection() takes over 1000ms.
Added more trace-level logging to ExecuteProduceConsume to
highlight slow produceTask() calls.
Symptoms were on Ubuntu with 8u222:
Exception in thread "Controller" java.awt.AWTError: Assistive Technology not found: org.GNOME.Accessibility.AtkWrapper
at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:807)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
at java.awt.SystemTray.isSupported(SystemTray.java:219)
at org.qora.gui.SysTray.<init>(SysTray.java:50)
at org.qora.gui.SysTray.getInstance(SysTray.java:249)
at org.qora.controller.Controller.updateSysTray(Controller.java:480)
at org.qora.controller.Controller.run(Controller.java:368)
(Underlying cause not known)
This would cause Controller thread to silently exit, and so no
synchronization would occur. Node would still have connections
and thus happily generate its own blocks on its on fork.
Maybe other peers would try to sync with this node but would
likely reject this node's chain after a short while.
New NTP class now runs as a simplistic NTP client, repeatedly polling
several NTP servers and maintaining a more accurate time independent
of operating system.
Several occurrences of System.currentTimeMillis() replaced with NTP.getTime()
particularly where block/transaction/networking is involved.
GET /admin/info now includes "currentTimestamp" as reported from NTP.
Added support for block timestamps determined by generator, instead of
supplied by clock. (BlockChain.newBlockTimestampHeight - not yet activated).
Incorrect timestamps will produce a TIMESTAMP_INCORRECT Block.ValidationResult.
Block.calcMinimumTimestamp repurposed as Block.calcTimestamp for above.
Block timestamps are now allowed to be max 2000ms in the future,
was previously max 500ms.
Block generation prohibited until initial NTP sync.
Instead of deleting INVALID unconfirmed transactions in BlockGenerator,
Controller now deletes EXPIRED unconfirmed transactions every so often.
This also fixes persistent expired unconfirmed transactions on nodes
that do not generate blocks, as BlockGenerator.deleteInvalidTransactions()
was never reached.
Abbreviated block sigs added to log entries declaring a new block is generated
in BlockGenerator.
Controller checks for NTP sync much faster during start-up and SysTray's
tooltip text starts as "Synchronizing clock" until NTP sync occurs.
After NTP sync, Controller logs NTP offset every so often (currently every 5 mins).
When considering synchronizing, Controller skips peers that have the same block sig
as last time when synchronization resulted in no action, e.g. INFERIOR_CHAIN,
NOTHING_TO_DO and also OK. OK is included as another sync attempt would result in
NOTHING_TO_DO.
Previously this skipping check only happened after prior INFERIOR_CHAIN.
During inbound peer handshaking, if we receive a peer ID that matches an existing inbound
peer then send peer ID of all zeros, then close connection.
Remote end should detect this and cleanly close connection instead of waiting for handshake timeout.
Randomly generated peer IDs have lowest bit set to avoid all zeros.
Might need further work.
Networking doesn't connect, or accept, until NTP has synced.
Transaction validation can fail with CLOCK_NOT_SYNCED if NTP not synced.
It seems unachievable for nodes to keep their clocks accurate to
within 500ms. It is unclear whether this is due to Windows'
implementation of client NTP or because of terrible network
conditions in China.
So increasing max NTP offset to allow block generation from
500ms to 30s. Correspondingly increasing max peer timestamp
delta from 5s to 30s.
The block consensus algorithm will need to change in the near
future to address clock issues.
Controller batches SysTray updates into one per second.
Block generation only allowed by Controller is clock
known to be accurate. ('not sure' stops generation).
NTP MAX_STDDEV increased from 25ms to 125ms to cater
for poorly connected nodes.
Controller sends peers list over outbound peer connections,
requests peers list from inbound peer connections.
When peer handshake completes, Network & Controller only send
initial messages over outbound peer connections.
This is to fix HEIGHT_V2 messages being processed out-of-order
breaking handshaking, as indicated by log entries like:
2019-07-26 16:16:35 DEBUG Network:702 - Unexpected HEIGHT_V2 message from xx.xxx.xx.xx:pppp, expected PROOF
2019-07-26 16:16:35 DEBUG Network:840 - Handshake completed with peer xx.xxx.xx.xx:pppp
Increased connection failure backoff from 1 minute to 5 minutes,
as handshake timeout is 1 minute and then nodes would immediately reconnect.
Changed default NTP servers from asia to cn.
Controller performs NTP check on startup (and every 5 minutes)
which determines whether block generation is allowed.
System Tray tooltip updated to reflect generating status.
Plus new translations.
Improved GuiTests.
BlockGenerator fetches forging accounts first, and sleeps
if none configured, which is less work than processing peer lists.
Now uses several NTP servers to determine mean offset from
system clock to internet time.
If abs(offset) > 500ms or NTP service not running then
user is 'nagged' via system tray pop-up notification
with instructions on how to fix.
Also improved system tray translations!
SysTray pop-up menu now includes entry for launching https://time.is
so node owners can check their system clocks against internet time.
Windows installs also have additional systray menu entry which
runs ntpcfg.bat script, included in resources.
Also available as download via node-UI servlet,
e.g. http://localhost:9880/downloads/ntpcfg.bat
ntpcfg.bat reconfigures Windows Time Service with many NTP servers,
restarts the service, and also makes sure it auto-starts on boot.
Added DEBUG-level logging when rejecting nodes due to excessive
time difference (during PROOF handshake stage).
Bumped default settings values for minOutboundPeers from 10 to 20.
Bumped default settings values for maxPeers from 30 to 50.
Defer the clearing of hasThreadPending flag until about to produce a task,
inside synchronized block.
This gives a new thread a chance to produce at least once before other threads
decide to spawn new threads.
Previously there could be an excessive number of unncessary threads,
all waiting for their initial attempt to produce a task.
Reworked networking execute-produce-consume threading.
Some networking task were wrongly performed during 'produce' phase,
and some producing was happening in 'consume' phase (also corrected).
Peer connection tasks are rate-limited to 1 per second to reduce CPU thrashing.
Show P2P listen port in logs on startup.
Tests for general purpose ExecuteProduceConsume class to cover both
random task scenario and mass-ping scenario.
Instead of 3 threads per peer:
1. peer main thread
2. peer's unsolicited messages processor
3. peer pinger
We now use a Jetty-style Execute-Produce-Consume server threading model.
For 60 connected peers, we no longer have 180 threads but typically only
the usual ~6 threads.
Also in this commit:
* merging peers locking changed from lock() to tryLock()
* PROOF handshake maximum time difference increased from 2000ms to 5000ms
* Peers still handshaking after 60s are considered stuck and hence disconnected
* We now use NIO SocketChannels instead of raw sockets
Change AutoUpdate to use 'quick' backup.
For HSQLDBRepository.backup, don't perform CHECKPOINT DEFRAG while repository in use
as it never completes.
Similarly, use BACKUP DATABASE ... NOT BLOCKING.
Controller requests 'quick' repository backup every 123 minutes.
On start-up, if repository fails to load, recovery is attempted using
backup (if present).
AutoUpdate also requests 'slow' repository backup just before
calling ApplyUpdate. ('Slow' means perform "CHECKPOINT DEFRAG" first).
Network.shutdown() called Peer.shutdown() on each Peer
while holding synchronization lock on this.connectedPeers.
This would cause a problem during Peer.shutdown() as some
other reachable code would also want synchronized access
to this.connectedPeers. Typical symptoms would be log
entries like:
2019-07-02 11:13:05 DEBUG Peer:512 - Message processor for peer 192.144.182.73:9889 failed to terminate
Eventually Network.shutdown() would exit, releasing synchronization
lock and awaking stuck Peer threads, which could then try to access
repository (now closed) causing further log spam.
Now it uses Network.getConnectedPeers to return duplicated
List<Peer>, minimizing lock time on this.connectedPeers.
Also made Peer main thread logging more informative when a IOException
occurs, as most situations are harmless EOF or connection reset by peer.
Refactored duplicate code into Transaction.setInitialApprovalStatus().
This is make sure transactions HAVE a group-approval status
in Synchronizer before Block.isValid is called.
This wasn't a problem for new, unconfirmed, individual transactions
arriving over the wire due to Transaction.importAsUnconfirmed()
doing the right thing.
Also added a groupApprovalTimestamp to BlockChain feature-triggers
to support legacy chains.
Tested by syncing mainnet from scratch.
lastHeight/blockSig/blockTimestamp, etc. moved from PeerData/repository
to Peer as it's transient so no need to store in repository.
Repository now keeps track of when/who added peer, e.g. API/INIT/another peer.
API calls DELETE /peers and DELETE /peers/known also disconnect peer
as well as deleting from repository.
Connection timestamp now reported by API call GET /peers
Some repository-updating code removed from Network/Controller as no
longer needed.
Removed obsolete Controller.hasShorterBlockchain predicate.
Previous version was prone to throwing exceptions during shutdown
sequence, especially in Peer/Network-related threads.
Shutdown now tries to wait for Peer/Network threads to cleanly exit.
API call POST /admin/forcesync now tries to grab blockchain lock,
with timeout, and then repeatedly sync with requested peer until
either fully synced or something unexpected happens.
BlockGenerator will now attempt to generate a new block if none of its
peers have a recent block either (in case of network stall). BlockGenerator
still needs a minimum number of peers before generating though.
Reduce BlockGenerator workload and use of blockchain lock if it
can't generate a block.
Reduce Synchronizer logging output.
Unify calculating timestamp threshold for 'recent' block into Controller.
Don't disconnect peers that fail to send a requested transaction,
as they may no longer have it. e.g. transaction might have expired
or become invalid.
For some other cases, e.g. we have transaction already, move on to
requesting the next transaction instead of giving up on the list.
NOTE: Downloaded update JARs are now expected to have been XORed with 0x5A!
This is to help prevent Windows Firewall from blocking update downloads
based on deep packet inspection.
Download read timeout reduced from 5s to 3s.
Download locations reordered so github entries are at the top as they have
better CDNs.
ApplyUpdate now assumes null response from GET /admin/stop means node
is not running.
ApplyUpdate now checks replacement JAR actually exists before attempting
to overwrite previous version.
ApplyUpdate now tries to use Windows EXE launcher in preference to raw
java command line. (This should improve Windows installer behaviour
in detecting running process and possibly firewall implications too).
Included unit test to cover change.
Modified test blockchain config "test-chain-v2.json" to add maxProxyRelationships.
Added comments to some proxy-related methods in AccountRepository class.
Added transaction [de]serialization test, along with corresponding random transaction generators.
Minor typo fix in Transaction.
Minor clarification in MessageTransactionTransformer.
Added debugging to Account.
Arbitrary transactions now [de]serialize data-type (raw/hash) for v4+ transactions.
Data type also stored in repository. Very small (<=255 byte) data payloads are also stored directly in HSQLDB.
Added ArbitraryDataManager which looks for hash-only data payloads and possibly requests raw data over network
depending on 'policy' (which currently is "fetch everything").
Added networking support for finding, and transferring, arbitrary data payloads.
Minor optimization to message ID generation in Peer.
Minor optimization in Serialization.serializeSizedString()
Bumped version
Controller no longer uses block height to determine whether to sync
but now uses peer's latest block's timestamp and signature.
Also BlockGenerator checks whether it's generating in isolation using
the same peer info (latest block timestamp and signature).
Added API call POST /admin/forcesync peer-address to help get wayward
nodes back on track.
Unified code around, and calling, Transaction.importAsUnconfirmed().
Tidied code around somelock.tryLock() to be more readable.
Controller (post-sync) now broadcasts new chaintip info if
our latest block's signature has changed, not simply the height.
Network.broadcast() only sends out via outbound peer if node has
more than one connection from the same peer. So Controller would
only update one of the peer records with chaintip info.
Controller now updates all connected peers with the ID when it
receives a HEIGHT or HEIGHT_V2 message.
Added node1 thru node7.mcfamily.io to default peers in Network.
Network ignores first "listen port" entry when receiving peers
list from an outbound-connection peer as it already knows
by virtue of having connected to it!
More network message debug logging (hopefully never to be seen).
[some old code left in, but commented out, for a while]
Controller now sets (volatile) requestSync flag when a peer sends new height info.
This allows much quicker response to new blocks which might hopefully improve synchronization
compared with the old periodic sync method.
"Unsolicited" network messages are now added to a BlockingQueue,
and a separate unsolicited message processing thread (one per peer)
deals with this messages in turn.
This allows "reply" network messages to propagate up to the
threads that are waiting for them, preventing deadlocks and
peer disconnections due to lost pings.
Controller tries to do as much new transaction processing
outside of the blockchain lock as possible, and only
broadcasts new transaction's signature if we successfully
import transaction to our unconfirmed pile.
Synchronizer.findSignaturesFromCommonBlock now returns null
to indicate some sort of connection issue (no cool-off)
and an empty list to indicate NO COMMON BLOCK.
That method also tries to work back to genesis block
instead of giving up too early if test block height
becomes negative.
Network.createConnection additionally filters out
candidates if their addresses resolve to the same
IP+port as an existing connection. So now it won't
connect to localhost:1234 if it has an existing
connection with 127.0.0.1:1234.
Network.broadcast only considers unique peers,
i.e. prefers outbound connection if a peer has
corresponding inbound connection.
Added Thread.currentThread().setName() where possible.
Notably: network messages passed up to Controller are now processed in their
own thread, as opposed to peer's thread.
So each message processor in Controller needs to thread-safe.
V2 network protocol asks for unconfirmed transactions, can send round lists
of transaction signatures and ask for individual transactions, to save
bandwidth/processing.
Remove fetching unconfirmed from Synchronizer
Add extra validity/reference/processable checks to
Transaction.isValidUnconfirmed
Update TransactionUtils to use Transaction.importAsUnconfirmed
for unit tests.
Transaction subclass isValid methods have last reference checks removed.
(Now handled by Transaction.hasValidReference)
Some checks in subclass' isValid moved to isProcessable, typically
for transaction types that require group-approval.
Removed fee extraction and last-reference update code from
subclass' process() method to Transaction.processReferencesAndFees.
Other changes to transaction/block processing.
Don't attempt to sync, or generate blocks, if we think we're not up to date.
Notify Controller of newly generated block AFTER releasing blockchain lock.
Loads of changes to synchronizer.
Added missing GET_PEERS handling to Controller.onNetworkMessage.
More detailed peer information (last block sig, last generator sig, last block timestamp, ...)
New HEIGHT_V2 network message to help support above.
More, and improved, logging.
Fix for HSQLDB serialization failure caused by trying to save the same new transaction
received by more than one peer/thread simultaneously.
Added mass (x1000) testing of key conversion and shared secret calculations.
Fix incorrect proxy private key test that has expected result from previous
algorithm.
Added another test HTML/JS file.
Use IPv6 wildcard address "::" for P2P bind address as null
seems to make listen socket bind to 127.0.0.1
Increase target minimum outbound connections from 3 to 10.
Increate maximum connection limit from 10 to 30.
Previously we preferred the outbound connection but actually
this leaves our info about inbound connection peers out of date.
Synchronizer still prefers outbound.
HTML/JS in src/test/resources/proxy-key-example.html updated accordingly.
Add handshake status to output of API call GET /peers
Add/correct @ApiErrors annotations on some API calls.
Add API call POST /admin/orphan (target height as body)
to force blockchain orphaning for when node is wildly out of sync.
Added support for above to BlockChain class.
BlockGenerator now requires a minimum number of peers
before it will generate any new blocks.
See "minBlockchainPeers" in settings.
Controller now requires a minimum number of peers
before it will consider synchronizing.
See "minBlockchainPeers" in settings.
Old "minPeers" entry in settings.json no longer valid!
Networking now allows both an outbound and inbound connection
to a peer although will use the outbound connection in preference.
Networking checks peer ID of inbound connections to detect,
and resolve, peer ID clashes/theft.
GET /addresses/proxykey/{privkey}/{pubkey} now POST /addresses/proxykey
GET /utils/fromBase58 now GET /utils/frombase58
GET /utils/fromBase64 now GET /utils/frombase64
GET /utils/toBase58/{hex} now GET /utils/tobase58/{hex}
GET /utils/toBase64/{hex} now GET /utils/tobase64/{hex}
GET /utils/privateKey/{entropy} now POST /utils/privatekey
GET /utils/publicKey/{privateKey} now POST /utils/publickey
Lots of edits to Transaction subclasses to change/remove 'delete'.
Corresponding extra changes to help reset some transaction fields to pre-process
state during orphaning.
Changed Block, GenesisBlock & Synchronizer to save transactions where appropriate.
Added enhanced GET_SIGNATURES_V2 network message to reduce the number of
block signatures sent over network.
Peers are now version 2 if they send a new-style build version string,
instead of using first digit from build version.
X25519 shared secrets now match those generated by libsodium.
New tests show that shared secrets are the same using either set
of private+public key combinations.
Changed proxy private key generation from using simple SHA256
of shared secret to using SHA256(shared secret + both public keys).
Added a temporary "BouncyCastle25519" shim class to provide missing
key conversion from Ed25519 to X25519.
Added a finer-grained result (enum) to synchronization to replace
stark boolean result. This allows Controller to decide whether
peer can be retried in the next round (e.g. network issue) or
cooled off for an hour (e.g. peer way too far behind)
Fixed bug with wrong UPDATE_GROUP serialized transaction length.
Added logging to BlockMessage when deserialization to help catch
future bugs.
Restrict synchronizer to only consider peers at most 60 blocks behind.
Synchronizer will only process a batch of 200 blocks at a time.
Improve debugging message in Peer.
Checking interval increased from 5 seconds to 5 minutes.
(Reduces repeated fails).
System tray icon should show message just prior to applying downloaded update.
Proxy forging recipient no longer needs a public key on the blockchain
at the point PROXY_FORGING transaction is submitted.
Proxy forging recipient is given a last-reference, if they don't have one,
when they receive their first block rewards.
Split block fees in proxy forging scenario, using same share proportion.
100% proxy sharing is now allowed.
Fixed account flags processing for accounts in genesis block.
AutoUpdate needs separate logfiles for its process as
log4j2's rolling appender doesn't support locking.
So we create AU_LOGGER in AutoUpdate and set it up
using static { } block.
Added support for starting/querying/stopping auto-update
Windows service.
Changed Controller's public connectionUrl field to
getRepositoryUrl() public static method so it
can incorporate repositoryPath from Settings.
Controller has added support for auto-update on startup.
Fix bugs with Settings not using userPath properly.
Removed obsolete standalone binary classes:
* blockgenerator
* txhex
* v1feeder
Removed all @Produces from API resources as response content type is
sorted by Swagger.
Added API /admin/info for generic node info.
Added API /arbitrary/ endpoints.
Moved arbitrary data storage from ArbitraryTransaction to ArbitraryRepository.
V4 arbitrary transaction signature is based on data's hash.
Original commit was d02f282, and commit message was:
Initial auto-update support, network MAGIC change, arbitrary tx improvements
Various fixes to synchronization
Added missing code for processing incoming block summaries in Network.
Fixed block summaries serialization and removed references to BlockData.
Fixed bug in transaction transformation where base transaction length
didn't include reference or fee lengths.
Original commit was ebbab7b
After opening repository connection with RepositoryManager.getRepostory(),
any 'read' from repository (e.g. SELECT) starts the transaction
even though HSQLDB documentation states there are no shared/read locks
in MVCC concurrency model.
The work-around for this is to 'reset' HSQLDB's in-transaction flag
by performing a ROLLBACK (Repository.discardChanges) immediately
after acquiring the blockchain lock (which is used to ringfence
changes that might collide like these).
Also adding an extra check to prevent payments to nonexistent AT
addresses as it touches Transaction.
API /addresses/{address} now returns lastReference taking unconfirmed into account.
Added DELETE /peers/known to remove all known peers from repository.
Added blockchain locking around Transaction methods like isValidUnconfirmed
as they (temporarily) update account lastReference.
Ditto getInvalidTransactions, etc.
Move hard-coded forging tiers to blockchain config.
Tests for granting forging rights.
Added API call to list top block forgers.
Fixed typo with Reward[s]ByHeight class name.
Remove old whispersystems, etc. *25519 and use new v1.61 BouncyCastle.
Fix proxy forging private key derivation from X25519 shared secret.
Also include Javascript test version for comparison.
Fix block rewards for proxy forging.
Add extra useful info to API call GET /admin/forgingaccounts.
Fix API response to POST/DELETE /admin/forgingaccounts when
passed invalid private keys.
Added block rewards and account flags to testchain config.
Tests to cover changes above.
Added /addresses/proxying to find proxy forging mappings.
Added /addresses/proxykey/{genprivkey}/{recipientpubkey} to calculate proxy private key.
New Block.regenerate factory method to create new Blocks
but without having to reprocess ATs, etc.
Added support for proxied generator in Block.calcGeneratorsTarget
BlockGenerator now generates and checks new blocks for various generators,
including proxy generators.
BlockGenerator now uses generator private keys supplied by Settings.
Corresponding changes to Settings to load base58-encoded private keys.
+ minor stuff
Also improved asset tests in that they don't use QORA any more
which takes fees/block rewards out of the picture.
More test assets are issued in the genesis block to
accomplish this:
1m TEST issued to Alice
1m OTHER issued to Bob
1m GOLD issued to Alice
e.g. supply "assetName" in JSON for TRANSFER_ASSET transactions
Also supply have/want asset names, amount asset ID/name, price-pair
and creator address in asset orders.
Show CREATE_ASSET_ORDER amount ID/name & price-pair in correct
format depending on whether transaction was placed before/after
'new' asset pricing took effect. (Orders are always in 'new' form).
Change API call /assets/transfers/{assetid}/{address} to
/assets/transfers/{assetid} with optional "address" query param.
Orders are back to having "amount" and "price".
(No more "unitPrice" or "wantAmount").
Order "amount" is expressed in terms of asset with highest
assetID.
"price" is expressed in (lowest-assetID)/(highest-assetID).
Given an order with two assets, e.g. QORA (0) and GOLD (31),
"amount" is in GOLD (31), "price" is in QORA/GOLD (0/31).
Order's "fulfilled" is in the same asset as "amount".
Yet more tests and debugging.
For simplicity's sake, the change to HSQLDB repository is
assumed to take place when 'new' pricing switch also
occurs.
Don't forget to change "newAssetPricingTimestamp" in
blockchain config JSON file.
Typical log entry would be:
2019-04-05 09:47:08 ERROR WadlGeneratorJAXBGrammarGenerator:401 - Failed to generate the schema for the JAX-B elements
com.sun.xml.internal.bind.v2.runtime.IllegalAnnotationsException: 4 counts of IllegalAnnotationExceptions
at com.sun.xml.internal.bind.v2.runtime.IllegalAnnotationsException$Builder.check(IllegalAnnotationsException.java:91) ~[?:1.8.0_191]
Changed API call GET /assets to NOT return asset "data" fields
as they can be huge. If need be, call GET /assets/info to fetch
a specific asset's data field.
Improve asset trade amount granularity, especially for indivisible
assets, under "new" pricing scheme only.
Added corresponding tests for granularity adjustments.
Fix/unify asset order logging text under "old" and "new"
pricing schemes.
Change asset order related API data models so that old "price" is
now "unitPrice" and add new "return" as in amount of want-asset
to receive if have-asset "amount" was fully matched.
(Affects OrderData, CreateAssetOrderTransactionData)
Some changes to the HSQLDB tables.
Don't forget to add "newAssetPricingTimestamp" to your blockchain config's
"featureTriggers" map.
Better order matching, especially in situations
where inexact fractional representations (e.g. 1/12)
or rounding issues might occur. Also better matching
with indivisible assets.
Essentially change ordering from have-amount & price
to have-amount and want-return, leaving unit price
to be calculated internally to a finer degree (in
some cases to 48 decimal points).
Corresponding unit tests to cover both legacy and new
scenarios. Support for tests to switch between
blockchain configs.
"New" pricing schema is its own 'feature trigger'
independent from general qorav2 switch.
Safety checks added during trading process.
HSQLDB schema changes (will probably need
careful conflict resolution on merge).
Still to do:
API changes
etc.
+ unit test
+ newer unit test harness
but still needs:
BlockChain config support for activating newer "price" arg
New unit test to check old "price" arg usage
Rework existing asset-related unit tests
Check API inputs/output pre/post "price" arg crossover
Non-native-coin payments (e.g. transfer-asset) didn't build map
for checking whether sender had enough balance.
Also fixed raw layout description for TRANSFER_ASSET.
Also fixed ISSUE_ASSET deserialization to use empty string (instead
of null) for v1 transactions.
GET /assets/trades/recent expanded to allow multiple otherAssetIds
When GET /assets/balances is called with address(es) but no assetIDs
then it will return balances for all assets, including zero balances
for assets the addresses don't own.
GET /addresses/{address} no long fakes a default groupID - in fact
defaultGroupId now removed from blockchain config.
Some Eclipse IDE files now hidden/removed from git repo.
Imported Block/BlockChain fixes from "minting" branch to do
with block timestamps.
GET /assets/holders/{assetid}
and
GET /assets/address/{address}
and
GET /assets/balance/{assetid}/{address}
all combined into
GET /assets/balances?address=...&address=...&assetid=...&assetid=...
New GET /assets/trades/recent?assetid=...&assetid=...
that returns most recent two trades for each asset-pair.
GET /assets/orders/{address}/{assetid}/{otherassetid} has
includeClosed and includeFulfilled repurposed as
isClosed (true/false/omitted) and isFulfilled (true/false/omitted).
ALSO, Order.isClosed is now set to true when isFulfilled is set to true
during processing (and correspondingly set to false during orphaning).
AccountBalanceData now includes optional assetName field for use with API
but generally not set for internal use.
Fixed bug in GenesisBlock which wasn't stripping
out ISSUE_ASSET transactions for v1 blockchains.
Updated blockchain.json for v1 qora blockchain to
fall into line with new BlockChain config
unmarshalling code.
Improved TransactionData.equals when comparing
exact same object.
Improved HSQLDBBlockRepository.getHeightFromTimestamp to use OffsetDateTime
object, which includes time zone info, to fix incorrect SQL full-scan
so that DB now does index-scan instead. Also converted use of MAX(column)
to ORDER BY column DESC as MAX() not optimized when MVCC in effect.
Added corresponding INDEX to Blocks on columns (generation, height);
Similar MAX(column) to ORDER BY column DESC improvement for getBlockchainHeight().
Bumped size of HSQLDB TYPE ATTags from VARCHAR(32) to VARCHAR(80)
Improved reflection-based method calls to rethrow underlying exceptions
like DataException, TransformationException, etc. instead of losing
them in general InvocationTargetException.
Re-added atTransaction.toBytes() method so v1 transactions can be verified.
Fixed extraneous additional copy of voter's public key when serializing
in VoteOnPollTransactionTransformer.toBytes()
Fix-up of v1feeder
UPDATE_NAME requires txGroupId to match original txGroupId used in REGISTER_NAME
UPDATE_GROUP requires txGroupId to match original txGroupId used in CREATE_GROUP
Extraneous JAXB unmarshal code removed from various transaction constructors.
Incorrect, old, groupID vs txGroupId checks removed from various transactions.
Fix bug in UPDATE_GROUP fromBase() method using mismatched column indexes.
Fix bug in API's pending transaction fetch not checking admin is of the same
group as the txGroupId. (Fix actually in HSQLDBTransactionRepository).
Similar fix in GroupApprovalTransaction.
GET /assets/orderbook/{assetid}/{otherassetid} renamed to
GET /assets/openorders/{assetid}/{otherassetid}
Replacement /assets/orderbook/{assetid}/{otherassetid} now
returns aggregated orders, with entries containing only
"price" and "unfulfilled" (amount).
Added /assets/orders/{assetid}/{otherassetid}/{address} to return
orders by specific account, for a specific asset-pair.
Block timestamp validity extracted to separate method so that
BlockGenerator can test timestamp and generate blocks at the usual
rate, even for testnets. This still allows testnets to a way to
generate blocks on demand as Block's isValid skips some timestamp
validity checks if testnet.
txGroupId was sometimes incorrectedly checked for approval-less tx types.
GenesisBlock (v4) now supports various transaction types (issue-asset, etc.)
with generated signatures (like genesis transaction signature) and
missing references inserted.
JUnit reverted back to v4 for Eclipse support (for now).
Some dev/testing API calls are now turned off by default in production mode,
see "restrictApi" settings entry, returning NON_PRODUCTION API error.
Corrections to how account's defaultGroupId works, removing "effective groupID"
which overly complicated matters.
In relation to above, DEFAULT_GROUP (0) no longer exists and NO_GROUP(-1) now has
the value 0 instead.
So transactions can no longer have txGroupId of DEFAULT_GROUP, which in turn
required all the erroneous "effective groupID" code.
API call /addresses/{address} now supplies blockchain-wide defaultGroupId if
account doesn't exist or if account's default not set and NO-GROUP not allowed.
API /transactions/pending now offloaded to repository instead of Java-based
processing and filtering.
Transaction approval checks added to Block.isValid
Groups now have min/max approval block delays.
Checks added to incoming unconfirmed, block generator, block.isValid, etc.
'needing approval' and 'meets approval threshold' now split into separate calls.
NB: settings.json no longer part of git repo
log4j2.properties now has debugging entries removed.
log4j2-test.properties (not in repo) takes priority
so using that in development instead.
Unconfirmed transactions no longer wiped on start-up
by default - see Settings
Reworking of {Public,Private,Genesis}Accounts as it seemed
possible to silently lose public key in repository.
The use of AccountData didn't work and so field-specific
repository calls have been made instead
(e.g. setLastReference) that try to opportunistically
store public key too, if available (i.e. caller is
PublicKeyAccount subclass, or better).
Added API call GET /addresses/{address} to return
general account info in one go. (Essentially the
AccountData object as fetched from repository).
Initial work on adding default groupID to accounts,
along with corresponding SET_GROUP transaction type.
In additional, added blockchain-wide default groupID
and flag to allow/disallow no-group/groupless
transactions.
Initial work on group-admin approval of transactions
tied to a specific group via txGroupId.
More work needed on transaction's "effective txGroupId"!
API call /transactions/pending to list transactions
pending group-admin approval. However, this needs more
work (see effective txGroupId above) and potentially
offloading to HSQLDB repository if possible.
Minor CIYAM AT renames to help static reflection initializers.
Block.orphan() no longer adds orphaned transactions back to
unconfirmed pile as they are themselves deleted during
Transaction.orphan(). Maybe the answer is to NOT delete
them during Transaction.orphan() but to add them to
unconfirmed pile at that point? Very old transactions
leftover from major resync would simply expire, whereas
recently transactions leftover from minor resync could
still make it into a new block on synced chain fork.
Changes/tidying/improvements to block generator regarding
removing invalid transactions and dealing with transactions
pending group approval.
Approval threshold added to groups.
Mass refactoring of transaction-related classes to unify
constructors, particularly field ordering, to fall in line
with raw transaction layout.
e.g. constructors now reflect that raw transactions mostly
start with type, timestamp, txGroupId, publicKey, reference
e.g. JAXB afterUnmarshal methods added where needed and corresponding
nasty code in Transaction subclass constructors ripped out.
e.g. TransactionTransformer subclasses contain less duplicated code.
Fixed bug with repository save points thanks to swapping to Deque.
Some fixes to do with missing transaction types being passed to JAXB
TransactionData subclass constructors.
Ripped out obsolete toJSON in TransactionTransformers as this
is all nicely taken care of by Swagger/OpenAPI (thanks @Kc)
Added short sleep() to GET /admin/stop to allow time for HTTP response body to be sent back.
Improved documentation for /peers API resources. Added examples, tidied API output models.
Fixed issue where IPv6 literals with port couldn't be parsed. Now uses RFC5952/RFC3986 style
literal IPv6 addresses with ports, e.g. [::1]:9084
Fixed NPE in Controller.potentiallySynchronize() where peer might not have sent height yet.
Improved Handshake to discard inbound connections if we already have an outbound connection
to a peer with that ID. This prevents us from having two connections to the same peer, one
in each direction.
Network.mergePeers() now runs in a separate thread as acquiring the lock might block.
Network.creationConnection() exits fast based on number of outbound connections, instead of
number of total connections.
Network no longer sends 'local' peer addresses to non-local peers.
e.g. it won't send localhost:9084 to node4.qora.org:9084
Added try-catch to Network.broadcast for when we try to broadcast while shutting down.
Added PeerAddress class to deal with the whole hostname/IPv4/IPv6 address situation.
Reworked PEERS_V2 message type to only send sized-strings instead of separate port,
and potentially IPv6 byte arrays.
Change to HSQLDB database shape.
Corresponding changes to HSQLDBNetworkRepository.
Peers now broadcast height after successful synchronization.
Added support for sending unconfirmed transactions to other peers.
This is done on connect and also after a new unconfirmed transaction is submitted via API.
Fixed synchronizer to handle blocks with transactions correctly.
Fixed network-related PoW to not use class-global SHA256 message digester!
(It was being corrupted by simulataneous access by different threads - whoops)
Surrounded Network.mergePeers with a lock to prevent HSQLDB deadlocks.
Also changed HSQLDB concurrency model to MVCC (only takes effect if DB rebuilt).
Added support for logging other HSQLDB sessions in the event of exception.
(Currently only used by HSQLDBSaver)
Transaction transformer modifications to help deserialize TransactionMessages.
Some pom.xml changes to reduce maven-shade-plugin conflicting classes warnings.
Repository now supports SAVEPOINT and ROLLBACK TO SAVEPOINT.
HSQLDB concurrency/transaction model changed from LOCKS to MVCC to help with
transaction deadlocks/rollbacks.
More XXXs and TODOs added to Block.java for investigation/fix/improvements.
Also used new repository SAVEPOINT feature when validating transactions
instead of rolling back entire transaction. This fixes problem during
synchronization where the rollback would undo previously synchronized,
but not yet committed, blocks!
Transactions orphaned by Block.orphan ARE now added to unconfirmed pile,
unlike before.
Concurrent lock now prevents simultaneous block generation and synchronization,
including synchronization via multiple peers.
Lots of new networking code: peer lists, block signatures, blocks,
blockchain synchronization.
PEERS_V2 message now supports hostnames, IPv6 and port numbers.
Fixed bug with block serialization for transport over network.
DB shape change from v29 to add peer info.
New NetworkRepository to handle above.
Peer handshaking with v2 anti-DoS PoW code.
Handshaking refactored into a state-machine-like enum.
Some peer-related API calls added.
Peers exchange pings, heights, peers.
No actual peer sync yet.
Other changes:
Peer version info taken from Maven build properties/resource file.
AnnotationPostProcessor more resilient when fetching PathItems.
Per-repository session debugging flag that can be toggled at will.
HSQLDBRepository.delete() now returns int so callers can detect
whether anything was actually deleted.
Some renaming to settings.
Moved as much reflection out to class-static initializers as possible.
Renamed some classes to fall in line with transaction type name
to class name conversion, e.g. DEPLOY_AT -> DeployAt
API calls that return lists now take limit, offset and reverse params.
API calls that used to return data & optional list (e.g. blockWithTransactions)
now only return base data. The optional lists can be fetched via
a different API call.
Also: SLF4J now routes logging to log4j2 so start up output cleaned up.
Suppressed extraneous Jersey warning about Providers during start-up injection.
Account group join for a group that is closed/invite-only
and has no corresponding invite is now turned into a "join request".
This can be accepted by an admin sending a corresponding invite tx.
(Also minor fix for orphan.java).
Note use of afterUnmarshal() in TransactionData-subclass to replace trash code in Transaction-subclass constructor.
See UpdateGroupTransactionData.afterUnmarshal() compared to RegisterNameTransaction constructor.
Added POST /names/update for building an UPDATE-NAME transaction.
BlockGenerator now tries to validate new block after adding each
unconfirmed transaction in turn. If block becomes invalid then
that transaction is removed/skipped. This should further prevent
block jams. Skipped transactions might be deleted as the next block
is forged when unconfirmed transactions are collated/filtered/expired.
Add Block.deleteTransaction() for use during block generation above.
Block.addTransaction() and Block.deleteTransaction() use transaction
signatures to test for presence in Block's existing transactions.
Names shouldn't have stored registrant's public key!
"registrantPublicKey" removed from NameData Java object/bean.
Corresponding column removed from HSQLDB using ALTER TABLE but
also from the original CREATE TABLE definition. Remove the ALTER
TABLE statement just prior to rebuilding database!
(This needs to be applied to Polls too as some point).
Also, UpdateNameTransactions and BuyNameTransactions tables now
allow name_reference to be NULL as this column value isn't set
until the corresponding transactions are processed/added to a
block. (name_reference is a link to previous name-related
transaction that altered Name data like "owner" or "data" so
that name-related transactions can be orphaned/undone).
Added GET /names to list all registered name.
Added GET /names/{name} for more info on a specific name.
Added GET /names/address/{address} for names owned by address.
Renamed GET /assets/all to GET /assets in line with above.
Fixed edge cases with AnnotationPostProcessor.
Fixed incorrectly exposed "blockHeight" in API UI examples/values.
Changed example transaction timestamp.
Added checks on building/signing/processing new transactions via API
so that they are not too old (older than latest block's timestamp),
too new (more than 24 hours in the future) or the tx creator doesn't
already have a lot of existing unconfirmed transactions (default 100).
Configurable via settings.json properties maxUnconfirmedPerAccount
and maxTransactionTimestampFuture.
Improved /transactions/search to not return unconfirmed transactions
and to order by timestamp.
Transaction.getCreator() now returns PublicKeyAccount, not Account.
Now uses working RIPE-MD160 by default but can be switched to broken MD160 using flag in blockchain config,
e.g. for Qora v1 blockchain.
Replaced API signature/reference examples with descriptive text as they weren't very useful.
Replaced API address examples with ones generated using working MD160.
Added GET /transactions/signature/{signature}/raw that returns raw transaction in base58 encoding.
Added "ignoreValidityChecks" query param to POST /transactions/decode to bypass INVALID_REFERENCE errors
if supplying an old/speculative transaction that can't be added to unconfirmed transaction pile.
Finally fixed creating inital assets in BlockChain.
Controller now inserts BouncyCastle as highest priority Security Provider.
TransactionData & transaction repository now tries to return transaction's block height in data when possible.
Also added test for missing creator public key on API-submitted
transaction creation calls, like /payments/pay or /asset/issue.
(Needs to be an OpenAPI validator added at some point).
Invalid/expired unconfirmed transactions are cleaned during various calls,
e.g. requesting list of unconfirmed transactions,
or requesting account's last reference (including considering unconfirmed),
or generating a new block.
BlockGenerator now calls repository.discardChanges before sleep to
release any repository-level locks.
Added settings.json toggle "wipeUnconfirmedOnStart" (default: true)
to aid testing.
REMOVED API call /addresses/lastreference/{address}/unconfirmed as
/addresses/lastreference/{address} now considers unconfirmed
transactions regardless.
Added useful error to /transactions/sign if an invalid private key
is supplied.
Improved API "invalid transaction" error to include actual apsect
that caused validity check to fail (e.g. invalid reference)
Refactored to standard Maven layout:
src/main/java
src/main/resources
src/test/java
etc.
New translation code that uses locale-specific ResourceBundles
to load translations on demand.
Reworked API error/exceptions code to a shorter, simpler
@ApiErrors annotation. Processing of @ApiErrors annotations
produces an example for each possible API error and includes
API error string in HTTP response code, e.g.
400 INVALID_SIGNATURE
Missing API error cases added to each API call.
Translation of openAPI.json removed (for now).
block-explorer.html and BIP39 wordlists now read as resources
instead of direct from disk.
Java compile warnings fixed.
Some runtime warnings remain:
WARNING: A provider api.resource.ApiDefinition registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime.
WARNING: A provider api.resource.AnnotationPostProcessor registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime.
WARN org.reflections.Reflections - given scan urls are empty. set urls in the configuration
Added slow query check to HSQLDB repository to help
isolate cases where transaction searching takes too long.
Added BigDecimalTypeAdapter for normalizing API inputs
but doesn't seem to get reliably called so also added
.setScale(8) to BigDecimal serialization method.
API-built transactions are now validated before emitting
base58 raw transaction to help callers.
API's transaction decoder accepts signed/unsigned raw transactions.
XmlJavaTypeAdapter api.Base58TypeAdapter converts byte[] to Base58.
This XmlAdapter is applied at package-level to all packages inside data and api.models.
So no need to annotate every byte[] property!
Added package-info-maven-plugin to pom.xml to do this.
block-explorer.html fixed to show/use base58 again
Some data objects (e.g. TransactionData) have added XmlElements that
convert public keys to addresses, for convenience.
Several API calls updated to return specifically text/plain instead
of ambiguous application/json and/or text/plain. (Typically
API calls that return a single value, e.g. an integer, like /blocks/height).
Cleaned up responses from /addresses/* endpoints
in that some return text/plain instead of application/json.
Removed need for class-local copy of ApiErrorFactory in
AddressesResource - using getInstance() instead.
Some work still needs to be done on annotating API errors.
API error examples in API UI rendered incorrectly - swagger-ui issue?
Removed repository-accessing code from api.models.*
Added /assets/order/{orderId} for fetching info on specific asset order.
NOTE: AssetRepository.getOrdersTrades() now returns trades where order is
initiating or target. (Previously was initiating order only).
qora.assets.Order.orphan() updated to reflect above change.
block-explorer.html fixed to use new API output.
Added more global parameters to /admin/unused API endpoint (formally /admin/dud)
and also managed to remove /admin/unused from API documentation UI.
Added results slicing to /assets/all
Added /assets/orderbook API call that returns open asset orders
Added /assets/trades that returns successful asset trades
Added POST /assets/issue stub
Unified HSQLDB connectionUrl to public variable inside Controller class.
Can't deploy v1 ATs with isFinished=true flag as that prevents later
transactions sending messages (during import of v1 chain).
Some future hard-fork code will need to set all v1 ATs to "finished".
Changed DB's "TransactionRecipients" to "TransactionParticipants" to
properly support API call to find all transactions 'involving' a
specific address. Support code needed in Block and Transaction with
some transaction-specific overrides for Genesis and AT transactions.
Removed old, deprecated calls from Transaction/TransactionRepository
Moved HSQLDB database properties from connection URL to explicit
SQL statements in HSQLDBDatabaseUpdates. They didn't work in
connection URL during DB creation anyway.
Retrofitted HSQLDB Accounts table with public_key column instead of
rebuilding it later.
Fixed incorrect comments in IssueAssetTransactionTransformer regarding
v1 serialization for signing.
Re-imported v1 chain to test latest changes.
Converted AddressesResource to full base64, removing base58.
Narrowed range of API errors returnable while there.
Added support for looking up public key of address.
Added support for converting public key TO address.
Added API endpoint for returning a range of block signatures,
to aid block explorers.
Added API support for fetching unconfirmed transactions.
Added API endpoint for searching transactions to meet criteria like:
- participating address (only recipients supported ATM)
- block height range
- transaction type
- result count limit/offset
---
Added storage of account's public key in repository
along with supporting code in AccountData and Account
business object to save public key where possible.
Rejigged pom.xml, extracting common dependency versions as properties.
Removed extraneous HSQLDB dependency (v2.4.1) as we're using svn r5836 for now.
Removed calls to Security.checkApiCallAllowed() for all API calls EXCEPT /admin/stop.
Throws error if remote IP is not localhost.
Added 'global' OpenAPI parameters to fake /admin/dud endpoint to save copy&pasting.
This will need more tidying in the future, or at least future support from swagger-core.
Code added in AnnotationPostProcessor to insert global parameters in top-level
OpenAPI components section.
/block-explorer.html hidden from API UI
BlocksResource now expects Base64 block signatures instead of Base58.
Endpoints that return block data also accept optional "includeTransactions"
query param which does exactly that.
BlockWithTransactions API model added for above.
Some attempt to get transaction-specific data returned but no luck as yet.
(TransactionData, GenesisTransactionData, PaymentTransactionData touched).
See https://github.com/swagger-api/swagger-core/issues/3046
TransactionsResource now has support for optional query params "limit" and "offset"
so that only a subset of large results can be requested.
UtilsResource added to provide convenient Base64<->Base58 conversions.
/admin/uptime fixed to return uptime from application launch instead of
instantiation of AdminResource class!
Controller improved to detect repository and API startup failures.
HSQLDBRepositoryFactory now detects when it can't open database and throws.
(Before it would simply hang).
Removed extraneous import from qora.account.Account
Switched from maven-assembly-plugin to maven-shade-plugin for
building FATJAR.
When running from FATJAR, class-path is ". .." to help find
log4j2.properties file.
Swagger-UI can now be served direct from inside FATJAR instead
of requiring resources in filesystem.
Default package Start now controller/Controller
block-explorer.html now served via Jetty and modified to use
relative URLs instead of absolute http://localhost:9085/... style
Improved shutdown code in controller
/admin/stop API call disabled for now
Highly permissive settings.json added
Added FATJAR packaging support to pom.xml
Added some "summary" fields to API calls but more need doing.
Corrected path clash from having unnecessary @OpenAPIDefinition annotations.
Added API "tags" to group similar calls (address-based, block-related, etc.)
Fixed addresses/lastreference/{address}
Implemented addresses/lastreference/{address}/unconfirmed
Implemented addresses/assets/{address}
Added /admin/stop and /admin/uptime API calls.
Moved general API info into new src/api/ApiDefinition.java
Added CORS support to ApiService
Added /transactions/address/{address} and /transactions/block/{signature}
Replaced references to test.Common.* to do with repository factory.
This fixes issues with building FATJAR due to references to test classes
that are omitted from FATJAR.
Changes to AccountBalanceData, BlockData and TransactionData
to support JAX-RS rendering to JSON.
Added getUnconfirmedLastReference() to Account.
Added getAllBalances(address) to account repository
- returns all asset balances for that address.
Added getAllSignaturesInvolvingAddress(address) to account repository
but currently only uses TransactionRecipients HSQLDB table.
(And even that wasn't automatically populated).
Included: very basic block explorer to be opened in browser as a file:
block-explorer.html
Some initial BTC cross-chain support. (Needs more work).
Unified timestamp for V2 switchover to block version 4,
applicable to several transaction types.
Qora-specific interface to CIYAM ATv2 library.
Beware: some areas still work-in-progress!
Now using ATv2 dated 20181101172102
ATData now uses byte[] creatorPublicKey instead of String creator.
TransactionData now has hashCode() and equals() methods,
which is needed for new Transaction Comparator,
used to sort transactions within a block,
AT-first, then timestamp, then signature.
AT-Transactions generate their own signatures using SHA2-256 of serialized data.
Arbitrary Transactions try to clean up their files when orphaned.
Deploy AT Transactions now check creation bytes (even for old v1 ATs).
Deprecated Transaction.getBlock() as it doesn't seem used
and would be better to simply have getHeight() rather than
a method that 'knows too much' about Blocks/BlockData.
Corresponding TransactionRepository.getBlockDataFromSignature()
also deprecated.
Loads more comments.
Tidied up some SQL: mainly correcting case,
moving PRIMARY KEY clauses to end of CREATE TABLE,
removing unnecessary columns from indexes.
Added "type" column to TransactionCreatorIndex so users can find
their transactions and optionally filter by type.
In BlockTransactions table, transaction_signature is now UNIQUE
as a transaction cannot be included in more than one block.
Various AT-related HSQLDB table and index changes.
ArbitraryTransactions transformer fixed to always return a list of payments,
even if empty. (Previously could return null which broke things).
Added simplistic block generator.
NOTE: unit tests broken due to pending upgrade to JUnit 5
ATData no longer needs deploySignature as a link back to DeployATTransaction,
but does need creator and creation [timestamp].
"creation" is critical for ordering ATs when creating/validating blocks.
Similar changes to ATStateData, adding creation, stateHash (for quicker
comparison with blocks received over the network), and fees incurred
by running AT on that block.
Also added more explicit constructors for different scenarios.
BlockData upgraded from simplistic "atBytes" to use ATStateData (above)
which has details on ATs run for that block, fees incurred, and a hash
of the AT's state. atCount added to keep track of how many ATs ran.
ATTransactions essentially reuse the GenesisAccount's publickey as
creator/sender as they're brought into existence by the Qora code
rather than an end user. ATTransactionData updated to reflect this
and the AT's address used as a "sender" field.
Account tidied up with respect to CIYAM ATs and setConfirmedBalance
ensures there is a corresponding record in Accounts (DB table).
Account, and subclasses, don't need "throws DataException" on
constructor any more.
Fixed bug in Asset Order matching where the matching engine
would give up after first potential match instead of trying others.
Lots more work on CIYAM AT, albeit mainly blind importing of old
v1 ATs from static JSON file as they're all dead and new
v2 implementation is not backwards compatible.
More work on Blocks, mostly AT stuff, but also fork-based corruption
prevention using fix from Qora v1.
Payment-related transactions (multipayment, etc.) always expect/use
non-null (albeit maybe empty) list of PaymentData when validating,
processing or orphaning.
Mainly a change in HSQLDBTransactionRepository.getPayments()
Payment.isValid(byte[], PaymentData, BigDecimal, boolean isZeroAmountValid)
didn't pass on isZeroAmountValid to called method - whoops!
Lots of work on ATTransactions themselves.
MessageTransactions incorrectly assumed the optional payment was always
in Qora. Now fixed to use the transaction's provided assetId.
Mass of fixes/additions to HSQLDBATRepository, especially fixing
incorrect reference to Assets DB table!
In HSQLDBDatabaseUpdates, bump QoraAmount type from DECIMAL(19,8)
to DECIMAL(27,8) to allow for huge asset quantities.
You WILL have to rebuild your database!
