This table was previously defined using the TEMPORARY keyword as the rows were used as a cached/index to speed up AT trimming SQL statements.
However, the table definition was lacking the "ON COMMIT PRESERVE ROWS" clause, and possibly the "GLOBAL" keyword, which caused the table contents to be emptied, immediately after being filled, when <tt>repository.saveChanges()</tt> was called (i.e. "COMMIT").
This caused latest AT states to be trimmed in error.
AtRepositoryTests.testGetLatestATStatePostTrimming also fixed so that it fails with the previous, broken code.
This table was previously defined using the TEMPORARY keyword as the rows were used as a cached/index to speed up AT trimming SQL statements.
However, the table definition was lacking the "ON COMMIT PRESERVE ROWS" clause, and possibly the "GLOBAL" keyword, which caused the table contents to be emptied, immediately after being filled, when <tt>repository.saveChanges()</tt> was called (i.e. "COMMIT").
This caused latest AT states to be trimmed in error.
AtRepositoryTests.testGetLatestATStatePostTrimming also fixed so that it fails with the previous, broken code.
Add caching of transactions fetched via ElectrumX to reduce network load and speed up API response.
Fix handling ElectrumX servers that don't want to supply verbose transaction JSON.
Hide lots of data in BitcoinyTransaction that isn't needed by current API users.
Previous fixes for "transaction rollback: serialization failure" when updating trim heights
in commits 16397852 and 58ed7205 had the right idea but were broken due to being synchronized
on different objects.
this.repository.trimHeightsLock would be a new Object() for each repository connection/session
and so not actually synchronize concurrent updates.
Implicit saveChanges()/COMMIT is still needed.
Fix is to use a repository-wide object for synchronization - in this case the repositoryFactory
object as held by RepositoryManager.
Added test to cover.
Also reduced DB trim height read to one call at start of thread for both trimming threads.
This involves a database reshape, but before this happens the node-local
data is exported to local files, giving the user the option to use a
bootstrap file instead of waiting.
Additional benefit is to speed up syncing if node has trade-bot entries,
as a batch of blocks processed within 30s will have the same HTLC responses
as neither Bitcoin nor Litecoin blocks are that fast. Once synced, the next
Qortal block (~60s) should pick up any new changes. Generally users will be
synced when using trade-bot anyway.
Also moved bitcoiny.getAddressTransactions(p2shAddress)
into BitcoinyHTLC.findHtlcSecret() so only called if secret,
or lack thereof, is not cached.
Added tests to cover caching.
Several cross-chain API calls moved into separate classes,
although most of the URLs remain roughly the same to provide
backwards compatibility.
API /crosschain/at/build moved into /crosschain/BitcoinACCTv1
Converted DELETE /crosschain/tradeoffer to be ACCT-agnostic.
Changes to ACCT interface, etc. to support above.
Changes applied to other crosschain API calls to make them
independent of Bitcoin/Litecoin.
Corrections to fee calculations and usage in BitcoinACCTv1.
Added new LitecoinACCTv1 trade-bot, using LitecoinACCTv1.
Some minor typo corrections, rename of secretHash to hashOfSecret.
Some more Bitcoin-specific fields deprecated, but values duplicated
from newly-named fields for now.
Lower default fee (10sats/byte) for Litecoin spending transactions.
(Not P2SH fees which are 1000sats).
Changed ApiError INSUFFICIENT_BALANCE HTTP status from 422 to 402
as 422 isn't supported by Jetty?
CrossChainTradeSummary.btcAmount deprecated, use: foreignAmount
Modified pom.xml to generated package-info.java files for classes
inside org.qortal.api.model.** subdirectories.
API support for Litecoin wallet balance and sending LTC.
TradeBotCreateRequest rejigged to use blockchain-agnostic
field names, e.g. bitcoinAmount now foreignAmount,
and added foreignBlockchain field.
The massive API CrossChainResource class has been split into:
CrossChainAtResource: for building TRADE/REDEEM/CANCEL messages
(OFFER missing?)
CrossChainBitcoinResource: for Bitcoin wallet balance/spend
CrossChainLitecoinResource: ditto for Litecoin
CrossChainHtlcResource: for Bitcoiny-HTLC actions like:
deriving P2SH address
checking HTLC status
eventually: building refund/redeem transactions
CrossChainResource: for creating/cancelling/listing trade offers.
CrossChainTradeBotResource: for creating/cancelling trade-bot
entries, including responding to trade offers.
---
Other general trading changes:
TradeBot states are now specific to each individual trade-bot,
e.g. BitcoinACCTv1TradeBot or LitecoinACCTv1TradeBot, etc.
TradeBot states now a combination of int & String, instead of
enums due to above.
Extra columns added to DB TradeBotStates to store
blockchain, which ACCT in use, etc.
---
UNTESTED at this point!
Extracted AcctMode from BitcoinACCTv1.Mode as the values are
common to both Bitcoin/Litecoin ACCTs.
Added test apps for deploy, cancel, trade and redeem of LitecoinACCTv1.
Added altcoinj library as Maven dependency.
Added new Litecoin subclass of Bitcoiny,
with mainnet and testnet ElectrumX server lists.
Added litecoinNet settings variable and getter.
Added LitecoinTests.
Most tests work but testFindHtclSecret()
needs a redeemed HTLC on chain (not yet done).
Added litecoinNet to some test settings files
in resources.
Added Litecoin BuildHTLC, Refund test apps.
Added SendLTC app as Electrum-LTC seems a bit flaky?
So far managed to build HTLC P2SH, fund it and then
refund it!
---
As Bitcoin and Litecoin are both subclasses of Bitcoiny,
could unify some test apps with added Bitcoin/Litecoin
switch as first arg?
Bitcoin/Litecoin common aspects extracted in a "Bitcoiny" common class.
So:
Bitcoin (was BTC) extends Bitcoiny
Litecoin (future code) will also extend Bitcoiny
ElectrumX is now a BitcoinyBlockchainProvider
to allow easier future replacement and also tidier integration.
BTCP2SH is now BitcoinyHTLC as they are generic hash time-locked contracts,
probably Bitcoin/Litecoin agnostic.
BTCACCT is now BitcoinACCTv1, allowing for v2+ and also LitecoinACCTv1, etc.
BitcoinTransaction is now BitcoinyTransaction
as they are pretty much the same in Litecoin.
BitcoinException is now a more generic ForeignBlockchainException.
---
Bitcoiny subclasses instantiate a new BitcoinyBlockchainProvider
when creating their singleton instance. They pass relevant network
details to the BBP, like server lists, genesis block hash, etc.
Bitcoiny.WalletAwareUTXOProvider now only has the one key search mode
that is equivalent to the old REQUEST_MORE_IF_ANY_SPENT.
Tests tidied up.
---
Still to do:
Modifying TradeBot to handle multiple types of ACCTs,
like BitcoinACCTv2, LitecoinACCTv1...
Modifying API to support multiple types of ACCTs.
Actually add Litecoin support.
Build new ACCT without needing P2SH-B if possible.
Added separate method to determine status of P2SH transactions,
returning UNFUNDED, FUNDING_IN_PROGRESS, REDEEMED, etc.
Added code to trade-bot to increase robustness. Lots more
changes including unified state change/logging, checking
for existing MESSAGEs, etc.
Added missing websocket methods to silence log noise.
Trade-bot now called per block during synchronization,
instead of per batch, to pick up edge cases where some
potential trade-bot transitions were missed, resulting
in failed trades.
Corresponding changes in Controller, such as notifying
event bus of new block in same thread (thus blocking)
instead of using executor.
Added slightly more robust common block determination
to Synchronizer.
Refactored code in BTC class to use new BitcoinException
rather than simply returning null, with added sub-classes
allowing differentiation between network issues or fund
issues.
Changed BTC.buildSpend to try harder to find UXTOs to
address false "insufficient funds" issues.
Repository change to add index on MessageTransactions
for quicker look-up of trade-related messages.
Reduced reliance on bitcoinj library in BTCP2SH.
Reworked ElectrumX to better detect errors rather than
continuously try more servers to no avail.
Also added genesis block check in case of servers on
different Bitcoin networks.
Now tries to extract upstream bitcoind error codes
and pass those up to caller via exceptions.
Updated list of testnet servers.
MemoryPoW now detects thread interrupt and exits fast.
Moved some non-generic transaction-related repository
methods to their own subclass. For example:
moved TransactionRepository.getMessagesByRecipient
to MessageRepository.getMessagesByParticipants
Updated and added more tests.
Added Qortal-side HSQLDB PreparedStatement cache, hashed
by SQL query string, to reduce re-preparing statements.
(HSQLDB actually does the work in avoiding re-preparing
by comparing its own query-to-statement cache map, but we
need to keep an 'open' statement on our side for this to
happen).
Support added for batched INSERT/UPDATE SQL statements to
update many rows in one call.
Several specific repository calls, e.g. modifyMintedBlockCount
or modifyAssetBalance, now have batch versions that allow
many rows to be updated in one call.
In Block, when distributing block rewards, although we still
build a map of balance changes to apply after all calculations,
this map is now handed off wholesale to the repository to
apply in one (or two) queries, instead of a repository call
per account. The balanceChanges map is now keyed by account
address, as opposed to actual Account.
Also in Block, we try to cache the fetched online reward-shares
(typically in Block.isValid et al) to avoid re-fetching them
later when calculating block rewards.
In addition, actually fetching online reward-shares is no longer
done index-by-index, but the whole array of indexes is passed
wholesale to the repository which then returns the corresponding
reward-shares as a list.
In Block.increaseAccountLevels, blocks minted counts are also
updated in one single repository call, rather than one
repository call per account.
When distributing Block rewards to legacy QORA holders,
all necessary info is fetched from the repository in one hit
instead of two-phases of: 1. fetching eligible QORA holders,
and 2. fetching extra data for that QORA holder as needed.
In addition, updated QORT_FROM_QORA asset balances are done
via one batch repository call, rather than per update.
Symptoms include this in logs:
Unexpected zero effective minter level for reward-share %s - using 1 instead!
This occurs when Synchronizer compares two sub-chains from a common block,
and one of the blocks is signed by a reward-share key that has
subsequently been cancelled.
Although this is catered for, excessive log-spam is emited.
So in addition to demoting the log level from WARN to DEBUG,
more code has been added to try harder to find the actual data needed,
thus preventing the logging in the first place.
New repository transaction search method added to support above,
along with corresponding tests.
ApplyUpdate is the 2nd-stage of the auto-update system, called
after core has downloaded the update.
As old versions of the Windows launcher EXE selects a 'client'
JVM mode, heap memory could be limited to only 256MB.
Until users upgrade via Windows installer, which replaces the EXE
with 'server' JVM mode baked-in, then a work-around is to
pass -XX:MaxRAMFraction=4 to the new JVM in order to emulate
heap size in 'server' JVM mode.
Added "minimumTimestamp" param to same API call to allow fetching results for scenarios like:
* completed trades since midnight
* completed trades within last 24 hours
Added corresponding tests for above API call, including checking call response times.
Also improved BTC.WalletAwareUTXOProvider to derive more keys itself
instead of throwing and relying on caller to do the work.
Added benefit of cleaning up caller code and being more efficient.
Needed because not all receiving/change addresses were being picked up.
Bitcoin receive address no longer stored in AT but dealt with by trade-bot.
This allows 'Bob' to have his BTC sent anywhere he likes when redeeming P2SH-A
thus saving a step, typically incurred by UI. DB shape change due to this.
Similarly, AT code has been updated to expect a Qortal receiving address when
Alice sends MESSAGE to redeem AT.
This means both trade-bot entries (Alice/Bob) can be safely wiped once trade completes.
Some terms were confusing like "trade recipient" which actually referred to
Alice and so have been unified as "trade partner" as to not be confused with
(say) "recipient address"
The MESSAGEs sent from Alice to Bob, from Bob to AT and from Alice to AT have been
given more useful names: 'offer', 'trade' and 'redeem'. There is also a cancel
MESSAGE sent from Bob to AT to cancel AT before trading occurs.
Some API calls have been renamed in light of above.
AT's 'mode' has been expanded from simply OFFER/TRADE to:
OFFERING, TRADING, REFUNDED, REDEEMED, CANCELLED
Tests updated, but MORE TESTING REQUIRED BEFORE RELEASE
Previous version fetched all the blocks from previous 'timestamp'
to current height, checking each transaction. (very slow)
New implementation leverages repository to do the heavy lifting.
Could potentially benefit from some DB indexes in the future?
Added unit test to cover.
