Commit Graph

597 Commits

Author SHA1 Message Date
catbref
a6fa4fc613 WIP: trade-bot MESSAGE support 2020-07-17 08:39:45 +01:00
catbref
593b61ea4b Reduce bitcoinj exposure to classes outside of org.qortal.crosschain package.
BTC.getBalance() now returns Long instead of Coin.

BTC.FORMAT.format(Coin) changed to BTC.format(Coin or long).

Added BTC.deriveP2shAddress(byte[] redeemScriptBytes).
2020-07-17 08:39:45 +01:00
catbref
04d691991a WIP: more work on trade-bot 2020-07-17 08:39:45 +01:00
catbref
faa6e82bef WIP on trade-bot 2020-07-17 08:39:45 +01:00
catbref
65ccb80aa4 AT-related changes: new Qortal functions, tests, etc.
Added GET_MESSAGE_LENGTH_FROM_TX_IN_A
and PUT_PARTIAL_MESSAGE_FROM_TX_IN_A_INTO_B.

Replaced AT-1.3.4 with version including bug-fix for off-by-one
data address bounds checking.

Moved long-from-bytes method to BitTwiddling class.

Renamed some methods to make it more obvious they work with
little/big endian data.
2020-07-17 08:39:45 +01:00
catbref
cc13d1d0f1 WIP commit 2020-07-17 08:39:45 +01:00
catbref
ead84d70d1 Initial skeleton code for Trade Bot 2020-07-17 08:39:45 +01:00
catbref
275146fb55 Return REWARD_SHARE_UNKNOWN when trying to cancel non-existent reward-share 2020-07-14 09:53:14 +01:00
catbref
e74a249388 Collate network PoW computes into a fixed-sized pool, with dead peer detection.
Also added Named/DaemonThreadFactory classes.

Network EPC now uses NamedThreadFactory for easier debugging.

Added settings field "networkPoWComputePoolSize", default 2, which
seems to work with both low-power ARM boards and high-power desktops.
2020-07-03 09:31:46 +01:00
catbref
d8c5e557d8 Update uiLocalServers, autoUpdateRepos and bump to v1.2.1 2020-06-30 15:41:53 +01:00
catbref
984e8b5227 Network optimizations: if we're not up to date then don't request, or send, unconfirmed transaction lists 2020-06-30 14:26:12 +01:00
catbref
469bf2a63e Improve inbound peer handshaking
If a node accepts a connection from an inbound peer
then remote peer will send RESPONSE first
and local node would previously change handshaking state
to COMPLETED while computing their own RESPONSE.

This meant that the local node would sometimes also start
sending post-handshake messages to the remote peer,
e.g. TRANSACTION_SIGNATURES.

Remote peer is only expecting a RESPONSE message, so would
close connection.

So we introduce an extra handshaking state "RESPONDING" for use
by local node while they compute RESPONSE in a separate thread.
Once the RESPONSE has been sent, local node moves to COMPLETED
state and called onHandshakeCompleted() as per usual.

Note that the code path when connecting outbound to a remote peer
is not changed, and the RESPONDING state is not used.

Also in this commit:

Network.onPeerReady now bypasses call to onMessage and instead
calls onHandshakingMessage() directly to avoid race condition
where peer's handshake status could change between
onPeerReady's caller and onMessage() calling peer.getHandshakeStatus()
2020-06-30 13:37:14 +01:00
catbref
e05fcd6655 Final genesis block & bump to v1.2.0 2020-06-29 15:45:10 +01:00
catbref
3a7751910e Fix /websockets/chat/messages so it disregards group-membership change notifications 2020-06-29 08:38:19 +01:00
catbref
3c139f3e53 Minor fix-up to allow go-live:
Re-add (for now) Ed25519 HSQLDB conversion.
Catch DataException in BlockChain.isGenesisBlockValid and return false.
Remove duplicate bug-fix for LeaveGroupTransactions DB table.

Interim, near-final blockchain.json
2020-06-26 15:06:04 +01:00
catbref
2c14a12464 Check for unknown Qortal-only AT function codes & add safety to AT running in general 2020-06-25 14:22:28 +01:00
catbref
faa6405d5f Reference fixes for MESSAGE transactions & tests to cover 2020-06-24 17:15:17 +01:00
catbref
e2e4555009 Added /websockets/admin/status and improved GET version.
NodeStatus contructor now fills in fields, which themselves are now 'final'.
NodeStatus also includes numberOfConnections and height as per systray.

AdminResource.status() unified with websocket version.
2020-06-24 11:54:06 +01:00
catbref
448e984995 Fix some minor group-related bugs
Incorrect column names when saving a group ban.

Missing column in LeaveGroupTransactions.

More stringent validity checks in group-kick, group-ban and remove-group-admin.

Added loads more tests to cover group actions.
2020-06-24 11:24:19 +01:00
catbref
ec1954bae1 Notify Chat websocket listeners if group membership changes
Also unified newTransactionExecutor and newBlockExecutor into
callbackExecutor.
2020-06-24 11:22:26 +01:00
catbref
66276a6f65 Merge branch 'ssl' into launch 2020-06-23 14:29:45 +01:00
catbref
c00ab2f87c Add support for HTTPS for API
Requires entries 'sslKeystorePathname' and 'sslKeystorePassword'
in settings.json.

With SSL enabled, API will auto-detect HTTP or HTTPs on the same port.

Included tools/build-keystore.sh to help build keystore from
Let's Encrypt certificates.
2020-06-23 14:27:40 +01:00
catbref
99f3ab9921 /chat/active/{address} now produces entries for groups where {address} is a member, even if there are no messages 2020-06-22 14:16:57 +01:00
catbref
75b15c6639 Improve /chat/active/{address} output to include latest message's sender address, and registered name if applicable 2020-06-19 13:38:38 +01:00
catbref
e5e60a5032 Fix badly named API calls refering to block signers as block minters!
Renamed GET /blocks/minters to /blocks/signers
Renamed GET /blocks/minter/{address} to /blocks/signer/{address}

Changed corresponding repository methods and data classes.
2020-06-16 16:58:34 +01:00
catbref
b9d2bbb78b New /websockets/blocks & some controller/block tidying
Controller.onBlockMinted() now .onNewBlock(BlockData)
which saves having to fetch from repository.
Controller.onNewBlock also takes care of updating Controller's
cached chain tip, requesting SysTray refresh, broadcasting
new tip info to peers and notifying websockets.

BlockMinter and Controller.actuallySynchronize updated
to use unified .onNewBlock.

BlocksWebsocket also returns blocks on demand, given either
integer block height or base58 block signature.

Added support to return ApiError via websockets.
2020-06-15 14:07:09 +01:00
catbref
3d79408574 API GET /blocks/signature/{signature} now returns BLOCK_UNKNOWN instead of null
Also removed unnecessary catch-and-throw of ApiExceptions
2020-06-15 13:18:33 +01:00
catbref
67b184acc9 Chat websockets!
ws://hostname:port/websockets/chat/active/{address}
ws://hostname:port/websockets/chat/messages?txGroupId=XXX
ws://hostname:port/websockets/chat/messages?involving=AAA&involving=BBB
2020-06-12 15:10:55 +01:00
catbref
11040ae60a Added ws://hostname:apiport/websockets/chat/active/{address}
Unified Transaction.importAsUnconfirmed() and Controller.onNetworkTransactionMessage()
to both call Controller.onNewTransaction().

Modified Controller.onNewTransaction() to only send transaction signature to
other peers, instead of full transaction. Peers can request full transaction if they
don't have it.

Controller.onNewTransaction() also calls ChatNotifier, which in turn
notifies websocket handlers about new CHAT transactions.

Added jetty websocket dependency to pom.xml
2020-06-12 10:24:22 +01:00
catbref
a338202ded Fix incorrect PoW buffer usage length in verify & adjust difficulties
CHAT: 8 or 14
MESSAGE: 14
PUBLICIZE: 15
Handshake: 8

Added test to cover verify bug
2020-06-10 10:09:06 +01:00
catbref
e0398490ae Override reference checking for PUBLICIZE transaction type 2020-06-08 09:16:10 +01:00
catbref
847093edac Fix incorrect PoW buffer length usage 2020-06-08 09:01:55 +01:00
catbref
758a42db36 Fix incorrect txType in PUBLICIZE layout 2020-06-05 12:04:59 +01:00
catbref
c2b253df55 Remove extraneous import to cure warning 2020-06-04 18:10:49 +01:00
catbref
cc3adc6720 Correct API models from using "Bitcoin refund/redeem address" to pubkeyhash 2020-06-04 15:51:44 +01:00
catbref
d77acd9eb9 Delete "old" peer from in-memory known peer cache too 2020-06-04 10:57:49 +01:00
catbref
5ad2bc1940 Merge branch 'MESSAGE-PoW' into launch 2020-06-04 10:22:20 +01:00
catbref
d0b4a1f12f Added PoW to MESSAGE (for zero fee). DB and tx layout changes. 2020-06-04 10:20:02 +01:00
catbref
5ffddd0169 Changes to block reward distribution
Any reward leftover from ditributing to legacy QORA holders is reallocated to either:
founders if any online
or
account-level-based reward candidates, if no founders online

We should get pretty close to 100% block reward distribution, barring rounding artifacts.

More documentation and tests.

Removed BlockChain's founderShare as it is calculated in Block on a per-block basis instead.
2020-06-02 10:42:45 +01:00
catbref
b5512dfa91 Rework block rewards to be faster and only reward *online* founders.
Now we sum generic block reward + transaction fees before performing
distribution only once.

Added Map to collate account-balance changes during block reward
distribution so the final changes can be applied in one batch,
reducing DB load.

Some other optimizations like a faster ExpandedAccount.getShareBin().

Passes test EXCEPT RewardTests.testLegacyQoraReward(), pending decision
on how to reallocate 'unspent' block reward.
2020-06-01 16:50:28 +01:00
catbref
bef1828404 Add support for multiple P2SH funding transactions rather than requiring only one 2020-05-29 19:10:20 +01:00
catbref
0ae232b8ba Fix return result from ElectrumX.broadcastTransaction 2020-05-29 19:09:45 +01:00
catbref
31e85226f4 WASM version of MemoryPoW! 2020-05-28 14:09:53 +01:00
catbref
6eea7c2aa1 Disallow registering/updating to a name that looks like an address 2020-05-27 10:56:03 +01:00
catbref
9aabf93523 Merge branch 'PUBLICIZE-txn' into launch 2020-05-27 10:46:27 +01:00
catbref
322e2cdc41 API call /crosschain/p2sh/redeem returns BTC_BALANCE_ISSUE in preference to INVALID_ADDRESS when P2SH balance is too low/zero 2020-05-27 10:44:37 +01:00
catbref
df395c77db Fix up BTCACCT.findP2shSecret given reduced data available since switch to ElectrumX 2020-05-27 10:43:14 +01:00
catbref
274002c473 Fix BTC.getMedianBlockTime() and update tests 2020-05-27 09:29:11 +01:00
catbref
3d4fc38fcb Replaced bitcoinj networking with ElectrumX.
No more bitcoinj peer-group stalls, or slow startups,
or downloading tons of block headers, or checkpoint files.

Now we use ElectrumX protocol to query info from random servers.

Also:
BTC.hash160 callers now use Crypto.hash160 instead.
Added BitTwiddling.fromLEBytes() returns int.

Unit tests seem OK, but needs complete testnet ACCT walkthrough.
2020-05-26 17:47:37 +01:00
catbref
d50f16b8a9 PUBLICIZE transaction for on-chain record of public key 2020-05-25 15:20:21 +01:00
catbref
59de22883b Use CHAT, not MESSAGE, MAX_DATA_SIZE in ChatTransactionTransformer 2020-05-25 08:30:06 +01:00
catbref
db73afaf88 Remove Block.orphan() forced repository debugging 2020-05-25 07:27:42 +01:00
catbref
3afbd7aa51 Recheck for duplicate connection after handshaking to cover race condition with simultaneous bi-directional connections 2020-05-25 07:24:19 +01:00
catbref
0c32afa07f New network handshaking. NOT backwards compatible!
Old Qora v1 message types removed.
Message type values changed.

Network handshaking reworked to fix multiple-connections issue.
Instead of using some random peerID, we now use proper keypairs and a challenge-response handshake to prevent doppelgangers/ID-theft.
This results in simpler handshaking code as we don't have to perform some arcane doppelganger resolution.

Handshaking still uses proof-of-work for challenge-response, but switched to newer MemoryPoW.

API call GET /peers no longer has 'buildTimestamp' field, but does now have 'nodeId' field.

Network no longer has a whole raft of getXXXpeers() due to simplified handshaking.
Quite a few method calls changed to simply Network.getHandshakedPeers(), which is also faster.
2020-05-22 17:16:45 +01:00
catbref
bd543a526b Update uses of old Public/PrivateKeyAccount static methods to Crypto 2020-05-22 17:13:55 +01:00
catbref
b262044a52 Move some crypto methods from Public/PrivateKeyAccount to Crypto for reuse by new network handshaking 2020-05-22 17:06:48 +01:00
catbref
200a97184c Include transaction reference in chat messages returned by API call GET /chat/messages 2020-05-22 08:00:43 +01:00
catbref
5f4b66e5b0 Save public keys from CHAT transactions so they can be fetched via API. 2020-05-20 15:53:43 +01:00
catbref
9c48343581 Potential fix for rare HSQLDB "serialization failure" in Transaction.importAsUnconfirmed() 2020-05-20 07:33:21 +01:00
catbref
219f82f562 Modify API call GET /chats/active/{address} to return info on all joined groups.
Previously GET /chats/active/{address} would only return an active group chat
entry where 'address' was a member AND there was an existing CHAT
transaction with the same tx_group_id (and no recipient).

Now the response contains entries for ALL groups where 'address' is a member,
regardless of an existing CHAT transactions, omitting the 'timestamp' entry
if there are none.
2020-05-19 17:12:41 +01:00
catbref
51bfd49e25 Re-add missing senderName & recipientName to output of API call GET /chat/messages 2020-05-19 15:29:12 +01:00
catbref
7102f4a727 Fix for incorrect amounts reported by API 2020-05-19 15:20:20 +01:00
catbref
28991a926f Fix incorrect getDataLength() in RegisterNameTransactionTransformer 2020-05-19 14:55:26 +01:00
catbref
74f89af841 Enforce version 2+ for DEPLOY_AT 2020-05-19 08:35:25 +01:00
catbref
b4284515e7 Unify transaction NAME_NOT_LOWER_CASE checks to Unicode NAME_NOT_NORMALIZED version 2020-05-19 08:31:36 +01:00
catbref
032c5d0d07 Add missing fee check to TRANSFER_PRIVS 2020-05-19 08:25:53 +01:00
catbref
72100fe1d8 Refactor Unicode 'reduced' name code from side-effects into 'data' objects.
CREATE_GROUP, ISSUE_ASSET, REGISTER_NAME and UPDATE_NAME transactions affected.

The code to actually generate 'reduced' name was called inside isValid() and
relied on setting the corresponding transaction data object field so that it would
be saved by isValid()'s caller. Although this worked, it wasn't a very clean
solution.

Now the 'reduced' name is generated by transaction data object's constructors so
it is always present.

Also removed name/group/asset reduceName(String) methods as they were all the
same single-line call to Unicode.sanitize().
2020-05-19 08:08:21 +01:00
catbref
ed178e744d Merge branch 'asset-unicode' into launch 2020-05-19 07:57:06 +01:00
catbref
94f7079c2e Unicode homoglyph support to Assets 2020-05-19 07:56:17 +01:00
catbref
f1638aa9d9 Removed "owner" from CREATE_GROUP and added Unicode homoglyph support.
Group owner now derived from CREATE_GROUP transaction creator's public key.

Added 'reduced' group name to GroupData, with corresponding change to DB.
Renamed GroupData.getIsOpen() to simply isOpen().

Tidied up CreateGroupTransactionData, adding 'reduced' group name.
Renamed getIsOpen() to simply isOpen().
Added code to generated reduced group name when building genesis block.

Added Group.MIN_NAME_SIZE of 3.

DB tables changed to add reduced_group_name where appropriate,
removing owner where necessary.

Added GroupRepository.reducedGroupNameExists(String).

Fixed up test blockchain configs in src/test/resources/test-chain-v2*.json.
2020-05-18 17:27:32 +01:00
catbref
a7b9215ace Merge branch 'message-wo-recipient' into launch 2020-05-18 10:12:54 +01:00
catbref
956ad7bfa8 Merge branch 'asset-fixes' into launch 2020-05-18 10:12:42 +01:00
catbref
4baf442cb8 Merge branch 'name-fixes' into launch 2020-05-18 10:12:31 +01:00
catbref
24eb7c6933 Allow MESSAGE transactions to have no recipient.
This allows on-chain messages to a group, including NO_GROUP / groupID zero.

No-recipient messages cannot have an amount - where would it go?

Changed MESSAGE serialization layout to add boolean indicating
whether recipient is present.

Changed MESSAGE serialization layout so assetID is after amount,
and only present if amount is non-zero.

Changed DB table structures to cover above.

Added unit tests to cover above.
2020-05-18 09:09:35 +01:00
catbref
38a2af8cd5 Tidy up Assets by removing 'owner' from ISSUE_ASSET.
Owner now derived from issuer's public key.
Maximum asset name length reduced to 40 characters.

Repository table changes.

"owner" removed from test blockchain configs and "issuerPublicKey" used instead
where applicable.

Some getters in the form of "getIs___()" renamed to simply "is____()".
2020-05-15 16:22:13 +01:00
catbref
7447ab20a9 Add index for finding Registered Names using 'reduced' form 2020-05-15 14:18:51 +01:00
catbref
197c742ce7 Major work on Registered Names
Changes include:

* Allowing renaming
* Tracking last-updated timestamps
* More stringent Unicode processing
* Way more unit tests
* Max name length reduction to 40 chars

Note: HSQLDB repository table changes
2020-05-15 14:08:46 +01:00
catbref
f6ed3388a4 BTC tidy-up 2020-05-15 07:45:24 +01:00
catbref
c61690f3e6 BTC class does not need to extend Thread! 2020-05-14 13:18:44 +01:00
catbref
9a94873d0e Fix broken Long vs Long comparison in Block.areAtsValid() 2020-05-14 13:18:17 +01:00
catbref
5c8bda37d1 Rework BTC class for better startup & shutdown.
Controller no longer starts up BTC support during main startup.
This does mean that BTC startup is deferred until first BTC-related
action, and that the first BTC-related action will take much longer
to complete.

Added tests to cover startup/shutdown.

This also fixes splash logo stuck on-screen and broken Controller
shutdown when using REGTEST bitcoin network AND there is no
local regtest bitcoin server running.
2020-05-14 12:52:26 +01:00
catbref
fa08041696 BlockTransformer should skip AT transactions when calculating block length 2020-05-14 12:51:44 +01:00
catbref
f01a34a461 Throw an API error for inappropriate calls on OFFER-state cross-chain ATs.
P2SH-related API calls under /crosschain/ aren't applicable for
cross-chain ATs that are still in OFFER state, only TRADE state.
2020-05-14 12:48:41 +01:00
catbref
ef790a8cb1 Fix missing groupId 0 entry in output from API call GET /chat/active/{address} 2020-05-13 16:30:28 +01:00
catbref
cea0cee9a8 Names: fixes to allow name change and tests to cover 2020-05-13 15:07:36 +01:00
catbref
d9f784ed2b Registered names: changing 'owner' and allowing renaming.
REGISTER_NAME has an "owner" field which can be different from the actual
registrant (transaction creator's public key, used for signing transaction).

This allowed people to register names to be owned by someone else, thus breaking
the whole "one name per account" aspect.

So now "owner" is removed from REGISTER_NAME, and the actual owner address is
derived from transaction creator's public key, as you would expect.

Similarly, UPDATE_NAME has a corresponding "newOwner" field which has been removed.

In addition, UPDATE_NAME now allows users to change their registered name using a new
"newName" field.

Various changes made to DB, Name class, etc. to accomodate above, along with some minor
bug-fixes and comment improvements/corrections.

Needs new unit tests to cover both new functionality and old!
2020-05-13 10:19:56 +01:00
catbref
f29ae656b9 More work on CHAT
Always add group 0 info to output of API call GET /chats/active/{address}.
No groupName entry as it's "no group" or "group-less" or "not group related".
Timestamp also might be omitted if no message found.

Fix output of POST /chats/compute so it doesn't include zeroed 64-byte signature.
2020-05-12 20:27:09 +01:00
catbref
a9852e5305 More work on CHAT API / support.
Renamed GET /chats/search to /chats/messages.

Added GET /chats/active/{address} to return lists of group chats
and direct chats involving {address}, where a chat message exists.
2020-05-12 14:28:41 +01:00
catbref
32470fa641 Improve CHAT API and repository support.
Change CHAT API call GET /chat/search to better support the two
main scenarios of:

group-based chatting: supply txGroupId only
private chatting: supply 2 'involving' addresses only

Added some DB indexes to cater for above.

GET /chat/search now returns specialized ChatMessage objects
instead of ChatTransactions. This is to reduce unnecessary fetching
of data from repository, and onward sending to API client.
2020-05-12 10:02:41 +01:00
catbref
0d1c08bf96 Add index on DB Names.owner to help find names by owner 2020-05-12 10:02:26 +01:00
catbref
026c904ce4 Change processing of network TRANSACTION_SIGNATURES message.
Previously Controller would loop through the transaction signatures,
discard those already known, and then requesting the full transaction
via peer.getResponse(). This would tie up a networking thread for some
time and also potentially cause repository deadlocks, although the latter
could have been fixed another way.

However, the code after peer.getResponse() was identical to the code
processing an incoming TRANSACTION message. Now instead of requesting
and waiting for then processing each transaction, Controller simply
sends the peer a GET_TRANSACTION for each unknown transaction signature.

As the peer responds with corresponding TRANSACTION messages, these can
be processed individually with shorter period of locking.
2020-05-12 08:03:11 +01:00
catbref
59ae070c83 Fix API call POST /peers so it returns "false" for existing peer, instead of throwing / Internal Server Error 2020-05-11 12:59:56 +01:00
catbref
2ab695f308 NTP: don't call shutdownNow() on null instanceExecutor
When using fixed NTP offset, e.g. via "testNtpoffset" in settings.json,
Controller calls NTP.shutdownNow() which throws a NPE because
NTP.instanceExecutor is null.
2020-05-11 12:59:45 +01:00
catbref
f0ff77cd31 Fix ChatTransaction w.r.t. txGroupId meaning. Relax no-QORT PoW difficulty. 2020-05-11 12:58:46 +01:00
catbref
e241d9fa67 Split CHAT compute into separate API call to help UI 2020-05-11 12:58:05 +01:00
catbref
5e9b0cd03c Fix GroupInvites.expires_when in DB to re-allow NULL 2020-05-11 12:55:58 +01:00
catbref
a5c437913f Transaction.isValidTxGroupId() changed from private to protected - needed so ChatTransaction can override 2020-05-11 12:55:22 +01:00
catbref
3fa7da5115 Fix for incorrect blocksMinted count. Added test to cover 2020-05-08 08:51:56 +01:00
catbref
6d8f41ab05 Fix long overflow in Block.distributeBlockRewardToQoraHolders()
Sadly no native 128bit integer support in Java 11 so resorting to using
BigInteger.

Added/improved unit tests to cover.
2020-05-07 16:37:40 +01:00
catbref
3094ec3c26 Massive clean-up of DB & conversion to long for timestamps
Collated all development changes to DB so now we build
initial DB structure directly with final layout.
i.e. no ALTER TABLE, etc.

Reordered HSQLDB 'CREATE TYPE' statements into alphabetical order
for easier maintainability.

Replaced TIMESTAMP WITH TIME ZONE with simple BIGINT ("EpochMillis").
Timezone conversion is now a presentation task, rather than having
pretty values in database.

Removed associated conversion methods, like toOffsetDateTime(),
fromOffsetDateTime() and getZonedTimestampMilli().

Renamed some DB columns to make them more obviously timestamps, like:
Names.registered is now Names.registered_when.

Removed IFNULL(balance, 0) from HSQLDBAccountRepository as balances
are never null, or actually never 0 either.

Added more tests to increase API call, and hence repository, coverage.

Removed unused "milestone block" from Transactions.
2020-05-07 12:16:22 +01:00