AlphaX/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Issues Pull Requests Projects Releases Wiki Activity
1,825 Commits 77 Branches 112 Tags 430 MiB
29bd8203b5
Commit Graph

1620 Commits

Author SHA1 Message Date
CalDescent
3e0574e563 Added another missing block archive lookup relating to trade timestamps. 
Note that this is unlikely to be the cause of some of the zero timestamps issue seen on a subset of nodes - there is still likely to be another problem that needs fixing.
 2021-10-25 18:21:40 +01:00
CalDescent
69e557e70d Delete .sha256 file which was left lying around after running the bootstrap unit tests. 2021-10-25 18:20:58 +01:00
CalDescent
1b846be5fc Fixed bug in arbitrary transaction builder 2021-10-24 22:43:58 +01:00
CalDescent
707eb58068 Added testPatchBeforePut() unit test 2021-10-24 22:43:28 +01:00
CalDescent
8630f3be96 Added first end-to-end test of data storage 2021-10-24 19:08:09 +01:00
CalDescent
c90aeba286 Return ArbitraryTransactionData instead of a byte array, as it is more useful if the bytes are transformed separately. 2021-10-24 19:06:30 +01:00
CalDescent
5055cfc6cb Removed unused imports 2021-10-24 17:43:28 +01:00
CalDescent
c222c4eb29 Updated expected hash of demo data as it has been updated. 2021-10-24 17:41:51 +01:00
CalDescent
6c01955561 Refactor to move arbitrary transaction building to its own class. 2021-10-24 17:41:28 +01:00
CalDescent
305e0f1772 Disable validation of previous hash unless validateAllDataLayers is true 
We may not need to validate this at all now that we have the ability to validate the current layer, but I'll leave it as it could be useful for debugging. It is disabled by default so not an issue.
 2021-10-24 15:35:51 +01:00
CalDescent
52a94e3256 Added "validateAllDataLayers" setting (default false) 
When true, the hashes of every layer are validated when building a data resource. When false, only the final layer's hash is validated.
 2021-10-24 14:37:29 +01:00
CalDescent
a418fb18b6 Hash the current state when creating a patch 
This is included in the patch metadata and then validated every time it is rebuilt.
 2021-10-24 13:00:21 +01:00
CalDescent
9cd579d3db Another typo 2021-10-24 12:20:49 +01:00
CalDescent
e1a6ba7377 Fixed incorrect comment. 2021-10-24 12:03:22 +01:00
CalDescent
04aabe0921 Include the original file instead of a patch if the patch is larger than the original file. 
This saves processing and disk space, as there is no point in applying a patch when the original file is smaller and can be included in its entirety.
 2021-10-24 12:02:09 +01:00
CalDescent
8dd4d71d75 Significant rework of patches 
- The "diff type" is now specified per file, allowing for different diff methods in each modified file.
- Patches will only be created when both the before and after files are less than 100kiB in size.
- Patches are validated after creation, and if invalid it will fall back to including the entire file.

This has identified a bug where patching fails for files without trailing newline characters, which still needs to be fixed. Until then, it will fall back to including the entire file in these cases.
 2021-10-24 10:47:47 +01:00
QuickMythril
49dd63af1e
updated BTC electrum servers 2021-10-23 00:46:02 -04:00
CalDescent
18c6f0ccc3
Merge pull request #60 from Tocoolmh58/master 2021-10-22 18:05:38 +01:00
QuickMythril
55c50a4b5b
add API option to return inverse price (#61) 2021-10-22 18:04:53 +01:00
CalDescent
12b3267d5c Added arbitrary data merge tests. 2021-10-22 11:49:15 +01:00
CalDescent
d6d564c027 Fixed refresh interval of loading screen 2021-10-17 16:55:32 +01:00
CalDescent
1fbd5f7922 Fix for issue causing tradeTimestamp to report as 0 for trades in archived blocks. 2021-10-17 09:52:59 +01:00
CalDescent
f0e13fa492 Arbitrary transaction names are now case insensitive 2021-10-15 13:58:27 +01:00
CalDescent
c8d5ac9248 Fixed bug in ArbitraryTransactionTransformer.getDataLength() when missing a name. 2021-10-15 11:32:08 +01:00
CalDescent
aa4f77d4de Fixed merge issues relating to database updates. Existing data nodes will need to delete their db folder and resync. 2021-10-15 09:13:15 +01:00
CalDescent
f3ef112297 Merge remote-tracking branch 'qortal/master' 
# Conflicts:
#	.gitignore
#	pom.xml
#	src/main/java/org/qortal/controller/Controller.java
#	src/main/java/org/qortal/gui/SysTray.java
#	src/main/java/org/qortal/settings/Settings.java
#	src/main/resources/i18n/ApiError_en.properties
#	src/test/java/org/qortal/test/CryptoTests.java
#	src/test/resources/test-settings-v2.json
 2021-10-15 09:03:28 +01:00
CalDescent
581fe17b58 Added message to check the internet connection if the download cannot start. 2021-10-12 08:08:48 +01:00
CalDescent
af8608f302 Show full stack trace when bootstrapping fails for any reason. 2021-10-12 08:08:05 +01:00
CalDescent
290a19b6c6 Log the URL when downloading a bootstrap, to help with problem solving. 2021-10-12 08:01:47 +01:00
CalDescent
73eaa93be8 Added missing space in log entry. 2021-10-11 23:00:59 +01:00
CalDescent
7ab17383a6 Fix for NPE when serialized block bytes are unavailable. 2021-10-10 13:38:10 +01:00
CalDescent
b7d8a83017 Log "Downloading bootstrap..." as well as showing it in the splash screen. 2021-10-09 17:46:08 +01:00
CalDescent
e7bf4f455d Added missing repository.saveChanges() when reimporting data after creating a bootstrap. 2021-10-09 16:57:53 +01:00
CalDescent
a7f212c4f2 Create a .sha256 file to accompany each bootstrap 
This can ultimately be validated after download, and can also be used to help coordinate updates on the various bootstrap hosts.
 2021-10-09 16:57:19 +01:00
CalDescent
eb991c6026 Fixed issue causing bootstrap validation to be ignored before creation. 2021-10-09 16:29:40 +01:00
CalDescent
a78af8f248 Added SHA-256 file digest utility methods. 
These read the file in small chunks, to reduce memory.
 2021-10-09 16:22:21 +01:00
CalDescent
f34bdf0f58 Fixed issue causing minting accounts to be lost in some cases when auto bootstrapping. 2021-10-09 14:31:13 +01:00
CalDescent
9f488b7b77 Sleep for 5s before cleaning up temp path, in case this improves reliability on Windows. 2021-10-09 13:03:32 +01:00
CalDescent
3fb7df18a0 Delete temp directories at the beginning of the bootstrap process too, as Windows doesn't like deleting it at the end of the process. 2021-10-09 13:02:47 +01:00
CalDescent
00401080e0 Simplified cleanup process. Individual deletions aren't needed as they are all inside the main temp directory. 2021-10-09 13:02:00 +01:00
CalDescent
b265dc3bfb Don't log the complete stack trace for exceptions generated by bootstrap.checkRepositoryState(). The error message is enough in these cases. 2021-10-09 11:47:49 +01:00
CalDescent
63cabbe960 Log the full exception details and stack trace when creating bootstraps. 2021-10-09 11:39:08 +01:00
CalDescent
f6c1a7e6db Disregard exceptions in the bootstrap creation cleanup process because these don't affect the created bootstrap - instead just log the exception and full stack trace. 2021-10-09 11:38:13 +01:00
CalDescent
a3dcacade9 Now showing errors directly in the POST /bootstrap/create API response. 
This avoids needing to check the log file each time.
 2021-10-09 11:02:21 +01:00
CalDescent
f53e2ffa47 Add initial peers on node startup if we don't have any in the repository. 
This will be needed for future bootstraps, which don't contain any peers. It is also useful for those who have used the DELETE /peers/known API.
 2021-10-08 19:10:02 +01:00
CalDescent
a1e4047695 Rework of bootstrap finalization process. 2021-10-08 18:06:41 +01:00
CalDescent
47ce884bbe Delete all known peers when creating a bootstrap 2021-10-08 15:24:10 +01:00
CalDescent
1b17c2613d Show "full node" or "top-only" in the "Downloading bootstrap" message. 2021-10-08 13:12:47 +01:00
CalDescent
dedc8d89c7 Handle case when attempting to load a block from the archive by reference, but the referenced block is in the main block repository, not the archive. This is the case with the genesis block. 
Should fix issue where no block summaries were returned when syncing from block 1
 2021-10-08 12:51:02 +01:00
CalDescent
d00fce86d2 Treat the genesis block as unpruned, as we leave this in the HSQLDB repository. 2021-10-08 12:42:23 +01:00
CalDescent
abab2d1cde Fixed issue preventing blocks from being served from the archive. 
Now prefixing the byte buffer with the block height to mimic a cached block message.
 2021-10-08 12:22:21 +01:00
CalDescent
33b715eb4e Merge branch 'networking' into v2.0-beta 
# Conflicts:
#	src/main/java/org/qortal/settings/Settings.java
 2021-10-07 18:53:49 +01:00
CalDescent
f6effbb6bb Removed unnecessary repository parameter from PruneManager.isBlockPruned() 2021-10-07 18:51:52 +01:00
CalDescent
dff9ec0704 Don't attempt to cache blocks from the archive, as they will never be recent 2021-10-07 18:50:59 +01:00
CalDescent
bfaf4c58e4 Make sure to check the archive when serving block summaries and signatures 2021-10-07 18:50:25 +01:00
CalDescent
ab7d24b637 Updated status wording 2021-10-07 09:02:28 +01:00
CalDescent
c256dae736 Ensure that the temp directory is always in the parent directory of the db folder. 2021-10-07 09:02:13 +01:00
CalDescent
045026431b Create a cleaner base directory path, without the "/./" 2021-10-06 19:50:32 +01:00
CalDescent
4dff91a0e5 Initial bootstrap import retry interval reduced from 5 minutes to 1 minute 2021-10-06 19:45:18 +01:00
CalDescent
7105872a37 Improved exception message 2021-10-06 19:44:30 +01:00
CalDescent
179bd8e018 Moved repository reopen to the finally {} block, so that we're never left without a repository instance. Should fix occasional "No repository available" error seen when retrying. 2021-10-06 19:44:04 +01:00
CalDescent
c82293342f Show full exception stack trace when a bootstrap import fails 2021-10-06 19:32:49 +01:00
CalDescent
8d6dffb3ff Added test for bootstrap random host selection. 2021-10-06 18:23:17 +01:00
CalDescent
2f6a8f793b Invert the colours in the splash screen 2021-10-06 18:22:52 +01:00
CalDescent
9bcd0bbfac Reduce log spam 2021-10-06 18:22:38 +01:00
CalDescent
cd359de7eb Scheduled maintenance now enabled by default, but uses a min and a max, to reduce the chances of multiple nodes running maintenance at the same time. Default to min: 7 days, max: 30 days. 2021-10-06 18:22:31 +01:00
Tocoolmh58
000f9ed459
Update ApiError_de.properties 2021-10-06 17:23:16 +02:00
Tocoolmh58
c5b2c0b4ec
Create SysTray_de.properties 2021-10-06 17:01:52 +02:00
CalDescent
b7e9af100a Added scheduled repository maintenance feature. Currently disabled by default. 2021-10-06 08:52:27 +01:00
CalDescent
0d6409098f Added another bootstrap host 2021-10-05 22:08:18 +01:00
CalDescent
e07238ded8 Fixed variable name 2021-10-04 22:52:47 +01:00
CalDescent
ddf966d08c Show progress status when extracting files 2021-10-04 22:44:51 +01:00
CalDescent
65dca36ae1 Show progress status when downloading a bootstrap 2021-10-04 22:38:58 +01:00
CalDescent
289dae0780 Fixed issue causing the local repository data backup to be overwritten with an empty list. 2021-10-04 09:28:16 +01:00
CalDescent
71f802ef35 Exponentially backoff when bootstrapping fails, to reduce bandwidth 
The retry interval starts at 5 minutes and doubles with each failure.
 2021-10-04 09:25:23 +01:00
CalDescent
0135f25b9d Delete existing repository before extracting bootstrap 
This limits the amount of additional space needed to the size of the compressed bootstrap (currently just under 4GB for full nodes, or 200MB for top-only nodes).
 2021-10-04 09:15:54 +01:00
CalDescent
de3ebf664f Fixed issue with format specifier 2021-10-04 09:11:11 +01:00
CalDescent
850d879726 Use a "tmp" folder in the Qortal directory rather than a system generated temp folder. 
This avoids the need to move files between partitions, and we also can't assume that the system partition has enough space to do the extraction.
 2021-10-04 09:10:56 +01:00
CalDescent
889f6fc5fc Add a "testnet-" prefix in filenames when creating or importing bootstraps on testnet, so that the two databases can be kept separate. 2021-10-03 22:57:38 +01:00
CalDescent
41c2ed7c67 Fixed out of memory errors when copying AT states. 2021-10-03 22:51:15 +01:00
CalDescent
cdf47d4719 Reduce log spam. 2021-10-03 22:33:36 +01:00
CalDescent
4f48751d0b Fixed issue caused when trying to update the splash frame status in a headless environment. 2021-10-03 19:43:10 +01:00
CalDescent
b6d3e82304 Update status when performing repository maintenance 2021-10-03 19:31:05 +01:00
CalDescent
4f892835b8 Show maximum time estimations in archiving and pruning statuses 2021-10-03 18:41:47 +01:00
CalDescent
ac49221639 Show warning status on startup if the database is missing the AtStatesHeightIndex. 2021-10-03 18:34:21 +01:00
CalDescent
75ed5db3e4 Test multiple files when bulk archiving. 2021-10-03 17:33:02 +01:00
CalDescent
59c8e4e6a2 Fixed bug in earlier commit 2021-10-03 16:09:52 +01:00
CalDescent
52b322b756 Take a backup of local data before overwriting with a bootstrap. 
Also moved the import phase to after the validation phase, so that the data returns after the bootstrap.
 2021-10-03 16:09:40 +01:00
CalDescent
dc876d9c96 Force a bootstrap if the block archive isn't intact on launch 
This allows the topOnly setting to be disabled and the node will automatically bootstrap to the archive version. A rebuild isn't attempted if bootstrapping is disabled, in order to reduce risk.
 2021-10-03 16:00:11 +01:00
CalDescent
5b028428c4 Checkpoint immediately after starting/upgrading the repository 
This should fix a longstanding issue where quitting the core before the first checkpoint (1-2 hours after first launch) causes the database to become corrupt.
 2021-10-03 15:47:10 +01:00
CalDescent
f67a0469fc SplashFrame styling improvements 2021-10-03 15:33:53 +01:00
CalDescent
494cd0efff Added white background to splash frame - I think it looks nicer this way, and it may solve the X2Go issues too. 2021-10-03 15:27:19 +01:00
CalDescent
fc8e38e862 Show version number in the "Starting Qortal Core" status. 2021-10-03 15:22:21 +01:00
CalDescent
f09fb5a209 Run the bulk block archiver any time that it isn't reported to be up to date. This should allow it to re-run if the user quits the core before it completes. 2021-10-03 15:17:31 +01:00
CalDescent
b00c1c1575 Update splash frame statuses when reshaping, pruning, or building the block archive 2021-10-03 15:16:36 +01:00
CalDescent
7e5dd62a92 Show bootstrap statuses on splash frame. 2021-10-03 15:01:50 +01:00
CalDescent
35718f6215 Fixed issue with bootstrap retries. 2021-10-03 15:00:40 +01:00
CalDescent
a6d3891a95 Fixed bugs when importing bootstrap. 2021-10-03 09:48:14 +01:00
CalDescent
9591c4eb58 Added support for creating top-only bootstraps 2021-10-02 19:13:18 +01:00
CalDescent
8aaf720b0b Force archiveEnabled to false if we're in top only mode. 
Most of the code handles this case anyway, but it's an easy place for bugs to be created. So it's safest to enforce it at the settings level.
 2021-10-02 19:11:47 +01:00
CalDescent
63a35c97bc Fixed bug in path returned to POST /bootstrap/create API. 2021-10-02 19:10:23 +01:00
CalDescent
8eddaa3fac Small refactor to update wording. 2021-10-02 15:48:31 +01:00
CalDescent
1b3f37eb78 Delete the "archive" folder when in top-only mode 
This allows a block archive node to be switched into top-only mode.
 2021-10-02 15:37:06 +01:00
CalDescent
1f8fbfaa24 Missed a few topOnly references from the last commit. 2021-10-02 15:34:55 +01:00
CalDescent
ea92ccb4c1 "pruningEnabled" setting renamed to "topOnly" 
Pruning is still a concept in the code, but since it relates to both archived and topOnly mode, it makes it cleaner to use "topOnly" to refer to the pruned db with no archive.
 2021-10-02 15:13:23 +01:00
CalDescent
51bb776e56 Select a random host when importing a bootstrap, and started adding support for multiple bootstrap types. 2021-10-02 15:03:31 +01:00
CalDescent
47b1b6daba Retry the entire bootstrap import process on failure, rather than just the download. 2021-10-02 14:43:26 +01:00
CalDescent
adeb654248 Rework of repository maintenance and backups 
It will now attempt to wait until there are no other active transactions before starting, to avoid deadlocks. A timeout for this process is specified - generally 60 seconds - so that callers can give up or retry if something is holding a transaction open for too long. Right now we will give up in all places except for bootstrap creation, where it will keep retrying until successful.
 2021-10-02 13:23:26 +01:00
CalDescent
c4d7335fdd Fixed more issues that could cause transactions to be held open. 2021-10-02 13:05:16 +01:00
CalDescent
ca7f42c409 Reduced unnecessary database queries in the block archiver. 2021-10-02 11:52:20 +01:00
CalDescent
ca02cd72ae Fixed issue in block archiver, which caused it to hold a transaction open for a very long time. This caused deadlocks when trying to create bootstraps or perform repository maintenance. 2021-10-02 11:51:53 +01:00
CalDescent
1ba542eb50 Simplified minting code in block archive tests. 2021-10-01 14:51:45 +01:00
CalDescent
53cd967541 Added defrag (repository maintenance) tests. 2021-10-01 14:51:28 +01:00
CalDescent
49749a0bc7 Added more precise checking of database states to the bulk pruning test. This highlighted a major bug in the bulk prune process whereby the recent AT states weren't being retained. 2021-10-01 11:03:56 +01:00
CalDescent
446f924380 Added bulk pruning test, which highlighted some bugs in both bulk and regular pruning. 2021-10-01 09:32:35 +01:00
CalDescent
5b231170cd Fixed small issues with block archive tests. 2021-10-01 08:10:28 +01:00
CalDescent
7375357b11 Added bootstrap tests 
This involved adding a feature to the test suite in include the option of using a repository located on disk rather than in memory. Also moved the bootstrap compression/extraction working directories to temporary folders.
 2021-10-01 07:44:33 +01:00
CalDescent
347d799d85 Reduce log spam. 2021-09-28 20:30:06 +01:00
CalDescent
0d17f02191 Pass a repository instance into the bulk archiving and pruning methods. 
This is a better approach than opening a new session for each, and it makes it easier to write unit tests.
 2021-09-28 20:29:53 +01:00
CalDescent
ce5bc80347 Increased threshold of BlockArchiveWriter.isArchiverUpToDate() from 90 to 95%. 
In practice, the reading from a correctly archived chain with 550k blocks is currently around 99.5%, but it will be lower if starting with a chain that isn't fully synced.
 2021-09-28 20:21:19 +01:00
CalDescent
0a4479fe9e Initial implementation of automatic bootstrapping 
Currently supports block archive (i.e. full) bootstraps only. Still need to add support for top-only bootstraps.
 2021-09-28 20:17:19 +01:00
CalDescent
de8e96cd75 Added Blockchain.validateAllBlocks() to check every block back to genesis. 
This is extremely slow and shouldn't be needed in normal use cases. It currently checks that each block references the one before, but can ultimately be expanded to check more information about each block and its derived data.
 2021-09-28 09:28:47 +01:00
CalDescent
e2a62f88a6 Modified repository backup and recovery to allow a custom filename to be specified. 2021-09-28 09:26:18 +01:00
CalDescent
8926d2a73c Rework of import/export process. 
- Adds support for minting accounts as well as trade bot states
- Includes automatic import of both types on node startup, and automatic export on node shutdown
- Retains legacy trade bot states in a separate "TradeBotStatesArchive.json" file, whilst keeping the current active ones in "TradeBotStates.json". This prevents states being re-imported after they have been removed, but still keeps a copy of the data in case a key is ever needed.
- Uses indentation in the JSON files for easier readability.
 2021-09-28 09:24:21 +01:00
CalDescent
114833cf8e Fixed NPE when attempting to lookup a block signature that doesn't exist. 2021-09-27 09:13:19 +01:00
CalDescent
656896d16f Fixed issue causing base block prune/trim heights to not be updated on the final pass. 2021-09-24 21:36:05 +01:00
CalDescent
19bf8afece Fixed bug in pruning phase on node startup 
This was causing very recent AT states to be deleted accidentally, because we weren't rebuilding the LatestATStates table before running the query. We should add unit tests to cover this process in case there are any other undiscovered problems.
 2021-09-24 20:52:45 +01:00
CalDescent
841b6c4ddf Fixed another issue causing ATStatesHeightIndex to go missing after pruning. 2021-09-24 15:58:09 +01:00
CalDescent
4c171df848 Disable archiving and pruning if the AtStatesHeightIndex is missing, and log it so that the user knows they should bootstrap or resync. 2021-09-24 11:14:18 +01:00
CalDescent
1f79d88840 Fixed errors found in unit tests. 2021-09-24 09:35:36 +01:00
CalDescent
4856223838 Fixed error in rebase. 2021-09-24 08:50:00 +01:00
CalDescent
74ea2a847d Added unit tests for trimming, pruning, and archiving. 2021-09-24 08:37:36 +01:00
CalDescent
9813dde3d9 Added importFromArchive() feature 
This allows archived blocks to be imported back into HSQLDB in order to make them SQL-compatible again.
 2021-09-24 08:37:36 +01:00
CalDescent
fea7b62b9c Fixed some bugs found in unit testing. 2021-09-24 08:37:36 +01:00
CalDescent
37e03bf2bb Removed BLOCK_LIMIT_REACHED result from the block archive writer. 
This wasn't needed, and is now instead caught by the NOT_ENOUGH_BLOCKS result.
 2021-09-24 08:37:36 +01:00
CalDescent
5656de79a2 Removed maxDuplicatedBlocksWhenArchiving setting as it's no longer needed. 2021-09-24 08:37:36 +01:00
CalDescent
70c6048cc1 Added block archive mode 
This takes all trimmed blocks (which should now be all but the last 1450 or so) and moves them into flat files. Each file contains the serialized bytes of as many blocks that can fit within the file size target of 100MiB.

As a result, the HSQLDB size drops to less than 1GB, making it much faster and easier to maintain. It also significantly reduces the total size of each full node, because the data is stored in a highly optimized way.

HSQLDB then works similarly to the way it does in pruning mode - it holds all transactions, the latest state of every AT, as well as the full AT states data and hashes for the past 1450 blocks.

Each archive file contains headers and indexes in order to quickly locate blocks. When a peer requests a block that is within the archive, the serialized bytes are sent directly without the need to go via a BlockData object. Now that there are no slow queries or data serialization processes needed, it should greatly speed up the block serving.

The /block API endpoints have been modified in such a way that they will also check and retrieve blocks from the archive when needed.

A lightweight "BlockArchive" table is needed in HSQLDB to map block heights to signatures minters and timestamps. It made more sense to keep SQL support for these basic attributes of each block. These are located in a separate table from the full blocks, in order to create a clear distinction between HSQLDB blocks and archived blocks, and also to speed up query times in the Blocks table, which is the one we are using 99% of the time.

There is currently a restriction on the /admin/orphan API endpoint to prevent orphaning beyond the threshold of the block archive.
 2021-09-24 08:37:36 +01:00
CalDescent
87595fd704 Synchronized LatestATStates, to make rebuildLatestAtStates() thread safe. 2021-09-24 08:37:15 +01:00
CalDescent
dc030a42bb Moved trimming and pruning classes into a single package (org.qortal.controller.repository) 2021-09-24 08:37:15 +01:00
CalDescent
89283ed179 Increased atStatesPruneBatchSize from 10 to 25. 2021-09-24 08:36:52 +01:00
CalDescent
64e8a05a9f Prune ATStatesData as well as the ATStates when switching to pruning mode. 2021-09-24 08:36:52 +01:00
CalDescent
676320586a Updated tests to use the renamed method. 2021-09-24 08:36:52 +01:00
CalDescent
734fa51806 Unified the code to build the LatestATStates table, as it's now used by more than one class. 
Note - the rebuildLatestAtStates() must never be used by two different classes at the same time, or AT states could be incorrectly deleted. It is okay at the moment as we don't run the AT states trimmer and pruner in the same app session. However we should probably synchronize this method so that we don't accidentally call it from two places in the future.
 2021-09-24 08:36:52 +01:00
CalDescent
f056ecc8d8 Added bulk pruning phase on node startup the first time that pruning mode is enabled. 
When switching from a full node to a pruning node, we need to delete most of the database contents. If we do this entirely as a background process, it is very slow and can interfere with syncing. However, if we take the approach of transferring only the necessary rows to a new table and then deleting the original table, this makes the process much faster. It was taking several days to delete the AT states in the background, but only a couple of minutes to copy them to a new table.

The trade off is that we have to go through a form of "reshape" when starting the app for the first time after enabling pruning mode. But given that this is an opt-in mode, I don't think it will be a problem.

Once the pruning is complete, it automatically performs a CHECKPOINT DEFRAG in order to shrink the database file size down to a fraction of what it was before.

From this point, the original background process will run, but can be dialled right down so not to interfere with syncing.
 2021-09-24 08:36:52 +01:00
CalDescent
1a722c1517 Break out of the AT pruning inner loops if we're stopping the app. 2021-09-24 08:36:51 +01:00
CalDescent
44607ba6a4 Fixed NPE introduced in earlier commit. 2021-09-24 08:36:51 +01:00
CalDescent
01d66212da Updated AT states pruner as it previously relied on blocks being present in the db to make decisions. As a side effect, this now prunes ATs up the the pruneBlockLimit too, rather than keeping the last 35 days or so. Will review this later but I don't think we will need the missing ones. 2021-09-24 08:36:51 +01:00
CalDescent
925e10b19b Rework of Blockchain.validate() to account for pruning mode. 2021-09-24 08:36:51 +01:00
CalDescent
1b4c75a76e Prune all blocks up until the blockPruneLimit 
By default, this leaves only the last 1450 blocks in the database. Only applies when pruning mode is enabled.
 2021-09-24 08:36:51 +01:00
CalDescent
3400e36ac4 Started work on pruning mode (top-only-sync) 
Initially just deleting old and unused AT states, to get this table under control. I have had to delete them individually as the table can't handle complex queries due to its size.

Nodes in pruning mode will be unable to serve older blocks to peers.
 2021-09-24 08:36:51 +01:00