AlphaX/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Issues Pull Requests Projects Releases Wiki Activity
1,409 Commits 77 Branches 112 Tags 430 MiB
0f1927b4b1
Commit Graph

1409 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
CalDescent
0f1927b4b1 Take a hash of the previous state's directory structure and file contents, and then include that hash in the patch metadata (when creating a new patch). This allows the integrity of the layers to be validated as each one is applied. 2021-08-17 08:43:08 +01:00
CalDescent
9baccc0784 Improved HTTP response generation when serving websites. 2021-08-16 22:40:55 +01:00
CalDescent
95c9cc7f99 Added ArbitraryDataCache 
This decides whether to build a new state or use an existing cached state when serving a data resource. It will cache a built resource until a new transaction (i.e. layer) arrives. This drastically reduces load, and still allows for almost instant propagation of new layers.
 2021-08-16 22:40:05 +01:00
CalDescent
0ed8e04233 Arbitrary data metadata classes moved to their own package. 2021-08-15 21:54:21 +01:00
CalDescent
b46c328811 Fixed bug in FilesystemUtils.copyAndReplaceDirectory() 2021-08-15 21:50:19 +01:00
CalDescent
c7d88ed95b Added "cache" file to the .qortal metadata folder. 
This is used to store the transaction signature and build timestamp for each built data resource. It involved a refactor of the ArbitraryDataMetadata class to introduce a subclass for each file ("patch" and "cache"). This allows more files to be easily added later.
 2021-08-15 21:49:45 +01:00
CalDescent
94da1a30dc When merging two states, validate that the transaction signature we are using for the "before" layer matches the "previous transaction signature" that is baked into the "after" layer. 
This defends against a missing or out-of-order transaction. If this ever fails validation, we may need to rethink the way we are requesting transactions. But in theory this shouldn't happen, given that the "last reference" field of a transaction ensures that out-of-order transactions are invalid already.
 2021-08-15 20:27:16 +01:00
CalDescent
219a5db60c Fixed circular bug in ArbitraryDataMetadata.getPreviousSignature() 2021-08-15 20:21:47 +01:00
CalDescent
a5cfedcae9 Exclude the .qortal metadata directory in all diffs. Also improved logging. 2021-08-15 20:02:14 +01:00
CalDescent
9b8a632f37 Include the .qortal folder in the merge output, since it will be needed for validation. We may choose to exclude it from the final output path, but for now it is left there to make debugging easier. 2021-08-15 19:55:56 +01:00
CalDescent
be0426d9a2 Improved logging 2021-08-15 19:19:55 +01:00
CalDescent
8fac0a02e5 Use the /.qortal/patch file to apply differences when merging together two layers, rather than walking through the file tree like we did before. 2021-08-15 18:49:30 +01:00
CalDescent
fa696a2901 Log exceptions when publishing data updates. 2021-08-15 18:37:27 +01:00
CalDescent
f5615b1c54 Don't exclude hidden files in the zips, as the .qortal folder needs to be included. 2021-08-15 17:45:37 +01:00
CalDescent
9850c294d1 Fixed various issues relating to syncing data for transactions without any chunks. 2021-08-15 11:28:26 +01:00
CalDescent
8929f32068 No longer creating the ".removed" files. 2021-08-15 10:43:00 +01:00
CalDescent
3019bb5c97 Enforce minBlockchainPeers in ArbitraryDataManager, as there is no point in trying to request data files when we don't have the minimum number of peers. 2021-08-15 10:39:00 +01:00
CalDescent
95044d27ce Fixed NPE caused by having an arbitrary transaction with no chunks (which is expected if the total data size is less than the chunk size). 2021-08-15 10:32:37 +01:00
CalDescent
16ac92b2ef Write patch metadata to a file inside a hidden ".qortal" folder which is included with each patch. This can be used in place of the existing ".removed" placeholder files to track removals. 2021-08-15 10:13:32 +01:00
CalDescent
f095964f7b Fixed edge case. 2021-08-14 18:28:22 +01:00
CalDescent
cfba793fcf Fixed bugs in ArbitraryDataFile, introduced when switching to the new temporary path. 2021-08-14 18:10:59 +01:00
CalDescent
c50a11e58a Cleanup intermediate paths in ArbitraryDataWriter. 2021-08-14 17:32:25 +01:00
CalDescent
a7282a5794 Renamed encrypted and zipped files. 2021-08-14 17:31:58 +01:00
CalDescent
efaf313422 When creating an ArbitraryDataFile from a path, make sure to move the file to the correct location within the data directory. 
This bug was introduced now that the temp directory is contained within the data directory. Without this, it would leave it in the temp folder and then fail at a later stage.
 2021-08-14 16:58:10 +01:00
CalDescent
d9f5753f58 Ensure parent directories exist when creating a diff. 2021-08-14 16:55:19 +01:00
CalDescent
179318f1d8 Ensure a patch exists after creating it. 2021-08-14 16:54:35 +01:00
CalDescent
dd33d24346 Fixed bug with identifier in ArbitraryDataWriter 2021-08-14 16:53:57 +01:00
CalDescent
fa684eabab Removed ArbitraryDataPatches class 
This was essentially just a wrapper for a single method.
 2021-08-14 15:51:16 +01:00
CalDescent
4b5bd6eed7 Cleanup temporary files used when building data directories. Also added a safety net to only delete files that are without our data or temp directories. 2021-08-14 15:43:08 +01:00
CalDescent
63f5946527 Switched all system-generated temp paths to a user specified "tempDataPath". 
This ensures that the temporary files are being kept with the rest of the data, rather than somewhere inappropriate such as on flash storage. It also allows the user to locate them somewhere else, such as on a dedicated drive.
 2021-08-14 14:30:06 +01:00
CalDescent
8dac3ebf96 More miscellaneous refactoring 2021-08-14 14:05:33 +01:00
CalDescent
09e783fbf6 Renamed a few variables and methods that weren't caught in previous refactors. 2021-08-14 14:01:30 +01:00
CalDescent
b18b686545 Renamed DATA_FILE to ARBITRARY_DATA_FILE and GET_DATA_FILE to GET_ARBITRARY_DATA_FILE. 
This will break existing data nodes, but this is okay, as nothing is in production yet.
 2021-08-14 13:57:44 +01:00
CalDescent
e99ea41117 Renamed DataFileMessage to ArbitraryDataFileMessage and GetDataFileMessage to GetArbitraryDataFileMessage 2021-08-14 13:52:41 +01:00
CalDescent
172a37ec8c Renamed org.qortal.storage package to org.qortal.arbitrary 2021-08-14 13:49:19 +01:00
CalDescent
da6b341b63 Renamed DataFile to ArbitraryDataFile, and DataFileChunk to ArbitraryDataFileChunk 2021-08-14 13:25:45 +01:00
CalDescent
16d93b1775 Renamed newly added classes to ArbitraryData*.java instead of DataFile*.java 2021-08-14 13:18:02 +01:00
CalDescent
e15cf063c6 Initial implementation of data patches/updates 
This adds support for the PATCH method in addition to the existing PUT method.

Currently, a patch includes only files that have been added or modified, as well as placeholder files to indicate those that have been removed.

This is not production ready, as I am hoping to create patches on a more granular level - i.e. just the modified bytes of each file. It would also make sense to track deletions using a metadata/manifest file in a hidden folder.

It also adds early support of accessing files using a name rather than a signature or hash.
 2021-08-14 13:11:36 +01:00
CalDescent
5ac9e3e47a Fixed recently introduced bugs in arbitrary transaction transformation. 2021-08-13 19:14:44 +01:00
CalDescent
743a61bf49 Fixed bug which overwrote the data file, causing it to be invalid. 2021-08-13 06:55:44 +01:00
CalDescent
c790ea07dd Started abstracting the file processing code away from the API handlers, and making it more modular. 2021-08-12 20:53:42 +01:00
CalDescent
e259a09b89 Fixed merge issues. 2021-08-01 09:54:15 +01:00
CalDescent
6472d8438a Merge remote-tracking branch 'qortal/master' 
# Conflicts:
#	pom.xml
#	src/main/java/org/qortal/api/ApiError.java
#	src/main/java/org/qortal/settings/Settings.java
#	src/main/resources/i18n/ApiError_en.properties
 2021-07-31 22:57:07 +01:00
CalDescent
dc8a402a4a Revert "Removed all cross-chain code. It's not needed for data nodes." 
This reverts commit cc59510cd0.

# Conflicts:
#	src/main/java/org/qortal/controller/Controller.java
 2021-07-31 22:55:27 +01:00
CalDescent
f72374488e Revert "Removed block 212937" 
This reverts commit b917da765c.
 2021-07-31 22:54:19 +01:00
CalDescent
3c6d9a4b8e Merge branch 'new-coins' 2021-07-31 21:27:40 +01:00
CalDescent
3073388403 Fix for missing transactions in getWalletTransactions() response 
The previous criteria was to stop searching for more leaf keys as soon as we found a batch of keys with no transactions, but it seems that there are occasions when subsequent batches do actually contain transactions. The solution/workaround is to require 5 consecutive empty batches before giving up. There may be ways to improve this further by copying approaches from other BIP32 implementations, but this is a quick fix that should solve the problem for now.
 2021-07-31 18:10:24 +01:00
CalDescent
67f856c997 Increased minimum order size to 3 DOGE, as some transactions for 2.1 DOGE were being rejected due to the "dust" error. 2021-07-31 12:20:08 +01:00
CalDescent
742fd0b444 Added /crosschain/htlc/refundAll API endpoint 
This is the equivalent of the redeemAll API but can be used from the buyer's side to get their LTC or DOGE back in the event of a problem.
 2021-07-31 12:19:12 +01:00
CalDescent
e1d69c0eae Removed /refund/{ataddress}/{receivingAddress} API 
This was causing confusion and isn't generally needed.
 2021-07-31 11:55:48 +01:00