Commit Graph

1707 Commits

Author SHA1 Message Date
CalDescent
f51a082049 Validate the previous state's hash each time a new layer is applied.
It's possible that this concept will struggle in the real world if operating systems, virus scanners, etc start interfering with our file stucture. Right now it is using a zero tolerance approach when checking the validity of each layer. We may choose to loosen this slightly if we encounter problems, e.g. by excluding hidden files. But for now it is best to be as strict as possible.
2021-08-17 09:07:46 +01:00
CalDescent
0f1927b4b1 Take a hash of the previous state's directory structure and file contents, and then include that hash in the patch metadata (when creating a new patch). This allows the integrity of the layers to be validated as each one is applied. 2021-08-17 08:43:08 +01:00
CalDescent
9baccc0784 Improved HTTP response generation when serving websites. 2021-08-16 22:40:55 +01:00
CalDescent
95c9cc7f99 Added ArbitraryDataCache
This decides whether to build a new state or use an existing cached state when serving a data resource. It will cache a built resource until a new transaction (i.e. layer) arrives. This drastically reduces load, and still allows for almost instant propagation of new layers.
2021-08-16 22:40:05 +01:00
CalDescent
0ed8e04233 Arbitrary data metadata classes moved to their own package. 2021-08-15 21:54:21 +01:00
CalDescent
b46c328811 Fixed bug in FilesystemUtils.copyAndReplaceDirectory() 2021-08-15 21:50:19 +01:00
CalDescent
c7d88ed95b Added "cache" file to the .qortal metadata folder.
This is used to store the transaction signature and build timestamp for each built data resource. It involved a refactor of the ArbitraryDataMetadata class to introduce a subclass for each file ("patch" and "cache"). This allows more files to be easily added later.
2021-08-15 21:49:45 +01:00
CalDescent
94da1a30dc When merging two states, validate that the transaction signature we are using for the "before" layer matches the "previous transaction signature" that is baked into the "after" layer.
This defends against a missing or out-of-order transaction. If this ever fails validation, we may need to rethink the way we are requesting transactions. But in theory this shouldn't happen, given that the "last reference" field of a transaction ensures that out-of-order transactions are invalid already.
2021-08-15 20:27:16 +01:00
CalDescent
219a5db60c Fixed circular bug in ArbitraryDataMetadata.getPreviousSignature() 2021-08-15 20:21:47 +01:00
CalDescent
a5cfedcae9 Exclude the .qortal metadata directory in all diffs. Also improved logging. 2021-08-15 20:02:14 +01:00
CalDescent
9b8a632f37 Include the .qortal folder in the merge output, since it will be needed for validation. We may choose to exclude it from the final output path, but for now it is left there to make debugging easier. 2021-08-15 19:55:56 +01:00
CalDescent
be0426d9a2 Improved logging 2021-08-15 19:19:55 +01:00
CalDescent
8fac0a02e5 Use the /.qortal/patch file to apply differences when merging together two layers, rather than walking through the file tree like we did before. 2021-08-15 18:49:30 +01:00
CalDescent
fa696a2901 Log exceptions when publishing data updates. 2021-08-15 18:37:27 +01:00
CalDescent
f5615b1c54 Don't exclude hidden files in the zips, as the .qortal folder needs to be included. 2021-08-15 17:45:37 +01:00
CalDescent
9850c294d1 Fixed various issues relating to syncing data for transactions without any chunks. 2021-08-15 11:28:26 +01:00
CalDescent
8929f32068 No longer creating the ".removed" files. 2021-08-15 10:43:00 +01:00
CalDescent
3019bb5c97 Enforce minBlockchainPeers in ArbitraryDataManager, as there is no point in trying to request data files when we don't have the minimum number of peers. 2021-08-15 10:39:00 +01:00
CalDescent
95044d27ce Fixed NPE caused by having an arbitrary transaction with no chunks (which is expected if the total data size is less than the chunk size). 2021-08-15 10:32:37 +01:00
CalDescent
16ac92b2ef Write patch metadata to a file inside a hidden ".qortal" folder which is included with each patch. This can be used in place of the existing ".removed" placeholder files to track removals. 2021-08-15 10:13:32 +01:00
CalDescent
f095964f7b Fixed edge case. 2021-08-14 18:28:22 +01:00
CalDescent
cfba793fcf Fixed bugs in ArbitraryDataFile, introduced when switching to the new temporary path. 2021-08-14 18:10:59 +01:00
CalDescent
c50a11e58a Cleanup intermediate paths in ArbitraryDataWriter. 2021-08-14 17:32:25 +01:00
CalDescent
a7282a5794 Renamed encrypted and zipped files. 2021-08-14 17:31:58 +01:00
CalDescent
efaf313422 When creating an ArbitraryDataFile from a path, make sure to move the file to the correct location within the data directory.
This bug was introduced now that the temp directory is contained within the data directory. Without this, it would leave it in the temp folder and then fail at a later stage.
2021-08-14 16:58:10 +01:00
CalDescent
d9f5753f58 Ensure parent directories exist when creating a diff. 2021-08-14 16:55:19 +01:00
CalDescent
179318f1d8 Ensure a patch exists after creating it. 2021-08-14 16:54:35 +01:00
CalDescent
dd33d24346 Fixed bug with identifier in ArbitraryDataWriter 2021-08-14 16:53:57 +01:00
CalDescent
fa684eabab Removed ArbitraryDataPatches class
This was essentially just a wrapper for a single method.
2021-08-14 15:51:16 +01:00
CalDescent
4b5bd6eed7 Cleanup temporary files used when building data directories. Also added a safety net to only delete files that are without our data or temp directories. 2021-08-14 15:43:08 +01:00
CalDescent
63f5946527 Switched all system-generated temp paths to a user specified "tempDataPath".
This ensures that the temporary files are being kept with the rest of the data, rather than somewhere inappropriate such as on flash storage. It also allows the user to locate them somewhere else, such as on a dedicated drive.
2021-08-14 14:30:06 +01:00
CalDescent
8dac3ebf96 More miscellaneous refactoring 2021-08-14 14:05:33 +01:00
CalDescent
09e783fbf6 Renamed a few variables and methods that weren't caught in previous refactors. 2021-08-14 14:01:30 +01:00
CalDescent
b18b686545 Renamed DATA_FILE to ARBITRARY_DATA_FILE and GET_DATA_FILE to GET_ARBITRARY_DATA_FILE.
This will break existing data nodes, but this is okay, as nothing is in production yet.
2021-08-14 13:57:44 +01:00
CalDescent
e99ea41117 Renamed DataFileMessage to ArbitraryDataFileMessage and GetDataFileMessage to GetArbitraryDataFileMessage 2021-08-14 13:52:41 +01:00
CalDescent
172a37ec8c Renamed org.qortal.storage package to org.qortal.arbitrary 2021-08-14 13:49:19 +01:00
CalDescent
da6b341b63 Renamed DataFile to ArbitraryDataFile, and DataFileChunk to ArbitraryDataFileChunk 2021-08-14 13:25:45 +01:00
CalDescent
16d93b1775 Renamed newly added classes to ArbitraryData*.java instead of DataFile*.java 2021-08-14 13:18:02 +01:00
CalDescent
e15cf063c6 Initial implementation of data patches/updates
This adds support for the PATCH method in addition to the existing PUT method.

Currently, a patch includes only files that have been added or modified, as well as placeholder files to indicate those that have been removed.

This is not production ready, as I am hoping to create patches on a more granular level - i.e. just the modified bytes of each file. It would also make sense to track deletions using a metadata/manifest file in a hidden folder.

It also adds early support of accessing files using a name rather than a signature or hash.
2021-08-14 13:11:36 +01:00
CalDescent
5ac9e3e47a Fixed recently introduced bugs in arbitrary transaction transformation. 2021-08-13 19:14:44 +01:00
CalDescent
743a61bf49 Fixed bug which overwrote the data file, causing it to be invalid. 2021-08-13 06:55:44 +01:00
CalDescent
c790ea07dd Started abstracting the file processing code away from the API handlers, and making it more modular. 2021-08-12 20:53:42 +01:00
CalDescent
b4f980b349 Restrict lists API endpoints to local/apiKey requests only. 2021-08-12 19:52:49 +01:00
CalDescent
673f23b6a0 Improvement to commit f71516f
Now only skipping the HTLC redemption if the AT is finished and the balance has been redeemed by the buyer. This allows HTLCs to be refunded for ATs that have been refunded or cancelled.
2021-08-12 08:29:52 +01:00
8c325f3a8a
original design 2021-08-11 18:18:10 -04:00
CalDescent
f71516f36f Skip finished ATs in the refund API endpoints. 2021-08-11 21:26:29 +01:00
CalDescent
1752386a6c Fixed logging errors in previous commit. 2021-08-11 20:33:54 +01:00
CalDescent
112675c782 Better handling of RPC errors.
Previously, if an error was returned from an Electrum server (such as "server busy") it would throw a NetworkException that would be caught outside of the server loop and cause the entire request to fail.

Instead of throwing an exception, I am now logging the error and returning null, in the same way we do for IOException and NoSuchElementException further up in the same method.

This allows the caller - most likely connectedRpc() - to move on to the next server in the list and try again.

This should fix an issue seen where a "server busy" response from a single server was essentially breaking our implementation, as we would give up altogether instead of trying another server.
2021-08-11 19:22:53 +01:00
CalDescent
3b6ba7641d Updated icon for Qortal.exe 2021-08-10 19:35:03 +01:00
CalDescent
477a35a685 Fixed response schema for GET /lists/blacklist/addresses endpoint 2021-08-10 08:43:47 +01:00