crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
1,731 Commits 85 Branches 118 Tags 530 MiB
c63d238316
Commit Graph

1731 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
CalDescent
d22a03f1a5 Fixed misleading exception string. 2021-08-20 07:51:58 +01:00
CalDescent
ab0aeec434 Default method of serving websites switched from signature to name. 
Before:
GET /site/:signature
GET /site/name/:name

After:
GET /site/signature/:signature
GET /site/:name
 2021-08-20 07:51:27 +01:00
CalDescent
47ff51ce4e Use a random last reference on the very first transaction for an account 
This is needed because we want to allow brand new accounts to publish data without a fee. A similar approach to CrossChainResource.buildAtMessage(). We already require PoW on all arbitrary transactions, so no additional logic beyond this should be needed.
 2021-08-20 07:49:05 +01:00
CalDescent
1d62ef357d Check for null last references at the beginning of the process when creating an arbitrary tx. 2021-08-19 09:10:13 +01:00
CalDescent
59ef66f46d Handle shutdowns when zipping a large number of files. 2021-08-19 08:48:39 +01:00
CalDescent
77479215a6 Throw an exception if a patch file doesn't exist. 2021-08-18 20:32:47 +01:00
CalDescent
11da1f72b1 Added convenience method to make the code more readable. 2021-08-18 20:30:00 +01:00
CalDescent
6375b9d14d Fixed bug in getLatestTransaction() filtering. 2021-08-18 20:27:18 +01:00
CalDescent
42bc12f56d Call cleanupQueues() from the Controller 2021-08-18 16:25:47 +01:00
CalDescent
8515112811 Fixed unused variable issue 2021-08-18 16:24:37 +01:00
CalDescent
bedb87674b FAILURE_TIMEOUT set to 5 minutes 2021-08-18 16:24:13 +01:00
CalDescent
029c038a49 Catch runtime exceptions (e.g. IllegalStateException) when using the arbitrary data reader/writer. 2021-08-18 16:23:46 +01:00
CalDescent
95e905a5ae If a build fails, prevent any rebuilds for 5 minutes. 
This prevents a resource with build problems from getting into a loop due to the browser requesting a rebuild as soon as it fails.
 2021-08-18 16:04:57 +01:00
CalDescent
8a8ec32f2c Added java-diff-utils to pom.xml 
This was accidentally missing from commit cb6fc46.
 2021-08-18 09:59:56 +01:00
CalDescent
cd958398af Exclude all data* paths from gitignore. 2021-08-18 09:59:12 +01:00
CalDescent
2eedafd506 Log the error if a build fails. 2021-08-18 09:58:59 +01:00
CalDescent
9a88c0d579 Apply the unified-diff patch when combining layers. 2021-08-18 09:58:30 +01:00
CalDescent
cb6fc466d1 Create a "unified-diff" patch using java-diff-utils instead of including the entire modified file. I still need to test how well this works with binary files. 2021-08-18 09:57:44 +01:00
CalDescent
a6154cbb43 Don't allow a new layer to be created if it matches the existing state. 2021-08-18 07:51:50 +01:00
CalDescent
9c20967d24 Catch NPE seen a couple of times in Systray.setTrayIcon() 2021-08-18 07:51:26 +01:00
CalDescent
79bbadad2f Initial implementation of arbitrary data build queue 
This adds the loadAsynchronously() method to ArbitraryDataReader, in addition to the existing loadSynchronously() method.

When requesting a website in a browser, previously the building of the resource's layers would be done synchronously in the API handler. This understandably caused many issues, so the building is now done asynchronously by a dedicated thread. A loading screen is shown in its place which auto refreshes every second until the build has completed.
 2021-08-18 07:50:45 +01:00
CalDescent
c3b44cee94 Fixed bugs when computing directory digest. The order was being lost when adding to the second List, which was producing different hashes on different systems. It also doesn't make sense to convert paths to lowercase, given that we want our validation to be case sensitive. 2021-08-18 07:38:08 +01:00
CalDescent
1968496ce1 Invalidate the cache if hash validation fails, so that it can be rebuilt next time. 2021-08-17 09:30:27 +01:00
CalDescent
e1feb46de9 Put test data in a separate folder to real data. 2021-08-17 09:08:03 +01:00
CalDescent
f51a082049 Validate the previous state's hash each time a new layer is applied. 
It's possible that this concept will struggle in the real world if operating systems, virus scanners, etc start interfering with our file stucture. Right now it is using a zero tolerance approach when checking the validity of each layer. We may choose to loosen this slightly if we encounter problems, e.g. by excluding hidden files. But for now it is best to be as strict as possible.
 2021-08-17 09:07:46 +01:00
CalDescent
0f1927b4b1 Take a hash of the previous state's directory structure and file contents, and then include that hash in the patch metadata (when creating a new patch). This allows the integrity of the layers to be validated as each one is applied. 2021-08-17 08:43:08 +01:00
CalDescent
9baccc0784 Improved HTTP response generation when serving websites. 2021-08-16 22:40:55 +01:00
CalDescent
95c9cc7f99 Added ArbitraryDataCache 
This decides whether to build a new state or use an existing cached state when serving a data resource. It will cache a built resource until a new transaction (i.e. layer) arrives. This drastically reduces load, and still allows for almost instant propagation of new layers.
 2021-08-16 22:40:05 +01:00
CalDescent
0ed8e04233 Arbitrary data metadata classes moved to their own package. 2021-08-15 21:54:21 +01:00
CalDescent
b46c328811 Fixed bug in FilesystemUtils.copyAndReplaceDirectory() 2021-08-15 21:50:19 +01:00
CalDescent
c7d88ed95b Added "cache" file to the .qortal metadata folder. 
This is used to store the transaction signature and build timestamp for each built data resource. It involved a refactor of the ArbitraryDataMetadata class to introduce a subclass for each file ("patch" and "cache"). This allows more files to be easily added later.
 2021-08-15 21:49:45 +01:00
CalDescent
94da1a30dc When merging two states, validate that the transaction signature we are using for the "before" layer matches the "previous transaction signature" that is baked into the "after" layer. 
This defends against a missing or out-of-order transaction. If this ever fails validation, we may need to rethink the way we are requesting transactions. But in theory this shouldn't happen, given that the "last reference" field of a transaction ensures that out-of-order transactions are invalid already.
 2021-08-15 20:27:16 +01:00
CalDescent
219a5db60c Fixed circular bug in ArbitraryDataMetadata.getPreviousSignature() 2021-08-15 20:21:47 +01:00
CalDescent
a5cfedcae9 Exclude the .qortal metadata directory in all diffs. Also improved logging. 2021-08-15 20:02:14 +01:00
CalDescent
9b8a632f37 Include the .qortal folder in the merge output, since it will be needed for validation. We may choose to exclude it from the final output path, but for now it is left there to make debugging easier. 2021-08-15 19:55:56 +01:00
CalDescent
be0426d9a2 Improved logging 2021-08-15 19:19:55 +01:00
CalDescent
8fac0a02e5 Use the /.qortal/patch file to apply differences when merging together two layers, rather than walking through the file tree like we did before. 2021-08-15 18:49:30 +01:00
CalDescent
fa696a2901 Log exceptions when publishing data updates. 2021-08-15 18:37:27 +01:00
CalDescent
f5615b1c54 Don't exclude hidden files in the zips, as the .qortal folder needs to be included. 2021-08-15 17:45:37 +01:00
CalDescent
9850c294d1 Fixed various issues relating to syncing data for transactions without any chunks. 2021-08-15 11:28:26 +01:00
CalDescent
8929f32068 No longer creating the ".removed" files. 2021-08-15 10:43:00 +01:00
CalDescent
3019bb5c97 Enforce minBlockchainPeers in ArbitraryDataManager, as there is no point in trying to request data files when we don't have the minimum number of peers. 2021-08-15 10:39:00 +01:00
CalDescent
95044d27ce Fixed NPE caused by having an arbitrary transaction with no chunks (which is expected if the total data size is less than the chunk size). 2021-08-15 10:32:37 +01:00
CalDescent
16ac92b2ef Write patch metadata to a file inside a hidden ".qortal" folder which is included with each patch. This can be used in place of the existing ".removed" placeholder files to track removals. 2021-08-15 10:13:32 +01:00
CalDescent
f095964f7b Fixed edge case. 2021-08-14 18:28:22 +01:00
CalDescent
cfba793fcf Fixed bugs in ArbitraryDataFile, introduced when switching to the new temporary path. 2021-08-14 18:10:59 +01:00
CalDescent
c50a11e58a Cleanup intermediate paths in ArbitraryDataWriter. 2021-08-14 17:32:25 +01:00
CalDescent
a7282a5794 Renamed encrypted and zipped files. 2021-08-14 17:31:58 +01:00
CalDescent
efaf313422 When creating an ArbitraryDataFile from a path, make sure to move the file to the correct location within the data directory. 
This bug was introduced now that the temp directory is contained within the data directory. Without this, it would leave it in the temp folder and then fail at a later stage.
 2021-08-14 16:58:10 +01:00
CalDescent
d9f5753f58 Ensure parent directories exist when creating a diff. 2021-08-14 16:55:19 +01:00