crowetic/qortal
1
0
Fork 0
forked from Qortal/qortal
Code Pull Requests Activity
1,414 Commits 85 Branches 118 Tags
Commit Graph

1414 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
CalDescent
6d549b0754 Updated AdvancedInstaller project for v1.5.5 2021-06-23 19:57:03 +01:00
CalDescent
02dd64558f Bump version to 1.5.5 v1.5.5 2021-06-23 18:21:55 +01:00
CalDescent
ea5e2f5580 Added POST /site/preview API 
This can be used to preview a site before signing a transaction and announcing it to the network. The response will need reworking to return JSON (along with most of the other new APIs)
 2021-06-23 09:28:38 +01:00
CalDescent
b65c7a75fe Handle relative links when parsing HTML. 2021-06-23 09:26:41 +01:00
CalDescent
39f5dce51c Moved "directory" data uploads to new POST /site/upload API. 
Directory uploads don't make much sense outside of website hosting, so it's best to make this API specific to that purpose.
 2021-06-23 09:10:06 +01:00
CalDescent
f77ec1faf6 A very crude proof of concept which serves a website from a zipped (and in future, chunked) data blob. This forms the beginnings of the "website hosting" layer on top of the data storage. It needs a significant rework - most importantly so that we aren't serving every asset from memory, and also so that the correct content-type headers are returned, etc. 2021-06-23 08:43:21 +01:00
CalDescent
cd3a1e0159 Increased max file size to 1GiB. Will review this again later. 2021-06-23 08:33:53 +01:00
CalDescent
3f20fadb81 When zipping files, rename the outer folder to "data" instead of using the original folder name. This means that the data can be accessed deterministically without the need to first lookup the folder name. 2021-06-23 08:09:59 +01:00
CalDescent
1c6428dd3b Added equivalent split and join test but this time using a 5.5MiB source file. 2021-06-23 08:07:35 +01:00
CalDescent
aca620241a More work on data file split/join, and added a test. 2021-06-22 08:58:16 +01:00
CalDescent
808b36e088 Specify chunk size when splitting. 2021-06-22 07:44:34 +01:00
CalDescent
1613375cc0 Added more validation of files received in GET /data/file/frompeer 2021-06-21 19:03:34 +01:00
CalDescent
787ef957d2 Added support for uploading an entire directory via POST /data/upload/path 
If a directory is specified instead of a file, the directory is automatically zipped before being split into chunks.
 2021-06-21 19:02:49 +01:00
CalDescent
b915d0aed5 Only create the output file directories when we are actually writing a file there. This should prevent empty directories being created when initializing a nonexistent DataFile using a hash. 2021-06-21 08:40:52 +01:00
CalDescent
16dc5b5327 Include the data length in DataFileMessage, which is more similar to the approach used by ArbitraryDataMessage. These two message types are very similar, except arbitrary code currently has a requirement of one piece of data per signature, whereas DataFile code is independent and can support multiple files per transaction. Maybe the two can be combined at some point, but for now I'll keep them separate. 2021-06-20 19:49:10 +01:00
CalDescent
d25e98d9c4 Include peer connection ID in recently created log message. 2021-06-20 10:26:14 +01:00
CalDescent
227cdc1ec8 Log each sync attempt when our blockchain isn't up to date 
Without this it can appear as though nothing is happening for a while after the app launches.
 2021-06-20 07:25:25 +01:00
CalDescent
c2d0c63db0 Improved error logging. 2021-06-19 20:31:04 +01:00
CalDescent
f5c9807a48 Use contains() rather than equals() when matching a peer in /data/file/frompeer, so that the port can be optionally left out. 2021-06-19 20:29:05 +01:00
CalDescent
7e9b1d5e16 Rework of DataFile.base58Digest() 
This fixes an NPE when trying to send a file that doesn't exist. It also removes the caching, which we can add again later if it turns out to be needed.
 2021-06-19 20:25:25 +01:00
CalDescent
5070c4eea9 Better handling of data file responses in the /data/file/frompeer API endpoint. 2021-06-19 20:23:33 +01:00
CalDescent
33d9c51b6f Validate supplied base58 string in /data/file/frompeer API endpoint 2021-06-19 19:26:13 +01:00
CalDescent
d0f9d478c2 Fixed bug which prevented the DATA_FILE message ID from making it through to the reply queue. 2021-06-19 19:05:11 +01:00
CalDescent
f296ec46c8 Removed unused headers. 2021-06-19 17:32:55 +01:00
CalDescent
64d19e480b Chunk size set to 1MB for now, as it seems that our networking code has problems when transferring 2MB chunks. We can increase this later once that problem has been fixed. 2021-06-19 17:32:48 +01:00
CalDescent
2c585a9328 Upper connection time limit reduced from 60 mins to 20 mins. 
Now that we aren't disconnecting mid sync, we can get away with more frequent disconnections. This brings the average connection length to around 9 mins.
 2021-06-19 17:25:50 +01:00
CalDescent
45b0d9e19b Added more initial peers, submitted by CWDSYSTEMS 2021-06-19 16:19:53 +01:00
CalDescent
026a4b896c Added BTC and LTC electrum nodes submitted by CWDSYSTEMS 2021-06-19 14:50:18 +01:00
CalDescent
78237fcd11 Don't intentionally disconnect peers if we are currently syncing with them. 2021-06-19 13:12:16 +01:00
CalDescent
73cc3dcb92 Force a disconnect of each peer when the connection age reaches the maximum allowed time. 
Connection limits are defined in settings (denominated in seconds):
"minPeerConnectionTime": 120,
"maxPeerConnectionTime": 3600

Peers will disconnect after a randomly chosen amount of time between the min and the max. The default range is 2 minutes to 1 hour, as above.

This encourages nodes to connect to a wider range of peers across the course of each day, rather than staying connected to an "island" of peers for an extended period of time. Hopefully this will reduce the amount of parallel chains that can form due to permanently connected clusters of peers.

We may find that we need to reduce the defaults to get optimal results, however it is best to do this incrementally, with the option for reducing further via each node's settings. Being too aggressive here could cause some of the earlier problems (e.g. 20% missing blocks minted) to reappear. We can re-evaluate this in the next version. Note that if defaults are reduced significantly, we may need to add code to prevent this from happening mid-sync. With higher defaults, this is less of an issue.

Thanks to @szisti for supplying some base code for this commit, and also to @CWDSYSTEMS for diagnosing the original problem.
 2021-06-19 13:03:46 +01:00
CalDescent
4cff03e7fe Include "size" value in the "Synchronized with peer" logs. 
This indicates the size of the re-org/rollback that was required in order to perform this sync operation. It is only included if it's greater than 0 blocks.
 2021-06-19 09:04:14 +01:00
CalDescent
8e35f131d5 Removed debugging log that wasn't intended to be committed. 2021-06-18 08:33:51 +01:00
CalDescent
aafb9d7e4f Include running total in "Sent X bytes" log entry. 2021-06-18 08:05:35 +01:00
CalDescent
652f30bdbd Added DATA_FILE and GET_DATA_FILE message types. 2021-06-18 08:02:57 +01:00
CalDescent
f4ba7b2a0c Added onNetworkGetDataFileMessage() handler 2021-06-18 08:02:13 +01:00
CalDescent
592490d709 Added GET /data/file/frompeer API endpoint 
This requests a file from the supplied peer address, and stores a copy locally if successful. Still a work in progress.
 2021-06-18 07:59:46 +01:00
CalDescent
5ac676d201 Added DataFileMessage and GetDataFileMessage, used for requesting and sending files between peers. 2021-06-17 19:05:00 +01:00
CalDescent
abfe0a925a More DataFile methods and improvements 2021-06-17 18:20:42 +01:00
CalDescent
fa11f4f45b Moved DataFileChunk(byte[] fileContent) constructor to the superclass so it can be used by regular data files too. 2021-06-17 18:20:11 +01:00
CalDescent
1e8dbfe4b7 Delete chunk if it fails the hash validation in the constructor. 2021-06-17 18:18:19 +01:00
CalDescent
f82f2bd287 Added DELETE /data/file API endpoint 
This deletes a file referenced by a user supplied SHA256 digest string (which we will use as the file's "ID" in the Qortal data system). In the future this could be extended to delete all associated chunks, but first we need to build out the data chain so we have a way to look up chunks associated with a file hash.
 2021-06-16 20:03:15 +01:00
CalDescent
76742c3869 Removed .dat extension, and use an extra level in the directory structure (data/aB/cD/aBcDeF... etc) 2021-06-16 19:49:12 +01:00
CalDescent
9407e7e418 Data storage location moved to settings ("dataPath") 2021-06-16 19:22:50 +01:00
CalDescent
120552b36e Added resource file missing from last commit. 2021-06-16 19:13:24 +01:00
CalDescent
9fb58c7ae3 Added the beginnings of an upload API, with some basic data file management. 2021-06-16 19:10:35 +01:00
CalDescent
b917da765c Removed block 212937 2021-06-15 09:29:07 +01:00
CalDescent
cc59510cd0 Removed all cross-chain code. It's not needed for data nodes. 2021-06-15 09:27:05 +01:00
CalDescent
86aab7023c Initial settings for data node development. Most to be decided later. 2021-06-14 19:40:37 +01:00
CalDescent
e3b0a41ba9 Merge branch 'sync-multiple-blocks' 2021-06-14 19:01:17 +01:00
CalDescent
802c55dfc8 Merge branch 'networking' 2021-06-14 19:00:17 +01:00