Commit Graph

1378 Commits

Author SHA1 Message Date
CalDescent
ebfa941a4f Fixed some file separators. 2021-06-26 10:27:32 +01:00
CalDescent
47c70eea9e Use system temp directory instead of making a "temp" subfolder when zipping files. 2021-06-26 10:11:34 +01:00
CalDescent
fe7c40cb7c Reduced log spam. 2021-06-25 19:31:28 +01:00
CalDescent
8973626a4b Fixed issue with temp directory on Linux. 2021-06-25 19:31:13 +01:00
CalDescent
ace5d999e2 Log a comma separated list of hashes after splitting a file into chunks, so they can easily be requested from another node using the //data/files/frompeer/{peer} API endpoint. Again temporary until the sync happens automatically. 2021-06-25 19:30:45 +01:00
CalDescent
52829a244b More work on APIs to request files from peers. I won't spend too long making these perfect as they are mostly a temporary measure. 2021-06-25 19:28:39 +01:00
CalDescent
71c247fe56 Added POST /data/file/{hash}/build API, used to join multiple chunks together. 2021-06-25 19:28:01 +01:00
CalDescent
b34066f579 More work on HTML parsing.
The style tag parsing ideally needs rewriting using an actual CSS parser, but we can get away with this hacky approach in the short term.
2021-06-25 08:34:44 +01:00
CalDescent
b286c15c51 Optimized website serving, and added code to return the correct content types.
This is probably the most efficient way to process the data on the fly, but it's still not very scalable. A better approach would be to pre-process the HTML when building the file structure, and then serve them completely statically (i.e. using a standard webserver rather than via application memory). But it makes sense to keep it this way for development and maybe early beta testing.
2021-06-25 08:32:22 +01:00
Marco Moesman
aff4f6c859
Create TransactionValidity_nl.properties 2021-06-24 19:13:55 +02:00
Marco Moesman
1f8f73fa30
Create ApiError_nl.properties 2021-06-24 18:44:53 +02:00
Marco Moesman
620d6624a9
Create SysTray_nl.properties 2021-06-24 18:30:50 +02:00
JaymenChou
287f42ae64
Update and rename SysTray_zh.properties to SysTray_zh_SC.properties
Rename to zh_SC for better distinguish between zh_SC (Simple Chinese)and zh_TC(Traditional Chinese)
Rephrase some of the words for better understanding.
2021-06-24 11:42:48 +08:00
JaymenChou
d976c97d13
Create SysTray_zh_TC.properties 2021-06-24 11:31:46 +08:00
CalDescent
6d549b0754 Updated AdvancedInstaller project for v1.5.5 2021-06-23 19:57:03 +01:00
CalDescent
02dd64558f Bump version to 1.5.5 2021-06-23 18:21:55 +01:00
CalDescent
ea5e2f5580 Added POST /site/preview API
This can be used to preview a site before signing a transaction and announcing it to the network. The response will need reworking to return JSON (along with most of the other new APIs)
2021-06-23 09:28:38 +01:00
CalDescent
b65c7a75fe Handle relative links when parsing HTML. 2021-06-23 09:26:41 +01:00
CalDescent
39f5dce51c Moved "directory" data uploads to new POST /site/upload API.
Directory uploads don't make much sense outside of website hosting, so it's best to make this API specific to that purpose.
2021-06-23 09:10:06 +01:00
CalDescent
f77ec1faf6 A very crude proof of concept which serves a website from a zipped (and in future, chunked) data blob. This forms the beginnings of the "website hosting" layer on top of the data storage. It needs a significant rework - most importantly so that we aren't serving every asset from memory, and also so that the correct content-type headers are returned, etc. 2021-06-23 08:43:21 +01:00
CalDescent
cd3a1e0159 Increased max file size to 1GiB. Will review this again later. 2021-06-23 08:33:53 +01:00
CalDescent
3f20fadb81 When zipping files, rename the outer folder to "data" instead of using the original folder name. This means that the data can be accessed deterministically without the need to first lookup the folder name. 2021-06-23 08:09:59 +01:00
CalDescent
1c6428dd3b Added equivalent split and join test but this time using a 5.5MiB source file. 2021-06-23 08:07:35 +01:00
CalDescent
aca620241a More work on data file split/join, and added a test. 2021-06-22 08:58:16 +01:00
CalDescent
808b36e088 Specify chunk size when splitting. 2021-06-22 07:44:34 +01:00
CalDescent
1613375cc0 Added more validation of files received in GET /data/file/frompeer 2021-06-21 19:03:34 +01:00
CalDescent
787ef957d2 Added support for uploading an entire directory via POST /data/upload/path
If a directory is specified instead of a file, the directory is automatically zipped before being split into chunks.
2021-06-21 19:02:49 +01:00
CalDescent
b915d0aed5 Only create the output file directories when we are actually writing a file there. This should prevent empty directories being created when initializing a nonexistent DataFile using a hash. 2021-06-21 08:40:52 +01:00
CalDescent
16dc5b5327 Include the data length in DataFileMessage, which is more similar to the approach used by ArbitraryDataMessage. These two message types are very similar, except arbitrary code currently has a requirement of one piece of data per signature, whereas DataFile code is independent and can support multiple files per transaction. Maybe the two can be combined at some point, but for now I'll keep them separate. 2021-06-20 19:49:10 +01:00
CalDescent
d25e98d9c4 Include peer connection ID in recently created log message. 2021-06-20 10:26:14 +01:00
CalDescent
227cdc1ec8 Log each sync attempt when our blockchain isn't up to date
Without this it can appear as though nothing is happening for a while after the app launches.
2021-06-20 07:25:25 +01:00
CalDescent
c2d0c63db0 Improved error logging. 2021-06-19 20:31:04 +01:00
CalDescent
f5c9807a48 Use contains() rather than equals() when matching a peer in /data/file/frompeer, so that the port can be optionally left out. 2021-06-19 20:29:05 +01:00
CalDescent
7e9b1d5e16 Rework of DataFile.base58Digest()
This fixes an NPE when trying to send a file that doesn't exist. It also removes the caching, which we can add again later if it turns out to be needed.
2021-06-19 20:25:25 +01:00
CalDescent
5070c4eea9 Better handling of data file responses in the /data/file/frompeer API endpoint. 2021-06-19 20:23:33 +01:00
CalDescent
33d9c51b6f Validate supplied base58 string in /data/file/frompeer API endpoint 2021-06-19 19:26:13 +01:00
CalDescent
d0f9d478c2 Fixed bug which prevented the DATA_FILE message ID from making it through to the reply queue. 2021-06-19 19:05:11 +01:00
CalDescent
f296ec46c8 Removed unused headers. 2021-06-19 17:32:55 +01:00
CalDescent
64d19e480b Chunk size set to 1MB for now, as it seems that our networking code has problems when transferring 2MB chunks. We can increase this later once that problem has been fixed. 2021-06-19 17:32:48 +01:00
CalDescent
2c585a9328 Upper connection time limit reduced from 60 mins to 20 mins.
Now that we aren't disconnecting mid sync, we can get away with more frequent disconnections. This brings the average connection length to around 9 mins.
2021-06-19 17:25:50 +01:00
CalDescent
45b0d9e19b Added more initial peers, submitted by CWDSYSTEMS 2021-06-19 16:19:53 +01:00
CalDescent
026a4b896c Added BTC and LTC electrum nodes submitted by CWDSYSTEMS 2021-06-19 14:50:18 +01:00
CalDescent
78237fcd11 Don't intentionally disconnect peers if we are currently syncing with them. 2021-06-19 13:12:16 +01:00
CalDescent
73cc3dcb92 Force a disconnect of each peer when the connection age reaches the maximum allowed time.
Connection limits are defined in settings (denominated in seconds):
"minPeerConnectionTime": 120,
"maxPeerConnectionTime": 3600

Peers will disconnect after a randomly chosen amount of time between the min and the max. The default range is 2 minutes to 1 hour, as above.

This encourages nodes to connect to a wider range of peers across the course of each day, rather than staying connected to an "island" of peers for an extended period of time. Hopefully this will reduce the amount of parallel chains that can form due to permanently connected clusters of peers.

We may find that we need to reduce the defaults to get optimal results, however it is best to do this incrementally, with the option for reducing further via each node's settings. Being too aggressive here could cause some of the earlier problems (e.g. 20% missing blocks minted) to reappear. We can re-evaluate this in the next version. Note that if defaults are reduced significantly, we may need to add code to prevent this from happening mid-sync. With higher defaults, this is less of an issue.

Thanks to @szisti for supplying some base code for this commit, and also to @CWDSYSTEMS for diagnosing the original problem.
2021-06-19 13:03:46 +01:00
CalDescent
4cff03e7fe Include "size" value in the "Synchronized with peer" logs.
This indicates the size of the re-org/rollback that was required in order to perform this sync operation. It is only included if it's greater than 0 blocks.
2021-06-19 09:04:14 +01:00
CalDescent
8e35f131d5 Removed debugging log that wasn't intended to be committed. 2021-06-18 08:33:51 +01:00
CalDescent
aafb9d7e4f Include running total in "Sent X bytes" log entry. 2021-06-18 08:05:35 +01:00
CalDescent
652f30bdbd Added DATA_FILE and GET_DATA_FILE message types. 2021-06-18 08:02:57 +01:00
CalDescent
f4ba7b2a0c Added onNetworkGetDataFileMessage() handler 2021-06-18 08:02:13 +01:00
CalDescent
592490d709 Added GET /data/file/frompeer API endpoint
This requests a file from the supplied peer address, and stores a copy locally if successful. Still a work in progress.
2021-06-18 07:59:46 +01:00