Commit Graph

2036 Commits

Author SHA1 Message Date
CalDescent
f3e9dfe734 Return a 404 instead of a 500 if a resource is unavailable.
Could be improved in the future to return different codes depending on its status (e.g. doesn't exist = 404, 102 for loading, 500 for error, etc), but 404 makes the most sense until that has been developed
2021-11-27 16:43:29 +00:00
CalDescent
a7b31ab1f9 Small bug fixes 2021-11-27 16:40:12 +00:00
CalDescent
644ab27186 Updated wording 2021-11-27 14:22:09 +00:00
CalDescent
e90ecd2085 Adapted GET /arbitrary/resources endpoint to allow filtering by identifier
- If an identifier parameter is missing or empty, it will return an unfiltered list of all possible identifiers.
- If an identifier is specified, only resources with a matching identifier will be returned.
- If default is set to true, only resources without identifiers will be returned.
2021-11-27 14:21:36 +00:00
CalDescent
bc38184ebf Major rework of local data directory structure
Files are now keyed by signature, in the format:
data/si/gn/signature/hash

For times when there is no signature available (i.e. at the time of initial upload), files are keyed by hash, in the format:
data/_misc/ha/sh/hash

Files in the _misc folder are subsequently relocated to a path that is keyed by the resulting signature.

The end result is that chunks are now grouped on the filesystem by signature. This allows more transparency as to what is being hosted, and will also help simplify the reporting and management of local files.
2021-11-27 13:00:32 +00:00
CalDescent
d9de27e6f2 Updated AdvancedInstaller project for v2.1.0 2021-11-24 19:39:37 +00:00
CalDescent
6930bf0200 Bump version to 2.1.0 2021-11-24 18:36:40 +00:00
CalDescent
199833bdd4 Fixed issue with GET /crosschain/trades API endpoint where the minimumTimestamp parameter returned inconsistent results. 2021-11-24 15:21:08 +00:00
CalDescent
0dcd2e6e93 Fixed bug in GET ​/crosschain​/price​/{blockchain} inverse price API endpoint when the "inverse" parameter is null. 2021-11-24 15:09:45 +00:00
CalDescent
0dd43d5c9a Fixed bug in storage calculation 2021-11-24 14:22:57 +00:00
CalDescent
e879bd0fc5 Delete some random chunks when we reach the storage capacity
This should allow for a relatively even distribution of chunks, but there is a (currently unavoidable) risk of files with very few mirrors being deleted altogether.

Longer term this could be improved by checking that one of our peers has a file, before it's deleted locally
2021-11-24 14:15:22 +00:00
CalDescent
8bf7daff65 Track the storage capacity and the total data/temp directory sizes
Nodes will stop proactively storing new data when they reach 90% capacity.

A new "maxStorageCapacity" setting has been added to allow the user to optionally limit the allocated space for this node. Limits are approximate only, not exact.
2021-11-24 13:43:45 +00:00
CalDescent
ae0f01d326 Added storage policy unit tests 2021-11-24 11:02:54 +00:00
CalDescent
af8d0a3965 Separated computeNonce() from build() in the transaction builder.
This gives the option of the nonce to be computed elsewhere, such as in the UI, and also allows transaction unit tests to run much more quickly.
2021-11-24 11:02:17 +00:00
CalDescent
1b170c74c0 Modified storage code to support 2 new settings:
publicDataEnabled - whether to store decryptable data (default true)
privateDataEnabled - whether to store data without a decryption key (default false)
2021-11-24 09:38:18 +00:00
CalDescent
f6b9ff50c3 More loading screen improvements 2021-11-23 22:21:57 +00:00
CalDescent
9ef75ebcde Improved styling of loading panel 2021-11-23 21:15:45 +00:00
CalDescent
f76a618768 Display the latest status on the loading screen, updated via API calls on a timer 2021-11-23 20:53:09 +00:00
CalDescent
098d7baa4d Bump version to 2.1.0-prerelease.0 2021-11-23 19:03:27 +00:00
CalDescent
59a57d3d28 Increased frequency of automatic repository maintenance attempts
repositoryMaintenanceMinInterval reduced from 7 to 3 days
repositoryMaintenanceMaxInterval reduced from 30 to 14 days
2021-11-23 18:47:27 +00:00
CalDescent
cce95e09de Default min level for block submissions increased to 3
This doesn't affect minting rewards; it is simply a means of reducing block candidates. There should be no noticeable difference other than hopefully less re-orgs. We can ultimately do a hard fork and increase Blockchain.minAccountLevelToMint but this allows us to test the approach in a lower risk way.
2021-11-23 18:45:35 +00:00
CalDescent
ec48ebcd79 Improved resource statuses 2021-11-23 09:14:44 +00:00
CalDescent
908f80a15d Fixed bug when checking if all files exist locally in /arbitrary/status 2021-11-22 08:43:07 +00:00
CalDescent
02eab89d82 Fixed bug when trying to delete a file instead of a directory. 2021-11-21 19:24:20 +00:00
CalDescent
c588786a06 Added /base64 variation of POST /arbitrary/* APIs
This can be used to upload base64 encoded file data directly from the UI. Using base64 because base58 is unusably slow
2021-11-21 19:12:01 +00:00
CalDescent
b4f3105035 Added /render/authorize/{service}/{resourceId}* APIs
These allow the UI to pre-authorize a resource and therefore avoid having to pass a sensitive API key to a website or app.
2021-11-21 14:57:26 +00:00
CalDescent
d018f11877 Log the initial height of the block archiver on startup 2021-11-21 09:12:32 +00:00
CalDescent
d0000c6131 If "build=true" is specified in query string of GET /resource/status/{service}/{name}, build the resource before returning the status 2021-11-20 18:52:03 +00:00
CalDescent
c05ffefd7d Added a dynamic background to the loading screen. Still needs a lot of work. 2021-11-20 14:09:00 +00:00
CalDescent
530fc67a05 List available services in API docs when requesting arbitrary data 2021-11-20 11:46:03 +00:00
CalDescent
c79ec11b07 Fixed incorrect comment 2021-11-19 23:16:28 +00:00
CalDescent
668ef26056 Fixed major performance issue with GET /arbitrary/resources?includestatus=true
The missing data check was triggering decryptions, extractions, etc. Replaced with some code which checks for the presence of chunks on the local machine, without getting involved with any build process overhead.
2021-11-19 23:00:36 +00:00
CalDescent
75ec7723ef Improved accuracy of statuses
- "NOT_STARTED" is now "DOWNLOADED"
- "DOWNLOADING" is now "MISSING_DATA"
- Removed "DOWNLOAD_FAILED"

Some of these could be reintroduced once the system is able to support them.
2021-11-19 21:48:41 +00:00
CalDescent
73e609fa29 Replaced all IllegalStateException with DataException in arbitrary code
This was necessary to ensure that all exceptions are caught intentionally, as otherwise it creates endless amounts of edge cases.
2021-11-19 21:42:03 +00:00
CalDescent
8cb06bf451 Return statuses in GET /arbitrary/resources endpoint when the "includestatus" parameter is true. 2021-11-19 20:20:45 +00:00
CalDescent
1be8a059f4 Fixed bug caused by not catching a IllegalStateException
I may replace all IllegalStateExceptions with DataExceptions soon so we can ensure they are always caught.
2021-11-19 20:18:56 +00:00
CalDescent
7f41c7ab0e Added "BLACKLISTED" status for arbitrary data resources. 2021-11-19 20:18:00 +00:00
CalDescent
3860c5d8ec Fixed some failing tests. 2021-11-19 16:12:31 +00:00
CalDescent
a061a7cc4d Fixed various warnings raised by the IDE. 2021-11-19 16:11:37 +00:00
CalDescent
844501d6cd Added GET /arbitrary/resource/status/* API endpoints
These can be used to check the current status of a resource. The different statuses are:

NOT_STARTED,
DOWNLOADING
DOWNLOADED
BUILDING
READY
DOWNLOAD_FAILED
BUILD_FAILED
UNSUPPORTED

Not all statuses are returned yet. The build process needs more functionality to be able to support DOWNLOADED and DOWNLOAD_FAILED. Also, BUILDING and BUILD_FAILED are currently unable to distinguish between different resources with the same registered name, so need some attention.
2021-11-19 15:26:52 +00:00
CalDescent
020bd00b8f Removed incorrect @SecurityRequirement annotation 2021-11-19 13:40:02 +00:00
CalDescent
0706b0d287 Added some /site endpoints to the gateway, for backwards support of the demo sites 2021-11-19 13:26:47 +00:00
CalDescent
ce56cd2b16 Disallow local (loopback address) requests when using the gateway
This removes the possibility of some locally running javascript in a website or app requesting unvetted data via the local gateway.
2021-11-19 13:20:53 +00:00
CalDescent
b7a0a7eea4 Removed API authentication when using the gateway, as that would defeat the purpose of it. 2021-11-19 13:14:57 +00:00
CalDescent
824d14e793 Removed unnecessary check for isApiRestricted() when previewing.
The API key authentication will be enough to restrict requests.
2021-11-19 13:08:56 +00:00
CalDescent
83e0ed2b5d Reduced log spam 2021-11-19 12:59:25 +00:00
CalDescent
c8b70b51c3 Added gateway service, to allow websites to be served directly on a domain or IP.
This replaces the existing GET /site/{name} API endpoints.

Example settings:

"gatewayServiceEnabled": true,
"gatewayServicePort": 80

Websites can then be served using URL:

http://localhost/RegisteredName

Or, if node is behind public DNS:

http://example.com/RegisteredName

Or, if a custom port (such as 12393) is used:

http://localhost:12393/RegisteredName
http://example.com:12393/RegisteredName

This is currently for serving websites only, but can be adapted to serve other services if needed.
2021-11-19 12:59:15 +00:00
CalDescent
c0fedaa3a4 Attempt to request files directly from a peer if it isn't returned in the general network broadcast. 2021-11-19 12:05:40 +00:00
CalDescent
e74dcff010 Wait for 3 seconds between attempts to allow time for data to arrive. 2021-11-19 11:21:40 +00:00
CalDescent
3b5b45b463 Give up after 5 attempts to request data in the GET /arbitrary APIs 2021-11-19 11:21:14 +00:00