- "NOT_STARTED" is now "DOWNLOADED"
- "DOWNLOADING" is now "MISSING_DATA"
- Removed "DOWNLOAD_FAILED"
Some of these could be reintroduced once the system is able to support them.
These can be used to check the current status of a resource. The different statuses are:
NOT_STARTED,
DOWNLOADING
DOWNLOADED
BUILDING
READY
DOWNLOAD_FAILED
BUILD_FAILED
UNSUPPORTED
Not all statuses are returned yet. The build process needs more functionality to be able to support DOWNLOADED and DOWNLOAD_FAILED. Also, BUILDING and BUILD_FAILED are currently unable to distinguish between different resources with the same registered name, so need some attention.
Each service supports basic validation params, plus has the option for an entirely custom validation function.
Initial validation settings:
- IMAGE must be less than 10MiB
- THUMBNAIL must be less than 500KiB
- METADATA must be less than 10KiB and must contain JSON keys "title", "description", and "tags"
This is needed to avoid triggering a CORS preflight (which occurs when using an X-API-KEY header). The core isn't currently capable of responding to a preflight and the UI therefore blocks the entire request. See: https://stackoverflow.com/a/43881141
This allows users to set only their data path, and for the temp folder to automatically follow it. The temp folder can be moved to a custom location by setting the "tempDataPath" setting.
An API key is now _required_ for sensitive API calls that would previously have allowed local loopback authentication.
Previously, a request would have been considered authenticated if it originated from the same machine, however this creates a security issue when running third party code (particularly javascript) via the data network.
The solution is to now require an API key to authenticate sensitive API calls no matter where the request originates from.
It works as follows:
- When the core is first installed, it has no API key generated and will block sensitive calls until generated.
- A new POST /admin/apikey/generate API endpoint has been added, which can be used the generate an API key for a newly installed node. The UI will ultimately call this automatically.
- This API returns the generated key so that it can be stored by the requesting app (most likely the UI).
- From then on, the generate API requires authentication via the existing API key in order to regenerate a key. It can be used as a security measure if the existing key is compromised.
- The API key must be passed to all sensitive API endpoints from then on, even when calling it from the same local machine.
- If the core already has a legacy API key specified via the 'apiKey' setting, this will be automatically copied to the new format so that a new one doesn't need to be generated.
- The API key itself is stored in a flat file in the qortal directory (the path can be customized using the `apiKeyPath` setting). Deleting this file and restarting the core will allow a new one to be regenerated.
The process of serving resources to a browser will likely be needed for more than just websites (e.g. it will be needed for apps too) so it makes sense to abstract it to its own class.
This should help keep the peer lookup table size down, as there is no need to locate files for transactions that existed before the most recent PUT transaction.
Built resources are deleted when either:
- The resource reaches the expiry interval specified in the builtDataExpiryInterval setting (default 30 days)
- The resource is published by a name that is in the local blacklist
Resources only exist in the reader cache once they have been viewed, to remove the loading time on subsequent views. But some may prefer to reduce this expiry time (at the expense of longer load times and more CPU), as data is held unencrypted in the cache.
We still allow it to be fetched even if it's outside of the storage policy, as the cleanup manager will delete the files very soon after, and they won't be allowed to be served to other peers due to other checks already in place.
This allows other peers to find out where they can obtain these files if we were to stop hosting them later. Or even if we continue hosting copies, it still informs the network on other locations, for better decentralization.
We don't want the network being spammed when a file isn't available by any reachable peers. This feature ensures retries are spaced out over longer timeframes. Basic logic:
- Wait 5 minutes in between failed attempts
- After 5 failed attempts (i.e. 25 mins) only try once per day from then on
- A core restart resets the counters
The stats gathered here can also be used to inform the core of when it should attempt a direct connection with a peer to obtain the data. That part isn't implemented yet.
