Qortal-Nextcloud-Integration/README.md

# Qortal Nextcloud Integration

This repository bootstraps a local development setup for integrating Qortal identity/auth flows with Nextcloud.

## Current scope

1. Local Nextcloud development stack via Docker Compose.
2. A TypeScript broker service for:
   - Nextcloud provisioning bridge (create/link users).
   - OIDC authorization-code flow (`/authorize`, `/token`, `/userinfo`, `/jwks`).
3. Nextcloud app for:
   - admin setup + onboarding workflows,
   - user self‑service wallet import/linking,
   - Q‑Apps gateway proxy + embedded launcher,
   - Qortal Account dashboard (live auth validation, purchases, CHD announcements, activity notifications),
   - CHD billing sync + checkout integration,
   - branded PWA install surface for cloud-wide install flows.
4. Bridge apps for:
   - Files -> QDN publish/delete flows from Nextcloud Files,
   - Talk <-> Qortal mapping and relay controls (including cron-driven Qortal -> Talk relay),
   - instance-wide branded PWA installer/device registration hooks.
5. Initial architecture and phased implementation notes.

## Repo layout

- `docker-compose.yml`: MySQL/MariaDB local stack.
- `docker-compose.postgres.yml`: PostgreSQL local stack.
- `.env.example`: environment variables for local stack.
- `.env.postgres.example`: environment variables for PostgreSQL stack.
- `Makefile`: helper commands for startup and Nextcloud `occ` tasks.
- `docs/concept.md`: phase plan, contracts, and milestones.
- `docs/external-auth-integration.md`: Qortal External Auth hookup and commands.
- `docs/master-goals.md`: full goals, architecture boundaries, and delivery phases.
- `docs/nextcloud-admin-app.md`: usage for the custom Nextcloud admin app.
- `docs/nextcloud-app-roadmap.md`: roadmap for the custom Nextcloud app.
- `docs/nextcloud-oidc-setup.md`: working `user_oidc` configuration + policy notes.
- `docs/nuqloud-scrum.md`: NuQloud Scrum setup, per-conversation enablement, and v1 usage.
- `docs/nextcloud-vm-setup.md`: production-style setup for Nextcloud VM.
- `docs/devprod-ssl.md`: HTTPS-enabled dev-prod Docker stack.
- `docs/features-list.md`: full current feature inventory.
- `docs/plugin-roadmap.md`: current NuQloud for Nextcloud package/add-on inventory and launch-prep summary.
- `docs/settings-matrix.md`: admin/personal settings behavior matrix.
- `docs/scripts-reference.md`: script usage and recommended execution flows.
- `docs/testing-guide.md`: end-to-end onboarding test paths and expected results.
- `nextcloud/custom_apps/qortal_integration`: custom admin app for Qortal management inside Nextcloud.
- `nextcloud/custom_apps/qortal_files_bridge`: Files bridge add-on for QDN publishing.
- `nextcloud/custom_apps/qortal_talk_bridge`: Talk bridge add-on for mapping/relay controls.
- `nextcloud/custom_apps/nuqloud_scrum`: Talk-first scrum workspace automation and Deck/files mirror layer.
- `nextcloud/custom_apps/custom_pwa`: branded install/PWA add-on.
- `nextcloud/custom_apps/chd_admin`: CHD-only operational/admin app (not customer-facing).
- `nextcloud/custom_apps/user_oidc`: bundled upstream OIDC app used by the overall integration stack.
- `nextcloud/html`: bind-mounted Nextcloud web root for Docker stacks.
- `nextcloud/data`: bind-mounted Nextcloud data directory for Docker stacks.
- `services/qortal-oidc-broker`: Node/TypeScript broker + provisioning bridge starter.
- `external-auth/data`: bind-mounted data for bundled External Auth container.
- `qortal/data`: bind-mounted data for bundled Qortal node container.

## Quickstart

1. Run the guided dev setup:
   - `./start-dev.sh`
2. Follow logs until Nextcloud is installed:
   - `make logs`
3. Open Nextcloud:
   - `http://localhost:8080`
4. Install the OIDC app in Nextcloud:
   - `make install-oidc`
5. Enable the Qortal integration app:
   - `make occ cmd="app:enable qortal_integration"`

Runtime/version note:

- Docker stacks default to `NEXTCLOUD_IMAGE=nextcloud:34-apache`.
- Tested image majors are 33 and 34.
- Guided installers prompt for image selection (34 default, 33 tested fallback, or custom tag with warning).
- One-click/non-interactive flows keep the default unless explicitly overridden.
- Upgrade intentionally with:
  - `./upgrade-devprod-nextcloud.sh --nc34`
  - or pinned patch tag: `./upgrade-devprod-nextcloud.sh --nextcloud-image nextcloud:34.0.0-apache`

Dev‑prod (HTTPS) quickstart:

1. Run guided dev‑prod setup:
   - `./start-devprod.sh`
2. Rebuild/recreate after env changes:
   - `./recreate-devprod.sh`
   - (this now disables/enables the app automatically)
3. Safe Nextcloud image upgrade (maintenance mode + backup + occ upgrade):
   - `./upgrade-devprod-nextcloud.sh --nc34`

Production-oriented Docker bootstrap:

1. Run installer baseline:
   - `./scripts/install-production-docker.sh --mode nossl --with-external-auth`
   - Optional feature flags:
     - `--with-office`
     - `--with-signaling` (enables signaling + talk + turn + janus + recording)
     - `--with-talk-signaling` (Talk HPB only; use `--with-turn`/`--with-janus` to add media components)
     - `--with-recording` / `--without-recording` (override recording backend profile)
   - Optional explicit domains:
     - `--nextcloud-domain cloud.example.com`
     - `--broker-domain broker.example.com`
     - `--office-domain office.example.com`
     - `--signaling-domain signal.example.com`
2. Fully automatic one-click bootstrap:
   - `./scripts/install-production-docker.sh --one-click --mode nossl`
3. Verify services:
   - `docker compose -f docker-compose.devprod.nossl.yml --env-file .env.devprod ps`

PostgreSQL option:

1. (Optional) copy PG env file:
   - `cp .env.postgres.example .env.postgres`
2. Start PostgreSQL stack:
   - `make up-pg`
3. Stop PostgreSQL stack:
   - `make down-pg`

## Useful commands

- MySQL env behavior: `Makefile` uses `.env` if present, otherwise `.env.example`.
- PostgreSQL env behavior: `Makefile` uses `.env.postgres` if present, otherwise `.env.postgres.example`.
- Start: `make up`
- Stop: `make down`
- Logs: `make logs`
- List services: `make ps`
- Run Nextcloud `occ`: `make occ cmd="app:list"`
- Start (PostgreSQL): `make up-pg`
- Stop (PostgreSQL): `make down-pg`
- Logs (PostgreSQL): `make logs-pg`
- List services (PostgreSQL): `make ps-pg`
- Run Nextcloud `occ` (PostgreSQL): `make occ-pg cmd="app:list"`
- Broker logs: `docker compose logs -f broker`

## Qortal External Auth hookup

The broker now supports authenticated calls to your External Auth API.

1. Register an app in your external auth API (or reuse an existing app):
   - `POST /apps/register`
2. Set these in `.env` (or `.env.postgres`):
   - `BROKER_INTERNAL_API_TOKEN` (required for broker internal APIs)
   - `BROKER_CORS_ALLOWED_ORIGINS` (recommended: your Nextcloud public URL)
   - `QORTAL_EXTERNAL_AUTH_BASE_URL` (for Docker Desktop: usually `http://gateway.docker.internal:3191`)
   - `QORTAL_EXTERNAL_AUTH_APP_ID`
   - `QORTAL_EXTERNAL_AUTH_APP_SECRET`
   - If your node requires API keys, also set:
     - `QORTAL_AUTH_NODE_API_KEY`
     - `QORTAL_AUTH_NODE_API_KEY_MODE`
     - `QORTAL_AUTH_NODE_API_KEY_PATHS`
3. Restart broker:
   - `make up`

Useful broker endpoints:

- `GET /api/qortal/health` (connectivity check)
- `GET /api/qortal/wallets` (wallets accessible to configured app creds)
- `POST /api/qortal/wallets` (create wallet via configured app creds)
- `GET /api/oidc/allowlist` (list allowlisted Qortal addresses for auto-provision)
- `POST /api/oidc/allowlist` (add allowlisted Qortal address)
- `POST /api/oidc/allowlist/remove` (remove allowlisted Qortal address)
- `GET /api/oidc/invites` (list invite tokens)
- `POST /api/oidc/invites` (create invite token)
- `POST /api/oidc/invites/revoke` (revoke invite token)
- `POST /api/provision/upsert-from-wallet` (provision/link via walletId -> address0)
- `POST /api/provision/import-seed-link` (import existing seed + link to Nextcloud user)
- `POST /api/provision/import-backup-link` (import encrypted backup JSON + link to Nextcloud user)
- `POST /api/provision/unlink` (remove mapping)
- `GET /api/provision/mappings/by-nextcloud/:nextcloudUserId` (list mappings for one Nextcloud user)
- `GET /api/provision/mappings` (list persisted mappings)

Internal API security:

- Broker internal APIs under `/api/qortal/*`, `/api/provision/*`, and `/api/oidc/*` require `BROKER_INTERNAL_API_TOKEN`.
- Nextcloud app sends this token automatically when configured via:
  - Env in app container: `QORTAL_BROKER_INTERNAL_API_TOKEN` (recommended for docker stacks), or
  - Nextcloud Admin setting: `Broker Internal API Token`.
- In bundled devprod stacks, helper scripts auto-generate the token in `.env.devprod`.

Custom Nextcloud app:

- Open Nextcloud Admin settings and go to `Qortal Integration` section.
- Configure broker URL and test connectivity directly from Nextcloud UI.
- Run setup actions from UI:
  - wallet create/list
  - mapping link/list
  - allowlist and invite management for guarded auto-provision
- User self-service panel in Personal Settings:
  - import existing wallet by seed or backup JSON + link to current user
  - remove linked accounts
  - list linked accounts for current user
- Qortal Account dashboard:
  - `/apps/qortal_integration/account` (live auth validation)
- Q-Apps launcher:
  - `/apps/qortal_integration/qapps` (embedded gateway)
- Branded PWA installer:
  - `/apps/custom_pwa/install` (cloud-wide themed install page + custom manifest)
- CHD admin app (early-stage admin operations surface):
  - `/apps/chd_admin/admin`
- Files bridge:
  - `/apps/qortal_files_bridge/` (status + quick links)
  - primary publish workflow lives in Files actions/details panels
- Talk bridge:
  - `/apps/qortal_talk_bridge/admin` (admin relay + mapping controls)
  - personal settings panel for moderator-level mapping management
  - Talk conversation sidebar embed for in-context mapping/relay controls
- NuQloud Scrum:
  - `/apps/nuqloud_scrum/` (workspace index)
  - admin settings panel for Talk workspace sync configuration

Plugin roadmap:

- `docs/plugin-roadmap.md`

Broker persistence:

- Broker mappings are now stored in PostgreSQL (`identity_mapping` table).
- Configure via `BROKER_DATABASE_URL` and broker DB env vars in `.env`.

## Bundled Qortal node

All compose stacks now include a `qortal_node` container by default:

- Build context defaults to `QORTAL_NODE_CONTEXT=../qortal`
- Install/rebuild scripts use the primary upstream repo for this context:
  - `./scripts/install-production-docker.sh` ensures the context is cloned from `https://github.com/Qortal/qortal.git`
  - `./recreate-devprod.sh --full` refreshes the local clone to latest upstream before rebuild
- Node API is published to host via:
  - `QORTAL_NODE_API_BIND_HOST` / `QORTAL_NODE_API_HOST_PORT` (default `127.0.0.1:12391`)
- Node P2P is published to host via:
  - `QORTAL_NODE_P2P_BIND_HOST` / `QORTAL_NODE_P2P_HOST_PORT` (default `0.0.0.0:12392`)
- Node QDN data is published to host via:
  - `QORTAL_NODE_QDN_BIND_HOST` / `QORTAL_NODE_QDN_HOST_PORT` (default `0.0.0.0:12394`)
- External Auth defaults to use the internal node URL:
  - `QORTAL_AUTH_NODE_URL=http://qortal_node:12391`
  - `QORTAL_AUTH_NODE_API_KEY` (set this when the node requires `X-API-KEY`)
  - `QORTAL_AUTH_NODE_API_KEY_MODE=paths`
  - `QORTAL_AUTH_NODE_API_KEY_PATHS=/` (send key header for all node API calls)
- Talk/signaling env naming:
  - `SIGNALING_DOMAIN` is the canonical domain variable.
  - `TALK_DOMAIN` is retained as a legacy alias for compatibility.
  - `TALK_SIGNALING_PUBLIC_URL` should resolve to the same public host as `SIGNALING_DOMAIN`.
- Default node settings template:
  - `deploy/templates/qortal/default-node-settings.json`
  - `scripts/ensure-qortal-settings.sh` initializes `qortal/data/settings.json` from this template when missing.
  - It also ensures `autoUpdateEnabled` is present and set to `false` for Docker node stability.
- Default JVM start args:
  - `scripts/ensure-qortal-start-args.sh` initializes `qortal/data/start-arguments.txt` when missing and refreshes auto-managed files on restart.
  - Container startup reads that file on every restart, so you can tune memory without reinstall.
  - Auto-sizing target: roughly 66% of available RAM, clamped to about 4-8 GiB on hosts with at least 6 GiB RAM.
  - Optional override for first-run/manual sizing: `QORTAL_JVM_MEMORY_ARGS`

When using `./start-dev.sh`, `./start-devprod.sh`, or `./recreate-devprod.sh`,
the script auto-selects `QORTAL_NODE_API_HOST_PORT` in this order:

1. `12391`
2. `22391`, `32391`, `42391`, `52391`
3. `12291`, `22291`, `32291`, `42291`, `52291`

Then it sets:

- `QORTAL_NODE_P2P_HOST_PORT` to `QORTAL_NODE_API_HOST_PORT + 1`
- `QORTAL_NODE_QDN_HOST_PORT` to `QORTAL_NODE_API_HOST_PORT + 3`

## Qortal reverse proxy templates

Installer-ready templates for exposing a local node as both API/render and gateway:

- `deploy/templates/proxy/nginx-qortal-node.conf.template`
- `deploy/templates/proxy/apache-qortal-node.conf.template`
- `deploy/templates/proxy/README.md`

## Smoke test

Create/link a Nextcloud user from the broker:

```bash
curl -sS http://localhost:3000/api/provision/upsert \
  -H "X-Broker-Internal-Token: ${BROKER_INTERNAL_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "qortalAddress": "QTESTADDRESS1234567890ABCDEFG",
    "displayName": "Qortal Test",
    "email": "qortal-test@example.com"
  }'
```

Expected first-call behavior:

- `created: true` with a new mapping created.
- If Nextcloud user creation is needed, broker uses an internal random password that is never returned.

Expected repeated-call behavior:

- `created: false` with same `nextcloudUserId` mapping.

Provision/link from wallet (uses Qortal External Auth):

```bash
curl -sS http://localhost:3000/api/provision/upsert-from-wallet \
  -H "X-Broker-Internal-Token: ${BROKER_INTERNAL_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "walletId": "YOUR_WALLET_ID",
    "displayName": "Qortal Wallet User",
    "email": "wallet-user@example.com"
  }'
```

## Next step focus

- Add admin onboarding campaign APIs + audit events.
- Add Nextcloud app UI for mapping management and campaign triggering.
- Add stricter signed-nonce verification once External Auth exposes a dedicated verify endpoint.
- Add Q-App launch policy + signed launch token flow.