209 lines
12 KiB
Markdown
209 lines
12 KiB
Markdown
# NP Replica
|
||
|
||
This project provides a local Chinese web interface for the official NUPACK Analysis, Design, and Utilities workflows.
|
||
|
||
## v5.4 · NUPACK 4.1 mixed-material Design
|
||
|
||
The `v5.4-nupack4.1.0.1` image extends mixed-material support through the complete Design workflow:
|
||
|
||
- RNA/DNA and RNA/2′OMe-RNA Domain constraints using explicit `r`, `d`, `m`, and wildcard `w` prefixes;
|
||
- material-aware IUPAC parsing, fixed-domain reverse complements, and true nucleotide lengths;
|
||
- mixed-material Pattern, Similarity, Window, and Library sequence inputs;
|
||
- a mixed-segment preset, dynamic syntax guidance, and client-side validation in the Design editor;
|
||
- material prefixes preserved in designed Domains, strands, reproducibility exports, and Analysis handoff;
|
||
- Docker version metadata moved after dependency layers so version-only rebuilds reuse the expensive NUPACK/ViennaRNA layers.
|
||
|
||
## v5.3 · NUPACK 4.1 mixed-material analysis
|
||
|
||
The `v5.3-nupack4.1.0.1` image adds:
|
||
|
||
- material-aware salt defaults and server-side physical-range validation;
|
||
- RNA/DNA (`rna-dna06`) and RNA/2′OMe-RNA (`rna-merna06`) Analysis and Utilities inputs using `r`, `d`, and `m` prefixes;
|
||
- NUPACK version and resolved parameter-set provenance in results;
|
||
- `indistinguishable_search` and `max_subopt_count` controls for MFE/Subopt;
|
||
- per-complex analysis bonus support.
|
||
|
||
|
||
## Run
|
||
|
||
```bash
|
||
docker compose up --build -d
|
||
```
|
||
|
||
Open `http://127.0.0.1:18765`.
|
||
|
||
Application pages:
|
||
|
||
- `/`: operational overview
|
||
- `/workspace`: calculation workspace
|
||
- `/cloud`: account jobs and shares
|
||
- `/account`: identity and usage
|
||
|
||
The system design and deployment boundaries are documented in
|
||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
|
||
|
||
## Reverse Proxy
|
||
|
||
The service listens on port `18765`. The current local deployment is reverse proxied as `npt.lihato.icu`.
|
||
|
||
Traefik only performs TLS termination, compression, and reverse proxying for `npt`; authentication is enforced by this application.
|
||
|
||
## Deploying the independent np instance
|
||
|
||
`np.lihato.icu` runs independently on `100.64.0.11`. Do not copy the `npt`
|
||
OIDC values unchanged. In the `np-replica` service of the
|
||
`docker-compose.yml` deployed on `.11`, use:
|
||
|
||
```yaml
|
||
ports:
|
||
- "100.64.0.11:18765:18765"
|
||
environment:
|
||
NP_AUTH_REQUIRED: 1
|
||
NP_OIDC_ISSUER: https://auth.lihato.icu/application/o/nupack-account/
|
||
NP_OIDC_CLIENT_ID: np-replica-web
|
||
NP_OIDC_REDIRECT_URI: https://np.lihato.icu/auth/callback
|
||
NP_OIDC_POST_LOGOUT_URI: https://np.lihato.icu/
|
||
```
|
||
|
||
The `NP_OIDC_*` variables belong on the web service (`np-replica`); the
|
||
worker does not perform browser login. Preserve the calculation settings on
|
||
both application containers. The recommended portable defaults are
|
||
`NP_WORKER_CONCURRENCY=auto`, `NP_PER_JOB_THREAD_LIMIT=4`,
|
||
`NP_WORKER_MEMORY_RESERVE_GB=4.0`, `NP_ESTIMATED_JOB_MEMORY_GB=9.0`,
|
||
and `NP_NUPACK_CACHE_GB=8.0`.
|
||
|
||
After editing the Compose file on `.11`, recreate the web container:
|
||
|
||
```bash
|
||
docker compose up -d --force-recreate np-replica
|
||
```
|
||
|
||
If the deployed image predates the account/OIDC changes, rebuild both
|
||
application containers instead:
|
||
|
||
```bash
|
||
docker compose up -d --build --force-recreate np-replica np-worker
|
||
```
|
||
|
||
Verify the result:
|
||
|
||
```bash
|
||
curl -I https://np.lihato.icu/
|
||
curl -I 'https://np.lihato.icu/auth/login?next=%2F'
|
||
```
|
||
|
||
The first response should redirect to `/auth/login?next=%2F`. The second
|
||
`Location` header must contain all of the following:
|
||
|
||
- `client_id=np-replica-web`
|
||
- `redirect_uri=https%3A%2F%2Fnp.lihato.icu%2Fauth%2Fcallback`
|
||
- `code_challenge_method=S256`
|
||
|
||
If it contains `npt-replica-web` or `npt.lihato.icu`, the `.11` container
|
||
is still running with the wrong environment and must be recreated.
|
||
|
||
## 容器资源配置
|
||
|
||
当前 `docker-compose.yml` 可随 CPU 和容器内存限制自动调整 worker 并发，服务拆成三个容器：
|
||
|
||
- `redis`：只保存实时队列、运行状态和会话缓存，限制为 `mem_limit: 3g`。Redis 开启 AOF，数据写入 `./runtime/redis`。
|
||
- `np-replica`：Web/API 容器，限制为 `mem_limit: 4g`，负责页面、登录、历史记录、任务提交和状态查询。它不应该承担大规模计算。
|
||
- `np-worker`：后台计算容器，限制为 `mem_limit: 56g`，负责实际 NUPACK 计算。大任务应该由这个容器消耗 CPU 和内存。
|
||
|
||
计算相关环境变量需要在 `np-replica` 和 `np-worker` 中保持一致：
|
||
|
||
- `NP_WORKER_CONCURRENCY=auto`：根据容器可用 CPU、每任务线程数、内存限制、预留内存和估算的单任务内存，自动选择同时运行的任务数。也可填正整数进行手动覆盖。
|
||
- `NP_PER_JOB_THREAD_LIMIT=4`：单个任务最多使用 4 个 native 计算线程，同时写入 `OMP_NUM_THREADS`、`OPENBLAS_NUM_THREADS`、`MKL_NUM_THREADS`、`NUMEXPR_NUM_THREADS`、`VECLIB_MAXIMUM_THREADS`、`GOTO_NUM_THREADS`，并设置 `nupack.config.threads`。
|
||
- `NP_NUPACK_CACHE_GB=8.0`：单个 NUPACK 进程可使用的缓存上限。
|
||
- `NP_WORKER_MEMORY_GB=0`：为 0 时自动读取 cgroup 内存限制或系统内存；在特殊运行环境中可填实际可用 GB 数覆盖检测值。
|
||
- `NP_WORKER_MEMORY_RESERVE_GB=4.0`：不给计算任务使用的内存，留给系统、Redis、Web 服务和运行时开销。
|
||
- `NP_ESTIMATED_JOB_MEMORY_GB=9.0`：调度器用于计算并发的单任务内存预算。根据其他设备上观测到的峰值灵活调整；它不限制单任务实际内存。
|
||
- `NP_JOB_TTL_SECONDS=3600`：Redis 中实时任务状态的保留时间；长期历史记录写入 SQLite。
|
||
|
||
`auto` 取 CPU 容量 `有效 CPU 数 / 单任务线程数` 与内存容量
|
||
`(worker 内存 - 预留内存) / 单任务内存估算` 中较小者。CPU 检测会考虑
|
||
进程 affinity 和 cgroup quota，内存检测会考虑 cgroup limit，因此同一份配置可迁移到
|
||
不同设备。若出现 OOM 或交换分区抖动，提高 `NP_ESTIMATED_JOB_MEMORY_GB` 或直接降低
|
||
`NP_WORKER_CONCURRENCY`；空闲资源很多时则可谨慎反向调整。`/health` 会返回 worker
|
||
最终采用的资源计划。
|
||
|
||
增加线程不会保证单个 design 等比例提速：NUPACK 的部分设计阶段实际上主要使用一个
|
||
核心，线程更适合底层可并行计算；同时运行多个独立任务通常更容易利用多核。增加内存也
|
||
不会直接缩短搜索，只能减少 OOM、换页和缓存压力。当前 NUPACK 4.1.0.1 wheel 只暴露 CPU
|
||
线程配置，不能通过环境变量直接调用 GPU；CUDA 加速需要重新构建或替换 NUPACK 核心
|
||
wheel，并重新做数值一致性验证。
|
||
|
||
设计任务默认使用官方行为：固定序列 target 也参与 `tube_design` / `complex_design`。如果确认某些 target 完全固定、只需要展示结果、不需要参与优化，可在页面中把 “固定目标策略” 改为 “剥离固定目标以提速”，这样会减少 off-target 集合和优化搜索量。
|
||
|
||
## Endpoints
|
||
|
||
- `GET /`
|
||
- `GET /health`
|
||
- `GET /api/example`
|
||
- `POST /api/analyze`
|
||
- `POST /api/jobs`
|
||
- `GET /api/jobs/<job_id>`
|
||
|
||
Submitting an identical active payload for the same account returns
|
||
`409 duplicate`. The Workspace asks the user whether to cancel or explicitly
|
||
submit another calculation through `POST /api/jobs?force=1`. Fingerprints are
|
||
account-scoped, so different users are never deduplicated against one another.
|
||
|
||
## Admin Login
|
||
|
||
Admin uses a dedicated password and does not require OIDC. Generate a strong
|
||
password and store it in the ignored `.env`:
|
||
|
||
```bash
|
||
openssl rand -hex 24
|
||
```
|
||
|
||
```dotenv
|
||
NP_ADMIN_TOKEN=replace-with-the-generated-password
|
||
```
|
||
|
||
After deployment, open `https://<host>/admin` from the Account page or directly.
|
||
The password is submitted in a JSON `POST /api/admin/login` request and never
|
||
appears in the URL. A successful login sets a random, short-lived HttpOnly
|
||
session cookie and redirects to `/admin/panel`.
|
||
|
||
Admin can query all task metadata, stop active jobs, move terminal jobs to the
|
||
trash, restore them, permanently delete trashed jobs, and configure trash
|
||
retention (two days by default). Normal account deletion is a soft delete and
|
||
also disables that job's public shares. Set `NP_ADMIN_SESSION_TTL_SECONDS` to
|
||
control session duration and keep `NP_ADMIN_COOKIE_SECURE=1` on HTTPS deployments.
|
||
|
||
|
||
## Persistent Shares
|
||
|
||
Live jobs and sessions are stored in Redis when `NP_REDIS_URL` is enabled. Account history and share links are stored in `./runtime/account/np-replica.sqlite3`; back up both runtime directories when moving the deployment.
|
||
|
||
## Notes
|
||
|
||
- `tube` mode supports strand concentrations and the official `SetSpec(max_size=..., include=..., exclude=...)` semantics.
|
||
- `complex` mode supports explicit complex definitions such as `A+B`.
|
||
- Pair matrices are returned as a preview block to keep responses manageable.
|
||
- Tube `pairs` now runs inside the same official `tube_analysis` call as the other selected calculations, including `ensemble_pair_fractions` and `fraction_bases_unpaired`.
|
||
- `result_limit` limits display and structure plotting only; it does not silently reduce the Tube calculation ensemble.
|
||
- Analysis and design results include phase timings and workload diagnostics to explain long-running jobs without changing thermodynamic results.
|
||
- Design target tubes support per-tube `SetSpec(max_size/include/exclude)`, target-complex bonuses, weighted-objective trial selection, and multi-tube result tables.
|
||
- Utilities directly exposes the official `pfunc`, `structure_energy`, `structure_probability`, `sample`, `pairs`, `mfe`, `subopt`, `ensemble_size`, `des`, `defect`, `seq_distance`, and `struc_distance` calls.
|
||
- Active calculations write only small stage-transition records. The result panel performs one status request per manual refresh and does not poll or serialize NUPACK intermediate results.
|
||
- Identical active submissions from the same account reuse the existing job instead of consuming another worker slot.
|
||
- The Docker image compiles ViennaRNA `RNAplot` from `rna/ViennaRNA-2.7.2.tar.gz` and embeds SVG structure plots into MFE results.
|
||
- Multistrand MFE plots now use a split-strand layout by default when a structure contains `+`, with `RNAplot` retained as the fallback path.
|
||
- The web UI supports Chinese and English switching, account-scoped cloud history, result export, share controls, and manually refreshed async job status.
|
||
- Docker now starts three services: `np-replica`, `np-worker`, and `redis`.
|
||
- Async jobs use Redis-backed storage and queueing when `NP_REDIS_URL` is configured; otherwise the app falls back to the older in-memory mode.
|
||
- The browser-facing API and page behavior stay the same after enabling Redis-backed jobs.
|
||
- Worker concurrency is controlled by `NP_WORKER_CONCURRENCY` and defaults to `auto`; a positive integer remains available as an explicit override.
|
||
- Per-job native thread usage is capped by `NP_PER_JOB_THREAD_LIMIT` and defaults to `1` to prevent a single analysis from monopolizing the machine.
|
||
- `NP_STRUCTURE_PLOT_MODE` supports `auto` (default), `split`, and `rnaplot`.
|
||
- Static pages are served with ETag-based browser caching; JSON API responses remain uncached.
|
||
- The force-directed structure viewer lazy-loads D3 only when that view is opened, so the initial page load is not blocked by the external CDN.
|
||
- `/health` uses Redis queue length plus a running-job set instead of scanning every historical job on each poll.
|
||
- Application login uses the public Authentik issuer at `https://auth.lihato.icu/` with Authorization Code + PKCE.
|
||
- The current `npt` instance uses issuer `/application/o/nupack-account-npt/`, client `npt-replica-web`, callback `https://npt.lihato.icu/auth/callback`, and logout return `https://npt.lihato.icu/`.
|
||
- For the independent `np` deployment on `100.64.0.11`, set `NP_OIDC_ISSUER=https://auth.lihato.icu/application/o/nupack-account/`, `NP_OIDC_CLIENT_ID=np-replica-web`, `NP_OIDC_REDIRECT_URI=https://np.lihato.icu/auth/callback`, and `NP_OIDC_POST_LOGOUT_URI=https://np.lihato.icu/`.
|
||
- Redis stores the live queue and login sessions. SQLite WAL at `/data/np-replica.sqlite3` stores users, owned jobs, compressed inputs/results/errors, usage totals, and durable share links.
|
||
- History and job APIs enforce ownership by the Authentik OIDC subject. Public share links expose only the selected record and can be disabled or given an expiry by its owner.
|