升级 NUPACK 4.1 并支持混合材料设计

2026-08-20 16:33:16 +08:00 · 2026-08-20 16:33:16 +08:00 · c6189d857d
commit c6189d857d
parent 5daa60a464
33 changed files with 5830 additions and 466 deletions
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@ -0,0 +1,159 @@
+# NP Replica Architecture
+
+## Goals
+
+NP Replica separates user navigation, durable account data, live queue state,
+and NUPACK compute processes. Scientific calculation code remains isolated from
+page rendering and account administration so UI changes do not alter numerical
+semantics.
+
+## Runtime Topology
+
+```text
+Browser
+  |
+  | HTTPS / OIDC session
+  v
+np-replica (Web/API, 4 GB)
+  |                 |
+  | live state      | durable metadata/results
+  v                 v
+Redis             SQLite
+  |
+  | job queue
+  v
+np-worker (NUPACK compute, 56 GB)
+```
+
+- **Web/API** serves the multi-page UI, validates ownership, accepts jobs, and
+  serializes API responses. It does not run Redis-backed NUPACK jobs.
+- **Redis** owns queue order, active job state, heartbeats, duplicate claims,
+  sessions, and temporary result state.
+- **SQLite** owns users, durable job inputs/results, usage totals, and shares.
+- **Worker** consumes Redis jobs and runs NUPACK in isolated child processes.
+
+## Frontend Structure
+
+The UI uses a small multi-page application structure. Each page has one primary
+operational purpose.
+
+| Route | File | Responsibility |
+| --- | --- | --- |
+| `/` | `service/home.html` | Queue overview, usage summary, recent jobs |
+| `/workspace` | `service/index.html` | Analysis, Design, Utilities, result inspection |
+| `/cloud` | `service/cloud.html` | Job search, JSON import/export, result links, cancellation, deletion, shares |
+| `/account` | `service/account.html` | Identity, account usage, worker resource information |
+| `/admin` | `service/admin-login.html` | Dedicated Admin password login |
+| `/admin/panel` | `service/admin.html` | Cross-account task and trash management |
+
+`service/static/app-shell.css` provides the shared application shell and
+responsive operational layout. `service/static/portal.js` contains shared
+request, formatting, navigation, table, Cloud, Account, and Admin controllers.
+The calculation workspace retains its self-contained renderer because it
+contains specialized structure, heatmap, force-layout, and result-export code.
+
+## Backend Boundaries
+
+- `service/server.py`
+  - HTTP routes and authentication gates
+  - Redis live-job repository and queue
+  - resource planning and worker orchestration
+  - NUPACK request validation and calculation adapters
+- `service/account.py`
+  - SQLite schema and durable repository
+  - account-scoped job/share queries
+  - cross-account Admin queries and trash lifecycle
+- `service/split_strand_svg.py`
+  - structure rendering adapter
+
+The next safe backend extraction point is moving Redis job coordination into a
+`job_repository.py` module. NUPACK adapters can then move into separate
+`analysis_service.py`, `design_service.py`, and `utilities_service.py`
+modules without changing HTTP contracts.
+
+## Job Submission And Duplicate Control
+
+Duplicate detection is scoped by `user_id` and a canonical SHA-256 hash of the
+JSON payload. Different users never share duplicate state.
+
+```text
+POST /api/jobs
+  |
+  +-- no active match --> 202 accepted
+  |
+  +-- active match ----> 409 duplicate + existing job metadata
+                           |
+                           +-- user cancels: no write
+                           |
+                           +-- user confirms:
+                               POST /api/jobs?force=1 --> 202 accepted
+```
+
+Redis uses an atomic claim key. Terminal updates delete a claim only when it
+still belongs to that job, preventing an older job from deleting a newer forced
+submission's claim. The in-memory development backend mirrors this behavior
+under `JOB_LOCK`.
+
+## Resource Planning
+
+`NP_WORKER_CONCURRENCY=auto` selects the smaller of:
+
+- effective CPU capacity divided by `NP_PER_JOB_THREAD_LIMIT`;
+- usable worker memory divided by `NP_ESTIMATED_JOB_MEMORY_GB`.
+
+CPU detection considers process affinity and cgroup quota. Memory detection
+uses an explicit override, cgroup memory limit, or host memory in that order.
+The published plan is visible through `/health`.
+
+These controls affect scheduling only. They do not change models, sequence
+constraints, complexes, thermodynamic parameters, or result serialization.
+
+## Authentication And Authorization
+
+- Normal pages and account APIs require an OIDC-backed session.
+- Job, history, and share APIs enforce ownership by stable OIDC `user_id`.
+- Public share routes expose only the selected shared record.
+- Admin uses a dedicated password configured by `NP_ADMIN_TOKEN` and bypasses
+  OIDC. The password is accepted only in the JSON body of
+  `POST /api/admin/login`; it is never embedded in a route, redirect, cookie, or
+  local storage value.
+- Successful login creates a random session in Redis (or process memory in
+  development) and sets a short-lived HttpOnly, SameSite=Strict cookie.
+- Admin may list all durable job metadata, stop active jobs, move terminal jobs
+  to trash, restore jobs, permanently delete trashed jobs, and configure trash
+  retention. APIs do not return stored payload or result blobs.
+
+The Admin password must be random, kept in the ignored `.env`, and rotated after
+exposure. Keep `NP_ADMIN_COOKIE_SECURE=1` in HTTPS deployments.
+
+## Trash Lifecycle
+
+Normal user deletion sets `deleted_at` and `purge_after` rather than deleting a
+job row. Deleted jobs disappear from account history, usage, and result APIs;
+all share records for that job are deleted. Admin can inspect the trash, restore a
+job, or permanently delete it. Expired rows are purged lazily during job-list
+reads. Retention is stored in SQLite `app_settings`, defaults to two days, and
+can be changed from one hour to 365 days.
+
+## Persistence And Recovery
+
+- Redis AOF persists live queue state under `runtime/redis`.
+- SQLite WAL persists account data under `runtime/account`.
+- Worker recovery requeues jobs marked running after an interrupted worker.
+- Active Redis job records do not expire while heartbeat updates continue;
+  terminal records use `NP_JOB_TTL_SECONDS`.
+
+Back up both runtime directories. SQLite is the durable source for the Cloud and
+Admin pages; Redis is the source for current queue and heartbeat state.
+
+## Deployment
+
+1. Set `NP_ADMIN_TOKEN` in `.env`.
+2. Validate with `docker compose config --quiet`.
+3. Wait for active calculations to finish before recreating `np-worker`.
+4. Build and recreate Web and Worker from the same source revision.
+5. Verify `/health`, normal OIDC pages, duplicate confirmation, Admin password
+   login, task controls, and the trash lifecycle.
+
+The Web container may be recreated independently for page-only changes, but
+backend contract changes should deploy Web and Worker from the same image.
--- a/docs/FRONTEND_OPTIONS.md
+++ b/docs/FRONTEND_OPTIONS.md
@ -0,0 +1,28 @@
+# Workbench Frontend Options
+
+## Decision
+
+The current service is a server-rendered, dependency-light HTML/CSS/JavaScript application. The first UI pass keeps that boundary: `service/index.html` still owns the existing calculation payload and result rendering, while `service/static/workspace-refresh.css` owns the workbench presentation layer.
+
+This keeps the account/OIDC flow and calculation API stable, avoids a frontend build pipeline, and makes it possible to replace the shell incrementally later.
+
+## Open-source shells that fit
+
+| Option | License | Fit for this service | Trade-off |
+| --- | --- | --- | --- |
+| [Tabler](https://tabler.io/) | MIT | Best direct fit. Static CSS/components, strong tables, forms, navigation, and responsive layout. It can be vendored and mounted without changing the Python service. | The existing large result renderer would still need custom components. |
+| [CoreUI Free](https://coreui.io/) | MIT | Good if the application grows into a full operations console with user, storage, and admin areas. | More opinionated and heavier than the current single-page workbench. |
+| [Apache ECharts](https://echarts.apache.org/) | Apache-2.0 | Good for the result dashboards and concentration/defect charts. | It is a visualization layer, not a complete application shell. |
+| [shadcn/ui](https://ui.shadcn.com/) | MIT components | Excellent visual quality for a future React/Next.js rewrite. | Requires introducing a Node build, React, and a new frontend boundary. |
+
+## Recommended migration path
+
+1. Keep `/api/*`, OIDC, duplicate detection, job polling, cancel, and exports unchanged.
+2. Vendor Tabler CSS and icons locally; use its shell, tabs, cards, tables, alerts, and progress components.
+3. Split the current workbench into three route-level views: `Setup`, `Run`, and `Results`. Each view reads and writes one shared draft object.
+4. Move the existing result renderers into isolated modules only after the new shell is stable. The computational payload stays a versioned JSON contract.
+5. Add Playwright smoke coverage for authenticated navigation, draft persistence, submit/duplicate confirmation, polling, cancel, and result export.
+
+## Why not replace it in one step?
+
+The current `index.html` contains the complete NUPACK result viewer, structure graph, pair heatmap, design preflight, history, and export behavior. A template can replace the shell quickly, but replacing the whole file at once would risk changing calculation semantics and result interpretation. An incremental shell migration gives the visual improvement immediately and keeps the scientific behavior testable.
--- a/docs/design(2).txt
+++ b/docs/design(2).txt
@ -0,0 +1,30 @@
+from nupack import *
+
+my_model = Model(material='dna', celsius=37,sodium=0.1, magnesium=0.02)
+
+c = Domain('N8',        name='c')
+cc = Domain('N3',        name='cc')
+cc2 = Domain('N3',        name='cc2')
+d = Domain('N8',        name='d')
+e = Domain('N7R1',        name='e')
+f = Domain('N10',        name='f')
+g = Domain('N8',        name='g')
+k = Domain('N8',        name='k')
+h = Domain('N15R1',        name='h')
+hh = Domain('N6',        name='hh')
+E = Domain('GGCTAGCTACAACGA',        name='E')
+
+E1 = TargetStrand([h, E, ~e, ~c], name='Strand E1')
+F1 = TargetStrand([~f, ~d, ~g, ~k, f, c, e, hh], name='Strand F1')
+
+E1F1 = TargetComplex([E1, F1], '.10(6.15(16+(10.24)32', name='E1F1')
+
+t1 = TargetTube(on_targets={E1F1: 1e-8}, name='t1',
+    off_targets=SetSpec(max_size=4))
+
+pattern = Pattern(['A4', 'C4', 'G4', 'U4', 'T4'])
+my_tubes = [t1,t2]
+my_design = tube_design(tubes=my_tubes,
+    hard_constraints=[pattern],model=my_model)
+my_result = my_design.run(trials=1)
+print(my_result)
--- a/docs/need.md
+++ b/docs/need.md
@ -0,0 +1,39 @@
+请修改当前 RNA 二级结构可视化中的“配对概率”视图，重点不是简单美化，而是修正可视化语义和可读性。
+如图：[[1.png]]
+当前问题：
+1. 现在用同一种绿色的不同透明度表示概率，用户几乎无法区分不同概率值。
+2. 当前视图容易让人误解“配对概率”到底表示节点概率还是边概率。
+3. 小尺寸视图下，透明度编码效果很差，截图后更不清楚。
+4. 右侧图例是 0~1，但图中节点颜色变化不够清晰。
+
+修改目标：
+1. 不要再使用“同色 + alpha透明度”作为主编码。
+2. 改成“固定不透明度 + 明确的颜色梯度”来表示数值大小。
+3. 优先使用单调、感知一致的 colormap（例如 viridis / plasma / magma），不要用当前这种浅绿色透明度方案。
+4. 节点颜色必须在 0~1 范围内有清楚区分，低值和高值要一眼能看出。
+5. 右侧 colorbar 要和图中实际颜色完全一致。
+6. 保持当前布局基本不变，不要先大改 UI。
+
+语义要求：
+1. 如果当前数据实际上是“每个碱基的边际配对倾向 / per-base probability”，那么按钮或图例文字不要再直接写 Pair probabilities，而要改成更准确的名称，例如：
+   - Per-base pairing probability
+   - Base-wise pairing score
+   - Marginal pairing probability
+2. 如果当前数据确实是配对矩阵 P(i,j)，那就不要只给节点上色，后续需要支持把概率映射到配对边上。
+
+这一步先做的事情：
+1. 先保留当前节点着色模式。
+2. 去掉 alpha 映射，改成纯颜色映射。
+3. 调整图例标题，使其准确表达“节点值”而不是“边值”。
+4. 检查节点、文字、主链、配对线在高低概率下是否仍然清晰可读。
+
+验收标准：
+1. 同一颜色不同透明度的问题被彻底移除。
+2. 在小图（类似当前截图尺寸）下，0.2、0.5、0.8 三档能明显区分。
+3. 用户不会再把这个图误解为“边概率图”。
+4. 代码尽量少改，先保证现有功能可用。
+
+请直接修改代码，并说明：
+1. 改了哪些绘图参数
+2. 改了哪些 label / title / legend 文案
+3. 如果当前数据语义和 Pair probabilities 不一致，请明确指出并给出更合适命名
--- a/docs/绘图自动分割.txt
+++ b/docs/绘图自动分割.txt
@ -0,0 +1,94 @@
+                                在ununtu上使用该命令安装draw_rna
+pip install matplotlib numpy draw_rna --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+-------------------------------打开原本步骤中lab中的记事本--------------------------
+# 依赖导入
+from nupack import *
+from draw_rna.ipynb_draw import draw_struct
+import os
+import matplotlib.pyplot as plt
+
+# 开启显示（恢复正常绘图展示）
+plt.ion()
+
+# ======================================
+# 仅保留两个核心函数，完全用 draw_rna
+# ======================================
+def get_mfe_structure(complex_name: str, nupack_result, strands: list):
+    """获取复合物MFE结构，支持同源二聚体 Y1+Y1"""
+    strand_names = complex_name.strip('()').split('+')
+    strand_map = {s.name: s for s in strands}
+    strand_objs = [strand_map[name] for name in strand_names]
+    complex_obj = Complex(strand_objs)
+    return str(nupack_result[complex_obj].mfe[0].structure)
+
+def draw_complex(complex_name: str, nupack_result, strands: list, output_folder="./"):
+    """draw_rna绘图：无页面显示，直接保存图片到目标路径"""
+    os.makedirs(output_folder, exist_ok=True)
+
+    # 获取结构和序列（完全不变）
+    struct = get_mfe_structure(complex_name, nupack_result, strands)
+    strand_names = complex_name.strip('()').split('+')
+    strand_map = {s.name: s for s in strands}
+    seq = " ".join([str(strand_map[name]) for name in strand_names])   # 序列用空格分隔
+    draw_structure = struct.replace('+', ' ')                            # 结构用空格分隔
+    # 设置全局字体大小，可以根据需要调整数字 (例如 14, 18, 20)
+    plt.rcParams['font.size'] = 12
+    # ---------- 修正点：显式创建画布并传递给 draw_struct ----------
+    fig, ax = plt.subplots(figsize=(15, 15))   # 可调尺寸，保证图形清晰
+    draw_struct(seq, draw_structure, ax=ax)   # 在指定 ax 上绘图
+
+    # 保存并关闭
+    save_name = complex_name.strip('()').replace('+', '_') + '.png'
+    save_path = os.path.join(output_folder, save_name)
+    plt.savefig(save_path, dpi=300, bbox_inches='tight')
+
+    print(f"\n✅ 图片已保存至：{save_path}")
+    print(f"MFE 结构：{struct}")
+def draw_all_complexes(nupack_result, strands: list, output_folder="./"):
+    print("\n========== 开始批量绘制所有复合物 ==========")
+    
+    # 过滤出 Complex 对象
+    complex_objs = [obj for obj in nupack_result.keys() if not isinstance(obj, Tube)]
+    total = len(complex_objs)
+    success_count = 0
+    
+    for idx, complex_obj in enumerate(complex_objs, 1):
+        complex_name = complex_obj.name   # ✅ 修正点
+        try:
+            draw_complex(complex_name, nupack_result, strands, output_folder)
+            success_count += 1
+            print(f"进度：{idx}/{total} 完成")
+        except Exception as e:
+            print(f"❌ 绘制复合物 {complex_name} 失败: {e}")
+    
+    print("\n========== 所有复合物绘制完成 ==========")
+    print(f"✅ 批量绘制完成！成功：{success_count} / 总计：{total}")
+    print(f"📁 图片保存在：{os.path.abspath(output_folder)}")
+--------------------------------  nupack计算--------------------------------------
+
+model1 = Model(material='dna', celsius=37,sodium=0.1, magnesium=0.02)
+Y1 = Strand('CGTTAACGCAGTGAGGACGGTAGTTTGTCGTTCCATCGCACC', name='Y1')
+Y2 = Strand('CGTTAACGGGTGCGATGGAACGACTTTCGACAGGCCTGGTGTAATTTCACCCATGTTAGTCGA', name='Y2')
+Y3 = Strand('ATACGGAAAATGGAGATAGGAAGAGTACAATGTCAGCGATAAATTCCGTATACGACACCAGGCCTGTCGATTACTACCGTCCTCACTG', name='Y3')
+
+t1 = Tube({Y1: 1e-8, Y2: 1e-8, Y3:1e-8 },complexes=SetSpec(max_size=4), name='Tube 1')
+
+my_result = tube_analysis(tubes=[t1], model=model1,
+    compute=['pfunc', 'pairs', 'mfe', 'sample', 'subopt'],
+    options={'num_sample': 1, 'energy_gap': 0.5})
+print(my_result)
+
+--------------------------- --------------- 单复合物绘图------------------------------
+target = '(Y1+Y1+Y2+Y3)'   # 替换为你的目标复合物名称
+all_strands = [Y1, Y2, Y3]  # 所有链的列表
+
+# 1. 获取并打印MFE结构
+mfe_struct = get_mfe_structure(target, my_result, all_strands)
+#print("MFE结构：", mfe_struct)
+
+# 2. 绘制二级结构
+draw_complex(target, my_result, all_strands, output_folder="/home/zsy")
+--------------------------------------------批量绘图-----------------------------------
+strands_list = [Y1, Y2, Y3]
+draw_all_complexes(my_result, strands_list, output_folder="/home/zsy/jihe")