v4.7 新增账号系统,云端存储保证计算的长久有效

This commit is contained in:
Lihatoo 2026-07-21 14:14:15 +08:00
parent cbbef10e5d
commit dd21623cb5
7 changed files with 1032 additions and 178 deletions

View file

@ -17,3 +17,4 @@ nupack/vendor/nupack-4.0.2.0/package/*.whl
!nupack/vendor/nupack-4.0.2.0/package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl !nupack/vendor/nupack-4.0.2.0/package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl
rna/* rna/*
!rna/ViennaRNA-2.7.2.tar.gz !rna/ViennaRNA-2.7.2.tar.gz
runtime

1
.gitignore vendored
View file

@ -18,3 +18,4 @@ nupack/vendor/nupack-4.0.2.0/package/*.whl
!nupack/vendor/nupack-4.0.2.0/package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl !nupack/vendor/nupack-4.0.2.0/package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl
# Runtime state mounted from Docker containers. # Runtime state mounted from Docker containers.
runtime/redis/ runtime/redis/
runtime/account/

View file

@ -31,14 +31,14 @@ RUN cd /tmp \
&& rm -rf /tmp/ViennaRNA-2.7.2 /tmp/ViennaRNA-2.7.2.tar.gz && rm -rf /tmp/ViennaRNA-2.7.2 /tmp/ViennaRNA-2.7.2.tar.gz
RUN python -m pip install --no-cache-dir \ RUN python -m pip install --no-cache-dir \
numpy \ numpy==2.4.6 \
scipy \ scipy==1.17.1 \
pandas \ pandas==3.0.3 \
pyyaml \ pyyaml==6.0.3 \
jinja2 \ jinja2==3.1.6 \
/tmp/nupack-package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl /tmp/nupack-package/nupack-4.0.2.0-cp312-cp312-linux_x86_64.whl
RUN python -m pip install --no-cache-dir redis RUN python -m pip install --no-cache-dir redis==7.4.0
WORKDIR /app/service WORKDIR /app/service
COPY service /app/service COPY service /app/service

View file

@ -12,7 +12,9 @@ Open `http://127.0.0.1:18765`.
## Reverse Proxy ## Reverse Proxy
The service listens on port `18765`. You can later reverse proxy `np.lihato.icu` to this port from your public server. The service listens on port `18765`. The current local deployment is reverse proxied as `npt.lihato.icu`.
Traefik only performs TLS termination, compression, and reverse proxying for `npt`; authentication is enforced by this application.
## Endpoints ## Endpoints
@ -26,7 +28,7 @@ The service listens on port `18765`. You can later reverse proxy `np.lihato.icu`
## Persistent Shares ## Persistent Shares
Shared result links are stored in Redis when `NP_REDIS_URL` is enabled. The compose file maps Redis data to `./runtime/redis` and enables AOF/RDB persistence, so shares survive container recreation and service restarts. Back up or migrate `./runtime/redis` if you move the deployment. Live jobs and sessions are stored in Redis when `NP_REDIS_URL` is enabled. Account history and share links are stored in `./runtime/account/np-replica.sqlite3`; back up both runtime directories when moving the deployment.
## Notes ## Notes
@ -35,7 +37,7 @@ Shared result links are stored in Redis when `NP_REDIS_URL` is enabled. The comp
- Pair matrices are returned as a preview block to keep responses manageable. - Pair matrices are returned as a preview block to keep responses manageable.
- The Docker image compiles ViennaRNA `RNAplot` from `rna/ViennaRNA-2.7.2.tar.gz` and embeds SVG structure plots into MFE results. - The Docker image compiles ViennaRNA `RNAplot` from `rna/ViennaRNA-2.7.2.tar.gz` and embeds SVG structure plots into MFE results.
- Multistrand MFE plots now use a split-strand layout by default when a structure contains `+`, with `RNAplot` retained as the fallback path. - Multistrand MFE plots now use a split-strand layout by default when a structure contains `+`, with `RNAplot` retained as the fallback path.
- The web UI now supports Chinese and English switching, browser-side history, result export, and async polling for submitted jobs. - The web UI supports Chinese and English switching, account-scoped cloud history, result export, share controls, and async polling for submitted jobs.
- Docker now starts three services: `np-replica`, `np-worker`, and `redis`. - Docker now starts three services: `np-replica`, `np-worker`, and `redis`.
- Async jobs use Redis-backed storage and queueing when `NP_REDIS_URL` is configured; otherwise the app falls back to the older in-memory mode. - Async jobs use Redis-backed storage and queueing when `NP_REDIS_URL` is configured; otherwise the app falls back to the older in-memory mode.
- The browser-facing API and page behavior stay the same after enabling Redis-backed jobs. - The browser-facing API and page behavior stay the same after enabling Redis-backed jobs.
@ -45,6 +47,11 @@ Shared result links are stored in Redis when `NP_REDIS_URL` is enabled. The comp
- Static pages are served with ETag-based browser caching; JSON API responses remain uncached. - Static pages are served with ETag-based browser caching; JSON API responses remain uncached.
- The force-directed structure viewer lazy-loads D3 only when that view is opened, so the initial page load is not blocked by the external CDN. - The force-directed structure viewer lazy-loads D3 only when that view is opened, so the initial page load is not blocked by the external CDN.
- `/health` uses Redis queue length plus a running-job set instead of scanning every historical job on each poll. - `/health` uses Redis queue length plus a running-job set instead of scanning every historical job on each poll.
- Application login uses the public Authentik issuer at `https://auth.lihato.icu/` with Authorization Code + PKCE.
- The current `npt` instance uses issuer `/application/o/nupack-account-npt/`, client `npt-replica-web`, callback `https://npt.lihato.icu/auth/callback`, and logout return `https://npt.lihato.icu/`.
- For the independent `np` deployment on `100.64.0.11`, set `NP_OIDC_ISSUER=https://auth.lihato.icu/application/o/nupack-account/`, `NP_OIDC_CLIENT_ID=np-replica-web`, `NP_OIDC_REDIRECT_URI=https://np.lihato.icu/auth/callback`, and `NP_OIDC_POST_LOGOUT_URI=https://np.lihato.icu/`.
- Redis stores the live queue and login sessions. SQLite WAL at `/data/np-replica.sqlite3` stores users, owned jobs, compressed inputs/results/errors, usage totals, and durable share links.
- History and job APIs enforce ownership by the Authentik OIDC subject. Public share links expose only the selected record and can be disabled or given an expiry by its owner.
@ -53,7 +60,8 @@ Shared result links are stored in Redis when `NP_REDIS_URL` is enabled. The comp
- 最多同时跑 16 个任务 - 最多同时跑 16 个任务
- 每个任务最多用 4 个计算线程 - 每个任务最多用 4 个计算线程
- 后续任务进入队列等待 - 后续任务进入队列等待
- 理论上最多占用约 64 个 native 计算线程,避免 16 × 8 这类过度超卖 - 每个任务的 NUPACK 缓存上限为 8 GB
- 理论上最多占用约 64 个 native 计算线程
如果后面你发现机器还会被压满,最直接的调法就是在 docker-compose.yml 里继续压: 如果后面你发现机器还会被压满,最直接的调法就是在 docker-compose.yml 里继续压:

View file

@ -9,7 +9,7 @@ services:
- ./runtime/redis:/data - ./runtime/redis:/data
np-replica: np-replica:
image: np-app:v4.5 image: np-app:v4.7
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
@ -18,6 +18,8 @@ services:
mem_limit: 4g mem_limit: 4g
depends_on: depends_on:
- redis - redis
volumes:
- ./runtime/account:/data
ports: ports:
- "100.64.0.2:18765:18765" - "100.64.0.2:18765:18765"
environment: environment:
@ -25,6 +27,12 @@ services:
NP_PORT: 18765 NP_PORT: 18765
NP_RUN_MODE: server NP_RUN_MODE: server
NP_REDIS_URL: redis://redis:6379/0 NP_REDIS_URL: redis://redis:6379/0
NP_ACCOUNT_DB_PATH: /data/np-replica.sqlite3
NP_AUTH_REQUIRED: 1
NP_OIDC_ISSUER: https://auth.lihato.icu/application/o/nupack-account-npt/
NP_OIDC_CLIENT_ID: npt-replica-web
NP_OIDC_REDIRECT_URI: https://npt.lihato.icu/auth/callback
NP_OIDC_POST_LOGOUT_URI: https://npt.lihato.icu/
NP_JOB_TTL_SECONDS: 3600 NP_JOB_TTL_SECONDS: 3600
NP_WORKER_CONCURRENCY: 16 NP_WORKER_CONCURRENCY: 16
NP_PER_JOB_THREAD_LIMIT: 4 NP_PER_JOB_THREAD_LIMIT: 4
@ -37,15 +45,18 @@ services:
GOTO_NUM_THREADS: 4 GOTO_NUM_THREADS: 4
np-worker: np-worker:
image: np-app:v4.5 image: np-app:v4.7
container_name: np-worker container_name: np-worker
restart: always restart: always
mem_limit: 56g mem_limit: 56g
depends_on: depends_on:
- redis - redis
volumes:
- ./runtime/account:/data
environment: environment:
NP_RUN_MODE: worker NP_RUN_MODE: worker
NP_REDIS_URL: redis://redis:6379/0 NP_REDIS_URL: redis://redis:6379/0
NP_ACCOUNT_DB_PATH: /data/np-replica.sqlite3
NP_JOB_TTL_SECONDS: 3600 NP_JOB_TTL_SECONDS: 3600
NP_WORKER_CONCURRENCY: 16 NP_WORKER_CONCURRENCY: 16
NP_PER_JOB_THREAD_LIMIT: 4 NP_PER_JOB_THREAD_LIMIT: 4

File diff suppressed because it is too large Load diff

View file

@ -14,7 +14,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path from pathlib import Path
import subprocess import subprocess
import tempfile import tempfile
from urllib.parse import parse_qs, urlparse from urllib.parse import parse_qs, urlencode, urlparse
from uuid import uuid4 from uuid import uuid4
import numpy import numpy
@ -45,6 +45,7 @@ from nupack import (
tube_design, tube_design,
) )
from nupack import config as nupack_config from nupack import config as nupack_config
from account import AccountStore, OIDCAuth
from split_strand_svg import render_split_strands_svg from split_strand_svg import render_split_strands_svg
try: try:
@ -70,6 +71,7 @@ JOB_RUNNING_KEY = os.environ.get("NP_JOB_RUNNING_KEY", "np_replica:jobs:running"
WORKER_CONCURRENCY = int(os.environ.get("NP_WORKER_CONCURRENCY", "2")) WORKER_CONCURRENCY = int(os.environ.get("NP_WORKER_CONCURRENCY", "2"))
PER_JOB_THREAD_LIMIT = int(os.environ.get("NP_PER_JOB_THREAD_LIMIT", "1")) PER_JOB_THREAD_LIMIT = int(os.environ.get("NP_PER_JOB_THREAD_LIMIT", "1"))
NUPACK_CACHE_GB = float(os.environ.get("NP_NUPACK_CACHE_GB", "2.0")) NUPACK_CACHE_GB = float(os.environ.get("NP_NUPACK_CACHE_GB", "2.0"))
ACCOUNT_DB_PATH = os.environ.get("NP_ACCOUNT_DB_PATH", "/data/np-replica.sqlite3")
UNIT_SCALE = { UNIT_SCALE = {
"M": 1.0, "M": 1.0,
@ -88,6 +90,7 @@ TERMINAL_JOB_STATUSES = {"success", "error", CANCELED_STATUS}
JOB_STORE = {} JOB_STORE = {}
JOB_LOCK = threading.Lock() JOB_LOCK = threading.Lock()
JOB_TTL_SECONDS = int(os.environ.get("NP_JOB_TTL_SECONDS", "3600")) JOB_TTL_SECONDS = int(os.environ.get("NP_JOB_TTL_SECONDS", "3600"))
JOB_HEARTBEAT_SECONDS = max(1, int(os.environ.get("NP_JOB_HEARTBEAT_SECONDS", "30")))
JOB_MAX_COUNT = int(os.environ.get("NP_JOB_MAX_COUNT", "64")) JOB_MAX_COUNT = int(os.environ.get("NP_JOB_MAX_COUNT", "64"))
SHARE_STORE = {} SHARE_STORE = {}
SHARE_LOCK = threading.Lock() SHARE_LOCK = threading.Lock()
@ -96,6 +99,8 @@ SHARE_MAX_BYTES = int(os.environ.get("NP_SHARE_MAX_BYTES", str(12 * 1024 * 1024)
SHARE_KEY_PREFIX = os.environ.get("NP_SHARE_KEY_PREFIX", "np_replica:shares") SHARE_KEY_PREFIX = os.environ.get("NP_SHARE_KEY_PREFIX", "np_replica:shares")
SHARE_INDEX_KEY = f"{SHARE_KEY_PREFIX}:index" SHARE_INDEX_KEY = f"{SHARE_KEY_PREFIX}:index"
REDIS_CLIENT = None REDIS_CLIENT = None
ACCOUNT_STORE = AccountStore(ACCOUNT_DB_PATH)
OIDC_AUTH = None
def json_bytes(payload, status=HTTPStatus.OK): def json_bytes(payload, status=HTTPStatus.OK):
@ -151,6 +156,9 @@ def redis_client():
return REDIS_CLIENT return REDIS_CLIENT
OIDC_AUTH = OIDCAuth(redis_client)
def queue_size(): def queue_size():
if not redis_enabled(): if not redis_enabled():
with JOB_LOCK: with JOB_LOCK:
@ -1562,7 +1570,14 @@ def prune_jobs(now=None):
def set_job_data(job): def set_job_data(job):
if redis_enabled(): if redis_enabled():
redis_client().setex(job_key(job["job_id"]), JOB_TTL_SECONDS, json.dumps(job, ensure_ascii=False)) client = redis_client()
key = job_key(job["job_id"])
encoded = json.dumps(job, ensure_ascii=False)
if job.get("status") in TERMINAL_JOB_STATUSES:
client.setex(key, JOB_TTL_SECONDS, encoded)
else:
# Active jobs may legitimately outlive the result-retention TTL.
client.set(key, encoded)
return return
with JOB_LOCK: with JOB_LOCK:
prune_jobs(job.get("updated_at")) prune_jobs(job.get("updated_at"))
@ -1613,7 +1628,11 @@ def update_job_data(job_id, **updates):
current.update(updates) current.update(updates)
current["updated_at"] = time.time() current["updated_at"] = time.time()
pipe.multi() pipe.multi()
pipe.setex(key, JOB_TTL_SECONDS, json.dumps(current, ensure_ascii=False)) encoded = json.dumps(current, ensure_ascii=False)
if current.get("status") in TERMINAL_JOB_STATUSES:
pipe.setex(key, JOB_TTL_SECONDS, encoded)
else:
pipe.set(key, encoded)
pipe.execute() pipe.execute()
return current return current
except redis.WatchError: except redis.WatchError:
@ -1634,11 +1653,12 @@ def update_job_data(job_id, **updates):
return dict(current) return dict(current)
def create_job(payload): def create_job(payload, owner):
job_id = uuid4().hex job_id = uuid4().hex
now = time.time() now = time.time()
job = { job = {
"job_id": job_id, "job_id": job_id,
"user_id": owner["user_id"],
"status": "queued", "status": "queued",
"error": None, "error": None,
"result": None, "result": None,
@ -1646,8 +1666,9 @@ def create_job(payload):
"updated_at": now, "updated_at": now,
"payload": payload, "payload": payload,
} }
ACCOUNT_STORE.create_job(job_id, owner, payload, status="queued", created_at=now)
set_job_data(job) set_job_data(job)
log_event(f"accepted job_id={job_id} mode={payload.get('mode', 'tube')}") log_event(f"accepted job_id={job_id} user_id={owner['user_id']} mode={payload.get('mode', 'tube')}")
if redis_enabled(): if redis_enabled():
redis_client().lpush(JOB_QUEUE_KEY, job_id) redis_client().lpush(JOB_QUEUE_KEY, job_id)
@ -1688,14 +1709,17 @@ def terminate_process(process):
def _run_job(job_id, payload): def _run_job(job_id, payload):
started_at = time.time() started_at = time.time()
last_heartbeat_at = started_at
current = get_job_data(job_id) current = get_job_data(job_id)
if current and current.get("status") in {CANCEL_REQUESTED_STATUS, CANCELED_STATUS}: if current and current.get("status") in {CANCEL_REQUESTED_STATUS, CANCELED_STATUS}:
elapsed = round(time.time() - started_at, 3) elapsed = round(time.time() - started_at, 3)
update_job_data(job_id, status=CANCELED_STATUS, payload=None, elapsed_seconds=elapsed) update_job_data(job_id, status=CANCELED_STATUS, payload=None, elapsed_seconds=elapsed)
ACCOUNT_STORE.update_job(job_id, CANCELED_STATUS, error={"message": "Job canceled by user."}, elapsed_seconds=elapsed)
return return
log_event(f"running job_id={job_id}") log_event(f"running job_id={job_id}")
update_job_data(job_id, status="running") update_job_data(job_id, status="running")
ACCOUNT_STORE.update_job(job_id, "running")
if redis_enabled(): if redis_enabled():
redis_client().sadd(JOB_RUNNING_KEY, job_id) redis_client().sadd(JOB_RUNNING_KEY, job_id)
@ -1719,8 +1743,15 @@ def _run_job(job_id, payload):
payload=None, payload=None,
elapsed_seconds=elapsed, elapsed_seconds=elapsed,
) )
ACCOUNT_STORE.update_job(
job_id, CANCELED_STATUS, error={"message": "Job canceled by user."}, elapsed_seconds=elapsed
)
log_event(f"canceled job_id={job_id} elapsed={elapsed}s") log_event(f"canceled job_id={job_id} elapsed={elapsed}s")
return return
now = time.time()
if now - last_heartbeat_at >= JOB_HEARTBEAT_SECONDS:
update_job_data(job_id, heartbeat_at=now)
last_heartbeat_at = now
try: try:
message = result_queue.get_nowait() message = result_queue.get_nowait()
break break
@ -1745,30 +1776,38 @@ def _run_job(job_id, payload):
payload=None, payload=None,
elapsed_seconds=elapsed, elapsed_seconds=elapsed,
) )
ACCOUNT_STORE.update_job(
job_id, CANCELED_STATUS, error={"message": "Job canceled by user."}, elapsed_seconds=elapsed
)
log_event(f"canceled job_id={job_id} elapsed={elapsed}s") log_event(f"canceled job_id={job_id} elapsed={elapsed}s")
return return
elapsed = round(time.time() - started_at, 3) elapsed = round(time.time() - started_at, 3)
if message and message.get("status") == "success": if message and message.get("status") == "success":
update_job_data(job_id, status="success", result=message["result"], payload=None, elapsed_seconds=elapsed) update_job_data(job_id, status="success", result=message["result"], payload=None, elapsed_seconds=elapsed)
ACCOUNT_STORE.update_job(job_id, "success", result=message["result"], elapsed_seconds=elapsed)
log_event(f"success job_id={job_id} elapsed={elapsed}s") log_event(f"success job_id={job_id} elapsed={elapsed}s")
elif message and message.get("status") == "error": elif message and message.get("status") == "error":
error = message.get("error") or {"message": "Job failed."}
update_job_data( update_job_data(
job_id, job_id,
status="error", status="error",
error=message.get("error") or {"message": "Job failed."}, error=error,
payload=None, payload=None,
elapsed_seconds=elapsed, elapsed_seconds=elapsed,
) )
ACCOUNT_STORE.update_job(job_id, "error", error=error, elapsed_seconds=elapsed)
log_event(f"error job_id={job_id} elapsed={elapsed}s message={message.get('error', {}).get('message')}") log_event(f"error job_id={job_id} elapsed={elapsed}s message={message.get('error', {}).get('message')}")
else: else:
error = {"message": f"Job process exited with code {process.exitcode}."}
update_job_data( update_job_data(
job_id, job_id,
status="error", status="error",
error={"message": f"Job process exited with code {process.exitcode}."}, error=error,
payload=None, payload=None,
elapsed_seconds=elapsed, elapsed_seconds=elapsed,
) )
ACCOUNT_STORE.update_job(job_id, "error", error=error, elapsed_seconds=elapsed)
log_event(f"error job_id={job_id} elapsed={elapsed}s exitcode={process.exitcode}") log_event(f"error job_id={job_id} elapsed={elapsed}s exitcode={process.exitcode}")
finally: finally:
terminate_process(process) terminate_process(process)
@ -1799,12 +1838,14 @@ def cancel_job(job_id):
result=None, result=None,
payload=None, payload=None,
) )
ACCOUNT_STORE.update_job(job_id, CANCELED_STATUS, error={"message": "Job canceled by user."})
else: else:
updated = update_job_data( updated = update_job_data(
job_id, job_id,
status=CANCEL_REQUESTED_STATUS, status=CANCEL_REQUESTED_STATUS,
error={"message": "Cancellation requested."}, error={"message": "Cancellation requested."},
) )
ACCOUNT_STORE.update_job(job_id, CANCEL_REQUESTED_STATUS, error={"message": "Cancellation requested."})
updated.pop("payload", None) updated.pop("payload", None)
return updated return updated
@ -1882,11 +1923,53 @@ def get_share(share_id):
return dict(item) if item else None return dict(item) if item else None
def recover_interrupted_jobs():
client = redis_client()
recovered = 0
canceled = 0
for job_id in client.smembers(JOB_RUNNING_KEY):
raw = client.get(job_key(job_id))
if raw is None:
client.srem(JOB_RUNNING_KEY, job_id)
continue
job = json.loads(raw)
status = job.get("status")
if status == "running":
client.lrem(JOB_QUEUE_KEY, 0, job_id)
update_job_data(
job_id,
status="queued",
recovered_at=time.time(),
recovery_count=int(job.get("recovery_count", 0)) + 1,
)
client.lpush(JOB_QUEUE_KEY, job_id)
ACCOUNT_STORE.update_job(job_id, "queued")
recovered += 1
elif status == CANCEL_REQUESTED_STATUS:
update_job_data(
job_id,
status=CANCELED_STATUS,
error={"message": "Job canceled while the worker was restarting."},
result=None,
payload=None,
)
ACCOUNT_STORE.update_job(
job_id,
CANCELED_STATUS,
error={"message": "Job canceled while the worker was restarting."},
)
canceled += 1
client.srem(JOB_RUNNING_KEY, job_id)
if recovered or canceled:
log_event(f"recovered jobs queued={recovered} canceled={canceled}")
def run_worker_loop(): def run_worker_loop():
if not redis_enabled(): if not redis_enabled():
raise RuntimeError("Worker mode requires NP_REDIS_URL and the redis package.") raise RuntimeError("Worker mode requires NP_REDIS_URL and the redis package.")
client = redis_client() client = redis_client()
recover_interrupted_jobs()
worker_count = max(1, WORKER_CONCURRENCY) worker_count = max(1, WORKER_CONCURRENCY)
log_event( log_event(
f"Starting worker loop on Redis queue {JOB_QUEUE_KEY} " f"Starting worker loop on Redis queue {JOB_QUEUE_KEY} "
@ -2072,18 +2155,101 @@ def get_example_payload(query):
class AppHandler(BaseHTTPRequestHandler): class AppHandler(BaseHTTPRequestHandler):
server_version = "NPReplica/0.1" server_version = "NPReplica/0.1"
def _session(self):
if not hasattr(self, "_cached_session"):
self._cached_session = OIDC_AUTH.current_session(self.headers)
if self._cached_session:
ACCOUNT_STORE.upsert_user(self._cached_session["user"])
return self._cached_session
def _require_user(self):
session = self._session()
if session is None:
self._respond(*json_bytes({"error": "Authentication required", "login_url": "/auth/login"}, status=HTTPStatus.UNAUTHORIZED))
return None
return session["user"]
def _read_json(self, max_bytes=16 * 1024 * 1024):
length = int(self.headers.get("Content-Length", "0"))
if length < 0 or length > max_bytes:
raise ValueError(f"Request payload is too large. Limit is {max_bytes} bytes.")
return json.loads(self.rfile.read(length).decode("utf-8"))
def _redirect(self, location, *, cookie=None):
headers = {"Location": location}
if cookie:
headers["Set-Cookie"] = cookie
self._respond(HTTPStatus.FOUND, "text/plain; charset=utf-8", b"Redirecting", extra_headers=headers)
def _require_page_user(self, next_path):
if self._session() is not None:
return True
self._redirect(f"/auth/login?{urlencode({'next': next_path})}")
return False
def _owned_job(self, user, job_id, include_content=True):
if ACCOUNT_STORE.owner_id(job_id) != user["user_id"]:
return None
live = get_job(job_id)
if live is not None:
live.pop("user_id", None)
return live
return ACCOUNT_STORE.get_job(user["user_id"], job_id, include_content=include_content)
def do_GET(self): def do_GET(self):
parsed = urlparse(self.path) parsed = urlparse(self.path)
if parsed.path == "/": if parsed.path == "/auth/login":
next_path = (parse_qs(parsed.query).get("next") or ["/"])[0]
try:
self._redirect(OIDC_AUTH.begin_login(next_path))
except Exception as exc:
self._respond(*json_bytes({"error": f"Unable to start login: {exc}"}, status=HTTPStatus.BAD_GATEWAY))
return
if parsed.path == "/auth/callback":
try:
session_id, user, next_path = OIDC_AUTH.complete_login(parse_qs(parsed.query))
ACCOUNT_STORE.upsert_user(user)
self._redirect(next_path, cookie=OIDC_AUTH.cookie_header(session_id))
except Exception as exc:
log_event(f"OIDC callback failed: {exc}")
self._respond(*json_bytes({"error": f"Login failed: {exc}"}, status=HTTPStatus.BAD_REQUEST))
return
if parsed.path == "/auth/logout":
session = self._session()
try:
location = OIDC_AUTH.logout_url(session)
except Exception:
location = "/"
OIDC_AUTH.delete_session(session)
self._redirect(location, cookie=OIDC_AUTH.clear_cookie_header())
return
share_page = re.fullmatch(r"/share/([A-Za-z0-9_-]{8,128})", parsed.path)
if share_page:
self._respond_file(INDEX_PATH, cache_control="public, max-age=60, stale-while-revalidate=86400") self._respond_file(INDEX_PATH, cache_control="public, max-age=60, stale-while-revalidate=86400")
return return
legacy_share = (parse_qs(parsed.query).get("share") or [""])[0]
if parsed.path == "/" and re.fullmatch(r"[A-Za-z0-9_-]{8,128}", legacy_share):
self._respond_file(INDEX_PATH, cache_control="public, max-age=60, stale-while-revalidate=86400")
return
if parsed.path == "/":
if not self._require_page_user(self.path):
return
self._respond_file(INDEX_PATH, cache_control="private, no-cache")
return
if parsed.path in {"/favicon.svg", "/favicon.ico"}: if parsed.path in {"/favicon.svg", "/favicon.ico"}:
self._respond_file(FAVICON_PATH, cache_control="public, max-age=86400") self._respond_file(FAVICON_PATH, cache_control="public, max-age=86400")
return return
if parsed.path == "/design-guide.html": if parsed.path == "/design-guide.html":
if not self._require_page_user(self.path):
return
self._respond_file(GUIDE_PATH, cache_control="public, max-age=3600") self._respond_file(GUIDE_PATH, cache_control="public, max-age=3600")
return return
@ -2108,13 +2274,53 @@ class AppHandler(BaseHTTPRequestHandler):
) )
return return
if parsed.path == "/api/me":
user = self._require_user()
if user is None:
return
self._respond(*json_bytes({"status": "success", "user": user, "usage": ACCOUNT_STORE.usage(user["user_id"])}))
return
if parsed.path == "/api/history":
user = self._require_user()
if user is None:
return
params = {key: values[0] for key, values in parse_qs(parsed.query).items() if values}
self._respond(*json_bytes({"status": "success", **ACCOUNT_STORE.list_jobs(user["user_id"], params)}))
return
if parsed.path.startswith("/api/history/"):
user = self._require_user()
if user is None:
return
job_id = parsed.path.rsplit("/", 1)[-1]
job = ACCOUNT_STORE.get_job(user["user_id"], job_id, include_content=True)
if job is None:
self._respond(*json_bytes({"error": "History record not found"}, status=HTTPStatus.NOT_FOUND))
return
self._respond(*json_bytes({"status": "success", "item": job}))
return
if parsed.path == "/api/account/shares":
user = self._require_user()
if user is None:
return
self._respond(*json_bytes({"status": "success", "items": ACCOUNT_STORE.list_shares(user["user_id"])}))
return
if parsed.path == "/api/example": if parsed.path == "/api/example":
user = self._require_user()
if user is None:
return
self._respond(*json_bytes(get_example_payload(parsed.query))) self._respond(*json_bytes(get_example_payload(parsed.query)))
return return
if parsed.path.startswith("/api/jobs/"): if parsed.path.startswith("/api/jobs/"):
user = self._require_user()
if user is None:
return
job_id = parsed.path.rsplit("/", 1)[-1] job_id = parsed.path.rsplit("/", 1)[-1]
job = get_job(job_id) job = self._owned_job(user, job_id)
if job is None: if job is None:
self._respond(*json_bytes({"error": "Job not found"}, status=HTTPStatus.NOT_FOUND)) self._respond(*json_bytes({"error": "Job not found"}, status=HTTPStatus.NOT_FOUND))
return return
@ -2123,7 +2329,7 @@ class AppHandler(BaseHTTPRequestHandler):
if parsed.path.startswith("/api/shares/"): if parsed.path.startswith("/api/shares/"):
share_id = parsed.path.rsplit("/", 1)[-1] share_id = parsed.path.rsplit("/", 1)[-1]
share = get_share(share_id) share = ACCOUNT_STORE.resolve_share(share_id) or get_share(share_id)
if share is None: if share is None:
self._respond(*json_bytes({"error": "Share not found"}, status=HTTPStatus.NOT_FOUND)) self._respond(*json_bytes({"error": "Share not found"}, status=HTTPStatus.NOT_FOUND))
return return
@ -2134,75 +2340,78 @@ class AppHandler(BaseHTTPRequestHandler):
def do_POST(self): def do_POST(self):
parsed = urlparse(self.path) parsed = urlparse(self.path)
user = self._require_user()
if parsed.path != "/api/analyze": if user is None:
return
try:
if parsed.path.startswith("/api/jobs/") and parsed.path.endswith("/cancel"): if parsed.path.startswith("/api/jobs/") and parsed.path.endswith("/cancel"):
job_id = parsed.path.split("/")[-2] job_id = parsed.path.split("/")[-2]
if ACCOUNT_STORE.owner_id(job_id) != user["user_id"]:
self._respond(*json_bytes({"error": "Job not found"}, status=HTTPStatus.NOT_FOUND))
return
job = cancel_job(job_id) job = cancel_job(job_id)
if job is None: if job is None:
self._respond(*json_bytes({"error": "Job not found"}, status=HTTPStatus.NOT_FOUND)) self._respond(*json_bytes({"error": "Job not found"}, status=HTTPStatus.NOT_FOUND))
return return
job.pop("user_id", None)
self._respond(*json_bytes({"status": "success", "job": job})) self._respond(*json_bytes({"status": "success", "job": job}))
return return
if parsed.path == "/api/jobs":
try:
length = int(self.headers.get("Content-Length", "0"))
raw = self.rfile.read(length)
payload = json.loads(raw.decode("utf-8"))
job_id = create_job(payload)
self._respond(*json_bytes({"status": "accepted", "job_id": job_id}, status=HTTPStatus.ACCEPTED))
except Exception as exc:
self._respond(
*json_bytes(
{
"status": "error",
"error": str(exc),
"traceback": traceback.format_exc(),
},
status=HTTPStatus.BAD_REQUEST,
)
)
return
if parsed.path == "/api/shares":
try:
length = int(self.headers.get("Content-Length", "0"))
if length > SHARE_MAX_BYTES:
raise ValueError(f"Share payload is too large. Limit is {SHARE_MAX_BYTES} bytes.")
raw = self.rfile.read(length)
record = json.loads(raw.decode("utf-8"))
share = create_share(record)
self._respond(
*json_bytes(
{
"status": "success",
"share_id": share["id"],
"url": f"/?share={share['id']}",
"max_count": SHARE_MAX_COUNT,
},
status=HTTPStatus.CREATED,
)
)
except Exception as exc:
self._respond(
*json_bytes(
{
"status": "error",
"error": str(exc),
"traceback": traceback.format_exc(),
},
status=HTTPStatus.BAD_REQUEST,
)
)
return
self._respond(*json_bytes({"error": "Not found"}, status=HTTPStatus.NOT_FOUND))
return
try: if parsed.path.startswith("/api/jobs/") and parsed.path.endswith("/shares"):
length = int(self.headers.get("Content-Length", "0")) job_id = parsed.path.split("/")[-2]
raw = self.rfile.read(length) body = self._read_json()
payload = json.loads(raw.decode("utf-8")) expires_in = body.get("expires_in")
result = run_job_payload(payload) if expires_in not in {None, "", 0}:
self._respond(*json_bytes({"status": "success", "result": result})) expires_in = int(expires_in)
if expires_in < 60:
raise ValueError("Share duration must be at least 60 seconds.")
else:
expires_in = None
share = ACCOUNT_STORE.create_share(user["user_id"], job_id, expires_in)
self._respond(*json_bytes({"status": "success", "share": share, "url": f"/share/{share['share_id']}"}, status=HTTPStatus.CREATED))
return
if parsed.path == "/api/jobs":
payload = self._read_json()
job_id = create_job(payload, user)
self._respond(*json_bytes({"status": "accepted", "job_id": job_id}, status=HTTPStatus.ACCEPTED))
return
if parsed.path == "/api/history/import":
body = self._read_json(max_bytes=64 * 1024 * 1024)
entries = body.get("history") if isinstance(body, dict) else None
if not isinstance(entries, list):
raise ValueError("History import requires a history array.")
imported = ACCOUNT_STORE.import_history(user, entries)
self._respond(*json_bytes({"status": "success", "imported": imported}))
return
if parsed.path == "/api/shares":
body = self._read_json()
job_id = str(body.get("job_id") or "")
if not job_id:
raise ValueError("Cloud shares require a job_id.")
share = ACCOUNT_STORE.create_share(user["user_id"], job_id, body.get("expires_in"))
self._respond(*json_bytes({"status": "success", "share_id": share["share_id"], "url": f"/share/{share['share_id']}"}, status=HTTPStatus.CREATED))
return
if parsed.path == "/api/analyze":
payload = self._read_json()
job_id = uuid4().hex
started = time.time()
ACCOUNT_STORE.create_job(job_id, user, payload, status="running", created_at=started)
try:
result = run_job_payload(payload)
except Exception as exc:
elapsed = round(time.time() - started, 3)
ACCOUNT_STORE.update_job(job_id, "error", error={"message": str(exc), "traceback": traceback.format_exc()}, elapsed_seconds=elapsed)
raise
elapsed = round(time.time() - started, 3)
ACCOUNT_STORE.update_job(job_id, "success", result=result, elapsed_seconds=elapsed)
self._respond(*json_bytes({"status": "success", "job_id": job_id, "result": result}))
return
self._respond(*json_bytes({"error": "Not found"}, status=HTTPStatus.NOT_FOUND))
except Exception as exc: except Exception as exc:
self._respond( self._respond(
*json_bytes( *json_bytes(
@ -2215,6 +2424,53 @@ class AppHandler(BaseHTTPRequestHandler):
) )
) )
def do_PATCH(self):
parsed = urlparse(self.path)
user = self._require_user()
if user is None:
return
try:
if parsed.path.startswith("/api/account/shares/"):
share_id = parsed.path.rsplit("/", 1)[-1]
body = self._read_json()
expires_in = body["expires_in"] if "expires_in" in body else "unchanged"
share = ACCOUNT_STORE.update_share(
user["user_id"], share_id, active=body.get("active"), expires_in=expires_in
)
if share is None:
self._respond(*json_bytes({"error": "Share not found"}, status=HTTPStatus.NOT_FOUND))
return
self._respond(*json_bytes({"status": "success", "share": share}))
return
self._respond(*json_bytes({"error": "Not found"}, status=HTTPStatus.NOT_FOUND))
except Exception as exc:
self._respond(*json_bytes({"status": "error", "error": str(exc)}, status=HTTPStatus.BAD_REQUEST))
def do_DELETE(self):
parsed = urlparse(self.path)
user = self._require_user()
if user is None:
return
try:
if parsed.path.startswith("/api/history/"):
job_id = parsed.path.rsplit("/", 1)[-1]
if not ACCOUNT_STORE.delete_job(user["user_id"], job_id):
self._respond(*json_bytes({"error": "History record not found"}, status=HTTPStatus.NOT_FOUND))
return
self._respond(*json_bytes({"status": "success"}))
return
if parsed.path.startswith("/api/account/shares/"):
share_id = parsed.path.rsplit("/", 1)[-1]
share = ACCOUNT_STORE.update_share(user["user_id"], share_id, active=False)
if share is None:
self._respond(*json_bytes({"error": "Share not found"}, status=HTTPStatus.NOT_FOUND))
return
self._respond(*json_bytes({"status": "success", "share": share}))
return
self._respond(*json_bytes({"error": "Not found"}, status=HTTPStatus.NOT_FOUND))
except Exception as exc:
self._respond(*json_bytes({"status": "error", "error": str(exc)}, status=HTTPStatus.BAD_REQUEST))
def log_message(self, format_, *args): def log_message(self, format_, *args):
print(f"{self.address_string()} - {format_ % args}") print(f"{self.address_string()} - {format_ % args}")
@ -2245,6 +2501,7 @@ class AppHandler(BaseHTTPRequestHandler):
def main(): def main():
apply_thread_limits() apply_thread_limits()
ACCOUNT_STORE.initialize()
if RUN_MODE == "worker": if RUN_MODE == "worker":
run_worker_loop() run_worker_loop()
return return