feat(tools): 网页提取白名单直提 + recall 记忆纳入已读文件

- extract_webpage/save_webpage 命中白名单域名时本机直提：GitHub 代码页走
  jsDelivr CDN 直链（备 GitHub API），其余白名单页用 trafilatura 正文识别，
  失败自动回退 Tavily；内置白名单 github.com，个人空间可关闭/追加域名
- trafilatura 进 requirements（可选依赖，缺失时静默降级全走 Tavily）
- recall_project_memory 返回记忆全文后标记为已读，可直接 edit_file

Co-authored-by: Astrion powered by Kimi-K3 <astrion-agent@users.noreply.github.com>

AGENTS.md

@@ -647,3 +647,24 @@ Web 端实时通道曾长期双轨（REST 任务轮询为主 + Socket.IO 辅助
 2. 确认后端 formatter（`tool_result_formatter`）与前端 renderer（`toolRenderers.ts`）已覆盖该工具。
 3. 个人空间勾选 UI 自动出现（注册表经 `/api/personalization` 的 `tool_loading_registry` 下发，类目标签 i18n key `personalization.toolLoadingCat.<key>` 需双语补齐）。
 4. 跑 `test/test_tool_loading.py`（注册表完整性断言会校验数量与结构）。
+
+---
+
+## 14) 网页提取白名单直提（2026-09-12 新增）
+
+> 实现唯一权威：`modules/webpage_extractor.py`。
+
+### 14.1 机制一句话
+
+`extract_webpage` / `save_webpage` 不再全部走 Tavily：命中白名单的域名在本机直提（免费、零配额），失败或未命中才回退 Tavily。开关与追加域名在个人空间（工具页），存 personalization.json 的 `webpage_direct_extract_enabled`（默认 true）/ `webpage_direct_extract_domains`。
+
+### 14.2 硬约束（改代码必须知道）
+
+1. **统一入口是 `extract_single_url()`**：白名单判定 → `_direct_extract()` → 失败回退 `tavily_extract()`。两个工具调用点都走它，不要绕过单写 tavily 路径。
+2. **降级链**：GitHub blob 页 → jsDelivr CDN（`cdn.jsdelivr.net/gh/{owner}/{repo}@{branch}/{path}`，国内可用、无限速）→ GitHub API contents（备用，匿名限 60 次/时）→ trafilatura 通用提取 → Tavily。**不要用 raw.githubusercontent.com**（国内被 DNS 污染，实测不可达）。
+3. **trafilatura 是可选依赖**：`import` 失败时通用直提静默失效（GitHub 直链不依赖它仍可用），白名单不会报错；已进 `requirements.txt`，正式部署应装上。
+4. **blob URL 的 branch 按单段解析**（`[^/?#]+`）；含斜杠的分支名解析失败会自然降级 trafilatura → Tavily，属预期行为。
+5. **域名匹配 = 精确或子域名后缀**：`github.com` 命中 `gist.github.com`；用户输入经 `_normalize_domain` 清洗（容忍粘贴完整 URL）。
+6. **新增 personalization 键必须双注册**（`DEFAULT_PERSONALIZATION_CONFIG` + `sanitize_personalization_payload`），否则保存后被静默丢弃（见 .astrion/memory/personalization_config_whitelist.md）。
+7. **返回格式**沿用 `🌐 网页内容 (N 字符)` 骨架，新增 `[提取方式: ...]` 标注（i18n key `webpage.method_label` / `webpage.method_*`），模型侧无感知。
+8. **read-before-edit 已读白名单**含 `recall_project_memory`（recall 返回记忆文件全文，视为已读）；`search_project_memory` 只回片段，刻意不标记。

core/main_terminal_parts/tools_execution.py

@@ -72,7 +72,12 @@ from modules.memory_manager import MemoryManager
 from modules.terminal_manager import TerminalManager
 from modules.todo_manager import TodoManager
 from modules.sub_agent import SubAgentManager
-from modules.webpage_extractor import extract_webpage_content, tavily_extract
+from modules.webpage_extractor import (
+    extract_single_url,
+    extract_webpage_content,
+    resolve_direct_extract_config,
+    tavily_extract,
+)
 from modules.ocr_client import OCRClient
 from modules.easter_egg_manager import EasterEggManager
 from modules.personalization_manager import (
@@ -741,7 +746,9 @@ class MainTerminalToolsExecutionMixin:
                 return visited
 
     def _mark_file_read_from_result(self, tool_name: str, arguments: Dict[str, Any], result: Dict[str, Any]) -> None:
-                if tool_name not in {"read_file", "read_skill"}:
+                # recall_project_memory 内部即读取记忆文件全文（走 _handle_read_tool），
+                # 视为已读，避免后续 edit_file 被要求重复 read_file。
+                if tool_name not in {"read_file", "read_skill", "recall_project_memory"}:
                     return
                 if not isinstance(result, dict) or not result.get("success"):
                     return
@@ -1768,11 +1775,16 @@ class MainTerminalToolsExecutionMixin:
                         try:
                             # 从config获取API密钥
                             from config import TAVILY_API_KEY
+                            try:
+                                _prefs = load_personalization_config(self.data_dir) or {}
+                            except Exception:
+                                _prefs = {}
                             full_content, _ = await extract_webpage_content(
-                                urls=url, 
+                                urls=url,
                                 api_key=TAVILY_API_KEY,
                                 extract_depth="basic",
-                                max_urls=1
+                                max_urls=1,
+                                direct_config=resolve_direct_extract_config(_prefs),
                             )
 
                             # 字符数检查
@@ -1815,92 +1827,57 @@ class MainTerminalToolsExecutionMixin:
                         except ImportError:
                             TAVILY_API_KEY = None
 
-                        if not TAVILY_API_KEY or TAVILY_API_KEY == "your-tavily-api-key":
+                        # 白名单直提优先（无需 Tavily key）；未命中/失败自动回退 Tavily
-                            result = {
+                        try:
-                                "success": False,
-                                "error": tr("tools_exec.tavily_key_missing"),
-                                "url": url,
-                                "path": target_path
-                            }
-                        else:
                             try:
-                                extract_result = await tavily_extract(
+                                _prefs = load_personalization_config(self.data_dir) or {}
-                                    urls=url,
+                            except Exception:
-                                    api_key=TAVILY_API_KEY,
+                                _prefs = {}
-                                    extract_depth="basic",
+                            extract_one = await extract_single_url(
-                                    max_urls=1
+                                url,
-                                )
+                                TAVILY_API_KEY,
+                                extract_depth="basic",
+                                direct_config=resolve_direct_extract_config(_prefs),
+                            )
 
-                                if not extract_result or "error" in extract_result:
+                            if not extract_one.get("success"):
-                                    error_message = extract_result.get("error", tr("tools_exec.extract_failed_no_content")) if isinstance(extract_result, dict) else tr("tools_exec.extract_failed")
+                                result = {
+                                    "success": False,
+                                    "error": extract_one.get("error", tr("tools_exec.extract_failed_no_content")),
+                                    "url": url,
+                                    "path": target_path
+                                }
+                            else:
+                                content_to_save = extract_one.get("content") or ""
+                                write_result = self.file_manager.write_file(target_path, content_to_save, mode="w")
+
+                                if not write_result.get("success"):
                                     result = {
                                         "success": False,
-                                        "error": error_message,
+                                        "error": write_result.get("error", tr("tools_exec.write_file_failed")),
                                         "url": url,
                                         "path": target_path
                                     }
                                 else:
-                                    results_list = extract_result.get("results", []) if isinstance(extract_result, dict) else []
+                                    char_count = len(content_to_save)
+                                    byte_size = len(content_to_save.encode("utf-8"))
+                                    result = {
+                                        "success": True,
+                                        "url": url,
+                                        "path": write_result.get("path", target_path),
+                                        "char_count": char_count,
+                                        "byte_size": byte_size,
+                                        "extract_method": extract_one.get("method"),
+                                        "message": tr("tools_exec.webpage_saved", path=write_result.get('path', target_path))
+                                    }
 
-                                    primary_result = None
+                        except Exception as e:
-                                    for item in results_list:
+                            result = {
-                                        if item.get("raw_content"):
+                                "success": False,
-                                            primary_result = item
+                                "error": tr("tools_exec.webpage_save_failed", error=str(e)),
-                                            break
+                                "url": url,
-                                    if primary_result is None and results_list:
+                                "path": target_path
-                                        primary_result = results_list[0]
+                            }
-
-                                    if not primary_result:
-                                        failed_list = extract_result.get("failed_results", []) if isinstance(extract_result, dict) else []
-                                        result = {
-                                            "success": False,
-                                            "error": tr("tools_exec.extract_result_empty"),
-                                            "url": url,
-                                            "path": target_path,
-                                            "failed": failed_list
-                                        }
-                                    else:
-                                        content_to_save = primary_result.get("raw_content") or primary_result.get("content") or ""
-
-                                        if not content_to_save:
-                                            result = {
-                                                "success": False,
-                                                "error": tr("tools_exec.webpage_content_empty"),
-                                                "url": url,
-                                                "path": target_path
-                                            }
-                                        else:
-                                            write_result = self.file_manager.write_file(target_path, content_to_save, mode="w")
-
-                                            if not write_result.get("success"):
-                                                result = {
-                                                    "success": False,
-                                                    "error": write_result.get("error", tr("tools_exec.write_file_failed")),
-                                                    "url": url,
-                                                    "path": target_path
-                                                }
-                                            else:
-                                                char_count = len(content_to_save)
-                                                byte_size = len(content_to_save.encode("utf-8"))
-                                                result = {
-                                                    "success": True,
-                                                    "url": url,
-                                                    "path": write_result.get("path", target_path),
-                                                    "char_count": char_count,
-                                                    "byte_size": byte_size,
-                                                    "message": tr("tools_exec.webpage_saved", path=write_result.get('path', target_path))
-                                                }
-
-                                                if isinstance(extract_result, dict) and extract_result.get("failed_results"):
-                                                    result["warnings"] = extract_result["failed_results"]
-
-                            except Exception as e:
-                                result = {
-                                    "success": False,
-                                    "error": tr("tools_exec.webpage_save_failed", error=str(e)),
-                                    "url": url,
-                                    "path": target_path
-                                }
 
                     elif tool_name == "run_command":
                         permission_mode = "unrestricted"

modules/i18n_messages/modules_misc.py

@@ -214,6 +214,26 @@ MESSAGES = {
         "zh-CN": "❌ 未能提取到任何内容",
         "en-US": "❌ No content could be extracted",
     },
+    "webpage.method_label": {
+        "zh-CN": "提取方式: {method}",
+        "en-US": "Extraction method: {method}",
+    },
+    "webpage.method_jsdelivr": {
+        "zh-CN": "直连 jsDelivr",
+        "en-US": "Direct jsDelivr",
+    },
+    "webpage.method_github_api": {
+        "zh-CN": "直连 GitHub API",
+        "en-US": "Direct GitHub API",
+    },
+    "webpage.method_trafilatura": {
+        "zh-CN": "直连 trafilatura",
+        "en-US": "Direct trafilatura",
+    },
+    "webpage.method_tavily": {
+        "zh-CN": "Tavily",
+        "en-US": "Tavily",
+    },
 
     # ── mcp_client_manager（manager.py + http_client.py） ──
     "mcp.server_not_found": {

modules/personalization_manager.py

@@ -3,9 +3,10 @@
 from __future__ import annotations
 
 import json
+import re
 from copy import deepcopy
 from pathlib import Path
-from typing import Any, Dict, Optional, Union
+from typing import Any, Dict, List, Optional, Union
 
 try:
     from config.limits import REASONING_EFFORT_LEVELS
@@ -31,6 +32,9 @@ ALLOWED_UI_LOCALES = {"zh-CN", "en-US"}
 GOAL_MAX_TURNS_MIN = 1
 GOAL_MAX_TURNS_MAX = 100
 GOAL_MAX_TURNS_DEFAULT = 5
+
+# 网页提取直提白名单：用户可追加的域名数量上限
+MAX_DIRECT_EXTRACT_DOMAINS = 50
 GOAL_MAX_TOKENS_MIN = 1_000
 GOAL_MAX_TOKENS_MAX = 100_000_000
 
@@ -98,6 +102,10 @@ DEFAULT_PERSONALIZATION_CONFIG: Dict[str, Any] = {
     # 仅在创建对话时快照一次；已有对话以其对话文件中的快照为准，改这里不影响。
     "tool_loading_enabled": True,
     "tool_loading_deferred": None,
+    # 网页提取白名单直提：命中白名单的域名用本机直提（GitHub 直链 / trafilatura），
+    # 未命中才走 Tavily；开关默认开启，domains 为用户追加的白名单域名（内置 github.com 不在此列）。
+    "webpage_direct_extract_enabled": True,
+    "webpage_direct_extract_domains": [],
     "default_model": None,
     "image_compression": "original",  # original / 1080p / 720p / 540p
     "auto_shallow_compress_enabled": False,
@@ -754,6 +762,27 @@ def sanitize_personalization_payload(
     else:
         base["review_agents"] = _sanitize_review_agents(base.get("review_agents"))
 
+    # 网页提取白名单直提：开关（默认开启）+ 用户追加的域名列表
+    if "webpage_direct_extract_enabled" in data:
+        base["webpage_direct_extract_enabled"] = bool(data.get("webpage_direct_extract_enabled"))
+    else:
+        base["webpage_direct_extract_enabled"] = bool(base.get("webpage_direct_extract_enabled", True))
+
+    raw_domains = data.get("webpage_direct_extract_domains", base.get("webpage_direct_extract_domains"))
+    if not isinstance(raw_domains, list):
+        raw_domains = []
+    clean_domains: List[str] = []
+    for item in raw_domains[:MAX_DIRECT_EXTRACT_DOMAINS]:
+        if not isinstance(item, str):
+            continue
+        d = item.strip().lower().strip(".")
+        if "//" in d:
+            d = d.split("//", 1)[1]
+        d = d.split("/")[0].split("?")[0]
+        if d and re.fullmatch(r"[a-z0-9.-]+", d) and "." in d and d not in clean_domains:
+            clean_domains.append(d)
+    base["webpage_direct_extract_domains"] = clean_domains
+
     return base
 
 

modules/webpage_extractor.py

@@ -1,24 +1,52 @@
 # modules/webpage_extractor.py - 网页内容提取模块
+#
+# 提取分两层：
+# 1. 白名单域名走本机直提（免费、零配额）：GitHub 代码文件页有专属直链适配
+#    （jsDelivr CDN → GitHub API 备用），其余白名单页面用 trafilatura 提取正文。
+# 2. 未命中白名单（或直提失败）回退 Tavily 云端提取。
+#
+# trafilatura 为可选依赖：未安装时通用直提静默失效，仅 GitHub 直链仍可用。
+
+import base64
+import re
+from typing import Any, Dict, List, Optional, Tuple, Union
+from urllib.parse import urlparse
 
 import httpx
-import json
-from typing import Dict, Any, List, Union, Tuple
-from utils.logger import setup_logger
 
+from utils.logger import setup_logger
 from modules.i18n import tr
 
 logger = setup_logger(__name__)
 
+try:
+    import trafilatura as _trafilatura
+except ImportError:  # 可选依赖
+    _trafilatura = None
+
+# 内置直提白名单域名（个人空间可追加；子域名自动匹配）
+BUILTIN_DIRECT_EXTRACT_DOMAINS: Tuple[str, ...] = ("github.com",)
+
+_DIRECT_REQUEST_HEADERS = {
+    "User-Agent": (
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
+    ),
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+}
+_DIRECT_TIMEOUT = 30
+
+
 async def tavily_extract(urls: Union[str, List[str]], api_key: str, extract_depth: str = "basic", max_urls: int = 1) -> Dict[str, Any]:
     """
     执行Tavily网页内容提取
-    
+
     Args:
         urls: 要提取的URL（字符串或列表）
         api_key: Tavily API密钥
         extract_depth: 提取深度 (basic/advanced)
         max_urls: 最大提取URL数量
-    
+
     Returns:
         提取结果字典
     """
@@ -65,10 +93,10 @@ async def tavily_extract(urls: Union[str, List[str]], api_key: str, extract_dept
 def format_extract_results(results: Dict[str, Any]) -> str:
     """
     格式化提取结果为简洁版本
-    
+
     Args:
         results: tavily_extract返回的结果
-    
+
     Returns:
         格式化后的内容字符串
     """
@@ -79,12 +107,12 @@ def format_extract_results(results: Dict[str, Any]) -> str:
         return tr("webpage.no_content")
 
     formatted_parts = []
-    
+
     # 成功提取的结果
     for i, result in enumerate(results["results"], 1):
         url = result.get("url", "N/A")
         raw_content = result.get("raw_content", "").strip()
-        
+
         if raw_content:
             content_length = len(raw_content)
             formatted_parts.append(f"🌐 网页内容 ({content_length} 字符):")
@@ -104,24 +132,238 @@ def format_extract_results(results: Dict[str, Any]) -> str:
     return "\n".join(formatted_parts)
 
 
-async def extract_webpage_content(urls: Union[str, List[str]], api_key: str, extract_depth: str = "basic", max_urls: int = 1) -> Tuple[str, str]:
+# ============================================================
+# 白名单直提
+# ============================================================
+
+def _normalize_domain(raw: Any) -> str:
+    """把用户输入规范化为小写裸域名（容忍粘贴完整 URL / 前导点 / 路径）。"""
+    if not isinstance(raw, str):
+        return ""
+    d = raw.strip().lower()
+    if not d:
+        return ""
+    if "://" in d:
+        d = _url_hostname(d) or d
+    d = d.split("/")[0].split("?")[0].strip(".")
+    if not re.fullmatch(r"[a-z0-9.-]+", d or "") or "." not in d:
+        return ""
+    return d
+
+
+def _url_hostname(url: str) -> str:
+    try:
+        return (urlparse(str(url)).hostname or "").lower()
+    except Exception:
+        return ""
+
+
+def resolve_direct_extract_config(personalization: Optional[Dict[str, Any]]) -> Dict[str, Any]:
+    """从 personalization 配置解析直提设置。
+
+    Returns:
+        {"enabled": bool, "domains": [...]} —— domains 已合并内置域名并规范化。
     """
-    完整的网页内容提取流程
+    personalization = personalization or {}
-    
+    enabled = bool(personalization.get("webpage_direct_extract_enabled", True))
+    extra = personalization.get("webpage_direct_extract_domains")
+    if not isinstance(extra, list):
+        extra = []
+    domains: List[str] = []
+    for item in list(BUILTIN_DIRECT_EXTRACT_DOMAINS) + extra:
+        nd = _normalize_domain(item)
+        if nd and nd not in domains:
+            domains.append(nd)
+    return {"enabled": enabled, "domains": domains}
+
+
+def is_whitelisted_url(url: str, domains: List[str]) -> bool:
+    """域名等于白名单条目或为其子域名即命中。"""
+    host = _url_hostname(url)
+    if not host:
+        return False
+    return any(host == d or host.endswith("." + d) for d in domains)
+
+
+_GITHUB_BLOB_RE = re.compile(
+    r"^https?://(?:www\.)?github\.com/([^/?#]+)/([^/?#]+)/blob/([^/?#]+)/([^?#]+?)(?:[?#].*)?$",
+    re.IGNORECASE,
+)
+
+
+def parse_github_blob_url(url: str) -> Optional[Tuple[str, str, str, str]]:
+    """解析 GitHub blob 页面 URL，返回 (owner, repo, branch, path)；非 blob 页返回 None。
+
+    注意：branch 按单段处理（覆盖 main/master 等常见场景）；含斜杠的分支名
+    会解析失败并自然降级到通用提取，不会报错。
+    """
+    m = _GITHUB_BLOB_RE.match(str(url).strip())
+    if not m:
+        return None
+    return m.group(1), m.group(2), m.group(3), m.group(4)
+
+
+async def _fetch_jsdelivr_raw(client: httpx.AsyncClient, owner: str, repo: str, branch: str, path: str) -> Optional[str]:
+    """经 jsDelivr CDN 拿 GitHub 文件原文（免费、无限速，替代被墙的 raw.githubusercontent.com）。"""
+    cdn_url = f"https://cdn.jsdelivr.net/gh/{owner}/{repo}@{branch}/{path}"
+    try:
+        resp = await client.get(cdn_url, timeout=_DIRECT_TIMEOUT)
+        if resp.status_code == 200 and resp.text:
+            return resp.text
+        logger.info(f"jsDelivr 直链返回 {resp.status_code}: {cdn_url}")
+    except Exception as e:
+        logger.info(f"jsDelivr 直链失败 {cdn_url}: {e}")
+    return None
+
+
+async def _fetch_github_api_raw(client: httpx.AsyncClient, owner: str, repo: str, branch: str, path: str) -> Optional[str]:
+    """经 GitHub 官方 contents API 拿文件原文（备用；匿名限 60 次/小时，>1MB 文件不返回内容）。"""
+    api_url = f"https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={branch}"
+    try:
+        resp = await client.get(api_url, headers={"Accept": "application/vnd.github+json"}, timeout=_DIRECT_TIMEOUT)
+        if resp.status_code != 200:
+            logger.info(f"GitHub API 返回 {resp.status_code}: {api_url}")
+            return None
+        data = resp.json()
+        if isinstance(data, dict) and data.get("encoding") == "base64" and data.get("content"):
+            return base64.b64decode(data["content"]).decode("utf-8", errors="replace")
+    except Exception as e:
+        logger.info(f"GitHub API 失败 {api_url}: {e}")
+    return None
+
+
+def _trafilatura_extract(html: str, url: str) -> Optional[str]:
+    """trafilatura 通用正文提取（markdown 输出，保留标题/代码块结构）。"""
+    if _trafilatura is None:
+        return None
+    try:
+        text = _trafilatura.extract(
+            html,
+            url=url,
+            output_format="markdown",
+            include_links=False,
+            include_images=False,
+        )
+        if text and text.strip():
+            return text.strip()
+    except Exception as e:
+        logger.info(f"trafilatura 提取失败 {url}: {e}")
+    return None
+
+
+async def _direct_extract(client: httpx.AsyncClient, url: str) -> Tuple[Optional[str], str]:
+    """白名单直提主流程。返回 (内容, method)；全部失败返回 (None, "")。"""
+    blob = parse_github_blob_url(url)
+    if blob:
+        owner, repo, branch, path = blob
+        content = await _fetch_jsdelivr_raw(client, owner, repo, branch, path)
+        if content is not None:
+            return content, "jsdelivr"
+        content = await _fetch_github_api_raw(client, owner, repo, branch, path)
+        if content is not None:
+            return content, "github_api"
+        # 直链均失败 → 继续走通用提取兜底
+
+    if _trafilatura is not None:
+        try:
+            resp = await client.get(url, timeout=_DIRECT_TIMEOUT)
+            if resp.status_code == 200 and resp.text:
+                text = _trafilatura_extract(resp.text, url)
+                if text:
+                    return text, "trafilatura"
+            else:
+                logger.info(f"直提抓取返回 {resp.status_code}: {url}")
+        except Exception as e:
+            logger.info(f"直提抓取失败 {url}: {e}")
+    return None, ""
+
+
+def _format_single_result(url: str, content: str, method: Optional[str] = None) -> str:
+    """格式化单条提取结果（沿用 🌐 骨架，附提取方式标注）。"""
+    header = f"🌐 网页内容 ({len(content)} 字符)"
+    if method:
+        header += f" [{tr('webpage.method_label', method=tr(f'webpage.method_{method}'))}]"
+    return "\n".join([
+        header + ":",
+        f"📍 URL: {url}",
+        "=" * 50,
+        content,
+        "=" * 50,
+    ])
+
+
+async def extract_single_url(
+    url: str,
+    api_key: Optional[str],
+    extract_depth: str = "basic",
+    direct_config: Optional[Dict[str, Any]] = None,
+) -> Dict[str, Any]:
+    """统一单 URL 提取：白名单直提优先，失败/未命中回退 Tavily。
+
+    Args:
+        url: 目标 URL
+        api_key: Tavily API 密钥（可为 None；直提命中时不需要）
+        extract_depth: Tavily 提取深度
+        direct_config: resolve_direct_extract_config() 的返回；None 视为关闭直提
+
+    Returns:
+        {"success": bool, "url": str, "content": str, "method": str} 或
+        {"success": False, "url": str, "error": str, "method": str}
+    """
+    direct_config = direct_config or {}
+
+    if direct_config.get("enabled") and is_whitelisted_url(url, direct_config.get("domains") or []):
+        content, method = None, ""
+        try:
+            async with httpx.AsyncClient(headers=_DIRECT_REQUEST_HEADERS, follow_redirects=True) as client:
+                content, method = await _direct_extract(client, url)
+        except Exception as e:
+            logger.info(f"白名单直提异常 {url}: {e}")
+        if content is not None:
+            return {"success": True, "url": url, "content": content, "method": method}
+        logger.info(f"白名单直提未取到内容，回退 Tavily: {url}")
+
+    results = await tavily_extract(url, api_key, extract_depth, 1)
+    if "error" in results:
+        return {"success": False, "url": url, "error": results["error"], "method": "tavily"}
+    for item in results.get("results") or []:
+        raw = (item.get("raw_content") or "").strip()
+        if raw:
+            return {"success": True, "url": url, "content": raw, "method": "tavily"}
+    return {"success": False, "url": url, "error": tr("webpage.no_content"), "method": "tavily"}
+
+
+async def extract_webpage_content(
+    urls: Union[str, List[str]],
+    api_key: str,
+    extract_depth: str = "basic",
+    max_urls: int = 1,
+    direct_config: Optional[Dict[str, Any]] = None,
+) -> Tuple[str, str]:
+    """
+    完整的网页内容提取流程（白名单直提优先，Tavily 兜底）
+
     Args:
         urls: 要提取的URL（字符串或列表）
         api_key: Tavily API密钥
         extract_depth: 提取深度 (basic/advanced)
         max_urls: 最大提取URL数量
-    
+        direct_config: 直提配置（resolve_direct_extract_config 返回）；None=关闭直提
+
     Returns:
         (完整内容, 完整内容) - 为了兼容性返回相同内容两份
     """
-    # 执行提取
+    if isinstance(urls, str):
-    results = await tavily_extract(urls, api_key, extract_depth, max_urls)
+        urls = [urls]
-    
+    urls = urls[:max_urls]
-    # 格式化结果
+
-    formatted_content = format_extract_results(results)
+    formatted_parts: List[str] = []
-    
+    for url in urls:
-    # 返回相同内容（简化版本，不需要长短版本区分）
+        result = await extract_single_url(url, api_key, extract_depth=extract_depth, direct_config=direct_config)
-    return formatted_content, formatted_content
+        if result.get("success"):
+            formatted_parts.append(_format_single_result(url, result["content"], result.get("method")))
+        else:
+            formatted_parts.append(tr("webpage.format_failed", error=result.get("error")))
+
+    formatted_content = "\n".join(formatted_parts)
+    return formatted_content, formatted_content

requirements.txt

@@ -8,3 +8,6 @@ openai
 cryptography
 pillow
 pyyaml>=6.0
+# 网页白名单直提的正文识别引擎（可选依赖：import 失败时代码自动降级为全走 Tavily，
+# 但正式部署应装上，否则直提白名单只剩 GitHub 直链可用）
+trafilatura

static/src/components/personalization/styles/settings-shared.css

@@ -561,6 +561,65 @@ body[data-theme='dark'] .settings-floating-menu {
   gap: 8px;
 }
 
+/* 直提白名单域名列表：分隔线区分行，不套圆角卡片 */
+.settings-domain-list {
+  display: flex;
+  flex-direction: column;
+  margin: 4px 0 10px;
+}
+
+.settings-domain-row {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  gap: 8px;
+  height: 36px;
+  padding: 0 2px;
+  border-bottom: 1px solid var(--theme-control-border);
+}
+
+.settings-domain-row:last-child {
+  border-bottom: none;
+}
+
+.settings-domain-row .settings-row-title {
+  min-width: 0;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+}
+
+.settings-domain-badge {
+  flex: none;
+  display: inline-flex;
+  align-items: center;
+  height: 28px; /* 与 .settings-domain-remove 同高同居中机制，保证文字视觉对齐 */
+  margin-right: 10px; /* 抵消删除按钮的水平 padding，使两者文字右缘对齐 */
+  font-size: 12px;
+  line-height: 1;
+  color: var(--text-secondary);
+}
+
+.settings-domain-remove {
+  flex: none;
+  height: 28px;
+  padding: 0 10px;
+  border: none;
+  border-radius: 6px;
+  background: transparent;
+  color: var(--text-secondary);
+  font-size: 12px;
+  line-height: 1;
+  cursor: pointer;
+}
+
+@media (hover: hover) {
+  .settings-domain-remove:hover {
+    background: var(--hover-bg);
+    color: var(--state-warning);
+  }
+}
+
 .settings-add-row input,
 .settings-number-row input {
   min-width: 0;

static/src/components/personalization/tabs/ToolsTab.vue

@@ -1,5 +1,5 @@
 <script setup lang="ts">
-import { inject, unref } from 'vue';
+import { inject, unref, ref } from 'vue';
 import FancyCheck from '@/components/common/FancyCheck.vue';
 
 defineOptions({ name: 'ToolsTab' });
@@ -58,6 +58,43 @@ const toggleDeferredCategory = (cat: { tools: string[] }, checked: boolean) => {
   }
   personalization.updateField({ key: 'tool_loading_deferred', value: Array.from(current) });
 };
+
+/** 追加的直提白名单域名（内置 github.com 不在此列，直接展示内置徽标） */
+const directExtractDomains = (): string[] => {
+  const f: any = unref(form);
+  return Array.isArray(f?.webpage_direct_extract_domains) ? f.webpage_direct_extract_domains : [];
+};
+
+/** 新增域名输入框的本地草稿 */
+const newDirectDomain = ref('');
+
+const addDirectDomain = () => {
+  let value = newDirectDomain.value.trim().toLowerCase();
+  if (!value) return;
+  // 容忍粘贴完整 URL：提取 hostname 部分
+  if (value.includes('://')) {
+    try {
+      value = new URL(value).hostname;
+    } catch {
+      return;
+    }
+  }
+  value = value.split('/')[0].replace(/^\.+|\.+$/g, '');
+  if (!value || !value.includes('.')) return;
+  const current = [...directExtractDomains()];
+  if (!current.includes(value)) {
+    current.push(value);
+    personalization.updateField({ key: 'webpage_direct_extract_domains', value: current });
+  }
+  newDirectDomain.value = '';
+};
+
+const removeDirectDomain = (domain: string) => {
+  personalization.updateField({
+    key: 'webpage_direct_extract_domains',
+    value: directExtractDomains().filter((d) => d !== domain)
+  });
+};
 </script>
 
 <template>
@@ -208,6 +245,54 @@ const toggleDeferredCategory = (cat: { tools: string[] }, checked: boolean) => {
                               @change="toggleCategory(category.id)" /><FancyCheck :checked="form.disabled_tool_categories.includes(category.id)" /></label>
                         </div>
                       </div>
+                      <div class="settings-group-block">
+                        <div class="settings-group-title">
+                          <span class="settings-row-title">{{ $t('personalization.webDirectExtractTitle') }}</span
+                          ><span class="settings-row-desc">{{ $t('personalization.webDirectExtractDesc') }}</span>
+                        </div>
+                        <label class="settings-toggle-row inner"
+                          ><span class="settings-row-title">{{ $t('personalization.webDirectExtractEnabledTitle') }}</span
+                          ><input
+                            type="checkbox"
+                            :checked="form.webpage_direct_extract_enabled"
+                            @change="
+                              personalization.updateField({
+                                key: 'webpage_direct_extract_enabled',
+                                value: $event.target.checked
+                              })
+                            " /><FancyCheck :checked="form.webpage_direct_extract_enabled" /></label>
+                        <template v-if="form.webpage_direct_extract_enabled">
+                          <div class="settings-domain-list">
+                            <div class="settings-domain-row">
+                              <span class="settings-row-title">github.com</span
+                              ><span class="settings-domain-badge">{{
+                                $t('personalization.webDirectExtractBuiltinBadge')
+                              }}</span>
+                            </div>
+                            <div v-for="domain in directExtractDomains()" :key="domain" class="settings-domain-row">
+                              <span class="settings-row-title">{{ domain }}</span
+                              ><button
+                                type="button"
+                                class="settings-domain-remove"
+                                @click="removeDirectDomain(domain)"
+                              >
+                                {{ $t('common.delete') }}
+                              </button>
+                            </div>
+                          </div>
+                          <div class="settings-add-row">
+                            <input
+                              v-model="newDirectDomain"
+                              type="text"
+                              :placeholder="$t('personalization.webDirectExtractDomainPlaceholder')"
+                              @keydown.enter="addDirectDomain"
+                            />
+                            <button type="button" class="settings-secondary-button" @click="addDirectDomain">
+                              {{ $t('personalization.webDirectExtractAdd') }}
+                            </button>
+                          </div>
+                        </template>
+                      </div>
                       <div class="settings-group-block" v-if="toolLoadingRegistry.length">
                         <div class="settings-group-title">
                           <span class="settings-row-title">{{ $t('personalization.toolLoadingTitle') }}</span

static/src/locales/en-US/personalization.ts

@@ -253,6 +253,14 @@ export default {
     mcp: 'MCP',
     misc: 'Easter egg'
   },
+  webDirectExtractTitle: 'Direct extraction whitelist',
+  webDirectExtractDesc:
+    'Whitelisted sites are extracted locally (GitHub code files via CDN raw links, others via readability-style parsing) at no Tavily quota cost; falls back to Tavily automatically on failure',
+  webDirectExtractEnabledTitle: 'Enable whitelist direct extraction',
+  webDirectExtractDomainsTitle: 'Additional whitelisted domains',
+  webDirectExtractBuiltinBadge: 'Built-in',
+  webDirectExtractDomainPlaceholder: 'Enter a domain, e.g. docs.python.org',
+  webDirectExtractAdd: 'Add',
 
   // ── Files & Images ──
   imageCompressionTitle: 'Image compression',

static/src/locales/zh-CN/personalization.ts

@@ -254,6 +254,14 @@ export default {
     mcp: 'MCP',
     misc: '彩蛋'
   },
+  webDirectExtractTitle: '网页直提白名单',
+  webDirectExtractDesc:
+    '命中白名单的网站直接在本机提取内容（GitHub 代码文件走 CDN 直链，其余用正文识别），无需消耗 Tavily 配额；提取失败会自动回退 Tavily',
+  webDirectExtractEnabledTitle: '启用白名单直提',
+  webDirectExtractDomainsTitle: '追加白名单域名',
+  webDirectExtractBuiltinBadge: '内置',
+  webDirectExtractDomainPlaceholder: '输入域名，如 docs.python.org',
+  webDirectExtractAdd: '添加',
 
   // ── 文件与图片 ──
   imageCompressionTitle: '图片压缩',

static/src/stores/personalization.ts

@@ -103,6 +103,10 @@ interface PersonalForm {
   tool_loading_enabled: boolean;
   /** 默认延迟加载的工具名列表（仅影响新建对话） */
   tool_loading_deferred: string[];
+  /** 网页提取白名单直提开关（默认开启） */
+  webpage_direct_extract_enabled: boolean;
+  /** 用户追加的直提白名单域名（内置 github.com 不在此列） */
+  webpage_direct_extract_domains: string[];
   skill_hints_enabled: boolean;
   skill_strict_terminal_enabled: boolean;
   skill_strict_sub_agent_enabled: boolean;
@@ -314,6 +318,8 @@ const defaultForm = (): PersonalForm => ({
   tool_intent_enabled: true,
   tool_loading_enabled: true,
   tool_loading_deferred: [],
+  webpage_direct_extract_enabled: true,
+  webpage_direct_extract_domains: [],
   skill_hints_enabled: false,
   skill_strict_terminal_enabled: false,
   skill_strict_sub_agent_enabled: false,
@@ -540,6 +546,12 @@ export const usePersonalizationStore = defineStore('personalization', {
         tool_loading_deferred: Array.isArray(data.tool_loading_deferred)
           ? data.tool_loading_deferred.filter((item: any) => typeof item === 'string')
           : [],
+        webpage_direct_extract_enabled: data.webpage_direct_extract_enabled !== false,
+        webpage_direct_extract_domains: Array.isArray(data.webpage_direct_extract_domains)
+          ? data.webpage_direct_extract_domains.filter(
+              (item: any) => typeof item === 'string' && item.trim()
+            )
+          : [],
         skill_hints_enabled: !!data.skill_hints_enabled,
         skill_strict_terminal_enabled: !!data.skill_strict_terminal_enabled,
         skill_strict_sub_agent_enabled: !!data.skill_strict_sub_agent_enabled,