feat(tools): 网页提取白名单直提 + recall 记忆纳入已读文件
- extract_webpage/save_webpage 命中白名单域名时本机直提：GitHub 代码页走 jsDelivr CDN 直链（备 GitHub API），其余白名单页用 trafilatura 正文识别， 失败自动回退 Tavily；内置白名单 github.com，个人空间可关闭/追加域名 - trafilatura 进 requirements（可选依赖，缺失时静默降级全走 Tavily） - recall_project_memory 返回记忆全文后标记为已读，可直接 edit_file Co-authored-by: Astrion powered by Kimi-K3 <astrion-agent@users.noreply.github.com>
This commit is contained in:
parent
281be248c2
commit
c76431a8f1
21
AGENTS.md
21
AGENTS.md
@ -647,3 +647,24 @@ Web 端实时通道曾长期双轨（REST 任务轮询为主 + Socket.IO 辅助
|
||||
2. 确认后端 formatter（`tool_result_formatter`）与前端 renderer（`toolRenderers.ts`）已覆盖该工具。
|
||||
3. 个人空间勾选 UI 自动出现（注册表经 `/api/personalization` 的 `tool_loading_registry` 下发，类目标签 i18n key `personalization.toolLoadingCat.<key>` 需双语补齐）。
|
||||
4. 跑 `test/test_tool_loading.py`（注册表完整性断言会校验数量与结构）。
|
||||
|
||||
---
|
||||
|
||||
## 14) 网页提取白名单直提（2026-09-12 新增）
|
||||
|
||||
> 实现唯一权威：`modules/webpage_extractor.py`。
|
||||
|
||||
### 14.1 机制一句话
|
||||
|
||||
`extract_webpage` / `save_webpage` 不再全部走 Tavily：命中白名单的域名在本机直提（免费、零配额），失败或未命中才回退 Tavily。开关与追加域名在个人空间（工具页），存 personalization.json 的 `webpage_direct_extract_enabled`（默认 true）/ `webpage_direct_extract_domains`。
|
||||
|
||||
### 14.2 硬约束（改代码必须知道）
|
||||
|
||||
1. **统一入口是 `extract_single_url()`**：白名单判定 → `_direct_extract()` → 失败回退 `tavily_extract()`。两个工具调用点都走它，不要绕过单写 tavily 路径。
|
||||
2. **降级链**：GitHub blob 页 → jsDelivr CDN（`cdn.jsdelivr.net/gh/{owner}/{repo}@{branch}/{path}`，国内可用、无限速）→ GitHub API contents（备用，匿名限 60 次/时）→ trafilatura 通用提取 → Tavily。**不要用 raw.githubusercontent.com**（国内被 DNS 污染，实测不可达）。
|
||||
3. **trafilatura 是可选依赖**：`import` 失败时通用直提静默失效（GitHub 直链不依赖它仍可用），白名单不会报错；已进 `requirements.txt`，正式部署应装上。
|
||||
4. **blob URL 的 branch 按单段解析**（`[^/?#]+`）；含斜杠的分支名解析失败会自然降级 trafilatura → Tavily，属预期行为。
|
||||
5. **域名匹配 = 精确或子域名后缀**：`github.com` 命中 `gist.github.com`；用户输入经 `_normalize_domain` 清洗（容忍粘贴完整 URL）。
|
||||
6. **新增 personalization 键必须双注册**（`DEFAULT_PERSONALIZATION_CONFIG` + `sanitize_personalization_payload`），否则保存后被静默丢弃（见 .astrion/memory/personalization_config_whitelist.md）。
|
||||
7. **返回格式**沿用 `🌐 网页内容 (N 字符)` 骨架，新增 `[提取方式: ...]` 标注（i18n key `webpage.method_label` / `webpage.method_*`），模型侧无感知。
|
||||
8. **read-before-edit 已读白名单**含 `recall_project_memory`（recall 返回记忆文件全文，视为已读）；`search_project_memory` 只回片段，刻意不标记。
|
||||
|
||||
@ -72,7 +72,12 @@ from modules.memory_manager import MemoryManager
|
||||
from modules.terminal_manager import TerminalManager
|
||||
from modules.todo_manager import TodoManager
|
||||
from modules.sub_agent import SubAgentManager
|
||||
from modules.webpage_extractor import extract_webpage_content, tavily_extract
|
||||
from modules.webpage_extractor import (
|
||||
extract_single_url,
|
||||
extract_webpage_content,
|
||||
resolve_direct_extract_config,
|
||||
tavily_extract,
|
||||
)
|
||||
from modules.ocr_client import OCRClient
|
||||
from modules.easter_egg_manager import EasterEggManager
|
||||
from modules.personalization_manager import (
|
||||
@ -741,7 +746,9 @@ class MainTerminalToolsExecutionMixin:
|
||||
return visited
|
||||
|
||||
def _mark_file_read_from_result(self, tool_name: str, arguments: Dict[str, Any], result: Dict[str, Any]) -> None:
|
||||
if tool_name not in {"read_file", "read_skill"}:
|
||||
# recall_project_memory 内部即读取记忆文件全文（走 _handle_read_tool），
|
||||
# 视为已读，避免后续 edit_file 被要求重复 read_file。
|
||||
if tool_name not in {"read_file", "read_skill", "recall_project_memory"}:
|
||||
return
|
||||
if not isinstance(result, dict) or not result.get("success"):
|
||||
return
|
||||
@ -1768,11 +1775,16 @@ class MainTerminalToolsExecutionMixin:
|
||||
try:
|
||||
# 从config获取API密钥
|
||||
from config import TAVILY_API_KEY
|
||||
try:
|
||||
_prefs = load_personalization_config(self.data_dir) or {}
|
||||
except Exception:
|
||||
_prefs = {}
|
||||
full_content, _ = await extract_webpage_content(
|
||||
urls=url,
|
||||
api_key=TAVILY_API_KEY,
|
||||
extract_depth="basic",
|
||||
max_urls=1
|
||||
max_urls=1,
|
||||
direct_config=resolve_direct_extract_config(_prefs),
|
||||
)
|
||||
|
||||
# 字符数检查
|
||||
@ -1815,61 +1827,28 @@ class MainTerminalToolsExecutionMixin:
|
||||
except ImportError:
|
||||
TAVILY_API_KEY = None
|
||||
|
||||
if not TAVILY_API_KEY or TAVILY_API_KEY == "your-tavily-api-key":
|
||||
result = {
|
||||
"success": False,
|
||||
"error": tr("tools_exec.tavily_key_missing"),
|
||||
"url": url,
|
||||
"path": target_path
|
||||
}
|
||||
else:
|
||||
# 白名单直提优先（无需 Tavily key）；未命中/失败自动回退 Tavily
|
||||
try:
|
||||
extract_result = await tavily_extract(
|
||||
urls=url,
|
||||
api_key=TAVILY_API_KEY,
|
||||
try:
|
||||
_prefs = load_personalization_config(self.data_dir) or {}
|
||||
except Exception:
|
||||
_prefs = {}
|
||||
extract_one = await extract_single_url(
|
||||
url,
|
||||
TAVILY_API_KEY,
|
||||
extract_depth="basic",
|
||||
max_urls=1
|
||||
direct_config=resolve_direct_extract_config(_prefs),
|
||||
)
|
||||
|
||||
if not extract_result or "error" in extract_result:
|
||||
error_message = extract_result.get("error", tr("tools_exec.extract_failed_no_content")) if isinstance(extract_result, dict) else tr("tools_exec.extract_failed")
|
||||
if not extract_one.get("success"):
|
||||
result = {
|
||||
"success": False,
|
||||
"error": error_message,
|
||||
"url": url,
|
||||
"path": target_path
|
||||
}
|
||||
else:
|
||||
results_list = extract_result.get("results", []) if isinstance(extract_result, dict) else []
|
||||
|
||||
primary_result = None
|
||||
for item in results_list:
|
||||
if item.get("raw_content"):
|
||||
primary_result = item
|
||||
break
|
||||
if primary_result is None and results_list:
|
||||
primary_result = results_list[0]
|
||||
|
||||
if not primary_result:
|
||||
failed_list = extract_result.get("failed_results", []) if isinstance(extract_result, dict) else []
|
||||
result = {
|
||||
"success": False,
|
||||
"error": tr("tools_exec.extract_result_empty"),
|
||||
"url": url,
|
||||
"path": target_path,
|
||||
"failed": failed_list
|
||||
}
|
||||
else:
|
||||
content_to_save = primary_result.get("raw_content") or primary_result.get("content") or ""
|
||||
|
||||
if not content_to_save:
|
||||
result = {
|
||||
"success": False,
|
||||
"error": tr("tools_exec.webpage_content_empty"),
|
||||
"error": extract_one.get("error", tr("tools_exec.extract_failed_no_content")),
|
||||
"url": url,
|
||||
"path": target_path
|
||||
}
|
||||
else:
|
||||
content_to_save = extract_one.get("content") or ""
|
||||
write_result = self.file_manager.write_file(target_path, content_to_save, mode="w")
|
||||
|
||||
if not write_result.get("success"):
|
||||
@ -1888,12 +1867,10 @@ class MainTerminalToolsExecutionMixin:
|
||||
"path": write_result.get("path", target_path),
|
||||
"char_count": char_count,
|
||||
"byte_size": byte_size,
|
||||
"extract_method": extract_one.get("method"),
|
||||
"message": tr("tools_exec.webpage_saved", path=write_result.get('path', target_path))
|
||||
}
|
||||
|
||||
if isinstance(extract_result, dict) and extract_result.get("failed_results"):
|
||||
result["warnings"] = extract_result["failed_results"]
|
||||
|
||||
except Exception as e:
|
||||
result = {
|
||||
"success": False,
|
||||
|
||||
@ -214,6 +214,26 @@ MESSAGES = {
|
||||
"zh-CN": "❌ 未能提取到任何内容",
|
||||
"en-US": "❌ No content could be extracted",
|
||||
},
|
||||
"webpage.method_label": {
|
||||
"zh-CN": "提取方式: {method}",
|
||||
"en-US": "Extraction method: {method}",
|
||||
},
|
||||
"webpage.method_jsdelivr": {
|
||||
"zh-CN": "直连 jsDelivr",
|
||||
"en-US": "Direct jsDelivr",
|
||||
},
|
||||
"webpage.method_github_api": {
|
||||
"zh-CN": "直连 GitHub API",
|
||||
"en-US": "Direct GitHub API",
|
||||
},
|
||||
"webpage.method_trafilatura": {
|
||||
"zh-CN": "直连 trafilatura",
|
||||
"en-US": "Direct trafilatura",
|
||||
},
|
||||
"webpage.method_tavily": {
|
||||
"zh-CN": "Tavily",
|
||||
"en-US": "Tavily",
|
||||
},
|
||||
|
||||
# ── mcp_client_manager（manager.py + http_client.py） ──
|
||||
"mcp.server_not_found": {
|
||||
|
||||
@ -3,9 +3,10 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Union
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
try:
|
||||
from config.limits import REASONING_EFFORT_LEVELS
|
||||
@ -31,6 +32,9 @@ ALLOWED_UI_LOCALES = {"zh-CN", "en-US"}
|
||||
GOAL_MAX_TURNS_MIN = 1
|
||||
GOAL_MAX_TURNS_MAX = 100
|
||||
GOAL_MAX_TURNS_DEFAULT = 5
|
||||
|
||||
# 网页提取直提白名单：用户可追加的域名数量上限
|
||||
MAX_DIRECT_EXTRACT_DOMAINS = 50
|
||||
GOAL_MAX_TOKENS_MIN = 1_000
|
||||
GOAL_MAX_TOKENS_MAX = 100_000_000
|
||||
|
||||
@ -98,6 +102,10 @@ DEFAULT_PERSONALIZATION_CONFIG: Dict[str, Any] = {
|
||||
# 仅在创建对话时快照一次；已有对话以其对话文件中的快照为准，改这里不影响。
|
||||
"tool_loading_enabled": True,
|
||||
"tool_loading_deferred": None,
|
||||
# 网页提取白名单直提：命中白名单的域名用本机直提（GitHub 直链 / trafilatura），
|
||||
# 未命中才走 Tavily；开关默认开启，domains 为用户追加的白名单域名（内置 github.com 不在此列）。
|
||||
"webpage_direct_extract_enabled": True,
|
||||
"webpage_direct_extract_domains": [],
|
||||
"default_model": None,
|
||||
"image_compression": "original", # original / 1080p / 720p / 540p
|
||||
"auto_shallow_compress_enabled": False,
|
||||
@ -754,6 +762,27 @@ def sanitize_personalization_payload(
|
||||
else:
|
||||
base["review_agents"] = _sanitize_review_agents(base.get("review_agents"))
|
||||
|
||||
# 网页提取白名单直提：开关（默认开启）+ 用户追加的域名列表
|
||||
if "webpage_direct_extract_enabled" in data:
|
||||
base["webpage_direct_extract_enabled"] = bool(data.get("webpage_direct_extract_enabled"))
|
||||
else:
|
||||
base["webpage_direct_extract_enabled"] = bool(base.get("webpage_direct_extract_enabled", True))
|
||||
|
||||
raw_domains = data.get("webpage_direct_extract_domains", base.get("webpage_direct_extract_domains"))
|
||||
if not isinstance(raw_domains, list):
|
||||
raw_domains = []
|
||||
clean_domains: List[str] = []
|
||||
for item in raw_domains[:MAX_DIRECT_EXTRACT_DOMAINS]:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
d = item.strip().lower().strip(".")
|
||||
if "//" in d:
|
||||
d = d.split("//", 1)[1]
|
||||
d = d.split("/")[0].split("?")[0]
|
||||
if d and re.fullmatch(r"[a-z0-9.-]+", d) and "." in d and d not in clean_domains:
|
||||
clean_domains.append(d)
|
||||
base["webpage_direct_extract_domains"] = clean_domains
|
||||
|
||||
return base
|
||||
|
||||
|
||||
|
||||
@ -1,14 +1,42 @@
|
||||
# modules/webpage_extractor.py - 网页内容提取模块
|
||||
#
|
||||
# 提取分两层：
|
||||
# 1. 白名单域名走本机直提（免费、零配额）：GitHub 代码文件页有专属直链适配
|
||||
# （jsDelivr CDN → GitHub API 备用），其余白名单页面用 trafilatura 提取正文。
|
||||
# 2. 未命中白名单（或直提失败）回退 Tavily 云端提取。
|
||||
#
|
||||
# trafilatura 为可选依赖：未安装时通用直提静默失效，仅 GitHub 直链仍可用。
|
||||
|
||||
import base64
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
import json
|
||||
from typing import Dict, Any, List, Union, Tuple
|
||||
from utils.logger import setup_logger
|
||||
|
||||
from utils.logger import setup_logger
|
||||
from modules.i18n import tr
|
||||
|
||||
logger = setup_logger(__name__)
|
||||
|
||||
try:
|
||||
import trafilatura as _trafilatura
|
||||
except ImportError: # 可选依赖
|
||||
_trafilatura = None
|
||||
|
||||
# 内置直提白名单域名（个人空间可追加；子域名自动匹配）
|
||||
BUILTIN_DIRECT_EXTRACT_DOMAINS: Tuple[str, ...] = ("github.com",)
|
||||
|
||||
_DIRECT_REQUEST_HEADERS = {
|
||||
"User-Agent": (
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
|
||||
),
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
}
|
||||
_DIRECT_TIMEOUT = 30
|
||||
|
||||
|
||||
async def tavily_extract(urls: Union[str, List[str]], api_key: str, extract_depth: str = "basic", max_urls: int = 1) -> Dict[str, Any]:
|
||||
"""
|
||||
执行Tavily网页内容提取
|
||||
@ -104,24 +132,238 @@ def format_extract_results(results: Dict[str, Any]) -> str:
|
||||
return "\n".join(formatted_parts)
|
||||
|
||||
|
||||
async def extract_webpage_content(urls: Union[str, List[str]], api_key: str, extract_depth: str = "basic", max_urls: int = 1) -> Tuple[str, str]:
|
||||
# ============================================================
|
||||
# 白名单直提
|
||||
# ============================================================
|
||||
|
||||
def _normalize_domain(raw: Any) -> str:
|
||||
"""把用户输入规范化为小写裸域名（容忍粘贴完整 URL / 前导点 / 路径）。"""
|
||||
if not isinstance(raw, str):
|
||||
return ""
|
||||
d = raw.strip().lower()
|
||||
if not d:
|
||||
return ""
|
||||
if "://" in d:
|
||||
d = _url_hostname(d) or d
|
||||
d = d.split("/")[0].split("?")[0].strip(".")
|
||||
if not re.fullmatch(r"[a-z0-9.-]+", d or "") or "." not in d:
|
||||
return ""
|
||||
return d
|
||||
|
||||
|
||||
def _url_hostname(url: str) -> str:
|
||||
try:
|
||||
return (urlparse(str(url)).hostname or "").lower()
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def resolve_direct_extract_config(personalization: Optional[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""从 personalization 配置解析直提设置。
|
||||
|
||||
Returns:
|
||||
{"enabled": bool, "domains": [...]} —— domains 已合并内置域名并规范化。
|
||||
"""
|
||||
完整的网页内容提取流程
|
||||
personalization = personalization or {}
|
||||
enabled = bool(personalization.get("webpage_direct_extract_enabled", True))
|
||||
extra = personalization.get("webpage_direct_extract_domains")
|
||||
if not isinstance(extra, list):
|
||||
extra = []
|
||||
domains: List[str] = []
|
||||
for item in list(BUILTIN_DIRECT_EXTRACT_DOMAINS) + extra:
|
||||
nd = _normalize_domain(item)
|
||||
if nd and nd not in domains:
|
||||
domains.append(nd)
|
||||
return {"enabled": enabled, "domains": domains}
|
||||
|
||||
|
||||
def is_whitelisted_url(url: str, domains: List[str]) -> bool:
|
||||
"""域名等于白名单条目或为其子域名即命中。"""
|
||||
host = _url_hostname(url)
|
||||
if not host:
|
||||
return False
|
||||
return any(host == d or host.endswith("." + d) for d in domains)
|
||||
|
||||
|
||||
_GITHUB_BLOB_RE = re.compile(
|
||||
r"^https?://(?:www\.)?github\.com/([^/?#]+)/([^/?#]+)/blob/([^/?#]+)/([^?#]+?)(?:[?#].*)?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def parse_github_blob_url(url: str) -> Optional[Tuple[str, str, str, str]]:
|
||||
"""解析 GitHub blob 页面 URL，返回 (owner, repo, branch, path)；非 blob 页返回 None。
|
||||
|
||||
注意：branch 按单段处理（覆盖 main/master 等常见场景）；含斜杠的分支名
|
||||
会解析失败并自然降级到通用提取，不会报错。
|
||||
"""
|
||||
m = _GITHUB_BLOB_RE.match(str(url).strip())
|
||||
if not m:
|
||||
return None
|
||||
return m.group(1), m.group(2), m.group(3), m.group(4)
|
||||
|
||||
|
||||
async def _fetch_jsdelivr_raw(client: httpx.AsyncClient, owner: str, repo: str, branch: str, path: str) -> Optional[str]:
|
||||
"""经 jsDelivr CDN 拿 GitHub 文件原文（免费、无限速，替代被墙的 raw.githubusercontent.com）。"""
|
||||
cdn_url = f"https://cdn.jsdelivr.net/gh/{owner}/{repo}@{branch}/{path}"
|
||||
try:
|
||||
resp = await client.get(cdn_url, timeout=_DIRECT_TIMEOUT)
|
||||
if resp.status_code == 200 and resp.text:
|
||||
return resp.text
|
||||
logger.info(f"jsDelivr 直链返回 {resp.status_code}: {cdn_url}")
|
||||
except Exception as e:
|
||||
logger.info(f"jsDelivr 直链失败 {cdn_url}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def _fetch_github_api_raw(client: httpx.AsyncClient, owner: str, repo: str, branch: str, path: str) -> Optional[str]:
|
||||
"""经 GitHub 官方 contents API 拿文件原文（备用；匿名限 60 次/小时，>1MB 文件不返回内容）。"""
|
||||
api_url = f"https://api.github.com/repos/{owner}/{repo}/contents/{path}?ref={branch}"
|
||||
try:
|
||||
resp = await client.get(api_url, headers={"Accept": "application/vnd.github+json"}, timeout=_DIRECT_TIMEOUT)
|
||||
if resp.status_code != 200:
|
||||
logger.info(f"GitHub API 返回 {resp.status_code}: {api_url}")
|
||||
return None
|
||||
data = resp.json()
|
||||
if isinstance(data, dict) and data.get("encoding") == "base64" and data.get("content"):
|
||||
return base64.b64decode(data["content"]).decode("utf-8", errors="replace")
|
||||
except Exception as e:
|
||||
logger.info(f"GitHub API 失败 {api_url}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _trafilatura_extract(html: str, url: str) -> Optional[str]:
|
||||
"""trafilatura 通用正文提取（markdown 输出，保留标题/代码块结构）。"""
|
||||
if _trafilatura is None:
|
||||
return None
|
||||
try:
|
||||
text = _trafilatura.extract(
|
||||
html,
|
||||
url=url,
|
||||
output_format="markdown",
|
||||
include_links=False,
|
||||
include_images=False,
|
||||
)
|
||||
if text and text.strip():
|
||||
return text.strip()
|
||||
except Exception as e:
|
||||
logger.info(f"trafilatura 提取失败 {url}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def _direct_extract(client: httpx.AsyncClient, url: str) -> Tuple[Optional[str], str]:
|
||||
"""白名单直提主流程。返回 (内容, method)；全部失败返回 (None, "")。"""
|
||||
blob = parse_github_blob_url(url)
|
||||
if blob:
|
||||
owner, repo, branch, path = blob
|
||||
content = await _fetch_jsdelivr_raw(client, owner, repo, branch, path)
|
||||
if content is not None:
|
||||
return content, "jsdelivr"
|
||||
content = await _fetch_github_api_raw(client, owner, repo, branch, path)
|
||||
if content is not None:
|
||||
return content, "github_api"
|
||||
# 直链均失败 → 继续走通用提取兜底
|
||||
|
||||
if _trafilatura is not None:
|
||||
try:
|
||||
resp = await client.get(url, timeout=_DIRECT_TIMEOUT)
|
||||
if resp.status_code == 200 and resp.text:
|
||||
text = _trafilatura_extract(resp.text, url)
|
||||
if text:
|
||||
return text, "trafilatura"
|
||||
else:
|
||||
logger.info(f"直提抓取返回 {resp.status_code}: {url}")
|
||||
except Exception as e:
|
||||
logger.info(f"直提抓取失败 {url}: {e}")
|
||||
return None, ""
|
||||
|
||||
|
||||
def _format_single_result(url: str, content: str, method: Optional[str] = None) -> str:
|
||||
"""格式化单条提取结果（沿用 🌐 骨架，附提取方式标注）。"""
|
||||
header = f"🌐 网页内容 ({len(content)} 字符)"
|
||||
if method:
|
||||
header += f" [{tr('webpage.method_label', method=tr(f'webpage.method_{method}'))}]"
|
||||
return "\n".join([
|
||||
header + ":",
|
||||
f"📍 URL: {url}",
|
||||
"=" * 50,
|
||||
content,
|
||||
"=" * 50,
|
||||
])
|
||||
|
||||
|
||||
async def extract_single_url(
|
||||
url: str,
|
||||
api_key: Optional[str],
|
||||
extract_depth: str = "basic",
|
||||
direct_config: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""统一单 URL 提取：白名单直提优先，失败/未命中回退 Tavily。
|
||||
|
||||
Args:
|
||||
url: 目标 URL
|
||||
api_key: Tavily API 密钥（可为 None；直提命中时不需要）
|
||||
extract_depth: Tavily 提取深度
|
||||
direct_config: resolve_direct_extract_config() 的返回；None 视为关闭直提
|
||||
|
||||
Returns:
|
||||
{"success": bool, "url": str, "content": str, "method": str} 或
|
||||
{"success": False, "url": str, "error": str, "method": str}
|
||||
"""
|
||||
direct_config = direct_config or {}
|
||||
|
||||
if direct_config.get("enabled") and is_whitelisted_url(url, direct_config.get("domains") or []):
|
||||
content, method = None, ""
|
||||
try:
|
||||
async with httpx.AsyncClient(headers=_DIRECT_REQUEST_HEADERS, follow_redirects=True) as client:
|
||||
content, method = await _direct_extract(client, url)
|
||||
except Exception as e:
|
||||
logger.info(f"白名单直提异常 {url}: {e}")
|
||||
if content is not None:
|
||||
return {"success": True, "url": url, "content": content, "method": method}
|
||||
logger.info(f"白名单直提未取到内容，回退 Tavily: {url}")
|
||||
|
||||
results = await tavily_extract(url, api_key, extract_depth, 1)
|
||||
if "error" in results:
|
||||
return {"success": False, "url": url, "error": results["error"], "method": "tavily"}
|
||||
for item in results.get("results") or []:
|
||||
raw = (item.get("raw_content") or "").strip()
|
||||
if raw:
|
||||
return {"success": True, "url": url, "content": raw, "method": "tavily"}
|
||||
return {"success": False, "url": url, "error": tr("webpage.no_content"), "method": "tavily"}
|
||||
|
||||
|
||||
async def extract_webpage_content(
|
||||
urls: Union[str, List[str]],
|
||||
api_key: str,
|
||||
extract_depth: str = "basic",
|
||||
max_urls: int = 1,
|
||||
direct_config: Optional[Dict[str, Any]] = None,
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
完整的网页内容提取流程（白名单直提优先，Tavily 兜底）
|
||||
|
||||
Args:
|
||||
urls: 要提取的URL（字符串或列表）
|
||||
api_key: Tavily API密钥
|
||||
extract_depth: 提取深度 (basic/advanced)
|
||||
max_urls: 最大提取URL数量
|
||||
direct_config: 直提配置（resolve_direct_extract_config 返回）；None=关闭直提
|
||||
|
||||
Returns:
|
||||
(完整内容, 完整内容) - 为了兼容性返回相同内容两份
|
||||
"""
|
||||
# 执行提取
|
||||
results = await tavily_extract(urls, api_key, extract_depth, max_urls)
|
||||
if isinstance(urls, str):
|
||||
urls = [urls]
|
||||
urls = urls[:max_urls]
|
||||
|
||||
# 格式化结果
|
||||
formatted_content = format_extract_results(results)
|
||||
formatted_parts: List[str] = []
|
||||
for url in urls:
|
||||
result = await extract_single_url(url, api_key, extract_depth=extract_depth, direct_config=direct_config)
|
||||
if result.get("success"):
|
||||
formatted_parts.append(_format_single_result(url, result["content"], result.get("method")))
|
||||
else:
|
||||
formatted_parts.append(tr("webpage.format_failed", error=result.get("error")))
|
||||
|
||||
# 返回相同内容（简化版本，不需要长短版本区分）
|
||||
formatted_content = "\n".join(formatted_parts)
|
||||
return formatted_content, formatted_content
|
||||
@ -8,3 +8,6 @@ openai
|
||||
cryptography
|
||||
pillow
|
||||
pyyaml>=6.0
|
||||
# 网页白名单直提的正文识别引擎（可选依赖：import 失败时代码自动降级为全走 Tavily，
|
||||
# 但正式部署应装上，否则直提白名单只剩 GitHub 直链可用）
|
||||
trafilatura
|
||||
|
||||
@ -561,6 +561,65 @@ body[data-theme='dark'] .settings-floating-menu {
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
/* 直提白名单域名列表：分隔线区分行，不套圆角卡片 */
|
||||
.settings-domain-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
margin: 4px 0 10px;
|
||||
}
|
||||
|
||||
.settings-domain-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 8px;
|
||||
height: 36px;
|
||||
padding: 0 2px;
|
||||
border-bottom: 1px solid var(--theme-control-border);
|
||||
}
|
||||
|
||||
.settings-domain-row:last-child {
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
.settings-domain-row .settings-row-title {
|
||||
min-width: 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.settings-domain-badge {
|
||||
flex: none;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
height: 28px; /* 与 .settings-domain-remove 同高同居中机制，保证文字视觉对齐 */
|
||||
margin-right: 10px; /* 抵消删除按钮的水平 padding，使两者文字右缘对齐 */
|
||||
font-size: 12px;
|
||||
line-height: 1;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.settings-domain-remove {
|
||||
flex: none;
|
||||
height: 28px;
|
||||
padding: 0 10px;
|
||||
border: none;
|
||||
border-radius: 6px;
|
||||
background: transparent;
|
||||
color: var(--text-secondary);
|
||||
font-size: 12px;
|
||||
line-height: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
@media (hover: hover) {
|
||||
.settings-domain-remove:hover {
|
||||
background: var(--hover-bg);
|
||||
color: var(--state-warning);
|
||||
}
|
||||
}
|
||||
|
||||
.settings-add-row input,
|
||||
.settings-number-row input {
|
||||
min-width: 0;
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
<script setup lang="ts">
|
||||
import { inject, unref } from 'vue';
|
||||
import { inject, unref, ref } from 'vue';
|
||||
import FancyCheck from '@/components/common/FancyCheck.vue';
|
||||
|
||||
defineOptions({ name: 'ToolsTab' });
|
||||
@ -58,6 +58,43 @@ const toggleDeferredCategory = (cat: { tools: string[] }, checked: boolean) => {
|
||||
}
|
||||
personalization.updateField({ key: 'tool_loading_deferred', value: Array.from(current) });
|
||||
};
|
||||
|
||||
/** 追加的直提白名单域名（内置 github.com 不在此列，直接展示内置徽标） */
|
||||
const directExtractDomains = (): string[] => {
|
||||
const f: any = unref(form);
|
||||
return Array.isArray(f?.webpage_direct_extract_domains) ? f.webpage_direct_extract_domains : [];
|
||||
};
|
||||
|
||||
/** 新增域名输入框的本地草稿 */
|
||||
const newDirectDomain = ref('');
|
||||
|
||||
const addDirectDomain = () => {
|
||||
let value = newDirectDomain.value.trim().toLowerCase();
|
||||
if (!value) return;
|
||||
// 容忍粘贴完整 URL：提取 hostname 部分
|
||||
if (value.includes('://')) {
|
||||
try {
|
||||
value = new URL(value).hostname;
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
}
|
||||
value = value.split('/')[0].replace(/^\.+|\.+$/g, '');
|
||||
if (!value || !value.includes('.')) return;
|
||||
const current = [...directExtractDomains()];
|
||||
if (!current.includes(value)) {
|
||||
current.push(value);
|
||||
personalization.updateField({ key: 'webpage_direct_extract_domains', value: current });
|
||||
}
|
||||
newDirectDomain.value = '';
|
||||
};
|
||||
|
||||
const removeDirectDomain = (domain: string) => {
|
||||
personalization.updateField({
|
||||
key: 'webpage_direct_extract_domains',
|
||||
value: directExtractDomains().filter((d) => d !== domain)
|
||||
});
|
||||
};
|
||||
</script>
|
||||
|
||||
<template>
|
||||
@ -208,6 +245,54 @@ const toggleDeferredCategory = (cat: { tools: string[] }, checked: boolean) => {
|
||||
@change="toggleCategory(category.id)" /><FancyCheck :checked="form.disabled_tool_categories.includes(category.id)" /></label>
|
||||
</div>
|
||||
</div>
|
||||
<div class="settings-group-block">
|
||||
<div class="settings-group-title">
|
||||
<span class="settings-row-title">{{ $t('personalization.webDirectExtractTitle') }}</span
|
||||
><span class="settings-row-desc">{{ $t('personalization.webDirectExtractDesc') }}</span>
|
||||
</div>
|
||||
<label class="settings-toggle-row inner"
|
||||
><span class="settings-row-title">{{ $t('personalization.webDirectExtractEnabledTitle') }}</span
|
||||
><input
|
||||
type="checkbox"
|
||||
:checked="form.webpage_direct_extract_enabled"
|
||||
@change="
|
||||
personalization.updateField({
|
||||
key: 'webpage_direct_extract_enabled',
|
||||
value: $event.target.checked
|
||||
})
|
||||
" /><FancyCheck :checked="form.webpage_direct_extract_enabled" /></label>
|
||||
<template v-if="form.webpage_direct_extract_enabled">
|
||||
<div class="settings-domain-list">
|
||||
<div class="settings-domain-row">
|
||||
<span class="settings-row-title">github.com</span
|
||||
><span class="settings-domain-badge">{{
|
||||
$t('personalization.webDirectExtractBuiltinBadge')
|
||||
}}</span>
|
||||
</div>
|
||||
<div v-for="domain in directExtractDomains()" :key="domain" class="settings-domain-row">
|
||||
<span class="settings-row-title">{{ domain }}</span
|
||||
><button
|
||||
type="button"
|
||||
class="settings-domain-remove"
|
||||
@click="removeDirectDomain(domain)"
|
||||
>
|
||||
{{ $t('common.delete') }}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="settings-add-row">
|
||||
<input
|
||||
v-model="newDirectDomain"
|
||||
type="text"
|
||||
:placeholder="$t('personalization.webDirectExtractDomainPlaceholder')"
|
||||
@keydown.enter="addDirectDomain"
|
||||
/>
|
||||
<button type="button" class="settings-secondary-button" @click="addDirectDomain">
|
||||
{{ $t('personalization.webDirectExtractAdd') }}
|
||||
</button>
|
||||
</div>
|
||||
</template>
|
||||
</div>
|
||||
<div class="settings-group-block" v-if="toolLoadingRegistry.length">
|
||||
<div class="settings-group-title">
|
||||
<span class="settings-row-title">{{ $t('personalization.toolLoadingTitle') }}</span
|
||||
|
||||
@ -253,6 +253,14 @@ export default {
|
||||
mcp: 'MCP',
|
||||
misc: 'Easter egg'
|
||||
},
|
||||
webDirectExtractTitle: 'Direct extraction whitelist',
|
||||
webDirectExtractDesc:
|
||||
'Whitelisted sites are extracted locally (GitHub code files via CDN raw links, others via readability-style parsing) at no Tavily quota cost; falls back to Tavily automatically on failure',
|
||||
webDirectExtractEnabledTitle: 'Enable whitelist direct extraction',
|
||||
webDirectExtractDomainsTitle: 'Additional whitelisted domains',
|
||||
webDirectExtractBuiltinBadge: 'Built-in',
|
||||
webDirectExtractDomainPlaceholder: 'Enter a domain, e.g. docs.python.org',
|
||||
webDirectExtractAdd: 'Add',
|
||||
|
||||
// ── Files & Images ──
|
||||
imageCompressionTitle: 'Image compression',
|
||||
|
||||
@ -254,6 +254,14 @@ export default {
|
||||
mcp: 'MCP',
|
||||
misc: '彩蛋'
|
||||
},
|
||||
webDirectExtractTitle: '网页直提白名单',
|
||||
webDirectExtractDesc:
|
||||
'命中白名单的网站直接在本机提取内容（GitHub 代码文件走 CDN 直链，其余用正文识别），无需消耗 Tavily 配额；提取失败会自动回退 Tavily',
|
||||
webDirectExtractEnabledTitle: '启用白名单直提',
|
||||
webDirectExtractDomainsTitle: '追加白名单域名',
|
||||
webDirectExtractBuiltinBadge: '内置',
|
||||
webDirectExtractDomainPlaceholder: '输入域名，如 docs.python.org',
|
||||
webDirectExtractAdd: '添加',
|
||||
|
||||
// ── 文件与图片 ──
|
||||
imageCompressionTitle: '图片压缩',
|
||||
|
||||
@ -103,6 +103,10 @@ interface PersonalForm {
|
||||
tool_loading_enabled: boolean;
|
||||
/** 默认延迟加载的工具名列表（仅影响新建对话） */
|
||||
tool_loading_deferred: string[];
|
||||
/** 网页提取白名单直提开关（默认开启） */
|
||||
webpage_direct_extract_enabled: boolean;
|
||||
/** 用户追加的直提白名单域名（内置 github.com 不在此列） */
|
||||
webpage_direct_extract_domains: string[];
|
||||
skill_hints_enabled: boolean;
|
||||
skill_strict_terminal_enabled: boolean;
|
||||
skill_strict_sub_agent_enabled: boolean;
|
||||
@ -314,6 +318,8 @@ const defaultForm = (): PersonalForm => ({
|
||||
tool_intent_enabled: true,
|
||||
tool_loading_enabled: true,
|
||||
tool_loading_deferred: [],
|
||||
webpage_direct_extract_enabled: true,
|
||||
webpage_direct_extract_domains: [],
|
||||
skill_hints_enabled: false,
|
||||
skill_strict_terminal_enabled: false,
|
||||
skill_strict_sub_agent_enabled: false,
|
||||
@ -540,6 +546,12 @@ export const usePersonalizationStore = defineStore('personalization', {
|
||||
tool_loading_deferred: Array.isArray(data.tool_loading_deferred)
|
||||
? data.tool_loading_deferred.filter((item: any) => typeof item === 'string')
|
||||
: [],
|
||||
webpage_direct_extract_enabled: data.webpage_direct_extract_enabled !== false,
|
||||
webpage_direct_extract_domains: Array.isArray(data.webpage_direct_extract_domains)
|
||||
? data.webpage_direct_extract_domains.filter(
|
||||
(item: any) => typeof item === 'string' && item.trim()
|
||||
)
|
||||
: [],
|
||||
skill_hints_enabled: !!data.skill_hints_enabled,
|
||||
skill_strict_terminal_enabled: !!data.skill_strict_terminal_enabled,
|
||||
skill_strict_sub_agent_enabled: !!data.skill_strict_sub_agent_enabled,
|
||||
|
||||
Loading…
Reference in New Issue
Block a user