JOJO/agent-Specialization
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(voice): Android 端侧离线语音识别集成

Browse Source 
基于 Sherpa-ONNX + SenseVoice int8 实现 APK 端侧语音识别：
- VoiceBridge: AudioRecord 录音 + 整段识别 + JS Bridge 回调
- ModelManager: 模型下载管理（自有服务器），支持断点续传/校验/删除
- 前端：语音按钮仅 APK 环境显示，识别结果回填 ProseMirror 编辑器
- 调试：文件日志 + /api/voice_debug 接收路由
- demo/sense_voice_demo.py: Python 端测 Demo

versionCode 24→36, versionName 1.0.22→1.0.34
This commit is contained in:
JOJO 2026-06-10 00:50:16 +08:00
parent 62e2c08f5a
commit d24e00b9b3
13 changed files with 1625 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
android-webview-app/APP_CHANGELOG.md
View File

@ -1,5 +1,50 @@
# App 更新说明
# 1.0.32
- 识别引擎改为预初始化：模型下载完成后立即在后台加载，用户点击时直接可用（不再每次点击等3-5秒初始化）
- startRecordingInternal 增加详细日志，便于排查录音问题
# 1.0.31
- 修复语音识别无结果：去掉多余的 decode() 调用，对齐 sherpa-onnx Android 官方 API（getResult 内部已含 decode）
- recognizeSegment 增加识别结果详细日志
# 1.0.30
- 修复语音识别无声问题：前端处理 initializing 状态，防止初始化期间重复触发 startListening 导致录音永不停止
- Android 端增加启动取消机制，初始化期间按停止不会意外开始录音
- 完善录音链路日志，便于后续调试
# 1.0.29
- 暂时禁用 VAD（Android 端 Silero VAD 存在 Native 兼容性问题），改为整段识别模式
- 录音结束后一次性识别，效果无差异，彻底消除闪退
# 1.0.28
- 修复 VAD 初始化导致闪退：maxSpeechDuration 30f→5.0f（默认值），VAD 失败时降级为整段识别不崩溃
# 1.0.27
- 语音调试日志改为直接上传服务器（/api/voice_debug），无需手动找文件
# 1.0.26
- 修复点击麦克风后闪退问题：initEngine 增加文件完整性前置校验，numThreads 降为 1 降低内存压力
- 新增「保存日志」按钮，调试日志写入手机 Download/voice_debug.log
- 下载/删除前自动释放引擎资源，避免文件锁定导致异常
# 1.0.25
- 修复下载进度一直显示 0% 的问题（整数除法 bug）
- 新增模型文件完整性校验（文件大小 ±5% 容差），避免下载中断后显示“已下载”
- 新增「删除模型」按钮，可清理不完整的模型文件
- 模型文件移至 agents 代码目录外（/opt/agent/voice_models/），避免部署时被覆盖
# 1.0.24
- 修复语音按钮在首次启动（模型未下载）时不显示的问题：VoiceBridge 改为随 App 启动即注册，不再等待模型下载完成
- 修复个人空间中「下载语音模型」按钮误提示「仅在 App 内支持」的问题
# 1.0.23
- 新增端侧语音输入功能：集成 sherpa-onnx SenseVoice int8 离线语音识别
- 支持中英混说 + 自动标点，完全离线运行，无需网络
- 语音模型约 228MB，首次使用自动下载（可在「个人空间 → 语音模型」手动下载）
- 输入框语音按钮仅在 App 端显示，桌面端隐藏
- 新增录音权限请求（RECORD_AUDIO）
# 1.0.22
- 修复 Android App 内从本地发送图片/视频无效的问题：补充 Android 13+ 所需的媒体读取权限（`READ_MEDIA_IMAGES` / `READ_MEDIA_VIDEO`）
- 修复 Android App 文件选择器在 Android 13+ 上回调丢失的问题：将已废弃的 `startActivityForResult` 升级为 `registerForActivityResult`（Activity Result API）

11
android-webview-app/app/build.gradle.kts
View File

@ -16,8 +16,8 @@ android {
applicationId = "com.cyjai.agent"
minSdk = 24
targetSdk = 35
versionCode = 24
versionName = "1.0.22"
versionCode = 36
versionName = "1.0.34"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
@ -63,4 +63,11 @@ dependencies {
implementation("androidx.appcompat:appcompat:1.7.0")
implementation("com.google.android.material:material:1.12.0")
implementation("androidx.activity:activity-ktx:1.9.2")
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.8.0")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.8.1")
// sherpa-onnx 语音识别库（需手动下载 AAR 放到 app/libs/）
// 下载地址：https://huggingface.co/csukuangfj/sherpa-onnx-libs/tree/main/android/aar
// 下载最新 sherpa-onnx-*.aar 放到 app/libs/ 目录即可
implementation(fileTree(mapOf("dir" to "libs", "include" to listOf("*.aar"))))
}

BIN
android-webview-app/app/libs/sherpa-onnx-1.10.45.aar Normal file
View File

Binary file not shown.

1
android-webview-app/app/src/main/AndroidManifest.xml
View File

@ -2,6 +2,7 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />
<uses-permission android:name="android.permission.READ_MEDIA_VIDEO" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" android:maxSdkVersion="32" />

94
android-webview-app/app/src/main/java/com/cyjai/agent/MainActivity.kt
View File

@ -1,11 +1,14 @@
package com.cyjai.agent
import android.Manifest
import android.annotation.SuppressLint
import android.content.Intent
import android.content.pm.PackageManager
import android.graphics.Color
import android.net.Uri
import android.os.Build
import android.os.Bundle
import android.util.Log
import android.view.ViewGroup
import android.view.View
import android.webkit.CookieManager
@ -21,15 +24,29 @@ import androidx.activity.OnBackPressedCallback
import androidx.activity.result.ActivityResultLauncher
import androidx.activity.result.contract.ActivityResultContracts
import androidx.appcompat.app.AppCompatActivity
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import androidx.core.view.ViewCompat
import androidx.core.view.WindowInsetsCompat
import androidx.core.view.WindowCompat
import androidx.lifecycle.lifecycleScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
import java.util.Locale
class MainActivity : AppCompatActivity() {
private lateinit var webView: WebView
private var filePathCallback: ValueCallback<Array<Uri>>? = null
private var voiceBridge: VoiceBridge? = null
companion object {
private const val TAG = "MainActivity"
private const val REQUEST_RECORD_AUDIO = 300
private const val HOME_URL = "https://agent.cyjai.com"
private const val WEB_ASSET_VERSION = "20260406_4"
}
private val fileChooserLauncher: ActivityResultLauncher<Intent> = registerForActivityResult(
ActivityResultContracts.StartActivityForResult()
@ -84,6 +101,51 @@ class MainActivity : AppCompatActivity() {
webView.addJavascriptInterface(ThemeBridge(), "AndroidThemeBridge")
// 语音识别桥接（立即注册，前端始终可检测到；引擎在首次使用时懒初始化）
voiceBridge = VoiceBridge(this@MainActivity, webView)
webView.addJavascriptInterface(voiceBridge!!, "AndroidVoiceBridge")
// 后台检查并下载模型
lifecycleScope.launch(Dispatchers.IO) {
ModelManager.init(this@MainActivity)
if (!ModelManager.isModelReady(this@MainActivity)) {
Log.i(TAG, "首次启动，开始下载语音模型...")
withContext(Dispatchers.Main) {
injectVoiceStatus("downloading")
}
val success = ModelManager.downloadModels(this@MainActivity) { pct, msg ->
Log.i(TAG, "模型下载: ${pct}% - $msg")
runOnUiThread {
injectVoiceDownloadProgress(pct, msg)
}
}
if (!success) {
Log.e(TAG, "模型下载失败")
withContext(Dispatchers.Main) {
injectVoiceStatus("error")
}
return@launch
}
}
Log.i(TAG, "模型就绪")
withContext(Dispatchers.Main) {
injectVoiceStatus("ready")
// 模型就绪后立即预初始化识别引擎，用户点击时直接可用
voiceBridge?.ensureEngine()
}
}
// 请求录音权限
if (ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO)
!= PackageManager.PERMISSION_GRANTED
) {
ActivityCompat.requestPermissions(
this,
arrayOf(Manifest.permission.RECORD_AUDIO),
REQUEST_RECORD_AUDIO
)
}
webView.webViewClient = object : WebViewClient() {
override fun onPageFinished(view: WebView?, url: String?) {
super.onPageFinished(view, url)
@ -152,6 +214,7 @@ class MainActivity : AppCompatActivity() {
}
override fun onDestroy() {
voiceBridge?.destroy()
webView.destroy()
super.onDestroy()
}
@ -278,6 +341,32 @@ class MainActivity : AppCompatActivity() {
webView.evaluateJavascript(script, null)
}
// ═══════════════════ 语音状态通知（JS 注入） ═══════════════════
private fun injectVoiceStatus(status: String) {
val script = """
(function() {
if (window.__onVoiceStatus) window.__onVoiceStatus('$status');
if (window.dispatchEvent) {
window.dispatchEvent(new CustomEvent('voicebridge:status', { detail: '$status' }));
}
})();
""".trimIndent()
webView.evaluateJavascript(script, null)
}
private fun injectVoiceDownloadProgress(pct: Int, msg: String) {
val safeMsg = msg.replace("'", "\\'")
val script = """
(function() {
if (window.__onVoiceDownloadProgress) {
window.__onVoiceDownloadProgress($pct, '$safeMsg');
}
})();
""".trimIndent()
webView.evaluateJavascript(script, null)
}
inner class ThemeBridge {
@JavascriptInterface
fun onThemeChanged(theme: String?) {
@ -297,11 +386,6 @@ class MainActivity : AppCompatActivity() {
}
}
companion object {
private const val HOME_URL = "https://agent.cyjai.com"
private const val WEB_ASSET_VERSION = "20260406_4"
}
private fun buildHomeUrl(): String {
val vc = getInstalledVersionCode()
val vn = Uri.encode(getInstalledVersionName())

194
android-webview-app/app/src/main/java/com/cyjai/agent/ModelManager.kt Normal file
View File

@ -0,0 +1,194 @@
package com.cyjai.agent
import android.content.Context
import android.util.Log
import kotlinx.coroutines.*
import java.io.*
import java.net.HttpURLConnection
import java.net.URL
/**
* 语音模型下载管理器
* 首次启动时自动下载 SenseVoice + Silero VAD 模型到内部存储
*
* 模型来源：HuggingFace (csukuangfj)
* 总大小：约 230MB（SenseVoice int8 228MB + tokens 308KB + Silero VAD 1.5MB）
*/
object ModelManager {
private const val TAG = "ModelManager"
// 模型下载地址（从自有服务器下载，3 个文件约 230MB）
// 需提前上传到服务器静态目录：/static/voice_models/
private const val BASE_URL = "https://agent.cyjai.com/static/voice_models"
private const val SENSEVOICE_MODEL_URL = "$BASE_URL/model.int8.onnx"
private const val SENSEVOICE_TOKENS_URL = "$BASE_URL/tokens.txt"
private const val SILERO_VAD_URL = "$BASE_URL/silero_vad.onnx"
private var _modelDir: File? = null
fun init(context: Context) {
_modelDir = File(context.filesDir, "voice_models")
}
private fun getModelDir(context: Context): File {
return _modelDir ?: File(context.filesDir, "voice_models").also { _modelDir = it }
}
// 模型文件预期大小（字节），用于校验下载完整性（±5% 容差）
private const val EXPECTED_MODEL_SIZE = 239_233_841L // model.int8.onnx ~228MB
private const val EXPECTED_TOKENS_SIZE = 316_000L // tokens.txt ~308KB
private const val EXPECTED_VAD_SIZE = 644_000L // silero_vad.onnx ~629KB
private const val SIZE_TOLERANCE = 0.05 // 5% 容差
private fun isSizeValid(file: File, expected: Long): Boolean {
if (!file.exists()) return false
val size = file.length()
val lower = (expected * (1.0 - SIZE_TOLERANCE)).toLong()
val upper = (expected * (1.0 + SIZE_TOLERANCE)).toLong()
return size in lower..upper
}
/**
* 检查模型是否已下载完毕且文件完整
*/
fun isModelReady(context: Context): Boolean {
val dir = getModelDir(context)
val senseVoiceModel = File(dir, "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17/model.int8.onnx")
val senseVoiceTokens = File(dir, "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17/tokens.txt")
val sileroVad = File(dir, "silero_vad.onnx")
return isSizeValid(senseVoiceModel, EXPECTED_MODEL_SIZE)
&& isSizeValid(senseVoiceTokens, EXPECTED_TOKENS_SIZE)
&& isSizeValid(sileroVad, EXPECTED_VAD_SIZE)
}
/**
* 删除所有已下载的模型文件
*/
fun deleteModels(context: Context): Boolean {
val dir = getModelDir(context)
return try {
dir.deleteRecursively()
} catch (e: Exception) {
Log.e(TAG, "删除模型文件失败", e)
false
}
}
/**
* 下载所有模型（应在后台线程调用）
* @param onProgress 进度回调 (总百分比: Int, 阶段描述: String)
* @return 是否全部成功
*/
suspend fun downloadModels(
context: Context,
onProgress: (Int, String) -> Unit = { _, _ -> }
): Boolean = withContext(Dispatchers.IO) {
val dir = getModelDir(context)
val senseVoiceDir = File(dir, "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17")
senseVoiceDir.mkdirs()
try {
// 1. Silero VAD (~1.5MB)
val sileroFile = File(dir, "silero_vad.onnx")
if (!sileroFile.exists()) {
onProgress(0, "下载 VAD 模型...")
downloadFile(SILERO_VAD_URL, sileroFile) { pct ->
onProgress(pct * 5 / 100, "下载 VAD 模型... ${pct}%")
}
}
// 2. tokens.txt (~308KB)
val tokensFile = File(senseVoiceDir, "tokens.txt")
if (!tokensFile.exists()) {
onProgress(5, "下载 tokens...")
downloadFile(SENSEVOICE_TOKENS_URL, tokensFile) { pct ->
onProgress(5 + pct * 2 / 100, "下载 tokens... ${pct}%")
}
}
// 3. SenseVoice int8 (~228MB)
val modelFile = File(senseVoiceDir, "model.int8.onnx")
if (!modelFile.exists()) {
onProgress(7, "下载 SenseVoice 模型（约 228MB）...")
downloadFile(SENSEVOICE_MODEL_URL, modelFile) { pct ->
onProgress(7 + pct * 93 / 100, "下载 SenseVoice 模型... ${pct}%")
}
}
onProgress(100, "模型就绪")
Log.i(TAG, "模型下载完成")
true
} catch (e: Exception) {
Log.e(TAG, "模型下载失败", e)
false
}
}
private fun downloadFile(urlStr: String, dest: File, onProgress: (Int) -> Unit) {
var attempt = 0
val maxAttempts = 3
while (attempt < maxAttempts) {
try {
attempt++
doDownload(urlStr, dest, onProgress)
return
} catch (e: Exception) {
Log.w(TAG, "下载失败 (第${attempt}次): ${dest.name}", e)
if (attempt >= maxAttempts) throw e
Thread.sleep(2000) // 重试前等待
}
}
}
private fun doDownload(urlStr: String, dest: File, onProgress: (Int) -> Unit) {
val url = URL(urlStr)
val conn = url.openConnection() as HttpURLConnection
conn.requestMethod = "GET"
conn.connectTimeout = 15000
conn.readTimeout = 120000
// 断点续传
var downloaded = if (dest.exists()) dest.length() else 0L
if (downloaded > 0) {
conn.setRequestProperty("Range", "bytes=$downloaded-")
}
conn.connect()
val totalSize = if (downloaded > 0) {
conn.getHeaderField("Content-Range")?.substringAfter("/")?.toLongOrNull()
?: (conn.contentLengthLong + downloaded)
} else {
conn.contentLengthLong
}
val inputStream = conn.inputStream
val outputStream = FileOutputStream(dest, downloaded > 0)
val buffer = ByteArray(65536)
var bytesRead: Int
var totalRead = downloaded
var lastProgressReport = System.currentTimeMillis()
while (inputStream.read(buffer).also { bytesRead = it } != -1) {
outputStream.write(buffer, 0, bytesRead)
totalRead += bytesRead
// 限流进度回调（每 200ms 最多报一次）
val now = System.currentTimeMillis()
if (totalSize > 0 && now - lastProgressReport > 200) {
val pct = (totalRead * 100 / totalSize).toInt()
onProgress(pct.coerceIn(0, 100))
lastProgressReport = now
}
}
if (totalSize > 0 && totalRead < totalSize) {
throw IOException("下载不完整: 期望 ${totalSize} 字节，实际 ${totalRead} 字节")
}
inputStream.close()
outputStream.close()
conn.disconnect()
}
}

501
android-webview-app/app/src/main/java/com/cyjai/agent/VoiceBridge.kt Normal file
View File

@ -0,0 +1,501 @@
package com.cyjai.agent
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.media.AudioFormat
import android.media.AudioRecord
import android.media.MediaRecorder
import android.util.Log
import android.webkit.JavascriptInterface
import android.webkit.WebView
import androidx.core.app.ActivityCompat
import com.k2fsa.sherpa.onnx.OfflineRecognizer
import com.k2fsa.sherpa.onnx.OfflineRecognizerConfig
import com.k2fsa.sherpa.onnx.OfflineSenseVoiceModelConfig
import com.k2fsa.sherpa.onnx.OfflineModelConfig
import com.k2fsa.sherpa.onnx.FeatureConfig
import com.k2fsa.sherpa.onnx.Vad
import com.k2fsa.sherpa.onnx.VadModelConfig
import com.k2fsa.sherpa.onnx.SileroVadModelConfig
import com.k2fsa.sherpa.onnx.SpeechSegment
import android.os.Environment
import kotlinx.coroutines.*
import java.io.File
import java.io.FileWriter
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
import java.util.concurrent.atomic.AtomicBoolean
/**
* 语音识别桥接 — 通过 JS Bridge 暴露给 WebView
*
* 前端调用：
* window.AndroidVoiceBridge.startListening() // 开始录音
* window.AndroidVoiceBridge.stopListening() // 停止录音
* window.AndroidVoiceBridge.isSupported() // 是否支持 → true
*
* 结果通过全局回调传回前端：
* window.__onVoiceResult(text) // 识别结果
* window.__onVoiceStatus(status) // 状态变化: "idle"|"listening"|"processing"
* window.__onVoiceError(error) // 错误
*/
class VoiceBridge(
private val context: Context,
private val webView: WebView
) {
companion object {
private const val TAG = "VoiceBridge"
private const val SAMPLE_RATE = 16000
private const val CHUNK_SIZE = 512 // 每次读取的采样数
}
// ── 状态 ──
private val isRecording = AtomicBoolean(false)
private var audioRecord: AudioRecord? = null
private var recordingJob: Job? = null
private val scope = CoroutineScope(Dispatchers.IO + SupervisorJob())
private var recordingStartTime = 0L // 录音开始时间戳（用于防抖）
private val MIN_RECORDING_MS = 400L // 最小录音时长（防抖）
// ── sherpa-onnx 引擎 ──
private var recognizer: OfflineRecognizer? = null
private var vad: Vad? = null
private var initialized = false
private var engineInitJob: Job? = null // 预初始化任务
// ── 调试日志 ──
private val logFile: File
get() = File(context.getExternalFilesDir(Environment.DIRECTORY_DOWNLOADS), "voice_debug.log")
private val dateFmt = SimpleDateFormat("MM-dd HH:mm:ss.SSS", Locale.getDefault())
private fun logToFile(msg: String) {
try {
val ts = dateFmt.format(Date())
logFile.parentFile?.mkdirs()
FileWriter(logFile, true).use { it.write("[$ts] $msg\n") }
} catch (_: Exception) {}
}
// ── 模型路径 ──
private val modelDir: File
get() = File(context.filesDir, "voice_models")
private val senseVoiceModel: File
get() = File(modelDir, "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17/model.int8.onnx")
private val senseVoiceTokens: File
get() = File(modelDir, "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17/tokens.txt")
private val sileroVadModel: File
get() = File(modelDir, "silero_vad.onnx")
// ═══════════════════ JS Bridge 接口 ═══════════════════
@JavascriptInterface
fun isSupported(): Boolean = true
/** 模型是否已就绪（含文件大小校验） */
@JavascriptInterface
fun isModelReady(): Boolean {
return ModelManager.isModelReady(context)
}
/** 是否有模型文件残留（存在但不完整，需清理） */
@JavascriptInterface
fun isModelPartial(): Boolean {
val filesExist = senseVoiceModel.exists() || senseVoiceTokens.exists() || sileroVadModel.exists()
return filesExist && !ModelManager.isModelReady(context)
}
/** 删除已下载的模型文件 */
@JavascriptInterface
fun deleteModel(): Boolean {
releaseEngine()
return ModelManager.deleteModels(context)
}
/** 触发模型下载（供个人空间手动下载），下载前先释放引擎并清理旧文件 */
@JavascriptInterface
fun downloadModel() {
scope.launch {
try {
// 先释放引擎（避免文件被 onnxruntime 锁定导致删除失败）
releaseEngine()
ModelManager.deleteModels(context)
postToJs("window.__onVoiceDownloadProgress(0, '开始下载...')")
val success = ModelManager.downloadModels(context) { pct, msg ->
postToJs("window.__onVoiceDownloadProgress($pct, '${escapeJs(msg)}')")
}
if (success) {
postToJs("window.__onVoiceDownloadProgress(100, '下载完成')")
// 下载完成后立即预初始化引擎，这样用户点击时直接可用
ensureEngine()
} else {
postToJs("window.__onVoiceError('模型下载失败，请检查网络后重试')")
}
} catch (e: Exception) {
Log.e(TAG, "下载流程异常", e)
postToJs("window.__onVoiceError('${escapeJs(e.message ?: "未知错误")}')")
}
}
}
/** 释放识别引擎资源 */
private fun releaseEngine() {
initialized = false
engineInitJob?.cancel()
engineInitJob = null
try { recognizer?.release() } catch (_: Exception) {}
recognizer = null
try { vad?.release() } catch (_: Exception) {}
vad = null
}
/** 预初始化引擎（后台，不阻塞调用者）。下载完成后或 App 启动时调用 */
fun ensureEngine() {
if (initialized) return
if (!isModelReady()) {
logToFile("ensureEngine: 模型未就绪，跳过")
return
}
if (engineInitJob?.isActive == true) {
logToFile("ensureEngine: 已有初始化任务运行中")
return
}
logToFile("ensureEngine: 开始后台预初始化引擎")
engineInitJob = scope.launch(Dispatchers.IO) {
val ok = initEngine()
logToFile("ensureEngine: initEngine 返回 $ok")
if (ok) {
withContext(Dispatchers.Main) {
postToJs("window.__onVoiceModelReady()")
}
}
}
}
/** 获取模型大小（用于 UI 展示） */
@JavascriptInterface
fun getModelSizeMB(): Double {
return 228.0
}
/** 收集调试日志并上传到服务器 */
@JavascriptInterface
fun debugLog(msg: String) {
logToFile("[JS] $msg")
}
@JavascriptInterface
fun collectDebugLog(): String {
val sb = StringBuilder()
sb.appendLine("=== 语音调试日志 ===")
sb.appendLine("时间: ${dateFmt.format(Date())}")
sb.appendLine("模型目录: ${modelDir.absolutePath}")
sb.appendLine("senseVoice model: 存在=${senseVoiceModel.exists()}, 大小=${if (senseVoiceModel.exists()) senseVoiceModel.length() else -1}")
sb.appendLine("senseVoice tokens: 存在=${senseVoiceTokens.exists()}, 大小=${if (senseVoiceTokens.exists()) senseVoiceTokens.length() else -1}")
sb.appendLine("sileroVad: 存在=${sileroVadModel.exists()}, 大小=${if (sileroVadModel.exists()) sileroVadModel.length() else -1}")
sb.appendLine("isModelReady: ${ModelManager.isModelReady(context)}")
sb.appendLine("引擎已初始化: $initialized")
sb.appendLine("recognizer: ${recognizer != null}, vad: ${vad != null}")
sb.appendLine("录音权限: ${hasRecordPermission()}")
sb.appendLine("isRecording: ${isRecording.get()}")
sb.appendLine("")
sb.appendLine("=== 文件日志 ===")
if (logFile.exists()) {
try { sb.append(logFile.readText()) } catch (e: Exception) { sb.appendLine("(读取日志失败: ${e.message})") }
} else {
sb.appendLine("(无文件日志)")
}
return sb.toString()
}
@JavascriptInterface
fun startListening() {
logToFile("startListening 被调用, initialized=$initialized")
if (!hasRecordPermission()) {
logToFile("startListening: 缺少录音权限")
postToJs("window.__onVoiceError('缺少录音权限')")
Log.e(TAG, "缺少录音权限")
return
}
// 检查模型是否就绪
if (!isModelReady()) {
logToFile("startListening: 模型未就绪")
postToJs("window.__onVoiceStatus('model_not_ready')")
Log.w(TAG, "模型未就绪")
return
}
if (isRecording.get()) {
logToFile("startListening: 已在录音中，忽略")
Log.w(TAG, "已经在录音中")
return
}
if (!initialized) {
// 触发后台初始化并等待，完成后自动开始录音
logToFile("startListening: 引擎未初始化，触发后台初始化并等待")
postToJs("window.__onVoiceStatus('initializing')")
pendingStartJob?.cancel()
pendingStartJob = scope.launch(Dispatchers.IO) {
val ok = initEngine()
logToFile("startListening: initEngine 返回 $ok")
withContext(Dispatchers.Main) {
if (pendingStartJob == null || !pendingStartJob!!.isActive) {
logToFile("startListening: 启动请求在初始化期间被取消")
return@withContext
}
pendingStartJob = null
if (!ok) {
postToJs("window.__onVoiceError('模型初始化失败')")
return@withContext
}
startRecordingInternal()
}
}
} else {
startRecordingInternal()
}
}
@JavascriptInterface
fun stopListening() {
logToFile("stopListening 被调用, isRecording=${isRecording.get()}")
// 取消待处理的启动（初始化期间的取消）
pendingStartJob?.cancel()
pendingStartJob = null
// 如果还没开始录音（初始化中），直接通知前端停止
if (!isRecording.get()) {
logToFile("stopListening: 录音尚未开始，取消启动请求")
postToJs("window.__onVoiceStatus('idle')")
return
}
// 防抖：录音开始后 MIN_RECORDING_MS 内不允许停止
val elapsed = System.currentTimeMillis() - recordingStartTime
if (elapsed < MIN_RECORDING_MS) {
logToFile("stopListening: 录音时长不足 ${MIN_RECORDING_MS}ms，忽略 (已录 ${elapsed}ms)")
Log.w(TAG, "录音时长不足 ${MIN_RECORDING_MS}ms，忽略停止请求 (已录 ${elapsed}ms)")
return
}
stopRecordingInternal()
}
// ═══════════════════ 引擎初始化 ═══════════════════
private suspend fun initEngine(): Boolean = withContext(Dispatchers.IO) {
try {
// 先做文件大小校验，避免用残缺文件初始化导致 Native crash
if (!ModelManager.isModelReady(context)) {
logToFile("initEngine: 模型文件不完整，拒绝初始化")
Log.e(TAG, "模型文件不完整，拒绝初始化")
return@withContext false
}
logToFile("initEngine: 模型文件完整，开始加载 SenseVoice...")
Log.i(TAG, "初始化 SenseVoice 识别器... model=${senseVoiceModel.length()} tokens=${senseVoiceTokens.length()}")
// SenseVoice 模型配置
val senseVoiceConfig = OfflineSenseVoiceModelConfig(
model = senseVoiceModel.absolutePath,
useInverseTextNormalization = true
)
val featConfig = FeatureConfig(
sampleRate = SAMPLE_RATE,
featureDim = 80
)
val modelConfig = OfflineModelConfig()
modelConfig.senseVoice = senseVoiceConfig
modelConfig.tokens = senseVoiceTokens.absolutePath
modelConfig.numThreads = 1 // 单线程降低内存压力
modelConfig.provider = "cpu"
val config = OfflineRecognizerConfig(
featConfig = featConfig,
modelConfig = modelConfig
)
Log.i(TAG, "开始创建 OfflineRecognizer...")
logToFile("initEngine: 开始创建 OfflineRecognizer (numThreads=1)...")
recognizer = OfflineRecognizer(
assetManager = null,
config = config
)
logToFile("initEngine: OfflineRecognizer 创建完成")
Log.i(TAG, "SenseVoice 识别器初始化完成")
// VAD 暂时禁用（Android 端 Silero VAD 存在兼容性问题，改用整段识别）
logToFile("initEngine: 跳过 VAD，使用整段识别模式")
vad = null
initialized = true
logToFile("initEngine: 全部初始化完成")
true
} catch (e: Exception) {
logToFile("initEngine: 异常 ${e.javaClass.simpleName}: ${e.message}")
false
}
}
// ═══════════════════ 录音 ═══════════════════
// ── 待处理的启动任务（用于在初始化期间取消）──
private var pendingStartJob: Job? = null
// ── 录音数据缓冲（无 VAD 模式，整段识别）──
private val audioBuffer = mutableListOf<Short>()
private fun startRecordingInternal() {
logToFile("startRecordingInternal: 开始, isRecording=${isRecording.get()}")
if (isRecording.getAndSet(true)) {
logToFile("startRecordingInternal: 已在录音中，跳过")
return
}
val bufferSize = AudioRecord.getMinBufferSize(
SAMPLE_RATE,
AudioFormat.CHANNEL_IN_MONO,
AudioFormat.ENCODING_PCM_16BIT
)
logToFile("startRecordingInternal: bufferSize=$bufferSize")
audioRecord = AudioRecord(
MediaRecorder.AudioSource.MIC,
SAMPLE_RATE,
AudioFormat.CHANNEL_IN_MONO,
AudioFormat.ENCODING_PCM_16BIT,
bufferSize * 2
)
if (audioRecord?.state != AudioRecord.STATE_INITIALIZED) {
logToFile("startRecordingInternal: AudioRecord 初始化失败 state=${audioRecord?.state}")
Log.e(TAG, "AudioRecord 初始化失败")
isRecording.set(false)
postToJs("window.__onVoiceError('麦克风初始化失败')")
return
}
audioRecord?.startRecording()
recordingStartTime = System.currentTimeMillis()
audioBuffer.clear()
postToJs("window.__onVoiceStatus('listening')")
logToFile("startRecordingInternal: 录音已开始")
Log.i(TAG, "开始录音")
recordingJob = scope.launch {
processAudioLoop()
}
}
private suspend fun processAudioLoop() {
val buffer = ShortArray(CHUNK_SIZE)
while (isRecording.get()) {
val readCount = audioRecord?.read(buffer, 0, buffer.size) ?: -1
if (readCount <= 0) continue
for (i in 0 until readCount) {
audioBuffer.add(buffer[i])
}
}
}
private fun recognizeFullAudio() {
logToFile("recognizeFullAudio: buffer size=${audioBuffer.size}")
if (audioBuffer.isEmpty()) {
logToFile("recognizeFullAudio: buffer 为空")
postToJs("window.__onVoiceError('未检测到语音')")
return
}
scope.launch {
postToJs("window.__onVoiceStatus('processing')")
val samples = FloatArray(audioBuffer.size) { audioBuffer[it] / 32768.0f }
logToFile("recognizeFullAudio: 开始识别 ${samples.size} 采样 (${samples.size / SAMPLE_RATE}s)")
val text = withContext(Dispatchers.IO) { recognizeSegment(samples) }
logToFile("recognizeFullAudio: 识别结果 text='$text' length=${text.length}")
if (text.isNotBlank()) {
Log.i(TAG, "识别结果: $text")
logToFile("recognizeFullAudio: 准备调用 postToJs __onVoiceResult")
postToJs("window.__onVoiceResult('${escapeJs(text)}')")
logToFile("recognizeFullAudio: postToJs __onVoiceResult 已提交")
} else {
logToFile("recognizeFullAudio: 识别结果为空")
postToJs("window.__onVoiceError('未识别到语音内容')")
}
postToJs("window.__onVoiceStatus('idle')")
}
}
private fun recognizeSegment(samples: FloatArray): String {
val rec = recognizer ?: return ""
return try {
val stream = rec.createStream()
stream.acceptWaveform(samples, SAMPLE_RATE)
rec.decode(stream)
val result = rec.getResult(stream)
stream.release()
val text = result.text?.trim() ?: ""
logToFile("recognizeSegment: text='$text' lang='${result.lang}' emotion='${result.emotion}' event='${result.event}'")
text
} catch (e: Exception) {
logToFile("recognizeSegment: 异常 ${e.javaClass.simpleName}: ${e.message}")
Log.e(TAG, "识别错误", e)
""
}
}
private fun stopRecordingInternal() {
logToFile("stopRecordingInternal 被调用, isRecording=${isRecording.get()}")
if (!isRecording.getAndSet(false)) {
logToFile("stopRecordingInternal: isRecording 已为 false，跳过")
return
}
recordingJob?.cancel()
recordingJob = null
try {
audioRecord?.stop()
audioRecord?.release()
} catch (e: Exception) {
Log.w(TAG, "AudioRecord 释放异常", e)
logToFile("stopRecordingInternal: AudioRecord 释放异常 ${e.message}")
}
audioRecord = null
logToFile("stopRecordingInternal: 录音已停止，共 ${audioBuffer.size} 采样")
Log.i(TAG, "录音已停止，共 ${audioBuffer.size} 采样")
recognizeFullAudio()
}
// ═══════════════════ 工具方法 ═══════════════════
private fun hasRecordPermission(): Boolean {
return ActivityCompat.checkSelfPermission(
context, Manifest.permission.RECORD_AUDIO
) == PackageManager.PERMISSION_GRANTED
}
private fun postToJs(script: String) {
webView.post {
webView.evaluateJavascript(script, null)
}
}
private fun escapeJs(text: String): String {
return text
.replace("\\", "\\\\")
.replace("'", "\\'")
.replace("\n", "\\n")
.replace("\r", "\\r")
}
fun destroy() {
stopRecordingInternal()
scope.cancel()
releaseEngine()
}
}

467
demo/sense_voice_demo.py Normal file
View File

@ -0,0 +1,467 @@
#!/usr/bin/env python3
"""
SenseVoice int8 语音识别 Demo
================================
功能：麦克风录音 → Silero VAD 语音分段 → SenseVoice 识别（带标点）→ 实时显示
同时监控并输出 CPU / 内存占用
用法：
python3 demo/sense_voice_demo.py
首次运行会自动下载模型（约 228MB），请耐心等待。
按 Ctrl+C 退出时会输出性能统计。
"""
import sys
import os
import time
import argparse
import threading
import queue
import signal
from pathlib import Path
import numpy as np
# ── ANSI 颜色 ─────────────────────────────────────────────────
C = {
"reset": "\033[0m",
"bold": "\033[1m",
"dim": "\033[2m",
"green": "\033[32m",
"cyan": "\033[36m",
"yellow": "\033[33m",
"red": "\033[31m",
"magenta":"\033[35m",
"blue": "\033[94m",
}
def color(s, c):
return f"{c}{s}{C['reset']}"
# ── 模型下载 ─────────────────────────────────────────────────
MODEL_DIR = Path(__file__).parent / "models"
SENSEVOICE_URL = (
"https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/"
"sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2"
)
SILERO_VAD_URL = (
"https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/"
"silero_vad.onnx"
)
SENSEVOICE_DIR = MODEL_DIR / "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17"
SENSEVOICE_MODEL = SENSEVOICE_DIR / "model.int8.onnx"
SENSEVOICE_TOKENS = SENSEVOICE_DIR / "tokens.txt"
SILERO_VAD_MODEL = MODEL_DIR / "silero_vad.onnx"
def download_with_progress(url, dest):
"""下载文件并显示进度条"""
import urllib.request
import shutil
dest.parent.mkdir(parents=True, exist_ok=True)
print(f"\n 📥 下载 {Path(dest).name} ...")
print(f" {url}")
try:
with urllib.request.urlopen(url) as resp:
total = int(resp.headers.get("Content-Length", 0))
downloaded = 0
block_size = 8192
with open(dest, "wb") as f:
while True:
chunk = resp.read(block_size)
if not chunk:
break
f.write(chunk)
downloaded += len(chunk)
if total > 0:
pct = downloaded / total * 100
bar_len = 30
filled = int(bar_len * downloaded / total)
bar = "█" * filled + "░" * (bar_len - filled)
size_mb = downloaded / (1024 * 1024)
total_mb = total / (1024 * 1024)
print(
f"\r [{bar}] {pct:5.1f}% {size_mb:.0f}/{total_mb:.0f}MB",
end="", flush=True,
)
print()
except Exception as e:
if dest.exists():
dest.unlink()
raise e
def ensure_models():
"""确保模型文件存在，不存在则下载"""
import tarfile
MODEL_DIR.mkdir(parents=True, exist_ok=True)
# SenseVoice
if not SENSEVOICE_MODEL.exists():
archive = MODEL_DIR / "sense_voice_int8.tar.bz2"
if not archive.exists():
download_with_progress(SENSEVOICE_URL, archive)
print(f" 📦 解压模型...")
with tarfile.open(archive, "r:bz2") as tar:
tar.extractall(path=MODEL_DIR)
archive.unlink()
print(f" ✅ SenseVoice 模型就绪 ({SENSEVOICE_MODEL.stat().st_size / 1024 / 1024:.0f}MB)")
# Silero VAD
if not SILERO_VAD_MODEL.exists():
download_with_progress(SILERO_VAD_URL, SILERO_VAD_MODEL)
print(f" ✅ Silero VAD 模型就绪")
print()
# ── 性能监控 ─────────────────────────────────────────────────
class PerfMonitor:
"""后台线程采集 CPU / 内存"""
def __init__(self, interval=1.0):
self.interval = interval
self._stop = threading.Event()
self._thread = None
self._lock = threading.Lock()
# 累计统计
self.cpu_percent_samples = []
self.rss_mb_samples = []
self.last_cpu = 0.0
self.last_rss_mb = 0.0
try:
import psutil
self._psutil = psutil
self._proc = psutil.Process()
self._available = True
except ImportError:
self._available = False
@property
def available(self):
return self._available
def start(self):
if not self._available:
return
self._stop.clear()
self._thread = threading.Thread(target=self._run, daemon=True)
self._thread.start()
def stop(self):
self._stop.set()
if self._thread:
self._thread.join(timeout=2)
def _run(self):
while not self._stop.is_set():
try:
cpu = self._proc.cpu_percent(interval=None)
rss_mb = self._proc.memory_info().rss / 1024 / 1024
except Exception:
cpu, rss_mb = 0, 0
with self._lock:
self.last_cpu = cpu
self.last_rss_mb = rss_mb
self.cpu_percent_samples.append(cpu)
self.rss_mb_samples.append(rss_mb)
self._stop.wait(self.interval)
def current(self):
with self._lock:
return self.last_cpu, self.last_rss_mb
def summary(self):
with self._lock:
if not self.cpu_percent_samples:
return "无数据"
cpu_avg = np.mean(self.cpu_percent_samples)
cpu_max = np.max(self.cpu_percent_samples)
rss_avg = np.mean(self.rss_mb_samples)
rss_max = np.max(self.rss_mb_samples)
return {
"cpu_avg": cpu_avg,
"cpu_max": cpu_max,
"rss_avg_mb": rss_avg,
"rss_max_mb": rss_max,
"samples": len(self.cpu_percent_samples),
}
# ── 识别统计 ─────────────────────────────────────────────────
class ASRStats:
"""记录每次识别的性能"""
def __init__(self):
self._lock = threading.Lock()
self.segments = [] # list of (audio_duration, asr_time, text)
def record(self, audio_duration, asr_time, text):
with self._lock:
self.segments.append((audio_duration, asr_time, text))
def summary(self):
with self._lock:
if not self.segments:
return None
total_audio = sum(s[0] for s in self.segments)
total_asr = sum(s[1] for s in self.segments)
total_text = sum(len(s[2]) for s in self.segments)
rtf = total_asr / total_audio if total_audio > 0 else 0
return {
"segments": len(self.segments),
"total_audio_s": total_audio,
"total_asr_s": total_asr,
"total_chars": total_text,
"rtf": rtf,
"speedup": 1 / rtf if rtf > 0 else float("inf"),
}
# ── 主程序 ────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser(description="SenseVoice int8 语音识别 Demo")
parser.add_argument("--device", type=int, default=None,
help="音频输入设备编号（不指定则用默认）")
parser.add_argument("--list-devices", action="store_true",
help="列出所有音频设备后退出")
parser.add_argument("--save-audio", type=str, default=None,
help="保存最后一次识别的音频到指定文件（用于调试）")
parser.add_argument("--num-threads", type=int, default=2,
help="ONNX 推理线程数（默认 2）")
parser.add_argument("--vad-threshold", type=float, default=0.5,
help="VAD 灵敏度 0~1（默认 0.5，越小越敏感）")
parser.add_argument("--silence-duration", type=float, default=0.8,
help="判定为句尾的静音时长秒数（默认 0.8）")
parser.add_argument("--max-speech", type=float, default=30,
help="单段最长语音秒数（默认 30）")
args = parser.parse_args()
# ── 列出设备 ──
import sounddevice as sd
if args.list_devices:
print("\n🎤 可用音频设备：\n")
devices = sd.query_devices()
for i, d in enumerate(devices):
in_ch = d["max_input_channels"]
if in_ch > 0:
print(f" [{i}] {d['name']} (输入通道: {in_ch}, 默认采样率: {d['default_samplerate']:.0f})")
print()
return
# ── 下载模型 ──
print(color("\n🔧 检查模型文件...", C["cyan"]))
ensure_models()
# ── 初始化 sherpa-onnx ──
print(color("🚀 初始化识别引擎...", C["cyan"]))
import sherpa_onnx
# 通过工厂方法创建 SenseVoice 识别器
recognizer = sherpa_onnx.OfflineRecognizer.from_sense_voice(
model=str(SENSEVOICE_MODEL),
tokens=str(SENSEVOICE_TOKENS),
num_threads=args.num_threads,
provider="cpu",
language="auto",
use_itn=True, # 启用标点 + 逆文本正则化
)
# Silero VAD 配置
silero_vad_config = sherpa_onnx.SileroVadModelConfig(
model=str(SILERO_VAD_MODEL),
threshold=args.vad_threshold,
min_silence_duration=args.silence_duration,
min_speech_duration=0.25,
max_speech_duration=args.max_speech,
)
vad_config = sherpa_onnx.VadModelConfig(
silero_vad=silero_vad_config,
sample_rate=16000,
num_threads=1,
)
vad = sherpa_onnx.VoiceActivityDetector(vad_config, buffer_size_in_seconds=60)
print(color("✅ 引擎就绪！", C["green"]))
# ── 启动性能监控 ──
perf = PerfMonitor(interval=1.0)
perf.start()
asr_stats = ASRStats()
# ── 音频参数 ──
SAMPLE_RATE = 16000
BLOCK_SIZE = 1024 # 每次读取的采样数
# ── 优雅退出 ──
running = True
def on_sigint(sig, frame):
nonlocal running
running = False
print(f"\n{color('⏹ 正在退出...', C['yellow'])}")
signal.signal(signal.SIGINT, on_sigint)
# ── 打印标题 ──
print()
print(color("╔════════════════════════════════════════════════╗", C["bold"]))
print(color("║ 🎙️ SenseVoice 语音识别 Demo ║", C["bold"]))
print(color("║ 中英混说 · 自带标点 · 低资源运行 ║", C["bold"]))
print(color("╚════════════════════════════════════════════════╝", C["bold"]))
print()
print(f" {color('🎤', C['cyan'])} 对着麦克风说话，说完停顿即可看到结果")
print(f" {color('⏸', C['cyan'])} 按 {color('Ctrl+C', C['yellow'])} 退出并查看统计")
print(f" {color('📊', C['cyan'])} VAD 阈值: {args.vad_threshold} | 静音判定: {args.silence_duration}s")
print()
# 状态指示
STATUS_IDLE = f" {color('⏳', C['dim'])} 等待语音..."
STATUS_LISTENING = f" {color('🔴', C['red'])} 正在听..."
STATUS_PROCESSING = f" {color('⚙️', C['yellow'])} 识别中..."
sys.stdout.write(STATUS_IDLE)
sys.stdout.flush()
def audio_callback(indata, frames, time_info, status):
"""麦克风回调：将音频送入 VAD"""
if status:
print(f"\n ⚠️ 音频状态: {status}")
# 转为 float32 并归一化到 [-1, 1]
if indata.dtype == np.int16:
samples = indata[:, 0].astype(np.float32) / 32768.0
else:
samples = indata[:, 0].astype(np.float32)
vad.accept_waveform(samples)
try:
with sd.InputStream(
samplerate=SAMPLE_RATE,
device=args.device,
channels=1,
dtype="float32",
blocksize=BLOCK_SIZE,
callback=audio_callback,
):
last_status = STATUS_IDLE
segment_count = 0
while running:
time.sleep(0.05) # 50ms 轮询
# 检测是否有语音段结束
while not vad.empty():
speech_segment = vad.front # sherpa_onnx.SpeechSegment
samples = np.array(speech_segment.samples, dtype=np.float32)
audio_duration = len(samples) / SAMPLE_RATE
vad.pop()
# ── 识别 ──
sys.stdout.write(f"\r{STATUS_PROCESSING} ({audio_duration:.1f}s 音频)")
sys.stdout.flush()
t0 = time.perf_counter()
stream = recognizer.create_stream()
stream.accept_waveform(SAMPLE_RATE, samples)
recognizer.decode_stream(stream)
text = stream.result.text.strip()
t1 = time.perf_counter()
asr_time = t1 - t0
asr_stats.record(audio_duration, asr_time, text)
# ── 显示结果 ──
segment_count += 1
rtf = asr_time / audio_duration if audio_duration > 0 else 0
speed = 1 / rtf if rtf > 0 else float("inf")
# 清除状态行
sys.stdout.write("\r" + " " * 60 + "\r")
# 输出识别文本
prefix = color(f"[{segment_count}]", C["dim"])
speed_info = color(
f"({asr_time:.2f}s / {audio_duration:.1f}s, RTF={rtf:.3f}, {speed:.0f}x)",
C["dim"],
)
print(f" {prefix} {color(text, C['green'])}")
print(f" {speed_info}")
# 保存音频（调试用）
if args.save_audio and segment_count == 1:
import sherpa_onnx
sherpa_onnx.write_wave(
str(args.save_audio),
samples.tolist(),
SAMPLE_RATE,
)
print(f" 💾 音频已保存到 {args.save_audio}")
# 恢复状态
sys.stdout.write(STATUS_IDLE)
sys.stdout.flush()
# 更新状态行
current_status = STATUS_LISTENING if vad.is_speech_detected() else STATUS_IDLE
if current_status != last_status:
sys.stdout.write(f"\r{current_status}")
sys.stdout.flush()
last_status = current_status
except KeyboardInterrupt:
pass
finally:
running = False
perf.stop()
# ── 输出统计 ──
print("\n")
print(color("╔════════════════════════════════════════════════╗", C["bold"]))
print(color("║ 📊 性能统计 ║", C["bold"]))
print(color("╚════════════════════════════════════════════════╝", C["bold"]))
print()
# ASR 统计
s = asr_stats.summary()
if s:
print(color(" 🎙️ 识别统计", C["cyan"]))
print(f" 识别段数: {s['segments']}")
print(f" 总音频时长: {s['total_audio_s']:.1f}s")
print(f" 总识别耗时: {s['total_asr_s']:.2f}s")
print(f" 总字符数: {s['total_chars']}")
print(f" RTF: {s['rtf']:.4f}")
print(f" 处理速度: {s['speedup']:.1f}x 实时")
print()
# 系统资源统计
ps = perf.summary()
if isinstance(ps, dict):
print(color(" 💻 系统资源", C["cyan"]))
print(f" CPU 平均: {ps['cpu_avg']:.1f}%")
print(f" CPU 峰值: {ps['cpu_max']:.1f}%")
print(f" 内存平均: {ps['rss_avg_mb']:.0f}MB")
print(f" 内存峰值: {ps['rss_max_mb']:.0f}MB")
print(f" 采样点数: {ps['samples']}")
print()
elif not perf.available:
print(color(" ⚠️ 未安装 psutil，无法采集 CPU/内存数据", C["yellow"]))
print(color(" 安装: pip3 install psutil", C["dim"]))
print()
if __name__ == "__main__":
main()

15
server/app_legacy.py
View File

@ -1384,6 +1384,21 @@ def resource_busy_page():
return app.send_static_file('resource_busy.html'), 503
@app.route('/api/voice_debug', methods=['POST'])
def voice_debug():
"""接收手机端语音调试日志"""
import os, datetime
data = request.get_data(as_text=True)
log_dir = os.environ.get('LOGS_DIR', os.path.join(os.path.dirname(__file__), '..', 'logs'))
voice_log_dir = os.path.join(log_dir, 'voice_debug')
os.makedirs(voice_log_dir, exist_ok=True)
ts = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
filename = os.path.join(voice_log_dir, f'voice_{ts}.log')
with open(filename, 'w') as f:
f.write(data)
return jsonify({'ok': True, 'file': filename})
if __name__ == "__main__":
args = parse_arguments()
run_server(

131
static/src/components/input/InputComposer.vue
View File

@ -207,13 +207,13 @@
</button>
<div class="input-actions-right">
<button
v-if="isWebSpeechSupported"
v-if="isVoiceInputSupported"
type="button"
class="stadium-btn voice-btn"
:class="{ 'voice-btn--recording': isRecording }"
:class="{ 'voice-btn--recording': isRecording, 'voice-btn--disabled': voiceModelStatus === 'downloading' || voiceModelStatus === 'model_not_ready' }"
@click="toggleVoiceRecording"
:disabled="!isConnected || inputLocked"
title="语音输入"
:title="voiceButtonTitle"
>
<template v-if="!isRecording">
<svg
@ -1185,6 +1185,114 @@ const recognitionInstance = ref<SpeechRecognition | null>(null);
let voiceAnchorPos = 0;
let lastVoiceTextLength = 0;
// 模型状态：idle | downloading | ready | model_not_ready | error
const voiceModelStatus = ref<string>('idle');
const voiceDownloadPercent = ref(0);
const voiceDownloadMsg = ref('');
// 是否支持语音输入（仅 Android APK 环境，通过 VoiceBridge）
const isVoiceInputSupported = computed(() => {
if (typeof window === 'undefined') return false;
return !!(window as any).AndroidVoiceBridge;
});
const voiceButtonTitle = computed(() => {
if (voiceModelStatus.value === 'downloading') return `模型下载中 ${voiceDownloadPercent.value}%`;
if (voiceModelStatus.value === 'model_not_ready') return '模型未下载，请在个人空间下载';
if (isRecording.value) return '停止录音';
return '语音输入';
});
// 注册全局回调（Android Bridge 会调用这些函数）
const setupVoiceBridgeCallbacks = () => {
if (typeof window === 'undefined') return;
(window as any).__onVoiceResult = (text: string) => {
console.log('[VoiceInput] Android 识别结果:', text);
const bridge = (window as any).AndroidVoiceBridge;
bridge?.debugLog('__onVoiceResult 被调用, text=' + JSON.stringify(text));
if (!text) {
bridge?.debugLog('__onVoiceResult: text 为空，跳过');
return;
}
const ed = editor.value;
if (!ed) {
bridge?.debugLog('__onVoiceResult: editor.value 为 null');
console.warn('[VoiceInput] editor is null');
return;
}
bridge?.debugLog('__onVoiceResult: editor.value 存在，准备插入文字');
bridge?.debugLog('__onVoiceResult: voiceAnchorPos=' + voiceAnchorPos + ' lastVoiceTextLength=' + lastVoiceTextLength);
const { state, view } = ed;
const docSize = state.doc.content.size;
const safeAnchor = Math.max(0, Math.min(voiceAnchorPos, docSize));
const oldEnd = Math.min(safeAnchor + lastVoiceTextLength, docSize);
bridge?.debugLog('__onVoiceResult: docSize=' + docSize + ' safeAnchor=' + safeAnchor + ' oldEnd=' + oldEnd);
try {
let tr = state.tr;
if (oldEnd > safeAnchor) tr = tr.delete(safeAnchor, oldEnd);
tr = tr.insertText(text, safeAnchor);
const newCursor = Math.min(safeAnchor + text.length, tr.doc.content.size);
const TextSelection = (window as any).TextSelection;
if (TextSelection) {
tr = tr.setSelection(TextSelection.create(tr.doc, newCursor));
}
view.dispatch(tr);
view.focus();
lastVoiceTextLength = text.length;
emit('update:input-message', getEditorPlainText());
emit('input-change');
nextTick(() => adjustTextareaSize());
bridge?.debugLog('__onVoiceResult: 文字插入成功');
} catch (e: any) {
bridge?.debugLog('__onVoiceResult: 插入异常 ' + (e.message || String(e)));
}
};
(window as any).__onVoiceStatus = (status: string) => {
console.log('[VoiceInput] 状态:', status);
if (status === 'listening' || status === 'processing' || status === 'initializing') {
// initializing 也视为录音中，防止用户重复点击触发多次 startListening
isRecording.value = true;
} else if (status === 'idle') {
isRecording.value = false;
voiceAnchorPos = 0;
lastVoiceTextLength = 0;
} else if (status === 'model_not_ready') {
voiceModelStatus.value = 'model_not_ready';
}
};
(window as any).__onVoiceError = (error: string) => {
console.warn('[VoiceInput] 错误:', error);
isRecording.value = false;
voiceAnchorPos = 0;
lastVoiceTextLength = 0;
};
(window as any).__onVoiceDownloadProgress = (pct: number, msg: string) => {
voiceModelStatus.value = 'downloading';
voiceDownloadPercent.value = pct;
voiceDownloadMsg.value = msg;
if (pct >= 100) {
voiceModelStatus.value = 'ready';
}
};
(window as any).__onVoiceModelReady = () => {
voiceModelStatus.value = 'ready';
};
};
// 初始化时设置回调和检查模型状态
if (typeof window !== 'undefined') {
setupVoiceBridgeCallbacks();
// 检查 Android Bridge 模型状态
const bridge = (window as any).AndroidVoiceBridge;
if (bridge && typeof bridge.isModelReady === 'function') {
try {
voiceModelStatus.value = bridge.isModelReady() ? 'ready' : 'model_not_ready';
} catch (_) {
voiceModelStatus.value = 'idle';
}
}
}
const isWebSpeechSupported = computed(() => {
if (typeof window === 'undefined') return false;
const hasSR = 'SpeechRecognition' in window;
@ -1340,6 +1448,23 @@ const initSpeechRecognition = () => {
const toggleVoiceRecording = () => {
console.log('[VoiceInput] toggleVoiceRecording, isRecording=', isRecording.value);
// Android Bridge 模式
const bridge = (window as any).AndroidVoiceBridge;
if (bridge) {
if (isRecording.value) {
bridge.stopListening();
return;
}
// 记录锚点
const instance = editor.value;
voiceAnchorPos = instance ? instance.state.selection.from : 0;
lastVoiceTextLength = 0;
bridge.startListening();
return;
}
// 浏览器原生 Web Speech API 模式
if (isRecording.value) {
stopVoiceRecording();
return;

170
static/src/components/personalization/PersonalizationDrawer.vue
View File

@ -1470,6 +1470,68 @@
</div>
</section>
<section
v-else-if="activeTab === 'voice'"
key="voice"
class="settings-page"
>
<div class="settings-section" style="margin-bottom: 16px;">
<p class="settings-section-desc" style="margin: 0; color: var(--text-secondary); font-size: 13px; line-height: 1.6;">
端侧语音识别模型（SenseVoice int8），支持中英混说 + 自动标点。
模型约 228MB，仅在手机本地运行，无需网络。
</p>
</div>
<div class="settings-action-row">
<span class="settings-row-copy">
<span class="settings-row-title">语音识别模型</span>
<span class="settings-row-desc">
<template v-if="voiceModelReady">已下载 (228MB)</template>
<template v-else-if="voiceDownloading">下载中 {{ voiceDownloadPercent }}% — {{ voiceDownloadMsg }}</template>
<template v-else-if="voiceModelPartial">下载不完整，请重新下载</template>
<template v-else>未下载 — 点击下载</template>
</span>
</span>
<div style="display: flex; gap: 6px;">
<button
type="button"
class="settings-secondary-button"
:disabled="voiceDownloading"
@click="downloadVoiceModel"
>
{{ voiceDownloading ? '下载中...' : voiceModelReady ? '重新下载' : '下载模型' }}
</button>
<button
v-if="voiceModelReady || voiceModelPartial"
type="button"
class="settings-secondary-button"
style="color: var(--state-error);"
:disabled="voiceDownloading"
@click="deleteVoiceModel"
>
删除
</button>
</div>
</div>
<div class="settings-action-row" style="margin-top: 8px;">
<span class="settings-row-copy">
<span class="settings-row-desc" style="font-size: 12px;">点击麦克风闪退？先复现一次，再点「上传日志」</span>
</span>
<button
type="button"
class="settings-secondary-button"
@click="saveDebugLog"
>
上传日志
</button>
</div>
<div v-if="voiceDownloading" class="voice-download-bar" style="margin: 8px 0 0;">
<div class="voice-download-track">
<div class="voice-download-fill" :style="{ width: voiceDownloadPercent + '%' }"></div>
</div>
<span style="font-size: 11px; color: var(--text-secondary); margin-top: 4px;">{{ voiceDownloadPercent }}%</span>
</div>
</section>
<section
v-else-if="activeTab === 'admin'"
key="admin"
@ -1641,7 +1703,8 @@ const baseTabs = [
{ id: 'context', label: '上下文', icon: 'chatBubble' },
{ id: 'tools', label: '工具与 Skills', icon: 'wrench' },
{ id: 'files', label: '文件与图片', icon: 'file' },
{ id: 'data', label: '数据管理', icon: 'layers' }
{ id: 'data', label: '数据管理', icon: 'layers' },
{ id: 'voice', label: '语音模型', icon: 'mic' },
] as const satisfies ReadonlyArray<{ id: PersonalTab; label: string; icon: IconKey }>;
const sessionRole = ref('');
@ -1683,6 +1746,91 @@ const activeTabLabel = computed(() => {
return personalTabs.value.find((tab) => tab.id === activeTab.value)?.label || '常规';
});
// ── 语音模型下载 ──
const voiceModelReady = ref(false);
const voiceModelPartial = ref(false); // 文件存在但不完整
const voiceDownloading = ref(false);
const voiceDownloadPercent = ref(0);
const voiceDownloadMsg = ref('');
const checkVoiceModel = () => {
const bridge = (window as any)?.AndroidVoiceBridge;
if (bridge) {
try {
if (typeof bridge.isModelReady === 'function') {
voiceModelReady.value = bridge.isModelReady();
}
if (!voiceModelReady.value && typeof bridge.isModelPartial === 'function') {
voiceModelPartial.value = bridge.isModelPartial();
}
} catch (_) { /* ignore */ }
}
};
const downloadVoiceModel = () => {
const bridge = (window as any)?.AndroidVoiceBridge;
if (!bridge) {
alert('仅在 App 内支持下载语音模型');
return;
}
voiceDownloading.value = true;
voiceDownloadPercent.value = 0;
voiceDownloadMsg.value = '准备下载...';
voiceModelPartial.value = false;
(window as any).__onVoiceDownloadProgress = (pct: number, msg: string) => {
voiceDownloadPercent.value = pct;
voiceDownloadMsg.value = msg;
if (pct >= 100) {
voiceDownloading.value = false;
voiceModelReady.value = true;
voiceModelPartial.value = false;
}
};
(window as any).__onVoiceModelReady = () => {
voiceModelReady.value = true;
voiceDownloading.value = false;
voiceModelPartial.value = false;
};
bridge.downloadModel();
};
const deleteVoiceModel = () => {
const bridge = (window as any)?.AndroidVoiceBridge;
if (!bridge) return;
if (typeof bridge.deleteModel === 'function') {
bridge.deleteModel();
}
voiceModelReady.value = false;
voiceModelPartial.value = false;
voiceDownloadPercent.value = 0;
voiceDownloadMsg.value = '';
};
const saveDebugLog = async () => {
const bridge = (window as any)?.AndroidVoiceBridge;
if (!bridge) return;
let log = '';
if (typeof bridge.collectDebugLog === 'function') {
log = bridge.collectDebugLog();
}
if (!log) { alert('无法收集日志'); return; }
try {
const res = await fetch('/api/voice_debug', { method: 'POST', body: log });
if (res.ok) {
alert('日志已上传到服务器');
} else {
alert('上传失败: ' + res.status);
}
} catch (e: any) {
alert('上传失败: ' + (e.message || e));
}
};
// 打开个人空间时检查模型状态
checkVoiceModel();
const updateFloatingMenuPosition = async () => {
if (!activeDropdown.value || typeof window === 'undefined') {
floatingMenuStyle.value = {};
@ -3300,4 +3448,24 @@ const applyDefaultHideWorkspaceOption = async (hidden: boolean) => {
grid-template-columns: 1fr;
}
}
/* ── 语音模型下载进度条 ── */
.voice-download-bar {
display: flex;
flex-direction: column;
gap: 4px;
}
.voice-download-track {
width: 100%;
height: 6px;
background: var(--surface-muted);
border-radius: 3px;
overflow: hidden;
}
.voice-download-fill {
height: 100%;
background: var(--accent-primary);
border-radius: 3px;
transition: width 0.3s ease;
}
</style>

6
static/src/styles/components/input/_composer.scss
View File

@ -838,6 +838,12 @@ body[data-theme='light'] {
background: var(--hover-bg) !important;
}
.voice-btn--disabled {
opacity: 0.5;
cursor: not-allowed;
color: var(--text-secondary);
}
.voice-wave {
display: flex;
align-items: center;

1
static/src/utils/icons.ts
View File

@ -25,6 +25,7 @@ export const ICONS = Object.freeze({
layers: '/static/icons/layers.svg',
keyboard: '/static/icons/keyboard.svg',
menu: '/static/icons/menu.svg',
mic: '/static/icons/mic.svg',
mcpLogo: '/static/icons/mcp-logo.svg',
monitor: '/static/icons/monitor.svg',
navigation: '/static/icons/navigation.svg',