简单记录一下实际部署过程。
总体思路
- Qwen3-ASR：运行语音转文字服务。
- llama.cpp：运行 LLM，用于对转出的文字进行润色或翻译。
- ASR 网关：转换浏览器上传的语音格式，并将请求转给 Qwen3-ASR。
上述功能使用三只容器实现：Qwen3-ASR、ASR 网关和 llama.cpp。
在我的实现中 ASR 和网关部署在一台后端服务器上，LLM 部署在另一台后端服务器上。
公网服务器运行 Nginx、FRPS 和静态页面。两台后端服务器通过 FRP 接入公网服务器，浏览器只需访问公网服务器 Nginx。
系统结构
TEXT
| |
浏览器
│ HTTPS
▼
Nginx
├─ /voice/ ──> Voice Workbench 静态页面
├─ /voice-asr/ ──> FRP ──> ASR 网关容器 ──> Qwen3-ASR 容器
└─ /voice-llm/ ──> FRP ──> llama.cpp 容器
- ASR 将音频转换为文字。
- ASR 网关先用 FFmpeg 将浏览器上传的音频统一转换为单声道、16 kHz、16-bit PCM WAV，再调用 Qwen3-ASR，同时清理模型输出中的标记。
- LLM 负责纠错、整理、润色和翻译。
第一部分：部署过程
启动 LLM
我使用 Qwen3.6-27B 的 Q6_K 量化版本作为 LLM 模型，实际文件为：
TEXT
| |
Qwen_Qwen3.6-27B-Q6_K.gguf
llama.cpp 加载的是 GGUF 模型。
下载前，先在 Hugging Face
搜索模型名和 GGUF，然后在仓库的 Files and versions 页面中找到需要的量化文件。文件下载链接的格式是：
TEXT
| |
https://huggingface.co/<作者>/<仓库>/resolve/main/<文件名>
GGUF 是单个文件，直接用 wget 下载：
BASH
| |
sudo mkdir -p /srv/llm-backend/models/qwen36-27b /srv/llm-backend/cache
cd /srv/llm-backend/models/qwen36-27b
sudo wget -c https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF/resolve/main/Qwen_Qwen3.6-27B-Q6_K.gguf
启动 Docker：
Docker Compose 配置为文末的 qwen36-27b-q6.yaml。将其放进 /srv/llm-backend/，然后启动：
BASH
| |
cd /srv/llm-backend
docker compose -f qwen36-27b-q6.yaml up -d
此时 llama.cpp 容器应当已经启动，并在宿主机的 18080 端口提供 OpenAI 兼容接口，模型名称为 qwen3.6-27b。
我的配置使用两张 24 GB 显卡。容器内设置 CUDA_VISIBLE_DEVICES=1,2，并用 --tensor-split 1,1 平分模型。
启动 ASR 和网关
ASR 使用 Qwen/Qwen3-ASR-1.7B。容器首次启动时会自动下载模型，并保存到宿主机的持久化缓存中。宿主机不需要安装 huggingface_hub。
创建目录：
BASH
| |
mkdir -p ~/asr-stt/model-cache
cd ~/asr-stt
启动 Docker：
将文末的 compose.yaml 和 gateway.py 放进当前目录。这个 Compose 会同时启动 Qwen3-ASR 和 ASR 网关：
BASH
| |
docker compose up -d
第一次启动需要等待容器下载模型。下载完成后，Qwen3-ASR 应当监听宿主机的 18101 端口，ASR 网关应当监听 18100 端口。
Nginx 应连接网关的 18100，而不是直接连接原始 ASR 的 18101。
用 FRP 接回公网服务器
公网服务器运行 FRPS，两台后端服务器运行宿主机上的 frpc.service。FRPC 不在 Docker 中运行。
以下配置省略公网地址、端口和 Token。
LLM 节点的 frpc.toml 将 llama.cpp 的 18080 端口映射到公网服务器：
TOML
| |
serverAddr = "<PUBLIC_SERVER>"
serverPort = <FRPS_PORT>
auth.token = "<FRP_TOKEN>"
[[proxies]]
name = "voice-llm"
type = "tcp"
localIP = "127.0.0.1"
localPort = 18080
remotePort = <LLM_TUNNEL_PORT>
ASR 节点的 frpc.toml 将网关的 18100 端口映射到公网服务器：
TOML
| |
serverAddr = "<PUBLIC_SERVER>"
serverPort = <FRPS_PORT>
auth.token = "<FRP_TOKEN>"
[[proxies]]
name = "voice-asr"
type = "tcp"
localIP = "127.0.0.1"
localPort = 18100
remotePort = <ASR_TUNNEL_PORT>
配置完成后启动 FRPC：
BASH
| |
sudo systemctl enable --now frpc
此时公网服务器上应当已经出现 ASR 和 LLM 对应的两个 FRP 映射端口。
Nginx 页面示例
将静态 Workbench 放到 /var/www/voice/index.html。Nginx 提供页面，并将两个 API 路径分别转发到 ASR 和 LLM 的 FRP 端口。
下面是省略了认证和限流配置的示例：
NGINX
| |
server {
listen 443 ssl;
server_name voice.example.com;
ssl_certificate /etc/letsencrypt/live/voice.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/voice.example.com/privkey.pem;
location = / {
return 302 /voice/;
}
location = /voice/ {
root /var/www/voice;
try_files /index.html =404;
add_header Cache-Control "no-store" always;
add_header Permissions-Policy "microphone=(self)" always;
}
location /voice-asr/ {
client_max_body_size 32m;
proxy_pass http://127.0.0.1:<ASR_TUNNEL_PORT>/;
proxy_request_buffering off;
proxy_buffering off;
proxy_read_timeout 900s;
}
location /voice-llm/ {
proxy_pass http://127.0.0.1:<LLM_TUNNEL_PORT>/;
proxy_read_timeout 300s;
}
}
Workbench 中对应的 API 地址为：
JAVASCRIPT
| |
const ROUTES = {
asrModels: "/voice-asr/v1/models",
asrTranscriptions: "/voice-asr/v1/audio/transcriptions",
llmModels: "/voice-llm/v1/models",
llmChat: "/voice-llm/v1/chat/completions",
};
重载 Nginx：
BASH
| |
sudo systemctl reload nginx
此时静态页面应当可以通过 /voice/ 打开，ASR 和 LLM 请求则分别进入 /voice-asr/ 和 /voice-llm/。
第二部分：完整配置
以下是部署时需要使用的完整 Compose 和网关源码。平时只需看上面的部署过程，重建服务时再复制这里的文件。
这些配置由运行中容器的镜像、命令、端口、挂载和 GPU 参数还原。代理地址、FRP Token、API Key、域名和隐藏路径均已脱敏。
线上没有自定义 Dockerfile；三个容器都直接使用上游镜像，因此需要保存的是 Compose 和网关源码。
LLM Compose
qwen36-27b-q6.yaml：
YAML
| |
name: llm-backend
services:
qwen36_27b_q6:
image: ghcr.io/ggml-org/llama.cpp:server-cuda@sha256:a50b12bb92de0253d2737824ca1887f410e07b4dd3e3028f74a5a0a67c789e4b
container_name: qwen36_27b_q6
restart: unless-stopped
ports:
- "18080:8080"
environment:
CUDA_VISIBLE_DEVICES: "1,2"
gpus: all
volumes:
- ./models/qwen36-27b:/models/qwen36-27b:ro
- ./cache:/cache
command:
- -m
- /models/qwen36-27b/Qwen_Qwen3.6-27B-Q6_K.gguf
- --alias
- qwen3.6-27b
- --host
- 0.0.0.0
- --port
- "8080"
- --ctx-size
- "65536"
- --batch-size
- "2048"
- --ubatch-size
- "512"
- --n-gpu-layers
- all
- --parallel
- "1"
- --flash-attn
- "on"
- --cache-type-k
- q8_0
- --cache-type-v
- q8_0
- --split-mode
- tensor
- --tensor-split
- 1,1
- --fit
- "off"
- --metrics
- --sleep-idle-seconds
- "1800"
- --jinja
- --reasoning
- "off"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
ASR Compose
compose.yaml：
YAML
| |
name: qwen3-asr
services:
qwen3_asr:
image: qwenllm/qwen3-asr@sha256:fb75b775f089e06e5a1aaebffd421e37505cc630d50c86d889d95ffa45a7e16a
container_name: qwen3_asr_17b
restart: unless-stopped
ports:
- "127.0.0.1:18101:8000"
shm_size: 4gb
volumes:
- ./model-cache:/root/.cache/huggingface
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["1"]
capabilities: [gpu]
command:
- qwen-asr-serve
- Qwen/Qwen3-ASR-1.7B
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- "0.80"
- --max-model-len
- "16384"
- --served-model-name
- qwen3-asr-1.7b
gateway:
image: qwenllm/qwen3-asr@sha256:fb75b775f089e06e5a1aaebffd421e37505cc630d50c86d889d95ffa45a7e16a
container_name: qwen3_asr_gateway
restart: unless-stopped
depends_on:
- qwen3_asr
ports:
- "127.0.0.1:18100:8080"
volumes:
- ./gateway.py:/opt/asr-gateway/gateway.py:ro
command:
- python3
- -m
- uvicorn
- gateway:app
- --app-dir
- /opt/asr-gateway
- --host
- 0.0.0.0
- --port
- "8080"
- --workers
- "1"
若下载模型必须经过代理，在 qwen3_asr.environment 中添加 HTTP_PROXY、HTTPS_PROXY 和 NO_PROXY。代理值不应提交到公开仓库。
ASR 网关
gateway.py：
PYTHON
| |
#!/usr/bin/env python3
"""Small adapter exposing one OpenAI-compatible ASR contract."""
import asyncio
import json
import os
import re
import tempfile
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, Response
from starlette.datastructures import UploadFile
UPSTREAM = "http://qwen3_asr:8000"
MAX_AUDIO_BYTES = 32 * 1024 * 1024
FFMPEG_TIMEOUT_SECONDS = 300
HOP_BY_HOP_HEADERS = {
"connection",
"content-encoding",
"content-length",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"te",
"trailer",
"transfer-encoding",
"upgrade",
}
ASR_METADATA_PATTERN = re.compile(
r"language\s+[^<\r\n]{1,80}\s*<asr_text>",
flags=re.IGNORECASE,
)
ASR_TEXT_TAG_PATTERN = re.compile(r"<asr_text>", flags=re.IGNORECASE)
def clean_asr_text(value: str) -> str:
# Qwen can emit a new language marker when the speaker switches language
# mid-utterance. Remove every metadata marker instead of only a leading one.
cleaned = ASR_METADATA_PATTERN.sub("", value)
return ASR_TEXT_TAG_PATTERN.sub("", cleaned).strip()
def clean_transcription_response(content: bytes, content_type: str) -> bytes:
if "application/json" in content_type:
try:
payload = json.loads(content)
except (json.JSONDecodeError, UnicodeDecodeError):
return content
if isinstance(payload, dict) and isinstance(payload.get("text"), str):
payload["text"] = clean_asr_text(payload["text"])
return json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode()
return content
if content_type.startswith("text/"):
try:
return clean_asr_text(content.decode()).encode()
except UnicodeDecodeError:
return content
return content
@asynccontextmanager
async def lifespan(app: FastAPI):
timeout = httpx.Timeout(connect=30, read=3600, write=3600, pool=30)
app.state.client = httpx.AsyncClient(timeout=timeout)
yield
await app.state.client.aclose()
app = FastAPI(
title="ASR Gateway",
docs_url=None,
redoc_url=None,
openapi_url=None,
lifespan=lifespan,
)
async def forward(request: Request, upstream_path: str, clean: bool = False) -> Response:
headers = {
key: value
for key, value in request.headers.items()
if key.lower() not in HOP_BY_HOP_HEADERS and key.lower() != "host"
}
async with app.state.client.stream(
request.method,
UPSTREAM + upstream_path,
params=request.query_params,
headers=headers,
content=request.stream(),
) as upstream:
content = await upstream.aread()
content_type = upstream.headers.get("content-type", "")
if clean and upstream.is_success:
content = clean_transcription_response(content, content_type)
response_headers = {
key: value
for key, value in upstream.headers.items()
if key.lower() not in HOP_BY_HOP_HEADERS
}
return Response(
content=content,
status_code=upstream.status_code,
headers=response_headers,
)
async def normalize_audio(upload: UploadFile, workdir: str) -> str:
"""Convert browser/vendor audio containers to deterministic PCM WAV."""
input_path = os.path.join(workdir, "input-audio")
output_path = os.path.join(workdir, "normalized.wav")
total = 0
with open(input_path, "wb") as destination:
while chunk := await upload.read(1024 * 1024):
total += len(chunk)
if total > MAX_AUDIO_BYTES:
raise HTTPException(status_code=413, detail="Audio file exceeds 32 MiB.")
destination.write(chunk)
if total == 0:
raise HTTPException(status_code=400, detail="Audio file is empty.")
process = await asyncio.create_subprocess_exec(
"ffmpeg",
"-nostdin",
"-hide_banner",
"-loglevel",
"error",
"-y",
"-i",
input_path,
"-vn",
"-ac",
"1",
"-ar",
"16000",
"-c:a",
"pcm_s16le",
output_path,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.PIPE,
)
try:
_, stderr = await asyncio.wait_for(
process.communicate(), timeout=FFMPEG_TIMEOUT_SECONDS
)
except TimeoutError as error:
process.kill()
await process.communicate()
raise HTTPException(status_code=504, detail="Audio conversion timed out.") from error
if process.returncode != 0 or not os.path.exists(output_path):
detail = stderr.decode(errors="replace").strip()[-500:]
raise HTTPException(
status_code=415,
detail="Unsupported or damaged audio file." + (" " + detail if detail else ""),
)
return output_path
async def forward_transcription(request: Request, parts: list[tuple]) -> Response:
headers = {
key: value
for key, value in request.headers.items()
if key.lower() not in HOP_BY_HOP_HEADERS
and key.lower() not in {"host", "content-type"}
}
upstream = await app.state.client.post(
UPSTREAM + "/v1/audio/transcriptions",
params=request.query_params,
headers=headers,
files=parts,
)
content_type = upstream.headers.get("content-type", "")
content = upstream.content
if upstream.is_success:
content = clean_transcription_response(content, content_type)
response_headers = {
key: value
for key, value in upstream.headers.items()
if key.lower() not in HOP_BY_HOP_HEADERS
}
return Response(
content=content,
status_code=upstream.status_code,
headers=response_headers,
)
@app.get("/")
async def service_info() -> JSONResponse:
return JSONResponse(
{
"service": "asr-gateway",
"contract": "openai-audio-transcriptions-v1",
"models": "/v1/models",
"transcriptions": "/v1/audio/transcriptions",
}
)
@app.get("/health")
async def health(request: Request) -> Response:
return await forward(request, "/health")
@app.get("/v1/models")
async def models(request: Request) -> Response:
return await forward(request, "/v1/models")
@app.post("/v1/audio/transcriptions")
async def transcriptions(request: Request) -> Response:
form = await request.form(max_files=4, max_fields=32, max_part_size=MAX_AUDIO_BYTES)
upload = form.get("file")
if not isinstance(upload, UploadFile):
raise HTTPException(status_code=400, detail="Multipart field 'file' is required.")
fields = [
(key, (None, str(value)))
for key, value in form.multi_items()
if key != "file" and not isinstance(value, UploadFile)
]
with tempfile.TemporaryDirectory(prefix="asr-gateway-") as workdir:
normalized_path = await normalize_audio(upload, workdir)
with open(normalized_path, "rb") as normalized:
parts = [("file", ("audio.wav", normalized, "audio/wav")), *fields]
return await forward_transcription(request, parts)