[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"consumer-news-detail-930":3,"consumer-news-interaction-930":41,"consumer-news-related-930":44},{"detail":4,"item":36},{"card":5,"schemaVersion":23,"fields":24,"content":30},{"id":6,"kind":7,"targetType":8,"targetId":9,"subtype":7,"typeLabel":10,"title":11,"subtitle":12,"summary":13,"coverUrl":14,"badgeText":15,"href":16,"sourceName":12,"meta":17,"metrics":20,"tags":21,"resolved":22},"NEWS_ARTICLE:930","news","NEWS_ARTICLE",930,"资讯","基于 vLLM+Nginx 构建负载均衡推理集群","博客园","企业内部私有环境部署大模型推理集群时，很容易遇到流量调度混乱、节点负载失衡、会话上下文丢失、接口缺少鉴权防护等一系列问题，单 vLLM 推理节点难以支撑并发请求。本文基于 Ubuntu 22.04 系统环境，搭建 Nginx + vLLM-Semantic-Router + vLLM-Router","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F1379525\u002F202609\u002F1379525-20260910165534385-2022408098.png","","\u002Fnews\u002F930",[18,19],"2026","人工智能",{},[19],true,"consumer-content-detail-v1",{"sourceName":12,"authorName":25,"categoryName":19,"summary":13,"description":13,"publishTime":26,"updateTime":27,"sourceUrl":28,"language":29},"lyshark","2026-09-11T10:56","2026-09-11T15:21:59","https:\u002F\u002Fwww.cnblogs.com\u002FLyShark\u002Fp\u002F22904364","中文",{"format":31,"policy":32,"normalized":22,"html":33,"text":34,"wordCount":35,"hasBody":22},"HTML","NEWS_CONTENT_V1","\u003Cp>企业内部私有环境部署大模型推理集群时，很容易遇到流量调度混乱、节点负载失衡、会话上下文丢失、接口缺少鉴权防护等一系列问题，单 vLLM 推理节点难以支撑并发请求。本文基于 Ubuntu 22.04 系统环境，搭建 Nginx + vLLM-Semantic-Router + vLLM-Router + vLLM Node 多层推理集群。全文从环境准备、模型拉取、组件分步部署、Nginx SSE 流式反向代理，一直到接口联调验证，完整记录一套可直接复现的落地方案。\u003C\u002Fp>\n架构拓扑\n\u003Cp>本次部署采用 Nginx + vLLM-Semantic-Router + vLLM-Router + vLLM Node 推理节点的多层分层推理集群架构，实现请求鉴权、负载均衡、语义智能路由、算力负载分流的全流程能力。\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"https:\u002F\u002Fi1.wp.com\u002Fimg2024.cnblogs.com\u002Fblog\u002F1379525\u002F202609\u002F1379525-20260910165534385-2022408098.png?w=720&quality=65&strip=all\" alt=\"image\">\u003C\u002Fp>\n\u003Cp>架构层级从上至下如下：\u003C\u002Fp>\n\u003Cul>\n \u003Cli>Nginx：作为集群唯一对外入口，提供接口鉴权、请求反向代理、全局负载均衡、SSE 流式响应适配，保障集群安全与流量稳定分发。\u003C\u002Fli>\n \u003Cli>vLLM-Semantic-Router（语义网关）：识别用户请求意图，区分通用对话、代码编写等场景，实现请求的智能分类路由，适配不同业务场景模型。\u003C\u002Fli>\n \u003Cli>vLLM-Router（算力路由层）：监控后端推理节点显存、算力负载，基于一致性哈希策略实现流量均匀分配，支持会话亲和、KV缓存复用，承接上下层流量转发。\u003C\u002Fli>\n \u003Cli>vLLM Node 推理节点：部署大模型推理服务，提供核心的对话、代码生成推理能力，双节点部署实现算力冗余与负载分担。\u003C\u002Fli>\n\u003C\u002Ful>\n环境部署\n\u003Ch2>vLLM Node 推理节点部署（双节点）\u003C\u002Fh2>\n\u003Cp>集群底层算力载体，运行 vLLM 推理服务并加载大模型，负责执行对话、代码生成等实际推理任务。部署双实例，实现算力负载分担与服务冗余。\u003C\u002Fp>\n\u003Cp>部署2个独立推理节点，分别绑定5001、5002端口，同时部署5003端口语义识别模型节点，为上层语义路由提供意图识别能力。\u003C\u002Fp>\n\u003Cp>1、安装vLLM兼容依赖包，能正常输出版本号即成功。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# python3 -m venv ~\u002Fvllm\nroot@localhost:~# source ~\u002Fvllm\u002Fbin\u002Factivate\n\nroot@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple bitsandbytes tiktoken\nroot@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple openvino modelscope vllm\nroot@localhost:~# python -c \"import vllm; print(vllm.__version__)\"\n0.29.0\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>2、通过魔搭社区下载轻量化模型，适配本地部署场景，分别用于对话推理、语义意图识别。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# mkdir -p \u002Fdata\u002Fmodel\u002F\n\n# 用于对话的模型\nroot@localhost:~# modelscope download --model Qwen\u002FQwen2.5-0.5B-Instruct --local_dir \u002Fdata\u002Fmodel\u002Fqwen2.5\n\n# 用于vLLM-Semantic-Router语义检查模型\nroot@localhost:~# modelscope download --model gongjy\u002Fminimind-3 --local_dir \u002Fdata\u002Fmodel\u002Fminimind3\n\nroot@localhost:\u002Fdata\u002Fmodel# ls -lh\ntotal 8.0K\ndrwxr-xr-x 3 root root 4.0K Sep 10 17:25 minimind3\ndrwxr-xr-x 2 root root 4.0K Sep 10 17:23 qwen2.5\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>开启离线模式、配置模型基础参数，分别准备三台Ubuntu系统，其中每台服务器有1-5颗GPU显卡，启动三个节点服务。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# export HF_HUB_OFFLINE=1\n\n# 节点 1 端口 5001\nroot@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\\n  --host 127.0.0.1 \\\n  --port 5001 \\\n  --trust-remote-code \\\n  --dtype bfloat16 \\\n  --max-model-len 512 \\\n  --tensor-parallel-size 1\n\n# 节点 2 端口 5002\nroot@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\\n  --host 127.0.0.1 \\\n  --port 5002 \\\n  --trust-remote-code \\\n  --dtype bfloat16 \\\n  --max-model-len 512 \\\n  --tensor-parallel-size 1\n\n# 语义检查模型 端口5003\nroot@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\\n  --host 127.0.0.1 \\\n  --port 5003 \\\n  --trust-remote-code \\\n  --dtype bfloat16 \\\n  --max-model-len 512 \\\n  --tensor-parallel-size 1\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>vLLM-Router（算力路由层）\u003C\u002Fh2>\n\u003Cp>vLLM-Router为算力负载路由核心，负责监控后端推理节点负载、实现流量均匀分发，支持会话亲和与KV缓存复用，承接语义网关的请求流量。\u003C\u002Fp>\n\u003Cp>1、安装路由节点。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# python3 -m venv vLLM-Router\nroot@localhost:~# source vLLM-Router\u002Fbin\u002Factivate\nroot@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple\u002F vllm-router\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>2、采用一致性哈希策略，绑定对应推理节点，实现会话固定路由。\u003C\u002Fp>\n\u003Cpre>\u003Ccode># 路由节点1：对接5001通用对话节点 端口6001\nroot@localhost:~# vllm-router \\\n--host 127.0.0.1 \\\n--port 6001 \\\n--worker-urls http:\u002F\u002F127.0.0.1:5001 \\\n--policy consistent_hash\n\n# 路由节点2：对接5002代码推理节点 端口6002\nroot@localhost:~# vllm-router \\\n--host 127.0.0.1 \\\n--port 6002 \\\n--worker-urls http:\u002F\u002F127.0.0.1:5002 \\\n--policy consistent_hash\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>3、测试可用性，带上\u003Ccode>X-Session-ID\u003C\u002Fcode> 同一个 session-id 永远路由到同一个 vllm 后端，实现会话亲和、复用 KV 缓存。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# curl http:\u002F\u002F127.0.0.1:6001\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"X-Session-ID: session-001\" \\\n  -d '{\n    \"model\": \"qwen3\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"你好\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 128\n  }'\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>vLLM-Semantic-Router（语义网关）\u003C\u002Fh2>\n\u003Cp>上层业务路由，解析用户输入识别请求意图，区分普通闲聊、代码编写等场景，按语义规则将请求转发到对应的算力路由后端，实现按业务场景智能分流。\u003C\u002Fp>\n\u003Cp>此处的网关服务只能在Docker容器内，无法直接在物理机中运行，此处只提供一个正确的配置文件仅供参考。\u003C\u002Fp>\n\u003Cp>1、安装语义网关服务，并调用启动命令生成默认配置文件。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# python3 -m venv vsr\nroot@localhost:~# source vsr\u002Fbin\u002Factivate\nroot@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple\u002F vllm-sr\nroot@localhost:~# vllm-sr serve\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>2、修改生成的\u003Ccode>config.yaml\u003C\u002Fcode>（对接下层 vllm-router，地址\u003Ccode>127.0.0.1:8001\u003C\u002Fcode>）\u003C\u002Fp>\n\u003Cp>替换为以下完整配置，对接下层算力路由节点，定义语义路由规则：\u003C\u002Fp>\n\u003Cpre>\u003Ccode>version: v0.3\nlisteners:\n  - name: http-7001\n    address: 0.0.0.0\n    port: 7001\n    timeout: 300s\n\nproviders:\n  defaults:\n    default_model: qwen-0.5b\n  models:\n    # 语义识别模型 minimind（图中5003端口 minimind-3）\n    - name: minimind\n      provider_model_id: minimind\n      api_format: openai\n      backend_refs:\n        - name: minimind-backend\n          endpoint: http:\u002F\u002F127.0.0.1:5003\u002Fv1\n          protocol: http\n          weight: 100\n    # 第一个路由节点6001：普通聊天\n    - name: qwen-0.5b\n      provider_model_id: qwen-0.5b\n      api_format: openai\n      backend_refs:\n        - name: router-6001\n          endpoint: http:\u002F\u002F127.0.0.1:6001\u002Fv1\n          protocol: http\n          weight: 100\n    # 第二个路由节点6002：代码编写\n    - name: qwen-code\n      provider_model_id: qwen-code\n      api_format: openai\n      backend_refs:\n        - name: router-6002\n          endpoint: http:\u002F\u002F127.0.0.1:6002\u002Fv1\n          protocol: http\n          weight: 100\n\nrouting:\n  # 预先定义自定义信号 code_intent\n  signals:\n    embeddings:\n      - name: code_intent\n        threshold: 0.7\n        aggregation_method: max\n        candidates:\n          - \"写代码\"\n          - \"编写脚本\"\n          - \"python代码\"\n          - \"java代码\"\n          - \"js代码\"\n          - \"函数实现\"\n          - \"算法编写\"\n          - \"代码调试\"\n          - \"代码改错\"\n          - \"写程序\"\n          - \"代码实现\"\n          - \"写一段代码\"\n\n  modelCards:\n    - name: minimind\n      description: \"语义识别模型，用于意图判断\"\n      capabilities:\n        - semantic_routing\n    - name: qwen-0.5b\n      description: \"通用对话模型，普通聊天，路由6001\"\n      capabilities:\n        - chat\n    - name: qwen-code\n      description: \"代码专用模型，代码编写，路由6002\"\n      capabilities:\n        - coding\n\n  decisions:\n    # 代码编写意图 → 路由到6002\n    - name: route_code_writing\n      description: \"用户请求编写代码、脚本、程序，路由到代码专用router 6002\"\n      priority: 100\n      rules:\n        operator: AND\n        conditions:\n          - type: embedding\n            name: code_intent\n      modelRefs:\n        - model: qwen-code\n\n    # 兜底：普通聊天 → 路由到6001\n    - name: default_chat\n      description: \"默认普通聊天场景，路由到6001\"\n      priority: 10\n      rules:\n        operator: AND\n        conditions: []\n      modelRefs:\n        - model: qwen-0.5b\n\nglobal:\n  health_check:\n    interval_seconds: 10\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>3、前台检测文档可用性，并运行。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# vllm-sr validate --config \u002Froot\u002Fconfig.yaml\n2026-09-10 17:25:56,551 - INFO - ============================================================\n2026-09-10 17:25:56,551 - INFO - vLLM Semantic Router - Validate Configuration\n2026-09-10 17:25:56,551 - INFO - ============================================================\n2026-09-10 17:25:56,551 - INFO - Validating: \u002Froot\u002Fconfig.yaml\n2026-09-10 17:25:56,551 - INFO - \n2026-09-10 17:25:56,558 - INFO - Configuration parsed successfully\n2026-09-10 17:25:56,558 - INFO -   Version: v0.3\n2026-09-10 17:25:56,559 - INFO -   Listeners: 1\n2026-09-10 17:25:56,559 - INFO -   Decisions: 2\n2026-09-10 17:25:56,559 - INFO -   Models: 3\n2026-09-10 17:25:56,559 - INFO - Validating user configuration...\n2026-09-10 17:25:56,559 - INFO - Configuration validation passed\n2026-09-10 17:25:56,559 - INFO - ============================================================\n2026-09-10 17:25:56,559 - INFO - Configuration is valid!\n2026-09-10 17:25:56,559 - INFO - ============================================================\n2026-09-10 17:25:56,559 - INFO - \nConfiguration summary:\n2026-09-10 17:25:56,559 - INFO -   Version: v0.3\n2026-09-10 17:25:56,559 - INFO -   Listeners: 1\n2026-09-10 17:25:56,559 - INFO -   Embedding signals: 1\n2026-09-10 17:25:56,559 - INFO -   Decisions: 2\n2026-09-10 17:25:56,559 - INFO -   Models: 3\n2026-09-10 17:25:56,559 - INFO -   Default model: qwen-0.5b\n2026-09-10 17:25:56,559 - INFO - \n\nroot@localhost:~# vllm-sr serve --config \u002Froot\u002Fconfig.yaml\n2026-09-10 17:26:16,703 - INFO - Using config file: \u002Froot\u002Fconfig.yaml\n2026-09-10 17:26:16,713 - INFO - Created effective runtime config: \u002Froot\u002F.vllm-sr\u002Fruntime-config.yaml\n\n       █     █     █▄   ▄█\n ▄▄ ▄█ █     █     █ ▀▄▀ █\n  █▄█▀ █     █     █     █\n   ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀\n  Semantic Router\n  Intelligent Routing for Mixture-of-Models\n\n2026-09-10 17:26:16,721 - INFO - Starting vLLM Semantic Router\n2026-09-10 17:26:16,721 - INFO - Runtime stack: vllm-sr (port offset 0)\n2026-09-10 17:26:16,721 - INFO - Config file: \u002Froot\u002Fconfig.yaml\n2026-09-10 17:26:16,721 - INFO - Configured listeners:\n2026-09-10 17:26:16,721 - INFO -   - http-7001: 0.0.0.0:7001\n2026-09-10 17:26:16,721 - INFO - Runtime topology: split\n2026-09-10 17:26:16,721 - ERROR - Docker not found in PATH\n2026-09-10 17:26:16,721 - ERROR - Please install Docker Desktop or Docker Engine to use this tool\n2026-09-10 17:26:16,722 - ERROR - \n2026-09-10 17:26:16,722 - ERROR - Installation instructions:\n2026-09-10 17:26:16,722 - ERROR -   Docker: https:\u002F\u002Fdocs.docker.com\u002Fget-docker\u002F\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Nginx（反向代理及鉴权）\u003C\u002Fh2>\n\u003Cp>Nginx作为集群唯一对外入口，实现接口基础鉴权、SSE流式响应适配、双路由节点负载均衡、请求超时统一管控，替代语义网关直接承接外部流量，简化部署架构。\u003C\u002Fp>\n\u003Cblockquote>\n \u003Cp>Nginx 入口端口：11433，保留基础鉴权，去掉语义网关，Nginx 直接负载均衡分发流量到两台 router。\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>1、安装Mginx组件，并编辑配置文件启用反向代理功能。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# apt install -y nginx apache2-utils\nroot@localhost:~# vim \u002Fetc\u002Fnginx\u002Fnginx.conf\n\nuser www-data;\nworker_processes auto;\npid \u002Frun\u002Fnginx.pid;\ninclude \u002Fetc\u002Fnginx\u002Fmodules-enabled\u002F*.conf;\n\nevents {\n        worker_connections 768;\n}\n\nhttp {\n        sendfile on;\n        tcp_nopush on;\n        types_hash_max_size 2048;\n        include \u002Fetc\u002Fnginx\u002Fmime.types;\n        default_type application\u002Foctet-stream;\n\n        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;\n        ssl_prefer_server_ciphers on;\n\n        access_log \u002Fvar\u002Flog\u002Fnginx\u002Faccess.log;\n        error_log \u002Fvar\u002Flog\u002Fnginx\u002Ferror.log;\n\n        gzip on;\n\n        include \u002Fetc\u002Fnginx\u002Fconf.d\u002F*.conf;\n        include \u002Fetc\u002Fnginx\u002Fsites-enabled\u002F*;\n\n# 后端真实vllm-router地址\nupstream router_real_6001 {\n    server 127.0.0.1:29000;\n}\nupstream router_real_6002 {\n    server 127.0.0.1:29001;\n}\n\n# 内部端口6001 仅本机127.0.0.1可访问\nserver {\n    listen 127.0.0.1:6001;\n    server_name localhost;\n\n    location \u002F {\n        proxy_pass http:\u002F\u002Frouter_real_6001;\n\n        # SSE流式必备参数\n        proxy_http_version 1.1;\n        proxy_set_header Connection \"\";\n        proxy_buffering off;\n        proxy_cache off;\n\n        proxy_connect_timeout 300s;\n        proxy_send_timeout 300s;\n        proxy_read_timeout 300s;\n    }\n}\n\n# 内部端口6002 仅本机127.0.0.1可访问\nserver {\n    listen 127.0.0.1:6002;\n    server_name localhost;\n\n    location \u002F {\n        proxy_pass http:\u002F\u002Frouter_real_6002;\n\n        # SSE流式必备参数\n        proxy_http_version 1.1;\n        proxy_set_header Connection \"\";\n        proxy_buffering off;\n        proxy_cache off;\n\n        proxy_connect_timeout 300s;\n        proxy_send_timeout 300s;\n        proxy_read_timeout 300s;\n    }\n}\n\n# 对外唯一入口：11433，鉴权 + 负载均衡分发到内部6001\u002F6002\nupstream vllm_main_pool {\n    least_conn;\n    server 127.0.0.1:6001 max_fails=2 fail_timeout=15s;\n    server 127.0.0.1:6002 max_fails=2 fail_timeout=15s;\n    keepalive 16;\n}\n\nserver {\n    listen 11433;\n    server_name localhost;\n\n    auth_basic \"Restricted Access\";\n    auth_basic_user_file \u002Fetc\u002Fnginx\u002F.htpasswd;\n\n    location \u002F {\n        proxy_pass http:\u002F\u002Fvllm_main_pool;\n\n        # SSE流式必备\n        proxy_http_version 1.1;\n        proxy_set_header Connection \"\";\n        proxy_buffering off;\n        proxy_cache off;\n\n        # 透传客户端信息\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n\n        proxy_connect_timeout 300s;\n        proxy_send_timeout 300s;\n        proxy_read_timeout 300s;\n    }\n}\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>2、校验 Nginx 语法可用性。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# nginx -t\nnginx: the configuration file \u002Fetc\u002Fnginx\u002Fnginx.conf syntax is ok\nnginx: configuration file \u002Fetc\u002Fnginx\u002Fnginx.conf test is successful\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>3、创建鉴权账号。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# htpasswd -c \u002Fetc\u002Fnginx\u002F.htpasswd vllmuser\n\nroot@localhost:~# chown www-data:www-data \u002Fetc\u002Fnginx\u002F.htpasswd\nroot@localhost:~# chmod 644 \u002Fetc\u002Fnginx\u002F.htpasswd\n\nroot@localhost:~# nginx -t\nroot@localhost:~# nginx -s reload\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>4、测试访问示例，此处使用\u003Ccode>-u\u003C\u002Fcode>指定用户名及密码。\u003C\u002Fp>\n\u003Cpre>\u003Ccode>root@localhost:~# curl http:\u002F\u002F127.0.0.1:11433\u002Fv1\u002Fchat\u002Fcompletions \\\n-u vllmuser:1234 \\\n-H \"Content-Type: application\u002Fjson\" \\\n-d '{\n\"model\": \"qwen2.5\",\n\"messages\": [{\"role\":\"user\",\"content\":\"你好\"}],\n\"stream\": true\n}'\n\u003C\u002Fcode>\u003C\u002Fpre>","企业内部私有环境部署大模型推理集群时，很容易遇到流量调度混乱、节点负载失衡、会话上下文丢失、接口缺少鉴权防护等一系列问题，单 vLLM 推理节点难以支撑并发请求。本文基于 Ubuntu 22.04 系统环境，搭建 Nginx + vLLM-Semantic-Router + vLLM-Router + vLLM Node 多层推理集群。全文从环境准备、模型拉取、组件分步部署、Nginx SSE 流式反向代理，一直到接口联调验证，完整记录一套可直接复现的落地方案。 架构拓扑 本次部署采用 Nginx + vLLM-Semantic-Router + vLLM-Router + vLLM Node 推理节点的多层分层推理集群架构，实现请求鉴权、负载均衡、语义智能路由、算力负载分流的全流程能力。 架构层级从上至下如下： Nginx：作为集群唯一对外入口，提供接口鉴权、请求反向代理、全局负载均衡、SSE 流式响应适配，保障集群安全与流量稳定分发。 vLLM-Semantic-Router（语义网关）：识别用户请求意图，区分通用对话、代码编写等场景，实现请求的智能分类路由，适配不同业务场景模型。 vLLM-Router（算力路由层）：监控后端推理节点显存、算力负载，基于一致性哈希策略实现流量均匀分配，支持会话亲和、KV缓存复用，承接上下层流量转发。 vLLM Node 推理节点：部署大模型推理服务，提供核心的对话、代码生成推理能力，双节点部署实现算力冗余与负载分担。 环境部署 vLLM Node 推理节点部署（双节点） 集群底层算力载体，运行 vLLM 推理服务并加载大模型，负责执行对话、代码生成等实际推理任务。部署双实例，实现算力负载分担与服务冗余。 部署2个独立推理节点，分别绑定5001、5002端口，同时部署5003端口语义识别模型节点，为上层语义路由提供意图识别能力。 1、安装vLLM兼容依赖包，能正常输出版本号即成功。 root@localhost:~# python3 -m venv ~\u002Fvllm root@localhost:~# source ~\u002Fvllm\u002Fbin\u002Factivate root@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple bitsandbytes tiktoken root@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple openvino modelscope vllm root@localhost:~# python -c \"import vllm; print(vllm.__version__)\" 0.29.0 2、通过魔搭社区下载轻量化模型，适配本地部署场景，分别用于对话推理、语义意图识别。 root@localhost:~# mkdir -p \u002Fdata\u002Fmodel\u002F # 用于对话的模型 root@localhost:~# modelscope download --model Qwen\u002FQwen2.5-0.5B-Instruct --local_dir \u002Fdata\u002Fmodel\u002Fqwen2.5 # 用于vLLM-Semantic-Router语义检查模型 root@localhost:~# modelscope download --model gongjy\u002Fminimind-3 --local_dir \u002Fdata\u002Fmodel\u002Fminimind3 root@localhost:\u002Fdata\u002Fmodel# ls -lh total 8.0K drwxr-xr-x 3 root root 4.0K Sep 10 17:25 minimind3 drwxr-xr-x 2 root root 4.0K Sep 10 17:23 qwen2.5 开启离线模式、配置模型基础参数，分别准备三台Ubuntu系统，其中每台服务器有1-5颗GPU显卡，启动三个节点服务。 root@localhost:~# export HF_HUB_OFFLINE=1 # 节点 1 端口 5001 root@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\ --host 127.0.0.1 \\ --port 5001 \\ --trust-remote-code \\ --dtype bfloat16 \\ --max-model-len 512 \\ --tensor-parallel-size 1 # 节点 2 端口 5002 root@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\ --host 127.0.0.1 \\ --port 5002 \\ --trust-remote-code \\ --dtype bfloat16 \\ --max-model-len 512 \\ --tensor-parallel-size 1 # 语义检查模型 端口5003 root@localhost:~# vllm serve \u002Fdata\u002Fmodel\u002Fqwen2.5 \\ --host 127.0.0.1 \\ --port 5003 \\ --trust-remote-code \\ --dtype bfloat16 \\ --max-model-len 512 \\ --tensor-parallel-size 1 vLLM-Router（算力路由层） vLLM-Router为算力负载路由核心，负责监控后端推理节点负载、实现流量均匀分发，支持会话亲和与KV缓存复用，承接语义网关的请求流量。 1、安装路由节点。 root@localhost:~# python3 -m venv vLLM-Router root@localhost:~# source vLLM-Router\u002Fbin\u002Factivate root@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple\u002F vllm-router 2、采用一致性哈希策略，绑定对应推理节点，实现会话固定路由。 # 路由节点1：对接5001通用对话节点 端口6001 root@localhost:~# vllm-router \\ --host 127.0.0.1 \\ --port 6001 \\ --worker-urls http:\u002F\u002F127.0.0.1:5001 \\ --policy consistent_hash # 路由节点2：对接5002代码推理节点 端口6002 root@localhost:~# vllm-router \\ --host 127.0.0.1 \\ --port 6002 \\ --worker-urls http:\u002F\u002F127.0.0.1:5002 \\ --policy consistent_hash 3、测试可用性，带上X-Session-ID 同一个 session-id 永远路由到同一个 vllm 后端，实现会话亲和、复用 KV 缓存。 root@localhost:~# curl http:\u002F\u002F127.0.0.1:6001\u002Fv1\u002Fchat\u002Fcompletions \\ -H \"Content-Type: application\u002Fjson\" \\ -H \"X-Session-ID: session-001\" \\ -d '{ \"model\": \"qwen3\", \"messages\": [ {\"role\": \"user\", \"content\": \"你好\"} ], \"temperature\": 0.7, \"max_tokens\": 128 }' vLLM-Semantic-Router（语义网关） 上层业务路由，解析用户输入识别请求意图，区分普通闲聊、代码编写等场景，按语义规则将请求转发到对应的算力路由后端，实现按业务场景智能分流。 此处的网关服务只能在Docker容器内，无法直接在物理机中运行，此处只提供一个正确的配置文件仅供参考。 1、安装语义网关服务，并调用启动命令生成默认配置文件。 root@localhost:~# python3 -m venv vsr root@localhost:~# source vsr\u002Fbin\u002Factivate root@localhost:~# pip install -i https:\u002F\u002Fmirrors.cloud.tencent.com\u002Fpypi\u002Fsimple\u002F vllm-sr root@localhost:~# vllm-sr serve 2、修改生成的config.yaml（对接下层 vllm-router，地址127.0.0.1:8001） 替换为以下完整配置，对接下层算力路由节点，定义语义路由规则： version: v0.3 listeners: - name: http-7001 address: 0.0.0.0 port: 7001 timeout: 300s providers: defaults: default_model: qwen-0.5b models: # 语义识别模型 minimind（图中5003端口 minimind-3） - name: minimind provider_model_id: minimind api_format: openai backend_refs: - name: minimind-backend endpoint: http:\u002F\u002F127.0.0.1:5003\u002Fv1 protocol: http weight: 100 # 第一个路由节点6001：普通聊天 - name: qwen-0.5b provider_model_id: qwen-0.5b api_format: openai backend_refs: - name: router-6001 endpoint: http:\u002F\u002F127.0.0.1:6001\u002Fv1 protocol: http weight: 100 # 第二个路由节点6002：代码编写 - name: qwen-code provider_model_id: qwen-code api_format: openai backend_refs: - name: router-6002 endpoint: http:\u002F\u002F127.0.0.1:6002\u002Fv1 protocol: http weight: 100 routing: # 预先定义自定义信号 code_intent signals: embeddings: - name: code_intent threshold: 0.7 aggregation_method: max candidates: - \"写代码\" - \"编写脚本\" - \"python代码\" - \"java代码\" - \"js代码\" - \"函数实现\" - \"算法编写\" - \"代码调试\" - \"代码改错\" - \"写程序\" - \"代码实现\" - \"写一段代码\" modelCards: - name: minimind description: \"语义识别模型，用于意图判断\" capabilities: - semantic_routing - name: qwen-0.5b description: \"通用对话模型，普通聊天，路由6001\" capabilities: - chat - name: qwen-code description: \"代码专用模型，代码编写，路由6002\" capabilities: - coding decisions: # 代码编写意图 → 路由到6002 - name: route_code_writing description: \"用户请求编写代码、脚本、程序，路由到代码专用router 6002\" priority: 100 rules: operator: AND conditions: - type: embedding name: code_intent modelRefs: - model: qwen-code # 兜底：普通聊天 → 路由到6001 - name: default_chat description: \"默认普通聊天场景，路由到6001\" priority: 10 rules: operator: AND conditions: [] modelRefs: - model: qwen-0.5b global: health_check: interval_seconds: 10 3、前台检测文档可用性，并运行。 root@localhost:~# vllm-sr validate --config \u002Froot\u002Fconfig.yaml 2026-09-10 17:25:56,551 - INFO - ============================================================ 2026-09-10 17:25:56,551 - INFO - vLLM Semantic Router - Validate Configuration 2026-09-10 17:25:56,551 - INFO - ============================================================ 2026-09-10 17:25:56,551 - INFO - Validating: \u002Froot\u002Fconfig.yaml 2026-09-10 17:25:56,551 - INFO - 2026-09-10 17:25:56,558 - INFO - Configuration parsed successfully 2026-09-10 17:25:56,558 - INFO - Version: v0.3 2026-09-10 17:25:56,559 - INFO - Listeners: 1 2026-09-10 17:25:56,559 - INFO - Decisions: 2 2026-09-10 17:25:56,559 - INFO - Models: 3 2026-09-10 17:25:56,559 - INFO - Validating user configuration... 2026-09-10 17:25:56,559 - INFO - Configuration validation passed 2026-09-10 17:25:56,559 - INFO - ============================================================ 2026-09-10 17:25:56,559 - INFO - Configuration is valid! 2026-09-10 17:25:56,559 - INFO - ============================================================ 2026-09-10 17:25:56,559 - INFO - Configuration summary: 2026-09-10 17:25:56,559 - INFO - Version: v0.3 2026-09-10 17:25:56,559 - INFO - Listeners: 1 2026-09-10 17:25:56,559 - INFO - Embedding signals: 1 2026-09-10 17:25:56,559 - INFO - Decisions: 2 2026-09-10 17:25:56,559 - INFO - Models: 3 2026-09-10 17:25:56,559 - INFO - Default model: qwen-0.5b 2026-09-10 17:25:56,559 - INFO - root@localhost:~# vllm-sr serve --config \u002Froot\u002Fconfig.yaml 2026-09-10 17:26:16,703 - INFO - Using config file: \u002Froot\u002Fconfig.yaml 2026-09-10 17:26:16,713 - INFO - Created effective runtime config: \u002Froot\u002F.vllm-sr\u002Fruntime-config.yaml █ █ █▄ ▄█ ▄▄ ▄█ █ █ █ ▀▄▀ █ █▄█▀ █ █ █ █ ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ Semantic Router Intelligent Routing for Mixture-of-Models 2026-09-10 17:26:16,721 - INFO - Starting vLLM Semantic Router 2026-09-10 17:26:16,721 - INFO - Runtime stack: vllm-sr (port offset 0) 2026-09-10 17:26:16,721 - INFO - Config file: \u002Froot\u002Fconfig.yaml 2026-09-10 17:26:16,721 - INFO - Configured listeners: 2026-09-10 17:26:16,721 - INFO - - http-7001: 0.0.0.0:7001 2026-09-10 17:26:16,721 - INFO - Runtime topology: split 2026-09-10 17:26:16,721 - ERROR - Docker not found in PATH 2026-09-10 17:26:16,721 - ERROR - Please install Docker Desktop or Docker Engine to use this tool 2026-09-10 17:26:16,722 - ERROR - 2026-09-10 17:26:16,722 - ERROR - Installation instructions: 2026-09-10 17:26:16,722 - ERROR - Docker: https:\u002F\u002Fdocs.docker.com\u002Fget-docker\u002F Nginx（反向代理及鉴权） Nginx作为集群唯一对外入口，实现接口基础鉴权、SSE流式响应适配、双路由节点负载均衡、请求超时统一管控，替代语义网关直接承接外部流量，简化部署架构。 Nginx 入口端口：11433，保留基础鉴权，去掉语义网关，Nginx 直接负载均衡分发流量到两台 router。 1、安装Mginx组件，并编辑配置文件启用反向代理功能。 root@localhost:~# apt install -y nginx apache2-utils root@localhost:~# vim \u002Fetc\u002Fnginx\u002Fnginx.conf user www-data; worker_processes auto; pid \u002Frun\u002Fnginx.pid; include \u002Fetc\u002Fnginx\u002Fmodules-enabled\u002F*.conf; events { worker_connections 768; } http { sendfile on; tcp_nopush on; types_hash_max_size 2048; include \u002Fetc\u002Fnginx\u002Fmime.types; default_type application\u002Foctet-stream; ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; access_log \u002Fvar\u002Flog\u002Fnginx\u002Faccess.log; error_log \u002Fvar\u002Flog\u002Fnginx\u002Ferror.log; gzip on; include \u002Fetc\u002Fnginx\u002Fconf.d\u002F*.conf; include \u002Fetc\u002Fnginx\u002Fsites-enabled\u002F*; # 后端真实vllm-router地址 upstream router_real_6001 { server 127.0.0.1:29000; } upstream router_real_6002 { server 127.0.0.1:29001; } # 内部端口6001 仅本机127.0.0.1可访问 server { listen 127.0.0.1:6001; server_name localhost; location \u002F { proxy_pass http:\u002F\u002Frouter_real_6001; # SSE流式必备参数 proxy_http_version 1.1; proxy_set_header Connection \"\"; proxy_buffering off; proxy_cache off; proxy_connect_timeout 300s; proxy_send_timeout 300s; proxy_read_timeout 300s; } } # 内部端口6002 仅本机127.0.0.1可访问 server { listen 127.0.0.1:6002; server_name localhost; location \u002F { proxy_pass http:\u002F\u002Frouter_real_6002; # SSE流式必备参数 proxy_http_version 1.1; proxy_set_header Connection \"\"; proxy_buffering off; proxy_cache off; proxy_connect_timeout 300s; proxy_send_timeout 300s; proxy_read_timeout 300s; } } # 对外唯一入口：11433，鉴权 + 负载均衡分发到内部6001\u002F6002 upstream vllm_main_pool { least_conn; server 127.0.0.1:6001 max_fails=2 fail_timeout=15s; server 127.0.0.1:6002 max_fails=2 fail_timeout=15s; keepalive 16; } server { listen 11433; server_name localhost; auth_basic \"Restricted Access\"; auth_basic_user_file \u002Fetc\u002Fnginx\u002F.htpasswd; location \u002F { proxy_pass http:\u002F\u002Fvllm_main_pool; # SSE流式必备 proxy_http_version 1.1; proxy_set_header Connection \"\"; proxy_buffering off; proxy_cache off; # 透传客户端信息 proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 300s; proxy_send_timeout 300s; proxy_read_timeout 300s; } } } 2、校验 Nginx 语法可用性。 root@localhost:~# nginx -t nginx: the configuration file \u002Fetc\u002Fnginx\u002Fnginx.conf syntax is ok nginx: configuration file \u002Fetc\u002Fnginx\u002Fnginx.conf test is successful 3、创建鉴权账号。 root@localhost:~# htpasswd -c \u002Fetc\u002Fnginx\u002F.htpasswd vllmuser root@localhost:~# chown www-data:www-data \u002Fetc\u002Fnginx\u002F.htpasswd root@localhost:~# chmod 644 \u002Fetc\u002Fnginx\u002F.htpasswd root@localhost:~# nginx -t root@localhost:~# nginx -s reload 4、测试访问示例，此处使用-u指定用户名及密码。 root@localhost:~# curl http:\u002F\u002F127.0.0.1:11433\u002Fv1\u002Fchat\u002Fcompletions \\ -u vllmuser:1234 \\ -H \"Content-Type: application\u002Fjson\" \\ -d '{ \"model\": \"qwen2.5\", \"messages\": [{\"role\":\"user\",\"content\":\"你好\"}], \"stream\": true }'",9643,{"id":6,"kind":7,"title":11,"summary":13,"image":14,"href":16,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":40},"2026 · 人工智能","#2563eb","16 \u002F 10",[19],{"targetType":8,"targetId":9,"likedByMe":42,"likeCount":43,"commentCount":43,"contentLikeCount":43,"contentCommentCount":43,"sourceLikeCount":43,"sourceCommentCount":43},false,0,[45,52,59,66,73,79,86,93],{"id":46,"kind":7,"title":47,"summary":48,"image":49,"href":50,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":51},"NEWS_ARTICLE:932","2026年AI编程工具大全，33个主流工具一次看懂","事情是这样的，前两天看到一张图，是某个社区官网的「支持的工具」清单，我数了数，整整31个AI编程工具。 两年前这份清单撑死5个，现在直接31个，而且我居然每一个都认识。。。 干脆整理成一篇，顺手把最近字节的TraeWork和豆包工作也补了进来，凑成33个。 今天给大家推荐一遍，每个工具说说它是干什么","https:\u002F\u002Fimage.kjdaohang.com\u002Fimg\u002F20260909210838518.png","\u002Fnews\u002F932",[19],{"id":53,"kind":7,"title":54,"summary":55,"image":56,"href":57,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":58},"NEWS_ARTICLE:935","[Agent Memory \u002F 强化学习] MemPO源码学习笔记 ---（1）--- 总体","[Agent Memory \u002F 强化学习] MemPO源码学习笔记 （1） 总体 目录[Agent Memory \u002F 强化学习] MemPO源码学习笔记 （1） 总体0x00 概要0x01 基础 & 背景1.1 用RL训练记忆系统的要点1.2 主要难点1.3 主要思路1.4 RL训练方案1.","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F1850883\u002F202609\u002F1850883-20260906190854637-986949885.jpg","\u002Fnews\u002F935",[19],{"id":60,"kind":7,"title":61,"summary":62,"image":63,"href":64,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":65},"NEWS_ARTICLE:940","LLama-Factory 实现大模型LoRA-SFT微调指南","LLaMA Factory 是一款开源、低代码、一站式大语言与多模态模型微调框架，用于降低大模型的微调落地门槛。框架兼容 Qwen、LLaMA、ChatGLM、LLaVA 等上百款主流开源模型，支持增量预训练、SFT 监督微调、DPO、KTO、ORPO 等多种训练对齐方案，原生集成 LoRA、QLo","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F1379525\u002F202609\u002F1379525-20260909140111199-832484524.png","\u002Fnews\u002F940",[19],{"id":67,"kind":7,"title":68,"summary":69,"image":70,"href":71,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":72},"NEWS_ARTICLE:950","每天白嫖 WorkBuddy 100 积分，我让WorkBuddy自己领","有没有小伙伴跟我一样，每天都去白嫖 WorkBuddy 的 100 积分，每天都怕忘记领。 其实我早就开了会员，但还是会有积分焦虑，天天惦记着今天的积分领没领。后来我发现 WorkBuddy 可以自己领积分，今天把这个技能分享给小伙伴们。 Buddy 加油站每天签到领 100 积分，连续签满 7 天","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F2381533\u002F202609\u002F2381533-20260910080820041-1607850292.png","\u002Fnews\u002F950",[19],{"id":74,"kind":7,"title":75,"summary":76,"image":15,"href":77,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":78},"NEWS_ARTICLE:949","AI知识库，是捷径吗？","从信息化到数字化，再到当下的智能化，技术进步带来的效率提升，如果往好处想应该是：节省更多的时间和成本，用来做更多的事情。但从现实的角度看，节流比开源来得容易。","\u002Fnews\u002F949",[19],{"id":80,"kind":7,"title":81,"summary":82,"image":83,"href":84,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":85},"NEWS_ARTICLE:977","【OpenClaw具身硬件】ZeroClaw 源码阅读笔记（4）--- 代码执行","【OpenClaw具身硬件】ZeroClaw 源码阅读笔记（4） 代码执行 目录【OpenClaw具身硬件】ZeroClaw 源码阅读笔记（4） 代码执行0x00 概要0x01 代码合成1.1 核心思想1.2 业务逻辑具体应用场景能力体现1.3 实现细节生成种类Rust 代码Arduino CLI的","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F1850883\u002F202608\u002F1850883-20260823212850242-1723558087.png","\u002Fnews\u002F977",[19],{"id":87,"kind":7,"title":88,"summary":89,"image":90,"href":91,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":92},"NEWS_ARTICLE:985","机器学习项目：客户分群——K-Means 聚类从选参到业务画像的完整实战","商场用户分群： 一、前言 在商场运营中，面对成千上万的顾客，如果用同一套营销策略对待所有人，效果往往事倍功半。高收入的中年人和月光族的年轻人等等群体，消费习惯和偏好截然不同——这就需要用户分群（Customer Segmentation）：根据用户的年龄、收入、消费行为等特征，将相似的用户归为一类，","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F3775135\u002F202609\u002F3775135-20260908154332888-1132576367.png","\u002Fnews\u002F985",[19],{"id":94,"kind":7,"title":95,"summary":96,"image":97,"href":98,"meta":37,"badge":10,"author":12,"stats":-1,"accent":38,"coverRatio":39,"tags":99},"NEWS_ARTICLE:986","千问大模型完整RLHF全参数微调指南","大模型微调是实现模型领域定制的核心方案，本文承接《千问大模型二次 LoRA‑SFT 指令微调指南》部分内容，聚焦 Qwen3.5‑Base 纯文本基座全参数微调，完整复现 ChatGPT 风格 RLHF 对齐工程链路，覆盖数据预处理、SFT 监督微调、RM 奖励模型训练、PPO 强化学习、DPO 直","https:\u002F\u002Fimg2024.cnblogs.com\u002Fblog\u002F1379525\u002F202609\u002F1379525-20260908150528296-1541712777.png","\u002Fnews\u002F986",[19]]