VLLM 服务参数

https://docs.vllm.ai/en/latest/configuration/serve_args.html#cli-arguments

# config.yml

host: 0.0.0.0
port: 8088
api-key: empty
model: ./Qwen3-32B/
enable-lora: true
#lora-modules: '{"name": "test", "path": "Qwen3-32B", "base_model_name": "qwen3-32b"}'
max-model-len: 8192
served-model-name: Qwen3-32B
gpu-memory-utilization: 0.9

启动命令

CUDA_VISIBLE_DEVICES=0 vllm serve --config config.yml

或者
CUDA_VISIBLE_DEVICES=0 vllm serve ./Qwen3-32B/ --host 0.0.0.0 --port 8088 --api-key empty --served-model-name qwen3-32b --enable-auto-tool-choice --tool-call-parser hermes --enable-prefix-caching --dtype bfloat16 --max-model-len 8192
# --tensor-parallel-size 1

curl调用,body部分

{
	"model": "qwen3-32b",
	"temperature": 0.7,
	"top_p": 0.8,
	"top_k": 20,
	"presence_penalty": 1.5,
	"max_tokens": 1024,
	"stream": false,
	"chat_template_kwargs": {
		"enable_thinking": false
	},
	"messages": [
		{
			"content": "智能小助手",
			"role": "system"
		},
		{
			"content": "你是谁",
			"role": "user"
		}
	]
}

客户端调用

from openai import OpenAI

client = OpenAI(
        api_key="empty",
        base_url="https://xxxx:8088/v1",
    )

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "你是谁"}
]

response= client.chat.completions.create(
    model="qwen3-32b",
    messages=messages,
    max_tokens=1024,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20, 
        "chat_template_kwargs": {"enable_thinking": False},
    },
    stream=False,
)
print(response.choices[0].message.content)


## if stream = True
#  for chunk in response:
#      if chunk.choices[0].delta.content is not None:
#          print(chunk.choices[0].delta.content, end="", flush=True)