环境配置

pip install vllm

有条件的话安装一下flash-attn

pip install flash-attn

模型部署

首先我们需要下载需要的模型，如果不下载的话，默认的模型会从huggingface的模型库中下载。这里我们本地模型的地址是/data/nlp/models/llama3_7b_instruct。那么只需要执行以下代码。

1、部署原始模型

CUDA_VISIBLE_DEVICES=0,1,2,3 nohup python -m vllm.entrypoints.openai.api_server --model /home/admin/workspace/data/model/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --port 4500 --tensor-parallel-size 4 --limit-mm-per-prompt image=5 --gpu-memory-utilization 0.90 --served-model-name qwen2_5_7b_instruct --dtype=half > vllm_test.out &

参数说明

调用方法
命令行使用curl调用

curl http://localhost:4500/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "qwen2_5_7b_instruct",
 "messages": [
 {"role": "system", "content": "You are a helpful assistant."},
 {"role": "user", "content": "Who won the world series in 2020?"}
 ]
 }'

使用python API调用

from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:4500/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="qwen2_5_7b_instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a joke."},
    ]
)
print("Chat response:", chat_response)

2、部署lora微调模型

nohup vllm serve meta-llama/Meta-Llama-3-8B --host 0.0.0.0 --port 4500 --enable-lora --lora-modules oasst={oasst_lora_path} xlam={xlam_lora_path} &

参数说明

调用方法

 from openai import OpenAI
 
 model_id = "meta-llama/Meta-Llama-3-8B"
 # Modify OpenAI's API key and API base to use vLLM's API server.
 openai_api_key = "EMPTY"
 openai_api_base = "http://localhost:4500/v1"
 client = OpenAI(
     api_key=openai_api_key,
     base_url=openai_api_base,
 )
 prompts = [
     "### Human: Check if the numbers 8 and 1233 are powers of two.### Assistant:",
     "### Human: What is the division result of 75 divided by 1555?### Assistant:",
 ]
 completion = client.completions.create(model="oasst",
                                       prompt=prompts, temperature=0.7, top_p=0.9, max_tokens=500)
 print("Completion result:", completion)
 
 prompts = [
     "<user>Check if the numbers 8 and 1233 are powers of two.</user>\n\n<tools>",
     "<user>What is the division result of 75 divided by 1555?</user>\n\n<tools>",
 ]
 completion = client.completions.create(model="xlam",
                                       prompt=prompts, temperature=0.0, max_tokens=500)
 print("Completion result:", completion)

关于vllm的参数详细介绍，可以这个参数解读。