tio-boot 案例 - 流失响应

Introudction

使用 tio-boot 框架的 handler 组件从 openai chatgpt 获取流程响应并以流式的方法返回给客户端 使用了tio-boot框架，集成了与 OpenAI ChatGPT 进行交互的功能，并以服务器发送事件（Server-Sent Events, SSE）方式流式返回数据给客户端。

概念扩展

• Server-Sent Events (SSE)：一种允许服务器向浏览器客户端发送更新的技术。适用于需要实时数据更新的场景，如股票行情、新闻直播等。
• OpenAI API：这是利用人工智能模型生成文本的 API，可用于聊天机器人、内容生成等多种应用。

代码讲解

Maven 配置 (pom.xml)

这个 pom.xml 文件定义了项目的构建配置和依赖管理。主要部分包括：

• 项目信息：定义了项目的基本元数据，如 groupId, artifactId, 和 version。
• 属性：定义了常用的属性，如 Java 版本和项目依赖版本。这样可以在整个项目中重复使用这些属性，便于维护和更新。
• 依赖：列出了项目所需的库，例如日志框架 logback, JSON 处理库 fastjson2, HTTP 客户端库 okhttp3。
• 构建配置：分为开发和生产两种配置，使用了 Spring Boot 的 Maven 插件来简化打包和运行过程。

HTTP 请求处理配置 (HttpServerRequestHandlerConfig)

该类配置了 HTTP 请求的处理逻辑：

• 创建 SimpleHttpRoutes 对象并添加路由，将特定的 HTTP 请求映射到对应的处理器。

消息处理器 (OpenaiV1ChatHandler)

该类负责处理来自客户端的 HTTP 请求，并与 OpenAI ChatGPT API 交互：

• SSE Header：设置 HTTP 响应头为 SSE 格式，允许服务器推送实时数据到客户端。
• HTTP 请求处理：接收 HTTP 请求，解析并发送请求到 OpenAI 服务器，然后将响应以 SSE 形式发送给客户端。
• API 请求构建：构建发送到 OpenAI 的 HTTP 请求，包括设置请求头和请求体。
• 流处理：读取 OpenAI 响应，并将每一行数据以 SSE 格式实时发送给客户端。

代码内容

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.litongjava</groupId>
  <artifactId>open-chat-server</artifactId>
  <version>1.0.0</version>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <java.version>1.8</java.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>
    <graalvm.version>23.1.1</graalvm.version>
    <tio-boot.version>1.5.2</tio-boot.version>
    <lombok-version>1.18.30</lombok-version>
    <hotswap-classloader.version>1.2.3</hotswap-classloader.version>
    <final.name>open-chat-server</final.name>
    <main.class>com.litongjava.open.chat.OpenChatServer</main.class>
  </properties>
  <dependencies>
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.2.3</version>
    </dependency>

    <dependency>
      <groupId>com.squareup.okhttp3</groupId>
      <artifactId>okhttp</artifactId>
      <version>3.11.0</version>
    </dependency>

    <dependency>
      <groupId>com.litongjava</groupId>
      <artifactId>tio-boot</artifactId>
      <version>${tio-boot.version}</version>
    </dependency>

    <dependency>
      <groupId>com.litongjava</groupId>
      <artifactId>hotswap-classloader</artifactId>
      <version>${hotswap-classloader.version}</version>
    </dependency>

    <dependency>
      <groupId>com.litongjava</groupId>
      <artifactId>jfinal-aop</artifactId>
      <version>1.2.4</version>
    </dependency>

    <dependency>
      <groupId>com.alibaba.fastjson2</groupId>
      <artifactId>fastjson2</artifactId>
      <version>2.0.12</version>
    </dependency>

    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <version>${lombok-version}</version>
      <optional>true</optional>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>


  </dependencies>
  <profiles>
    <!-- development -->
    <profile>
      <id>development</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.7.4</version>
            <configuration>
              <fork>true</fork>
              <mainClass>${main.class}</mainClass>
              <excludeGroupIds>org.projectlombok</excludeGroupIds>
              <arguments>
                <argument>--mode=dev</argument>
              </arguments>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

    <!-- production -->
    <profile>
      <id>production</id>
      <build>
        <plugins>
          <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.7.4</version>
            <configuration>
              <mainClass>${main.class}</mainClass>
              <excludeGroupIds>org.projectlombok</excludeGroupIds>
            </configuration>
            <executions>
              <execution>
                <goals>
                  <goal>repackage</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
</project>

OpenChatServer

启动类

package com.litongjava.open.chat;

import com.litongjava.jfinal.aop.annotation.AComponentScan;
import com.litongjava.open.chat.config.OpenChatServerConfig;
import com.litongjava.tio.boot.TioApplication;

@AComponentScan
public class OpenChatServer {
  public static void main(String[] args) {
    long start = System.currentTimeMillis();
    TioApplication.run(OpenChatServer.class, new OpenChatServerConfig(), args);
    long end = System.currentTimeMillis();
    System.out.println((end - start) + "ms");
  }
}

配置类

package com.litongjava.open.chat.config;

import com.litongjava.tio.boot.context.TioBootConfiguration;

public class OpenChatServerConfig implements TioBootConfiguration {

  @Override
  public void config() {
    new ExecutorServiceConfig().config();
    new HttpRequestHanlderConfig().config();
  }
}

package com.litongjava.open.chat.config;

import java.util.concurrent.ExecutorService;

import com.litongjava.tio.boot.server.TioBootServer;
import com.litongjava.tio.utils.thread.ThreadUtils;

public class ExecutorServiceConfig {

  public void config() {
    // 创建包含10个线程的线程池
    ExecutorService executor = ThreadUtils.newFixedThreadPool(10);

    // 项目关闭时，关闭线程池
    TioBootServer.me().addDestroyMethod(() -> {
      if (executor != null && !executor.isShutdown()) {
        executor.shutdown();
      }
    });
  }
}

```java
package com.litongjava.open.chat.config;

import com.litongjava.open.chat.handler.OpenaiV1ChatHandler;
import com.litongjava.tio.boot.server.TioBootServer;
import com.litongjava.tio.http.server.router.HttpReqeustSimpleHandlerRoute;

public class HttpRequestHanlderConfig {

  public void config() {
    // 获取router
    HttpReqeustSimpleHandlerRoute r = TioBootServer.me().getHttpReqeustSimpleHandlerRoute();

    OpenaiV1ChatHandler openaiV1ChatHandler = new OpenaiV1ChatHandler();
    r.add("/v1/chat/completions", openaiV1ChatHandler::completions);
  }
}

常量类

package com.litongjava.open.chat.constants;

public interface OpenAiConstatns {
  String server_url = "https://api.openai.com";
  String gpt_4o_2024_05_13 = "gpt-4o-2024-05-13";
}

OkHttpClientPool

package com.litongjava.open.chat.instance;

import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.TimeUnit;

import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import okhttp3.ConnectionPool;
import okhttp3.OkHttpClient;

public enum OkHttpClientPool {
  INSTANCE;

  static okhttp3.OkHttpClient.Builder builder;
  static {
    builder = new OkHttpClient().newBuilder();
    // 连接池
    builder.connectionPool(pool());
    // 信任连接
    builder.sslSocketFactory(sslSocketFactory(), x509TrustManager());
    // 连接超时
    builder.connectTimeout(120L, TimeUnit.SECONDS).readTimeout(120L, TimeUnit.SECONDS).build();

  }

  public static OkHttpClient getHttpClient() {
    return builder.build();
  }

  private static ConnectionPool pool() {
    return new ConnectionPool(200, 5, TimeUnit.MINUTES);
  }

  public static X509TrustManager x509TrustManager() {
    return new X509TrustManager() {
      @Override
      public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {
      }

      @Override
      public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {
      }

      @Override
      public X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[0];
      }
    };
  }

  public static SSLSocketFactory sslSocketFactory() {
    try {
      // 信任任何链接
      SSLContext sslContext = SSLContext.getInstance("TLS");
      sslContext.init(null, new TrustManager[] { x509TrustManager() }, new SecureRandom());
      return sslContext.getSocketFactory();
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
    } catch (KeyManagementException e) {
      e.printStackTrace();
    }
    return null;
  }

}

OpenAiClient

OpenAiClient 负责发送请求

package com.litongjava.open.chat.client;

import java.io.IOException;
import java.util.Map;

import com.litongjava.open.chat.constants.OpenAiConstatns;
import com.litongjava.open.chat.instance.OkHttpClientPool;

import okhttp3.Callback;
import okhttp3.Headers;
import okhttp3.MediaType;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.RequestBody;
import okhttp3.Response;

public class OpenAiClient {

  public static Response completions(Map<String, String> requestHeaders, String bodyString) {

    OkHttpClient httpClient = OkHttpClientPool.getHttpClient();

    MediaType mediaType = MediaType.parse("application/json");

    RequestBody body = RequestBody.create(mediaType, bodyString);

    Headers headers = Headers.of(requestHeaders);

    Request request = new Request.Builder() //
        .url(OpenAiConstatns.server_url + "/v1/chat/completions") //
        .method("POST", body).headers(headers) //
        .build();
    try {
      return httpClient.newCall(request).execute();
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  public static void completions(Map<String, String> requestHeaders, String bodyString, Callback callback) {
    OkHttpClient httpClient = OkHttpClientPool.getHttpClient();

    MediaType mediaType = MediaType.parse("application/json");

    RequestBody body = RequestBody.create(mediaType, bodyString);

    Headers headers = Headers.of(requestHeaders);

    Request request = new Request.Builder() //
        .url(OpenAiConstatns.server_url + "/v1/chat/completions") //
        .method("POST", body).headers(headers) //
        .build();
    httpClient.newCall(request).enqueue(callback);
  }

}

OpenaiV1ChatHandler

package com.litongjava.open.chat.handler;


import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import com.alibaba.fastjson2.JSONArray;
import com.alibaba.fastjson2.JSONObject;
import com.jfinal.kit.Kv;
import com.litongjava.open.chat.client.OpenAiClient;
import com.litongjava.open.chat.constants.OpenAiConstatns;
import com.litongjava.open.chat.services.OpenaiV1ChatService;
import com.litongjava.tio.boot.http.TioControllerContext;
import com.litongjava.tio.core.ChannelContext;
import com.litongjava.tio.core.Tio;
import com.litongjava.tio.http.common.HeaderName;
import com.litongjava.tio.http.common.HeaderValue;
import com.litongjava.tio.http.common.HttpRequest;
import com.litongjava.tio.http.common.HttpResponse;
import com.litongjava.tio.http.common.encoder.ChunkEncoder;
import com.litongjava.tio.http.common.sse.SseBytesPacket;
import com.litongjava.tio.http.server.util.HttpServerResponseUtils;
import com.litongjava.tio.http.server.util.SseUtils;
import com.litongjava.tio.utils.environment.EnvUtils;
import com.litongjava.tio.utils.json.FastJson2Utils;
import com.litongjava.tio.utils.json.JsonUtils;
import com.litongjava.tio.utils.resp.RespVo;

import lombok.extern.slf4j.Slf4j;
import okhttp3.Call;
import okhttp3.Callback;
import okhttp3.Response;
import okhttp3.ResponseBody;

@Slf4j
public class OpenaiV1ChatHandler {

  private OpenaiV1ChatService openaiV1ChatService = new OpenaiV1ChatService();

  public HttpResponse completions(HttpRequest httpRequest) {
    long start = System.currentTimeMillis();
    HttpResponse httpResponse = TioControllerContext.getResponse();
    // HttpServerResponseUtils.enableCORS(httpResponse, new HttpCors());

    String requestURI = httpRequest.getRequestURI();

    Map<String, String> headers = httpRequest.getHeaders();
    String bodyString = httpRequest.getBodyString();
    log.info("requestURI:{},header:{},bodyString:{}", requestURI, headers, bodyString);

    // 替换基本的一些值
    String authorization = EnvUtils.get("OPENAI_API_KEY");
    headers.put("authorization", "Bearer " + authorization);
    headers.put("host", "api.openai.com");

    Boolean stream = true;
    JSONObject openAiRequestVo = null;
    if (bodyString != null) {
      openAiRequestVo = FastJson2Utils.parseObject(bodyString);
      stream = openAiRequestVo.getBoolean("stream");
      openAiRequestVo.put("model", OpenAiConstatns.gpt_4o_2024_05_13);
    }

    if (stream != null && stream) {
      if (openAiRequestVo != null) {
        // 告诉默认的处理器不要将消息体发送给客户端,因为后面会手动发送
        httpResponse.setSend(false);
        ChannelContext channelContext = httpRequest.getChannelContext();
        openAiRequestVo = openaiV1ChatService.beforeCompletions(openAiRequestVo);
        streamResponse(channelContext, httpResponse, headers, openAiRequestVo, start);
      } else {
        return httpResponse.setJson(RespVo.fail("empty body"));
      }
    } else {
      openAiRequestVo = openaiV1ChatService.beforeCompletions(openAiRequestVo);
      Response response = OpenAiClient.completions(headers, openAiRequestVo.toString());
      HttpServerResponseUtils.fromOkHttp(response, httpResponse);
      httpResponse.setHasGzipped(true);
      httpResponse.removeHeaders("Transfer-Encoding");
      httpResponse.removeHeaders("Server");
      httpResponse.removeHeaders("Date");
      httpResponse.setHeader("Connection", "close");
      httpResponse.removeHeaders("Set-Cookie");
      long end = System.currentTimeMillis();
      log.info("finish llm in {} (ms):", (end - start));
    }

    return httpResponse;
  }

  /**
   * 流式请求和响应
   *
   * @param channelContext
   * @param httpResponse
   * @param headers
   * @param start
   */
  public void streamResponse(ChannelContext channelContext, HttpResponse httpResponse, Map<String, String> headers,
      JSONObject requestBody, long start) {

    OpenAiClient.completions(headers, requestBody.toString(), new Callback() {

      @Override
      public void onFailure(Call call, IOException e) {
        e.printStackTrace();
        // 直接发送
        httpResponse.setSend(true);
        httpResponse.setJson(RespVo.fail(e.getMessage()));
        Tio.send(channelContext, httpResponse);

      }

      @Override
      public void onResponse(Call call, Response response) throws IOException {
        if (!response.isSuccessful()) {
          httpResponse.setSend(true);
          HttpServerResponseUtils.fromOkHttp(response, httpResponse);
          httpResponse.setHasGzipped(true);
          httpResponse.removeHeaders("Content-Length");
          // 响应
          Tio.send(channelContext, httpResponse);
          return;
        }
        // 设置sse请求头
        httpResponse.setServerSentEventsHeader();
        // 60秒后客户端关闭连接
        httpResponse.addHeader(HeaderName.Keep_Alive, HeaderValue.from("timeout=60"));
        httpResponse.addHeader(HeaderName.Transfer_Encoding, HeaderValue.from("chunked"));
        if (!httpResponse.isSend()) { // 不要让处理器发送,我来发送
          // 发送http 响应头,告诉客户端保持连接
          Tio.send(channelContext, httpResponse);
        }

        try (ResponseBody responseBody = response.body()) {
          if (responseBody == null) {
            String message = "response body is null";
            log.error(message);
            SseBytesPacket ssePacket = new SseBytesPacket(ChunkEncoder.encodeChunk(message.getBytes()));
            Tio.send(channelContext, ssePacket);
            closeSeeConnection(channelContext);
            return;
          }
          StringBuffer completionContent = new StringBuffer();
          StringBuffer fnCallName = new StringBuffer();
          StringBuffer fnCallArgs = new StringBuffer();

          StringBuffer toolFnCallId = new StringBuffer();
          StringBuffer toolFnCallName = new StringBuffer();
          StringBuffer toolFnCallArgs = new StringBuffer();

          String line;
          while ((line = responseBody.source().readUtf8Line()) != null) {
            // 必须添加一个回车符号
            byte[] bytes = (line + "\n\n").getBytes();
            if (line.length() < 1) {
              continue;
            }
            line = openaiV1ChatService.processLine(line);

            if (line.length() > 6) {
              int indexOf = line.indexOf(':');
              String data = line.substring(indexOf + 1, line.length());
              openaiV1ChatService.processData(data);
              if (data.endsWith("}")) {
                JSONObject parseObject = FastJson2Utils.parseObject(data);
                JSONArray choices = parseObject.getJSONArray("choices");
                if (choices.size() > 0) {
                  String content = choices.getJSONObject(0).getJSONObject("delta").getString("content");
                  // 只发送content信息
                  if (content != null) {
                    SseUtils.pushChunk(channelContext, bytes);
                  }
                  extraChoices(choices, completionContent, fnCallName, fnCallArgs, toolFnCallId, toolFnCallName,
                      toolFnCallArgs);
                }
              }
            }
          }
          openaiV1ChatService.completionContent(completionContent);

          if (fnCallName.length() > 0) {
            processFnCall(channelContext, httpResponse, headers, requestBody, start, fnCallName, fnCallArgs, "user",
                null);
          } else if (toolFnCallName.length() > 0) {
            processFnCall(channelContext, httpResponse, headers, requestBody, start, toolFnCallName, toolFnCallArgs,
                "tool", toolFnCallId.toString());

          } else {
            closeSeeConnection(channelContext);
          }
        }
      }
    });
  }

  public void processFnCall(ChannelContext channelContext, HttpResponse httpResponse, Map<String, String> headers,
      JSONObject requestBody, long start, StringBuffer fnCallName, StringBuffer fnCallArgs, String roleName,
      String fnCallId) {

    Kv functionCallResult = openaiV1ChatService.functionCall(channelContext, fnCallName, fnCallArgs);
    // 再次发送到大模型
    if (functionCallResult != null) {
      long newStart = System.currentTimeMillis();
      JSONArray messages = requestBody.getJSONArray("messages");

      Kv functionCall = Kv.by("name", fnCallName).set("arguments", fnCallArgs);

      // 查询结果
      Kv result = Kv.by("content", JsonUtils.toJson(functionCallResult));
      Kv lastMesage = Kv.by("role", "assistant");

      // assistantMessage.set("role", "s").set("content", null);

      if (fnCallId.length() > 0) {
        result.set("role", "tool").set("tool_call_id", fnCallId).set("name", fnCallName);

        List<Kv> toolCalls = new ArrayList<>(1);
        toolCalls.add(Kv.by("id", fnCallId).set("function", functionCall).set("type", "function"));
        lastMesage.set("tool_calls", toolCalls);
      } else {
        result.set("role", "system");
        lastMesage.set("function_call", functionCall);

      }
      messages.add(lastMesage);
      messages.add(result);
      // 防止重复发送响应头
      httpResponse.setSend(true);
      streamResponse(channelContext, httpResponse, headers, requestBody, newStart);

    } else {
      long end = System.currentTimeMillis();
      log.info("finish llm in {} (ms):", (end - start));
      closeSeeConnection(channelContext);
    }
  }

  /**
   * 发送一个大小为 0 的 chunk 以表示消息结束
   * @param channelContext
   */
  public void closeSeeConnection(ChannelContext channelContext) {
    // 关闭连接
    byte[] zeroChunk = ChunkEncoder.encodeChunk(new byte[0]);
    SseBytesPacket endPacket = new SseBytesPacket(zeroChunk);
    Tio.send(channelContext, endPacket);

    try {
      // 给客户端足够的时间接受消息
      Thread.sleep(1000);
      Tio.remove(channelContext, "remove");
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
  }

  @SuppressWarnings("unused")
  private void test(ChannelContext channelContext) {
    for (int i = 0; i < 100; i++) {
      // String line = "data:鲁";
      String line = "data:{\"id\":\"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC\",\"object\":\"chat.completion.chunk\",\"created\":1715759355,\"model\":\"gpt-3.5-turbo-0125\",\"system_fingerprint\":null,\"choices\":[{\"index\":0,\"delta\":{\"content\":\"鲁"
          + i + "\"},\"logprobs\":null,\"finish_reason\":null}]}";
      log.info("send:{}", line);

      byte[] bytes = (line + "\n\n").getBytes();

      // 将数据编码成chunked格式并返回,这样客户端的流式输出会更流程
      SseBytesPacket ssePacket = new SseBytesPacket(ChunkEncoder.encodeChunk(bytes));
      // 再次向客户端发送消息
      Tio.send(channelContext, ssePacket);
      try {
        Thread.sleep(100);
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
    }
  }

  public void extraChoices(JSONArray choices, StringBuffer complectionContent, StringBuffer fnCallName,
      StringBuffer fnCallArgs, StringBuffer tooFnCallId, StringBuffer tooFnCallName, StringBuffer tooFnCallArgs) {
    if (choices.size() > 0) {
      for (int i = 0; i < choices.size(); i++) {
        JSONObject delta = choices.getJSONObject(i).getJSONObject("delta");
        String part = delta.getString("content");
        if (part != null) {
          complectionContent.append(part);
        }
        String functionCallString = delta.getString("function_call");
        if (functionCallString != null) {
          JSONObject functionCall = FastJson2Utils.parseObject(functionCallString);
          String name = functionCall.getString("name");
          if (name != null) {
            fnCallName.append(name);
          }

          String arguments = functionCall.getString("arguments");
          if (arguments != null) {
            // System.out.println("arguments:" + arguments);
            fnCallArgs.append(arguments);
          }
        }
        String toolCallsString = delta.getString("tool_calls");
        if (toolCallsString != null) {
          // 不考虑执行多个的问题
          JSONArray parseArray = FastJson2Utils.parseArray(toolCallsString);
          JSONObject toolCall = parseArray.getJSONObject(0);
          String id = toolCall.getString("id");
          if (id != null) {
            tooFnCallId.append(id);
          }

          JSONObject funcation = toolCall.getJSONObject("function");
          String name = funcation.getString("name");
          if (name != null) {
            tooFnCallName.append(name);
          }
          String arguments = funcation.getString("arguments");
          if (arguments != null) {
            tooFnCallArgs.append(arguments);
          }
        }
      }
    }
  }
}

OpenaiV1ChatService

OpenaiV1ChatService 可以根据自己的定制开发

package com.litongjava.open.chat.services;

import com.alibaba.fastjson2.JSONObject;
import com.jfinal.kit.Kv;
import com.litongjava.tio.core.ChannelContext;

import lombok.extern.slf4j.Slf4j;

@Slf4j
public class OpenaiV1ChatService {

  public JSONObject beforeCompletions(JSONObject openAiRequestVo) {
    // JSONArray jsonArray = openAiRequestVo.getJSONArray("messages");
    return openAiRequestVo;
  }

  public String processLine(String line) {
    return line;
  }

  public void completionContent(StringBuffer completionContent) {
    log.info("completionContent:{}", completionContent);
  }

  /**
   *
   * @return {result}
   */
  public Kv functionCall(ChannelContext channelContext, StringBuffer fnCallName, StringBuffer fnCallArgs) {
    log.info("fn:{},{}", fnCallName.toString(), fnCallArgs.toString());
    return null;
  }

  public void processData(String data) {

  }
}

测试请求

curl --location --request POST 'http://127.0.0.1/v1/chat/completions' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "messages": [
        {
            "role": "system",
            "content": "hi"
        }
    ],
    "model": "gpt-3.5-turbo",
    "stream":true
}'

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"Hi"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" How"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" can"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" I"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" assist"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" today"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"?"},"logprobs":null,"finish_reason":null}]}

data:{"id":"chatcmpl-9P3fvvyk4IuCprCnvMytoKN8UtskC","object":"chat.completion.chunk","created":1715759355,"model":"gpt-3.5-turbo-0125","system_fingerprint":null,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}

data:[DONE]