Package com.aispect.core.client.adapter.llamacpp

Class LlamaCppAdapter

java.lang.Object
com.aispect.core.client.adapter.llamacpp.LlamaCppAdapter
All Implemented Interfaces:
AiClient
public class LlamaCppAdapter extends Object implements AiClient
adaptation Llama.cpp compatible API client implementation (based on OkHttp）。

Structurally with OpenAiAdapter Highly similar, but specifically local Llama.cpp deployment (such as /v1/chat/completions） Customized.

  • Constructor Details

    • LlamaCppAdapter

      public LlamaCppAdapter()

  • Method Details

    • generate

      public Response generate(PromptContext ctx, AiClientConfig config) throws AiSpectException
      Description copied from interface: AiClient
      Generates a single response in a blocking manner.

      The client will report to the underlying AI The model sends the prompt context and waits for the complete calculation result. Suitable for non-streaming scenarios where the final result needs to be obtained in one go.

      Specified by:
      generate in interface AiClient
      Parameters:
      ctx - Prompt context of the current request, including system prompts, historical messages, etc.
      config - Configuration parameters requested by the client, such as model name, temperature, maximum Token Count and so on
      Returns:
      Full response generated by the model Response
      Throws:
      AiSpectException - Thrown when the network request fails, times out, or the model service returns an exception status code

    • streamGenerate

      public Stream<Response> streamGenerate(PromptContext ctx, AiClientConfig config) throws AiSpectException
      Description copied from interface: AiClient
      via streaming (Server-Sent Events mode) generates a response.

      The client will continue to receive partial results generated by the model in the form of a stream. The stream will be terminated when the model has been generated or the connection is interrupted. Ideal for chat applications or long text generation scenarios that require a real-time "typewriter" output experience.

      Specified by:
      streamGenerate in interface AiClient
      Parameters:
      ctx - Prompt context of the current request, including system prompts, historical messages, etc.
      config - Configuration parameters requested by the client (make sure to declare stream=true）
      Returns:
      Include Response Sequential flow of data blocks, automatically closed when receiving the terminator
      Throws:
      AiSpectException - Thrown when the streaming request initialization fails or a connection error occurs