LLMs (大型语言模型)

添加 LLMs 至我们的数据流

添加附加算子：

wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/keyboard_op.py
wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/microphone_op.py
wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/whisper_op.py
wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/sentence_transformers_op.py
wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/llm_op.py
wget https://raw.githubusercontent.com/dora-rs/dora/v0.3.6/examples/python-operator-dataflow/file_saver_op.py

并配置数据流：

nodes:
  - id: webcam
    operator:
      python: webcam.py
      inputs:
        tick: dora/timer/millis/50
      outputs:
        - image

  - id: object_detection
    operator:
      python: object_detection.py
      inputs:
        image: webcam/image
      outputs:
        - bbox

  - id: plot
    operator:
      python: plot.py
      inputs:
        image: webcam/image
        bbox: object_detection/bbox
        line: llm/line
        keyboard_buffer: keyboard/buffer
        user_message: keyboard/submitted
        assistant_message: llm/assistant_message

  ## 语音转文本
  - id: keyboard
    custom:
      source: keyboard_op.py
      outputs:
        - buffer
        - submitted
        - record
        - ask
        - send
        - change
      inputs:
        recording: whisper/text

  - id: microphone
    operator:
      python: microphone_op.py
      inputs:
        record: keyboard/record
      outputs:
        - audio

  - id: whisper
    operator:
      python: whisper_op.py
      inputs:
        audio: microphone/audio
      outputs:
        - text

  ## 代码修饰符
  - id: vectordb
    operator:
      python: sentence_transformers_op.py
      inputs:
        query: keyboard/change
        saved_file: file_saver/saved_file
      outputs:
        - raw_file

  - id: llm
    operator:
      python: llm_op.py
      inputs:
        code_modifier: vectordb/raw_file
        assistant: keyboard/ask
        message_sender: keyboard/send
      outputs:
        - modified_file
        - line
        - assistant_message

  - id: file_saver
    operator:
      python: file_saver_op.py
      inputs:
        file: llm/modified_file
      outputs:
        - saved_file

尝试一下：

dora up
dora start dataflow.yml --attach

录取 语音转文本

键盘将记录每个键输入。

如果键盘接收到提交关键字“record”，则将触发麦克风，并使用 OpenAI 耳语转录音频。

变更 源节点

代码变更流程的工作原理是首先提交带有关键字 change 的指令。 该指令将在 vectorDB 上传递，以检索最接近的节点源代码以进行更改。

源代码和指令馈送到要修改的 LLM，然后将其保存到源代码中。

稍后会触发要重新加载的节点的热重载序列。

最终结果应与指令相对应。

询问 助理一个问题

您可以提交关键字 ask 询问助理一个问题。

当问题被传递给LLM，将在窗口中回复。

发送 一个信息至数据流

您还可以使用关键字 send 直接向数据流发送消息，并指定您要的数据设置。

请注意，数据和主题都应在数据流和 LLM 节点中指定，以便发送数据。

开始

您可以尝试如下指令：

ask how are you
change bounding box plot to red
change confidence value to percentage
change object detection to only detect person
send 200 200 200 400 to topic line
record # Then speak