llamacpp-imagetransform

llama.cpp-based element that passes an image through a vision LLM and forwards the generated text.

It is possible to configure a system prompt, various sampling parameters and to keep a history of the last inputs/outputs for producing more consistent outputs.

The element can be used for example for image descriptions, which can then be used for text to speech, or for understanding the content of a video frame by frame.

Models

The model and the multimodal projection model can be selected via the model-path and mmproj-path properties. These expect local GGUF files, which can be downloaded e.g. from Hugging Face. The model-path should point to the GGUF file of a vision LLM and mmproj-path to the GGUF file of the corresponding multimodal projection model. Models that are known to work well are for example

Keep in mind that all these models have safeguards integrated, which can lead to rejections.

Examples

Image description

 gst-launch-1.0 filesrc location=photo.jpg ! image/jpeg ! llamacpp-imagetransform model-path=/path/to/Qwen3.5-9B.gguf mmproj-path=/path/to/mmproj-Qwen3.5-9B.gguf system-prompt="Describe the image in one sentence." ! fakesink

Hierarchy

GObject
    ╰──GInitiallyUnowned
        ╰──GstObject
            ╰──GstElement
                ╰──GstLlamaBaseTransform
                    ╰──llamacpp-imagetransform

Factory details

Authors: – Sebastian Dröge

Classification: – Text/LLM/Video

Rank – none

Plugin – llamacpp

Package – gst-plugin-llamacpp

Pad Templates

sink

image/jpeg:
image/png:
video/x-raw:
         format: RGB
          width: [ 1, 2147483647 ]
         height: [ 1, 2147483647 ]

Presence – always

Direction – sink

Object type – GstPad


src

text/x-raw:
         format: utf8

Presence – always

Direction – src

Object type – GstPad


Properties

mmproj-path

“mmproj-path” gchararray

Path to the GGUF multimodal projection model file

Flags : Read / Write

Default value : NULL


The results of the search are