classifiertensordec

Tensor decoder element for image classification models using a common output format: a vector of per-class confidence levels.

The tensor id selects how the values are interpreted: classification-generic-out holds raw scores that are softmaxed here, classification-generic-softmaxed-out holds probabilities that are used as they are. uint8 values are scaled to [0, 1] first.

The most confident class is attached to the buffer as classification metadata. Without labels-file the classes are labelled by their index.

Test image file, model file and labels file can be found here : https://gitlab.collabora.com/gstreamer/onnx-models

 gst-launch-1.0 filesrc location=onnx-models/images/bus.jpg \
  ! jpegdec ! videoconvertscale ! video/x-raw,width=224,height=224 \
  ! onnxinference execution-provider=cpu model-file=onnx-models/models/mobilenetv2-10.onnx \
  ! classifiertensordec labels-file=onnx-models/labels/synset.txt \
  ! fakesink

This takes a JPEG, classifies it via onnxinference and decodes the inferred tensor with classifiertensordec.

Tensor format must be:

  • Dims: [class_count] or [batch-size, class_count]
  • Datatype: float32 or uint8
  Tensor [M,N]
  |Batch 0   | Class 0 confidence level | ... | Class N-1 confidence level |
  |               ...                   |
  |Batch M-1 | Class 0 confidence level | ... | Class N-1 confidence level |

  In-memory tensor format:

  |Batch 0, Class 0 confidence level     |
  |Batch 0,           ...                |
  |Batch 0, Class N-1 confidence level   |
  |               ...                    |
  |Batch M-1, Class 0 confidence level   |
  |Batch M-1,           ...              |
  |Batch M-1, Class N-1 confidence level |

Hierarchy

GObject
    ╰──GInitiallyUnowned
        ╰──GstObject
            ╰──GstElement
                ╰──GstBaseTransform
                    ╰──classifiertensordec

Factory details

Authors: – Jeremy Whiting

Classification: – Tensordecoder

Rank – primary

Plugin – rsanalytics

Package – gst-plugin-analytics

Pad Templates

sink

video/x-raw(ANY):
        tensors: "tensorgroups\,\ classification-generic-out\=\(/uniquelist\)\{\ \(caps\)\"tensor/strided\\\,\\\ tensor-id\\\=\\\(string\\\)classification-generic-out\\\,\\\ dims\\\=\\\(int\\\)\\\<\\\ \\\[\\\ 0\\\,\\\ 1\\\ \\\]\\\,\\\ \\\[\\\ 1\\\,\\\ 2147483647\\\ \\\]\\\ \\\>\\\,\\\ dims-order\\\=\\\(string\\\)row-major\\\,\\\ type\\\=\\\(string\\\)\\\{\\\ float32\\\,\\\ uint8\\\ \\\}\\\;\\\ tensor/strided\\\,\\\ tensor-id\\\=\\\(string\\\)classification-generic-out\\\,\\\ dims\\\=\\\(int\\\)\\\<\\\ \\\[\\\ 1\\\,\\\ 2147483647\\\ \\\]\\\ \\\>\\\,\\\ dims-order\\\=\\\(string\\\)row-major\\\,\\\ type\\\=\\\(string\\\)\\\{\\\ float32\\\,\\\ uint8\\\ \\\}\"\ \}\;"

video/x-raw(ANY):
        tensors: "tensorgroups\,\ classification-generic-softmaxed-out\=\(/uniquelist\)\{\ \(caps\)\"tensor/strided\\\,\\\ tensor-id\\\=\\\(string\\\)classification-generic-softmaxed-out\\\,\\\ dims\\\=\\\(int\\\)\\\<\\\ \\\[\\\ 0\\\,\\\ 1\\\ \\\]\\\,\\\ \\\[\\\ 1\\\,\\\ 2147483647\\\ \\\]\\\ \\\>\\\,\\\ dims-order\\\=\\\(string\\\)row-major\\\,\\\ type\\\=\\\(string\\\)\\\{\\\ float32\\\,\\\ uint8\\\ \\\}\\\;\\\ tensor/strided\\\,\\\ tensor-id\\\=\\\(string\\\)classification-generic-softmaxed-out\\\,\\\ dims\\\=\\\(int\\\)\\\<\\\ \\\[\\\ 1\\\,\\\ 2147483647\\\ \\\]\\\ \\\>\\\,\\\ dims-order\\\=\\\(string\\\)row-major\\\,\\\ type\\\=\\\(string\\\)\\\{\\\ float32\\\,\\\ uint8\\\ \\\}\"\ \}\;"

Presence – always

Direction – sink

Object type – GstPad


src

ANY

Presence – always

Direction – src

Object type – GstPad


Properties

class-confidence-threshold

“class-confidence-threshold” gfloat

Classes with a confidence level inferior to this threshold will be excluded

Flags : Read / Write

Default value : 0


labels-file

“labels-file” gchararray

Path to a file containing class label following the COCO format

Flags : Read / Write

Default value : NULL


The results of the search are