GstAnalyticsSegmentationMtd

This type of metadata holds information on which pixels belongs to a region of the image representing a type of object.

It supports two types of segmentation, semantic or instance:

• Semantic: All objects of the same type have the same id
• Instance: Each instance of an object has a different id

The results of the segmentation are stored in a GstBuffer that has a GstVideoMeta associated with it. This buffer is stored in the GstAnalyticsSegmentationMtd using gst_analytics_relation_meta_add_segmentation_mtd(). The GstBuffer containing the segmentation mask is image-like but the color values are arbitrary values, referred by region-id in this API, without meaning beyond specifying that two pixels in the original image with the same values in their corresponding mask value belong to the same region.

To further describe a region, the GstAnalyticsSegmentationMtd can be associated with other GstAnalyticsMtd. Since region ids are generated by the segmentation process itself and are not always sequential, we use a map of indexes to region ids starting with 0 without discontinuity which facilitate N-to-N mapping with other GstAnalyticsMtd. For example it can be associated with GstAnalyticsClsMtd to describe the class of object matching the pixels of a segmented region.

Example: Associate Instance Segmentation with Classification

In the following example the segmentation process will fill segmask with values of 0 for background, 12 for the first region which correspond to a to a strawberry, 7 for the second region that also correspond to a strawberry in the image and 31 for the third region that correspond to a leaf in the image. region_ids is fill during segmentation post-processing

region_ids:

region-index	region-id
0	0
1	12
2	7
3	31

region_count = 4

  GstAnalyticsSegmentationMtd segmtd;
  GstAnalyticsClassificationMtd clsmtd;
  GstBuffer *segmask, *img;
  guint *region_ids;
  gsize region_count, class_count;
  gfloat *class_confidence;
  GQuark *classes;

  ... (segmentation filling segmask based on img)

  gst_analytics_relation_meta_add_segmentation_mtd (rmeta, segmask,
    GST_SEGMENTATION_TYPE_INSTANCE, region_count, region_ids, &segmtd);
  class_count = region_count;

  ... (class-index must match and correspond to region-index)
  classes [0]  = g_quark_from_string ("background");
  classes [1]  = g_quark_from_string ("strawberry");
  classes [2] = g_quark_from_string ("strawberry");
  classes [3] = g_quark_from_string ("leaf");

  ... (set confidence level for each class associated with a region
  ... where -1.0 mean undefined.)
  class_confidence [0] = -1.0;
  class_confidence [1] = 0.6;
  class_confidence [2] = 0.9;
  class_confidence [3] = 0.8;

  gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
    class_confidence, classes, &clsmtd);

  gst_analytics_relation_meta_set_relation (rmeta,
   GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);

Example: Associate Semantic Segmentation with Classification Assuming the same context as for Instance Segmentation above but instead a semantic segmentation is performed, therefore region-id-12 and region-id-7 are now represented by the same region-id-12

region_ids:

region-index	region-id
0	0
1	12
2	31

Code remain the same except that we set all confidence level to undefined (-1.0).

  ... (class-index must match and correspond to region-index)
  classes [0]  = g_quark_from_string ("background");
  classes [1]  = g_quark_from_string ("strawberry");
  classes [2] = g_quark_from_string ("leaf");

  ... (set confidence level for each class associated with a region
  ... where -1.0 mean undefined.)
  class_confidence [0] = -1.0;
  class_confidence [1] = -1.0;
  class_confidence [2] = -1.0;

  gst_analytics_relation_meta_add_cls_mtd (rmeta, class_count,
    class_confidence, classes, &clsmtd);

  gst_analytics_relation_meta_set_relation (rmeta,
   GST_ANALYTICS_REL_TYPE_RELATE_TO, segmtd.id, clsmtd.id);

Example: Retrieving class associated with a segmentation region-id-12 This the typical case for an overlay as we visit the segmentation mask we we find region-id values

gsize idx;
gst_analytics_segmentation_mtd_get_region_index (&segmtd, &idx, 12);
gst_analytics_relation_meta_get_direct_related (rmeta, segmtd.id,
  GST_ANALYTICS_REL_TYPE_RELATE_TO, gst_analytics_cls_mtd_get_type (),
  NULL, &clsmtd);

 GQuark region_class = gst_analytics_cls_mtd_get_quark (&segmtd, idx)
 ...

GstBuffer *
gst_analytics_segmentation_mtd_get_mask (const GstAnalyticsSegmentationMtd * handle,
                                         gint * masks_loc_x,
                                         gint * masks_loc_y,
                                         guint * masks_loc_w,
                                         guint * masks_loc_h)

function GstAnalytics.SegmentationMtd.prototype.get_mask(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mask'
}

def GstAnalytics.SegmentationMtd.get_mask (self):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_mask'

gsize
gst_analytics_segmentation_mtd_get_region_count (const GstAnalyticsSegmentationMtd * handle)

function GstAnalytics.SegmentationMtd.prototype.get_region_count(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_count'
}

def GstAnalytics.SegmentationMtd.get_region_count (self):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_count'

guint
gst_analytics_segmentation_mtd_get_region_id (const GstAnalyticsSegmentationMtd * handle,
                                              gsize index)

function GstAnalytics.SegmentationMtd.prototype.get_region_id(index: Number): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_id'
}

def GstAnalytics.SegmentationMtd.get_region_id (self, index):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_id'

gboolean
gst_analytics_segmentation_mtd_get_region_index (const GstAnalyticsSegmentationMtd * handle,
                                                 gsize * index,
                                                 guint id)

function GstAnalytics.SegmentationMtd.prototype.get_region_index(id: Number): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_region_index'
}

def GstAnalytics.SegmentationMtd.get_region_index (self, id):
    #python wrapper for 'gst_analytics_segmentation_mtd_get_region_index'

GstAnalyticsMtdType
gst_analytics_segmentation_mtd_get_mtd_type ()

function GstAnalytics.SegmentationMtd.prototype.get_mtd_type(): {
    // javascript wrapper for 'gst_analytics_segmentation_mtd_get_mtd_type'
}

def GstAnalytics.SegmentationMtd.get_mtd_type ():
    #python wrapper for 'gst_analytics_segmentation_mtd_get_mtd_type'