========
Overview
========

The LCEVC Decoder SDK provides a **C** API to a library that takes decoded video frames from a separate base decoder, along with encoded enhancement data, and produces enhanced video frames according to the LCEVC specification. It is the responsibility of the integration to manage the decoding of the base frames, and extracting the enhancement data from the overall video stream.

:ref:`Decoder<overview:Decoder>` objects are allocated, configured, initialized and release using the Decoder SDK API.

:ref:`Pictures<overview:Pictures>` are then allocated from a decoder to manage getting image data in and out of the decoders.

:ref:`PictureLocks<overview:PictureLocks>` are short-lived allocated object associated with a picture and decoder that enable reading or writing of the memory underlying the Picture objects.

:ref:`Buffers<overview:Buffers>` describe regions of memory external to the Decoder API that can be used by :ref:`Pictures<overview:Pictures>` - typically this used to avoid memory copies to and from some client image buffer mechanism.

:ref:`Handles<overview:Handles>` identify all the objects allocated by the Decoder API. They are opaque pointer sized numbers that uniquely identify the objects to the API - they cannot be meaningfully dereferenced or decoded.

:ref:`Events<overview:Events>` can be generated by a decoder to indicate the progress of :ref:`Pictures<overview:Pictures>` through the decoder. A single callback function is registered with a decoder, and a set of events of interest is provided during decoder configuration.

The decoder object's state progresses as the integration calls the ``LCEVC_Send...()`` and ``LCEVC_Receive...()`` functions. If a particular function call cannot advance the decoder, it will return the :cpp:enumerator:`LCEVC_Again` return code, indicating that one or more of the other ``LCEVC_Send...()`` or ``LCEVC_Receive...()`` functions will need to be called first before the original action can happen.

Decoder
-------

Each decoder object manages the decoding of one **LCEVC** enhanced video stream. The new decoder is created using :cpp:func:`LCEVC_CreateDecoder` which outputs an opaque :cpp:type:`LCEVC_DecoderHandle` value.
The decoder is created in a *Configuration* state, optionally configured, then initialized for use. After use, decoders are destroyed using :cpp:func:`LCEVC_DestroyDecoder`.

Creation and Configuration
..........................

A new decoder is configured with various options that will persist for its lifetime. Configuration is via *name*/*value*, or *name*/*value array* using the ``LCEVC_ConfigureDecoder...()`` functions:

* :cpp:func:`LCEVC_ConfigureDecoderBool` and :cpp:func:`LCEVC_ConfigureDecoderBoolArray`
* :cpp:func:`LCEVC_ConfigureDecoderInt` and :cpp:func:`LCEVC_ConfigureDecoderIntArray`
* :cpp:func:`LCEVC_ConfigureDecoderFloat` and :cpp:func:`LCEVC_ConfigureDecoderFloatArray`
* :cpp:func:`LCEVC_ConfigureDecoderString` and :cpp:func:`LCEVC_ConfigureDecoderStringArray`

These are used to describe:

* Maximum resource use.
* :ref:`Event<overview:Events>` subscription.
* Implementation specific configuration.
* Implementation specific debugging.
* Log levels.

See :ref:`Supported Configuration Options<api:Configuration Options>` for a full list of options and types.

When a configuration option is not recognized by the decoder implementation, the call will return cpp:enumerator:`LCEVC_NotSupported`, and the decoder state will not be changed. The integration can choose to ignore this or raise an error as appropriate.

Once configured, the decoder is moved to the *Initialized* state by calling :cpp:func:`LCEVC_InitializeDecoder`, after which it can no longer be configured. These is no way of moving a decoder back into the *Configuration* state, it can only be used, or destroyed.

.. literalinclude:: doc_sample.cpp
  :start-after: Creation Start
  :end-before: Creation End
  :name: sample_creation
  :Caption: Decoder Creation

.. uml::
  :align: center

  skinparam monochrome true

  autoactivate on
  Integration -> Global: **LCEVC_CreateDecoder**()
  return **LCEVC_Success** and //decoderHandle//
  |||
  loop Configuration
   Integration -> Decoder: **LCEVC_ConfigureDecoder**...(//decoderHandle//, ...)
   return **LCEVC_Success** or **LCEVC_NotSupported**
  end
  |||
  Integration -> Decoder: **LCEVC_InitilizeDecoder**(//decoderHandle//)
  return **LCEVC_Success**

Enhancement Decoding
....................

Once a decoder has been configured, it can be used to decode an **LCEVC** enhanced video stream. The integration and decoder communicate using a *Send*/*Receive* mechanism. A *Send* call transfers ownership of resources into the decoder, and a *Receive* call takes ownership of resources from the the decoder.

The integration sends:

* Base picture from base decoder.
* Allocated pictures to receive output.
* Arrays of encoded enhancement data.

The integration receives:

* Base pictures that are no longer needed by decoder.
* Enhanced pictures with output images.

It is likely that the integration will produce enhancement data in decoder order whist parsing the bitstream, whilst the integrations' base decoder will produce pictures in presentation order. Furthermore, the integrations' base decoder may impose several frames of latency due to implementation details or the structure of the base bitstream. The **LCEVC** decoder manages this by having the base decoded pictures and the enhancement data  associate using a common :ref:`timestamp<overview:Timestamps>` sent alongside the resources. This timestamp will then associate with the enhanced output as part of the :cpp:struct:`LCEVC_DecodeInformation` that is received with the output picture.

Progress with :cpp:enumerator:`LCEVC_Again`
...........................................

When the integration calls a *Send* function, and the **LCEVC** decoder does not have the capacity to receive the supplied resource (either due a hard limit, or because the are already resources buffered internally up to a configured maximum latency), the function will return :cpp:enumerator:`LCEVC_Again`.

When the integration calls a *Receive* function, and the **LCEVC** decoder does not have any resources to return to the integration (usually due to not having associated resources sent), the function will also return :cpp:enumerator:`LCEVC_Again`.

The following code extract advances the overall decode process by attempting to move each stage of the process (from the end towards the front):

* Receive enhanced picture and pass on to rest of integration.
* Receive base pictures that the decoder has finished using.
* Send empty enhancement pictures ready for filling by decoder.
* Send enhancement data
* Send base picture
* Advance base decoder

The base decoder's output is kept in ``struct Base`` - a vector of enhancement bytes and a picture handle, along with the associated timestamps. The ``updateBase()`` function will fill it if they are empty, and the above steps will empty it as it successfully sends data to the enhancement decoder.

Two pools of pictures are maintained:

* Base pictures - consumed by the ``updateBase()`` function, and the filled when base pictures are received from the decoder.
* Enhancement pictures - consumed by sending empty pictures to the decoder, filled by enhanced pictures being received from the decoder.

.. literalinclude:: doc_sample.cpp
  :start-after: Base Start
  :end-before: Base End
  :name: sample_decoding
  :Caption: Decoding

.. literalinclude:: doc_sample.cpp
  :start-after: Decoding Start
  :end-before: Decoding End

.. uml::
  :align: center

  skinparam monochrome true

  autoactivate on
  loop Decoding
    loop Output Done
      Integration -> Decoder: **LCEVC_ReceiveDecoderPicture(decoderHandle, ...)**
      return **LCEVC_Success** or **LCEVC_Again**
    end
    |||
    loop Base Done
      Integration -> Decoder: **LCEVC_ReceiveDecoderBase(decoderHandle, ...)**
      return **LCEVC_Success** or **LCEVC_Again**
    end
    |||
    loop Output Ready
      Integration -> Decoder: **LCEVC_SendDecoderPicture(decoderHandle, ...)**
      return **LCEVC_Success** or **LCEVC_Again**
    end
    |||
    Integration -> Decoder: **LCEVC_SendDecoderEnhancementData(decoderHandle, ...)**
    return **LCEVC_Success** or **LCEVC_Again**
    |||
    Integration -> Decoder: **LCEVC_SendDecoderBase(decoderHandle, ...)**
    return **LCEVC_Success** or **LCEVC_Again**
    |||
    Integration -> BaseDecoder: Update Base Data
    return Updated base and enhancement
  end

Destruction
...........

Once decoding is finished - the LCEVC enhancement decoder is released using ``LCEVC_DestroyDecoder()``. All outstanding pictures that have been allocated using the decoder, and not freed will be released.

If an :cpp:enumerator`LCEVC_Exit` event has been registered, then any event callback will be triggered.
The event callback will be made *before* the ``LCEVC_DestroyDecoder()`` function returns.

.. literalinclude:: doc_sample.cpp
  :start-after: Destruction Start
  :end-before: Destruction End
  :name: sample_destruction
  :Caption: Destruction

.. uml::
  :align: center

  skinparam monochrome true

  autoactivate on

  Integration -> Decoder: **LCEVC_DestroyDecoder(decoderHandle)**
  return **void**

Pictures
--------

Video images are moved in and out of the LCEVC enhancement decoder using ``Picture`` objects. These are allocated from a configured decoder, and either have their backing memory managed internally, or externally via an integration supplied buffer.

The picture format is described by the :cpp:struct:`LCEVC_PictureDesc` structure. The can hold *YUV*, *RGB*, and greyscale formats with between 8 to 16 bits per component.

The underlying picture data is read and written using ``PictureLock`` objects. THis can then be queried for pointers and strides per plane. The lock should only be held whist the data is being accessed- a picture with any outstanding locks cannot be sent to the decoder.

If decode performance is not an issue, then the pictures can be allocated and freed during the decode process:

.. code-block:: C++
  :name: sample_code_picture_alloc

  // Get a picture
  LCEVC_PictureHandle basePicture:
  LCEVC_AllocatePicture(decoderHandle, &basePicture);

  // Fill it with base data
  LCEVC_PictureHandle pictureLock:
  LCEVC_LockPicture(decoderHandle, basePicture, LCEVC_Access_Write, &pictureLock);
  LCEVC_PicturePlaneDesc planeDesc = {};
  for(uint32_t plane=0; plane <3; ++plane) {
    LCEVC_GetPictureLockPlaneDesc(decoderHandle, pictureLock, plane, &planeDesc);
    copyYUVData(basePlanes[plane], planeDesc);
  }
  LCEVC_UnlockPicture(decoderHandle, pictureLock);

  // Send to decoder
  if(LCEVC_SendDecoderBase(decoderHandle, basePicture, ... ) == LCEVC_Success) {
    ...
  }

  ...

  // Get completed base from  decoder
  LCEVC_PictureHandle completedBasePicture = {0};
  if(LCEVC_ReceiveDecoderBase(decoderHandle, &completedBasePicture) == LCEVC_Success) {
    LCEVC_FreePicture(decoderHandle, completedBasePicture);

  }

If performance is an issue, then pictures can be recycled in pools, as shown in the decoding :ref:`sample<sample_decoding>` above. Typically, there would be two pools, one for base pictures, and one for enhanced pictures. This avoids any reallocation or reformatting happening whist the LCEVC stream's global configuration remains unchanged. Whist the decoder will be able to make progress with a single image in each pool, there will be implementation specific recommendations for how many images should be in each pool.

Picture Description
...................

The external details of a picture are carried by the :cpp:struct:`LCEVC_PictureDesc` structure. This covers the:

* Dimensions :cpp:member:`~LCEVC_PictureDesc::width` and :cpp:member:`~LCEVC_PictureDesc::height`.
* Components and sampling (e.g. *YUV*, *RGB* or *Grayscale*) : :cpp:member:`~LCEVC_PictureDesc::colorFormat`.
* Components and sampling: :cpp:member:`~LCEVC_PictureDesc::colorFormat`.
* Sample aspect ratio:  :cpp:member:`~LCEVC_PictureDesc::sampleAspectRatioNum` and :cpp:member:`~LCEVC_PictureDesc::sampleAspectRatioDen`.
* Color range (e.g. *Full* or *Limited*): :cpp:member:`~LCEVC_PictureDesc::colorRange`.
* Color primaries: (e.g. *BT601_NTSC*, *BT709* or *BT2020*) :cpp:member:`~LCEVC_PictureDesc::colorPrimaries`.
* Transfer characteristics (e.g. *PQ*, *LINEAR*, or *HLG*) :cpp:member:`~LCEVC_PictureDesc::transferCharacteristics`.
* Static High Dynamic Range metadata:  :cpp:member:`~LCEVC_PictureDesc::hdrStaticInfo`.
* Picture cropping :cpp:member:`~LCEVC_PictureDesc::cropTop`, :cpp:member:`~LCEVC_PictureDesc::cropBottom`, :cpp:member:`~LCEVC_PictureDesc::cropLeft` and :cpp:member:`~LCEVC_PictureDesc::cropRight`.

For convenience, the function :cpp:func:`LCEVC_DefaultPictureDesc` can be used to fill in this structure with defaults for all fields other than :cpp:member:`~LCEVC_PictureDesc::colorFormat`, :cpp:member:`~LCEVC_PictureDesc::width` and :cpp:member:`~LCEVC_PictureDesc::height`.

The completed :cpp:struct:`LCEVC_PictureDesc` structure is passed in as an argument to the :cpp:func:`LCEVC_AllocPicture` function to set the initial layout of a picture. The decoder will take a copy of the description, and no longer reference it one the allocation returns.

A picture can have it's description adjusted, by using the :cpp:func:`LCEVC_SetPictureDesc` function. This will perform any necessary reallocations to update the picture object. There are two ways in which the performance implications of this might be avoided:

* If the decoder has been configured with **max_width**, **max_height** and  **max_bits** limit parameters, and the new picture description falls within those limits.
* If the picture was allocated with an underlying external buffer, and the size of that buffer is large enough to hold the new picture description.

For the pictures coming from the base decoder, the integration layer will set up picture description derived from the base decoder's configuration and frames.

For the pictures coming from enhancement decoder, the integration will allocate them with small default values, then send them to the decoder. The decoder will then set the new description according to the configuration of the LCEVC enhancement data stream. It picture pools are being used - the enhanced pictures will cycle around with an unchanged layout, until the next configuration change embedded in the LCEVC stream.

The integration can then query the description of the enhanced ``Picture`` objects using the :cpp:func:`LCEVC_GetPictureDesc` function.

Internal
........

All pictures allocated using the :cpp:func:`LCEVC_AllocPicture` function have their storage managed internally by the decoder - the layout can be changed using :cpp:func:`LCEVC_SetPictureDesc` - which will destroy any existing data. The memory can only gen accessed via the ``PictureLock`` objects - which can be queried for a :cpp:struct:`LCEVC_PicturePlaneDesc` structure that contains points and strides for a given plane's memory, see :ref:`Plane Description<overview:Plane Description>`



External
........

The pictures allocated using the :cpp:func:`LCEVC_AllocPictureExternal` function have their storage managed externally to the decoder. A linear block of memory is described using the :cpp:struct:`LCEVC_PictureBufferDesc` structure (see :ref:`Buffers<overview:Buffers>`).

This size of this memory buffer should be greater than or equal to that required to store the picture data described by the initial allocation, and any later calls to :cpp:func:`LCEVC_SetPictureDesc`. As well as any integration calls, The decoder will, in effect, make a call to  :cpp:func:`LCEVC_SetPictureDesc` when reformatting the enhanced output buffer according to the LCEVC stream configuration.

The external buffer memory can be accessed using the ``PictureLock`` mechanism - the decoder will calculate pointer and strides according to the pictures description.

Flags
.....

The base pictures being sent into the LCEVC decoder may be marked with per frame metadata :cpp:enum:`flags <LCEVC_PictureFlag>`. This is used to convey metadata from the base decoder into the LCEVC decoder. Currently, two flags are used:

* :cpp:enumerator:`LCEVC_PictureFlag_IDR` Signal that this base frame is an "Instantaneous Decoder Refresh" frame. The LCEVC decoded may need this to synchronize temporal decoding.
* :cpp:enumerator:`LCEVC_PictureFlag_Interlaced` Signal that this base frame is an interlaced - containing two fields. The LCEVC decoder may treat each field as separate output pictures.


The per picture flags are set using :cpp:func:`LCEVC_SetPictureFlag`:

.. code-block:: C++

  LCEVC_SetPictureFlag(decoderHandle, basePicture, LCEVC_PictureFlag_IDR, baseFrame->isIDR);
  LCEVC_SetPictureFlag(decoderHandle, basePicture, LCEVC_PictureFlag_IDR, baseFrame->isInterlaced);

PictureLocks
------------

The memory underlying a decoder's picture is accessed via the ``PictureLock`` objects. This access is valid only for the lifetime of the ``PictureLock`` objects. Pictures with allocated locks cannot be sent to the decoder - the expectation is that the lock will only exist whilst data is being copied in or out of the memory.

When the ``PictureLock`` object is created, the intended :cpp:enum:`access <LCEVC_Access>` is specified. This is one of:

* :cpp:enumerator:`LCEVC_Access_Read` - Data will only be read from the plane (e.g. reading the enhanced output picture from the decoder).
* :cpp:enumerator:`LCEVC_Access_Write` - Data will only be written to the plane - any previous data will be lost. (e.g. writing the base input picture to the decoder).
* :cpp:enumerator:`LCEVC_Access_Modify` - Data will be read *and* written. The planes previous data will be available. (e.g. an in-place image processing operation on a picture)

This locking mechanism and access gives the decoder implementation the option to defer and recycle allocations, and when using hardware acceleration, map and possibly reformat regions of memory for access.

Plane Description
.................

Once a picture is locked - the description of how each plane is laid out in memory is fetched using :cpp:func:`LCEVC_GetPictureLockPlaneDesc` into a :cpp:struct:`LCEVC_PicturePlaneDesc`:

.. code-block:: C++

  struct LCEVC_PicturePlaneDesc
  {
    uint8_t* firstSample;
    uint32_t sampleByteStride;
    uint32_t rowByteStride;
    uint32_t sampleByteSize;
  };

The :cpp:member:`~LCEVC_PicturePlaneDesc::firstSample` member will point to the top left sample from the underlying plane of samples. The address of a particular *row*,*column* sample can be calculated:

.. code-block:: C++

  uint8_t *computeSamplePointer(const LCEVC_PicturePlaneDesc *desc, uint32_t row, uint32_t column)
  {
    return = desc->firstSample + desc->rowByteStride * row + desc->sampleByteStride * column;
  }

Whilst it is possible to write generic image copy routines completely based on the above code, in practice, optimisations base on the relative strides and sizes are possible:

.. code-block:: C++

  void copyPictureData(const LCEVC_PicturePlaneDesc *desc, uint32_t rows, uint32_t columns, const uint8_t *srcData)
  {
    if(desc->sampleByteStride == desc->sampleByteSize) {
      // Contiguous rows
      const uint32_t sz = columns desc->sampleByteSize;;
      const uint8_t *src = srcData;
      uint8_t *dest = firstSample;
      for(uint32_r = 0; r < rows; ++r) {
        memcpy(src, dest, sz);
        src += sz;
        dest += desc->rowByteStride;
      }

      return;
    }

    // Unoptimized copy
    ...
  }

.. uml::
  :align: center

  skinparam monochrome true

  autoactivate on
  Integration -> Decoder: **LCEVC_LockPicture(decoderHandle, ...)**
  return **LCEVC_PictureLockHandle**
  |||
  loop For each plane
    Integration -> Decoder: **LCEVC_SendDecoderEnhancementData(decoderHandle, pictureLockHandle, ...)**
    return **LCEVC_Success** and **LCEVC_PicturePlaneDesc**

    Integration -> Integration: **memcpy**
    return
  |||
  end
  |||
  Integration -> Decoder: **LCEVC_UnlockPicture(decoderHandle, pictureLockHandle)**
  return **LCEVC_Success**

Timestamps
----------

The LCEVC enhancement decoder is designed to handle receiving it's input data out of order relative to the displayed picture order implied by the overall video stream. The two sorts of data input (enhancement and base picture) need to sent along with a ``uint64_t`` timestamp. The decoder will internally associate the two elements, and send the resulting enhanced output picture along with that timestamp, in a :cpp:struct:`LCEVC_DecodeInformation` structure a supplied by the integration.

Typically, this timestamp would be transcribed straight from overall stream container, or possibly derived from picture order counters. Any discontinuities in incoming streams can be handled with the ``getUniqueTimestamp`` utility where a ``discontinuityCount`` and associated ``pts`` are combined to a continuous timestamp that can be passed to the LCEVCdec API.

Decode Information
------------------

The enhanced output picture received from the LCEVC enhancement decoder has information about the decoding process written into an :cpp:struct:`LCEVC_DecodeInformation` structure as an output of the :cpp:func:`LCEVC_ReceiveDecoderPicture` function:

.. code-block:: C++

  struct LCEVC_DecodeInformation
  {
    uint64_t  timestamp;
    bool      hasBase;
    bool      hasEnhancement;
    bool      skipped;
    bool      enhanced;

    uint32_t  baseWidth;
    uint32_t  baseHeight;
    uint8_t   baseBitdepth;

    void*     baseUserData;
  };

Along with the :ref:`timestamp<overview:Timestamps>` describe above, the decoder indicates to the integration information about how the picture was generated:

* Was base and enhancement data used?
* Was the picture skipped? - either as a specific request from the integration, or due to performance limitations - for example a timeout.
* The layout of the the base image used to generate this picture.
* The ``userData`` supplied when the base picture was sent to the decoder.

The integration may decide to not display the output picture based on the ``skipped`` and ``enhanced`` flags.

Buffers
-------

When a picture using externally managed data is allocated by calling :cpp:func:`LCEVC_AllocPictureExternal`, the memory supplied by the integration is described by an :cpp:struct:`LCEVC_PictureBufferDesc` structure:

.. code-block:: C++

  struct LCEVC_PictureBufferDesc
  {
    uint8_t* data;
    uint32_t byteSize;
    LCEVC_AccelBufferHandle accelBuffer;
    LCEVC_Access access;
  };

The buffer description structure supplied by the integration is copied within the picture allocation function. and will not be further referenced by the decoder.

This buffer must be large enough for all potential layout of the pictures.

Whilst a picture with an external buffer is being used by the decoder (i.e. between the **Send...()** and **Receive...()** functions, the memory should not be written, and the results of any reads are not defined.

Access
......

The buffer description includes an :cpp:member:`~LCEVC_PictureBufferDesc::access` member, that describes whether the buffer is read only, or can be written. The base pictures would usually be read-only, and the enhancement pictures would be writeable. This access level will be checked when the integration tries to access the picture's data using  :cpp:func:`LCEVC_LockPicture`.

Events
------

The LCEVC enhancement decoding process can be completely managed using the API calls described above, however there are integration situations where it is very useful to have callbacks from the decoder to indicate when further progress can be made.

The events are generated with the various API functions - and queued up to a separate thread that then invokes any supplied integration callback. That thread will interlock with the entry and exit of the API functions, and can call API functions as needed:

* Events are generated whenever any of the **Send..()** calls can accept data - i.e. it will not return :cpp:enumerator:`LCEVC_Again`.
* Events are generated whenever any of the **Receive..()** calls can has data to return - i.e. it will not return :cpp:enumerator:`LCEVC_Again`.
* An event is generated when the decoder is about to be destroyed.
* An event is generated for any implementation specific log messages.


Handles
-------

All objects that are allocated by the LCEVC decoder are referenced by *Handles*. These are opaque integers - with the exception that a 0 value does not refer to an object.

The handles are defined in the SDK header as a struct containing one integer, which is than passed by value in and out of the API:

.. code-block:: C++

  struct LCEVC_DecoderHandle
  {
      uintptr_t hdl;
  };

A null handle can be created by initialising it to 0:

.. code-block:: C++

  LCEVC_DecoderHandle decoderHandle = {0};

A handle can be tested for null by checking the value:

.. code-block:: C++

  if(decoderHandle.hdl ==0) {
    // ...
  }
