The two most important parts of the MSeg framework are the MSeg Core Module on the one hand, and the Programming Language Interfaces (PLI’s) on the other hand. The following diagram shows a general overview.
The individual parts will be explained in detail in the following sections.
MSeg Core Module¶
The MSeg Core Module offers the MSeg GUI, a plugin for the ArmarX GUI, and consists of three components, namely:
- DataExchange: This component is a mediator between the MSeg Core Module
and a segmentation algorithm. With it, a segmentation algorithm can request
motion data which can be segmented, and it can report keyframes where a
segmentation occurs. Each segmentation algorithm has access to the
DataExchange component, exposed through a PLI as
- SegmentationController: This component orchestrates the available segmentation algorithms, triggers a segmentation process, and collects the reported keyframes.
- EvaluationController: This component implements several metrics and measures from the literature to access the quality of a segmentation compared to a ground truth. There are three general approaches which are used to derive the number of true positives, false positives, false negatives and true positives (Standard approach, Margin approach, Integrated Kernel approach). With these numbers, several metrics are evaluated, for example precision and recall, F1-score, etc. For more information, please refer to our publication.
The Core Module also defines several communication interfaces (Com. Interfaces) in a programming-language-agnostic language called Slice from the ZeroC Ice library. Slice interfaces can be transpiled into concrete interfaces in a specific programming language. In the context of MSeg, each PLI is such a concrete interface.
PLI (Programming Language Interface)¶
A PLI (Programming Language Interface) is a concrete implementation of the interfaces defined by the MSeg Core Module. In particular, and from the perspective of the Core Module, a PLI is an implementation in a supported programming language (hence the name). Supported are C++, Java, Python and MATLAB.
On another abstraction layer, namely from the perspective of the segmentation
algorithm, the PLI offers an interface which has to be implemented by the
segmentation algoritm (the class
Communication between the Core Module and an Algorithm¶
The following subsections discuss how the communication between the MSeg Core Module and an algorithm takes place in a particular direction.
Core Module ➜ Algorithm¶
The MSeg Core Module communicates with a segmentation algorithm only indirectly
via a PLI. The PLI is used as a proxy, since its interface is known to MSeg. It
is the responsibility of the PLI to forward the instructions of the Core Module
to the contrete motion segmentation algorithm implementation. For this purpose,
SegmentationAlgorithm interface is used, a class which should be derived
by the to-be implemented segmentation algorithm.
Algorithm ➜ Core Module¶
Only one part of the MSeg Core Module is exposed to the motion segmentation
algorithm, namely the DataExchange via the
data attribute of the class
SegmentationAlgorithm. This attribute is an object of the type
DataExchangeProxy and offers interfaces for the algorithm to fetch motion
recordings in several data formats for segmentation on the on hand, and to
report keyframes back on the other hand. The segmentation algorithm cannot
access arbitrary data – the segmentation controller decides what data is made
available for the segmentation algorithm. The segmentation controller also keeps
track of how the data was used (i.e. in an online manner “frame-by-frame”, or