Architectural Overview

The two most important parts of the MSeg framework are the MSeg Core Module on the one hand, and the Programming Language Interfaces (PLI’s) on the other hand. The following diagram shows a general overview.

The individual parts will be explained in detail in the following sections.

MSeg Core Module

The MSeg Core Module offers the MSeg GUI, a plugin for the ArmarX GUI, and consists of three components, namely:

• DataExchange: This component is a mediator between the MSeg Core Module and a segmentation algorithm. With it, a segmentation algorithm can request motion data which can be segmented, and it can report keyframes where a segmentation occurs. Each segmentation algorithm has access to the DataExchange component, exposed through a PLI as DataExchangeProxy.
• SegmentationController: This component orchestrates the available segmentation algorithms, triggers a segmentation process, and collects the reported keyframes.
• EvaluationController: This component implements several metrics and measures from the literature to access the quality of a segmentation compared to a ground truth. There are three general approaches which are used to derive the number of true positives, false positives, false negatives and true positives (Standard approach, Margin approach, Integrated Kernel approach). With these numbers, several metrics are evaluated, for example precision and recall, F₁-score, etc. For more information, please refer to our publication.

The Core Module also defines several communication interfaces (Com. Interfaces) in a programming-language-agnostic language called Slice from the ZeroC Ice library. Slice interfaces can be transpiled into concrete interfaces in a specific programming language. In the context of MSeg, each PLI is such a concrete interface.

PLI (Programming Language Interface)

A PLI (Programming Language Interface) is a concrete implementation of the interfaces defined by the MSeg Core Module. In particular, and from the perspective of the Core Module, a PLI is an implementation in a supported programming language (hence the name). Supported are C++, Java, Python and MATLAB.

On another abstraction layer, namely from the perspective of the segmentation algorithm, the PLI offers an interface which has to be implemented by the segmentation algoritm (the class SegmentationAlgorithm).

Communication between the Core Module and an Algorithm

The following subsections discuss how the communication between the MSeg Core Module and an algorithm takes place in a particular direction.

Core Module ➜ Algorithm

The MSeg Core Module communicates with a segmentation algorithm only indirectly via a PLI. The PLI is used as a proxy, since its interface is known to MSeg. It is the responsibility of the PLI to forward the instructions of the Core Module to the contrete motion segmentation algorithm implementation. For this purpose, the SegmentationAlgorithm interface is used, a class which should be derived by the to-be implemented segmentation algorithm.

Algorithm ➜ Core Module

Only one part of the MSeg Core Module is exposed to the motion segmentation algorithm, namely the DataExchange via the data attribute of the class SegmentationAlgorithm. This attribute is an object of the type DataExchangeProxy and offers interfaces for the algorithm to fetch motion recordings in several data formats for segmentation on the on hand, and to report keyframes back on the other hand. The segmentation algorithm cannot access arbitrary data – the segmentation controller decides what data is made available for the segmentation algorithm. The segmentation controller also keeps track of how the data was used (i.e. in an online manner “frame-by-frame”, or offline “all-at-once”).