Open source toolbox to support safety engineering of neural networks
NNDK is an open-source toolbox to support the data-driven engineering of neural networks for safety-critical domains. Its goal is to provide evidence of uncertainty reduction in key phases of the product life cycle, ranging from data collection, training & validation, testing & generalization, to operation. The toolbox consists of dependability metrics, a formal reasoning engine, and a runtime monitor.
During data preparation, one needs to ensure that the collected data covers all operating scenarios. NNDK provides quantitative projection coverage metrics, and its associated test case generation techniques are used to provide a relative form of completeness against the combinatorial explosion of scenarios. In the training and validation phase NNDK facilitates to consider the performance under different operating scenarios. For vision-based object detection, metrics such as interpretation precision and sensitivity of occlusion are provided.
NNDK also supports to test the generalization capability of the neural network subject to known perturbations, using the perturbation loss metric. Provided that domain knowledge can be formally specified, one can also apply formal verification to examine if the neural network demonstrates correct behaviour with respect to the specification.
As for the operation phase, NNDK provides a runtime monitoring routine, which relies recording binarized neuron activation patterns in training time and checking if an activation pattern produced during operation has been seen during the training. These solutions can be used in assurance cases to argue for the safety of a function deploying neural networks, and a simplified Goal Structuring Notation (GSN) diagram is provided to assist in understanding how features provided by NNDK contribute to the overall safety goal.
Currently, the toolset uses the PyTorch framework. However, most solutions are based on NumPy arrays and can be easily modified to the respective framework. Sample implementations are provided in the form of Jupyter notebooks. They use standard datasets like MNIST, GTSRB, VOC, and Kitti for computation and results.
NNDK has been used for various applications in domains of autonomous driving, medical image classification, space navigation. The simplicity of its application in DNN models makes it a domain-independent approach.
You can find all available downloads here.
