Action Repair as Optimization Problem

By using the prediction model, the repaired action can be computed by solving the following optimization problem

The objective of this optimization is to minimize the difference between the safety estimate from the prediction model and the maximum value. The rationale behind this is that, due to the well-acknowledged balance between ensuring safety and encouraging exploration for higher rewards, optimizing the output towards an excessively high value may result in actions that are overly restrictive for safety, potentially neglecting the completion of task requirements. Therefore, instead of directly maximizing the output, i.e., finding the safest action, we aim to keep it close to the maximum value during optimization.

By solving the optimization problem, we determine a repaired action that offers enhanced safety. 

Practical Optimization Solving

The major challenge in solving the optimization is ensuring a time-efficient computation to minimize overhead in the control operation. Considering this, traditional search-based methods, such as Nelder-Mead and gradient-based methods, such as BFGS, often fail to perform satisfactorily. These methods may require a considerable number of iterations to find a feasible solution or need to compute estimates of the Hessian matrix for gradients, which is computationally expensive.

The key to reducing computational overhead is minimizing the number of times the prediction model is used to calculate predictions. Motivated by this, we utilize the targeted Basic Iterative Method (BIM) in this work to determine the repaired action. BIM is a widely adopted method for generating adversarial examples. It aims to incrementally modify the input so that the neural network produces a specific, altered classification result. Similarly, the action repair process seeks to modify actions to achieve a different output from the prediction model, paralleling the objectives of adversarial example generation and thereby motivating the use of the BIM method. By setting the target to the maximum value, the targeted BIM attempts to find an adversarial action patch that moves the output of the prediction model towards the maximum STL value. Note that rather than approaching the maximum STL value as closely as possible, the primary objective of the action repair process is to identify an action considered safe by the prediction model. Therefore, to further enhance computational efficiency, we incorporate an early-stop strategy within the BIM search process. Specifically, the search is terminated when the repaired action yields a value from the prediction model that falls within the valid range.

It is also important to note that, unlike traditional adversarial example generation, where classification results can change immediately, controlling CPSs generally requires smoother command inputs.  Large and sudden action changes may lead to system instability or excessive oscillating behaviour. To address this, we additionally introduce a step size to control the magnitude of the identified adversarial action patch. Moreover, the computed repaired action must be constrained by valid action space to remain within valid action boundaries. Accordingly, the complete process for solving the optimization using targeted BIM is detailed in the Algorithm. The proposed algorithm is able to find a solution in a limited number of iterations while maintaining satisfactory quality, thereby achieving a better balance between performance and computational cost.