RQ3:  How do different optimization algorithms impact the performance of MORTAR?

A critical factor that can notably affect the performance of MORTAR is the choice of the algorithm used to solve the optimization problem. Hence, in this RQ, we compare the proposed BIM-based algorithm with two traditional optimization algorithms to investigate their impacts on repair effectiveness and computational overhead.

Setup

To examine the influence of optimization algorithms on the effectiveness of MORTAR, we compare the proposed BIM-based method with two traditional optimization algorithms: Nelder-Mead and COBYLA. The performance of these algorithms heavily depends on the maximum iterations allowed. To ensure a fair comparison, we test these traditional algorithms at two different settings for maximum iterations. The first setting limits the iterations to 3, matching the setup for the targeted BIM. The second setting allows for 100 iterations, giving these algorithms more opportunity to find better solutions. We then assess and compare the repair performance and computational overhead under these different optimization algorithms.

Result

Table 7 displays the performance of different optimization algorithms in repairing unsafe actions. With only 3 iterations, the proposed targeted BIM-based algorithm outperforms the other two traditional optimization methods in 15 out of 16 tasks. The only exception is the Cube Stacking task with the TRPO control policy under strict specifications, where Nelder-Mead achieves a slightly better result. When the maximum iteration limit is increased to 100, the traditional optimization methods are able to find better solutions, leading to success rates comparable to our proposed method.

However, using a higher number of maximum iterations notably increases the computational overhead.  As shown in Table 8, running the Nelder-Mead and COBYLA algorithms with 100 maximum iterations typically results in computational times exceeding 20 ms. Given that a typical AI-CPS often operates the controller at 60 Hz (16.67 ms per step), such extended computational times can introduce delays in system operation and result in latency issues. Therefore, when considering both the effectiveness of action repair and computational efficiency, our proposed method provides the most satisfactory performance.