Safe Reinforcement Learning for Start-up and Operation of Chemical Processes

Despite of advances in computational methods, autonomous operation of complex chemical processes remains a very challenging problem. It is still not possible to robustly operate the plant in all possible situations using pure model-based optimization strategies, e.g., under heavy disturbances or product switches. Complex operations are performed by following recipes and manual adaptations by operators. This leads to suboptimal plant operation, but also severely limits flexibility of many industrial processes, as start-up and shut-down procedures need to be avoided due to the lack of systematic strategies to deal with them.The increasing availability of large amounts of data opens the door for operation strategies based thereon, such as reinforcement learning (RL). However, some critical challenges of RL, as the rigorous consideration of process constraints or the large amount of data required, prevent RL from being applied on chemical processes.To mitigate these drawbacks of RL, synergies between three different disciplines are explored: expert / domain knowledge in the form of operational recipes, model-based control, and reinforcement learning.The main goal of this project is to develop a systematic approach to the real implementation and investigations under real conditions of RL in complex chemical processes in order to optimize dynamic operation by safeguarding process constraints, resulting in a methodology for the applicationof RL with nonlinear model predictive control (NMPC) for cases where many practical experiments are (1) expensive, (2) might frequently fail, and (3) plant-model mismatch is significant.This project employs a wide variety of methods from chemical engineering by using both detailed dynamic simulation models and experimental / historical data from real-life experiments performed on an already existing batch distillation column at TUB. The optimization of its operation cycle is representative of typical complex applications in chemical engineering.From the field of machine learning we introduce the idea of the parameter manifold on virtual experiments to obtain inclusive descriptions for cases where the representation of the real-life experiment is uncertain. Efficient sampling strategies for RL are provided, and highly accurate surrogate models based on large amounts of data are developed. We will also develop a new safe RL method that can be continuously improved but is safe because it uses an NMPC as policy approximator.The proposed workflow is constructed in a generic fashion, finding application in single unit operations as well as in processes with multiple units, tackling especially issues concerning plant-model mismatch. Both fields, CE and ML, can together develop the necessary methods for achieving a robust optimal control.