Frequency-Voltage Digital Intelligent Control Strategy of Microgrid with User-Side Energy Storage Based on Deep Q-Learning

doi:10.11930/j.issn.1004-9649.202207086

Abstract

Abstract: Frequency and voltage are important criteria for measuring energy indicators. Considering the problem of frequency/voltage regulation caused by load fluctuations in microgrids, a smart monitoring-control strategy for microgrids with user-side energy storage based on deep Q-learning (DQN) is proposed in this paper. Firstly, in view of the randomness of user behavior, the random constraints on the output of energy storage on the user side are added, and a four-quadrant charging and discharging model is introduced to build a charging and discharging model for user-side energy storage. On this basis, a cooperative control model of frequency and voltage in the microgrid is built. Secondly, the frequency/voltage controller structure and digital intelligent control platform based on DQN are designed. With the real-time frequency deviation and voltage deviation of the system and the upper and lower limit constraints of the output power of user-side energy storage as the state space and the output of each unit in the system as the action space, a global reward function including two local rewards is designed on the basis of the two control objectives of frequency and voltage. The results of the calculation examples indicate that compared with the traditional PID controller, the DQN controller proposed in this paper can simultaneously meet the control requirements of frequency and voltage and can more effectively deal with the power quality problems caused by load fluctuations.

Key words: island microgrid, user-side energy storage, coordinated frequency/voltage control, deep Q-learning, digital platform

LIN Rihui, CHEN Youli. Frequency-Voltage Digital Intelligent Control Strategy of Microgrid with User-Side Energy Storage Based on Deep Q-Learning[J]. Electric Power, 2022, 55(12): 43-50.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202207086

https://www.electricpower.com.cn/EN/Y2022/V55/I12/43

References

[1] 王岳, 杨国华, 庄家懿, 等. 基于一致性算法的微电网无差调频控制策略[J]. 中国电力, 2020, 53(10): 187–191
WANG Yue, YANG Guohua, ZHUANG Jiayi, et al. Zero-error frequency regulation control method for microgrids based on consensus algorithm[J]. Electric Power, 2020, 53(10): 187–191
[2] 随权, 魏繁荣, 林湘宁, 等. 一种基于可控负荷效率控制的孤岛微网新型调度策略[J]. 中国电机工程学报, 2019, 39(24): 7168–7178, 7490
SUI Quan, WEI Fanrong, LIN Xiangning, et al. A novel dispatching strategy for isolated microgrid based on controllable load efficiency control[J]. Proceedings of the CSEE, 2019, 39(24): 7168–7178, 7490
[3] RAHIMI F, IPAKCHI A. Demand response as a market resource under the smart grid paradigm[J]. IEEE Transactions on Smart Grid, 2010, 1(1): 82–88.
[4] 苏宏升, 江昆, 杨祯, 等. 基于虚拟同步发电机的微网频率与电压综合控制策略[J]. 电力自动化设备, 2020, 40(3): 21–28
SU Hongsheng, JIANG Kun, YANG Zhen, et al. Comprehensive control strategy of microgrid frequency and voltage based on virtual synchronous generator[J]. Electric Power Automation Equipment, 2020, 40(3): 21–28
[5] 康慨, 张超, 张云龙, 等. 交直流混合微电网二次频率电压协调控制策略[J]. 武汉大学学报(工学版), 2021, 54(3): 239–246
KANG Kai, ZHANG Chao, ZHANG Yunlong, et al. Secondary frequency and voltage coordinated control strategy for AC/DC hybrid microgrid[J]. Engineering Journal of Wuhan University, 2021, 54(3): 239–246
[6] 马经纬, 王晓梅, 陈淼, 等. 基于多智能体系统的智能微网分散协调控制策略[J]. 电网与清洁能源, 2017, 33(11): 15–23, 31
MA Jingwei, WANG Xiaomei, CHEN Miao, et al. A decentralized control method for smart micro-grid based on multi-agent system[J]. Power System and Clean Energy, 2017, 33(11): 15–23, 31
[7] 赵永良, 付鑫, 郭阳, 等. 基于深度学习和图像识别的电力配件智能出入库[J]. 中国电力, 2021, 54(3): 55–60
ZHAO Yongliang, FU Xin, GUO Yang, et al. Intelligent storage and retrieval of power accessories based on deep learning and image recognition[J]. Electric Power, 2021, 54(3): 55–60
[8] 程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1): 15–31
CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Machine learning for energy and electric power systems: state of the art and prospects[J]. Automation of Electric Power Systems, 2019, 43(1): 15–31
[9] 史建勋, 张冲标, 吴晗, 等. 高比例光伏微网无功均分控制中的Q学习方法[J]. 电力系统及其自动化学报, 2021, 33(8): 88–93
SHI Jianxun, ZHANG Chongbiao, WU Han, et al. Q-learning method in reactive power sharing control of high-proportion photovoltaic microgrid[J]. Proceedings of the CSU-EPSA, 2021, 33(8): 88–93
[10] 唐捷, 张泽宇, 程乐峰, 等. 基于CEQ(λ)强化学习算法的微电网智能发电控制[J]. 电测与仪表, 2017, 54(1): 39–45
TANG Jie, ZHANG Zeyu, CHENG Lefeng, et al. Smart generation control for micro-grids based on correlated equilibrium Q(λ) learning algorithm[J]. Electrical Measurement & Instrumentation, 2017, 54(1): 39–45
[11] 师瑞峰, 李少鹏. 电动汽车V2G问题研究综述[J]. 电力系统及其自动化学报, 2019, 31(6): 28–37
SHI Ruifeng, LI Shaopeng. Review on studies of V2G problem in electric vehicles[J]. Proceedings of the CSU-EPSA, 2019, 31(6): 28–37
[12] 崔全胜, 白晓民, 董伟杰, 等. 用户侧综合能源系统规划运行联合优化[J]. 中国电机工程学报, 2019, 39(17): 4967–4981,5279
CUI Quansheng, BAI Xiaomin, DONG Weijie, et al. Joint optimization of planning and operation in user-side multi-energy systems[J]. Proceedings of the CSEE, 2019, 39(17): 4967–4981,5279
[13] 吴科成, 高志华, 刘瑞宽, 等. 考虑分布式储能功率四象限输出的主动配电网鲁棒优化调度模型[J]. 南方电网技术, 2021, 15(11): 75–84
WU Kecheng, GAO Zhihua, LIU Ruikuan, et al. Robust optimal scheduling model of active distribution network considering four-quadrant output of distributed energy storage power[J]. Southern Power System Technology, 2021, 15(11): 75–84
[14] YANG J, ZENG Z L, TANG Y F, et al. Load frequency control in isolated micro-grids with electrical vehicles based on multivariable generalized predictive theory[J]. Energies, 2015, 8: 2145–2164.[LinkOut].
[15] 范培潇, 杨军, 肖金星, 等. 基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略[J]. 电力建设, 2022, 43(4): 91–99
FAN Peixiao, YANG Jun, XIAO Jinxing, et al. Load frequency control strategy based on deep Q learning for island microgrid with electric vehicles[J]. Electric Power Construction, 2022, 43(4): 91–99
[16] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1–27
LIU Quan, ZHAI JianWei, ZHANG ZongZhang, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1): 1–27
[17] 赵星宇, 丁世飞. 深度强化学习研究综述[J]. 计算机科学, 2018, 45(7): 1–6
ZHAO Xingyu, DING Shifei. Research on deep reinforcement learning[J]. Computer Science, 2018, 45(7): 1–6
[18] HUANG L W, FU M S, QU H, et al. A deep reinforcement learning-based method applied for solving multi-agent defense and attack problems[J]. Expert Systems With Applications, 2021, 176: 114896.
[19] 赵建立, 汤卓凡, 王桂林, 等. 具有储能作用的用户侧资源运行特性[J]. 综合智慧能源, 2022, 44(2): 8–14
ZHAO Jianli, TANG Zhuofan, WANG Guilin, et al. Operation characteristics of user-side resources with energy storage function[J]. Integrated Intelligent Energy, 2022, 44(2): 8–14
[20] 姚蓝霓, 李钦豪, 杨景旭, 等. 考虑电动汽车充放电支撑的配用电系统综合无功优化[J]. 电力系统自动化, 2022, 46(6): 39–47
YAO Lanni, LI Qinhao, YANG Jingxu, et al. Comprehensive reactive power optimization of power distribution and consumption system with support of electric vehicle charging and discharging[J]. Automation of Electric Power Systems, 2022, 46(6): 39–47
[21] RAO Y Q, YANG J, XIAO J X, et al. A frequency control strategy for multimicrogrids with V2 G based on the improved robust model predictive control[J]. Energy, 2021, 222: 119963.
[22] 刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794–3803
LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794–3803