基于深度Q学习的含用户侧储能微电网频率-电压数字化智能控制策略

doi:10.11930/j.issn.1004-9649.202207086

摘要/Abstract

摘要： 频率与电压是衡量电能指标的重要标准。针对微电网受到负荷波动而引起的频率/电压调控问题，提出基于深度Q学习(deep Q-learning, DQN)的含用户侧储能微电网智能监控-控制策略。首先，通过考虑用户行为的随机性，增加了用户侧储能输出的随机约束，并引入四象限充放电的模型，构建用户侧储能的集群充放电模型，从而搭建出微电网频率-电压的协同控制模型。其次，设计基于DQN的频率/电压控制器结构与数字化智能控制平台，以系统实时的频率偏差、电压偏差与用户侧储能输出功率的上、下限约束为状态空间，以系统各机组出力为动作空间，并基于频率及电压2个控制目标，完成包含2个本地奖励的全局奖励函数的设计。算例结果表明：与传统PID控制器相比，所提DQN控制器能同时满足频率与电压的控制需求，更有效地应对负荷波动所引起的电能质量问题。

关键词: 孤岛微电网, 用户侧储能, 频率/电压协调控制, 深度Q学习算法, 数字化平台

Abstract: Frequency and voltage are important criteria for measuring energy indicators. Considering the problem of frequency/voltage regulation caused by load fluctuations in microgrids, a smart monitoring-control strategy for microgrids with user-side energy storage based on deep Q-learning (DQN) is proposed in this paper. Firstly, in view of the randomness of user behavior, the random constraints on the output of energy storage on the user side are added, and a four-quadrant charging and discharging model is introduced to build a charging and discharging model for user-side energy storage. On this basis, a cooperative control model of frequency and voltage in the microgrid is built. Secondly, the frequency/voltage controller structure and digital intelligent control platform based on DQN are designed. With the real-time frequency deviation and voltage deviation of the system and the upper and lower limit constraints of the output power of user-side energy storage as the state space and the output of each unit in the system as the action space, a global reward function including two local rewards is designed on the basis of the two control objectives of frequency and voltage. The results of the calculation examples indicate that compared with the traditional PID controller, the DQN controller proposed in this paper can simultaneously meet the control requirements of frequency and voltage and can more effectively deal with the power quality problems caused by load fluctuations.

Key words: island microgrid, user-side energy storage, coordinated frequency/voltage control, deep Q-learning, digital platform

林日晖, 陈友立. 基于深度Q学习的含用户侧储能微电网频率-电压数字化智能控制策略[J]. 中国电力, 2022, 55(12): 43-50.

LIN Rihui, CHEN Youli. Frequency-Voltage Digital Intelligent Control Strategy of Microgrid with User-Side Energy Storage Based on Deep Q-Learning[J]. Electric Power, 2022, 55(12): 43-50.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202207086

https://www.electricpower.com.cn/CN/Y2022/V55/I12/43

参考文献

[1] 王岳, 杨国华, 庄家懿, 等. 基于一致性算法的微电网无差调频控制策略[J]. 中国电力, 2020, 53(10): 187–191
WANG Yue, YANG Guohua, ZHUANG Jiayi, et al. Zero-error frequency regulation control method for microgrids based on consensus algorithm[J]. Electric Power, 2020, 53(10): 187–191
[2] 随权, 魏繁荣, 林湘宁, 等. 一种基于可控负荷效率控制的孤岛微网新型调度策略[J]. 中国电机工程学报, 2019, 39(24): 7168–7178, 7490
SUI Quan, WEI Fanrong, LIN Xiangning, et al. A novel dispatching strategy for isolated microgrid based on controllable load efficiency control[J]. Proceedings of the CSEE, 2019, 39(24): 7168–7178, 7490
[3] RAHIMI F, IPAKCHI A. Demand response as a market resource under the smart grid paradigm[J]. IEEE Transactions on Smart Grid, 2010, 1(1): 82–88.
[4] 苏宏升, 江昆, 杨祯, 等. 基于虚拟同步发电机的微网频率与电压综合控制策略[J]. 电力自动化设备, 2020, 40(3): 21–28
SU Hongsheng, JIANG Kun, YANG Zhen, et al. Comprehensive control strategy of microgrid frequency and voltage based on virtual synchronous generator[J]. Electric Power Automation Equipment, 2020, 40(3): 21–28
[5] 康慨, 张超, 张云龙, 等. 交直流混合微电网二次频率电压协调控制策略[J]. 武汉大学学报(工学版), 2021, 54(3): 239–246
KANG Kai, ZHANG Chao, ZHANG Yunlong, et al. Secondary frequency and voltage coordinated control strategy for AC/DC hybrid microgrid[J]. Engineering Journal of Wuhan University, 2021, 54(3): 239–246
[6] 马经纬, 王晓梅, 陈淼, 等. 基于多智能体系统的智能微网分散协调控制策略[J]. 电网与清洁能源, 2017, 33(11): 15–23, 31
MA Jingwei, WANG Xiaomei, CHEN Miao, et al. A decentralized control method for smart micro-grid based on multi-agent system[J]. Power System and Clean Energy, 2017, 33(11): 15–23, 31
[7] 赵永良, 付鑫, 郭阳, 等. 基于深度学习和图像识别的电力配件智能出入库[J]. 中国电力, 2021, 54(3): 55–60
ZHAO Yongliang, FU Xin, GUO Yang, et al. Intelligent storage and retrieval of power accessories based on deep learning and image recognition[J]. Electric Power, 2021, 54(3): 55–60
[8] 程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1): 15–31
CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Machine learning for energy and electric power systems: state of the art and prospects[J]. Automation of Electric Power Systems, 2019, 43(1): 15–31
[9] 史建勋, 张冲标, 吴晗, 等. 高比例光伏微网无功均分控制中的Q学习方法[J]. 电力系统及其自动化学报, 2021, 33(8): 88–93
SHI Jianxun, ZHANG Chongbiao, WU Han, et al. Q-learning method in reactive power sharing control of high-proportion photovoltaic microgrid[J]. Proceedings of the CSU-EPSA, 2021, 33(8): 88–93
[10] 唐捷, 张泽宇, 程乐峰, 等. 基于CEQ(λ)强化学习算法的微电网智能发电控制[J]. 电测与仪表, 2017, 54(1): 39–45
TANG Jie, ZHANG Zeyu, CHENG Lefeng, et al. Smart generation control for micro-grids based on correlated equilibrium Q(λ) learning algorithm[J]. Electrical Measurement & Instrumentation, 2017, 54(1): 39–45
[11] 师瑞峰, 李少鹏. 电动汽车V2G问题研究综述[J]. 电力系统及其自动化学报, 2019, 31(6): 28–37
SHI Ruifeng, LI Shaopeng. Review on studies of V2G problem in electric vehicles[J]. Proceedings of the CSU-EPSA, 2019, 31(6): 28–37
[12] 崔全胜, 白晓民, 董伟杰, 等. 用户侧综合能源系统规划运行联合优化[J]. 中国电机工程学报, 2019, 39(17): 4967–4981,5279
CUI Quansheng, BAI Xiaomin, DONG Weijie, et al. Joint optimization of planning and operation in user-side multi-energy systems[J]. Proceedings of the CSEE, 2019, 39(17): 4967–4981,5279
[13] 吴科成, 高志华, 刘瑞宽, 等. 考虑分布式储能功率四象限输出的主动配电网鲁棒优化调度模型[J]. 南方电网技术, 2021, 15(11): 75–84
WU Kecheng, GAO Zhihua, LIU Ruikuan, et al. Robust optimal scheduling model of active distribution network considering four-quadrant output of distributed energy storage power[J]. Southern Power System Technology, 2021, 15(11): 75–84
[14] YANG J, ZENG Z L, TANG Y F, et al. Load frequency control in isolated micro-grids with electrical vehicles based on multivariable generalized predictive theory[J]. Energies, 2015, 8: 2145–2164.[LinkOut].
[15] 范培潇, 杨军, 肖金星, 等. 基于深度Q学习的含电动汽车孤岛微电网负荷频率控制策略[J]. 电力建设, 2022, 43(4): 91–99
FAN Peixiao, YANG Jun, XIAO Jinxing, et al. Load frequency control strategy based on deep Q learning for island microgrid with electric vehicles[J]. Electric Power Construction, 2022, 43(4): 91–99
[16] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1–27
LIU Quan, ZHAI JianWei, ZHANG ZongZhang, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1): 1–27
[17] 赵星宇, 丁世飞. 深度强化学习研究综述[J]. 计算机科学, 2018, 45(7): 1–6
ZHAO Xingyu, DING Shifei. Research on deep reinforcement learning[J]. Computer Science, 2018, 45(7): 1–6
[18] HUANG L W, FU M S, QU H, et al. A deep reinforcement learning-based method applied for solving multi-agent defense and attack problems[J]. Expert Systems With Applications, 2021, 176: 114896.
[19] 赵建立, 汤卓凡, 王桂林, 等. 具有储能作用的用户侧资源运行特性[J]. 综合智慧能源, 2022, 44(2): 8–14
ZHAO Jianli, TANG Zhuofan, WANG Guilin, et al. Operation characteristics of user-side resources with energy storage function[J]. Integrated Intelligent Energy, 2022, 44(2): 8–14
[20] 姚蓝霓, 李钦豪, 杨景旭, 等. 考虑电动汽车充放电支撑的配用电系统综合无功优化[J]. 电力系统自动化, 2022, 46(6): 39–47
YAO Lanni, LI Qinhao, YANG Jingxu, et al. Comprehensive reactive power optimization of power distribution and consumption system with support of electric vehicle charging and discharging[J]. Automation of Electric Power Systems, 2022, 46(6): 39–47
[21] RAO Y Q, YANG J, XIAO J X, et al. A frequency control strategy for multimicrogrids with V2 G based on the improved robust model predictive control[J]. Energy, 2021, 222: 119963.
[22] 刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794–3803
LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794–3803