基于RG-DDPG的直流微网能量管理策略

doi:10.11930/j.issn.1004-9649.202210086

摘要/Abstract

摘要： 针对分布式能源的随机性和间歇性给直流微网能量管理带来的巨大挑战，提出一种基于奖励指导深度确定性策略梯度（reward guidance deep deterministic policy gradient，RG-DDPG）的直流微网能量管理策略。该策略将直流微网的优化运行描述为一个马尔科夫决策过程，利用智能体与直流微网环境间的持续交互，自适应地学习能量管理决策，实现直流微网能量的优化管理。在策略训练过程中，采用基于时序差分误差（temporal difference error，TD-error）的优先经验回放机制减少RG-DDPG在直流微网运行环境中学习、探索的随机性和盲目性，提升所提能量优化管理策略的收敛速度。同时，在训练回合间利用累计奖励的大小构造直流微网能量管理的优秀回合集，加强RG-DDPG智能体在训练回合间的联系，最大化利用优秀回合的训练价值。算例仿真结果表明：所提策略能够实现直流微网内能量的合理分配。相较于基于深度Q网络（deep Q-network，DQN）和粒子群算法（particle swarm optimization，PSO）的能量管理策略，所提策略能使直流微网日平均运行成本分别降低11.16%和7.10%。

关键词: 直流微网, 能量管理, RG-DDPG, 优先经验回放, 优秀回合集

Abstract: The randomness and intermittency of distributed energy have brought great challenges to the energy management of direct current (DC) microgrids. In order to solve this challenge, a DC microgrid energy management strategy based on reward guidance deep deterministic policy gradient (RG-DDPG) is proposed in this paper. This strategy describes the optimal operation of the DC microgrid as a Markov decision process and uses the continuous interaction between the agent and the DC microgrid environment to adaptively learn energy management decisions, thus realizing the optimal management of the DC microgrid energy. In the strategy training process, the priority experience replay mechanism based on temporal difference error (TD-error) is used to reduce the randomness and blindness of RG-DDPG’s learning and exploration in the DC microgrid operating environment and improve the convergence speed of the energy optimization and management strategy proposed in this paper. At the same time, during the training rounds, the size of the accumulated rewards is used to construct an excellent round set of DC microgrid energy management, strengthen the connection between RG-DDPG agents in the training rounds, and maximize the use of the training value of the excellent round. The simulation results show that the proposed strategy can reasonably distribute energy in the DC microgrid. Compared with the energy management strategy based on deep Q learning (DQN) and particle swarm optimization (PSO), the proposed strategy can reduce the daily average operation cost of DC microgrids by 11.16% and 7.10%, respectively.

Key words: DC microgrid, energy management, RG-DDPG, priority experience replay, excellent round set

李建标, 陈建福, 高滢, 裴星宇, 吴宏远, 陆子凯, 周少雄, 曾杰. 基于RG-DDPG的直流微网能量管理策略[J]. 中国电力, 2023, 56(7): 85-94.

LI Jianbiao, CHEN Jianfu, GAO Ying, PEI Xingyu, WU Hongyuan, LU Zikai, ZHOU Shaoxiong, ZENG Jie. Strategy for DC Microgrid Energy Management Based on RG-DDPG[J]. Electric Power, 2023, 56(7): 85-94.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202210086

https://www.electricpower.com.cn/CN/Y2023/V56/I7/85

参考文献

[1] 国家能源局. 我国可再生发电装机总容量突破11亿千瓦[DB/OL]. (2022-06-24) [2022-11-08] http://www.nea.gov.cn/2022-06/24/c_1310631956.htm.
[2] 吴强, 韩俊, 金颋, 等. 冷热电联供型孤岛微电网环保经济调度模型[J]. 中国电力, 2021, 54(12): 137–142
WU Qiang, HAN Jun, JIN Ting, et al. Environmental and economic dispatch model for island microgrid of combined cooling, heating and power[J]. Electric Power, 2021, 54(12): 137–142
[3] 李相俊, 马会萌, 姜倩. 新能源侧储能配置技术研究综述[J]. 中国电力, 2022, 55(1): 13–25
LI Xiangjun, MA Huimeng, JIANG Qian. Review of energy storage configuration technology on renewable energy side[J]. Electric Power, 2022, 55(1): 13–25
[4] 林其友, 蒋文良, 李媛媛, 等. 基于母线电压分层的直流微网系统协调控制[J]. 中国电力, 2022, 55(2): 166–171, 180
LIN Qiyou, JIANG Wenliang, LI Yuanyuan, et al. Coordinated control of DC microgrid system based on bus voltage stratification[J]. Electric Power, 2022, 55(2): 166–171, 180
[5] 陈景文, 周媛, 李晓飞, 等. 光储直流微网混合储能控制策略研究[J]. 智慧电力, 2022, 50(1): 14–20, 87
CHEN Jingwen, ZHOU Yuan, LI Xiaofei, et al. Hybrid energy storage control strategy of optical storage DC microgrid[J]. Smart Power, 2022, 50(1): 14–20, 87
[6] 陈友芹, 蒋炯, 殷展翔, 等. 基于NCPSO的微网群优化调度策略研究[J]. 太阳能学报, 2022, 43(8): 477–483
CHEN Youqin, JIANG Jiong, YIN Zhanxiang, et al. Research on optimal scheduling strategy of microgrid clusters based on NCPSO[J]. Acta Energiae Solaris Sinica, 2022, 43(8): 477–483
[7] 张政林, 张惠娟, 孙文治, 等. 含混合储能的微电网能量优化管理[J]. 科学技术与工程, 2022, 22(25): 11049–11056
ZHANG Zhenglin, ZHANG Huijuan, SUN Wenzhi, et al. Optimal energy management of microgrid with hybrid energy storage[J]. Science Technology and Engineering, 2022, 22(25): 11049–11056
[8] 张祥宇, 舒一楠, 付媛, 等. 含虚拟储能直流微电网的源荷储能量协同优化控制[J/OL]. 高电压技术: 1–12[2023-05-17]. https: //doi. org/10.13336/j. 1003-6520. hve. 20221350.
ZHANG Xiangyu, SHU Yinan, FU Yuan, et al. Cooperative optimization control of source-load-storage energy in DC microgrid with virtual energy storage [J/OL]. High Voltage Engineering: 1–12[2023-05-17]. https://doi.org/10.13336/j.1003-6520.hve.20221350.
[9] 柏晨, 王念春, 卓青. 绿色数据中心的供电运行控制和能量管理[J]. 电源学报, 2021, 19(1): 107–114
BO Chen, WANG Nianchun, ZHUO Qing. Power supply operation control and energy management in green data center[J]. Journal of Power Supply, 2021, 19(1): 107–114
[10] 王毅, 张丽荣, 李和明, 等. 风电直流微网的电压分层协调控制[J]. 中国电机工程学报, 2013, 33(4): 16–24, 4
WANG Yi, ZHANG Lirong, LI Heming, et al. Hierarchical coordinated control of wind turbine-based DC microgrid[J]. Proceedings of the CSEE, 2013, 33(4): 16–24, 4
[11] 宋宪可, 樊艳芳, 刘群杰, 等. 基于功率信号判别的光-储-燃直流微网协调控制策略研究[J]. 电力电容器与无功补偿, 2020, 41(3): 197–204
SONG Xianke, FAN Yanfang, LIU Qunjie, et al. Study on DC microgrid coordination control strategy of optical-storage-fuel based on power signal discrimination[J]. Power Capacitor & Reactive Power Compensation, 2020, 41(3): 197–204
[12] 康家玉, 史晨雨, 王素娥, 等. 基于自适应下垂控制的多储能直流微网能量管理策略[J]. 现代电力, 2022, 39(2): 219–227
KANG Jiayu, SHI Chenyu, WANG Sue, et al. Energy management strategy for DC microgrid of multi-energy storage based on adaptive droop control[J]. Modern Electric Power, 2022, 39(2): 219–227
[13] 谭文娟, 陈燕东, 杨苓, 等. 光储直流微网能量协调控制方法[J]. 电源学报, 2018, 16(2): 76–85
TAN Wenjuan, CHEN Yandong, YANG Ling, et al. Energy coordinated control method for DC microgrid with photovoltaic[J]. Journal of Power Supply, 2018, 16(2): 76–85
[14] 王毅, 于明, 李永刚. 基于改进微分进化算法的风电直流微网能量管理[J]. 电网技术, 2015, 39(9): 2392–2397
WANG Yi, YU Ming, LI Yonggang. Energy management of wind turbine-based DC microgrid using improved differential algorithm[J]. Power System Technology, 2015, 39(9): 2392–2397
[15] 李瑞, 李占凯, 张福民, 等. 基于共识算法的直流微网群分布式优化调度策略[J]. 南方电网技术, 2022, 16(1): 49–57
LI Rui, LI Zhankai, ZHANG Fumin, et al. Distributed optimal scheduling strategy of DC microgrid cluster based on consensus algorithm[J]. Southern Power System Technology, 2022, 16(1): 49–57
[16] 茅靖峰, 武剑, 张珍梦, 等. 基于AFPSO的孤岛型直流微网多目标优化研究[J]. 太阳能学报, 2021, 42(6): 63–71
MAO Jingfeng, WU Jian, ZHANG Zhenmeng, et al. Multi-objective optimization of island-type DC microgrid based on AFPSO[J]. Acta Energiae Solaris Sinica, 2021, 42(6): 63–71
[17] 程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1): 15–31
CHENG Lefeng, YU Tao, ZHANG Xiaoshun, et al. Machine learning for energy and electric power systems: state of the art and prospects[J]. Automation of Electric Power Systems, 2019, 43(1): 15–31
[18] 冯昌森, 张瑜, 文福拴, 等. 基于深度期望Q网络算法的微电网能量管理策略[J]. 电力系统自动化, 2022, 46(3): 14–22
FENG Changsen, ZHANG Yu, WEN Fushuan, et al. Energy management strategy for microgrid based on deep expected Q network algorithm[J]. Automation of Electric Power Systems, 2022, 46(3): 14–22
[19] 刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794–3803
LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794–3803
[20] 高振海, 闫相同, 高菲, 等. 仿驾驶员DDPG汽车纵向自动驾驶决策方法[J]. 汽车工程, 2021, 43(12): 1737–1744
GAO Zhenhai, YAN Xiangtong, GAO Fei, et al. A driver-like decision-making method for longitudinal autonomous driving based on DDPG[J]. Automotive Engineering, 2021, 43(12): 1737–1744
[21] 余宏晖, 林声宏, 朱建全, 等. 基于深度强化学习的微电网在线优化[J/OL]. 电测与仪表: 1–7[2022-10-04]. http://kns.cnki.net/kcms/detail/23.1202.TH.20211021.1651.007.html.
YU Honghui, LIN Shenghong, ZHU Jiangquan et al. On-line optimization of micro grid based on deep reinforcement learning[J/OL]. Electrical Measurement & Instrumentation: 1–7[2022-10-04]. http://kns.cnki.net/kcms/detail/23.1202.TH.20211021.1651.007.html.
[22] 刘蓉晖, 李阳, 杨秀, 等. 考虑需求响应的社区综合能源系统两阶段优化调度[J]. 太阳能学报, 2021, 42(9): 46–54
LIU Ronghui, LI Yang, YANG Xiu, et al. Two-stage optimal scheduling of community integrated energy system considering demand response[J]. Acta Energiae Solaris Sinica, 2021, 42(9): 46–54