Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

doi:10.11930/j.issn.1004-9649.202503014

Abstract

Abstract:

To address the overestimation bias and poor decision accuracy of conventional microgrid cluster energy management methods, an energy management strategy based on improved double deep Q-network is proposed. Firstly, this study constructed a dual-objective value network framework based on clipped double Q-learning, which enhances decision-making precision by suppressing value overestimation bias through parallel computation of temporal difference (TD) targets for dual value networks and clipping high TD target values. And then, a dynamic greedy strategy was adopted to calculate the value function of all possible actions based on the current state, avoiding persistent exploitation of the greedy actions to ensure sufficient exploration and prevent premature convergence of the agent. Finally, a case study of a microgrid cluster with three sub-microgrids was conducted for verification. The simulation results show that compared to the energy management strategies based on model predictive control and conventional double deep Q-network, the proposed method achieves superior optimization performance and convergence characteristics, while reducing system operating costs by 44.62% and 26.39% respectively.

Key words: microgrid cluster, energy management, improved double deep Q-network, clipped double Q values, greedy strategy

CLC Number:

TM73

HE Jintao, WANG Can, WANG Mingchao, CHENG Bentao, LIU Yuzheng, CHANG Wenhan, WANG Rui, YU Han. Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network[J]. Electric Power, 2025, 58(10): 14-26.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202503014

https://www.electricpower.com.cn/EN/Y2025/V58/I10/14

Figures/Tables 15

Fig.1 MGC system structure

Fig.2 MGC energy management training process

Table 1 Parameters of MGC system

MG	电池额定容量/(kW·h)	充放电效率/%	微燃机出力上限/(kW·h)	爬坡速率/ (kW·s^–1)	价格响应负荷/kW
1	600	0.9	600	6	175
2	1 000	0.9	800	6	150
3	800	0.9	400	6	200

Fig.3 Forecast of wind power of each MG

Fig.4 User electrical load power for each MG

Table 2 Distribution network purchase and sale electricity price

时段	购电电价/(元·(kW·h)^–1)	售电电价/(元·(kW·h)^–1)
11:00—16:00 19:00—22:00	1.079	0.845
08:00—11:00 16:00—19:00 22:00—00:00	0.637	0.494
00:00—08:00	0.421	0.322

Table 3 MGC model training parameters

超参数		数值
奖励折扣率		0.99
学习率		0.001
目标网络Q网络更新权值的步数C		200
最大探索率$ {\alpha _{\max }} $		0.3
最小探索率$ {\alpha _{\min }} $		0.01

Fig.5 Comparison of training results for different algorithms

Fig.6 Comparison of training results with different discretization granularities

Fig.7 Change curves of true Q-values and estimated Q-values for different algorithms

Fig.8 Comparison of the absolute values of Q-value estimation bias

Table 4 Performance comparison of improved DDQN algorithm with different parameters

参数设置		平均奖励收敛值	收敛轮数	后50%训练周期方差$\sigma $值
$ {\alpha _{\max }} $	$ {\alpha _{\min }} $	平均奖励收敛值	收敛轮数	后50%训练周期方差$\sigma $值
0.1	0.01	–1153.2	843	32.74
0.2	0.01	–975.6	693	26.17
0.3	0.01	–813.7	540	12.49
0.4	0.01	–891.1	652	19.21
0.3	0	–873.9	581	15.76
0.3	0.10	–858.4	603	46.59

Fig.9 MGC energy management results based on improved DDQN algorithm

Fig.10 MG energy storage charging and discharging power and SOC state change

Fig.11 Benefit comparison for different methods

References 30

1	王灿, 张雪菲, 凌凯, 等. 基于区间概率不确定集的微电网两阶段自适应鲁棒优化调度[J]. 中国电机工程学报, 2024, 44 (5): 1750- 1764.
	WANG Can, ZHANG Xuefei, LING Kai, et al. Two-stage adaptive robust optimal scheduling based on the interval probability uncertainty set for microgrids[J]. Proceedings of the CSEE, 2024, 44 (5): 1750- 1764.
2	刘任, 刘洋, 许立雄, 等. 计及分布式需求响应的多微电网系统协同优化策略[J]. 电力建设, 2023, 44 (5): 72- 83.
	LIU Ren, LIU Yang, XU Lixiong, et al. Multi-microgrid system collaborative optimization strategy considering distributed demand response[J]. Electric Power Construction, 2023, 44 (5): 72- 83.
3	易文飞, 朱卫平, 郑明忠. 计及数据中心和风电不确定性的微电网经济调度[J]. 中国电力, 2024, 57 (2): 19- 26.
	YI Wenfei, ZHU Weiping, ZHENG Mingzhong. Economic dispatch of microgrid considering data center and wind power uncertainty[J]. Electric Power, 2024, 57 (2): 19- 26.
4	谭玲玲, 汤伟, 楚冬青, 等. 考虑电-氢一体化的微电网低碳-经济协同优化调度[J]. 中国电力, 2024, 57 (5): 137- 148.
	TAN Lingling, TANG Wei, CHU Dongqing, et al. Low-carbon-economic collaborative optimal dispatching of microgrid considering electricity-hydrogen integration[J]. Electric Power, 2024, 57 (5): 137- 148.
5	樊晓伟, 王瑞妙, 杨海峰, 等. 计及源荷不确定的综合能源微电网集群优化运行[J]. 电力建设, 2024, 45 (8): 128- 139.
	FAN Xiaowei, WANG Ruimiao, YANG Haifeng, et al. Optimization operation of integrated energy microgrid cluster considering source-load uncertainty[J]. Electric Power Construction, 2024, 45 (8): 128- 139.
6	凌凯, 王灿, 张高瑞, 等. 考虑路径损耗的热电联供型微网三层能量优化策略[J]. 中国电力, 2023, 56 (2): 102- 113.
	LING Kai, WANG Can, ZHANG Gaorui, et al. Three-layer energy optimization strategy for CHP microgrid considering path loss[J]. Electric Power, 2023, 56 (2): 102- 113.
7	姚文亮, 王成福, 赵雨菲, 等. 不确定性环境下基于合作博弈的综合能源系统分布式优化[J]. 电力系统自动化, 2022, 46 (20): 43- 53.
	YAO Wenliang, WANG Chengfu, ZHAO Yufei, et al. Distributed optimization of integrated energy system based on cooperative game in uncertain environment[J]. Automation of Electric Power Systems, 2022, 46 (20): 43- 53.
8	李建标, 陈建福, 高滢, 等. 基于RG-DDPG的直流微网能量管理策略[J]. 中国电力, 2023, 56 (7): 85- 94.
	LI Jianbiao, CHEN Jianfu, GAO Ying, et al. Strategy for DC microgrid energy management based on RG-DDPG[J]. Electric Power, 2023, 56 (7): 85- 94.
9	王勇, 吕华灿, 姚文亮, 等. 基于消费者耦合行为的综合需求响应建模与主从博弈运行策略研究[J]. 电网技术, 2024, 48 (7): 2873- 2883.
	WANG Yong, LÜ Huacan, YAO Wenliang, et al. Integrated demand response modeling and master-slave game operation strategy based on consumer coupling behavior[J]. Power System Technology, 2024, 48 (7): 2873- 2883.
10	肖白, 韩康琦, 张晓华. 含氢储能的独立微电网IGDT鲁棒规划[J]. 电力建设, 2024, 45 (4): 77- 88.
	XIAO Bai, HAN Kangqi, ZHANG Xiaohua. Robust IGDT planning for stand-alone microgrid with hydrogen energy storage[J]. Electric Power Construction, 2024, 45 (4): 77- 88.
11	AASLID P, KORPÅS M, BELSNES M M, et al. Stochastic optimization of microgrid operation with renewable generation and energy storages[J]. IEEE Transactions on Sustainable Energy, 2022, 13 (3): 1481- 1491.
12	WANG P B, TAN L L, ZHANG X C, et al. An energy management method for a microgrid group considering uncertainty models[C]//2019 22nd International Conference on Electrical Machines and Systems (ICEMS). Harbin, China. IEEE, 2019: 1–6.
13	李扬, 马文捷, 卜凡金, 等. 多智能体深度强化学习驱动的跨园区能源交互优化调度[J]. 电力建设, 2024, 45 (5): 59- 70.
	LI Yang, MA Wenjie, BU Fanjin, et al. Deep reinforcement learning-driven cross-community energy interaction optimal scheduling[J]. Electric Power Construction, 2024, 45 (5): 59- 70.
14	张宏涛, 吴怡之, 邓开连, 等. 一种基于强化学习的微电网能量管理算法[J]. 物联网技术, 2022, 12 (12): 74- 78.
15	YANG Y H, MA T F, LI H T, et al. Federated double DQN based multi-energy microgrid energy management strategy considering carbon emissions[J]. Global Energy Interconnection, 2023, 6 (6): 689- 699.
16	LIU D, ZANG C Z, ZENG P, et al. Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties[J]. Frontiers in Energy Research, 2023, 11, 1163053.
17	FAN L Q, ZHANG J, HE Y, et al. Optimal scheduling of microgrid based on deep deterministic policy gradient and transfer learning[J]. Energies, 2021, 14 (3): 584.
18	雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38 (5): 1- 11.
	LEI Jiaming, JIANG Aihua, WU Xinfei, et al. Proximal policy optimization dispatch of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38 (5): 1- 11.
19	刘向杰, 刘梓安, 孔小兵, 等. 基于深度Q网络的风光柴储微电网能量管理策略[J]. 控制工程, 2023, 30 (8): 1538- 1547.
	LIU Xiangjie, LIU Zian, KONG Xiaobing, et al. Energy management strategy of wind-PV-diesel-battery microgrid based on deep Q-network[J]. Control Engineering of China, 2023, 30 (8): 1538- 1547.
20	薛溟枫, 毛晓波, 肖浩, 等. 基于改进深度Q网络算法的多园区综合能源系统能量管理方法[J]. 电力建设, 2022, 43 (12): 83- 93.
	XUE Mingfeng, MAO Xiaobo, XIAO Hao, et al. A novel energy management method based on modified deep Q network algorithm for multi-park integrated energy system[J]. Electric Power Construction, 2022, 43 (12): 83- 93.
21	XIAO H, PU X W, PEI W, et al. A novel energy management method for networked multi-energy microgrids based on improved DQN[J]. IEEE Transactions on Smart Grid, 2023, 14 (6): 4912- 4926.
22	XIAO H, YANG Y H, ZHANG S, et al. Stochastic game based microgrid clusters energy management modeling and strategy[C]//2023 3rd Power System and Green Energy Conference (PSGEC). Shanghai, China. IEEE, 2023: 250–254.
23	张冲标, 钱辰雯, 俞红燕, 等. 基于ADMM的多场景县域多微电网交互运行策略[J]. 中国电力, 2024, 57 (2): 9- 18.
	ZHANG Chongbiao, QIAN Chenwen, YU Hongyan, et al. Interactive operation strategy for multi-scenario county-level multi-microgrid based on ADMM[J]. Electric Power, 2024, 57 (2): 9- 18.
24	张晓佳, 王灿, 张佳恒, 等. 基于多能需求响应与改进BiLSTM的综合能源系统负荷预测[J]. 电力建设, 2025, 46 (4): 113- 125.
	ZHANG Xiaojia, WANG Can, ZHANG Jiaheng, et al. Integrated energy system load forecasting based on multi-energy demand response and improved BiLSTM[J]. Electric Power Construction, 2025, 46 (4): 113- 125.
25	崔永玲, 王成福, 牛远方, 等. 考虑综合需求响应不确定性的综合能源系统两阶段随机优化决策[J]. 电网技术, 2025, 49 (6): 2232- 2242.
	CUI Yongling, WANG Chengfu, NIU Yuanfang, et al. Two-stage stochastic optimization decision of integrated energy system considering the uncertainty of integrated demand response[J]. Power System Technology, 2025, 49 (6): 2232- 2242.
26	王灿, 张羽, 田福银, 等. 基于双向主从博弈的储能电站与综合能源系统经济运行策略[J]. 电工技术学报, 2023, 38 (13): 3436- 3446, 3472.
	WANG Can, ZHANG Yu, TIAN Fuyin, et al. Economic operation of energy storage power stations and integrated energy systems based on bidirectional master-slave game[J]. Transactions of China Electrotechnical Society, 2023, 38 (13): 3436- 3446, 3472.
27	姜智霖, 郝峰杰, 袁志昌, 等. 考虑SOC优化设定的电-氢混合储能系统的运行优化[J]. 电力系统保护与控制, 2024, 52 (8): 65- 76.
	JIANG Zhilin, HAO Fengjie, YUAN Zhichang, et al. Optimal operation of an electro-hydrogen hybrid energy storage system considering SOC optimization setting[J]. Power System Protection and Control, 2024, 52 (8): 65- 76.
28	万玲玲, 陈中, 王毅, 等. 考虑能量时空转移的城市规模化共享电动汽车充放电优化调度[J]. 电力建设, 2023, 44 (6): 135- 143.
	WAN Lingling, CHEN Zhong, WANG Yi, et al. Optimal charging and discharging scheduling of urban large-scale shared electric vehicles considering energy temporal and spatial transfer[J]. Electric Power Construction, 2023, 44 (6): 135- 143.
29	吕金玲, 王小君, 窦嘉铭, 等. 考虑运行状态信息的综合能源系统图强化学习优化调度[J]. 电力系统保护与控制, 2024, 52 (2): 1- 14.
	LÜ Jinling, WANG Xiaojun, DOU Jiaming, et al. Optimal dispatch of an integrated energy system based on graph reinforcement learning considering operation state information[J]. Power System Protection and Control, 2024, 52 (2): 1- 14.
30	谢敬东, 徐振邦. 基于三方博弈的多能交互微网群系统双层优化模型[J]. 太阳能学报, 2024, 45 (2): 384- 394.
	XIE Jingdong, XU Zhenbang. Two-layer optimization model for multi-energy interactive microgrid cluster system based on tripartite game[J]. Acta Energiae Solaris Sinica, 2024, 45 (2): 384- 394.

[1]	GUO Qi, CHEN Mengxiao, YU Jiawei, HUANG Yiheng, GUO Haiping. Energy Management Strategy for Distributed Energy Systems Considering the Regulation Capability from Flexible Loads [J]. Electric Power, 2025, 58(8): 60-68.
[2]	Zhiyuan GAO, Weijin ZHUANG, Feng LI, Fang YU, Hong ZHANG, Yan WANG, Min XIA. High Reusability Verification Platform for Artificial Intelligence Applications in Power Grid Dispatching and Control Field [J]. Electric Power, 2025, 58(3): 142-150.
[3]	Pei ZHANG, Majing YANG, Fang ZHANG, Hua XIE, Guangyi LIU, Wenyun LI, Xuegang LU, Zhenyi WANG, Suwei ZHAI. Review of the Application of Graph Database and Graph Computing in Power System Dispatch and Operation [J]. Electric Power, 2025, 58(3): 119-131.
[4]	LIU Hua, XIONG Zaibao, JIANG Taoning, GAO Yu, JIN Yuhan, GE Leijiao. Distributed Reinforcement Learning-Driven Dynamic Energy Optimization Management Strategy for Microgrid Clusters [J]. Electric Power, 2025, 58(10): 50-62.
[5]	LI Jianbiao, CHEN Jianfu, GAO Ying, PEI Xingyu, WU Hongyuan, LU Zikai, ZHOU Shaoxiong, ZENG Jie. Strategy for DC Microgrid Energy Management Based on RG-DDPG [J]. Electric Power, 2023, 56(7): 85-94.
[6]	ZHANG Li, LIU Qinglei, ZHANG Hongwei. Home Load Optimization Scheduling Strategy Based on Improved Binary Particle Swarm Optimization Algorithm [J]. Electric Power, 2023, 56(5): 118-128.
[7]	ZENG Shuang, LIANG Anqi, WANG Liyong, LI Xianglong, MA Lin, WANG Zhe, WANG Linyu, LIU Lan, ZHAO Wei. Multi-agent Peak Shaving and Valley Filling Strategy Considering the Flexibility of Electric-Thermal System with Optical Storage [J]. Electric Power, 2023, 56(2): 133-142.
[8]	Linxinyan LIN, Junpeng ZHU, Yue YUAN. A Distributed Resilience Enhancement Strategy for Multi-microgrids Based on System of Systems Architecture [J]. Electric Power, 2023, 56(12): 87-99.
[9]	WANG Ziqi, ZHANG Huiyuan, XU Jun, CHENG Jiehui. An Energy Management Optimization Strategy for Regional Power Grid Energy Storage System Based on Improved Artificial Bee Colony Algorithm [J]. Electric Power, 2022, 55(9): 16-22,55.
[10]	SHI Xuntao, LEI Jinyong, HUANG Andi, YU Lei, GUO Xiaobin, ZOU Fuqiang, LIU Nian. An Energy Management Algorithm of PV-Assisted Smart Building Based on Offline Optimization and Online Decision [J]. Electric Power, 2019, 52(10): 123-131.
[11]	JI Yu, WANG Shengqiang, CAO Yang, XIE fei, XU Xiaoyi. Research on Comprehensive Control Strategy of Distributed Energy Storage MMC [J]. Electric Power, 2017, 50(10): 159-165.
[12]	FENG Jianghua, ZHAO Xin, SU Zhan，. Research on Energy Management Strategy for the Islanded Mode of Micro-Grid [J]. Electric Power, 2016, 49(8): 81-86.
[13]	CHENG Qiming, CHEN Gen, CHENG Yinman, BAI Yuanfei, LI Ming. The Simulative Research on Energy Management Strategy for Microgrid Structure [J]. Electric Power, 2016, 49(7): 128-134.
[14]	WU Zhi-feng, SHU Jie, ZHANG Xian-yong. Control System of a Stand-Alone Microgrid Based on Multi-Grade Structure Theory [J]. Electric Power, 2012, 45(10): 77-81.