基于改进双深度Q网络的微电网群能量管理策略

doi:10.11930/j.issn.1004-9649.202503014

中国电力 ›› 2025, Vol. 58 ›› Issue (10): 14-26.DOI: 10.11930/j.issn.1004-9649.202503014

• “十五五”电力系统源网荷储协同规划运行关键技术 • 上一篇下一篇

基于改进双深度Q网络的微电网群能量管理策略

何锦涛¹^,²(), 王灿¹^,²(), 王明超¹^,²(), 程本涛¹^,², 刘于正¹^,², 常文涵¹^,², 王锐³, 余涵⁴

1. 三峡大学电气与新能源学院，湖北宜昌　443002
2. 湖北省微电网工程技术研究中心（三峡大学），湖北宜昌　443002
3. 武汉长海高新技术有限公司，湖北武汉　430223
4. 湖北华中电力科技开发有限责任公司，湖北武汉　430077

收稿日期:2025-03-07 发布日期:2025-10-23 出版日期:2025-10-28
作者简介:
何锦涛（2001），男，硕士研究生，从事微电网优化运行与控制研究，E-mail：hejintao1017@163.com
王灿（1987），男，副教授，从事综合能源系统优化运行、微电网协调控制与优化运行研究，E-mail：xfcancan@163.com
王明超（2001），男，通信作者，从事微电网优化运行与控制研究，E-mail：2949948561@qq.com
基金资助:
国家自然科学基金资助项目（52107108）。

Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

HE Jintao¹^,²(), WANG Can¹^,²(), WANG Mingchao¹^,²(), CHENG Bentao¹^,², LIU Yuzheng¹^,², CHANG Wenhan¹^,², WANG Rui³, YU Han⁴

1. College of Electrical Engineering and New Energy, China Three Gorges University, Yichang 443002, China
2. Hubei Provincial Engineering Technology Research Center for Microgrid (China Three Gorges University), Yichang 443002, China
3. Wuhan Great Sea Hi-tech Co., Ltd., Wuhan 430223, China
4. State Grid Hubei Central China Technology Development of Electric Power Co., Ltd., Wuhan 430077, China

Received:2025-03-07 Online:2025-10-23 Published:2025-10-28
Supported by:
This work is supported by National Natural Science Foundation of China (No.52107108).

摘要/Abstract

摘要：

针对传统微电网群能量管理方法存在的高估偏差与决策精度不足问题，提出一种基于改进双深度Q网络的能量管理策略。首先，构建基于裁剪双Q值思想的双目标价值网络框架，通过并行计算双价值网络的时序差分（temporal difference，TD）目标值并裁剪高TD目标值，抑制价值函数的高估偏差，提高决策精度。然后，采用动态贪婪策略，基于当前状态计算所有可能动作的值函数，避免频繁选择最大Q值动作，使智能体充分探索动作以防止过早收敛。最后，以包含3个子微网的微电网群进行算例验证。仿真结果表明，相较于基于模型预测控制和传统双深度Q网络的能量管理策略，本文所提方法具有更好的寻优效果和收敛性，同时将系统运行成本分别降低了44.62%和26.39%。

关键词: 微电网群, 能量管理, 改进双深度Q网络, 裁剪双Q值, 贪婪策略

Abstract:

To address the overestimation bias and poor decision accuracy of conventional microgrid cluster energy management methods, an energy management strategy based on improved double deep Q-network is proposed. Firstly, this study constructed a dual-objective value network framework based on clipped double Q-learning, which enhances decision-making precision by suppressing value overestimation bias through parallel computation of temporal difference (TD) targets for dual value networks and clipping high TD target values. And then, a dynamic greedy strategy was adopted to calculate the value function of all possible actions based on the current state, avoiding persistent exploitation of the greedy actions to ensure sufficient exploration and prevent premature convergence of the agent. Finally, a case study of a microgrid cluster with three sub-microgrids was conducted for verification. The simulation results show that compared to the energy management strategies based on model predictive control and conventional double deep Q-network, the proposed method achieves superior optimization performance and convergence characteristics, while reducing system operating costs by 44.62% and 26.39% respectively.

Key words: microgrid cluster, energy management, improved double deep Q-network, clipped double Q values, greedy strategy

中图分类号:

TM73

何锦涛, 王灿, 王明超, 程本涛, 刘于正, 常文涵, 王锐, 余涵. 基于改进双深度Q网络的微电网群能量管理策略[J]. 中国电力, 2025, 58(10): 14-26.

HE Jintao, WANG Can, WANG Mingchao, CHENG Bentao, LIU Yuzheng, CHANG Wenhan, WANG Rui, YU Han. Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network[J]. Electric Power, 2025, 58(10): 14-26.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202503014

https://www.electricpower.com.cn/CN/Y2025/V58/I10/14

图/表 15

图 1 MGC系统结构

Fig.1 MGC system structure

图 2 MGC能量管理训练流程

Fig.2 MGC energy management training process

表 1 MGC 系统参数

Table 1 Parameters of MGC system

MG	电池额定容量/(kW·h)	充放电效率/%	微燃机出力上限/(kW·h)	爬坡速率/ (kW·s^–1)	价格响应负荷/kW
1	600	0.9	600	6	175
2	1 000	0.9	800	6	150
3	800	0.9	400	6	200

图 3 各MG风光出力预测

Fig.3 Forecast of wind power of each MG

图 4 各MG用户电负荷功率

Fig.4 User electrical load power for each MG

表 2 配网购售电电价

Table 2 Distribution network purchase and sale electricity price

时段	购电电价/(元·(kW·h)^–1)	售电电价/(元·(kW·h)^–1)
11:00—16:00 19:00—22:00	1.079	0.845
08:00—11:00 16:00—19:00 22:00—00:00	0.637	0.494
00:00—08:00	0.421	0.322

表 3 MGC模型训练参数

Table 3 MGC model training parameters

超参数		数值
奖励折扣率		0.99
学习率		0.001
目标网络Q网络更新权值的步数C		200
最大探索率$ {\alpha _{\max }} $		0.3
最小探索率$ {\alpha _{\min }} $		0.01

图 5 不同算法训练结果对比

Fig.5 Comparison of training results for different algorithms

图 6 不同离散粒度训练结果对比

Fig.6 Comparison of training results with different discretization granularities

图 7 不同算法真实Q值与估计Q值变化曲线

Fig.7 Change curves of true Q-values and estimated Q-values for different algorithms

图 8 Q值估计偏差绝对值对比

Fig.8 Comparison of the absolute values of Q-value estimation bias

表 4 不同参数下的改进DDQN算法性能对比

Table 4 Performance comparison of improved DDQN algorithm with different parameters

参数设置		平均奖励收敛值	收敛轮数	后50%训练周期方差$\sigma $值
$ {\alpha _{\max }} $	$ {\alpha _{\min }} $	平均奖励收敛值	收敛轮数	后50%训练周期方差$\sigma $值
0.1	0.01	–1153.2	843	32.74
0.2	0.01	–975.6	693	26.17
0.3	0.01	–813.7	540	12.49
0.4	0.01	–891.1	652	19.21
0.3	0	–873.9	581	15.76
0.3	0.10	–858.4	603	46.59

图 9 基于改进DDQN算法的MGC能量管理结果

Fig.9 MGC energy management results based on improved DDQN algorithm

图 10 MG储能充放电功率与SOC状态变化

Fig.10 MG energy storage charging and discharging power and SOC state change

图 11 各方法的效益对比

Fig.11 Benefit comparison for different methods

参考文献 30

1	王灿, 张雪菲, 凌凯, 等. 基于区间概率不确定集的微电网两阶段自适应鲁棒优化调度[J]. 中国电机工程学报, 2024, 44 (5): 1750- 1764.
	WANG Can, ZHANG Xuefei, LING Kai, et al. Two-stage adaptive robust optimal scheduling based on the interval probability uncertainty set for microgrids[J]. Proceedings of the CSEE, 2024, 44 (5): 1750- 1764.
2	刘任, 刘洋, 许立雄, 等. 计及分布式需求响应的多微电网系统协同优化策略[J]. 电力建设, 2023, 44 (5): 72- 83.
	LIU Ren, LIU Yang, XU Lixiong, et al. Multi-microgrid system collaborative optimization strategy considering distributed demand response[J]. Electric Power Construction, 2023, 44 (5): 72- 83.
3	易文飞, 朱卫平, 郑明忠. 计及数据中心和风电不确定性的微电网经济调度[J]. 中国电力, 2024, 57 (2): 19- 26.
	YI Wenfei, ZHU Weiping, ZHENG Mingzhong. Economic dispatch of microgrid considering data center and wind power uncertainty[J]. Electric Power, 2024, 57 (2): 19- 26.
4	谭玲玲, 汤伟, 楚冬青, 等. 考虑电-氢一体化的微电网低碳-经济协同优化调度[J]. 中国电力, 2024, 57 (5): 137- 148.
	TAN Lingling, TANG Wei, CHU Dongqing, et al. Low-carbon-economic collaborative optimal dispatching of microgrid considering electricity-hydrogen integration[J]. Electric Power, 2024, 57 (5): 137- 148.
5	樊晓伟, 王瑞妙, 杨海峰, 等. 计及源荷不确定的综合能源微电网集群优化运行[J]. 电力建设, 2024, 45 (8): 128- 139.
	FAN Xiaowei, WANG Ruimiao, YANG Haifeng, et al. Optimization operation of integrated energy microgrid cluster considering source-load uncertainty[J]. Electric Power Construction, 2024, 45 (8): 128- 139.
6	凌凯, 王灿, 张高瑞, 等. 考虑路径损耗的热电联供型微网三层能量优化策略[J]. 中国电力, 2023, 56 (2): 102- 113.
	LING Kai, WANG Can, ZHANG Gaorui, et al. Three-layer energy optimization strategy for CHP microgrid considering path loss[J]. Electric Power, 2023, 56 (2): 102- 113.
7	姚文亮, 王成福, 赵雨菲, 等. 不确定性环境下基于合作博弈的综合能源系统分布式优化[J]. 电力系统自动化, 2022, 46 (20): 43- 53.
	YAO Wenliang, WANG Chengfu, ZHAO Yufei, et al. Distributed optimization of integrated energy system based on cooperative game in uncertain environment[J]. Automation of Electric Power Systems, 2022, 46 (20): 43- 53.
8	李建标, 陈建福, 高滢, 等. 基于RG-DDPG的直流微网能量管理策略[J]. 中国电力, 2023, 56 (7): 85- 94.
	LI Jianbiao, CHEN Jianfu, GAO Ying, et al. Strategy for DC microgrid energy management based on RG-DDPG[J]. Electric Power, 2023, 56 (7): 85- 94.
9	王勇, 吕华灿, 姚文亮, 等. 基于消费者耦合行为的综合需求响应建模与主从博弈运行策略研究[J]. 电网技术, 2024, 48 (7): 2873- 2883.
	WANG Yong, LÜ Huacan, YAO Wenliang, et al. Integrated demand response modeling and master-slave game operation strategy based on consumer coupling behavior[J]. Power System Technology, 2024, 48 (7): 2873- 2883.
10	肖白, 韩康琦, 张晓华. 含氢储能的独立微电网IGDT鲁棒规划[J]. 电力建设, 2024, 45 (4): 77- 88.
	XIAO Bai, HAN Kangqi, ZHANG Xiaohua. Robust IGDT planning for stand-alone microgrid with hydrogen energy storage[J]. Electric Power Construction, 2024, 45 (4): 77- 88.
11	AASLID P, KORPÅS M, BELSNES M M, et al. Stochastic optimization of microgrid operation with renewable generation and energy storages[J]. IEEE Transactions on Sustainable Energy, 2022, 13 (3): 1481- 1491. DOI
12	WANG P B, TAN L L, ZHANG X C, et al. An energy management method for a microgrid group considering uncertainty models[C]//2019 22nd International Conference on Electrical Machines and Systems (ICEMS). Harbin, China. IEEE, 2019: 1–6.
13	李扬, 马文捷, 卜凡金, 等. 多智能体深度强化学习驱动的跨园区能源交互优化调度[J]. 电力建设, 2024, 45 (5): 59- 70. DOI
	LI Yang, MA Wenjie, BU Fanjin, et al. Deep reinforcement learning-driven cross-community energy interaction optimal scheduling[J]. Electric Power Construction, 2024, 45 (5): 59- 70. DOI
14	张宏涛, 吴怡之, 邓开连, 等. 一种基于强化学习的微电网能量管理算法[J]. 物联网技术, 2022, 12 (12): 74- 78.
15	YANG Y H, MA T F, LI H T, et al. Federated double DQN based multi-energy microgrid energy management strategy considering carbon emissions[J]. Global Energy Interconnection, 2023, 6 (6): 689- 699. DOI
16	LIU D, ZANG C Z, ZENG P, et al. Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties[J]. Frontiers in Energy Research, 2023, 11, 1163053. DOI
17	FAN L Q, ZHANG J, HE Y, et al. Optimal scheduling of microgrid based on deep deterministic policy gradient and transfer learning[J]. Energies, 2021, 14 (3): 584. DOI
18	雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38 (5): 1- 11.
	LEI Jiaming, JIANG Aihua, WU Xinfei, et al. Proximal policy optimization dispatch of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38 (5): 1- 11.
19	刘向杰, 刘梓安, 孔小兵, 等. 基于深度Q网络的风光柴储微电网能量管理策略[J]. 控制工程, 2023, 30 (8): 1538- 1547.
	LIU Xiangjie, LIU Zian, KONG Xiaobing, et al. Energy management strategy of wind-PV-diesel-battery microgrid based on deep Q-network[J]. Control Engineering of China, 2023, 30 (8): 1538- 1547.
20	薛溟枫, 毛晓波, 肖浩, 等. 基于改进深度Q网络算法的多园区综合能源系统能量管理方法[J]. 电力建设, 2022, 43 (12): 83- 93.
	XUE Mingfeng, MAO Xiaobo, XIAO Hao, et al. A novel energy management method based on modified deep Q network algorithm for multi-park integrated energy system[J]. Electric Power Construction, 2022, 43 (12): 83- 93.
21	XIAO H, PU X W, PEI W, et al. A novel energy management method for networked multi-energy microgrids based on improved DQN[J]. IEEE Transactions on Smart Grid, 2023, 14 (6): 4912- 4926. DOI
22	XIAO H, YANG Y H, ZHANG S, et al. Stochastic game based microgrid clusters energy management modeling and strategy[C]//2023 3rd Power System and Green Energy Conference (PSGEC). Shanghai, China. IEEE, 2023: 250–254.
23	张冲标, 钱辰雯, 俞红燕, 等. 基于ADMM的多场景县域多微电网交互运行策略[J]. 中国电力, 2024, 57 (2): 9- 18.
	ZHANG Chongbiao, QIAN Chenwen, YU Hongyan, et al. Interactive operation strategy for multi-scenario county-level multi-microgrid based on ADMM[J]. Electric Power, 2024, 57 (2): 9- 18.
24	张晓佳, 王灿, 张佳恒, 等. 基于多能需求响应与改进BiLSTM的综合能源系统负荷预测[J]. 电力建设, 2025, 46 (4): 113- 125.
	ZHANG Xiaojia, WANG Can, ZHANG Jiaheng, et al. Integrated energy system load forecasting based on multi-energy demand response and improved BiLSTM[J]. Electric Power Construction, 2025, 46 (4): 113- 125.
25	崔永玲, 王成福, 牛远方, 等. 考虑综合需求响应不确定性的综合能源系统两阶段随机优化决策[J]. 电网技术, 2025, 49 (6): 2232- 2242.
	CUI Yongling, WANG Chengfu, NIU Yuanfang, et al. Two-stage stochastic optimization decision of integrated energy system considering the uncertainty of integrated demand response[J]. Power System Technology, 2025, 49 (6): 2232- 2242.
26	王灿, 张羽, 田福银, 等. 基于双向主从博弈的储能电站与综合能源系统经济运行策略[J]. 电工技术学报, 2023, 38 (13): 3436- 3446, 3472.
	WANG Can, ZHANG Yu, TIAN Fuyin, et al. Economic operation of energy storage power stations and integrated energy systems based on bidirectional master-slave game[J]. Transactions of China Electrotechnical Society, 2023, 38 (13): 3436- 3446, 3472.
27	姜智霖, 郝峰杰, 袁志昌, 等. 考虑SOC优化设定的电-氢混合储能系统的运行优化[J]. 电力系统保护与控制, 2024, 52 (8): 65- 76.
	JIANG Zhilin, HAO Fengjie, YUAN Zhichang, et al. Optimal operation of an electro-hydrogen hybrid energy storage system considering SOC optimization setting[J]. Power System Protection and Control, 2024, 52 (8): 65- 76.
28	万玲玲, 陈中, 王毅, 等. 考虑能量时空转移的城市规模化共享电动汽车充放电优化调度[J]. 电力建设, 2023, 44 (6): 135- 143. DOI
	WAN Lingling, CHEN Zhong, WANG Yi, et al. Optimal charging and discharging scheduling of urban large-scale shared electric vehicles considering energy temporal and spatial transfer[J]. Electric Power Construction, 2023, 44 (6): 135- 143. DOI
29	吕金玲, 王小君, 窦嘉铭, 等. 考虑运行状态信息的综合能源系统图强化学习优化调度[J]. 电力系统保护与控制, 2024, 52 (2): 1- 14.
	LÜ Jinling, WANG Xiaojun, DOU Jiaming, et al. Optimal dispatch of an integrated energy system based on graph reinforcement learning considering operation state information[J]. Power System Protection and Control, 2024, 52 (2): 1- 14.
30	谢敬东, 徐振邦. 基于三方博弈的多能交互微网群系统双层优化模型[J]. 太阳能学报, 2024, 45 (2): 384- 394.
	XIE Jingdong, XU Zhenbang. Two-layer optimization model for multi-energy interactive microgrid cluster system based on tripartite game[J]. Acta Energiae Solaris Sinica, 2024, 45 (2): 384- 394.

[1]	郭琦, 陈孟晓, 余佳微, 黄以恒, 郭海平. 考虑负荷灵活调节潜力的分布式能源系统能量管理策略[J]. 中国电力, 2025, 58(8): 60-68.
[2]	高志远, 庄卫金, 李峰, 于芳, 张鸿, 王艳, 夏旻. 调控领域人工智能应用的高复用性验证平台[J]. 中国电力, 2025, 58(3): 142-150.
[3]	张沛, 杨马婧, 张放, 谢桦, 刘广一, 李文云, 路学刚, 王珍意, 翟苏巍. 图数据库与图计算在电力系统调度运行应用综述[J]. 中国电力, 2025, 58(3): 119-131.
[4]	柳华, 熊再豹, 蒋陶宁, 高宇, 金雨含, 葛磊蛟. 分布式强化学习驱动的微电网群动态能量优化管理策略[J]. 中国电力, 2025, 58(10): 50-62.
[5]	李建标, 陈建福, 高滢, 裴星宇, 吴宏远, 陆子凯, 周少雄, 曾杰. 基于RG-DDPG的直流微网能量管理策略[J]. 中国电力, 2023, 56(7): 85-94.
[6]	张丽, 刘青雷, 张宏伟. 基于改进二进制粒子群算法的家庭负荷优化调度策略[J]. 中国电力, 2023, 56(5): 118-128.
[7]	曾爽, 梁安琪, 王立永, 李香龙, 马麟, 王喆, 王林钰, 刘澜, 赵伟. 考虑光储型电热协同系统灵活性的多代理削峰填谷策略[J]. 中国电力, 2023, 56(2): 133-142.
[8]	林林馨妍, 朱俊澎, 袁越. 体系架构下的多微电网分布式韧性增强策略[J]. 中国电力, 2023, 56(12): 87-99.
[9]	王子琪, 张慧媛, 许军, 程杰慧. 基于改进人工蜂群算法的区域电网储能系统能量管理优化策略[J]. 中国电力, 2022, 55(9): 16-22,55.
[10]	史训涛, 雷金勇, 黄安迪, 喻磊, 郭晓斌, 邹福强, 刘念. 基于离线优化和在线决策的光伏智能楼宇能量管理算法[J]. 中国电力, 2019, 52(10): 123-131.
[11]	吉宇, 王生强, 曹炀, 谢飞, 徐晓轶. 分散式储能MMC综合控制策略研究[J]. 中国电力, 2017, 50(10): 159-165.
[12]	冯江华，赵新，苏展，. 独立运行微网系统的能量管理策略研究[J]. 中国电力, 2016, 49(8): 81-86.
[13]	程启明, 陈根, 程尹曼, 白园飞, 李明. 一种微网结构的能量管理策略仿真研究[J]. 中国电力, 2016, 49(7): 128-134.
[14]	吴志锋, 舒杰, 张先勇. 基于多级结构原理的独立微电网控制系统[J]. 中国电力, 2012, 45(10): 77-81.

基于改进双深度Q网络的微电网群能量管理策略

Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 30

相关文章 14

编辑推荐

Metrics

模态框（Modal）标题

基于改进双深度Q网络的微电网群能量管理策略

Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 30

相关文章 14

编辑推荐

Metrics