基于深度强化学习的孤岛微电网二次频率控制

doi:10.11930/j.issn.1004-9649.202411069

中国电力 ›› 2025, Vol. 58 ›› Issue (5): 176-188.DOI: 10.11930/j.issn.1004-9649.202411069

基于深度强化学习的孤岛微电网二次频率控制

王力¹^,²(), 蒋宇翔¹(), 曾祥君¹^,²^,³, 赵斌¹^,², 李均昊⁴

1. 长沙理工大学电气与信息工程学院，湖南长沙　410114
2. 长沙理工大学电网防灾减灾全国重点实验室，湖南长沙　410114
3. 湖南理工学院，湖南岳阳　414006
4. 眉山市特种设备监督检验所，四川眉山　620000

收稿日期:2024-11-20 发布日期:2025-05-30 出版日期:2025-05-28
作者简介:
王力（1990），男，通信作者，博士，副教授，从事新型电力系统运行与控制研究，E-mail：wangli@csust.edu.cn
蒋宇翔（1999），男，硕士研究生，从事微电网运行控制研究，E-mail：jyx395215402@163.com
基金资助:
国家自然科学基金资助项目（交直流混联独立微电网动态频率自律控制研究，52107071）；湖南省自然科学基金资助项目（风光水储交直流混联微电网调频特性及频率稳定控制研究，2023JJ40043）；2023年长沙理工大学研究生科研创新项目（新能源发电系统中双模式并网逆变器平滑切换控制，CSLGCX23065）。

Secondary Frequency Control of Islanded Microgrid Based on Deep Reinforcement Learning

WANG Li¹^,²(), JIANG Yuxiang¹(), ZENG Xiangjun¹^,²^,³, ZHAO Bin¹^,², LI Junhao⁴

1. School of Electrical and Information Engineering, Changsha University of Science and Technology, Changsha 410114, China
2. State Key Laboratory of Disaster Prevention & Reduction for Power Grid, Changsha University of Science & Technology, Changsha 410114, China
3. Hunan Institute of Science and Technology, Yueyang 414006, China
4. Meishan City Special Equipment Supervision and Inspection Institute, Meishan 620000, China

Received:2024-11-20 Online:2025-05-30 Published:2025-05-28
Supported by:
This work is supported by National Natural Science Foundation of China (Research on Dynamic Frequency Autonomous Control in AC-DC Hybrid Standalone Microgrids, No.52107071), Hunan Provincial Natural Science Foundation of China (Study on Frequency Regulation Characteristics and Stability Control of Wind-Solar-Hydro-Storage AC-DC Hybrid Microgrids, No.2023JJ40043) and 2023 Graduate Research Innovation Project of Changsha University of Science and Technology (Smooth switching control of dual-mode grid-connected inverter in new energy power generation system, No.CSLGCX23065).

摘要/Abstract

摘要：

随着分布式电源大量接入微电网，可再生能源发电波动性和系统随机扰动给孤岛微电网频率稳定和运行控制带来了严重威胁。为此，提出了基于深度强化学习的二次频率控制方法，分析孤岛微电网下垂控制特性，提出了基于深度Q网络的二次频率控制器结构。将频率偏差作为状态输入变量，依次完成深度Q网络算法中状态空间、动作空间、奖励函数、神经网络和超参数的设计，其中奖励函数兼顾了频率恢复和各分布式电源功率分配的目标，实现各智能体动作选择一致性；通过离线学习训练生成深度强化学习二次频率控制器。在Matlab/Simulink中搭建孤岛微电网仿真模型，设置多场景源荷扰动验证控制器性能。结果表明，与传统PID控制和基于Q学习算法控制器相比，该控制方法能够快速实现更稳定的二次频率控制，并能自适应协调各分布式电源按自身容量进行功率分配，确保系统稳定运行。

关键词: 深度强化学习, 孤岛微电网, 下垂控制, 深度Q网络, 二次频率控制, 功率分配

Abstract:

With the large-scale integration of distributed generation into microgrids, the volatility of renewable energy generation and system random disturbances pose significant threats to the frequency stability and operational control of islanded microgrids. To address this, a secondary frequency control method based on deep reinforcement learning is proposed. The droop control characteristics of islanded microgrids are analyzed, and a secondary frequency controller structure based on deep Q-Networks is presented. The frequency deviation is used as the state input variable, and the design of the state space, action space, reward function, neural network, and hyperparameters in the deep Q-Networks algorithm is carried out. The reward function balances the goals of frequency recovery and power allocation among distributed energy resources , ensuring consistency in action selection among the intelligent agents. An offline learning process is used to train the deep reinforcement learning-based secondary frequency controller. A simulation model of the islanded microgrid is developed in Matlab/Simulink, and multiple disturbance scenarios are tested to validate the controller's performance. The results show that, compared to traditional PID control and Q-Learning-based controllers, the proposed method achieves more stable secondary frequency control and adapts to coordinate the power allocation of distributed generation units according to their capacities, ensuring the stable operation of the system.

Key words: deep reinforcement learning, islanded microgrid, droop control, deep Q-Network, secondary frequency control, power allocation

王力, 蒋宇翔, 曾祥君, 赵斌, 李均昊. 基于深度强化学习的孤岛微电网二次频率控制[J]. 中国电力, 2025, 58(5): 176-188.

WANG Li, JIANG Yuxiang, ZENG Xiangjun, ZHAO Bin, LI Junhao. Secondary Frequency Control of Islanded Microgrid Based on Deep Reinforcement Learning[J]. Electric Power, 2025, 58(5): 176-188.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202411069

https://www.electricpower.com.cn/CN/Y2025/V58/I5/176

图/表 18

图 1 2台DG并联简化

Fig.1 Simplify of two DG in parallel

图 2 $P$-$f$下垂特性曲线

Fig.2 $P$-$f$droop characteristic curve

图 3 马尔可夫决策过程

Fig.3 Markov decision process

图 4 神经网络价值函数近似方法

Fig.4 Value function approximation method of neural network

图 5 DQN算法流程

Fig.5 Flow of DQN algorithm

图 6 基于DQN的孤岛微电网二次频率控制器原理

Fig.6 Principle of secondary frequency controller for islanded microgrid based on DQN

图 7 DQN控制器结构

Fig.7 DQN controller structure

表 1 设置不同权值参数下的收敛效果

Table 1 Convergence performance under different parameter settings

序号	$ {\lambda _{1}} $	$ {\lambda _2} $	$ \lambda _3 $	$ \lambda _4 $	最终奖励值
1	2	3	4	5	9.809956
2	2	4	6	8	9.239820
3	2	5	10	15	8.860610
4	2	8	10	20	8.176960
5	2	10	20	30	6.721230

图 8 神经网络结构

Fig.8 Neural network structure

表 2 不同神经网络结构下的收敛效果

Table 2 Convergence performance under different neural network structures

序号	$M$	$u$	最终奖励值
1	1	32	4.4423
2	1	64	6.9203
3	2	32	8.4634
4	2	64	9.8098
5	3	32	9.3675
6	3	64	8.9325

图 9 孤岛微电网拓扑

Fig.9 Islanded microgrid topology

图 10 光伏拓扑结构

Fig.10 Control strategy of photovoltaic power supply

表 3 微电网系统参数

Table 3 Microgrid system parameters

参数		数值
DG1额定功率 ${P_{{\text{n}}1}}$/kW		10
DG2额定功率 ${P_{{\text{n2}}}}$/kW		8
DG3额定功率 $ {P_{{\text{n3}}}} $/kW		12
DG1下垂系数 ${m_{{\text{DG1}}}}$		3.1416×10^–4
DG2下垂系数 ${m_{{\text{DG2}}}}$		3.9271×10^–4
DG3下垂系数 $ {m_{{\text{DG3}}}} $		2.6181×10^–4
交流母线电压 ${U_{\text{o}}}$/V		380

图 11 DQN离线学习情况

Fig.11 Cumulative reward graph for DQN training

图 12 负荷投切及线路阻抗变化工况下各DG频率偏差

Fig.12 Frequency deviations of each DG under load switching and increased line impedance conditions

图 13 随机负荷扰动时3种控制器性能对比

Fig.13 Performance comparison of three controllers under random load disturbance

图 14 DG2退出运行时3种控制器性能对比

Fig.14 Performance comparison of three controllers when DG2 disconnects from the power grid

图 15 复合扰动下加入控制器前后性能对比

Fig.15 Performance comparison before and after adding controller under compound disturbances

参考文献 33

1	王新宝, 葛景, 韩连山, 等. 构网型储能支撑新型电力系统建设的思考与实践[J]. 电力系统保护与控制, 2023, 51 (5): 172- 179.
	WANG Xinbao, GE Jing, HAN Lianshan, et al. Theory and practice of grid-forming BESS supporting the construction of a new type of power system[J]. Power System Protection and Control, 2023, 51 (5): 172- 179.
2	王大兴, 宁妍, 汪敬培, 等. 构建新型电力系统背景下的微电网鲁棒简化建模[J]. 中国电力, 2024, 57 (1): 148- 157.
	WANG Daxing, NING Yan, WANG Jingpei, et al. Robust simplified modeling of microgrid in the context of constructing new power systems[J]. Electric Power, 2024, 57 (1): 148- 157.
3	王德志, 张孝顺, 刘前进, 等. 基于集成学习的孤岛微电网源—荷协同频率控制[J]. 电力系统自动化, 2018, 42 (10): 46- 52. DOI
	WANG Dezhi, ZHANG Xiaoshun, LIU Qianjin, et al. Ensemble learning for generation-consumption coordinated frequency control in an islanded microgrid[J]. Automation of Electric Power Systems, 2018, 42 (10): 46- 52. DOI
4	张宇, 王洪希, 王璞. 交直流混合微电网互联变流器微分平坦控制[J]. 中国电力, 2022, 55 (7): 102- 109, 120.
	ZHANG Yu, WANG Hongxi, WANG Pu. Flatness-based control of AC/DC hybrid microgrid interconnected converter[J]. Electric Power, 2022, 55 (7): 102- 109, 120.
5	吴忠强, 程洪强. 考虑状态受限和一致性的微电网二次控制[J]. 控制理论与应用, 2024, 41 (1): 183- 188. DOI
	WU Zhongqiang, CHENG Hongqiang. Secondary control for microgrid considering state constraint and consensus[J]. Control Theory & Applications, 2024, 41 (1): 183- 188. DOI
6	董家伟, 王志新, 朱国忠, 等. 孤岛运行下垂逆变器二次调频方法[J]. 电力自动化设备, 2022, 42 (5): 40- 46.
	DONG Jiawei, WANG Zhixin, ZHU Guozhong, et al. Secondary frequency regulation method for droop inverters in island operation[J]. Electric Power Automation Equipment, 2022, 42 (5): 40- 46.
7	李忠文, 程志平, 张书源, 等. 考虑经济调度及电压恢复的直流微电网分布式二次控制[J]. 电工技术学报, 2021, 36 (21): 4482- 4492.
	LI Zhongwen, CHENG Zhiping, ZHANG Shuyuan, et al. Distributed secondary control for economic dispatch and voltage restoration of DC microgrid[J]. Transactions of China Electrotechnical Society, 2021, 36 (21): 4482- 4492.
8	VAISHNAV V, SHARMA D, JAIN A. Quadratic-droop-based distributed secondary control of microgrid with detail-balanced communication topology[J]. IEEE Systems Journal, 2023, 17 (3): 3401- 3412. DOI
9	曹晓, 李泽, 崔国增. 孤岛微电网固定时间分布式鲁棒二次控制[J]. 电力系统保护与控制, 2024, 52 (12): 143- 153.
	CAO Xiao, LI Ze, CUI Guozeng. Distributed robust fixed-time secondary control of islanded microgrids[J]. Power System Protection and Control, 2024, 52 (12): 143- 153.
10	李志军, 苗庆玉, 张家安. 基于虚拟领导节点改进的孤岛微电网完全分布式二次调频策略[J]. 太阳能学报, 2024, 45 (5): 333- 342.
	LI Zhijun, MIAO Qingyu, ZHANG Jia’an. Fully distributed secondary frequency control strategy for island microgrids based on improved virtual leader node[J]. Acta Energiae Solaris Sinica, 2024, 45 (5): 333- 342.
11	孙伟, 方昭, 杨建平, 等. 考虑随机时变延时的孤岛微电网分布式二次控制[J]. 中国电机工程学报, 2022, 42 (3): 864- 875.
	SUN Wei, FANG Zhao, YANG Jianping, et al. Distributed secondary control of islanded microgrid with stochastic time-varying delay[J]. Proceedings of the CSEE, 2022, 42 (3): 864- 875.
12	孙永辉, 孟雲帆, 葛磊蛟, 等. 人工智能赋能微电网运行优化的应用及展望[J]. 高电压技术, 2023, 49 (6): 2239- 2252.
	SUN Yonghui, MENG Yunfan, GE Leijiao, et al. Application and prospect of microgrid operation optimization enabled by artificial intelligence[J]. High Voltage Engineering, 2023, 49 (6): 2239- 2252.
13	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47 (10): 2321- 2340.
	XIONG Luolin, MAO Shuai, TANG Yang, et al. Reinforcement learning based integrated energy system management: a survey[J]. Acta Automatica Sinica, 2021, 47 (10): 2321- 2340.
14	汪超, 赵婵娟, 程志友, 等. 基于协同强化学习的微电网分布式两级电压优化控制[J]. 电力系统保护与控制, 2022, 50 (21): 22- 32.
	WANG Chao, ZHAO Chanjuan, CHENG Zhiyou, et al. Distributed secondary voltage optimization control for a microgrid based on cooperative reinforcement learning[J]. Power System Protection and Control, 2022, 50 (21): 22- 32.
15	LI J, CAI C, SU Q Y. Secondary restoration of islanded alternating current microgrids under a neural inverse optimal control[J]. Engineering Applications of Artificial Intelligence, 2024, 133, 108538. DOI
16	ADIBI M, VAN DER WOUDE J. Secondary frequency control of microgrids: an online reinforcement learning approach[J]. IEEE Transactions on Automatic Control, 2022, 67 (9): 4824- 4831. DOI
17	沈珺, 柳伟, 李虎成, 等. 基于强化学习的多微电网分布式二次优化控制[J]. 电力系统自动化, 2020, 44 (5): 198- 206. DOI
	SHEN Jun, LIU Wei, LI Hucheng, et al. Reinforcement learning based distributed secondary optimal control for multiple microgrids[J]. Automation of Electric Power Systems, 2020, 44 (5): 198- 206. DOI
18	符杨, 郭笑岩, 米阳, 等. 基于强化学习的直流微电网分布式经济下垂控制[J]. 电力自动化设备, 2021, 41 (11): 1- 7.
	FU Yang, GUO Xiaoyan, MI Yang, et al. Distributed economic droop control for DC microgrid based on reinforcement learning[J]. Electric Power Automation Equipment, 2021, 41 (11): 1- 7.
19	江昌旭, 刘晨曦, 林铮, 等. 基于深度强化学习的电力系统暂态稳定控制策略研究综述[J]. 高电压技术, 2023, 49 (12): 5171- 5186.
	JIANG Changxu, LIU Chenxi, LIN Zheng, et al. Review of power system transient stability control strategies based on deep reinforcement learning[J]. High Voltage Engineering, 2023, 49 (12): 5171- 5186.
20	LI J W, CHENG Y Y. Deep meta-reinforcement learning-based data-driven active fault tolerance load frequency control for islanded microgrids considering Internet of Things[J]. IEEE Internet of Things Journal, 2024, 11 (6): 10295- 10303. DOI
21	谢黎龙, 李勇汇, 范培潇, 等. 基于深度强化学习的孤立多微电网系统频率和电压综合控制[J]. 电力自动化设备, 2024, 44 (6): 118- 126.
	XIE Lilong, LI Yonghui, FAN Peixiao, et al. Deep reinforcement learning-based integrated frequency and voltage control for isolated multi-microgrid system[J]. Electric Power Automation Equipment, 2024, 44 (6): 118- 126.
22	LI J W, ZHOU T. Prior knowledge incorporated large-scale multiagent deep reinforcement learning for load frequency control of isolated microgrid considering multi-structure coordination[J]. IEEE Transactions on Industrial Informatics, 2024, 20 (3): 3923- 3934. DOI
23	毕聪博, 唐聿劼, 罗永红, 等. 电力系统优化控制中强化学习方法应用及挑战[J]. 中国电机工程学报, 2024, 44 (1): 1- 22.
	BI Congbo, TANG Yujie, LUO Yonghong, et al. Review on critical problems in reinforcement learning methods applied in power system optimization and control scenarios[J]. Proceedings of the CSEE, 2024, 44 (1): 1- 22.
24	LIN S W, CHU C C, TUNG C F. Data-driven distributed Q-learning droop control for frequency synchronization and voltage restoration in isolated AC micro-grids[J]. IEEE Transactions on Industry Applications, 2023, 59 (6): 7306- 7317. DOI
25	ALABDULLAH M H, ABIDO M A. Microgrid energy management using deep Q-network reinforcement learning[J]. Alexandria Engineering Journal, 2022, 61 (11): 9069- 9078. DOI
26	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44 (10): 3794- 3803.
	LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44 (10): 3794- 3803.
27	胡正伟, 夏思懿, 王文彬, 等. 基于深度强化学习的Π型阻抗匹配网络多参数最优求解方法[J]. 电力系统保护与控制, 2024, 52 (6): 152- 163.
	HU Zhengwei, XIA Siyi, WANG Wenbin, et al. Multi-parameter optimal solution method for Π-type impedance matching networks based on deep reinforcement learning[J]. Power System Protection and Control, 2024, 52 (6): 152- 163.
28	董光德, 李道明, 陈咏涛, 等. 基于粒子群优化与卷积神经网络的电能质量扰动分类方法[J]. 发电技术, 2023, 44 (1): 136- 142.
	DONG Guangde, LI Daoming, CHEN Yongtao, et al. Power quality disturbance classification method based on particle swarm optimization and convolutional neural network[J]. Power Generation Technology, 2023, 44 (1): 136- 142.
29	武同心, 纪鑫, 杨成月, 等. 基于双层注意力机制的电力文本分类模型[J/OL]. 中国电力, 1–8 [2025-05-09]. http://kns.cnki.net/kcms/detail/11.3265.TM.20250425.1604.002.html.
	WU Tongxin, JI Xin, YANG Chengyue, et al. A power text classification model based on deep learning[J/OL]. Electric power, 1–8[2025-05-09]. http://kns.cnki.net/kcms/detail/11.3265.TM.20250425.1604.002.html.
30	付红军, 朱劭璇, 王步华, 等. 基于长短期记忆神经网络的检修态电网低频振荡风险预测方法[J]. 发电技术, 2024, 45 (2): 353- 362.
	FU Hongjun, ZHU Shaoxuan, WANG Buhua, et al. Risk prediction method of low frequency oscillation in maintenance power network based on long short term memory neural network[J]. Power Generation Technology, 2024, 45 (2): 353- 362.
31	李卓, 王胤喆, 叶林, 等. 从感知-预测-优化综述图神经网络在电力系统中的应用[J]. 中国电力, 2024, 57 (12): 2- 16.
	LI Zhuo, WANG Yinzhe, YE Lin, et al. The application of graph neural networks in power systems from perspective of perception-prediction-optimization[J]. Electric Power, 2024, 57 (12): 2- 16.
32	邵宜祥, 刘剑, 胡丽萍, 等. 一种改进组合神经网络的超短期风速预测方法研究[J]. 发电技术, 2024, 45 (2): 323- 330.
	SHAO Yixiang, LIU Jian, HU Liping, et al. Research on an ultra-short-term wind speed prediction method based on improved combined neural networks[J]. Power Generation Technology, 2024, 45 (2): 323- 330.
33	闫志彬, 李立, 阳鹏, 等. 考虑构网型储能支撑能力的微电网优化调度策略[J]. 中国电力, 2025, 58 (2): 103- 110.
	YAN Zhibin, LI Li, YANG Peng, et al. Optimal scheduling strategy for microgrid considering the support capabilities of grid Forming energy storage[J]. Electric Power, 2025, 58 (2): 103- 110.

[1]	樊会丛, 段志国, 陈志永, 朱士加, 刘航, 李文霄, 杨阳. 基于多智能体深度策略梯度的离网型微电网双层优化调度[J]. 中国电力, 2025, 58(5): 11-20, 32.
[2]	王冠朝, 霍雨翀, 李群, 李强. 基于深度强化学习与改进Jensen模型的风电场功率优化[J]. 中国电力, 2025, 58(4): 78-89.
[3]	吴云锐, 张建坡, 田新成, 应恺锋, 陈忠. 计及频率适应和快速稳定特性的多端直流电网改进协调控制策略[J]. 中国电力, 2025, 58(3): 31-42.
[4]	曾仪, 周毅, 陆继翔, 周良才, 唐宁恺, 李红. 基于多智能体安全深度强化学习的电压控制[J]. 中国电力, 2025, 58(2): 111-117.
[5]	赵静波, 李文博, 朱鑫要, 孙庆斌, 郝全睿. 海上风电经柔直送出系统受端扰动自适应控制策略[J]. 中国电力, 2025, 58(1): 26-38.
[6]	么钟然, 孙丽颖. 考虑线路阻抗的分布式储能SOC均衡控制策略[J]. 中国电力, 2024, 57(9): 238-246.
[7]	张静忠, 蒙飞, 孙阳, 于慧霞. 考虑有功不确定性的配电网新能源无功优化控制[J]. 中国电力, 2024, 57(3): 51-59.
[8]	黄堃, 付明, 梁加本. 基于融合专家知识DDPG的孤岛微电网频率调节策略[J]. 中国电力, 2024, 57(2): 194-201.
[9]	夏向阳, 蒋戴宇, 曾小勇, 龚芬, 吴小忠, 华夏, 罗彦鹏, 石超. 基于虚拟振荡器控制的储能变流器控制策略研究[J]. 中国电力, 2024, 57(11): 70-77.
[10]	薛松萍, 高德荃, 赵子岩, 林彧茜, 广泽晶, 张大卫. 基于业务差异化传输需求下的电力通信网路由算法[J]. 中国电力, 2024, 57(11): 183-190.
[11]	穆怀天, 廉洪亮, 刘娟, 李艳琼. 基于数据物理融合驱动配电网三相线性化潮流及线损分析应用[J]. 中国电力, 2024, 57(10): 46-56.
[12]	张超, 赵冬梅, 季宇, 张颖. 基于改进深度Q网络的虚拟电厂实时优化调度[J]. 中国电力, 2024, 57(1): 91-100.
[13]	王激华, 叶夏明, 秦如意, 应芳义, 俞佳捷. 基于指数型下垂控制的氢电混合储能微网协调控制策略研究[J]. 中国电力, 2023, 56(7): 43-53.
[14]	孙佳航, 王小华, 黄景光, 曹豪, 梅诺男, 李浙栋. 基于MPC-VSG的孤岛微电网频率和电压动态稳定控制策略[J]. 中国电力, 2023, 56(6): 51-60,81.
[15]	石文喆, 李冰洁, 尤培培, 张泠. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56(6): 114-122.

基于深度强化学习的孤岛微电网二次频率控制

Secondary Frequency Control of Islanded Microgrid Based on Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 18

参考文献 33

相关文章 15

编辑推荐

Metrics

模态框（Modal）标题

基于深度强化学习的孤岛微电网二次频率控制

Secondary Frequency Control of Islanded Microgrid Based on Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 18

参考文献 33

相关文章 15

编辑推荐

Metrics