基于多智能体安全深度强化学习的电压控制

doi:10.11930/j.issn.1004-9649.202404047

中国电力 ›› 2025, Vol. 58 ›› Issue (2): 111-117.DOI: 10.11930/j.issn.1004-9649.202404047

• 基于数据驱动的电力系统安全稳定分析与控制 • 上一篇下一篇

基于多智能体安全深度强化学习的电压控制

曾仪¹(), 周毅²(), 陆继翔¹^,³(), 周良才²(), 唐宁恺¹, 李红¹

1. 国网电力科学研究院有限公司（南瑞集团有限公司），江苏南京　211106
2. 国家电网有限公司华东分部，上海　200120
3. 电网运行风险防御技术与装备全国重点实验室，江苏南京　211106

收稿日期:2024-04-10 出版日期:2025-02-28 发布日期:2025-02-25
作者简介:曾仪（1999—），女，硕士研究生，从事人工智能及电力系统自动化研究，E-mail：eezengyi@foxmail.com
周毅（1982—），男，高级工程师，从事电网调度、电力系统自动化研究，E-mail：joryie@163.com
陆继翔（1973—），男，通信作者，高级工程师（教授级），硕士生导师，从事人工智能及电力系统自动化研究，E-mail：lujixiang@sgepri.sgcc.com.cn
周良才（1984—），男，高级工程师，从事电网调度、电力系统自动化研究，E-mail：liangcaizhou@163.com
基金资助:
国家电网有限公司科技项目（5108-20233058A-1-1-ZN）。

Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning

Yi ZENG¹(), Yi ZHOU²(), Jixiang LU¹^,³(), Liangcai ZHOU²(), Ningkai TANG¹, Hong LI¹

1. State Grid Electric Power Research Institute (NARI Group Corporation), Nanjing 211106, China
2. East China Branch of State Grid Corporation of China, Shanghai 200120, China
3. State Key Laboratory of Technology and Equipment for Defense against Power System Operational Risks, Nanjing 211106, China

Received:2024-04-10 Online:2025-02-28 Published:2025-02-25
Supported by:
This work is supported by Science and Technology Project of SGCC (No.5108-20233058A-1-1-ZN).

摘要/Abstract

摘要：

针对分布式光伏在配电网中的高比例接入带来的电压越限和波动问题，提出了一种基于多智能体安全深度强化学习的电压控制方法。将含光伏的电压控制建模为分布式部分可观马尔可夫决策过程。在深度策略网络中引入安全层进行智能体设计，同时在智能体奖励函数定义时，使用基于传统优化模型电压约束的电压屏障函数。在IEEE 33节点算例上的测试结果表明：所提方法在光伏高渗透率场景下可生成符合安全约束的电压控制策略，可用于在线辅助调度员进行实时决策。

关键词: 无功电压控制, 安全深度强化学习, 多智能体

Abstract:

To address issues of voltage limit violations and fluctuations caused by the high penetration of distributed photovoltaic (PV) systems in the distribution network, a voltage control method based on multi-agent safe deep reinforcement learning is proposed. The voltage control with PV is modeled as a decentralized partially observable Markov decision process. A safety layer is introduced in the deep policy network for agent design, while the voltage barrier function based on traditional optimization model voltage constraints is used in defining the agent reward function. Testing results on the IEEE 33-bus system demonstrate that the proposed method can generate voltage control strategies that meet safety constraints under high photovoltaic penetration scenarios, and it can be used to assist dispatchers in making real-time decisions online.

Key words: volt-var control, safe deep reinforcement learning, multi-agent

中图分类号:

TM73

曾仪, 周毅, 陆继翔, 周良才, 唐宁恺, 李红. 基于多智能体安全深度强化学习的电压控制[J]. 中国电力, 2025, 58(2): 111-117.

Yi ZENG, Yi ZHOU, Jixiang LU, Liangcai ZHOU, Ningkai TANG, Hong LI. Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning[J]. Electric Power, 2025, 58(2): 111-117.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202404047

https://www.electricpower.com.cn/CN/Y2025/V58/I2/111

图/表 8

图 1 Safe-MADDPG算法结构

Fig.1 The structure of Safe-MADDPG algorithm

图 2 改进的IEEE 33节点系统

Fig.2 Modified IEEE 33-node system

表 1 超参数取值

Table 1 Hyperparameter values

超参数		神经网络隐藏层个数		经验回放池容量		批大小		折扣因子		值网络、策略网络学习率		目标网络软更新因子
取值		64		5 000		32		0.99		0.000 1		0.1

图 3 训练阶段智能体平均奖励曲线

Fig.3 Average reward curve of the agent during the training phase

表 2 测试集结果

Table 2 Results of test set

控制方法	电压偏差(p.u.)	网络损耗/MW	无功输出/(MV·A)
本文方法	0.014 2	0.069 4	0.103 1
MADDPG	0.014 5	0.066 7	0.122 6
DDPG	0.014 8	0.067 0	0.121 6

图 4 典型日光伏出力及负荷曲线

Fig.4 Photovoltaic output and load curve on a typical day

图 5 典型日各节点电压变化情况

Fig.5 Voltage variation at each node on a typical day

图 6 夏季典型日代表性节点电压曲线

Fig.6 Voltage curves of typical bus on a typical summer day

参考文献 26

1	郑国光. 支撑“双碳” 目标实现的问题辨识与关键举措研究[J]. 中国电力, 2023, 56 (11): 1- 8.
	ZHENG Guoguang. Problem identification and key measures to support the achievement of carbon peak and carbon neutrality[J]. Electric Power, 2023, 56 (11): 1- 8.
2	周勤勇, 李根兆, 秦晓辉, 等. 能源革命下的电力系统范式转换分析[J]. 中国电力, 2024, 57 (3): 1- 11.
	ZHOU Qinyong, LI Genzhao, QIN Xiaohui, et al. Analysis of power system paradigm shift under energy revolution[J]. Electric Power, 2024, 57 (3): 1- 11.
3	何奇, 张宇, 邓玲, 等. 基于水电储能调节的风光水发电联合优化调度策略[J]. 广东电力, 2024, 37 (3): 12- 24.
	HE Qi, ZHANG Yu, DENG Ling, et al. Joint optimal scheduling strategy of wind, photovoltaic and water storage power generation considering hydropower storage regulation[J]. Guangdong Electric Power, 2024, 37 (3): 12- 24.
4	陈文进, 杨晓丰, 祁炜雯, 等. 基于原型提取和聚类的光伏电站快速集群划分方法[J]. 浙江电力, 2024, 43 (4): 74- 84.
	CHEN Wenjin, YANG Xiaofeng, QI Weiwen, et al. A method for rapid cluster partitioning of photovoltaic plants based on prototype extraction and clustering[J]. Zhejiang Electric Power, 2024, 43 (4): 74- 84.
5	李翠萍, 朱文超, 李军徽, 等. 分布式电源接入中压配电网的运行方案研究[J]. 东北电力大学学报, 2023, 43 (4): 57- 64.
	LI Cuiping, ZHU Wenchao, LI Junhui, et al. Research on the operation scheme of distributed generation access to medium voltage distribution network[J]. Journal of Northeast Electric Power University, 2023, 43 (4): 57- 64.
6	徐恒辉, 姚杰, 周萍, 等. 基于运行数据的光伏电站状态评估方法研究[J]. 电力科技与环保, 2023, 39 (5): 450- 456.
	XU Henghui, YAO Jie, ZHOU Ping, et al. Research on photovoltaic power plant state evaluation based on operating data[J]. Electric Power Technology and Environmental Protection, 2023, 39 (5): 450- 456.
7	XU Y, DONG Z Y, ZHANG R, et al. Multi-timescale coordinated voltage/var control of high renewable-penetrated distribution systems[J]. IEEE Transactions on Power Systems, 2017, 32 (6): 4398- 4408. DOI
8	ZHANG B S, LAM A Y S, DOMÍNGUEZ-GARCÍA A D, et al. An optimal and distributed method for voltage regulation in power distribution systems[J]. IEEE Transactions on Power Systems, 2015, 30 (4): 1714- 1726. DOI
9	LI J Y, XU Z, ZHAO J, et al. Distributed online voltage control in active distribution networks considering PV curtailment[J]. IEEE Transactions on Industrial Informatics, 2019, 15 (10): 5519- 5530. DOI
10	姜涛, 张东辉, 李雪, 等. 含分布式光伏的主动配电网电压分布式优化控制[J]. 电力自动化设备, 2021, 41 (9): 102- 109, 125.
	JIANG Tao, ZHANG Donghui, LI Xue, et al. Distributed optimal control of voltage in active distribution network with distributed photovoltaic[J]. Electric Power Automation Equipment, 2021, 41 (9): 102- 109, 125.
11	武海涛, 庞春林, 张宁宁. 兼顾提升功率分配精度与抑制电压偏差的自适应下垂控制[J]. 电力系统保护与控制, 2024, 52 (4): 109- 120.
	WU Haitao, PANG Chunlin, ZHANG Ningning. Adaptive sag control with improved power distribution accuracy and voltage deviation suppression[J]. Power System Protection and Control, 2024, 52 (4): 109- 120.
12	巨云涛, 陈希, 李嘉伟, 等. 基于分布式深度强化学习的微网群有功无功协调优化调度[J]. 电力系统自动化, 2023, 47 (1): 115- 125.
	JU Yuntao, CHEN Xi, LI Jiawei, et al. Active and reactive power coordinated optimal dispatch of networked microgrids based on distributed deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47 (1): 115- 125.
13	陈池瑶, 苗世洪, 姚福星, 等. 基于多智能体算法的多微电网-配电网分层协同调度策略[J]. 电力系统自动化, 2023, 47 (10): 57- 65.
	CHEN Chiyao, MIAO Shihong, YAO Fuxing, et al. Hierarchical cooperative dispatching strategy of multi-microgrid and distribution networks based on multi-agent algorithm[J]. Automation of Electric Power Systems, 2023, 47 (10): 57- 65.
14	李超英, 檀勤良. 基于智能体建模的新型电力系统下火电企业市场交易策略[J]. 中国电力, 2024, 57 (2): 212- 225.
	LI Chaoying, TAN Qinliang. Market trading strategy for thermal power enterprise in new power system based on agent modeling[J]. Electric Power, 2024, 57 (2): 212- 225.
15	王蓓蓓, 刘飞宇, 杨朋朋, 等. 考虑轻微利他效用的发售电一体化集团成员合谋行为的多智能体深度双Q网络推演研究[J]. 中国电机工程学报, 2023, 43 (7): 2640- 2652.
	WANG Beibei, LIU Feiyu, YANG Pengpeng, et al. Study on collusion behavior between the members in electricity producers-retailers integration group considering slightly altruistic utility based on multi-agent deep double Q network[J]. Proceedings of the CSEE, 2023, 43 (7): 2640- 2652.
16	杨悦, 王丹, 胡博, 等. 基于改进多智能体Q学习的多源最优联合调频控制策略研究[J]. 电力系统保护与控制, 2022, 50 (7): 135- 144.
	YANG Yue, WANG Dan, HU Bo, et al. Multi-source optimal joint frequency modulation control strategy based on improved multi-agent Q-learning[J]. Power System Protection and Control, 2022, 50 (7): 135- 144.
17	YAN Z M, XU Y. A multi-agent deep reinforcement learning method for cooperative load frequency control of a multi-area power system[J]. IEEE Transactions on Power Systems, 2020, 35 (6): 4599- 4608. DOI
18	WANG J H, XU W K, GU Y J, et al. Multi-agent reinforcement learning for active voltage control on power distribution networks[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM, 2024: 3271–3284.
19	CAO D, HU W H, ZHAO J B, et al. A multi-agent deep reinforcement learning based voltage regulation using coordinated PV inverters[J]. IEEE Transactions on Power Systems, 2020, 35 (5): 4120- 4123. DOI
20	CAO D, ZHAO J B, HU W H, et al. Data-driven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of PVs[J]. IEEE Transactions on Smart Grid, 2021, 12 (5): 4137- 4150. DOI
21	DALAL G, DVIJOTHAM K, VECERIK M, et al. Safe exploration in continuous action spaces[EB/OL]. (2018-01-08)[2024-03-05]. https://arxiv.org/abs/1801.08757v1.
22	MA Q, DENG C H. Simplified deep reinforcement learning based volt-var control of topologically variable power system[J]. Journal of Modern Power Systems and Clean Energy, 2023, 11 (5): 1396- 1404.
23	VU T L, MUKHERJEE S, HUANG R K, et al. Safe reinforcement learning for grid voltage control[EB/OL]. (2021-02-13)[2024-03-17]. https://arxiv.org/abs/2112.01484v1.
24	唐巍, 蔡永翔, 李天锐, 等. 低压配电网消纳分布式光伏的控制策略及性能分析[J]. 分布式能源, 2018, 3 (6): 1- 12.
	TANG Wei, CAI Yongxiang, LI Tianrui, et al. Distributed PV consumption control strategy and performance analysis in low voltage distribution network[J]. Distributed Energy, 2018, 3 (6): 1- 12.
25	WANG S Y, DUAN J J, SHI D, et al. A data-driven multi-agent autonomous voltage control framework using deep reinforcement learning[J]. IEEE Transactions on Power Systems, 2020, 35 (6): 4644- 4654. DOI
26	SHEEBAELHAMD Z, ZISIS K, NISIOTI A, et al. Safe deep reinforcement learning for multi-agent systems with continuous action spaces[C]//International Conference on Machine Learning (ICML). Online: IMLS, 2021.

[1]	李超英, 檀勤良. 基于智能体建模的新型电力系统下火电企业市场交易策略[J]. 中国电力, 2024, 57(2): 212-225.
[2]	张兴平, 王腾, 张馨月, 张浩楠. 基于多智能体深度确定策略梯度算法的火力发电商竞价策略[J]. 中国电力, 2024, 57(11): 161-172.
[3]	丁雨, 于艾清, 高纯. 基于改进一致性算法的独立光储直流微电网电压稳定能量协调策略[J]. 中国电力, 2022, 55(3): 74-79.
[4]	李咸善, 陈奥博, 程杉, 陈敏睿. 基于生态博弈的含云储能微电网多智能体协调优化调度[J]. 中国电力, 2021, 54(7): 166-177.
[5]	金子冉, 吴红斌, 周亦尧, 徐斌, 丁津津, 魏薇. 基于电压趋危系数的配电网多时间尺度自适应无功电压控制[J]. 中国电力, 2021, 54(11): 69-75.
[6]	刘京波，崔正湃，吴宇辉，刘辉. 大规模风电汇集系统动态无功补偿装置性能测试及改进措施探讨[J]. 中国电力, 2016, 49(12): 139-143.
[7]	赵凤贤，吴静，孙丽颖. 基于MAPSO优化的智能配电网大面积断电供电恢复[J]. 中国电力, 2016, 49(1): 85-90.

基于多智能体安全深度强化学习的电压控制

Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 26

相关文章 7

编辑推荐

Metrics

模态框（Modal）标题

基于多智能体安全深度强化学习的电压控制

Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 26

相关文章 7

编辑推荐

Metrics