Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning

doi:10.11930/j.issn.1004-9649.202404047

Abstract

Abstract:

To address issues of voltage limit violations and fluctuations caused by the high penetration of distributed photovoltaic (PV) systems in the distribution network, a voltage control method based on multi-agent safe deep reinforcement learning is proposed. The voltage control with PV is modeled as a decentralized partially observable Markov decision process. A safety layer is introduced in the deep policy network for agent design, while the voltage barrier function based on traditional optimization model voltage constraints is used in defining the agent reward function. Testing results on the IEEE 33-bus system demonstrate that the proposed method can generate voltage control strategies that meet safety constraints under high photovoltaic penetration scenarios, and it can be used to assist dispatchers in making real-time decisions online.

Key words: volt-var control, safe deep reinforcement learning, multi-agent

CLC Number:

TM73

Yi ZENG, Yi ZHOU, Jixiang LU, Liangcai ZHOU, Ningkai TANG, Hong LI. Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning[J]. Electric Power, 2025, 58(2): 111-117.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202404047

https://www.electricpower.com.cn/EN/Y2025/V58/I2/111

Figures/Tables 8

Fig.1 The structure of Safe-MADDPG algorithm

Fig.2 Modified IEEE 33-node system

Table 1 Hyperparameter values

超参数		神经网络隐藏层个数		经验回放池容量		批大小		折扣因子		值网络、策略网络学习率		目标网络软更新因子
取值		64		5 000		32		0.99		0.000 1		0.1

Fig.3 Average reward curve of the agent during the training phase

Table 2 Results of test set

控制方法	电压偏差(p.u.)	网络损耗/MW	无功输出/(MV·A)
本文方法	0.014 2	0.069 4	0.103 1
MADDPG	0.014 5	0.066 7	0.122 6
DDPG	0.014 8	0.067 0	0.121 6

Fig.4 Photovoltaic output and load curve on a typical day

Fig.5 Voltage variation at each node on a typical day

Fig.6 Voltage curves of typical bus on a typical summer day

References 26

1	郑国光. 支撑“双碳” 目标实现的问题辨识与关键举措研究[J]. 中国电力, 2023, 56 (11): 1- 8.
	ZHENG Guoguang. Problem identification and key measures to support the achievement of carbon peak and carbon neutrality[J]. Electric Power, 2023, 56 (11): 1- 8.
2	周勤勇, 李根兆, 秦晓辉, 等. 能源革命下的电力系统范式转换分析[J]. 中国电力, 2024, 57 (3): 1- 11.
	ZHOU Qinyong, LI Genzhao, QIN Xiaohui, et al. Analysis of power system paradigm shift under energy revolution[J]. Electric Power, 2024, 57 (3): 1- 11.
3	何奇, 张宇, 邓玲, 等. 基于水电储能调节的风光水发电联合优化调度策略[J]. 广东电力, 2024, 37 (3): 12- 24.
	HE Qi, ZHANG Yu, DENG Ling, et al. Joint optimal scheduling strategy of wind, photovoltaic and water storage power generation considering hydropower storage regulation[J]. Guangdong Electric Power, 2024, 37 (3): 12- 24.
4	陈文进, 杨晓丰, 祁炜雯, 等. 基于原型提取和聚类的光伏电站快速集群划分方法[J]. 浙江电力, 2024, 43 (4): 74- 84.
	CHEN Wenjin, YANG Xiaofeng, QI Weiwen, et al. A method for rapid cluster partitioning of photovoltaic plants based on prototype extraction and clustering[J]. Zhejiang Electric Power, 2024, 43 (4): 74- 84.
5	李翠萍, 朱文超, 李军徽, 等. 分布式电源接入中压配电网的运行方案研究[J]. 东北电力大学学报, 2023, 43 (4): 57- 64.
	LI Cuiping, ZHU Wenchao, LI Junhui, et al. Research on the operation scheme of distributed generation access to medium voltage distribution network[J]. Journal of Northeast Electric Power University, 2023, 43 (4): 57- 64.
6	徐恒辉, 姚杰, 周萍, 等. 基于运行数据的光伏电站状态评估方法研究[J]. 电力科技与环保, 2023, 39 (5): 450- 456.
	XU Henghui, YAO Jie, ZHOU Ping, et al. Research on photovoltaic power plant state evaluation based on operating data[J]. Electric Power Technology and Environmental Protection, 2023, 39 (5): 450- 456.
7	XU Y, DONG Z Y, ZHANG R, et al. Multi-timescale coordinated voltage/var control of high renewable-penetrated distribution systems[J]. IEEE Transactions on Power Systems, 2017, 32 (6): 4398- 4408. DOI
8	ZHANG B S, LAM A Y S, DOMÍNGUEZ-GARCÍA A D, et al. An optimal and distributed method for voltage regulation in power distribution systems[J]. IEEE Transactions on Power Systems, 2015, 30 (4): 1714- 1726. DOI
9	LI J Y, XU Z, ZHAO J, et al. Distributed online voltage control in active distribution networks considering PV curtailment[J]. IEEE Transactions on Industrial Informatics, 2019, 15 (10): 5519- 5530. DOI
10	姜涛, 张东辉, 李雪, 等. 含分布式光伏的主动配电网电压分布式优化控制[J]. 电力自动化设备, 2021, 41 (9): 102- 109, 125.
	JIANG Tao, ZHANG Donghui, LI Xue, et al. Distributed optimal control of voltage in active distribution network with distributed photovoltaic[J]. Electric Power Automation Equipment, 2021, 41 (9): 102- 109, 125.
11	武海涛, 庞春林, 张宁宁. 兼顾提升功率分配精度与抑制电压偏差的自适应下垂控制[J]. 电力系统保护与控制, 2024, 52 (4): 109- 120.
	WU Haitao, PANG Chunlin, ZHANG Ningning. Adaptive sag control with improved power distribution accuracy and voltage deviation suppression[J]. Power System Protection and Control, 2024, 52 (4): 109- 120.
12	巨云涛, 陈希, 李嘉伟, 等. 基于分布式深度强化学习的微网群有功无功协调优化调度[J]. 电力系统自动化, 2023, 47 (1): 115- 125.
	JU Yuntao, CHEN Xi, LI Jiawei, et al. Active and reactive power coordinated optimal dispatch of networked microgrids based on distributed deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47 (1): 115- 125.
13	陈池瑶, 苗世洪, 姚福星, 等. 基于多智能体算法的多微电网-配电网分层协同调度策略[J]. 电力系统自动化, 2023, 47 (10): 57- 65.
	CHEN Chiyao, MIAO Shihong, YAO Fuxing, et al. Hierarchical cooperative dispatching strategy of multi-microgrid and distribution networks based on multi-agent algorithm[J]. Automation of Electric Power Systems, 2023, 47 (10): 57- 65.
14	李超英, 檀勤良. 基于智能体建模的新型电力系统下火电企业市场交易策略[J]. 中国电力, 2024, 57 (2): 212- 225.
	LI Chaoying, TAN Qinliang. Market trading strategy for thermal power enterprise in new power system based on agent modeling[J]. Electric Power, 2024, 57 (2): 212- 225.
15	王蓓蓓, 刘飞宇, 杨朋朋, 等. 考虑轻微利他效用的发售电一体化集团成员合谋行为的多智能体深度双Q网络推演研究[J]. 中国电机工程学报, 2023, 43 (7): 2640- 2652.
	WANG Beibei, LIU Feiyu, YANG Pengpeng, et al. Study on collusion behavior between the members in electricity producers-retailers integration group considering slightly altruistic utility based on multi-agent deep double Q network[J]. Proceedings of the CSEE, 2023, 43 (7): 2640- 2652.
16	杨悦, 王丹, 胡博, 等. 基于改进多智能体Q学习的多源最优联合调频控制策略研究[J]. 电力系统保护与控制, 2022, 50 (7): 135- 144.
	YANG Yue, WANG Dan, HU Bo, et al. Multi-source optimal joint frequency modulation control strategy based on improved multi-agent Q-learning[J]. Power System Protection and Control, 2022, 50 (7): 135- 144.
17	YAN Z M, XU Y. A multi-agent deep reinforcement learning method for cooperative load frequency control of a multi-area power system[J]. IEEE Transactions on Power Systems, 2020, 35 (6): 4599- 4608. DOI
18	WANG J H, XU W K, GU Y J, et al. Multi-agent reinforcement learning for active voltage control on power distribution networks[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM, 2024: 3271–3284.
19	CAO D, HU W H, ZHAO J B, et al. A multi-agent deep reinforcement learning based voltage regulation using coordinated PV inverters[J]. IEEE Transactions on Power Systems, 2020, 35 (5): 4120- 4123. DOI
20	CAO D, ZHAO J B, HU W H, et al. Data-driven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of PVs[J]. IEEE Transactions on Smart Grid, 2021, 12 (5): 4137- 4150. DOI
21	DALAL G, DVIJOTHAM K, VECERIK M, et al. Safe exploration in continuous action spaces[EB/OL]. (2018-01-08)[2024-03-05]. https://arxiv.org/abs/1801.08757v1.
22	MA Q, DENG C H. Simplified deep reinforcement learning based volt-var control of topologically variable power system[J]. Journal of Modern Power Systems and Clean Energy, 2023, 11 (5): 1396- 1404.
23	VU T L, MUKHERJEE S, HUANG R K, et al. Safe reinforcement learning for grid voltage control[EB/OL]. (2021-02-13)[2024-03-17]. https://arxiv.org/abs/2112.01484v1.
24	唐巍, 蔡永翔, 李天锐, 等. 低压配电网消纳分布式光伏的控制策略及性能分析[J]. 分布式能源, 2018, 3 (6): 1- 12.
	TANG Wei, CAI Yongxiang, LI Tianrui, et al. Distributed PV consumption control strategy and performance analysis in low voltage distribution network[J]. Distributed Energy, 2018, 3 (6): 1- 12.
25	WANG S Y, DUAN J J, SHI D, et al. A data-driven multi-agent autonomous voltage control framework using deep reinforcement learning[J]. IEEE Transactions on Power Systems, 2020, 35 (6): 4644- 4654. DOI
26	SHEEBAELHAMD Z, ZISIS K, NISIOTI A, et al. Safe deep reinforcement learning for multi-agent systems with continuous action spaces[C]//International Conference on Machine Learning (ICML). Online: IMLS, 2021.

[1]	FAN Huicong, DUAN Zhiguo, CHEN Zhiyong, ZHU Shijia, LIU Hang, LI Wenxiao, YANG Yang. Two-layer Optimization Scheduling for Off-grid Microgrids Based on Multi-agent Deep Policy Gradient [J]. Electric Power, 2025, 58(5): 11-20, 32.
[2]	ZHOU Feihang, WANG Hao, WANG Haili, WANG Meng, JIN Yaojie, LI Zhongchun, ZHANG Zhongde, WANG Peng. Multi-entity Behaviors in Electricity-Carbon-Green Certificate Coupled Markets Based on Multi-agent Reinforcement Learning [J]. Electric Power, 2025, 58(4): 44-55.
[3]	Jiawu WANG, Dianyun ZHAO, Changfeng LIU, Kang CHEN, Yumin ZHANG. Analytical Target Cascading Based Active Distribution Network Level Multi-agent Autonomous Collaborative Optimization [J]. Electric Power, 2024, 57(7): 214-226.
[4]	Chaoying LI, Qinliang TAN. Market Trading Strategy for Thermal Power Enterprise in New Power System Based on Agent Modeling [J]. Electric Power, 2024, 57(2): 212-225.
[5]	Xingping ZHANG, Teng WANG, Xinyue ZHANG, Haonan ZHANG. Bidding Strategy for Thermal Power Generation Companies Based on Multi-agent Deep Deterministic Policy Gradient Algorithm [J]. Electric Power, 2024, 57(11): 161-172.
[6]	ZENG Shuang, LIANG Anqi, WANG Liyong, LI Xianglong, MA Lin, WANG Zhe, WANG Linyu, LIU Lan, ZHAO Wei. Multi-agent Peak Shaving and Valley Filling Strategy Considering the Flexibility of Electric-Thermal System with Optical Storage [J]. Electric Power, 2023, 56(2): 133-142.
[7]	LI Xianshan, CHEN Aobo, CHENG Shan, CHEN Minrui. Multi-agent Coordination and Optimal Dispatch of Microgrid with CES Based on Ecological Game [J]. Electric Power, 2021, 54(7): 166-177.
[8]	ZHAO Fengxian, WU Jing, SUN Liying. Service Restoration for Large Area Blackout of Smart Distribution System Based on MAPSO [J]. Electric Power, 2016, 49(1): 85-90.
[9]	ZHAI Dang-guo, FU Zhou-xing, YUN Bao-ji, LI Zhong, LI Lei. Study on Hierarchical Regional Protection of Power System Based on MAS Information Fusion [J]. Electric Power, 2014, 47(7): 118-121.
[10]	ZHAO Feng-xian, SUN Li-ying. Study on Regional Protection System of Microgrid Based on Multi-Agent System [J]. Electric Power, 2014, 47(5): 59-63.
[11]	WANG Jing, WANG Zong-li, ZHANG Ying. A Review of the Application of Multi-Agent Coordinated Technology in Microgrids [J]. Electric Power, 2013, 46(10): 67-73.