基于安全强化学习的主动配电网有功-无功协调优化调度

doi:10.11930/j.issn.1004-9649.202311065

中国电力 ›› 2024, Vol. 57 ›› Issue (3): 43-50.DOI: 10.11930/j.issn.1004-9649.202311065

• 数字化技术驱动的新型配电网 • 上一篇下一篇

基于安全强化学习的主动配电网有功-无功协调优化调度

焦昊¹(), 殷岩岩²(), 吴晨³(), 刘建¹, 徐春雷³, 徐贤³, 孙国强²

1. 国网江苏省电力有限公司电力科学研究院，江苏南京　211103
2. 河海大学电气与动力工程学院，江苏南京　211100
3. 国网江苏省电力有限公司，江苏南京　210024

收稿日期:2023-11-15 出版日期:2024-03-28 发布日期:2024-03-26
作者简介:焦昊（1991—），男，工程师，从事人工智能在电网中的应用研究，E-mail：jiaohaopeter@163.com
殷岩岩（1997—），男，通信作者，硕士研究生，从事配电网优化研究，E-mail：1109851959@qq.com
吴晨（1989—），女，高级工程师，从事电网规划、电力市场研究，E-mail：cwusgcc@gmail.com
基金资助:
国家自然科学基金资助项目（U1966205）；国网江苏省电力有限公司科技项目（J2023121）。

Coordinated Optimization of Active and Reactive Power of Active Distribution Network Based on Safety Reinforcement Learning

Hao JIAO¹(), Yanyan YIN²(), Chen WU³(), Jian LIU¹, Chunlei XU³, Xian XU³, Guoqiang SUN²

1. State Grid Jiangsu Electric Power Science Research Institute, Nanjing 211103, China
2. College of Electrical and Power Engineering, Hohai University, Nanjing 211100, China
3. State Grid Jiangsu Electric Power Co., Ltd., Nanjing 210024, China

Received:2023-11-15 Online:2024-03-28 Published:2024-03-26
Supported by:
This work is supported by National Natural Science Foundation of China (No.U1966205) and Science and Technology Project of State Grid Jiangsu Electric Power Co., Ltd. (No.J2023121).

摘要/Abstract

摘要：

提出一种基于离线策略的安全强化学习方法，通过离线训练大量配电网历史运行数据，摆脱了传统优化方法对完备且准确模型的依赖。首先，结合配电网络参数信息，建立了基于约束马尔可夫决策过程的有功无功优化模型；其次，基于原始对偶优化法设计了新型安全强化学习方法，该方法在最大化未来折扣奖励的同时最小化成本函数；最后，在配电系统上进行仿真。仿真结果表明：所提方法能够根据配电网实时观测信息，在线生成满足复杂约束条件且具有经济效益的调度策略。

关键词: 主动配电网, 有功无功协调优化, 安全强化学习

Abstract:

A safe reinforcement learning method based on offline strategies is proposed. Through offline training of a large amount of historical operating data of the distribution network, it gets rid of the traditional optimization method. Dependence on complete and accurate models. First, combined with the distribution network parameter information, an active and reactive power optimization model based on the constrained Markov decision process (CMDP) was established; then, a new safety reinforcement learning method was designed based on the original dual optimization method. The cost function is minimized while maximizing future discount rewards; finally, simulations are performed on power distribution system. The simulation results show that the proposed method can online generate a dispatching strategy that satisfies complex constraints and has economic benefits based on real-time observation information of the distribution network.

Key words: active distribution network, active and reactive power coordination optimization, safety reinforcement learning

焦昊, 殷岩岩, 吴晨, 刘建, 徐春雷, 徐贤, 孙国强. 基于安全强化学习的主动配电网有功-无功协调优化调度[J]. 中国电力, 2024, 57(3): 43-50.

Hao JIAO, Yanyan YIN, Chen WU, Jian LIU, Chunlei XU, Xian XU, Guoqiang SUN. Coordinated Optimization of Active and Reactive Power of Active Distribution Network Based on Safety Reinforcement Learning[J]. Electric Power, 2024, 57(3): 43-50.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202311065

https://www.electricpower.com.cn/CN/Y2024/V57/I3/43

图/表 11

表 1 DESS和MT设备参数

Table 1 DESS and MT equipment parameters

MT 节点	$\bar S_i^{{\text{mt}}}$/ (kV·A)	$ \underline {\varphi} _i^{{\text{mt}}} $	$a_i^{{\text{mt}}}$/ (元·(kW·h)^–1)		$b_i^{{\text{mt}}}$/ (元·(kW·h)^–1)		$c_i^{{\text{mt}}}$/(元·h^–1)
25	825	0.8	0		0.20		0
95	625	0.8	0		0.15		0
115	625	0.8	0		0.18		0

DESS 节点	$\overline E_i^{{\text{dess}}}$/ (kW·h)	$\underline {E} _i^{{\text{dess}}}$/ (kW·h)	$\bar P_{i,{\text{ch}}}^{{\text{dess}}}$/ kW	$\bar P_{i,{\text{dch}}}^{{\text{dess}}}$/ kW		${\eta _{{\text{dess}}}}$	$ \kappa _i^{{\text{dess}}} $/ (元·(kW·h)^–1)
21, 57	2 000	200	500	500		0.98	0.1

表 2 所提方法参数设置

Table 2 Parameter settings of the proposed method

参数		数值
$\gamma $		0.995
Critic网络学习率		0.001
Actor网络学习率		0.000 5
$\lambda $学习率		0.000 1
$\lambda $初始值		0
$\tau $		0.02
$d$		0.1
经验回放池大小		50 000

表 3 不同算法的训练和测试时间

Table 3 Training and testing time of different algorithms

算法	离线训练时间/h	在线测试时间/s
PD-DDPG	12.638	0.223
DDPG（$\rho = 1$）	11.050	0.236
DDPG（$\rho = 10$）	10.626	0.229
DDPG（$\rho = 20$）	10.462	0.232

图 1 不同算法奖励值曲线

Fig.1 Different algorithm reward value curves

图 2 不同算法成本值曲线

Fig.2 Cost value curves of different algorithms

图 3 在线测试成本值分布情况

Fig.3 Online test cost value distribution

图 4 在线测试阶段运行成本

Fig.4 Running costs during online testing phase

图 5 有功调度结果

Fig.5 Active power dispatch results

图 6 无功调度结果

Fig.6 Reactive power scheduling results

图 7 OLTC和SCB档位变化

Fig.7 OLTC and SCB gear change

图 8 不同节点的电压分布

Fig.8 Voltage distribution at different nodes

参考文献 25

1	王鹤, 王钲淇, 韩皓, 等. 使用蒙特卡罗逐时估算模型的住宅配电网光伏准入容量研究[J]. 东北电力大学学报, 2023, 43 (1): 9- 19, 2, 99.
	WANG He, WANG Zhengqi, HAN Hao, et al. Research on photovoltaic hosting capacity of residential distribution network based on Monte Carlo hourly estimation framework[J]. Journal of Northeast Electric Power University, 2023, 43 (1): 9- 19, 2, 99.
2	丁琦欣, 覃洪培, 万灿, 等. 基于机会约束规划的配电网分布式光伏承载能力评估[J]. 东北电力大学学报, 2022, 42 (6): 28- 38.
	DING Qixin, QIN Hongpei, WAN Can, et al. Chance-constrained optimization-based distributed photovoltaic hosting capacity assessment of distribution networks[J]. Journal of Northeast Electric Power University, 2022, 42 (6): 28- 38.
3	杨亘烨, 孙荣富, 丁然, 等. 计及光伏多状态调节能力的配电网多时间尺度电压优化[J]. 中国电力, 2022, 55 (3): 105- 114.
	YANG Genye, SUN Rongfu, DING Ran, et al. Multi-time scale reactive power and voltage optimization of distribution network considering photovoltaic multi state regulation capability[J]. Electric Power, 2022, 55 (3): 105- 114.
4	白晶, 金广厚, 孙鹤林, 等. 高渗透光伏配电网第三方主体调压辅助服务补偿与获取[J]. 中国电力, 2023, 56 (4): 95- 103.
	BAI Jing, JIN Guanghou, SUN Helin, et al. Third-party entity voltage regulation ancillary service compensation and procurement in distribution networks with high-penetration PV[J]. Electric Power, 2023, 56 (4): 95- 103.
5	黄南天, 郭玉, 赵暄远. 计及辐照区间划分的含光伏电源配电网源-荷联合场景生成[J]. 东北电力大学学报, 2023, 43 (5): 78- 84.
	HUANG Nantian, GUO Yu, ZHAO Xuanyuan. Combined source-load scenario generation for PV-containing distribution networks with calculation and irradiation interval classification[J]. Journal of Northeast Electric Power University, 2023, 43 (5): 78- 84.
6	祁晓婧. 计及不确定性的主动配电网有功无功联合优化调度技术研究[D]. 南京: 东南大学, 2019.
	QI Xiaojing. Research on joint optimal dispatch technology of active and reactive power in active distribution networks considering uncertainty[D]. Nanjing: Southeast University, 2019.
7	王耀翔, 戴朝波, 杨志昌, 等. 考虑风电机组无功潜力的风电场无功电压控制策略[J]. 电力系统保护与控制, 2022, 50 (24): 83- 90.
	WANG Yaoxiang, DAI Chaobo, YANG Zhichang, et al. Voltage control strategy for a wind farm considering the reactive capability of DFIGs[J]. Power System Protection and Control, 2022, 50 (24): 83- 90.
8	马君亮, 王智冬, 张述铭. 考虑县域光伏潜力评估的源网荷储协同规划[J]. 东北电力大学学报, 2023, 43 (3): 82- 90.
	MA Junliang, WANG Zhidong, ZHANG Shuming. Collaborative planning of source-grid-load-storage considering County PV potential assessment[J]. Journal of Northeast Electric Power University, 2023, 43 (3): 82- 90.
9	马跃, 孟润泉, 魏斌, 等. 考虑阶梯式碳交易机制的微电网两阶段鲁棒优化调度[J]. 电力系统保护与控制, 2023, 51 (10): 22- 33.
	MA Yue, MENG Runquan, WEI Bin, et al. Two-stage robust optimal scheduling of a microgrid with a stepped carbon trading mechanism[J]. Power System Protection and Control, 2023, 51 (10): 22- 33.
10	孙端航, 李本新. 考虑风电不确定性的电网状态检修策略[J]. 东北电力大学学报, 2023, 43 (4): 65- 73.
	SUN Duanhang, LI Benxin. Condition-based maintenance scheduling for power transmission system considering wind power uncertainty[J]. Journal of Northeast Electric Power University, 2023, 43 (4): 65- 73.
11	朱建昆, 高红均, 贺帅佳, 等. 考虑VSC与光-储-充协同配置的交直流混合配电网规划[J]. 智慧电力, 2023, 51 (11): 7- 14.
	ZHU Jiankun, GAO Hongjun, HE Shuaijia, et al. AC-DC hybrid distribution network planning considering VSC and photovoltaic-storage-charging coordinated configuration[J]. Smart Power, 2023, 51 (11): 7- 14.
12	李笑竹, 王维庆. 基于贝叶斯理论的分布鲁棒优化在储能配置上的应用[J]. 电网技术, 2022, 46 (10): 4001- 4011.
	LI Xiaozhu, WANG Weiqing. Application of distributed robust optimization based on bayesian theory in allocation of energy storage[J]. Power System Technology, 2022, 46 (10): 4001- 4011.
13	徐澄莹, 朱旭, 窦真兰, 等. 基于数据驱动鲁棒优化的用户侧综合能源舱低碳规划[J]. 电力建设, 2022, 43 (12): 27- 36.
	XU Chengying, ZHU Xu, DOU Zhenlan, et al. Research on low carbon planning based on data driven robust optimization for user-side integrated energy module[J]. Electric Power Construction, 2022, 43 (12): 27- 36.
14	KOU P, LIANG D L, WANG C, et al. Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks[J]. Applied Energy, 2020, 264, 114772. DOI
15	CHU Y F, WEI Z N, FANG X C, et al. A multiagent federated reinforcement learning approach for plug-In electric vehicle fleet charging coordination in a residential community[J]. IEEE Access, 2022, 10, 98535- 98548. DOI
16	戴武昌, 刘艾冬, 申鑫, 等. 基于MADDPG算法的家用电动汽车集群充放电行为在线优化[J]. 东北电力大学学报, 2021, 41 (5): 80- 89.
	DAI Wuchang, LIU Aidong, SHEN Xin, et al. Online optimization of charging and discharging behavior of household electric vehicle cluster based on MADDPG algorithm[J]. Journal of Northeast Electric Power University, 2021, 41 (5): 80- 89.
17	HOU S R, SALAZAR E M, VERGARA P P, et al. Performance comparison of deep RL algorithms for energy systems optimal scheduling[C]//2022 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe). Novi Sad, Serbia. IEEE, 2022: 1–6.
18	LIU H T, WU W C. Two-stage deep reinforcement learning for inverter-based volt-VAR control in active distribution networks[J]. IEEE Transactions on Smart Grid, 2021, 12 (3): 2037- 2047. DOI
19	GAO Y Q, WANG W, SHI J, et al. Batch-constrained reinforcement learning for dynamic distribution network reconfiguration[J]. IEEE Transactions on Smart Grid, 2020, 11 (6): 5357- 5369. DOI
20	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]//4th International Conference on Learning Representations. San Juan, 2016.
21	LI H P, HE H B. Learning to operate distribution networks with safe deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2022, 13 (3): 1860- 1872. DOI
22	季颖, 王建辉. 基于深度强化学习的微电网在线优化调度[J]. 控制与决策, 2022, 37 (7): 1675- 1684.
	JI Ying, WANG Jianhui. Microgrid online optimal dispatch based on deep reinforcement learning[J]. Control and Decision-making, 2022, 37 (7): 1675- 1684.
23	兰飞, 林立成, 黎静华. 基于改进变分模态分解和信息融合的故障选线[J]. 东北电力大学学报, 2022, 42 (5): 1- 14.
	LAN Fei, LIN Licheng, LI Jinghua. Fault line selection based on improved variational mode decomposition and information fusion[J]. Journal of Northeast Electric Power University, 2022, 42 (5): 1- 14.
24	DING Y, LAVAEI J. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints[C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 7396–7404.
25	PASZKE A, GROSS S, MASSA F, et al. Pytorch: an imperative style, high-performance deep learning library[J]. Advances in Neural Information Processing Systems, 2019, 32, 8024- 8035.

[1]	凡鹏飞, 李宝琴, 侯江伟, 李嵘, 宋崇明, 林凯骏. 配电网分布式电源经济可承载力评估[J]. 中国电力, 2024, 57(7): 196-202.
[2]	王家武, 赵佃云, 刘长锋, 陈康, 张玉敏. 基于目标级联法的多主体主动配电网自治协同优化[J]. 中国电力, 2024, 57(7): 214-226.
[3]	龚逊东, 郭维嘉, 杨晨, 周力, 董晓峰, 朱俊澎. 基于动态孤岛的主动配电网多阶段故障恢复策略[J]. 中国电力, 2024, 57(1): 158-166.
[4]	岳园园, 王主丁, 王辉, 罗璇, 苏舟, 惠子珈. 考虑设备调节成本的主动配电网日前优化调度[J]. 中国电力, 2023, 56(8): 136-142.
[5]	田宇, 黄婧, 谢枭, 王若昕, 沈丹青, 何丽娜, 杨凯帆, 陈汝科. 主动配电网无功补偿和OLTC鲁棒多目标优化配置[J]. 中国电力, 2023, 56(3): 94-99.
[6]	周蠡, 熊川羽, 柯方超, 李智威, 张洪, 陈然, 刘万方, 喻亚洲, 任丽琴. 基于区块链和禀赋效应的主动配电网经济运行方法[J]. 中国电力, 2023, 56(12): 58-68.
[7]	茹秋实, 米雪峰, 宋志刚, 刘金涛, 雷霞. 考虑储能-智能软开关的主动配电网混合时间尺度鲁棒优化[J]. 中国电力, 2022, 55(9): 129-139.
[8]	张进, 胡存刚, 芮涛. 基于交替方向乘子法的主动配电网日前两阶段分布式优化调度策略[J]. 中国电力, 2021, 54(5): 91-100.
[9]	翁嘉明, 刘东, 安宇, 殷浩洋, 黄植, 秦汉. 馈线功率控制下的主动配电网信息物理风险演化分析[J]. 中国电力, 2021, 54(3): 13-22.
[10]	徐艳春, 赵彩彩, 孙思涵, MI Lu. 基于改进LMD和能量相对熵的主动配电网故障定位方法[J]. 中国电力, 2021, 54(11): 133-143.
[11]	邢海军, 谢宝江, 秦建, 罗扬帆, 娄伟明. 考虑CVR的主动配电网优化运行方法[J]. 中国电力, 2021, 54(11): 76-81.
[12]	赵晋泉, 张强, 方嵩, 彭嵩. 最小化用户停电损失的主动配电网黑启动分区优化策略[J]. 中国电力, 2020, 53(7): 114-121.
[13]	何英静, 沈舒仪, 何东, 张靖瑞, 李帆, 朱克平, 王松岩. 兼顾网损优化效益与投资成本的SNOP容量规划方法[J]. 中国电力, 2020, 53(4): 32-40.
[14]	张伊, 陈启鑫, 夏清, 施磊, 喻乐, 谢旭, 张晶. 基于分布因子法的主动配电网成本分摊方法[J]. 中国电力, 2020, 53(4): 13-21.
[15]	李昱佳, 董川. 基于两种仿生算法的低碳主动配电网规划[J]. 中国电力, 2017, 50(4): 66-70.

基于安全强化学习的主动配电网有功-无功协调优化调度

Coordinated Optimization of Active and Reactive Power of Active Distribution Network Based on Safety Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 25

相关文章 15

编辑推荐

Metrics

模态框（Modal）标题

基于安全强化学习的主动配电网有功-无功协调优化调度

Coordinated Optimization of Active and Reactive Power of Active Distribution Network Based on Safety Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 25

相关文章 15

编辑推荐

Metrics