基于深度强化学习的海上风电集群自进化功率平滑控制方法

doi:10.11930/j.issn.1004-9649.202206099

中国电力 ›› 2023, Vol. 56 ›› Issue (3): 36-46.DOI: 10.11930/j.issn.1004-9649.202206099

• 双碳目标下可再生能源新型利用及关键技术 • 上一篇下一篇

基于深度强化学习的海上风电集群自进化功率平滑控制方法

宋伟业¹, 刘灵玥², 阎洁¹, 王航宇¹, 何书凯¹, 韩爽¹, 王明辉³, 刘永前¹

1. 新能源电力系统国家重点实验室（华北电力大学），北京 102206;
2. 南京工程学院，江苏南京 211167;
3. 江苏金风软件技术有限公司，江苏无锡 21400

收稿日期:2022-06-13 修回日期:2022-12-18 出版日期:2023-03-28 发布日期:2023-03-28
作者简介:宋伟业(1999-),男,硕士研究生,从事风电场控制、强化学习算法研究,E-mail:120222111005@ncepu.edu.cn;阎洁(1987-),女,通信作者,博士,副教授,从事新能源场站功率预测与控制研究,E-mail:yanjie@ncepu.edu.cn;王航宇(1996-),男,硕士研究生,从事风电场控制、强化学习算法研究,E-mail:hangyuw@ncepu.edu.cn
基金资助:
国家重点研发计划资助项目（2019YFE0104800）；中国科协青年人才托举工程项目（2019QNRC001）；华能集团总部科技项目“海上风电与智慧能源系统科技专项（一期）”(HNKJ20-H88)。

Self-evolving Power Smooth Control Method for Offshore Wind Power Cluster Based On Deep Reinforcement Learning

SONG Weiye¹, LIU Lingyue², YAN Jie¹, WANG Hangyu¹, HE Shukai¹, HAN Shuang¹, WANG Minghui³, LIU Yongqian¹

1. State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources, North China Electric Power University, Beijing 102206, China;
2. Nanjing Institute of Technology, Nanjing 211167, China;
3. Jiangsu Goldwind Software Technology Co., Ltd., Wuxi 214000, China

Received:2022-06-13 Revised:2022-12-18 Online:2023-03-28 Published:2023-03-28
Supported by:
This work is supported by the National Key Research and Development Program of China (No.2019YFE0104800) , the Young Elite Scientists Sponsorship Program by CAST (No.2019QNRC001) and Offshore Wind Power and Intelligent Energy System Project (Phase I) of Huaneng Group (No.HNKJ20-H88).

摘要/Abstract

摘要： 海上风电集群的风速时空相关性强，加剧了整体有功输出的波动幅度，大规模并网对电力系统的影响更为突出，海上风电集群有功输出平滑控制是解决上述问题的关键手段。传统方法的优化效率低、难以支撑高频率控制，且对预测误差、执行偏差灵敏度过高。因此，提出了“策略离线训练、在线快速寻优、控制效果自进化”的控制架构，建立了深度强化学习的海上风电集群有功输出平滑控制模型。首先，提出了面向集群功率平滑控制的短期收益函数，基于马尔科夫决策过程模型求解最优指令；其次，提出了面向功率策略校准的长期收益Policy函数，根据历史反馈数据有效矫正控制偏差；最后，建立了智能体状态、控制收益和控制决策之间映射的深度神经网络模型，实现基于深度确定性策略梯度算法的智能体训练与求解。算例结果表明：在平均风速为7.5m/s的给定风况下，所提方法能够降低功率波动幅度达20%，同时将发电量损失控制在5%以内。

关键词: 风电集群, 有功功率控制, 深度强化学习, 波动平抑, 控制偏差修正

Abstract: The offshore wind speed has a high spatial-temporal correlation, which aggravates the power fluctuation of the whole wind farm and poses significant challenges to the power system, especially when large-scale offshore wind power is integrated. Smoothing control of large-scale offshore wind power clusters is a key solution to mitigating the above problems. However, most existing methods are inefficient and difficult to support higher frequency control and are susceptible to wind power forecast errors and the deviation of actual action from the optimal control command. Therefore, this paper proposes a new control framework for “offline-training, online-optimization and self-evolution”, and establishes a deep-reinforcement-learning-based model for the smoothing control of the active power of offshore wind power clusters. Firstly, a short-term revenue function for cluster power smoothing control is proposed to solve the optimal command based on the Markov decision process model. Secondly, a long-term revenue policy function for power policy calibration is proposed to effectively correct the control deviation according to the historical feedback data. Finally, a deep neural network model is established for the mapping between the state of the agent, the control benefit and the control decision to realize the training and solution of the agent based on the deep deterministic policy gradient algorithm. The results show that the proposed method can reduce the power fluctuation by 20% and control the power loss within 5% under the given wind condition of 7.5 m/s average wind speed.

Key words: wind power cluster, active power control, deep reinforcement learning, fluctuation smoothing, control deviation correction

宋伟业, 刘灵玥, 阎洁, 王航宇, 何书凯, 韩爽, 王明辉, 刘永前. 基于深度强化学习的海上风电集群自进化功率平滑控制方法[J]. 中国电力, 2023, 56(3): 36-46.

SONG Weiye, LIU Lingyue, YAN Jie, WANG Hangyu, HE Shukai, HAN Shuang, WANG Minghui, LIU Yongqian. Self-evolving Power Smooth Control Method for Offshore Wind Power Cluster Based On Deep Reinforcement Learning[J]. Electric Power, 2023, 56(3): 36-46.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202206099

https://www.electricpower.com.cn/CN/Y2023/V56/I3/36

参考文献

[1] 时智勇, 王彩霞, 李琼慧. “十四五”中国海上风电发展关键问题[J]. 中国电力, 2020, 53(7): 8–17
SHI Zhiyong, WANG Caixia, LI Qionghui. Key issues of China's offshore wind power development in the “14 th five-year plan”[J]. Electric Power, 2020, 53(7): 8–17
[2] 房方, 梁栋炀, 刘亚娟, 等. 海上风电智能控制与运维关键技术[J]. 发电技术, 2022, 43(2): 175–185
FANG Fang, LIANG Dongyang, LIU Yajuan, et al. Key technologies for intelligent control and operation and maintenance of offshore wind power[J]. Power Generation Technology, 2022, 43(2): 175–185
[3] 林俐, 朱晨宸, 郑太一, 等. 风电集群有功功率控制及其策略[J]. 电力系统自动化, 2014, 38(14): 9–16
LIN Li, ZHU Chenchen, ZHENG Taiyi, et al. Active power control of wind farm cluster and its strategy[J]. Automation of Electric Power Systems, 2014, 38(14): 9–16
[4] 鲁宗相, 叶一达, 郭莉, 等. 电力电子化电力系统的调频挑战与多层级协调控制框架[J]. 中国电力, 2019, 52(4): 8–17,110
LU Zongxiang, YE Yida, GUO Li, et al. Frequency regulation challenge of power electronics dominated power systems and its new multi-level coordinated control framework[J]. Electric Power, 2019, 52(4): 8–17,110
[5] 樊新东, 杨秀媛, 金鑫城, 等. 风电场有功功率控制综述[J]. 发电技术, 2018, 39(3): 268–276
FAN Xindong, YANG Xiuyuan, JIN Xincheng. An overview of active power control in wind farms[J]. Power Generation Technology, 2018, 39(3): 268–276
[6] 吴涛, 刘立红, 王岱岚, 等. 风电场智能化远程集控系统设计[J]. 中国电力, 2018, 51(4): 161–167
WU Tao, LIU Lihong, WANG Dailan. Design of remote centralized intelligent monitoring system for wind farms[J]. Electric Power, 2018, 51(4): 161–167
[7] YE L, ZHANG C H, TANG Y, et al. Hierarchical model predictive control strategy based on dynamic active power dispatch for wind power cluster integration[J]. IEEE Transactions on Power Systems, 2019, 34(6): 4617–4629.
[8] 施贵荣, 孙荣富, 徐海翔, 等. 大规模集群可再生能源有功分层协调控制策略[J]. 电网技术, 2018, 42(7): 2160–2167
SHI Guirong, SUN Rongfu, XU Haixiang, et al. Active power stratification coordination control strategy for large-scale cluster of renewable energy[J]. Power System Technology, 2018, 42(7): 2160–2167
[9] MA S K, GENG H, YANG G, et al. Clustering-based coordinated control of large-scale wind farm for power system frequency support[J]. IEEE Transactions on Sustainable Energy, 2018, 9(4): 1555–1564.
[10] KHALID M, SAVKIN A V. A model predictive control approach to the problem of wind power smoothing with controlled battery storage[J]. Renewable Energy, 2009, 35(7): 1520–1526.
[11] 鹿婷, 贾继超, 彭晓涛. 一种考虑经济调度的风电场储能控制策略[J]. 分布式能源, 2019, 4(3): 40–49
LU Ting, JIA Jichao, PENG Xiaotao. An energy storage control strategy for wind farm considering economic dispatching[J]. Distributed Energy, 2019, 4(3): 40–49
[12] 席裕庚, 李德伟, 林姝. 模型预测控制-现状与挑战[J]. 自动化学报, 2013, 39(3): 222–236
XI Yugeng, LI Dewei, LIN Shu. Model predictive control-status and challenges[J]. Acta Automatica Sinica, 2013, 39(3): 222–236
[13] GAO X D, MENG K, DONG Z Y, et al. Cooperation-driven distributed control scheme for large-scale wind farm active power regulation[J]. IEEE Transactions on Energy Conversion, 2017, 32(3): 1240–1250.
[14] 武晗, 鲁宗相, 白恺, 等. 风光储联合电站有功控制的响应迟滞“拖尾现象”分析与改进[J]. 电网技术, 2017, 41(4): 1068–1076
WU Han, LU Zongxiang, BAI Kai, et al. Analysis and optimization of ‘tailing phenomenon’ in active power control of hybrid wind-PV-ES power system[J]. Power System Technology, 2017, 41(4): 1068–1076
[15] 黄鹏翔, 周云海, 徐飞, 等. 基于灵活性裕度的含风电电力系统源荷储协调滚动调度[J]. 中国电力, 2020, 53(11): 78–88
HUANG Pengxiang, ZHOU Yunhai, XU Fei, et al. Source-load-storage coordinated rolling dispatch for wind power integrated power system based on flexibility margin[J]. Electric Power, 2020, 53(11): 78–88
[16] 仲悟之, 李梓锋, 肖洋, 等. 高渗透联网风电集群有功分层递阶控制策略[J]. 电网技术, 2018, 42(6): 1868–1875
ZHONG Wuzhi, LI Zifeng, XIAO Yang, et al. Active hierarchical progressive control strategy of highly penetrated networked wind power cluster[J]. Power System Technology, 2018, 42(6): 1868–1875
[17] 张斌, 左剑, 周年光, 等. 考虑变桨机组动态响应的风电场闭环有功控制方法[J]. 电力系统及其自动化学报, 2017, 29(7): 136–143
ZHANG Bin, ZUO Jian, ZHOU Nianguang, et al. Closed-loop active power control method for wind farm considering the dynamic response of variable-pitch wind turbine[J]. Proceedings of the CSU-EPSA, 2017, 29(7): 136–143
[18] 张晋华, 朱悦榕, 李旭强, 等. 考虑风电不确定性的含海上风电场电力系统优化调度策略研究[J]. 分布式能源, 2021, 6(5): 33–43
ZHANG Jinhua, ZHU Yuerong, LI Xuqiang, et al. Optimization scheduling strategy of offshore wind farm power system considering wind power uncertainty[J]. Distributed Energy, 2021, 6(5): 33–43
[19] 葛晓琳, 郝广东, 夏澍, 等. 高比例风电系统的优化调度方法[J]. 电网技术, 2019, 43(2): 390–400
GE Xiaolin, HAO Guangdong, XIA Shu, et al. An optimal system scheduling method with high proportion of wind power[J]. Power System Technology, 2019, 43(2): 390–400
[20] 朱瑛, 高云波, 臧海祥, 等. 风电机组输出功率平滑技术综述[J]. 电力系统自动化, 2018, 42(18): 182–191
ZHU Ying, GAO Yunbo, ZANG Haixiang, et al. Review of output power smoothing technologies for wind turbine[J]. Automation of Electric Power Systems, 2018, 42(18): 182–191
[21] 刘军, 张彬彬, 赵晨聪. 基于数据驱动的风电场有功功率分配算法[J]. 电力系统自动化, 2019, 43(17): 125–131
LIU Jun, ZHANG Binbin, ZHAO Chencong. Data-driven based active power distribution algorithm in wind farm[J]. Automation of Electric Power Systems, 2019, 43(17): 125–131
[22] 路朋, 叶林, 裴铭, 等. 风电集群有功功率模型预测协调控制策略[J]. 中国电机工程学报, 2021, 41(17): 5887–5900
LU Peng, YE Lin, PEI Ming, et al. Coordinated control strategy for active power of wind power cluster based on model predictive control[J]. Proceedings of the CSEE, 2021, 41(17): 5887–5900
[23] 陈晓光, 杨秀媛, 王镇林, 等. 考虑多目标优化模型的风电场储能容量配置方案[J]. 发电技术, 2022, 43(5): 718–730
CHEN Xiaoguang, YANG Xiuyuan, WANG Zhenlin, et al. Energy storage capacity allocation scheme of wind farm considering multi-objective optimization model[J]. Power System Technology:, 2022, 43(5): 718–730
[24] SULTANA W R, SAHOO S K, SUKCHAI S, et al. A review on state of art development of model predictive control for renewable energy applications[J]. Renewable & sustainable energy reviews, 2017, 76: 391–406.
[25] LIAO S Y, XU J, SUN Y Z, et al. Control of energy-intensive load for power smoothing in wind power plants[J]. IEEE Transactions on Power Systems, 2018, 33(6): 6142–6154.
[26] 张中泉, 刘晓光, 钟天宇. 风电机组有功功率集散优化控制系统研究[J]. 发电技术, 2018, 39(6): 574–579
ZHANG Zhongquan, LIU Xiaoguang, ZHONG Tianyu. Research on optimal distributed control system of active power of wind turbine generators[J]. Power Generation Technology, 2018, 39(6): 574–579
[27] YE Y J, QIU D W, SUN M Y, et al. Deep reinforcement learning for strategic bidding in electricity markets[J]. IEEE Transactions on Smart Grid, 2020, 11(2): 1343–1355.
[28] 张淑兴, 马驰, 杨志学, 等. 基于深度确定性策略梯度算法的风光储系统联合调度策略[J/OL]. 中国电力: 1–9[2022-06-08].
ZHANG Shuxing, MA Chi, YANG Zhixue, et al. Joint dispatch of wind-photovoltaic-storage hybrid system based on deep deterministic policy gradient algorithm [J/OL]. Electric Power: 1–9[2022-06-08].
[29] 孙宏斌, 黄天恩, 郭庆来, 等. 基于仿真大数据的电网智能型超前安全预警技术[J]. 南方电网技术, 2016, 10(3): 42–46
SUN Hongbin, HUANG Tianen, GUO Qinglai, et al. Power grid intelligent security early warning technology based on big simulation data[J]. Southern Power System Technology, 2016, 10(3): 42–46
[30] LIN Z W, CHEN Z, QU C Z, et al. A hierarchical clustering-based optimization strategy for active power dispatch of large-scale wind farm[J]. International Journal of Electrical Power & Energy Systems, 2020, 121: 106155–106163.

基于深度强化学习的海上风电集群自进化功率平滑控制方法

Self-evolving Power Smooth Control Method for Offshore Wind Power Cluster Based On Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 8

编辑推荐

Metrics

[1]	曾仪, 周毅, 陆继翔, 周良才, 唐宁恺, 李红. 基于多智能体安全深度强化学习的电压控制[J]. 中国电力, 2025, 58(2): 111-117.
[2]	薛松萍, 高德荃, 赵子岩, 林彧茜, 广泽晶, 张大卫. 基于业务差异化传输需求下的电力通信网路由算法[J]. 中国电力, 2024, 57(11): 183-190.
[3]	张超, 赵冬梅, 季宇, 张颖. 基于改进深度Q网络的虚拟电厂实时优化调度[J]. 中国电力, 2024, 57(1): 91-100.
[4]	石文喆, 李冰洁, 尤培培, 张泠. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56(6): 114-122.
[5]	张淑兴, 马驰, 杨志学, 王尧, 吴昊, 任洲洋. 基于深度确定性策略梯度算法的风光储系统联合调度策略[J]. 中国电力, 2023, 56(2): 68-76.
[6]	周庆锋, 王思淳, 李德鑫, 刘佳琪, 李同. 基于DDPG的风电场动态参数智能校核知识学习模型[J]. 中国电力, 2022, 55(5): 32-38.
[7]	杨朋朋, 王蓓蓓, 胥鹏, 王高琴, 郑亚先. 不完全信息下基于深度双Q网络的发电商三段式竞价策略[J]. 中国电力, 2021, 54(11): 47-58.
[8]	林俐，李丹，齐军，张红光. 风电集群功率波动下系统关键变量提取及特性分析[J]. 中国电力, 2015, 48(9): 113-118.

模态框（Modal）标题

基于深度强化学习的海上风电集群自进化功率平滑控制方法

Self-evolving Power Smooth Control Method for Offshore Wind Power Cluster Based On Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 8

编辑推荐

Metrics