基于混合博弈强化学习的虚拟电厂市场交易策略

doi:10.11930/j.issn.1004-9649.202506047

摘要/Abstract

摘要：

随着地区分布式能源快速发展，其单机装机容量小和出力随机性强的问题愈发凸显，导致分布式能源在单独参与市场交易时竞争力不足。为提升其市场参与能力，整合分布式能源形成虚拟电厂（virtual power plant，VPP）已成为一种有效途径。因此，针对含分布式能源的VPP市场交易策略进行研究，提出一种基于混合博弈强化学习的交易策略。首先，根据虚拟电厂内部单元的运行特性构建能源供应商和负荷聚合商的收益模型；然后，为了保证虚拟电厂内部运营商的整体收益建立社会福利最大化模型；最后，基于Stackelberg博弈和演化博弈的混合博弈强化学习算法求解该交易模型。算例分析表明，基于混合博弈强化学习算法的双层模型求解效果优于其他传统智能算法，求解时间减小近50%；此外，VPP同时参与能量市场和辅助服务市场时，可获得更高的收益。

关键词: 虚拟电厂运营商, 市场交易, 混合博弈

Abstract:

With the rapid development of regional distributed energy, the issues of small installed capacity and strong output variability have become increasingly prominent, resulting in insufficient competitiveness when distributed energy participates in market transactions independently. To enhance its market participation capabilities, integrating distributed energy resources into virtual power plant has emerged as an effective approach. Therefore, this study investigates market trading strategies for virtual power plant incorporating distributed energy resources and proposes a trading strategy based on hybrid game-based reinforcement learning. First, establish revenue models for energy suppliers and load aggregators based on the operational characteristics of internal units within the virtual power plant. Then, to ensure the overall profitability of operators within the virtual power plant, a social welfare maximization model is established. Finally, the transaction model is solved using a hybrid game-based reinforcement learning algorithm combining Stackelberg and evolutionary game theory. Case studies demonstrate that the two-layer model based on hybrid game-theoretic reinforcement learning algorithms outperforms traditional intelligent algorithms, reducing computation time by nearly 50%. Furthermore, when virtual power plants participate in both energy markets and ancillary service markets, they can achieve higher returns.

Key words: virtual power plant operators, market trading, hybrid game

郑峰, 孙电, 黄丽丽, 杨峰, 倪芸. 基于混合博弈强化学习的虚拟电厂市场交易策略[J]. 中国电力, 2026, 59(3): 94-102.

ZHENG Feng, SUN Dian, HUANG Lili, YANG Feng, NI Yun. Virtual power plant market trading strategy based on hybrid game reinforcement learning[J]. Electric Power, 2026, 59(3): 94-102.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202506047

https://www.electricpower.com.cn/CN/Y2026/V59/I3/94

图/表 11

图 1 市场交易模型

Fig.1 Market trading model

图 2 求解流程

Fig.2 Solution process

图 3 风光预测出力

Fig.3 Wind and solar forecast output

图 4 负荷聚合商的负荷预测情况

Fig.4 Load forecasting situation of load aggregator

表 1 储能装置参数

Table 1 Parameters of energy storage devices

最大充放电功率/MW	容量/ (MW·h)	充放电效率	初始荷电状态	最小荷电状态	最大荷电状态
20	80	0.95	0.5	0.4	0.9

图 5 调度结果

Fig.5 Scheduling results

图 6 能源供应商出力情况

Fig.6 Output Status of VPP operator

图 7 能源供应商的储能出力情况

Fig.7 Energy storage output of VPP operator

图 8 能源供应商的风光消纳情况

Fig.8 Wind and solar power consumption situation of VPP operator

表 2 不同算法下优化结果对比

Table 2 Comparison of optimization results under different algorithms

算法	社会效益/10⁴元	求解时间/s
粒子群算法	4.81	25.42
KKT条件	4.83	12.15
遗传算法	4.83	23.57
混合强化学习算法	4.86	20.11

表 3 能源供应商参与不同市场的交易结果对比

Table 3 Comparison of transaction results of VPP operators participating in different markets

场景	能源供应商收益		负荷聚合商收益/10⁴元	社会效益/ 10⁴元
场景	能量收益/10⁴元	辅助服务收益/10⁴元	负荷聚合商收益/10⁴元	社会效益/ 10⁴元
场景1	32.65	0	–32.88	–2.30
场景2	32.92	4.88	–32.95	4.86

参考文献 32

1	杨胡萍, 龚家宁, 程明, 等. 计及多重不确定性的综合能源系统两阶段鲁棒低碳优化调度[J]. 中国电力, 2025, 58 (11): 101- 110, 121.
	YANG Huping, GONG Jianing, CHENG Ming, et al. Two-stage robust low-carbon optimal scheduling for integrated energy systems considering for multiple uncertainties[J]. Electric Power, 2025, 58 (11): 101- 110, 121.
2	赵峰, 孙超, 寇凌岳, 等. 分布式储能聚合商参与调频市场竞价策略与协调调度研究[J]. 浙江电力, 2025, 44 (3): 30- 41.
	ZHAO Feng, SUN Chao, KOU Lingyue, et al. Research on a bidding strategy and coordinated scheduling of distributed energy storage aggregators in frequency regulation market[J]. Zhejiang Electric Power, 2025, 44 (3): 30- 41.
3	刘洋, 陆秋瑜, 徐展鹏, 等. 适用于大规模海上风电集中送电的能源基地运行调度策略[J]. 广东电力, 2025, 38 (5): 16- 29.
	LIU Yang, LU Qiuyu, XU Zhanpeng, et al. Energy base operation and scheduling strategy suitable for large-scale centralized transmission of offshore wind power[J]. Guangdong Electric Power, 2025, 38 (5): 16- 29.
4	周洋, 黄德志, 李培栋, 等. 考虑平衡端点相位不对称及光伏接入的低压配电网三相潮流模型[J]. 中国电力, 2024, 57 (10): 190- 198.
	ZHOU Yang, HUANG Dezhi, LI Peidong, et al. A three-phase power flow model for low-voltage distribution networks considering balanced bus phase asymmetry and photovoltaic access[J]. Electric Power, 2024, 57 (10): 190- 198.
5	段玉, 陈军, 刘震, 等. 面向新型电力系统动态分析的多时间尺度构网型储能系统仿真建模方法[J]. 广东电力, 2024, 37 (12): 27- 38.
	DUAN Yu, CHEN Jun, LIU Zhen, et al. Modelling of multi-time scale grid-forming energy storage systems for dynamic analysis of new power systems[J]. Guangdong Electric Power, 2024, 37 (12): 27- 38.
6	黄灿兵, 熊妮, 吴伟, 等. 虚拟电厂参与一次调频的优化调度策略研究[J]. 浙江电力, 2025, 44 (2): 42- 52.
	HUANG Canbing, XIONG Ni, WU Wei, et al. Research on an optimal scheduling strategy for a virtual power plant participating in primary frequency regulation[J]. Zhejiang Electric Power, 2025, 44 (2): 42- 52.
7	王宇绅, 陈皓勇, 黄宇翔, 等. 多重不确定性下的虚拟电厂参与电能量和需求响应市场鲁棒优化调度策略[J]. 发电技术, 2024, 45 (6): 1173- 1185.
	WANG Yushen, CHEN Haoyong, HUANG Yuxiang, et al. Robust optimal scheduling strategy for virtual power plant participation in electric energy and demand response markets under multiple uncertainties[J]. Power Generation Technology, 2024, 45 (6): 1173- 1185.
8	江海燕, 李晨阳, 孟诗语, 等. 考虑配电网分区的虚拟电厂分布式协同优化运行模型[J]. 浙江电力, 2025, 44 (2): 53- 63.
	JIANG Haiyan, LI Chenyang, MENG Shiyu, et al. A distributed cooperative optimal operating model for virtual power plants considering distribution network partitions[J]. Zhejiang Electric Power, 2025, 44 (2): 53- 63.
9	钟永洁, 汤成俊, 王紫东, 等. 我国虚拟电厂的发展演进和关键技术及难点分析[J]. 浙江电力, 2025, 44 (2): 13- 31.
	ZHONG Yongjie, TANG Chengjun, WANG Zidong, et al. Analysis of the evolution, key technologies, and challenges of virtual power plants in China[J]. Zhejiang Electric Power, 2025, 44 (2): 13- 31.
10	杨彪, 李佳蓉, 魏子轲. 美国分布式能源发展对我国的启示[J]. 中国电力企业管理, 2024 (25): 94- 96.
11	张燕, 乔松博, 徐奇锋, 等. 基于纳什议价理论的分布式绿色电力交易优化分析[J]. 中国电力, 2022, 55 (12): 168- 178.
	ZHANG Yan, QIAO Songbo, XU Qifeng, et al. Analysis of distributed green power transaction optimization based on Nash bargaining theory[J]. Electric Power, 2022, 55 (12): 168- 178.
12	WU S, LIN J, LI J, et al. Multi-timescale trading strategy for renewable power to ammonia virtual power plant in the electricity, hydrogen, and ammonia markets[J]. Policy and Regulation IEEE Transactions on Energy Markets, 2023, 1 (4): 322- 335.
13	GONG X, LI X M, ZHONG Z M. Strategic bidding of virtual power plants in integrated electricity-carbon-green certificate market with renewable energy uncertainties[J]. Sustainable Cities and Society, 2025, 121, 106176.
14	许星原, 陈皓勇, 黄宇翔, 等. 虚拟电厂市场化交易中的挑战、策略与关键技术[J]. 发电技术, 2023, 44 (6): 745- 757.
	XU Xingyuan, CHEN Haoyong, HUANG Yuxiang, et al. Challenges, strategies and key technologies for virtual power plants in market trading[J]. Power Generation Technology, 2023, 44 (6): 745- 757.
15	曾垂辉, 李文正, 李宝伟, 等. 含多分布式能源聚合商的本地电力市场最优能源交易策略[J]. 中国电力, 2023, 56 (11): 236- 245.
	ZENG Chuihui, LI Wenzheng, LI Baowei, et al. Optimal energy trading strategies for local electricity markets with multiple distributed energy resource aggregators[J]. Electric Power, 2023, 56 (11): 236- 245.
16	杨谭洪, 杨彪, 张延旭, 等. 虚拟电厂设计与运营[M]. 北京: 中国电力出版社, 2025.
17	何斯强, 张俊岭, 顾宗奇, 等. 面向分布式资源聚合管控的虚拟电厂建模与优化控制综述[J]. 山东电力技术, 2024, 51 (2): 11- 24, 45.
	HE Siqiang, ZHANG Junling, GU Zongqi, et al. Overview of virtual power plant modeling and optimization control for distributed resource aggregation and control[J]. Shandong Electric Power, 2024, 51 (2): 11- 24, 45.
18	崔一阳, 黎灿兵, 潘斗南, 等. 高比例新能源电力系统中机组组合研究综述[J]. 湖南电力, 2025, 45 (1): 37- 45.
	CUI Yiyang, LI Canbing, PAN Dounan, et al. A review of unit commitment in high proportion renewable energy power systems[J]. Hunan Electric Power, 2025, 45 (1): 37- 45.
19	耿健, 金玉龙, 杨宇峰, 等. 考虑调峰辅助服务的虚拟电厂日前运行优化研究[J]. 山东电力技术, 2024, 51 (12): 44- 52.
	GENG Jian, JIN Yulong, YANG Yufeng, et al. Virtual power plant day-ahead operation optimization considering peak shaving auxiliary service[J]. Shandong Electric Power, 2024, 51 (12): 44- 52.
20	杨珂, 王栋, 李达, 等. 虚拟电厂网络安全风险评估指标体系构建及量化计算[J]. 中国电力, 2024, 57 (8): 130- 137.
	YANG Ke, WANG Dong, LI Da, et al. Network security risk assessment index system and calculation for virtual power plant[J]. Electric Power, 2024, 57 (8): 130- 137.
21	何知纯, 谢敏, 黄莹, 等. 基于连续隐马尔可夫模型的风水火联合低碳检修优化[J]. 南方能源建设, 2023, 10 (4): 43- 56.
	HE Zhichun, XIE Min, HUANG Ying, et al. Wind power, hydropower and thermal power combined low-carbon maintenance optimization based on continuous hidden Markov model[J]. Southern Energy Construction, 2023, 10 (4): 43- 56.
22	吴垠, 牛文娟, 诸晓骏. 考虑不同场景的虚拟电厂建模与优化分析[J]. 电测与仪表, 2024, 61 (11): 40- 45.
	WU Yin, NIU Wenjuan, ZHU Xiaojun. Modeling and optimization analysis of virtual power plant considering different scenarios[J]. Electrical Measurement & Instrumentation, 2024, 61 (11): 40- 45.
23	李继传, 罗朝春, 王宇, 等. 适应大峰谷差的虚拟电厂市场化交易机制研究[J]. 湖南电力, 2025, 45 (2): 129- 136.
	LI Jichuan, LUO Chaochun, WANG Yu, et al. Study on market trading mechanism of virtual power plant adapted to large peak-valley difference[J]. Hunan Electric Power, 2025, 45 (2): 129- 136.
24	刘继春, 刘可欣, 柯贤波, 等. 新型电力系统下储能参与电能量-惯量-一次调频多市场交易方法[J]. 电网技术, 2025, 49 (3): 1018- 1031.
	LIU Jichun, LIU Kexin, KE Xianbo, et al. Participation of energy storage in electric energy-inertia-primary frequency regulation multi-market trading method under a new power system[J]. Power System Technology, 2025, 49 (3): 1018- 1031.
25	丁泉, 窦晓波, 温鹏, 等. 电力市场环境下配电网侧多类型主体纳什议价交易模型[J]. 电力自动化设备, 2024, 44 (12): 170- 177.
	DING Quan, DOU Xiaobo, WEN Peng, et al. Nash bargaining based transaction model for multi-type entities on distribution network side under electricity market environment[J]. Electric Power Automation Equipment, 2024, 44 (12): 170- 177.
26	郭祚刚, 喻磊, 胡洋, 等. 基于合作博弈的综合能源服务商现货市场风险规避策略[J]. 中国电力, 2019, 52 (11): 28- 34.
	GUO Zuogang, YU Lei, HU Yang, et al. Research on competition strategy of integrated energy service provider under the pool-based market mechanism[J]. Electric Power, 2019, 52 (11): 28- 34.
27	包涛, 李昊飞, 余涛, 等. 考虑市场因素的电力系统供需互动混合博弈强化学习算法[J]. 控制理论与应用, 2020, 37 (4): 907- 917.
	BAO Tao, LI Haofei, YU Tao, et al. Mixed game reinforcement learning of supply-demand interaction in power system dis-patch on electricity market[J]. Control Theory & Applications, 2020, 37 (4): 907- 917.
28	顼佳宇, 王晓冰, 李冰, 等. 考虑灵活性资源配置的配电网光伏承载力评估方法[J]. 电测与仪表, 2025, 62 (2): 143- 153.
	XU Jiayu, WANG Xiaobing, LI Bing, et al. Distributed PV hosting capacity assessment method for distribution network considering planning of flexible resource[J]. Electrical Measurement & Instrumentation, 2025, 62 (2): 143- 153.
29	易文飞, 朱卫平, 郑明忠. 计及数据中心和风电不确定性的微电网经济调度[J]. 中国电力, 2024, 57 (2): 19- 26.
	YI Wenfei, ZHU Weiping, ZHENG Mingzhong. Economic dispatch of microgrid considering data center and wind power uncertainty[J]. Electric Power, 2024, 57 (2): 19- 26.
30	张冬冬, 单琳珂, 刘天皓. 人工智能技术在风力与光伏发电数据挖掘及功率预测中的应用综述[J]. 综合智慧能源, 2025, 47 (3): 32- 46.
	ZHANG Dongdong, SHAN Linke, LIU Tianhao. Review on the application of artificial intelligence in data mining and wind and photovoltaic power forecasting[J]. Integrated Intelligent Energy, 2025, 47 (3): 32- 46.
31	王辉, 李乃慧, 梁凌, 等. 耦合系统下含光热电站的需求响应虚拟电厂优化调度[J]. 电测与仪表, 2025, 62 (12): 30- 40.
	WANG Hui, LI Naihui, LIANG Ling, et al. Optimal scheduling of virtual power plant with concentrating solar power based on demand response in coupled system[J]. Electrical Measurement & Instrumentation, 2025, 62 (12): 30- 40.
32	张玉敏, 尹延宾, 吉兴全, 等. 计及热网不同运行状态下灵活性供给能力的综合能源系统优化调度[J]. 中国电力, 2025, 58 (2): 88- 102.
	ZHANG Yumin, YIN Yanbin, JI Xingquan, et al. Optimal dispatch of integrated electric-heat energy system considering supply flexibility of heat networks under different operation states[J]. Electric Power, 2025, 58 (2): 88- 102.

[1]	尚静怡, 张艺涵, 杨福旺, 向明旭. 发电商策略性竞价对省间-省内现货市场影响与应对策略[J]. 中国电力, 2025, 58(11): 1-13.
[2]	高志远, 黄海峰, 孙芊, 张鸿, 曹阳, 唐保国. 跨国调度与交易系统互联的信息通信支撑研究[J]. 中国电力, 2019, 52(1): 40-47.
[3]	胡飞雄, 周保荣, 卢斯煜. 南方电网促进可再生能源消纳的实践及发展展望[J]. 中国电力, 2018, 51(1): 22-28.