Bidding Strategy for Thermal Power Generation Companies Based on Multi-agent Deep Deterministic Policy Gradient Algorithm

doi:10.11930/j.issn.1004-9649.202309119

Abstract

Abstract:

Thermal power is an important support for the new power system. It is of great significance to study the bidding strategy for thermal power generation companies and the influence of different clearing mechanisms to ensure their low-carbon and efficient operation. A bidding strategy model is constructed based on the multi-agent deep deterministic policy gradient algorithm to analyze the differential bidding strategies for different combinations of thermal power generation companies. The multi-agent price and quantity bidding strategy is optimized, and the market impact of different market clearing mechanisms is explored. The simulation results indicate that the proposed bidding strategy model can guide the thermal power generation companies to optimize their bidding methods and improve the market efficiency. When the penetration rate of new energy is low, the applicability of different clearing mechanisms varies for various types of units; with the increase of the penetration rate of new energy, the pay as bid mechanism can be used to enhance the economic and environmental efficiency of the electricity market; when the penetration rate of new energy reaches a high level, the random matching clearing mechanism can effectively address market volatility risks.

Key words: thermal power generation companies, multi-agent, clearing mechanism, bidding strategy, new energy penetration

Xingping ZHANG, Teng WANG, Xinyue ZHANG, Haonan ZHANG. Bidding Strategy for Thermal Power Generation Companies Based on Multi-agent Deep Deterministic Policy Gradient Algorithm[J]. Electric Power, 2024, 57(11): 161-172.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202309119

https://www.electricpower.com.cn/EN/Y2024/V57/I11/161

Figures/Tables 13

References 31

1	王林, 李晨, 刘嘉佳, 等. 基于复式竞价撮合的电力市场交易模式设计与实践[J]. 电力系统自动化, 2018, 42 (24): 188- 195.
	WANG Lin, LI Chen, LIU Jiajia, et al. Design and practice of electricity market trading mode based on compound bidding matchmaking[J]. Automation of Electric Power Systems, 2018, 42 (24): 188- 195.
2	黄远明, 卢恩, 赖晓文, 等. 广东月度集中竞争交易机制设计与实践[J]. 南方电网技术, 2018, 12 (8): 51- 58.
	HUANG Yuanming, LU En, LAI Xiaowen, et al. Design and practice of monthly centralized trading mechanism in Guangdong Province[J]. Southern Power System Technology, 2018, 12 (8): 51- 58.
3	程乐峰, 余涛. 发电市场长期竞价均衡自发形成过程中的一般多策略演化博弈决策行为研究[J]. 中国电机工程学报, 2020, 40 (21): 6936- 6955.
	CHENG Lefeng, YU Tao. Decision-making behavior investigation for general multi-strategy evolutionary games in the spontaneous formation of long-term bidding equilibria of A power generation market[J]. Proceedings of the CSEE, 2020, 40 (21): 6936- 6955.
4	何洋, 黄龙, 陈皓勇, 等. 基于协同进化算法的集中竞价市场模拟分析[J]. 浙江电力, 2019, 38 (7): 7- 13.
	HE Yang, HUANG Long, CHEN Haoyong, et al. Simulation analysis of centralized bidding market based on co-evolutionary algorithm[J]. Zhejiang Electric Power, 2019, 38 (7): 7- 13.
5	齐世雄, 王秀丽, 张炜, 等. 基于博弈论和改进PageRank的电力市场出清方式评价方法[J]. 电力建设, 2019, 40 (5): 107- 117. DOI
	QI Shixiong, WANG Xiuli, ZHANG Wei, et al. Evaluation method for electricity market clearing model based on game theory and improved PageRank[J]. Electric Power Construction, 2019, 40 (5): 107- 117. DOI
6	石可, 陈皓勇, 李鹏, 等. 基于协同进化的两种电力市场出清机制分析[J]. 电力系统自动化, 2019, 43 (9): 68- 74.
	SHI Ke, CHEN Haoyong, LI Peng, et al. Analysis on two kinds of electricity market clearance mechanism based on co-evolution[J]. Automation of Electric Power Systems, 2019, 43 (9): 68- 74.
7	陈皓勇, 付超. 统一价格和PAB竞价的实验分析[J]. 电力系统自动化, 2007, 31 (4): 12- 17.
	CHEN Haoyong, FU Chao. Experimental analysis of uniform price and PAB auctions in electricity markets[J]. Automation of Electric Power Systems, 2007, 31 (4): 12- 17.
8	HAN D, SUN M. The design of a Probability Bidding Mechanism in electricity auctions by considering trading constraints[J]. Simulation, 2015, 91 (10): 916- 924. DOI
9	LIU Z, ZHANG X L, LIEU J. Design of the incentive mechanism in electricity auction market based on the signaling game theory[J]. Energy, 2010, 35 (4): 1813- 1819. DOI
10	薛贵元, 吴垠, 诸晓骏, 等. 计及用能权配额约束的发电商竞价策略双层优化方法[J]. 中国电力, 2023, 56 (5): 51- 61.
	XUE Guiyuan, WU Yin, ZHU Xiaojun, et al. Bi-level optimization method for bidding strategy of power suppliers considering energy-consuming right[J]. Electric Power, 2023, 56 (5): 51- 61.
11	田福银, 马骏, 王灿, 等. 基于双层主从博弈的综合能源系统多主体低碳经济运行策略[J]. 中国电力, 2022, 55 (11): 184- 193.
	TIAN Fuyin, MA Jun, WANG Can, et al. Multi-agent low-carbon and economy operation strategy of integrated energy system based on Bi-level master-slave game[J]. Electric Power, 2022, 55 (11): 184- 193.
12	杨朋朋, 王蓓蓓, 胥鹏, 等. 不完全信息下基于深度双Q网络的发电商三段式竞价策略[J]. 中国电力, 2021, 54 (11): 47- 58.
	YANG Pengpeng, WANG Beibei, XU Peng, et al. Three-stage bidding strategy of generation company based on double deep Q-network under incomplete information condition[J]. Electric Power, 2021, 54 (11): 47- 58.
13	NAMALOMBA E, HU F H, SHI H J. Agent based simulation of centralized electricity transaction market using bi-level and Q-learning algorithm approach[J]. International Journal of Electrical Power & Energy Systems, 2022, 134, 107415.
14	高宇, 李昀, 曹蓉蓉, 等. 基于多代理Double DQN算法模拟发电侧竞价行为[J]. 电网技术, 2020, 44 (11): 4175- 4182.
	GAO Yu, LI Yun, CAO Rongrong, et al. Simulation of generators' bidding behavior based on multi-agent double DQN[J]. Power System Technology, 2020, 44 (11): 4175- 4182.
15	周翔, 王继业, 陈盛, 等. 基于深度强化学习的微网优化运行综述[J]. 全球能源互联网, 2023, 6 (3): 240- 257.
	ZHOU Xiang, WANG Jiye, CHEN Sheng, et al. Review of microgrid optimization operation based on deep reinforcement learning[J]. Journal of Global Energy Interconnection, 2023, 6 (3): 240- 257.
16	徐尔丰. 基于A3C强化学习的电力市场发电商报价策略研究[D]. 北京: 华北电力大学(北京), 2019.
	XU Erfeng. Research on bidding strategy of generators in electricity market based on asynchronous advantage actor-critic reinforcement learning[J]. Beijing: North China Electric Power University(Beijing), 2019.
17	马丽莹, 魏云冰. 基于DDPG算法的发电企业报价策略研究[J]. 电气工程学报, 2023, 18 (2): 192- 200.
	MA Liying, WEI Yunbing. Research on bidding strategy of power generation enterprise based on DDPG algorithm[J]. Journal of Electrical Engineering, 2023, 18 (2): 192- 200.
18	LIANG Y C, GUO C L, DING Z H, et al. Agent-based modeling in electricity market using deep deterministic policy gradient algorithm[J]. IEEE Transactions on Power Systems, 2020, 35 (6): 4180- 4192. DOI
19	YE Y J, QIU D W, SUN M Y, et al. Deep reinforcement learning for strategic bidding in electricity markets[J]. IEEE Transactions on Smart Grid, 2020, 11 (2): 1343- 1355. DOI
20	YE Y J, QIU D W, LI J, et al. Multi-period and multi-spatial equilibrium analysis in imperfect electricity markets: a novel multi-agent deep reinforcement learning approach[J]. IEEE Access, 2019, 7, 130515- 130529. DOI
21	DU Y, LI F X, ZANDI H L, et al. Approximating Nash equilibrium in day-ahead electricity market bidding with multi-agent deep reinforcement learning[J]. Journal of Modern Power Systems and Clean Energy, 2021, 9 (3): 534- 544. DOI
22	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California, USA. ACM, 2017: 6382–6393.
23	于申, 申建建, 程春田, 等. 耦合梯级水电调蓄价值的月度集中撮合交易出清方法[J]. 中国电机工程学报, 2022, 42 (16): 5858- 5868.
	YU Shen, SHEN Jianjian, CHENG Chuntian, et al. Centralized matchmaking transaction clearing method embedded the regulation value of cascade hydropower[J]. Proceedings of the CSEE, 2022, 42 (16): 5858- 5868.
24	张兴平, 何澍, 王泽嘉, 等. 不同新能源渗透率下燃煤机组行为策略分析[J]. 电力建设, 2022, 43 (5): 9- 17.
	ZHANG Xingping, HE Shu, WANG Zejia, et al. Behavior strategy of coal-fired units under different new energy penetration rate[J]. Electric Power Construction, 2022, 43 (5): 9- 17.
25	ZHANG X Y, GUO X P, ZHANG X P. Collaborative strategy within China's emission trading scheme: evidence from a tripartite evolutionary game model[J]. Journal of Cleaner Production, 2023, 382, 135255. DOI
26	李姚旺, 刘昱良, 杨晓斌, 等. 计及电量交易信息的用电碳计量方法[J]. 中国电机工程学报, 2024, 44 (2): 439- 451.
	LI Yaowang, LIU Yuliang, YANG Xiaobin, et al. Electricity carbon metering method considering electricity transaction information[J]. Proceedings of the CSEE, 2024, 44 (2): 439- 451.
27	万里鹏, 兰旭光, 张翰博, 等. 深度强化学习理论及其应用综述[J]. 模式识别与人工智能, 2019, 32 (1): 67- 81.
	WAN Lipeng, LAN Xuguang, ZHANG Hanbo, et al. A review of deep reinforcement learning theory and application[J]. Pattern Recognition and Artificial Intelligence, 2019, 32 (1): 67- 81.
28	赵立阳, 常天庆, 褚凯轩, 等. 完全合作类多智能体深度强化学习综述[J]. 计算机工程与应用, 2023, 59 (12): 14- 27. DOI
	ZHAO Liyang, CHANG Tianqing, CHU Kaixuan, et al. Survey of fully cooperative multi-agent deep reinforcement learning[J]. Computer Engineering and Applications, 2023, 59 (12): 14- 27. DOI
29	王军, 曹雷, 陈希亮, 等. 多智能体博弈强化学习研究综述[J]. 计算机工程与应用, 2021, 57 (21): 1- 13. DOI
	WANG Jun, CAO Lei, CHEN Xiliang, et al. Overview on reinforcement learning of multi-agent game[J]. Computer Engineering and Applications, 2021, 57 (21): 1- 13. DOI
30	李鹏, 黄文琦, 王鑫, 等. 数据与知识联合驱动的人工智能方法在电力调度中的应用综述[J]. 电力系统自动化, 2024, 48 (1): 160- 175. DOI
	LI Peng, HUANG Wenqi, WANG Xin, et al. Review on application of combined data-knowledge-driven artificial intelligence methods in power dispatching[J]. Automation of Electric Power Systems, 2024, 48 (1): 160- 175. DOI
31	刘福国, 蒋学霞, 李志. 燃煤发电机组负荷率影响供电煤耗的研究[J]. 电站系统工程, 2008, 24 (4): 47- 49.
	LIU Fuguo, JIANG Xuexia, LI Zhi. Investigation on affects of generator load on coal consumption rate in fossil power plant[J]. Power System Engineering, 2008, 24 (4): 47- 49.

编号	机组容量/MW	a	b	c	可申报电量/(MW·h)	边际成本/ (元·(kW·h)^–1）
G1	330	0.00140	–0.8391	484.2	46012.9	0.3877
G2	330	0.00140	–0.8439	495.6	46012.9	0.3960
G3	330	0.00130	–0.8700	477.8	48801.6	0.3465
G4	330	0.00080	–0.5428	416.6	47407.3	0.3119
G5	330	0.00090	–0.5910	417.0	92025.8	0.3210
G6	330	0.00090	–0.6375	422.3	92025.8	0.2956
G7	660	0.00030	–0.3747	413.8	92025.8	0.3112
G8	600	0.00030	–0.3532	437.3	83659.9	0.3375
G9	660	0.00020	–0.3241	430.8	92025.8	0.2643
G10	600	0.00030	–0.3384	439.3	83659.9	0.3572
G11	1000	0.00005	–0.1208	356.1	139433.1	0.2645
G12	1000	0.00010	–0.2442	420.5	147799.1	0.2320

编号	机组容量/MW	a	b	c	可申报电量/(MW·h)	边际成本/ (元·(kW·h)^–1）
G1	330	0.00140	–0.8391	484.2	46012.9	0.3877
G2	330	0.00140	–0.8439	495.6	46012.9	0.3960
G3	330	0.00130	–0.8700	477.8	48801.6	0.3465
G4	330	0.00080	–0.5428	416.6	47407.3	0.3119
G5	330	0.00090	–0.5910	417.0	92025.8	0.3210
G6	330	0.00090	–0.6375	422.3	92025.8	0.2956
G7	660	0.00030	–0.3747	413.8	92025.8	0.3112
G8	600	0.00030	–0.3532	437.3	83659.9	0.3375
G9	660	0.00020	–0.3241	430.8	92025.8	0.2643
G10	600	0.00030	–0.3384	439.3	83659.9	0.3572
G11	1000	0.00005	–0.1208	356.1	139433.1	0.2645
G12	1000	0.00010	–0.2442	420.5	147799.1	0.2320

机组容量/ MW	设置发电机组为智能体
机组容量/ MW	S1	S2	S3	S4	S5	S6	S7	S8
300	√	×	×	√	√	×	√	×
600	×	√	×	√	×	√	√	×
1000	×	×	√	×	√	√	√	×

机组容量/ MW	设置发电机组为智能体
机组容量/ MW	S1	S2	S3	S4	S5	S6	S7	S8
300	√	×	×	√	√	×	√	×
600	×	√	×	√	×	√	√	×
1000	×	×	√	×	√	√	√	×

出清机制	排放收益率/(元·t^–1)
出清机制	S1	S2	S3	S4	S5	S6	S7	S8
1	0.154	0.146	0.168	0.183	0.187	0.151	0.228	0.137
2	0.141	0.154	0.094	0.219	0.186	0.125	0.227	0.092
3	0.128	0.140	0.146	0.181	0.163	0.17	0.213	0.117