Real Time Optimal Dispatch of Virtual Power Plant Based on Improved Deep Q Network

doi:10.11930/j.issn.1004-9649.202307006

Abstract

Abstract:

The deep reinforcement learning algorithm is data-driven and does not rely on specific models, which can effectively address the complexity issues in virtual power plant (VPP) operation. However, existing algorithms are difficult to strictly enforce operational constraints, which limits their application in practical systems. To overcome this problem, an improved deep Q-network (MDQN) algorithm based on deep reinforcement learning is proposed. This algorithm expresses deep neural networks as mixed integer programming formulas to ensure strict execution of all operational constraints within the action space, thus ensuring the feasibility of the formulated scheduling in actual operation. In addition, sensitivity analysis is conducted to flexibly adjust hyperparameters, providing greater flexibility for algorithm optimization. Finally, the superior performance of the MDQN algorithm is verified through comparative experiments. An effective solution is provided to address the complexity issues in the operation of VPP.

Key words: virtual power plant, real time optimization, deep reinforcement learning, cloud edge collaboration, optimal dispatch

Chao ZHANG, Dongmei ZHAO, Yu JI, Ying ZHANG. Real Time Optimal Dispatch of Virtual Power Plant Based on Improved Deep Q Network[J]. Electric Power, 2024, 57(1): 91-100.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202307006

https://www.electricpower.com.cn/EN/Y2024/V57/I1/91

Figures/Tables 10

Fig.1 VPP cloud edge collaboration architecture

Fig.2 VPP regulation cloud platform logic diagram

Fig.3 VPP structural framework

Fig.4 The layer structure of DNN for action value function Q(s, a)

Table 1 Parameters of each micro gas turbine

设备	a	b	c	$ P_{{\text{mt}}}^{\min }/\rm kW $	$ P_{{\text{mt}}}^{\max }/\rm kW$	$ \Delta {P_{{\text{mt}}}}/\rm kW $
微燃机1	0.0034	3	30	10	150	100
微燃机2	0.0010	10	40	50	375	200
微燃机3	0.0010	15	70	100	500	200

Table 2 Parameters of each DRL algorithm

算法	样本数	学习率	折扣因子	网络维度	缓冲区大小
DDPG	256	1×10^–4	0.995	（64，64，64）	5×10⁴
SAC	256	1×10^–4	0.995	（64，64，64）	5×10⁴
TD3	256	1×10^–4	0.995	（64，64，64）	5×10⁴
MDQN	256	1×10^–4	0.995	（64，64，64）	5×10⁴

Fig.5 Rewards and power imbalance during DDPG, SAC, TD3, MDQN training

Fig.6 Accumulated 10-day operating costs and imbalance of DDPG, SAC, TD3, and MDQN

Fig.7 MDQN optimized operation plans for all units

Fig.8 MDQN operating costs and imbalance with different σ2 values

References 24

1	林毓军, 苗世洪, 杨炜晨, 等. 面向多重不确定性环境的虚拟电厂日前优化调度策略[J]. 电力自动化设备, 2021, 41 (12): 143- 150.
	LIN Yujun, MIAO Shihong, YANG Weichen, et al. Day-ahead optimal scheduling strategy of virtual power plant for environment with multiple uncertainties[J]. Electric Power Automation Equipment, 2021, 41 (12): 143- 150.
2	尚文强, 李广磊, 丁月明, 等. 考虑源荷不确定性和新能源消纳的综合能源系统协同调度方法[J/OL]. 电网技术: 1–17.[2023-10-23]. https: //doi. org/10.13335/j. 1000-3673. pst. 2023.0577.
	SHANG Wenqiang, LI Guanglei, DING Yueming, et al. A collaborative dispatching method for comprehensive energy systems considering source load uncertainty and new energy consumption [J/OL]. Power System Technology: 1–17[2023-10-23].https://doi.org/10.13335/j.1000-3673.pst.2023.0577.
3	陶力, 杨夏喜, 顾金辉, 等. 基于SAC和TD3的含电动汽车虚拟电厂调度策略[J]. 电气传动, 2023, 53 (9): 25- 34.
	TAO Li, YANG Xiaxi, GU Jinhui, et al. Scheduling strategy of virtual power plant with electric vehicle based on SAC and TD3[J]. Electric Drive, 2023, 53 (9): 25- 34.
4	CHEN C S, XIAO L L, DUAN S D, et al. Cooperative optimization of electric vehicles in microgrids considering across-time-and-space energy transmission[J]. IEEE Transactions on Industrial Electronics, 2019, 66 (2): 1532- 1542. DOI
5	程逸帆, 乔飞, 侯珂, 等. 区域微电网群两级能量调度策略优化研究[J]. 仪器仪表学报, 2019, 40 (5): 68- 77.
	CHENG Yifan, QIAO Fei, HOU Ke, et al. Research on bi-level energy dispatching strategy optimization for regional microgrid cluster[J]. Chinese Journal of Scientific Instrument, 2019, 40 (5): 68- 77.
6	梁涛, 孙博峰, 谭建鑫, 等. 基于深度强化学习算法的风光互补可再生能源制氢系统调度方案[J]. 高电压技术, 2023, 49 (6): 2264- 2275.
	LIANG Tao, SUN Bofeng, TAN Jianxin, et al. Scheduling scheme of wind-solar complementary renewable energy hydrogen production system based on deep reinforcement learning[J]. High Voltage Engineering, 2023, 49 (6): 2264- 2275.
7	王尧, 樊伟, 王佳伟, 等. 考虑风光不确定性的分布式能源集成虚拟电厂收益-风险均衡模型[J]. 可再生能源, 2020, 38 (1): 76- 83. DOI
	WANG Yao, FAN Wei, WANG Jiawei, et al. Revenue-risk equilibrium model for distributed energy integrated virtual power plant benefits considering wind and light uncertainty[J]. Renewable Energy Resources, 2020, 38 (1): 76- 83. DOI
8	LI Z M, XU Y. Temporally-coordinated optimal operation of a multi-energy microgrid under diverse uncertainties[J]. Applied Energy, 2019, 240, 719- 729. DOI
9	罗建勋, 张玮, 王辉, 等. 基于深度强化学习的微电网优化调度研究[J]. 电力学报, 2023, 38 (1): 54- 63. DOI
	LUO Jianxun, ZHANG Wei, WANG Hui, et al. Research on optimal scheduling of micro-grid based on deep reinforcement learning[J]. Journal of Electric Power, 2023, 38 (1): 54- 63. DOI
10	郭庆来, 兰健, 周艳真, 等. 基于混合智能的新型电力系统运行方式分析决策架构及其关键技术[J]. 中国电力, 2023, 56 (9): 1- 13.
	GUO Qinglai, LAN Jian, ZHOU Yanzhen, et al. Architecture and key technologies of hybrid-intelligence-based decision-making of operation modes for new type power systems[J]. Electric Power, 2023, 56 (9): 1- 13.
11	石文喆, 李冰洁, 尤培培, 等. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56 (6): 114- 122.
	SHI Wenzhe, LI Bingjie, YOU Peipei, et al. Optimization strategy of building energy system based on deep reinforcement learning[J]. Electric Power, 2023, 56 (6): 114- 122.
12	蔺伟山, 王小君, 孙庆凯, 等. 不确定性环境下基于深度强化学习的综合能源系统动态调度[J]. 电力系统保护与控制, 2022, 50 (18): 50- 60.
	LIN Weishan, WANG Xiaojun, SUN Qingkai, et al. Dynamic dispatch of an integrated energy system based on deep reinforcement learning in an uncertain environment[J]. Power System Protection and Control, 2022, 50 (18): 50- 60.
13	刘林鹏, 朱建全, 陈嘉俊, 等. 基于柔性策略-评价网络的微电网源储协同优化调度策略[J]. 电力自动化设备, 2022, 42 (1): 79- 85.
	LIU Linpeng, ZHU Jianquan, CHEN Jiajun, et al. Cooperative optimal scheduling strategy of source and storage in microgrid based on soft actor-critic[J]. Electric Power Automation Equipment, 2022, 42 (1): 79- 85.
14	ALI K H, ABUSARA M, ALI TAHIR A, et al. Dual-layer Q-learning strategy for energy management of battery storage in grid-connected microgrids[J]. Energies, 2023, 16 (3): 1334. DOI
15	张宏涛, 吴怡之, 邓开连, 等. 一种基于强化学习的微电网能量管理算法[J]. 物联网技术, 2022, 12 (12): 74- 78.
16	CHEN P Z, LIU M C, CHEN C X, et al. A battery management strategy in microgrid for personalized customer requirements[J]. Energy, 2019, 189, 116245. DOI
17	吴利刚, 张梁, 周倩, 等. 基于强化学习的微电网能量调度优化策略研究[J]. 控制工程, 2022, 29 (7): 1162- 1172.
	WU Ligang, ZHANG Liang, ZHOU Qian, et al. Research on energy scheduling optimization strategy of micro-grid based on reinforcement learning[J]. Control Engineering of China, 2022, 29 (7): 1162- 1172.
18	马苗苗, 董利鹏, 刘向杰. 基于Q-learning算法的多智能体微电网能量管理策略[J]. 系统仿真学报, 2023, 35 (7): 1487- 1496.
	MA Miaomiao, DONG Lipeng, LIU Xiangjie. Energy management strategy of multi-agent microgrid based on Q-learning algorithm[J]. Journal of System Simulation, 2023, 35 (7): 1487- 1496.
19	冯斌, 胡轶婕, 黄刚, 等. 基于深度强化学习的新型电力系统调度优化方法综述[J]. 电力系统自动化, 2023, 47 (17): 187- 199.
	FENG Bin, HU Yijie, HUANG Gang, et al. Review on optimization methods for new power system dispatch based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47 (17): 187- 199.
20	张华赢, 艾精文, 汪伟. 基于约束型深度强化学习的主动配电网电压控制策略[J]. 电测与仪表, 2023, 60 (5): 159- 166.
	ZHANG Huaying, AI Jingwen, WANG Wei. Volt/Var control strategy for active distribution network based on constrained deep reinforcement learning[J]. Electrical Measurement & Instrumentation, 2023, 60 (5): 159- 166.
21	朱卉乔, 王卫东. 网络拥塞控制的优化研究[J]. 保密科学技术, 2022, (5): 56- 60.
22	胡维昊, 曹迪, 黄琦, 等. 深度强化学习在配电网优化运行中的应用[J]. 电力系统自动化, 2023, 47 (14): 174- 191.
	HU Weihao, CAO Di, HUANG Qi, et al. Application of deep reinforcement learning in optimal operation of distribution network[J]. Automation of Electric Power Systems, 2023, 47 (14): 174- 191.
23	陈兴国, 孙丁源昊, 杨光, 等. 不动点视角下的强化学习算法综述[J]. 计算机学报, 2023, 46 (6): 1246- 1271.
	CHEN Xingguo, SUN Dingyuanhao, YANG Guang, et al. A survey of reinforcement learning algorithms from a fixed point perspective[J]. Chinese Journal of Computers, 2023, 46 (6): 1246- 1271.
24	FISCHETTI M, JO J. Deep neural networks and mixed integer linear optimization[J]. Constraints, 2018: 296–309.

[1]	XU Shijie, HU Bangjie, ZHAO Liang, WANG Pei. Research on Optimal Dispatch with Source-Load Coordination for Micro-energy Grid Based on Energy-Carbon Coupling Model [J]. Electric Power, 2025, 58(4): 1-12.
[2]	WANG Jinfeng, LI Jinpeng, XU Yinliang, LIU Haitao, HE Jinxiong, XU Jianyuan. Distributed Optimization for VPP and Distribution Network Operation Considering Uncertainty and Green Certificate Market [J]. Electric Power, 2025, 58(4): 21-30, 192.
[3]	WANG Guanchao, HUO Yuchong, LI Qun, LI Qiang. Power Optimization of Wind Farms Based on Improved Jensen Model and Deep Reinforcement Learning [J]. Electric Power, 2025, 58(4): 78-89.
[4]	Mingbing LI, Qiang LI, Xiyang GUAN, Haoyang ZHOU, Rui LU, Yankun FENG. Market Oriented Low-Carbon Optimal Scheduling of Virtual Power Plants Considering Multiple User-Side Resources Coordination [J]. Electric Power, 2025, 58(2): 66-76.
[5]	Yumin ZHANG, Yanbin Yin, Xingquan JI, Pingfeng YE, Donglei SUN, Aiquan SONG. Optimal Dispatch of Integrated Electric-Heat Energy System Considering Supply Flexibility of Heat Networks Under Different Operation States [J]. Electric Power, 2025, 58(2): 88-102.
[6]	Yi ZENG, Yi ZHOU, Jixiang LU, Liangcai ZHOU, Ningkai TANG, Hong LI. Voltage Control Based on Multi-Agent Safe Deep Reinforcement Learning [J]. Electric Power, 2025, 58(2): 111-117.
[7]	Ke YANG, Dong WANG, Da LI, Wangjun ZHANG, Ga XIANG, Jun LI. Network Security Risk Assessment Index System and Calculation for Virtual Power Plant [J]. Electric Power, 2024, 57(8): 130-137.
[8]	Zhipeng LV, Zhenhao SONG, Lisheng LI, Yang LIU. Optimization Scheduling of Integrated Energy System Scheduling in Industrial Park containing Electric Vehicles [J]. Electric Power, 2024, 57(4): 25-31.
[9]	Dongshun ZHANG, Hengli QUAN, Hua XIE, Zhihong XU, Yayun TAO, Huisheng WANG. Dispatching Strategy of Park-Level Integrated Energy System Considering Carbon Trading Mechanism and Hydrogen Blending Natural Gas [J]. Electric Power, 2024, 57(2): 183-193.
[10]	Zhongkai YI, Langbo HOU, Ying XU, Yongfeng WU, Zhimin LI, Junfei WU, Teng FENG, Liu HAN. Aggregation and Operation Key Technology of Virtual Power Plant with Flexible Resources in Electricity Market Environment: Review [J]. Electric Power, 2024, 57(12): 82-96.
[11]	Songping XUE, Dequan GAO, Ziyan ZHAO, Yuqian LIN, Zejing GUANG, Dawei ZHANG. Routing Algorithm for Power Communication Networks Based on Serivce Differentiated Transmission Requirements [J]. Electric Power, 2024, 57(11): 183-190.
[12]	Tianqi SONG, Zhipeng LV, Zhenhao SONG, Yunting MA, Zhihui ZHANG, Shan ZHOU, Hao LI. Research and Thinking on the Aggregation and Dispatching Control Framework of Virtual Power Plant's Large Scale Flexible Resources [J]. Electric Power, 2024, 57(1): 2-8.
[13]	Ying ZHOU, Xuefeng BAI, Yang WANG, Min QIU, Chong SUN, Yajie WU, Bin LI. Analysis and Evolution Trend of Temperature-Sensitive Loads for Virtual Power Plant Operation [J]. Electric Power, 2024, 57(1): 9-17.
[14]	Rui ZHU, Yiding OU, Xiaotian LI, Xingyu LEI, Yuqing ZHOU, Poyang ZHANG, Rui OU. Evaluation of Virtual Power Plant Flexibility Resources Based on External Characteristic Equivalence [J]. Electric Power, 2024, 57(1): 30-39.
[15]	Mengfei XIE, Gaoquan MA, Bin LIU, Zhenning PAN, Yunfeng SHANG. Virtual Power Plant Quotation Strategy Based on Information Gap Decision Theory [J]. Electric Power, 2024, 57(1): 40-50.