DDQN Based Resource Allocation Algorithm for Power Sensor Networks

doi:10.11930/j.issn.1004-9649.202305116

Abstract

Abstract:

The power sensor network can collect and obtain information on the power grid equipment's working status and working environment in real time, which plays an important role in monitoring and quick response of the grid facilities. Aiming at the special requirements of system such as data queuing time delay and packet loss rate, this paper proposes a resource allocation scheme based on reinforcement learning (RL) for power sensing networks. Under the resource constraints, the scheme optimizes the queuing time and packet loss rate of sensor nodes through the resource allocation algorithms, and the optimization problem is modeled as a Markov decision process (MDP), which is solved by double deep Q-network(DDQN) algorithm. Simulation results and numerical analysis show that the proposed scheme outperforms the benchmark scheme in convergence, time delay and packet loss rate.

Key words: power sensor network, resource allocation, Markov decision process, double deep Q-network

Xueqiong ZHU, Chengbo HU, Jinggang YANG, Yongling LU. DDQN Based Resource Allocation Algorithm for Power Sensor Networks[J]. Electric Power, 2023, 56(11): 60-66.

Add to citation manager EndNote|Ris|BibTeX

URL: https://www.electricpower.com.cn/EN/10.11930/j.issn.1004-9649.202305116

https://www.electricpower.com.cn/EN/Y2023/V56/I11/60

Figures/Tables 6

Fig.1 The power sensor network architecture

Fig.2 The convergence performance of different DDQN parameters

Table 1 DDQN and DQN neural network parameters

算法	学习率	折扣因子	贪婪因子	最大训练次数	经验池大小	最小取样大小
DDQN	0.0050	0.9	0.90	1000	10000	64
DQN	0.0001	0.9	0.99	1000	10000	64

Fig.3 The convergence performance of DDQN and DQN

Fig.4 The weighted queuing time of DDQN and DQN

Fig.5 The weighted packet loss of DDQN and DQN

References 25

1	PENG F N, LV L, CHEN W R, et al. A multi-objective evolutionary algorithm for wireless sensor networks node deployment[C]//2022 IEEE 8th International Conference on Computer and Communications (ICCC). Chengdu, China. IEEE, 2023: 983–988.
2	MANUEL E M, PANKAJAKSHAN V, MOHAN M T. Data aggregation in low-power wireless sensor networks with discrete transmission ranges: sensor signal aggregation over graph[J]. IEEE Sensors Journal, 2022, 22 (21): 21135- 21144. DOI
3	谢伟, 王哲斐, 蔡秋烨, 等. 基于最优系统能效的配电网规划方法[J]. 电力工程技术, 2021, 40 (2): 128- 134.
	XIE Wei, WANG Zhefei, CAI Qiuye, et al. Distribution network planning based on optimal system efficiency[J]. Electric Power Engineering Technology, 2021, 40 (2): 128- 134.
4	林日晖, 陈友立. 基于深度Q学习的含用户侧储能微电网频率-电压数字化智能控制策略[J]. 中国电力, 2022, 55 (12): 43- 50. DOI
	LIN Rihui, CHEN Youli. Frequency-voltage digital intelligent control strategy of microgrid with UserSide energy storage based on deep Q-learning[J]. Electric Power, 2022, 55 (12): 43- 50. DOI
5	梅冰笑, 周金辉, 孙翔. 基于知识图谱与细胞自动机模型的配电网信息系统风险分析[J]. 中国电力, 2022, 55 (10): 23- 31, 44. DOI
	MEI Bingxiao, ZHOU Jinhui, SUN Xiang. Analysis of distribution network information risks based on knowledge graph and cellular automata[J]. Electric Power, 2022, 55 (10): 23- 31, 44. DOI
6	LAM T D, HUYNH D T. Improved algorithms in directional wirelesssensor networks[C]//2022 IEEE Wireless Communications and Networking Conference (WCNC). Austin, TX, USA. IEEE, 2022: 2352–2357.
7	MAJID M, HABIB S, JAVED A R, et al. Applications of wireless sensor networks and Internet of Things frameworks in the industry revolution 4.0: a systematic literature review[J]. Sensors, 2022, 22 (6): 2087. DOI
8	LILHORE U K, KHALAF O I, SIMAIYA S, et al. A depth-controlled and energy-efficient routing protocol for underwater wireless sensor networks[J]. International Journal of Distributed Sensor Networks, 2022, 18(9): 155013292211171.
9	TEMENE N, SERGIOU C, GEORGIOU C, et al. A survey on mobility in wireless sensor networks[J]. Ad Hoc Networks, 2022, 125, 102726. DOI
10	YETGIN H, CHEUNG K T K, EL-HAJJAR M, et al. A survey of network lifetime maximization techniques in wireless sensor networks[J]. IEEE Communications Surveys & Tutorials, 2017, 19 (2): 828- 854.
11	杜欣. 无线传感网在电力系统中关键技术的研究和应用[D]. 北京: 华北电力大学, 2012.
	DU Xin. Research and application of key technology of wireless sensor networks in power system[D]. Beijing: North China Electric Power University, 2012.
12	ESMAEILI H, HAKAMI V, BIDGOLI B M, et al. Application-specific clustering in wireless sensor networks using combined fuzzy firefly algorithm and random forest[J]. Expert Syst Appl, 2022, 210, 118365. DOI
13	ZHAI D S, WANG C, ZHANG R N, et al. Energy-saving deployment optimization and resource management for UAV-assisted wireless sensor networks with NOMA[J]. IEEE Transactions on Vehicular Technology, 2022, 71 (6): 6609- 6623. DOI
14	ZHAN C, ZENG Y, ZHANG R. Energy-efficient data collection in UAV enabled wireless sensor network[J]. IEEE Wireless Communications Letters, 2018, 7 (3): 328- 331. DOI
15	王姣姣. 无线传感器网络分布式资源分配优化算法研究[D]. 秦皇岛: 燕山大学, 2020.
	WANG Jiaojiao. Study on distributed resource allocation optimization algorithm for wireless sensor network[D]. Qinhuangdao: Yanshan University, 2020.
16	王立元. 基于多目标优化的无线传感器网络资源分配算法[D]. 秦皇岛: 燕山大学, 2019.
	WANG Liyuan. Resource allocation algorithm based on multi-objective optimization for wireless sensor network[D]. Qinhuangdao: Yanshan University, 2019.
17	李桐. 基于能量传输的无线传感器网络资源分配策略研究[D]. 绵阳: 西南科技大学, 2019.
	LI Tong. Research on resource allocation strategy of wireless sensor networks based on energy transfer[D]. Mianyang: Southwest University of Science and Technology, 2019.
18	LV P. Research on resource scheduling algorithms in the new 5G network architecture[D]. Beijing: Beijing University of Posts and Telecommunications, 2021.
19	LI K W, ZHANG T, WANG R, et al. Deep reinforcement learning for combinatorial optimization: covering salesman problems[J]. IEEE Transactions on Cybernetics, 2022, 52 (12): 13142- 13155. DOI
20	ZHU C X, DASTANI M, WANG S H. A survey of multi-agent reinforcement learning with communication[EB/OL]. 2022: arXiv: 2203.08975.https://arxiv.org/abs/2203.08975.pdf.
21	LUO F M, XU T, LAI H, et al. A survey on model-based reinforcement learning[EB/OL]. 2022: arXiv: 2206.09328. https://arxiv.org/abs/2206.09328.pdf.
22	WU D, LEI Y, HE M E, et al. Deep reinforcement learning-based path control and optimization for unmanned ships[J]. Wireless Communications and Mobile Computing, 2022, 2022, 1- 8.
23	GANESH A H, XU B. A review of reinforcement learning based energy management systems for electrified powertrains: progress, challenge, and potential solution[J]. Renewable and Sustainable Energy Reviews, 2022, 154, 111833. DOI
24	ZHAO N, YE Z Y, PEI Y Y, et al. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing[J]. IEEE Transactions on Wireless Communications, 2022, 21 (9): 6949- 6960. DOI
25	ZHENG J J, LI K, MHAISEN N, et al. Federated learning for online resource allocation in mobile edge computing: a deep reinforcement learning approach[C]//2023 IEEE Wireless Communications and Networking Conference (WCNC). Glasgow, United Kingdom. IEEE, 2023: 1–6.

[1]	YAO Jun, WU Yongfei, WANG Yali, HE Jiao, DAI Shuangfeng. Impact Analysis of Two-Part Tariff Policy Executive Mode on Market Resource Allocation Efficiency [J]. Electric Power, 2023, 56(3): 154-161.
[2]	YAO Pengchao, YAN Bingjing, Hao Weijie, YANG Qiang. An Intrusion Tolerance Assessment Method for Cyber-Physical Power System [J]. Electric Power, 2022, 55(4): 13-22.
[3]	LU Xu, CHEN Ying, XU Zhongping, ZHANG Haiquan, MU Chunfang, CHEN Kai, SUN Yi. Joint Demand Response and Task Offloading Strategy for 5G Edge Computing Network [J]. Electric Power, 2022, 55(10): 209-218.