基于DDQN的电力传感网资源分配算法

doi:10.11930/j.issn.1004-9649.202305116

中国电力 ›› 2023, Vol. 56 ›› Issue (11): 60-66.DOI: 10.11930/j.issn.1004-9649.202305116

• 面向电网设备状态感知的低功耗无线传感网技术及应用 • 上一篇下一篇

基于DDQN的电力传感网资源分配算法

朱雪琼(), 胡成博, 杨景刚, 路永玲

国网江苏省电力有限公司电力科学研究院，江苏南京　211103

收稿日期:2023-05-26 接受日期:2023-10-12 出版日期:2023-11-28 发布日期:2023-11-28
作者简介:朱雪琼（1990—），男，博士，高级工程师，从事电力设备状态智能传感研究，E-mail： 18795897606@163.com
胡成博（1984—），男，硕士，高级工程师，从事智能运检和电力设备物联网通信算法研究
杨景刚（1984—），男，硕士，高级工程师（教授级），从事高压输变电设备状态评估、故障诊断与检测等研究
路永玲（1988—），女，硕士，高级工程师，从事智能运检和电力设备物联网通信算法研究
基金资助:
国家电网有限公司科技项目（与电力传感装置融合的低功耗宽窄融合无线传感网络系统研发及应用，5108-202218280A-2-201-XG）。

DDQN Based Resource Allocation Algorithm for Power Sensor Networks

Xueqiong ZHU(), Chengbo HU, Jinggang YANG, Yongling LU

State Grid Jiangsu Electric Power Co., Ltd. Research Institute, Nanjing 211103, China

Received:2023-05-26 Accepted:2023-10-12 Online:2023-11-28 Published:2023-11-28
Supported by:
This work is supported by Science and Technology Project of SGCC (Development and Application of Low-Power Wideband and Narrowband Fusion Wireless Sensor Network System Integrated with Power Sensing Device, No.5108-202218280A-2-201-XG).

摘要/Abstract

摘要：

电力传感网可以用于对电力网络的设备工作状态和工作环境等信息实时采集和获取，对于电力网络设施的实时监控与快速响应具有重要作用。针对系统在数据排队时延和丢包率上的特殊要求，提出了一种基于强化学习的电力传感网资源分配方案。在资源受限的情况下，通过资源分配算法来优化传感器节点的排队时延和丢包率，并将该优化问题建模为马尔可夫决策过程（Markov decision process，MDP），通过双深度Q网络（double deep Q-learning，DDQN）来对优化目标函数求解。仿真结果与数值分析表明，所提方案在收敛性、排队时延和丢包率等方面的性能均优于基准方案。

关键词: 电力传感网, 资源分配, 马尔可夫决策过程, 双深度Q网络

Abstract:

The power sensor network can collect and obtain information on the power grid equipment's working status and working environment in real time, which plays an important role in monitoring and quick response of the grid facilities. Aiming at the special requirements of system such as data queuing time delay and packet loss rate, this paper proposes a resource allocation scheme based on reinforcement learning (RL) for power sensing networks. Under the resource constraints, the scheme optimizes the queuing time and packet loss rate of sensor nodes through the resource allocation algorithms, and the optimization problem is modeled as a Markov decision process (MDP), which is solved by double deep Q-network(DDQN) algorithm. Simulation results and numerical analysis show that the proposed scheme outperforms the benchmark scheme in convergence, time delay and packet loss rate.

Key words: power sensor network, resource allocation, Markov decision process, double deep Q-network

朱雪琼, 胡成博, 杨景刚, 路永玲. 基于DDQN的电力传感网资源分配算法[J]. 中国电力, 2023, 56(11): 60-66.

Xueqiong ZHU, Chengbo HU, Jinggang YANG, Yongling LU. DDQN Based Resource Allocation Algorithm for Power Sensor Networks[J]. Electric Power, 2023, 56(11): 60-66.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202305116

https://www.electricpower.com.cn/CN/Y2023/V56/I11/60

图/表 6

图 1 电力传感网结构

Fig.1 The power sensor network architecture

图 2 DDQN不同参数的收敛性能

Fig.2 The convergence performance of different DDQN parameters

表 1 DDQN和DQN神经网络参数

Table 1 DDQN and DQN neural network parameters

算法	学习率	折扣因子	贪婪因子	最大训练次数	经验池大小	最小取样大小
DDQN	0.0050	0.9	0.90	1000	10000	64
DQN	0.0001	0.9	0.99	1000	10000	64

图 3 DDQN算法和DQN算法的收敛性能

Fig.3 The convergence performance of DDQN and DQN

图 4 DDQN和DQN的加权排队时延性能

Fig.4 The weighted queuing time of DDQN and DQN

图 5 DDQN和DQN的加权丢包个数

Fig.5 The weighted packet loss of DDQN and DQN

参考文献 25

1	PENG F N, LV L, CHEN W R, et al. A multi-objective evolutionary algorithm for wireless sensor networks node deployment[C]//2022 IEEE 8th International Conference on Computer and Communications (ICCC). Chengdu, China. IEEE, 2023: 983–988.
2	MANUEL E M, PANKAJAKSHAN V, MOHAN M T. Data aggregation in low-power wireless sensor networks with discrete transmission ranges: sensor signal aggregation over graph[J]. IEEE Sensors Journal, 2022, 22 (21): 21135- 21144. DOI
3	谢伟, 王哲斐, 蔡秋烨, 等. 基于最优系统能效的配电网规划方法[J]. 电力工程技术, 2021, 40 (2): 128- 134.
	XIE Wei, WANG Zhefei, CAI Qiuye, et al. Distribution network planning based on optimal system efficiency[J]. Electric Power Engineering Technology, 2021, 40 (2): 128- 134.
4	林日晖, 陈友立. 基于深度Q学习的含用户侧储能微电网频率-电压数字化智能控制策略[J]. 中国电力, 2022, 55 (12): 43- 50. DOI
	LIN Rihui, CHEN Youli. Frequency-voltage digital intelligent control strategy of microgrid with UserSide energy storage based on deep Q-learning[J]. Electric Power, 2022, 55 (12): 43- 50. DOI
5	梅冰笑, 周金辉, 孙翔. 基于知识图谱与细胞自动机模型的配电网信息系统风险分析[J]. 中国电力, 2022, 55 (10): 23- 31, 44. DOI
	MEI Bingxiao, ZHOU Jinhui, SUN Xiang. Analysis of distribution network information risks based on knowledge graph and cellular automata[J]. Electric Power, 2022, 55 (10): 23- 31, 44. DOI
6	LAM T D, HUYNH D T. Improved algorithms in directional wirelesssensor networks[C]//2022 IEEE Wireless Communications and Networking Conference (WCNC). Austin, TX, USA. IEEE, 2022: 2352–2357.
7	MAJID M, HABIB S, JAVED A R, et al. Applications of wireless sensor networks and Internet of Things frameworks in the industry revolution 4.0: a systematic literature review[J]. Sensors, 2022, 22 (6): 2087. DOI
8	LILHORE U K, KHALAF O I, SIMAIYA S, et al. A depth-controlled and energy-efficient routing protocol for underwater wireless sensor networks[J]. International Journal of Distributed Sensor Networks, 2022, 18(9): 155013292211171.
9	TEMENE N, SERGIOU C, GEORGIOU C, et al. A survey on mobility in wireless sensor networks[J]. Ad Hoc Networks, 2022, 125, 102726. DOI
10	YETGIN H, CHEUNG K T K, EL-HAJJAR M, et al. A survey of network lifetime maximization techniques in wireless sensor networks[J]. IEEE Communications Surveys & Tutorials, 2017, 19 (2): 828- 854.
11	杜欣. 无线传感网在电力系统中关键技术的研究和应用[D]. 北京: 华北电力大学, 2012.
	DU Xin. Research and application of key technology of wireless sensor networks in power system[D]. Beijing: North China Electric Power University, 2012.
12	ESMAEILI H, HAKAMI V, BIDGOLI B M, et al. Application-specific clustering in wireless sensor networks using combined fuzzy firefly algorithm and random forest[J]. Expert Syst Appl, 2022, 210, 118365. DOI
13	ZHAI D S, WANG C, ZHANG R N, et al. Energy-saving deployment optimization and resource management for UAV-assisted wireless sensor networks with NOMA[J]. IEEE Transactions on Vehicular Technology, 2022, 71 (6): 6609- 6623. DOI
14	ZHAN C, ZENG Y, ZHANG R. Energy-efficient data collection in UAV enabled wireless sensor network[J]. IEEE Wireless Communications Letters, 2018, 7 (3): 328- 331. DOI
15	王姣姣. 无线传感器网络分布式资源分配优化算法研究[D]. 秦皇岛: 燕山大学, 2020.
	WANG Jiaojiao. Study on distributed resource allocation optimization algorithm for wireless sensor network[D]. Qinhuangdao: Yanshan University, 2020.
16	王立元. 基于多目标优化的无线传感器网络资源分配算法[D]. 秦皇岛: 燕山大学, 2019.
	WANG Liyuan. Resource allocation algorithm based on multi-objective optimization for wireless sensor network[D]. Qinhuangdao: Yanshan University, 2019.
17	李桐. 基于能量传输的无线传感器网络资源分配策略研究[D]. 绵阳: 西南科技大学, 2019.
	LI Tong. Research on resource allocation strategy of wireless sensor networks based on energy transfer[D]. Mianyang: Southwest University of Science and Technology, 2019.
18	LV P. Research on resource scheduling algorithms in the new 5G network architecture[D]. Beijing: Beijing University of Posts and Telecommunications, 2021.
19	LI K W, ZHANG T, WANG R, et al. Deep reinforcement learning for combinatorial optimization: covering salesman problems[J]. IEEE Transactions on Cybernetics, 2022, 52 (12): 13142- 13155. DOI
20	ZHU C X, DASTANI M, WANG S H. A survey of multi-agent reinforcement learning with communication[EB/OL]. 2022: arXiv: 2203.08975.https://arxiv.org/abs/2203.08975.pdf.
21	LUO F M, XU T, LAI H, et al. A survey on model-based reinforcement learning[EB/OL]. 2022: arXiv: 2206.09328. https://arxiv.org/abs/2206.09328.pdf.
22	WU D, LEI Y, HE M E, et al. Deep reinforcement learning-based path control and optimization for unmanned ships[J]. Wireless Communications and Mobile Computing, 2022, 2022, 1- 8.
23	GANESH A H, XU B. A review of reinforcement learning based energy management systems for electrified powertrains: progress, challenge, and potential solution[J]. Renewable and Sustainable Energy Reviews, 2022, 154, 111833. DOI
24	ZHAO N, YE Z Y, PEI Y Y, et al. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing[J]. IEEE Transactions on Wireless Communications, 2022, 21 (9): 6949- 6960. DOI
25	ZHENG J J, LI K, MHAISEN N, et al. Federated learning for online resource allocation in mobile edge computing: a deep reinforcement learning approach[C]//2023 IEEE Wireless Communications and Networking Conference (WCNC). Glasgow, United Kingdom. IEEE, 2023: 1–6.

基于DDQN的电力传感网资源分配算法

DDQN Based Resource Allocation Algorithm for Power Sensor Networks

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 25

相关文章 1

编辑推荐

Metrics

模态框（Modal）标题

基于DDQN的电力传感网资源分配算法

DDQN Based Resource Allocation Algorithm for Power Sensor Networks

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 25

相关文章 1

编辑推荐

Metrics