基于深度强化学习的建筑能源系统优化策略

doi:10.11930/j.issn.1004-9649.202210043

中国电力 ›› 2023, Vol. 56 ›› Issue (6): 114-122.DOI: 10.11930/j.issn.1004-9649.202210043

基于深度强化学习的建筑能源系统优化策略

石文喆^1,2, 李冰洁^1,2, 尤培培³, 张泠^1,2

1. 湖南大学土木工程学院，湖南长沙 410082;
2. 建筑安全与节能教育部重点实验室，湖南长沙 410082;
3. 国网能源研究院有限公司，北京 102209

收稿日期:2022-10-13 修回日期:2022-11-04 发布日期:2023-07-04
作者简介:石文喆（1998—），男，硕士研究生，从事深度强化学习在建筑能源系统中的理论研究，E-mail：shiwenzhe@hnu.edu.cn;张泠张泠（1969—），女，通信作者，博士，教授，从事建筑能源系统柔性用能研究，E-mail：zhangling@hnu.edu.cn
基金资助:
国家自然科学基金资助项目（51878253）。

Optimization Strategy of Building Energy System Based on Deep Reinforcement Learning

SHI Wenzhe^1,2, LI Bingjie^1,2, YOU Peipei³, ZHANG Ling^1,2

1. College of Civil Engineering, Hunan University, Changsha 410082, China;
2. Key Laboratory of Building Safety and Energy Conservation, Ministry of Education, Changsha 410082, China;
3. State Grid Energy Research Institute Co., Ltd., Beijing 102209, China

Received:2022-10-13 Revised:2022-11-04 Published:2023-07-04
Supported by:
This work is supported by National Natural Science Foundation of China (No.51878253).

摘要/Abstract

摘要： 针对建筑能源系统中需求侧的负荷不确定性与供给侧的可再生能源随机性，提出一种基于深度强化学习的建筑能源系统管理优化策略。首先，搭建能源系统供需侧研究框架并建立设备模型。然后，将实时阶段下的建筑能源管理问题构建为马尔可夫决策过程，利用深度强化学习理论，以最小化用电成本、保证室内热舒适水平和最大化消纳可再生能源为优化目标，采用决斗双重深度Q网络算法进行训练，得到训练后的算法可以根据实时环境参数做出自适应控制决策。最后，通过在建筑能源系统案例中的应用，将该策略与传统的基于规则的控制策略相比较，结果表明，所提出的优化策略使用电成本降低11.03%，热不舒适时长降低89.62%，未消纳光伏发电量降低10.43%。

关键词: 建筑能源系统, 深度强化学习, 实时控制策略, 储能优化

Abstract: Aiming at the load uncertainty on the demand side of the building energy system and the randomness of renewable energy on the supply side, a building energy system management optimization strategy is proposed based on deep reinforcement learning. Firstly, a supply-demand side research framework for the energy system and device model is built. The building energy management problem under the real-time stage is constructed as Markov decision-making process, and the deep reinforcement learning theory is used to minimize the cost of electricity, ensure the indoor heat comfort level and maximize the consumption of renewable energy as the optimization goals, and the duel dual deep Q network algorithm is used for model training, and the trained model can make adaptive control decisions according to real-time environmental parameters. Finally, through the application in the building energy system case, the results show that the proposed optimization strategy reduces the cost of electricity by 11.03%, the duration of thermal discomfort by 89.62%, and the amount of unconsumed photovoltaic power generation by 10.43%, comparing with the traditional rule-based control strategy.

Key words: building energy systems, deep reinforcement learning, real-time control strategy, energy storage optimization

石文喆, 李冰洁, 尤培培, 张泠. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56(6): 114-122.

SHI Wenzhe, LI Bingjie, YOU Peipei, ZHANG Ling. Optimization Strategy of Building Energy System Based on Deep Reinforcement Learning[J]. Electric Power, 2023, 56(6): 114-122.

导出引用管理器 EndNote|Ris|BibTeX

链接本文: https://www.electricpower.com.cn/CN/10.11930/j.issn.1004-9649.202210043

https://www.electricpower.com.cn/CN/Y2023/V56/I6/114

参考文献

[1] 中国建筑能耗研究报告2020[J]. 建筑节能(中英文), 2021, 49(2): 1–6.
[2] GASSER J, CAI H M, KARAGIANNOPOULOS S, et al. Predictive energy management of residential buildings while self-reporting flexibility envelope[J]. Applied Energy, 2021, 288: 116653.
[3] MASON K, GRIJALVA S. A review of reinforcement learning for autonomous building energy management[J]. Computers and Electrical Engineering, 2019, 78(C): 300–312.
[4] XIONG L, LI P, WANG Z, et al. Multi-agent based multi objective renewable energy management for diversified community power consumers[J]. Applied Energy, 2020, 259: 114140.
[5] 曹雅琦, 赵波, 王丽婕, 等. 基于遗传蚁群的光储电站运行效益提升策略研究[J]. 中国电力, 2022, 55(2): 9–18
CAO Yaqi, ZHAO Bo, WANG Lijie, et al. Research on operational benefit improvement strategy of optical storage power station based on genetic ant colony algorithm[J]. Electric Power, 2022, 55(2): 9–18
[6] 王子琪, 张慧媛, 许军, 等. 基于改进人工蜂群算法的区域电网储能系统能量管理优化策略[J]. 中国电力, 2022, 55(9): 16–22, 55
WANG Ziqi, ZHANG Huiyuan, XU Jun, et al. An energy management optimization strategy for regional power grid energy storage system based on improved artificial bee colony algorithm[J]. Electric Power, 2022, 55(9): 16–22, 55
[7] LIU J, LIU Z, WU Y, et al. Impact of climate on photovoltaic battery energy storage system optimization[J]. Renewable Energy, 2022, 191: 625–638.
[8] 苏鹏伟, 赵军, 邓帅, 等. 基于预测技术的建筑可再生能源系统匹配特性分析[J]. 太阳能学报, 2019, 40(8): 2360–2367
SU Pengwei, ZHAO Jun, DENG Shuai, et al. Analysis of matching performance of building renewable energy system based on forecasting technology[J]. Acta Energiae Solaris Sinica, 2019, 40(8): 2360–2367
[9] 黄猛, 赵柏扬, 李勇, 等. 以光伏空调为中心的建筑能源系统控制优化[J]. 制冷技术, 2019, 39(1): 1–5
HUANG Meng, ZHAO Baiyang, LI Yong, et al. Control optimization of building energy systems centered on photovoltaic air conditioners[J]. Chinese Journal of Refrigeration Technology, 2019, 39(1): 1–5
[10] MOCANU E, MOCANU D C, NGUYEN P H, et al. On-line building energy optimization using deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 3698–3708.
[11] WANG Z, HONG T. Reinforcement learning for building controls: the opportunities and challenges[J]. Applied Energy, 2020, 269: 115036.
[12] 郑洁云, 宋倩芸, 吴桂联, 等. 基于Q学习的区域综合能源系统低碳运行策略[J]. 电力科学与技术学报, 2022, 37(2): 106–115, 128
ZHENG Jieyun, SONG Qianyun, WU Guilian, et al. Low-carbon operation strategy of regional integrated energy system based on the Q learning algorithm[J]. Journal of Electric Power Science and Technology, 2022, 37(2): 106–115, 128
[13] 沈国辉, 赵荣生, 董晓, 等. 基于多信息交互与深度强化学习的电动汽车充电导航策略[J]. 南方电网技术, 2022, 16(1): 108–116
SHEN Guohui, ZHAO Rongsheng, DONG Xiao, et al. Electric vehicle charging navigation strategy based on multi-information interaction and deep reinforcement learning[J]. Southern Power System Technology, 2022, 16(1): 108–116
[14] 毛亚哲, 何柏娜, 王德顺, 等. 基于改进深度强化学习的智能微电网群控制优化方法[J]. 智慧电力, 2021, 49(3): 19–25, 58
MAO Yazhe, HE Baina, WANG Deshun, et al. Optimization method for smart multi-microgrid control based on improved deep reinforcement learning[J]. Smart Power, 2021, 49(3): 19–25, 58
[15] 蔺伟山, 王小君, 孙庆凯, 等. 不确定性环境下基于深度强化学习的综合能源系统动态调度[J]. 电力系统保护与控制, 2022, 50(18): 50–60
LIN Weishan, WANG Xiaojun, SUN Qingkai, et al. Dynamic dispatch of an integrated energy system based on deep reinforcement learning in an uncertain environment[J]. Power System Protection and Control, 2022, 50(18): 50–60
[16] 傅质馨, 李潇逸, 朱俊澎, 等. 基于马尔科夫决策过程的家庭能量管理智能优化策略[J]. 电力自动化设备, 2020, 40(7): 141–152
FU Zhixin, LI Xiaoyi, ZHU Junpeng, et al. Intelligent optimization strategy of home energy management based on Markov decision process[J]. Electric Power Automation Equipment, 2020, 40(7): 141–152
[17] 杨思明, 单征, 丁煜, 等. 深度强化学习研究综述[J]. 计算机工程, 2021, 47(12): 19–29
YANG Siming, SHAN Zheng, DING Yu, et al. Survey of research on deep reinforcement learning[J]. Computer Engineering, 2021, 47(12): 19–29
[18] LIU Y K, ZHANG D X, BENG GOOI H. Optimization strategy based on deep reinforcement learning for home energy management[J]. CSEE Journal of Power and Energy Systems, 2020, 6(3): 572–582.
[19] 梁宏, 李鸿鑫, 张华赢, 等. 基于深度强化学习的微网储能系统控制策略研究[J]. 电网技术, 2021, 45(10): 3869–3877
LIANG Hong, LI Hongxin, ZHANG Huaying, et al. Control strategy of microgrid energy storage system based on deep reinforcement learning[J]. Power System Technology, 2021, 45(10): 3869–3877
[20] WU Y, LIU Z, LIU J, et al. Optimal battery capacity of grid-connected PV-battery systems considering battery degradation[J]. Renewable Energy, 2022, 181: 10–23.
[21] LIU J, YANG X, LIU Z, et al. Investigation and evaluation of building energy flexibility with energy storage system in hot summer and cold winter zones[J]. Journal of Energy Storage, 2022, 46: 103877.
[22] BACHER P, MADSEN H. Identifying suitable models for the heat dynamics of buildings[J]. Energy & Buildings, 2011, 43(7): 1511–1522.
[23] 王丰, 顾佼佼, 曹倩, 等. 马尔可夫过程的传导转移概率及工程研究应用[J]. 华中师范大学学报(自然科学版), 2018, 52(6): 773–777
WANG Feng, GU Jiaojiao, CAO Qian, et al. The transfer probability of Markov process and its application in engineering research[J]. Journal of Central China Normal University (Natural Sciences), 2018, 52(6): 773–777
[24] 刘建伟, 高峰, 罗雄麟. 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报, 2019, 42(6): 1406–1438
LIU Jianwei, GAO Feng, LUO Xionglin. Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019, 42(6): 1406–1438
[25] 杨朋朋, 王蓓蓓, 胥鹏, 等. 不完全信息下基于深度双Q网络的发电商三段式竞价策略[J]. 中国电力, 2021, 54(11): 47–58
YANG Pengpeng, WANG Beibei, XU Peng, et al. Three-stage bidding strategy of generation company based on double deep Q-network under incomplete information condition[J]. Electric Power, 2021, 54(11): 47–58

基于深度强化学习的建筑能源系统优化策略

Optimization Strategy of Building Energy System Based on Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

编辑推荐

Metrics

[1]	曾仪, 周毅, 陆继翔, 周良才, 唐宁恺, 李红. 基于多智能体安全深度强化学习的电压控制[J]. 中国电力, 2025, 58(2): 111-117.
[2]	徐峰亮, 王克谦, 王文豪, 王鹏, 王文烨, 张帅, 赵凤展. 计及激励型需求响应的低压配电网混合储能优化配置[J]. 中国电力, 2024, 57(6): 90-101.
[3]	薛松萍, 高德荃, 赵子岩, 林彧茜, 广泽晶, 张大卫. 基于业务差异化传输需求下的电力通信网路由算法[J]. 中国电力, 2024, 57(11): 183-190.
[4]	张超, 赵冬梅, 季宇, 张颖. 基于改进深度Q网络的虚拟电厂实时优化调度[J]. 中国电力, 2024, 57(1): 91-100.
[5]	宋伟业, 刘灵玥, 阎洁, 王航宇, 何书凯, 韩爽, 王明辉, 刘永前. 基于深度强化学习的海上风电集群自进化功率平滑控制方法[J]. 中国电力, 2023, 56(3): 36-46.
[6]	张淑兴, 马驰, 杨志学, 王尧, 吴昊, 任洲洋. 基于深度确定性策略梯度算法的风光储系统联合调度策略[J]. 中国电力, 2023, 56(2): 68-76.
[7]	孙东磊, 王耀, 张惠雯, 刘蕊, 石冰珂. 基于矩差分析的配电网分布式储能优化配置[J]. 中国电力, 2023, 56(12): 31-40.
[8]	周庆锋, 王思淳, 李德鑫, 刘佳琪, 李同. 基于DDPG的风电场动态参数智能校核知识学习模型[J]. 中国电力, 2022, 55(5): 32-38.
[9]	任大伟, 金晨, 肖晋宇, 侯金鸣, 杜尔顺, 周原冰. 计及灵活性基于时序的“十四五”储能需求分析[J]. 中国电力, 2021, 54(8): 190-198.
[10]	杨朋朋, 王蓓蓓, 胥鹏, 王高琴, 郑亚先. 不完全信息下基于深度双Q网络的发电商三段式竞价策略[J]. 中国电力, 2021, 54(11): 47-58.

模态框（Modal）标题

基于深度强化学习的建筑能源系统优化策略

Optimization Strategy of Building Energy System Based on Deep Reinforcement Learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

编辑推荐

Metrics