Safe SAC Model for RDMA Congestion Control with Queue Length Barrier Constraints
Authors
Lena Kaleta, Hanna Kędziora, Michał KosińskiFiles
Abstract
Safe rate adaptation for remote direct memory access requires concurrent foreground throughput, transfer completion and bounded switch-queue occupancy under delayed telemetry. This study evaluates a soft actor-critic controller whose proposed sender rates pass through a queue-length barrier projection before execution. Four sender work queues feed a shared fluid-model bottleneck. Fourteen actor descriptors summarize delayed occupancy, congestion feedback, pending work and previous rates, while a trusted counter and admission history form a conservative current-queue envelope. A 100 Gbit/s reference link and 50 microsecond control slot define 625,000 bytes per traffic unit. The admissible aggregate rate uses a 0.90-unit service lower bound, 0.10-unit background upper allowance and barrier gain 0.35. Training uses 12,000 transitions, and evaluation uses twelve independently seeded traffic episodes across four dimensionless offered-load levels. Safe SAC yields mean foreground goodput 79.55 Gbit/s and mean episode-level 99th percentile flow completion time 2.863 ms. Relative to additive-increase multiplicative-decrease control, goodput rises 11.30% and completion tail falls 34.55%; mean queue 99th percentile falls 52.11%. Fixed-proposal filtering through the same barrier also has zero observed nominal safety-boundary violations, while learned proposals improve goodput by 2.06% and completion tail by 17.53% against that comparator. Projection modifies 99.22% of learned proposals versus 25.65% of fixed proposals. When actual service falls below the assumed bound, violation fractions increase, reaching 30.73% at service multiplier 0.90. Median local actor-plus-projection time is 27.30 microseconds, with three of 1,000 calls above the 50 microsecond slot. These results separate policy efficiency, conditional barrier protection and execution cost within the declared queue simulator.
References
Hu, J., Shen, H., Liu, X., & Wang, J. (2024). RDMA transports in datacenter networks: Survey. IEEE Network, 38(6), 380–387. https://doi.org/10.1109/MNET.2024.3397781
Zhang, J., Wang, Y., Zhong, X., Yu, M., Pan, H., Zhang, Y., Guan, Z., Che, B., Wan, Z., Pan, T., & Huang, T. (2024). PACC: A proactive CNP generation scheme for datacenter networks. IEEE/ACM Transactions on Networking, 32(3), 2586–2599. https://doi.org/10.1109/TNET.2024.3361771
Tian, H., Liao, X., Zeng, C., Sun, D., Zhang, J., & Chen, K. (2024). Efficient DRL-based congestion control with ultra-low overhead. IEEE/ACM Transactions on Networking, 32(3), 1888–1903. https://doi.org/10.1109/TNET.2023.3330737
Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., & Knoll, A. (2024). A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 11216–11235. https://doi.org/10.1109/TPAMI.2024.3457538
Liu, Y., Yao, X., Yang, Z., & Li, W. (2024). A multi-queue-based ECN marking strategy for multi-class QoS guarantee in programmable networks. PeerJ Computer Science, 10, e2382. https://doi.org/10.7717/peerj-cs.2382
Du, Z., Zheng, J., Yu, H., Zhang, H., & Chen, G. (2025). Libra: A congestion control framework for diverse application preferences and network conditions. IEEE/ACM Transactions on Networking, 33(2), 510–525. https://doi.org/10.1109/TNET.2024.3492096
Sun, H., Huang, Q., Lee, P. P. C., Bai, W., Zhu, F., & Bao, Y. (2024). Distributed network telemetry with resource efficiency and full accuracy. IEEE/ACM Transactions on Networking, 32(3), 1857–1872. https://doi.org/10.1109/TNET.2023.3327345
Giacomoni, L., & Parisis, G. (2024). Reinforcement learning-based congestion control: A systematic evaluation of fairness, efficiency and responsiveness. In IEEE INFOCOM 2024—IEEE Conference on Computer Communications (pp. 1451–1460). IEEE. https://doi.org/10.1109/INFOCOM52122.2024.10621288
Vielhaus, C. L., von Lengerke, C., Latzko, V., Rischke, J., Reisslein, M., & Fitzek, F. H. P. (2026). Evaluating transport layer congestion control algorithms: A comprehensive survey. IEEE Communications Surveys & Tutorials, 28, 278–340. https://doi.org/10.1109/COMST.2025.3533303
Tan, Z., & Hong, P. (2024). FRCS: Congestion control in datacenter networks based on fair rate calculation by switches. In 2024 10th International Conference on Computer and Communications (ICCC) (pp. 1445–1450). IEEE. https://doi.org/10.1109/ICCC62609.2024.10941816
Che, B., Wang, Y., Wan, Z., Chen, Y., Wang, Z., Tian, Y., Zhao, J., & Zhang, J. (2026). A fast-converging low-latency RDMA congestion control algorithm. IEEE Network, 40(1), 219–226. https://doi.org/10.1109/MNET.2025.3589913
Zhao, Y., Wang, S., Han, S., Zhou, D., Peng, G., & Huang, T. (2024). RACC: Rapid and accurate INT-based congestion control in RDMA network. In GLOBECOM 2024—2024 IEEE Global Communications Conference (pp. 3811–3816). IEEE. https://doi.org/10.1109/GLOBECOM52923.2024.10901068
Boussaoud, K., En-Nouaary, A., & Ayache, M. (2025). Adaptive congestion detection and traffic control in software-defined networks via data-driven multi-agent reinforcement learning. Computers, 14(6), 236. https://doi.org/10.3390/computers14060236
Banerjee, C., Chen, Z., & Noman, N. (2024). Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experiences. IEEE Transactions on Neural Networks and Learning Systems, 35(3), 3121–3129. https://doi.org/10.1109/TNNLS.2022.3174051
Jha, M. S., & Kiumarsi, B. (2025). Off-policy safe reinforcement learning for nonlinear discrete-time systems. Neurocomputing, 611, 128677. https://doi.org/10.1016/j.neucom.2024.128677
Yang, Y., Zhang, Y., Zou, W., Chen, J., Yin, Y., & Li, S. E. (2024). Synthesizing control barrier functions with feasible region iteration for safe reinforcement learning. IEEE Transactions on Automatic Control, 69(4), 2713–2720. https://doi.org/10.1109/TAC.2023.3336263
Kong, X., Xia, Y., Sun, Z., Zhai, D.-H., Deng, Y., & Zhang, S. (2025). Differential high order control barrier function-based safe reinforcement learning. IEEE Robotics and Automation Letters, 10(7), 7524–7531. https://doi.org/10.1109/LRA.2025.3575310
Alan, A., Molnar, T. G., Ames, A. D., & Orosz, G. (2025). Generalizing robust control barrier functions from a controller design perspective. IEEE Open Journal of Control Systems, 4, 54–69. https://doi.org/10.1109/OJCSYS.2025.3529364
Wang, Z., Wan, X., Li, L., Sun, Y., Xie, P., Wei, X., Ning, Q., Zhang, J., & Chen, K. (2024). Fast, scalable, and accurate rate limiter for RDMA NICs. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 568–580). ACM. https://doi.org/10.1145/3651890.3672215
Pichetti, L., De Sensi, D., Sivalingam, K., Nassyr, S., Cesarini, D., Turisini, M., Pleiter, D., Artigiani, A., & Vella, F. (2024). Benchmarking Ethernet interconnect for HPC/AI workloads. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 869–875). IEEE. https://doi.org/10.1109/SCW63240.2024.00124
Wan, Z., Zhang, J., Yu, M., Liu, J., Yao, J., Zhao, X., & Huang, T. (2024). BiCC: Bilateral congestion control in cross-datacenter RDMA networks. In IEEE INFOCOM 2024—IEEE Conference on Computer Communications (pp. 1381–1390). IEEE. https://doi.org/10.1109/INFOCOM52122.2024.10621412
Wojnar, M., Szott, S., Rusek, K., & Ciezobka, W. (2024). Reinforced-lib: Rapid prototyping of reinforcement learning solutions. SoftwareX, 26, 101706. https://doi.org/10.1016/j.softx.2024.101706
Flageat, M., Lim, B., & Cully, A. (2024). Beyond expected return: Accounting for policy reproducibility when evaluating reinforcement learning algorithms. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11), 12024–12032. https://doi.org/10.1609/aaai.v38i11.29090
Lu, Y., Ma, X., & Cui, C. (2024). DCCS: A dual congestion control signals-based TCP for datacenter networks. Computer Networks, 247, 110457. https://doi.org/10.1016/j.comnet.2024.110457
Zheng, S., Liu, J., Yan, X., Xing, Z., Di, X., & Qi, H. (2024). BBR-R: Improving BBR performance in multi-flow competition scenarios. Computer Networks, 254, 110816. https://doi.org/10.1016/j.comnet.2024.110816
Arghavani, M., Zhang, H., Eyers, D., & Arghavani, A. (2024). SUSS: Improving TCP performance by speeding up slow-start. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 151–165). ACM. https://doi.org/10.1145/3651890.3672234
Hu, J., Zeng, C., Wang, Z., Zhang, J., Guo, K., Xu, H., Huang, J., & Chen, K. (2024). Load balancing with multi-level signals for lossless datacenter networks. IEEE/ACM Transactions on Networking, 32(3), 2736–2748. https://doi.org/10.1109/TNET.2024.3366336
Brown, L., Gran Alcoz, A., Cangialosi, F., Narayan, A., Alizadeh, M., Balakrishnan, H., Friedman, E., Katz-Bassett, E., Krishnamurthy, A., Schapira, M., & Shenker, S. (2024). Principles for Internet congestion management. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 166–180). ACM. https://doi.org/10.1145/3651890.3672247
Wiegrebe, S., Kopper, P., Sonabend, R., Bischl, B., & Bender, A. (2024). Deep learning for survival analysis: A review. Artificial Intelligence Review, 57, 65. https://doi.org/10.1007/s10462-023-10681-3
van Wijk, D. E. J., Coogan, S., Molnar, T. G., Majji, M., & Hobbs, K. L. (2024). Disturbance-robust backup control barrier functions: Safety under uncertain dynamics. IEEE Control Systems Letters, 8, 2817–2822. https://doi.org/10.1109/LCSYS.2024.3514998
Hu, J., Zhou, Z., Rao, S., Peng, Y., Bao, B., & Ruan, C. (2024). Learning-based hierarchical adaptive congestion control with low training overhead. In 2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) (pp. 262–267). IEEE. https://doi.org/10.1109/ISPA63168.2024.00041
Wang, R., Tian, Y., & Kashima, K. (2024). Density estimation based soft actor-critic: Deep reinforcement learning for static output feedback control with measurement noise. Advanced Robotics, 38(6), 398–409. https://doi.org/10.1080/01691864.2024.2309621
Ji, X., Han, B., Xu, Y., & Su, J. (2026). ICCP: Toward congestion control agent via controlling logic decoupling and algorithm integration. IEEE Transactions on Network and Service Management, 23, 6463–6479. https://doi.org/10.1109/TNSM.2026.3714992
Lindemann, L., Robey, A., Jiang, L., Das, S., Tu, S., & Matni, N. (2024). Learning robust output control barrier functions from safe expert demonstrations. IEEE Open Journal of Control Systems, 3, 158–172. https://doi.org/10.1109/OJCSYS.2024.3385348