Top

Safe SAC Model for RDMA Congestion Control with Queue Length Barrier Constraints

Authors

Files

PDF

Abstract

Safe rate adaptation for remote direct memory access requires concurrent foreground throughput, transfer completion and bounded switch-queue occupancy under delayed telemetry. This study evaluates a soft actor-critic controller whose proposed sender rates pass through a queue-length barrier projection before execution. Four sender work queues feed a shared fluid-model bottleneck. Fourteen actor descriptors summarize delayed occupancy, congestion feedback, pending work and previous rates, while a trusted counter and admission history form a conservative current-queue envelope. A 100 Gbit/s reference link and 50 microsecond control slot define 625,000 bytes per traffic unit. The admissible aggregate rate uses a 0.90-unit service lower bound, 0.10-unit background upper allowance and barrier gain 0.35. Training uses 12,000 transitions, and evaluation uses twelve independently seeded traffic episodes across four dimensionless offered-load levels. Safe SAC yields mean foreground goodput 79.55 Gbit/s and mean episode-level 99th percentile flow completion time 2.863 ms. Relative to additive-increase multiplicative-decrease control, goodput rises 11.30% and completion tail falls 34.55%; mean queue 99th percentile falls 52.11%. Fixed-proposal filtering through the same barrier also has zero observed nominal safety-boundary violations, while learned proposals improve goodput by 2.06% and completion tail by 17.53% against that comparator. Projection modifies 99.22% of learned proposals versus 25.65% of fixed proposals. When actual service falls below the assumed bound, violation fractions increase, reaching 30.73% at service multiplier 0.90. Median local actor-plus-projection time is 27.30 microseconds, with three of 1,000 calls above the 50 microsecond slot. These results separate policy efficiency, conditional barrier protection and execution cost within the declared queue simulator.

PDF

References

Hu, J., Shen, H., Liu, X., & Wang, J. (2024). RDMA transports in datacenter networks: Survey. IEEE Network, 38(6), 380–387. https://doi.org/10.1109/MNET.2024.3397781

Zhang, J., Wang, Y., Zhong, X., Yu, M., Pan, H., Zhang, Y., Guan, Z., Che, B., Wan, Z., Pan, T., & Huang, T. (2024). PACC: A proactive CNP generation scheme for datacenter networks. IEEE/ACM Transactions on Networking, 32(3), 2586–2599. https://doi.org/10.1109/TNET.2024.3361771

Tian, H., Liao, X., Zeng, C., Sun, D., Zhang, J., & Chen, K. (2024). Efficient DRL-based congestion control with ultra-low overhead. IEEE/ACM Transactions on Networking, 32(3), 1888–1903. https://doi.org/10.1109/TNET.2023.3330737

Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., & Knoll, A. (2024). A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 11216–11235. https://doi.org/10.1109/TPAMI.2024.3457538

Liu, Y., Yao, X., Yang, Z., & Li, W. (2024). A multi-queue-based ECN marking strategy for multi-class QoS guarantee in programmable networks. PeerJ Computer Science, 10, e2382. https://doi.org/10.7717/peerj-cs.2382

Du, Z., Zheng, J., Yu, H., Zhang, H., & Chen, G. (2025). Libra: A congestion control framework for diverse application preferences and network conditions. IEEE/ACM Transactions on Networking, 33(2), 510–525. https://doi.org/10.1109/TNET.2024.3492096

Sun, H., Huang, Q., Lee, P. P. C., Bai, W., Zhu, F., & Bao, Y. (2024). Distributed network telemetry with resource efficiency and full accuracy. IEEE/ACM Transactions on Networking, 32(3), 1857–1872. https://doi.org/10.1109/TNET.2023.3327345

Giacomoni, L., & Parisis, G. (2024). Reinforcement learning-based congestion control: A systematic evaluation of fairness, efficiency and responsiveness. In IEEE INFOCOM 2024—IEEE Conference on Computer Communications (pp. 1451–1460). IEEE. https://doi.org/10.1109/INFOCOM52122.2024.10621288

Vielhaus, C. L., von Lengerke, C., Latzko, V., Rischke, J., Reisslein, M., & Fitzek, F. H. P. (2026). Evaluating transport layer congestion control algorithms: A comprehensive survey. IEEE Communications Surveys & Tutorials, 28, 278–340. https://doi.org/10.1109/COMST.2025.3533303

Tan, Z., & Hong, P. (2024). FRCS: Congestion control in datacenter networks based on fair rate calculation by switches. In 2024 10th International Conference on Computer and Communications (ICCC) (pp. 1445–1450). IEEE. https://doi.org/10.1109/ICCC62609.2024.10941816

Che, B., Wang, Y., Wan, Z., Chen, Y., Wang, Z., Tian, Y., Zhao, J., & Zhang, J. (2026). A fast-converging low-latency RDMA congestion control algorithm. IEEE Network, 40(1), 219–226. https://doi.org/10.1109/MNET.2025.3589913

Zhao, Y., Wang, S., Han, S., Zhou, D., Peng, G., & Huang, T. (2024). RACC: Rapid and accurate INT-based congestion control in RDMA network. In GLOBECOM 2024—2024 IEEE Global Communications Conference (pp. 3811–3816). IEEE. https://doi.org/10.1109/GLOBECOM52923.2024.10901068

Boussaoud, K., En-Nouaary, A., & Ayache, M. (2025). Adaptive congestion detection and traffic control in software-defined networks via data-driven multi-agent reinforcement learning. Computers, 14(6), 236. https://doi.org/10.3390/computers14060236

Banerjee, C., Chen, Z., & Noman, N. (2024). Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experiences. IEEE Transactions on Neural Networks and Learning Systems, 35(3), 3121–3129. https://doi.org/10.1109/TNNLS.2022.3174051

Jha, M. S., & Kiumarsi, B. (2025). Off-policy safe reinforcement learning for nonlinear discrete-time systems. Neurocomputing, 611, 128677. https://doi.org/10.1016/j.neucom.2024.128677

Yang, Y., Zhang, Y., Zou, W., Chen, J., Yin, Y., & Li, S. E. (2024). Synthesizing control barrier functions with feasible region iteration for safe reinforcement learning. IEEE Transactions on Automatic Control, 69(4), 2713–2720. https://doi.org/10.1109/TAC.2023.3336263

Kong, X., Xia, Y., Sun, Z., Zhai, D.-H., Deng, Y., & Zhang, S. (2025). Differential high order control barrier function-based safe reinforcement learning. IEEE Robotics and Automation Letters, 10(7), 7524–7531. https://doi.org/10.1109/LRA.2025.3575310

Alan, A., Molnar, T. G., Ames, A. D., & Orosz, G. (2025). Generalizing robust control barrier functions from a controller design perspective. IEEE Open Journal of Control Systems, 4, 54–69. https://doi.org/10.1109/OJCSYS.2025.3529364

Wang, Z., Wan, X., Li, L., Sun, Y., Xie, P., Wei, X., Ning, Q., Zhang, J., & Chen, K. (2024). Fast, scalable, and accurate rate limiter for RDMA NICs. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 568–580). ACM. https://doi.org/10.1145/3651890.3672215

Pichetti, L., De Sensi, D., Sivalingam, K., Nassyr, S., Cesarini, D., Turisini, M., Pleiter, D., Artigiani, A., & Vella, F. (2024). Benchmarking Ethernet interconnect for HPC/AI workloads. In SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 869–875). IEEE. https://doi.org/10.1109/SCW63240.2024.00124

Wan, Z., Zhang, J., Yu, M., Liu, J., Yao, J., Zhao, X., & Huang, T. (2024). BiCC: Bilateral congestion control in cross-datacenter RDMA networks. In IEEE INFOCOM 2024—IEEE Conference on Computer Communications (pp. 1381–1390). IEEE. https://doi.org/10.1109/INFOCOM52122.2024.10621412

Wojnar, M., Szott, S., Rusek, K., & Ciezobka, W. (2024). Reinforced-lib: Rapid prototyping of reinforcement learning solutions. SoftwareX, 26, 101706. https://doi.org/10.1016/j.softx.2024.101706

Flageat, M., Lim, B., & Cully, A. (2024). Beyond expected return: Accounting for policy reproducibility when evaluating reinforcement learning algorithms. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11), 12024–12032. https://doi.org/10.1609/aaai.v38i11.29090

Lu, Y., Ma, X., & Cui, C. (2024). DCCS: A dual congestion control signals-based TCP for datacenter networks. Computer Networks, 247, 110457. https://doi.org/10.1016/j.comnet.2024.110457

Zheng, S., Liu, J., Yan, X., Xing, Z., Di, X., & Qi, H. (2024). BBR-R: Improving BBR performance in multi-flow competition scenarios. Computer Networks, 254, 110816. https://doi.org/10.1016/j.comnet.2024.110816

Arghavani, M., Zhang, H., Eyers, D., & Arghavani, A. (2024). SUSS: Improving TCP performance by speeding up slow-start. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 151–165). ACM. https://doi.org/10.1145/3651890.3672234

Hu, J., Zeng, C., Wang, Z., Zhang, J., Guo, K., Xu, H., Huang, J., & Chen, K. (2024). Load balancing with multi-level signals for lossless datacenter networks. IEEE/ACM Transactions on Networking, 32(3), 2736–2748. https://doi.org/10.1109/TNET.2024.3366336

Brown, L., Gran Alcoz, A., Cangialosi, F., Narayan, A., Alizadeh, M., Balakrishnan, H., Friedman, E., Katz-Bassett, E., Krishnamurthy, A., Schapira, M., & Shenker, S. (2024). Principles for Internet congestion management. In Proceedings of the ACM SIGCOMM 2024 Conference (pp. 166–180). ACM. https://doi.org/10.1145/3651890.3672247

Wiegrebe, S., Kopper, P., Sonabend, R., Bischl, B., & Bender, A. (2024). Deep learning for survival analysis: A review. Artificial Intelligence Review, 57, 65. https://doi.org/10.1007/s10462-023-10681-3

van Wijk, D. E. J., Coogan, S., Molnar, T. G., Majji, M., & Hobbs, K. L. (2024). Disturbance-robust backup control barrier functions: Safety under uncertain dynamics. IEEE Control Systems Letters, 8, 2817–2822. https://doi.org/10.1109/LCSYS.2024.3514998

Hu, J., Zhou, Z., Rao, S., Peng, Y., Bao, B., & Ruan, C. (2024). Learning-based hierarchical adaptive congestion control with low training overhead. In 2024 IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) (pp. 262–267). IEEE. https://doi.org/10.1109/ISPA63168.2024.00041

Wang, R., Tian, Y., & Kashima, K. (2024). Density estimation based soft actor-critic: Deep reinforcement learning for static output feedback control with measurement noise. Advanced Robotics, 38(6), 398–409. https://doi.org/10.1080/01691864.2024.2309621

Ji, X., Han, B., Xu, Y., & Su, J. (2026). ICCP: Toward congestion control agent via controlling logic decoupling and algorithm integration. IEEE Transactions on Network and Service Management, 23, 6463–6479. https://doi.org/10.1109/TNSM.2026.3714992

Lindemann, L., Robey, A., Jiang, L., Das, S., Tu, S., & Matni, N. (2024). Learning robust output control barrier functions from safe expert demonstrations. IEEE Open Journal of Control Systems, 3, 158–172. https://doi.org/10.1109/OJCSYS.2024.3385348

Details