Efficient schemes to scale the interconnection network bandwidth in a ring-based multiprocessor system

In the multiprocessors adopting ring topology with high-speed unidirectional point-to-point links, the performance of microprocessors, local busses and memory systems continue to improve at a very fast rate, whereas bandwidth of point-to-point links does not increase. Therefore ring network configured by current off-the-shelf point-to-point links cannot meet bandwidth requirements for system level interconnection, which may eventually limit scalability of system. For this reason, necessity to expand bandwidth of ring interconnect is emphasized. In this paper, we focus on the bandwidth of the interconnection network in a ring-based CC-NUMA system with snooping protocol. We show that ring interconnect is a bottleneck of performance in the high-end environment, if each node(cluster) is connected by point-to-point links based on the IEEE Scalable Coherent Interface(SCI) specifications. Bandwidth expansion by more than 200% is needed to achieve scalable performance. In order to expand ring bandwidth, it does not seem a cost effective way to develop a new link with doubled data line width. As an alternative to doubling data line width, we propose a dual ring architecture. In this paper, several ways to implement dual ring are suggested: simple dual ring, transaction-separated dual ring and direction-separated dual ring. We analyze pros. and cons. of each model compared with wide single ring (which means single ring using links with doubled data line width) model and we also simulate each model with On-Line Transaction Processing (OLTP) workloads. This paper demonstrates that dual ring architecture, in spite of some defects against wide single ring, scales system performance comparable to wide single ring, and that direction-separated dual ring outperforms wide single ring when there are large number of processors in the system.