Delay tolerant bulk data transfers on the internet SIGMETRICS '09 Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems, 2009
| Many emerging scientific and industrial applications require transferring multiple terabytes of data on a daily basis. Examples include pushing scientific datasets from particle accelerators/colliders to laboratories around the world, synchronizing data centers across continents, and replicating collections of high definition videos from events taking place at different time-zones. A convenient property of all above applications is their ability to tolerate delivery delays ranging from a few hours to a few days. Such Delay Tolerant Bulk (DTB) transfers are currently being serviced mostly by the postal system using hard drives and DVDs, or by expensive dedicated networks. In this paper we develop store-and-forward (SnF) scheduling policies for performing low-cost transfers of DTB data over existing public networks. We use traffic data from 200+ links of a large transit ISP to show that the naive approach of performing end-to-end (E2E) connection-oriented transfers can be prohibitively expensive under common percentile pricing schemes of transit bandwidth. The problem is that despite strong existing diurnal load patterns that leave lots of free bandwidth at individual PoPs during off-peak hours, E2E transfers are unable to take advantage of them because such load valleys appear at different times at the sender and the receiver. By utilizing network storage at intermediate PoPs, our SnF policies time-shift DTB transfers to bridge the gap between non-coinciding valleys, and thus reduce, even zero the resulting transit costs. |
Delay tolerant bulk data transfers on the internet