Packets take turns: routing, queues and why faster links still make you wait

By How things work · myAudience

Image: illustration, not a documentary photograph. Separate fictional FIFO store-and-forward packet schedules; queues are conceptual components, not instantaneous occupancy. No measured Internet performance claim. · Operator-authored generated teaching illustration; scientific concept/model review retained separately.

Packets take turns: routing, queues and why faster links still make you wait

An image in an online lesson may seem to arrive as one object, but the network moves smaller units of data along shared links. An application supplies bytes to a transport service; those bytes are carried in packets with information that helps intermediate equipment forward them. Other people's traffic can use the same output link between our packets. That sharing allows a network to serve many conversations without reserving a separate physical wire for each one, but it also creates waiting. We will follow one fictional route from a sender through two routers to a receiver. Each packet's supplied size includes the protocol bytes counted by this exercise. We exclude additional link framing, processing, acknowledgements, retransmissions and competing traffic, rather than pretending those costs disappear in a real network. The boundary matters: time to move our counted packets is not automatically time to load a complete webpage, establish a connection, decode an image or display it. A useful explanation first identifies which journey is being timed and which work takes place outside it.

A router does not normally receive a complete instruction listing every wire the packet must traverse. For ordinary destination-based forwarding, it consults information about reachable address prefixes and selects a next hop and output interface. A prefix describes a shared initial part of an address. Among applicable matching routes, a more specific prefix takes precedence under longest-prefix matching; this is not a promise that the chosen route is the shortest geographical distance. Imagine a deliberately simplified binary address beginning 101101. Entries beginning 101 and 10110 both match, but the latter describes a smaller set and has the longer matching prefix. Real IP addresses and forwarding rules contain more information than this toy string. Routing mechanisms establish and update the table; forwarding uses the table for individual packets. Keeping those jobs separate explains why a network can retain working forwarding while control information changes, and why a stale or mistaken table can send traffic toward a problem. This lesson gives a conceptual account, not instructions for configuring a router.

After selecting an output, a router may have to wait for it to become free. Transmission then places the counted packet's bits on that link. If the counted size is L bits and the supplied transmission rate is R bits per second, serialization takes L/R seconds. Propagation is a different interval: a signal takes time to travel from one end of the link to the other. Increasing R shortens serialization under this model; it does not make a given propagation interval vanish. We assume store-and-forward operation, so each router waits for the complete packet before sending it onward. Some equipment uses other forwarding behaviour, so that assumption must remain visible. A first packet can therefore incur serialization on every link, propagation on every link and any queue waiting along the route. A train of packets can overlap those stages: while one packet crosses a later link, another can occupy an earlier one. Confusing first-packet delay with the spacing between successive arrivals hides that pipeline and leads to incorrect completion-time estimates.

Our four packets each contain 1,500 counted bytes, or 12,000 bits. The three link rates are 12, 6 and 12 megabits per second, where mega means one million. Their serialization intervals are consequently 1, 2 and 1 milliseconds. Each link also contributes a supplied 2 milliseconds of propagation. With initially empty queues and zero processing time, packet 1 reaches the receiver after 10 milliseconds. Later packets reach the first router only 1 millisecond apart, but its output needs 2 milliseconds for each packet. A queue builds there even though no unrelated traffic exists in this simplified exercise. Packet 4 waits 3 milliseconds before its middle-link transmission begins. The receiver's final-bit arrival times are 10, 12, 14 and 16 milliseconds, rather than four copies of the entire 10-millisecond journey placed consecutively. Against an inclusive 12-millisecond deadline, packets 1 and 2 arrive in time; packets 3 and 4 miss it. A deadline assessment must state whether equality counts as success.

The middle link is the original pipeline's bottleneck because it provides the smallest sustained counted-bit transmission rate. This does not mean it determines every part of the first packet's delay: the other links and all propagation intervals still matter. Nor does a faster advertised link guarantee that an application receives useful bytes at that rate. Real queues have finite storage, other traffic shares resources, and packets can be dropped or corrupted. TCP adds an endpoint service that presents a reliable, ordered byte stream, using sequence information, error detection and retransmission mechanisms. That service is different from a claim that IP forwarding always delivers every packet once and in order. TCP sending can also be constrained by the sender's congestion window and the receiver's advertised window. The 1995 IPv4 router source includes historical congestion material; its Source Quench instructions must not be reused as current guidance because RFC 6633 explicitly supersedes them. Here, loss, acknowledgement timing and congestion-control dynamics are outside the numerical schedule, not properties we have proved absent.

Now replace only the middle link's supplied rate with 24 megabits per second. Its packet serialization interval falls from 2 milliseconds to 0.5 milliseconds, but the other links still each take 1 millisecond and propagation still totals 6 milliseconds. The first packet arrives at 8.5 milliseconds. A complete stage-by-stage schedule gives later arrivals at 9.5, 10.5 and 11.5 milliseconds. All four now meet the same inclusive 12-millisecond deadline. The bottleneck has moved to the two 12-megabit-per-second links; the whole path is not suddenly a 24-megabit-per-second path. Although the middle link's rate increased fourfold, completion of these four packets improved by only 16/11.5, or about 1.39 times. A finite task's completion time and a long train's steady spacing answer different questions. This distinction matters when judging an upgrade: spending more on one component may help until another resource or an unchanged delay becomes limiting. Our numbers illustrate that causal change without establishing a commercial performance rating.

Arrival order is another property that must be checked separately. In an independent, explicitly supplied route-change example, let packet 1 start at time zero on the original three-link route. It arrives at 10 milliseconds. Let packet 2 start 1 millisecond later on a different two-link route, with both rates 12 megabits per second and 1 millisecond propagation on each link. That second route takes 4 milliseconds, so packet 2 arrives at absolute time 5 milliseconds, before packet 1. This is a counterexample showing that later departure need not imply later packet arrival when routes or delays differ. It is not a claim that normal per-flow load balancing randomly sends successive packets down different paths. TCP can hold later-arriving bytes until the missing earlier bytes become available; applications using other transports may need their own ordering policy. A hop limit bounds indefinite circulation rather than promising delivery. IPv4 calls its field TTL; IPv6 uses Hop Limit and decrements it at forwarding nodes. Their specifications must not be treated as identical header or fragmentation rules.

A useful network comparison combines capacity, delay, reliability and the service people need. A large download may benefit from sustained capacity, while an interactive lesson or assistive conversation can be harmed by long waits even when average throughput seems high. For a separate illustrative window bound, a 6-megabit-per-second path with a supplied 20-millisecond round-trip time has a bandwidth-delay product of 120,000 bits, or 15,000 bytes. A supplied 6,000-byte outstanding-data limit divided by that round-trip time gives an upper bound of 2.4 megabits per second before other limitations. This is not a prediction from our four-packet schedule or a recommendation to change operating-system buffers. Measurement must distinguish those boundaries and report the conditions. Fair access also means asking who receives scarce capacity, which services require dependable response and whether proposed priorities disadvantage others. An accessible design can offer smaller media and readable text without assuming everybody has a fast connection. No queue diagram or fictional calculation certifies network safety, availability, device operation or assessed practical competence.

Worked example: Four packets of 1,500 counted bytes are ready at time zero. They follow the same three-link FIFO store-and-forward route at 12, 6 and 12 Mbps. Each link has 2 ms propagation. All queues initially contain no other packets, processing takes zero time, and the supplied packet size already includes the protocol bytes counted here. Exclude additional framing, acknowledgements, losses and retransmissions. Find each receiver final-bit arrival time, the fourth packet's queue wait at router 1, the bottleneck and which packets meet an inclusive 12 ms deadline.

Convert the size before dividing by a bit rate: 1,500 bytes × 8 bits per byte = 12,000 bits. One Mbps is 1,000,000 bits per second. Dividing 12,000 by 12,000,000 gives 0.001 s = 1 ms; dividing by 6,000,000 gives 0.002 s = 2 ms. The three serialization intervals are therefore 1, 2 and 1 ms. These account for the supplied counted packet, not additional link framing that the model excludes.

For packet 1, the sender transmits from 0 to 1 ms. Add the first link's 2 ms propagation: its final bit reaches router 1 at 3 ms. The middle link transmits it from 3 to 5 ms; propagation brings its final bit to router 2 at 7 ms. The last link transmits from 7 to 8 ms, then propagation completes arrival at 10 ms. Serialization totals 4 ms and propagation totals 6 ms.

The first link can send the four ready packets consecutively, completing transmission at 1, 2, 3 and 4 ms. Adding its 2 ms propagation gives router-1 final-bit arrivals at 3, 4, 5 and 6 ms. Those arrivals are separated by 1 ms. A router cannot begin a packet's store-and-forward transmission before that arrival, and it also cannot overlap two transmissions on the same supplied output link.

The middle link's 2 ms service time makes its transmission starts 3, 5, 7 and 9 ms, and finishes 5, 7, 9 and 11 ms. Packet 4 arrives there at 6 ms but starts at 9 ms, so its queue wait is 9 − 6 = 3 ms. Adding propagation gives router-2 final-bit arrivals 7, 9, 11 and 13 ms. Queue waiting is a consequence of the arrival/service mismatch, not an additional propagation interval.

The last link needs only 1 ms per packet, so it is free at each supplied arrival. Transmissions finish at 8, 10, 12 and 14 ms. Its 2 ms propagation produces receiver final-bit arrivals 10, 12, 14 and 16 ms. Earlier-link transmissions overlap later-link activity, so multiplying the first packet's 10 ms by four would wrongly ignore the pipeline. The receiver sees packets spaced 2 ms apart after the first one.

The original smallest rate is 6 Mbps, making the middle link the counted-bit bottleneck for a sustained train under these assumptions. The fourth packet finishes at 16 ms. With an inclusive deadline, arrival ≤ 12 ms succeeds: packets 1 and 2 meet it, while packets 3 and 4 miss it. Report that exact result rather than only saying the last packet is late. Neither the bottleneck nor the deadline calculation predicts unmodelled TCP dynamics or real application goodput.

Answer: The receiver final-bit arrivals are 10, 12, 14 and 16 ms. Packet 4 waits 3 ms at router 1. The middle 6 Mbps link is the original counted-bit bottleneck. Packets 1 and 2 meet the inclusive 12 ms deadline; packets 3 and 4 miss it. These are conditional predictions for the supplied store-and-forward model, not measured Internet performance.

Check your understanding: Change only the middle link to 24 Mbps and retain all original assumptions and the inclusive 12 ms deadline. Derive all four final-bit arrival times and the new bottleneck. Is the completion speedup fourfold? Then consider a separate supplied route-change case: packet 1 starts at time zero on the original route, while packet 2 starts at 1 ms on a different two-link route at 12/12 Mbps with 1 ms propagation per link. Which arrives first, and must TCP applications consequently receive bytes out of order?

Answer: The changed middle serialization interval is 0.5 ms. Router-1 arrivals remain 3, 4, 5 and 6 ms; middle transmissions finish 3.5, 4.5, 5.5 and 6.5 ms. After 2 ms propagation, router-2 arrivals are 5.5, 6.5, 7.5 and 8.5 ms. The final 1 ms serialization plus 2 ms propagation gives receiver arrivals 8.5, 9.5, 10.5 and 11.5 ms. All four meet the inclusive deadline. The 12 Mbps first and last links are now bottlenecks. Finite completion speedup is 16/11.5 = 32/23 ≈ 1.391, not four. In the independent route-change case, packet 1 arrives at 10 ms; packet 2 takes 1 + 1 + 1 + 1 = 4 ms after its 1 ms departure and arrives at 5 ms, so packet 2 arrives first. TCP provides an ordered byte stream at endpoints, so this packet arrival order does not itself require out-of-order application byte delivery.

Why: The changed route preserves packet size, first/last link rates and propagation; only one service interval changes. Each stage starts at the later of complete packet arrival and output availability. Counting that schedule prevents an invented 24 Mbps end-to-end rate. The alternative path is a separate explicitly supplied counterexample, not a claim that ordinary per-flow routing routinely alternates paths. TCP ordering is an endpoint service; other transports may require an application ordering policy. The calculation excludes processing, framing, acknowledgements, competing traffic, loss and congestion-control dynamics.