Creating Wi-Fi Test Metrics

The constant mobility of the WLAN user coupled with the inherent instability of the unwired medium - air - make the 802.11 protocol an order of magnitude more complex than equivalent wired protocols.

Testing techniques developed for wired devices and networks fall short when applied to the WLAN market. The inherent instability of the unwired medium — air — in which the wireless world operates and the constant mobility of the WLAN user make the 802.11 protocol an order of magnitude more complex than equivalent wired protocols. As a result, the metrics used to benchmark wired protocols are only a starting point for the WLAN industry.

Differences between wired and wireless networks require metrics and methodology for performance benchmarking that address the intricacies of the 802.11 protocol. The discussion in the article focuses on parameters critical to WLAN performance and discusses methodology to measure them. It addresses wireless-specific functions central to business-critical applications on a Wi-Fi infrastructure and includes typical results and interpretations.

WLAN Performance Metrics
Wi-Fi protocols address differences between wired and wireless networks, and the implementation of the more advanced wireless protocol demands performance validation. Algorithms used in a network's clients and APs (and the capacity of these devices to process the algorithms) limit the network's performance. The objective of the validation process and test metrics is to identify critical test parameters and find the correct method of testing them. A listing of wireless performance metrics shows that they outnumber traditional Ethernet metrics by a ratio of roughly five to one (see Table 1).

Testing Ethernet network performance is essentially a measure of packet forwarding rate. In addition to packet forwarding measurements, WLANs must undergo tests related to the unstable physical layer and end-user mobility, including automatic data rate adaptation, roaming, verification of security, QoS and overlapping BSSs, as well as behavioral tests that measure performance under abnormal network conditions. The primary focus of the testing effort should be parameters that eventually affect network efficiency and operation.

click the image to enlarge

Data Rate Adaptation: Wired LANs support fixed data rates: 10/100/1000 Mb/s. Wireless networks support multiple data rates: 11/5.5/2/1 Mb/s for 802.11b; 54/48/36/24/18/12/9/6 Mb/s for 802.11a and 802.11g. The critical difference is that WLANs support dynamic rate adaptation and can operate at multiple data rates automatically determined by the end point (AP or client), based on the condition of the physical layer between it and the client device.

In addition, because the 802.11 standard does not specify exact criteria for data rate adaptation, the algorithms can vary from device to device. The rate adaptation algorithm should be based on optimizing throughput; that is, when the number of errors at a specific data rate increases to the point where throughput is severely affected, the device should drop to a lower data rate to recover the best throughput at that distance from the AP.

The challenge is how to measure this repeatably, while creating a metric that can be set as the golden standard. To measure throughput at fixed points, many vendors often use an interference-free environment with a long, direct line-of-site area where they can simulate data rate adaptation by wheeling a client up and down on a cart. This method, however, lacks accurate rate adaptation data and is less efficient than newer devices that offer controlled RF environments and accurate signal attenuation through test setup automation. While characterizing range vs. data rate, the test should simultaneously characterize range vs. throughput and range vs. packet error rate.

Roaming: As a client moves out of range of one AP, it dissociates from the AP and must associate and authenticate with another. If the client predicts this roam will occur by noticing the drop in signal and searching for an alternate AP before it is actually disassociated from the first, it can optimize the roam time and network disruption caused by the roam.

The client device makes the decision to roam based on its position relative to different APs and their signal strengths. The client might periodically analyze signal strength of the APs that surround it and decide which one to associate with if it needs to roam. Load-sharing protocols used by some WLAN network vendors depart from the traditional client-based decision process, orchestrating client devices to associate with specific APs and spreading the load evenly among APs and optimizing the entire network throughput. In addition, the IEEE is advancing its work on roaming through better RF measurement (802.11k) and fast roaming processes (802.11r). As the roaming process increases in complexity, it is critical to have standard roaming metrics for testing WLAN networks and equipment.

Roaming is critical because it takes time, which can cause data loss that can ultimately disrupt a communication session. Data loss is particularly important for time-sensitive enterprise applications, such as VoWLAN, that are especially susceptible to packet delay caused by roaming. Roaming metrics include roaming time, packet loss and session continuity . Roaming time can be broken down into the following stages: scanning, associating to the new AP, authentication with the new AP and data flow. Analyzing the time of each phase of this process will help ensure the most efficient roam time.

( click the image to enlarge)

Figure 3. Roaming test configuration.

Behavior should be verified in a number of different roaming scenarios that will emulate the cause and speed of the roam. A roam caused by an AP failure is significantly different from a roam caused by a person walking down a hallway with a wireless device. Typical results show huge differences between a fail-over roam time — in which the client is not expecting to roam — and a motion roam — where the client is expecting the roam.

Packet Forwarding: Forwarding rate is a function of a device: in wired networks, the Ethernet switch; in WLANs, the AP. Packet forwarding rate testing is always done at the highest signal strength and at the highest data rate because this puts the most demand on the device and measures its packet processing power in the most extreme case.

Like wired throughput tests, a wireless packet forwarding test varies the packet size to ensure the ability of the device to work with diverse traffic. But unlike wired devices, there are other factors to consider. The most critical is security, because wireless network devices must encrypt each packet. This additional overhead must be added to evaluate its effect on the packet forwarding rate. Another important factor is client capacity. Running the test with a large number of users stresses the AP's ability to handle a large number of users, each sending a portion of the bandwidth. This also affects how the AP functions under such conditions.

( click the image to enlarge)

Figure 4. The test script measures and tabulates the duration of each phase of the roaming process.

( click the image to enlarge)

Figure 5. Test results of the fastest roaming client reported by an automated test script. The test script performed seven roaming cycles between two APs.

QoS:

( click the image to enlarge)

A summary of wired and wireless metrics. The significantly higher number of wireless metrics speaks to the relative complexity of wireless protocols. Note the enormous effect that an unstable physical layer would have on practically every measurement.