REI is a very well-known company in the outdoor gear space. They have 153 stores all over the world and are a respected leader in retail. They’re also a trailblazer in adoption of SD-WAN, as their recent plan to roll out Viptela SD-WAN appliances to their stores demonstrated. In a presentation for FutureWAN 18, Adam Burton, Rich Russell, and Mayra Guia detailed their reasons for moving to SD-WAN to increase stability and ease of use, standardize on protocols, and gain visibility and reporting into their wide network of retail stores.
Planning For Implementation
One of the biggest takeaways from the REI SD-WAN story is their planning process. Throughout the presentation, Adam, Rich, and Mayra talked about the importance of having a solid plan to deal with the cutover process. The process that REI came up with was very simple. They would cut over no more than 1-2 sites per timezone a night. That gave them the time they needed to resolve issues with the sites before they were scheduled to come back online. They also planned 1.5-2 hours per site, yet also understood that more time would be needed as issues arose.
The night of the cutover, the plan was clear. After the store’s systems were closed out for the day the onsite technician would unbox the new Viptela SD-WAN appliance and verify everything was there for the installation. Once everything was verified, the technician connected an LTE modem for backup connectivity before cutting the main systems over. Once everything was in place with the LTE backup, the old router was powered down and the new Viptela appliance was installed. The cabling diagram was followed and the new router came online. Connectivity was verified through the LTE modem and the new site was connected once the routing processes converged. After installation, the existing WAN optimization device was also removed, leaving the Viptela device to connect the store back to the rest of the REI network.
Once the cutover was accomplished, the on-site technician ran some post-installation tests to ensure that the cutover worked properly. Once the tests were completed, the old equipment was boxed up and readied to ship back to the main office. Then the site was ready to bring online permanently and the cutover was completed. This process was tested with local stores around the main headquarters first before being rolled out to remote stores. The team maintained a conference bridge during the cutover process to talk the on-site technical contacts through any issues that might crop up during installation.
Exceptions To The Rule
The plan, even when executed correctly, was never a perfect representation of all of the issues that might be encountered. During the presentation, the REI team recounted some of their common issues and gave some recommendations to help alleviate them both for future deployments as well as for anyone looking to use the REI plan as a prototype for their own deployment.
The biggest issue faced by the team was the sprawl that came from using a large number of Viptela templates to deploy the systems. Even though they were only using two different types of hardware, the amount of templates kept growing as each site came online because the primary template was exported to a new file and applied to the site each time. Adam said that he wished they had determined a naming convention for the templates earlier to help reduce the amount of confusion during the early deployments.
Another issue that was encountered was something simple with far-reaching ramifications: cabling diagrams. The REI team created detailed cabling diagrams for each installation to help the on-site installers connect the correct cables to the right ports on the Viptela appliances. The were in full color and easy to follow. During the first few remote installations, the on-site techs were getting confused looking at the diagrams. It quickly surfaced that they were printing them in black and white instead of color. Whether this was habit or due to a lack of a color printer in the office is unknown, but it caused some issues. The REI team fixed this problem by printing a full-color diagram and adding it to the box before shipping. That way they knew there was at least one proper diagram.
The remaining issues came from the installation teams. Using local installation help allowed the REI team to drive the installation from a central location and saved time and money because there was no need to go to the store location. However, there were some issues that came up. Cutting over stores in different timezones caused confusion on the phone bridge dedicated to supporting the technicians. A store that was a hour or two further into the cutover would bring up issues not being seen by the store that was behind. The REI team realized that it was sometimes easier to have the store further ahead drop off the bridge and call them directly to figure out the issues.
Also, some of the technical help was ready to leave at the drop of a hat, whether it be from technical trouble or not running the post-install tests. The REI team referred to this as the “Cinderella Problem”. A good recommendation is to make sure your local install teams know what is expected of them before the project kicks off.
Bringing It All Together
The REI experience with installing Viptela’s SD-WAN solution was eye opening. They had a solid plan and used it to the fullest to make everything go as smoothly as possible. Issues did occur but the team was able to sort them out quickly and get the stores online with the new equipment in time to open the next day to continue operations. This represents the flexibility that SD-WAN offers to easily replace the existing equipment easily and get back online with a minimum of effort. It would be interesting to see how the final deployment goes and how easily REI can bring new store online with their SD-WAN configuration going forward.
