I’m going to start this post with a disclaimer. Firstly, I spent a decade working for a Value-Added Reseller (VAR) that focused on a specific market segment: K-12 education. Second, I’m working from the position that I disagree with Peyton Koran’s assessment of the market in this video:
If you read the link above summarized by my friend Greg Ferro (@EtherealMind), you’ll see that he believes there are four major areas where open source and whitebox switching are taking over. Let’s look at each one.
In It To Win It
Premise: Vendors and VARs are no longer working to benefit the customer but to benefit themselves with increased sales and services.
What is the point of a business? If I recall from my business degree, it’s to stay in business. Business provide a product or a service. In the networking world, that means you’re either making gear or you’re providing a service for that gear. Vendors make equipment, VARs provide a service to install it.
If you sell a thing, how much should you sell it for? Should you sell it for what it cost you to make it? That wouldn’t keep you in business. Should you sell it for a huge fat profit? That depends on your business model. Most people will tell you that you should sell it for the price that “the market will bear”. So, if you make a thing that costs X to build and the market will let you sell it for 3X, should you sell it for 2X just because it makes you feel good? Or should you take the money that you can make and use the profits to invest in other things? It’s a tough call, right?
As your shareholders how they feel about profit. Yes, the shareholders get a say in this too. If you’re a publicly traded company you find quickly that your path doesn’t always line up perfectly with your goal. Shareholders are fickle and driven by a number at the end of every quarter. They don’t care about altruism or goodwill. They want return on their investment, and that comes from beating profit estimates.
Ask a company like Electronic Arts (EA) how they feel about profit. Should they start offering cheaper games or free online service? What about loot boxes sold for real money? Businesses exist for profit. Even a non-profit organization covers the costs of operating the business, including paying salaries. To think that a networking vendor or VAR is working for the benefit of the customer first might mean your priorities for business are misaligned.
Feature Presentation
Premise: Customers don’t need all the features in a modern network operating system.
That’s actually a true point. Most everyone that ever installs a switch won’t touch 50% of the features that it uses. In the customers that I worked with, that number was closer to 75% of the features. What does that mean for the future of networking?
I’m fond of saying that every crazy feature you’ve ever seen in a network operating system was the result of a million-dollar sale that needed that feature to go through. If you’re evaluating a solution and it’s missing a key feature, you’re not going to buy that solution. If you’re a vendor and you’re in danger of losing a sale because you don’t support a key feature, you’re going to develop that key feature. That’s supply and demand in a nutshell from Macroeconomics.
So, if you need a feature and it’s not present, how are you going to get that feature? Well, there are some in the industry that think that you should have an army of programmers waiting in the wings to write a Linux app that runs on your switch and executes that feature when needed. That’s well and good for a feature that you need once a month or once a quarter. But what about those features you need daily? Or constantly? Do you and your programming team feel comfortable enough with the architecture of a switch to write a routine that won’t leak memory and cause the switch to reboot once a week? Do you feel confident that your program isn’t going to slow down the switch and cause network bottlenecks?
Telling an application developer that they’re good at coding operating systems is like telling me that because I took an intro course to programming that I should be able to code a video game. General knowledge in one area doesn’t always translate into the specialized areas needed for other places. Sure, we’re always going to have to deal with feature bloat in code bases. Desktop and server operating systems have it. Mobile operating systems have it. Protesting that networking operating systems should be immune from it is a rosy view of the world that doesn’t work in reality.
I Feel The Need For Speed
Premise: Customers can’t wait months or years for vendors to fix or update their software. Speed is what’s important.
If you’ll excuse my pointedness, this may be one of the more hypocritical statements of the whole argument.
You want a bug free software release to happen hours or days after a vulnerability is found. Or you want a new standard implemented in software as soon as it’s ratified. But you want it to be stable and not introduce any new bugs into the code. That’s a tall order.
Applications get patched all the time. In fact, Star Wars Battlefront 2 from EA launched with patch waiting to download. Seems they couldn’t fix all the bugs before it went live. And given that it got at least three patches during the open beta, you might say they didn’t deliver the most stable and entertaining code they could have if they had just waited to release it.
I’m highlighting video games here because they provide the perfect example of the patching process. You create a new feature and you need it tested. A small group of dedicated people test it and bless that it’s ready to deploy. Then, the larger population of users gets the new patch and finds out that when it is pushed to the limits it has bugs and breaks under certain special circumstances. Those circumstances weren’t possible to replicate in testing or were absent from the pool. Perhaps the setup only happens on old hardware or for someone who is using the software in an unintended way? You can’t test for every possible scenario. You have to do the best you can with the limited time you have to fix the issues and get the patch out the door.
All software development works like this. You can deliver a solid patch eventually or a “good enough” patch tomorrow. You can’t have both. And for those that might be crying about DevOps/CI/CD mentalities for delivering consistent software quality, just remember that you’re not really working with a stable codebase at that point. You’re just working with one that’s getting constant micro changes.
Buyer’s Market
Premise: Network spending is down because buying from VARs or direct from vendors is just too hard.
This is a sentiment echoed by Teren Bryson in this post. Teren is like me, so I believe his viewpoint. But, if you’re a Small-to-Medium Enterprise (SME) spending in the tens or hundreds of thousands per year, I want you to go try to buy network gear directly from the big networking vendors. Go on to their website and just click the “Buy Now” button for a Nexus or a big Arista switch.
Back already? No “Buy Now” button? Strange. You’d think that large companies would just make it super easy to buy directly from them if VARs were so useless. Maybe there’s a bit more to the story.
Turns out, having hundreds of thousands of customers in your database can be a bit of a drain. And trying to make one or two switches for the majority of them a year takes way more time than it should. That’s why most vendors and resellers work through a distribution partner. Distribution partners agree to buy the basic configurations of devices and sell them direct to smaller customers and partners. This increases the amount of sales of equipment and keeps the smaller customers happy because they don’t have to wait a month for something to get built. It’s the same model used by Apple Stores and Best Buy. Keep stock in the things people want and if they want something special and custom they can order it direct and wait.
VARs are also incentivized to sell to customers. They get discounts that they often pass along to their customers to provide for service attachment. Want to buy direct from the vendor? I’d bet that discount disappears. It’s there because the VAR has made an investment in the brand. Discounts happen to make sure that VARs are concentrating on selling equipment and installing it promptly. The customer pays less overall because they’re getting something of value added to the transaction.
Do vendors that don’t have a channel model sell things cheaper? Sure. But you lose the value portion of it. If you don’t need that then you wouldn’t be looking to buy from a VAR in the first place. It doesn’t stop people from trying though. In my VAR days, we had several companies that wanted to buy directly from us but not use our services. They got upset when we quoted them a price that was higher than the discount they knew we got. We told them the discount existed for our use, not for theirs. These passthrough customers would get made and go off to try and buy direct. And in more than a few cases, they came back when they realized that they either couldn’t buy direct or the cost they had to pay was higher than our price.
If the purchasing model is truly broken, don’t rush to lay blame at the feet of the vendors and the VARs. Instead, think about where it got broken along the way. Think about all the marketing funding that goes into selling your product. And ask yourself how differently things might look if the deciding factor in most sales was something other than the bottom line price.
Bringing It All Together
Is the world of vendors and VARs all nice and rosy? No. There are always improvements to be made. We could streamline the process or make it easier to customize gear. And no one is ever really happy with the price they are paying for something. But to suggest that everything that whitebox switching and open source software offers is the Golden Path forward is just as short sighted. You have tradeoffs. You have to spend a lot of time retraining your workers to act in a different capacity. You gain flexibility at the cost of speed. And it won’t work for more than half the businesses out there. The model that Peyton Koran espouses may work for a number of large organizations. But the only way to know that is to investigate for yourself. Please do the research before jumping into the deep end of this pool.
Excellent piece:
What brought it home to me was this: “You have to spend a lot of time retraining your workers to act in a different capacity. ” And that is one of the key issues.. 99% of the enterprises out there are not hyper scale, startup Stanford-grad staffed companies, but business trying to get by with a limited IT budget and struggling to get the right IT/networking staff. Although I am a bit biased working for a vendor independent consultancy firm (we do not sell hardware nor a vendor, so we don’t fit the usual VAR/vendor category) I see enterprises asking us to provide knowledge because they can’t attract the right people anymore. So what are you going to do now. Move to white box/ open source with no staff to support it? Or take smaller steps towards a more automated future of networking clearly considering all the constraints beyond those of a vendor or VAR. Indeed research is needed to address these constraints and making sure the networking strategy fits these. Only then a choice can be made with what technology and with whom. And it’s never as clear cut as choosing for whitebox/open source in most cases.
Hey Tom, there’s one significant misconception here that I want to address, because it’s a very common one in the enterprise space: that in the new nirvana of open source and software-defined everything, users who want a new feature can and will write it themselves. This is very seldom be the case, even at very large companies, including major service providers with lots of coders on staff–and IMO will continue not to be a common thing for a very long time, if ever. Most companies who are in the throes of embracing open source and SDN/NFV still look for specialist partners who are familiar with their preferred (software) platforms, just as they always have with their VARs specializing in their preferred hardware platforms. And they’re obviously not throwing out all their physical gear either; they’re just buying progressively less of it compared to other areas of investment. It is, therefore, an extremely critical time for VARs to be reorienting their own business models. While the VAR world has seen shrinking margins for some time, the shift (back) to roll-your-own (with a lot of help from your friends) is a golden opportunity for those VARs who are willing to shift their own business models and staffing towards software expertise. Then there’s the commercial ecosystem of vendors, some of whom are figuring out how to effectively package and sell solutions built around OSS code. The second group is still in its infancy for now, but the point is, few users should be expecting to download, compile and maintain their own OSS networking stacks–that’s exactly what your supplier ecosystem should be begging to do for you. From a customer standpoint, OSS-based commercial solutions need not be noticeably different from any other packaged solution in terms of the familiar purchasing and consumption model, but they can be more customizable and extensible if the solution provider is strongly OSS-centric, eg maintaining compatibility with the source code vs forking for the purposes of their branded solution.
Tl;dr – the VAR/SI ecosystem doesn’t go away in this brave new world at all–quite the opposite, in all likelihood; but the skills and services that VARs will need to provide to stay competitive will continue to shift. Note, again, I’m NOT saying that all or even most user-side network engineers will need to know how to code. Successful tech vendors will be those who make it so that the users don’t have to code AND don’t have to spend their time on box-by-box configuration anymore.
I think your points fall pretty flat. But I love the discussion. So let’s break it down.
1) Of course companies are designed to make money, but to do that they have to present some form of value. The value derived by most Networking companies and VARs continues to decline, to the point that most of these companies are now pushing bad architectures to try to get customers hooked on them. In the EA case, EA provides their product to gamers, aren’t they supposed to do that in the most efficient and cost effective way? This helps them maximize profit. And they also increase profit by building better gaming experiences. So these types of companies will work with other companies that want to make money, but they have to be very conscious abut who they work with to make sure they are working as efficiently as possible. And again, VARs and Networking Vendors are not providing that much value, but actually more headache now.
2) The answer to this is easy, you develop the feature, or work in an open source community that works together to do it. You try it in small areas, and as you gain confidence you expand.
3) This is laughable. 95% of companies expect software and features from networking vendors that is broken. Your argument flies directly in the face of the proved experience. This is why more and more advanced companies are building their own. In fact, there are now whole programs around OCP networking, and open source communities, that are working on this. The idea that a networking vendors are the only ones that can deliver this is just hilarious. And again, they want us to use extremely unscalable solutions that just hook us on them. Project Calico can do 95% of all the things that Networking vendors have worked years on and it costs much less, and completely proves my point.
4) Basically you just defended VARs, with the same language they have used for years. They don’t offer discounts, as they take their take off the top, meaning they add points to the top of what the Network Vendor will sell it to you for. And they really don’t provide me anything in purchasing that can’t be handled more efficiently through a webpage. They are propping up an old ecosystem. And 95%+ of companies can begin building what they need in a cloud provider, today, instead of months later while waiting on a VAR to deliver hardware. And then they completely remove the need to have Network engineers.
Basically your post is just FUD (Fear, Uncertainty, and Doubt), with the same worn out old VAR and Network Vendor excuses. I am sorry I am being so blunt, but this stuff has been said for 20+ years. The ecosystem I am proposing actually makes network engineers engineers again, not just vendor implementors.
Lisa,
I completely disagree. I bet 95%+ of enterprises can, and should, operate in the cloud. They will actually find it more cost effective and efficient. That dramatically changes the whole paradigm. Companies don’t need network engineers. and as we are seeing with Facebook’s open-sourced networking platform. https://code.facebook.com/posts/291641674683314/open-r-open-routing-for-modern-networks/ Companies can now configure their network with code, not Network engineers.
Enterprise networks, the way we see them today, are going to almost virtually disappear.
Stephen (or Payton?), I don’t think we completely disagree, actually. Cloud is major reason enterprise IT spend is declining. And it’s also why spend with traditional equipment vendors overall is also declining–the major cloud providers are equipped to, and do, roll their own (as the majority of enterprises are not).
That said, nothing is ever 100%, and while the economics of cloud are pretty clear for smaller enterprises, as well as new ones with no sunk IT costs, it’s often less clear for larger ones. And there are other, strategic reasons besides cost that lead many enterprises to be selective about their use of cloud. I doubt I will live to see a time when the majority of enterprises of all sizes are entirely cloud-based. You are undoubtedly correct that in-house enterprise networks in their current form will evolve, but that, too, is likely to be a very slow process in most sectors.
However, the in-house vs cloud discussion is in fact a separate issue than the white box/compute-centric/OSS/DIY discussion that Tom started, although they can intersect. The quasi-conflation of vendors and their VARs in Tom’s own post is worth examining a bit, though–mainly because the WB/CC/OSS/DIY option offers VARs an opportunity for greater independence from the vendors and their existential challenges. I think the VAR industry is at a major crossroads, but is less aware of it, collectively speaking, than the vendors they rely on. It’s not going to be pretty. =/
Peyton,
I’d like to have a conversation about the topic for sure. Without fear, uncertainty, or doubt. Which I don’t believe to have been present in my original post. I have never said that whitebox switching or open source solutions are bad or inappropriate. I just have issues with the idea that they are best solution for every vertical or customer, which is the impression given when they are discussed without qualification.
A) If VARs and vendors truly add zero value to the equation, why do they still exist? Plenty of vendors of all stripes have gone out of business because customers aren’t purchasing their equipment. VARs frequently fail because they don’t have customers. But to lump all vendors and VARs together as “pushing bad architectures to get customers hooked” is somewhat disingenuous.
There are resellers that focus on pushing equipment instead of providing services. From my experience in the industry, the parter vendors dislike these companies as much as you might. Because they aren’t adding value. They’re just taking an order and filling it for a customer. There are distribution partners that do the same thing. I would love to see companies push for more “value” being added. Even if that value is more from a support perspective instead of a services viewpoint. There are even VARs that provide significant development support to companies that can’t hire the staff to do it themselves. Are they helping to build a better experience for their customers?
B) As pointed out in the excellent comment above from Ms. Caywood of the Linux Foundation, most open source customers aren’t going to invest the time and resources into writing their own software stack. Customers may take existing projects and modify them for their own needs. Or they may have a partner do it for them. The smart VARs are starting to diversify away from hardware and bring more software teams on board.
When you build a network from scratch, are you truly building from scratch? Or are you using BiRD, Quagga, and other components like Project Calico to build a platform? And, most importantly, are the companies that utilize those platforms giving back in the form of contributions? I ask that question because some conversations I’ve had, especially on the financial side of the equation, are a solid “no”. They take their own customizations and hold them as a competitive advantage over others. That strikes me as selfish and against the ideas that open source stand for.
For what it’s worth, I really wish that Rob Sherwood’s presentation from Future:Net was able to be shared online. I think his discussion of the true costs of “build your own switch” were very in-touch with the market today.
C) If companies are purchasing equipment that they expect to be broken on installation, where does the real issue lie? Should those customers not be pushing back against buying a defective product? And if the answer is “vendors don’t listen”, why are we still buying from them at all?
Part of the key in this point is the “more advanced” companies. In your original talk, you discuss having teams of programmers at the ready to build features and test code. In my own VAR experience there is a distinct lack of programming experience in most IT departments. That’s the so-called sweet spot for a VAR. They can provide experience and talent to help customers in an area where they have none.
Those “more advanced” companies that can write their own software in days or weeks see more value in doing it than companies that have to weigh the costs of spending weeks or months training programmers versus waiting that same time for features to be deployed. And support costs can’t be understated here. If I spend my time writing a software feature, I own that feature. All the support that has to be done comes from me. If I’m a well-versed programmer then I feel comfortable solving my own problems. But if I’m not sure about my skills I would feel a lot better having someone to talk to about things.
D) Yes, I did defend VARs here as a counterpoint to the statement that they are “too hard” to deal with or don’t add any value.
If a VAR truly doesn’t offer any discounts above and beyond the list price of equipment, then it does make a lot of sense that you shouldn’t buy from them. However, in my experience working with a VAR, we did offer discounts to the customers even after we accounted for our margin. Are you going to get the gear at 40% off direct from a VAR? No, because that’s what the VAR pays. If the VAR can’t make money, they go out of business. If you can buy the gear at 40% from anywhere, you should if you are the one that’s doing the majority of the configuration and support. Even at my VAR which had to buy from a distribution partner, we had to pay that distribution partner an extra 2%. That’s the cost of doing business.
As to the comments about building directly in the cloud, that’s an entirely different conversation about the value of cloud platforms and how companies perceive the value of their infrastructure.
I’m not anti-whitebox. I’m not advocating that software development teams need to be disbanded and everyone should be forced to buy from VARs. My goal in this post and comment thread is to point out that VARs have a place in the ecosystem just as much as the DIY architecture. Large, technically advanced companies will get more value from whitebox and open source. Smaller, less technically advanced companies will still see value in partnering with a reseller. There is no “one size fits all” to this problem. Which is why there is still significant room all around for a variety of solutions.
Lisa,
I go my Peyton, :). I see your point. However I just wonder how valuable a VAR is in the next 2 years. One of my main points is that most Enterprises do not derive value from their network. They derive value from the thing that is running on top of that network. So if the software engineers come to them and say, we can build this faster and more efficiently in the cloud, or if we own the whole stack suing things like Open/R – https://code.facebook.com/posts/291641674683314/open-r-open-routing-for-modern-networks/ – and doing things like Fastly – https://www.fastly.com/blog/building-and-scaling-fastly-network-part-1-fighting-fib/ – then it is going to be hard for the business to say no. And the network engineers have no real way to control their destiny and build something comparable – in the current VAR model.
Now, if we are talking about companies that only consume software and hardware, and don’t really have engineers, but implementers, then I agree, that company won’t change. But I also believe that company won’t be around very much longer.
Thank you for these, love the discussion.
Tom,
A) I would actually prefer a webpage to order my equipment. I can get that from EdgeCor, Celestica, etc. If I need a lower price, I make a phone call. As for the services part, if my team can’t manage our network, and needs that kind of help, we should just farm the whole thing out. That kind of team only slows me down and continually creates inefficiencies. They also put you at the whim of the VAR to tell you what you “have” to have. That is a very dangerous position and just leads to you bleeding cash for very little value.
VARs exist much like KMarts exist, because they are there. They are shrinking, but to part of the engineering eco system, they are the only way to work. No it is my opinion, that companies that operate that way will be gone shortly – 3 to 5 years – or have to radically rethink the way thy manage their model.
B) I think that most companies will use BiRD, Quagga, and other components like Project Calico, Open/R, Platina, and SnapRoute – just to name a few of an ever growing space – to build a platform. You are right, open source contribution is low, but it ever growing. And it still moves faster than most vendors.
C) The problem is not with the company buying, its with the company making. Network Vendors have to try to please as may people as they can to make the most money they can, and that leads to multiple code errors. JUNOS has been a disaster every time I have deployed it, no matter the platform, because they are trying to juggle so many code changes all the time. And those changes are usually to code I don’t need, but affect me.
And this is a closed eco system. Even if the companies that are buying push back, what can they do. They have closed hardware that only works in a closed software ecosystem. They can start buying from a new vendor, but wouldn’t it be amazing if they could put any software on any hardware .. you know, like servers?
As fo you writing software, you should! And you should realize that software developers do not write all their code. They can choose open source or an integration partner to come do it for them. Besides, if you network is amazing today, you really aren’t doing anything, so why not write code? It just makes you more valuable.
D) I want to be very clear on this, VARs do not offer discounts. VARs and vendors walk in with a price list that always has a “Basic” discount. You haggle with the VAR and Vendor on a lower cost, but the VAR takes a portion of that. Somewhere between 5 and 10%. And they usually work with the vendor that gives them the highest %age, regardless of history or technical ability.
I have been able to get 92% discounts from multiple hardware vendors, with no VAR involved. If people can’t read Gartner Market reports to see the standard costs of things, then shame on them. This information is out there. Regardless of what your VAR is telling you. Oh and because they make a %age of the whole cost, they are incentivized to keep that cost as high as possible.
Look, dealing with a VAR is like dealing with a used car salesman. He keeps running back to his boss to fight for you to get the best price. (yeah right) Just go talk to his boss, and cut out the middle man. “the cost of doing business” is a euphemism, especially when network companies have 40 – 60% margin.
I guess we just fundamentally disagree. We can go back and forth here, but time will tell.
Based on the last few quarters for Juniper and Arista, and teh increase in cloud, I think these comments have aged pretty well.