IT Test Lab Ensures Quality Storage Solutions

All VARs must walk the fine line between offering cutting-edge technology solutions to customers and supplying reliable products. A few years ago, Robert Brumwell, president of systems integrator Daytona Storage, learned a hard lesson. If you don't fully test the new products hitting the technology market, you end up losing customers when those products fail to meet expectations. That's when Brumwell made the decision to invest in a lab facility where his engineers test every product and solution Daytona sells. The end result was 40% growth last year.

In fact, the lab has become so integral to the growth of Daytona that Brumwell will not deploy or even start to train technicians on a product until it is determined — in the lab — to be a dependable fit for existing and potential customers in the verticals Daytona serves (education, state and local governments, banking, healthcare, and small business). "Daytona's future really solidified once I hired engineers and launched our lab," says Brumwell. "Having a lab has allowed us to train our staff faster, provide better professional services, and find exciting new products."

After launching Daytona out of his family room in 2001, Brumwell's business eventually grew to the point where he was leasing office space to have room for staff and customer meetings. Then he ran into a roadblock. After pushing out new solutions to two customers, the products failed, and he lost those accounts. "We had a couple accounts where we flipped out new equipment, including one school district where a storage area network [SAN] product completely and utterly failed," explains Brumwell. "My contact, who went to the school board to get that project money, still shakes my hand, but he will never buy from me again." Brumwell says that single failure has probably cost his company tens of thousands a year in sales given the size of the school district. "I knew right then, we needed to test everything we sold," he says.

Balance Talent, Cost When Hiring Engineers
"The first engineer you hire has to be a great engineer, and you should be ready to give a lot to him," says Brumwell. But beyond finding the perfect technologist, you need a partner who shares the vision of your company's evolution. "You have to talk to your engineer about your vision as well as the role he will play and how he will be involved in the business. Your engineer needs to be your ‘#2 guy'; you will lean on him a lot."

Brumwell cautions that finding that first perfect engineer can take time. In his case, serendipity interceded. After hiring and firing two engineers, Brumwell found his partner and head engineer, Chuck Struthers. "I made him an offer that was 60% of my total professional services revenue at that time, and I agreed to pay for any additional training he would need to do his job well and to establish the test lab," says Brumwell. With Struthers in place, Brumwell then needed additional engineers. He recommends looking at a pool of applicants ranging from graduates fresh out of college to potential hires with five years experience at Best Buy or similar technology retailers. Then, put them through their technical paces in the lab, but also look at their personalities. "We give technology tests to our potential engineers — sometimes we give them an impossible task — to check not only their skills but how they handle the stress of the ticking clock on the wall," says Brumwell. From there, he looks for engineers with an appetite to learn, good business sense, and strong people skills. "Sometimes that guy from Best Buy is a better choice than the one with a fouryear degree from a major university because he has spent the last five years not only working with technology, but handling customers in their homes on service calls," says Brumwell.

Once an engineer is hired — Daytona has five — that person is trained on every product in the company's portfolio and then used in the lab to determine what products will land on the line card. Because they are in the lab setting, junior engineers can learn both broad competencies and the product set in relatively short amount of time. Daytona pays for all testing and certifications for its engineers, budgeting as much as $25,000 a year. The company does require engineers to sign a noncompete agreement, and if they leave within three years, they must reimburse the company for any training costs.

Prepare To Invest In Storage Components
If you are considering a lab, do not underestimate the financial sacrifice you'll make. Brumwell says the initial investment for the Daytona lab was approximately $100,000, not including the cost of the engineers. The lab began with an empty server rack, which Brumwell and Struthers slowly populated with components. After starting with a small SAN and a couple servers, the men added VMware, which allowed them to create more environments with far less equipment. Brumwell has a litany of advice for any VAR ready to build a lab. For example, Brumwell says if you are going to partner with a vendor, and they offer 50% off a demo product, only take the deal if that product is going to be a mainstay in your environment — and your customers' networks. Another tip is to get a slightly older version of the equipment or a trade-in of the current version. Once you've vetted that appliance, ask for a demo unit of the newest version and compare. "Don't give in to their pressure to buy demo units," says Brumwell. "If you do buy a demo, make sure there are no restrictions on reselling it in the next six months." When it comes to servers, Brumwell advises units that are current and relevant, but not necessarily the latest and greatest. "You don't need a new server each year, but you can't be running on 5-year-old servers either," he cautions. Rather, cycle your products in and out, keeping them as long as they are relevant. If you really insist on getting new products, lease them.

In terms of software, Brumwell pushes for not-for-resale licenses from vendors. If you must buy something from vendors, don't be shy about asking for something in return. "If you are buying anything from a vendor you are planning to sell, ask for leads at the time of purchase," he says. In all, Daytona's lab took nearly a year of building, modifying, and rebuilding to be fully operational. Be prepared for that.

Set Benchmarks For Storage Performance
Once the lab was up and running, Daytona's engineers developed a list of must-haves and a method for benchmarking performance. The process starts with the normal deployment steps — set up, plug in, use the graphical user interface (GUI) — before the technical probing begins. "We try to break it," says Brumwell. "We unplug the controller, pull out the disk drive, shut down the network, and then turn it back on." Along the way, the engineers look at how the product handles those interruptions and failures.

Examples of products tested by Daytona are VMware, backup, and storage solutions. With VMware, the lab team concentrates on testing application compatibility and the impact upon a virtualized session or server. "We track CPU, memory, disk, and network utilization for the most common applications that our customers will be using," explains Brumwell. That allows the VAR to give accurate estimates for what can or cannot be done. It also provides Daytona time to proactively troubleshoot potential problems and pretest all patches and upgrades before deploying them in the field. When it comes to backup solutions, Daytona concentrates on what servers and applications can be backed up, how long the backup process takes, how easy it is to manage the backups, and how the reporting and diagnostic tools perform. "For storage, we focus on both the ease of use and the performance, as well as the failover capabilities of the product," explains Brumwell. "If those tests work out well, we then look at the additional features offered and weigh the benefits." Daytona only tests products that fit with its vision for growth, which focuses on virtualization, backup, storage, archiving, and security. Once it has tested its products and is moving forward with a customer project, the VAR prebuilds each project in the lab.

Leverage Storage Expertise Into Revenue
The lab's impact on Daytona's growth is illustrated by the profiles of its customers. Brumwell says, "You don't get to work for a major customer like the state of Ohio without a system for testing and prebuilding; you simply run into too many problems." Not only does the lab allow Daytona to deliver reliable solutions, it helps its engineers learn how to troubleshoot problems and the sales staff to talk with more firsthand knowledge than competitors. That makes the lab a differentiator, says Brumwell. Clearly, Daytona's ability to produce reliable, unique solutions has translated into growth. This year, it is on track for 25% revenue growth.

Brumwell says if you are a VAR thinking about building a lab, there are a handful of questions to answer before you start. The fundamental point is determining what underlying technology you as a VAR want to work with and what broad vendor set you want to align with. "You have to decide if you're a Dell guy, a VMware guy, whatever," advises Brumwell. "Then pick what products play in that space and start to test. For example, if you are doing infrastructure work, build out an Internet small computer system interface [iSCSI] and a fibre channel, and then add servers and VMware. Once you've done that, then start looking for new, game-changing products in your field." From there, you can enjoy the solid growth that has brought Daytona from being a one-man business in a family room to the systems integrator of choice for the state of Ohio.