Best Channel Products 2009 Survey Methodology

How were the Best Channel Products survey and special report generated? Our first step was to partner with Penn State University to help ensure the survey’s statistical accuracy. Penn State agreed with our philosophy of conducting a Web-based survey (as opposed to a phone-solicitation survey) of our subscribers to capture significant data from our most active resellers. For the January 2009 “Best (and Worst) Channel Vendors” issue of Business Solutions, we surveyed our VAR subscribers and asked them to rate their vendors in seven categories, including three product-related categories: Product Features, Product Reliability, and Product Innovation. Vendors who scored highest in those three categories were contacted and asked to submit a product for each applicable technology category. The technology categories were determined by the BSM editorial staff.

Who participated in the survey? How did they vote? From late May until mid-June, BSM VAR subscribers were asked to participate in the Best Channel Products survey. We asked those subscribers to rate the nominated products in five categories. We also provided some guiding questions to ensure consistency in how VARs interpreted each category:

• Richness Of Features/Functionality: Does the product meet your customers’ functionality needs? Is it easily configurable? Is it easily customizable (i.e. scalable and flexible)?
• Product Reliability/Durability: Does the product meet your quality needs? Do you frequently need to fix the product? Is it durable enough to withstand regular usage?
• Ease Of Integration: Is it easily integrated with other products — new or existing?
• Ease Of Upgrade: Is it easily upgradeable to meet your customers’ needs and technology changes?
• VAR’s Ability To Service: Is the product easy for you to service?

How many VARs participated? Did you throw away any votes? The volume of responses was overwhelming. We collected 19,833 votes from 2,186 unique reseller subscribers, eclipsing the 17,771 votes from 1,604 resellers gathered in our Best Vendor survey. The 2,186 resellers represent an estimated $14.3 billion in annual sales, an average of $6.5 million per participating VAR. Yes, we eliminated several ballots that could not be verified as BSM VAR subscribers.

How were the Best Channel Products determined? An overall average score for each product was determined by adding the cumulative scores of the five categories and dividing that score by the number of VAR subscribers who rated that product. From that list, our editorial staff chose the Best Channel Products by applying a “Scree Test.” We won’t go into the full definition here — we’ll leave that up to the Penn State statisticians to explain — but basically our editors applied reasonableness to the results and drew the line where there was a clear break between the highest scorers and the “rest of the pack.” Our goal was to reflect the sense of the survey respondents and let our readership know who they felt produced the Best Channel Products.

How are the Best Channel Products listed? They are listed first by technology category (Bar Code Labeling through VoIP) and then by technology subcategory. Best Channel Products are listed in alphabetical order. We did not rank them according to their overall average score. One of the main purposes of this special report is to give our resellers a guide to what their peers think are Best Channel Products. Ranking the products according to their overall average score would mislead readers and defeat that aforementioned purpose.

In the Best Channel Vendors special report in January, you listed scores from their top categories. Why didn’t you do that here? Based on reseller feedback over the past several months, we learned that many readers misinterpreted the statistics. When they saw Company A scored 3.73 in Channel Program and Company B scored 3.68 in that same category, they were led to believe that Company A was clearly better in that category. The scores should not be applied in that way because this survey entails a level of subjectivity. The report should be used as a guide to what your peers think are the Best Channel Products. Every product on the list is worthy of high praise.

How did you make sure the product comparisons were “apples-to-apples”? To make the survey as valid and valuable as possible, we divided the products into appropriate, specific subcategories. These subcategories helped us avoid “apples-to-oranges” comparisons. To explain the details of each product subcategory, those sections feature an “Editor’s Comment” from the BSM editor for that technology.

Did you “fudge” any scores to make sure your advertisers were included on the list? Did any vendors buy their way onto the list? No. Other magazines have skewed their surveys to include current advertisers or entice new advertisers. We believe engaging in that behavior is not only a disservice to the industry, but it’s dishonest. We would not compromise the reputation our magazine has built over the past two-plus decades for the extra revenue a few ads would generate. Plus, Penn State would not compromise its reputation as an outstanding university by condoning such a practice.

OK, so the survey is accurate. But is it perfect? Certainly not. Penn State stressed to us that all surveys have their limitations. So, this survey is not perfect, but it should be an excellent tool for the channel because it indicates products and vendors that have highly satisfied VARs.

What should I do if I have a comment about this special report? There is a featured discussion on BSM’s LinkedIn (BSMinfo.com/LinkedIn) about the Best Channel Products issue. Please add your comment to that discussion.