Best Channel Products 2011 Survey Methodology

How were the Best Channel Products survey and special report generated? Just like our previous surveys, we partnered with Penn State University to help ensure the survey's statistical accuracy. Penn State agreed with our philosophy of conducting a Web-based survey (as opposed to a phone solicitation survey) of our subscribers to capture significant data from our most active resellers. For the January 2011 "Best Channel Vendors" issue of Business Solutions (BSM), we surveyed our VAR subscribers and asked them to rate their vendors in seven categories, including three product-related categories: Product Features, Product Reliability, and Product Innovation. Vendors who scored highest in those three categories were contacted and asked to submit a product for each applicable technology category. The technology categories were determined by the BSM editorial staff.

Who participated in the survey? How did they vote? From late April until early May, BSM VAR subscribers were asked to participate in the Best Channel Products survey. We asked those subscribers to rate the nominated products in five categories. We also provided some guiding questions to ensure consistency in how VARs interpreted each category:

• Richness Of Features/Functionality: Does the product meet your customers' functionality needs? Is it easily configurable? Is it easily customizable (i.e. scalable and flexible)?
• Product Reliability/Durability: Does the product meet your quality needs? Do you frequently need to fix the product? Is it durable enough to withstand regular usage?
• Ease Of Integration: Is it easily integrated with other products — new or existing?
• Ease Of Upgrade: Is it easily upgradeable to meet your customers' needs and technology changes?
• VAR's Ability To Service: Is the product easy for you to service?

How many VARs participated? Did you throw away any votes? Despite the survey being open for a shorter window of time compared with previous surveys, we had one of the best voter turnouts. In the end, we collected 11,711 votes from 1,490 discrete reseller subscribers, eclipsing the 1,416 ballots gathered in our 2011 Best Channel Vendors survey. The total number of votes was down compared with last year's survey due to reducing the number of vendors included in the survey (in an attempt to shorten the survey). We also eliminated many ballots that could not be verified as BSM VAR subscribers, votes from vendors, or those that were obvious attempts to drop a vendor's score.

How were the Best Channel Products determined? An overall average score for each product was determined by adding the cumulative scores of the five categories and dividing that score by the number of VAR subscribers who rated that product. From that list, our editorial staff chose the Best Channel Products by applying a "Scree Test." We won't go into the full definition here — we'll leave that up to the Penn State statisticians to explain — but basically our editors applied reasonableness to the results and drew the line where there was a clear break between the highest scorers and the "rest of the pack." Our goal was to reflect the sense of the survey respondents and let our readership know who they felt produced the Best Channel Products.

How are the Best Channel Products listed? They are listed first by technology category (Data Collection through VoIP [Voice over Internet Protocol]) and then by technology subcategory. Best Channel Products are listed in alphabetical order. We did not rank them according to their overall average score. One of the main purposes of this special report is to give our resellers a guide to what their peers think are Best Channel Products. Ranking the products according to their overall average score would mislead readers and defeat that aforementioned purpose.

Additionally, based on reseller feedback on past surveys, we learned that many readers misinterpret statistics. For instance, if they saw Company A scored 3.73 in Ease Of Integration and Company B scored 3.68 in that same category, they were led to believe that Company A was clearly better in that category. The scores should not be applied in that way because this survey entails a level of subjectivity. The report should be used as a guide to what your peers think are the Best Channel Products. Every product on the list is worthy of high praise.

How did you make sure the product comparisons were "apples to apples"? To make the survey as valid and valuable as possible, we divided the products into appropriate, specific subcategories. These subcategories helped us avoid "apples-to-oranges" comparisons. If necessary, to explain the details of each product subcategory, those sections feature "Editor's Notes" from the BSM editor for that technology.

Did you "fudge" any scores to make sure your advertisers were included on the list? Did any vendors buy their way onto the list? No. Other magazines have skewed their surveys to include current advertisers or entice new advertisers. We believe engaging in that behavior is not only a disservice to the industry, but it is dishonest. We would not compromise the reputation our magazine has built during the past two+ decades for the extra revenue a few ads would generate. Plus, Penn State would not compromise its reputation as an outstanding university by condoning such a practice.

OK, so the survey is accurate. But, is it perfect? Certainly not. Penn State stressed to us that all surveys have their limitations. So, this survey is not perfect, but it should be an excellent tool for the channel because it indicates products and vendors that have highly satisfied VARs.

What should I do if I have a comment about this special report? There is a featured discussion on BSM's LinkedIn (BSMinfo.com/LinkedIn) about the Best Channel Products issue. Please add your comment to that discussion.