As part of our updated evaluations, we are including a numerical roll-up score with our summary reports. We want to be precise when we explain what this number means, and what it doesn't mean.
The numerical roll-up score should only be used as part of a decision-making process that includes the rest of the evaluation. The number's best use is as an indicator of how much additional work a person will need to do to make an informed decision about a service. This use is directly related to the core work driving the evaluations: to help people make informed decisions about a service with less effort. The higher the number, the less effort required to make an informed and appropriate decision.
The numerical roll-up is not a grade, rating, indication of the overall quality of the service, or measure that has any use outside the context of the rest of the evaluation. We understand the temptation to misuse the roll-up in this way, but that is not an accurate or reliable use of the calculation.
The evaluations come with five numerical roll-up scores: one for each of our core categories (Safety, Privacy, Security, and Compliance) and a top-level roll-up score for the entire service. Our calculations emphasize transparency and disclosure within a vendor's terms, with additional points earned for disclosures in policies that protect student data. A current snapshot of evaluated services shows numerical roll-up scores ranging from a low of 10 to a high of 75, with an average of 43 and a standard deviation of 13. These numbers will fluctuate almost daily as evaluations are added and updated, so any examples are best understood as snapshots in time.
To explain how questions affect a numerical roll-up score, we will look at question 3.2.4 from our published list of questions: Do the policies clearly indicate whether or not personal information is shared with third parties for advertising or marketing purposes?
At a high level, this question has three possible responses:
- The policies say nothing about whether or not personal information is shared with third parties for advertising or marketing purposes.
- The policies clearly indicate that personal information is shared with third parties for advertising or marketing purposes.
- The policies clearly indicate that personal information is not shared with third parties for advertising or marketing purposes.
In calculating the numerical roll-up, not sharing any information (the first option) brings the roll-up down the most, because without any information, making a fully informed decision is not possible.
Clearly specifying that personal information is not shared (the third option) increases the roll-up score the most. If a vendor is transparent and discloses that their practices protect information, the score increases the most.
The second option -- if a vendor clearly indicates that they do share personal information for marketing purposes -- still earns points toward the roll-up, because the disclosure helps a person make an informed decision. Because we look at privacy through the lens of making an informed decision, we encourage transparency in policy disclosures as a needed tool to help people make informed decisions.
Our evaluation process and numerical roll-up scores also take into account whether or not a question is relevant. This radial graph shows the relationships and dependencies among questions. Not every question is relevant for every service, so we exclude irrelevant questions from our overall scoring.
While we recognize that calculating a numerical summary to a service invites misuse of the number, we are cautiously optimistic that people will use this numerical summary as intended: as one piece of information among many designed to help people make an informed and appropriate decision for their school or district.