Evaluation Process

Our evaluation process for apps attempts to address some common barriers to evaluating privacy practices. Privacy concerns and needs vary widely based on the app and the context where it is used. For example, it makes sense for a student information system to collect a home address; it wouldn't make sense for an online calculator to get a home address. Because we pair a transparency evaluation with a qualitative evaluation, we can track what policies cover alongside the strengths and weaknesses of how data is handled. Then, our summary evaluation allows us to highlight the implications of privacy practice alongside the goals and contexts within which the app can be used.

The evaluation process contains five steps.

  • Step 1: Retrieve privacy policies.
  • Step 2: Complete transparency review.
  • Step 3: Complete qualitative evaluation.
  • Step 4: Highlight the qualitative responses that best demonstrate both the strengths and concerns of the app regarding privacy and security practices.
  • Step 5: Prepare a summary evaluation that highlights how the app manages privacy and security in these categories: Digital Footprint, Advertising and Data Profiling, Information Security and Data Breaches, and Legal Compliance.

All steps of the evaluation process will be reviewed by trained staff. In cases where questions or concerns arise about a specific product, we will attempt to contact vendors directly and give them the opportunity to address them. Obviously, any security issues will be addressed via responsible disclosure.

Over the next few months, we will be writing in more detail about the different steps of the evaluation process. Our goal is to be as transparent as possible. We're taking this approach for many reasons, but to start, the irony of doing privacy work in an opaque manner is more than we can bear. Additionally, we don't want anybody to be surprised by the work we're doing -- we want to be clear on how we're framing concerns and why.