District Criteria for Evaluating AI Tools: A Brisk Model Review

This product review is free based on our criteria that are grounded in the recently released US Department of Education Developers report.  We share other product reviews with our school (district) partners

TLDR

• Schools need criteria for evaluating AI apps they are being sold

• The USDOE offers some useful criteria

• Evaluating against the criteria, Brisk did well in some areas but not in others

• Districts need to carefully review Terms of Use and Privacy Policies before teachers start experimenting with apps

Introduction

Schools are currently inundated with “AI apps.” This “inundation” comes from two places –

• Company sales reps pitching districts

• Teachers using trial versions of the app and requesting paid upgrades.

Ask any district leader and they’ll tell you the requests are high and the information they need to make decisions is hard to quickly come by and organize.

In this post, we outline the criteria we use to help guide districts through decisions related to supporting particular applications.

It is based on the following assumptions

• *Applications should comply with the law and district policies

• *While districts may not want to ban teachers from using apps, their instructional technology department and “coaches” can only support teachers using so many apps. There are also financial limits to app purchases.
  *It is always possible to add more applications to the list, but it is hard to take apps away from teachers.
  *Different districts will put different weights on different criteria.
  *Experimentation isn’t inconsistent with an officially approved list

This evaluation is not designed to affirmatively support or oppose any particular app; that is the job of the district, and different districts may give weight to different criteria. We already see that with some districts choosing to block ChatGPT, others choosing to allow but not support use, and others agreeing to support use.

In some cases, this analysis exists simply to highlight what districts need to consider when implementing the product. Our goal is simply to give some initial guidance as to issues to consider and, in many instances, dig deeper with the help of IT and legal.

Our Criteria

This is our set of criteria. The criteria have been chosen based on the US Department of Education’s Developers report; the criteria are given some additional detail based on common expectations in various guidance reports and commonly supported criteria.  

USDOE criteria

• Ensuring Safety and Security

• Designed for Teaching and Learning

• Providing Evidence for Rational and Impact

• Advancing Equity and Providing Civil Rights

• Promoting Transparency and Earning Trust
  

Ensuring safety and security includes compliance with relevant laws, general protection of student and staff privacy, appropriate use based on age, cyber security, and efforts to limit misuse.

Designing for teaching and learning includes its purpose, a usable interface, teacher support, expectations of student use, 

Providing evidence for rationale and impact includes evidence of positive academic impact, an academic rationale for adopting the product, minimizing hallucinations

Advancing equity and providing civil rights includes making the technology available to all, training for equitable use, ADA compliance, 

Promoting transparency and earning trust includes transparency of tools, customer support, credible contracts/user agreements.

Districts may choose to add their own criteria.

Brisk Review

In this essay, we review one tool – Brisk – against some of the criteria we mention. Again, this is not designed to support or oppose the tool, but rather to raise some positive and negative flags based on the criteria above.

Brisk Teaching offers a comprehensive suite of core features designed to enhance educators' efficiency and effectiveness. At its heart, the extension leverages advanced AI to automate time-consuming tasks such as creating lesson plans, quizzes, and providing feedback. Seamlessly integrating with popular educational platforms like Google Docs, Google Classroom, and Canvas, Brisk enables teachers to work within their familiar digital environments. The content creation feature empowers educators to quickly generate personalized curriculum materials, lesson plans, and quizzes, saving valuable time and effort. To support diverse learning needs, particularly for ESL and special education students, Brisk includes a text leveling tool that allows teachers to adjust the reading level of various materials. The extension also provides multiple feedback modes, including "Glows & Grows," targeted feedback, and rubric criteria assessment, helping teachers offer constructive and personalized guidance to their students. Additionally, Brisk's "Inspect Writing" feature enhances academic integrity by allowing teachers to review students' revision history and detect potential AI-generated content.

Our review is based on an analysis of their “Terms of Use,” their “Privacy Notice,” our own small uses of the product, and what we’ve heard from others.

TLDR

• The language related to parental consent and potential district obligations to parents is confusing and needs to be carefully reviewed by legal counsel.

• They appear to be/claim to comply with relevant federal and state laws

• IT needs to closely evaluate Brisks’s security

• The tool has an easy and friendly user interface

• The academic quality of the output cannot be generally assessed because the user never knows what model is being queried and it doesn’t seem they have added any additional features to reduce bias, hallucination, or harmful content.

• The “Terms of Use” are not at all friendly to the customer. In some instances, the terms of use appear hostile and disrespectful

Safety and Security

Age/Parental Consent

Students 13-18 must “have and shall maintain permission of your Parent or Educational Institution.”

Students under 13 – Brisk can be used with students under 13, subject to some additional requirements (see below).

If a Parent consents, the parent becomes responsible for accepting the Terms of Service: “If you are a Parent who allows your minor child to use the Service, you accept these Terms on behalf of your child and are responsible for your child’s activity on the Service.” 

This makes sense, but if a district requests the parents provide permission, will the district first share the Terms of Use with the parents so the parents can agree to the Terms of Use? Will they notify the parents of changes (see below)?

Brisk does say schools should share their Privacy Notice with parents and notify them of Brisk’s use of third-party contractors, at least for students under 13.

This area needs more careful review and consideration, but it appears that if a school requires parental permission, both the parents and the school have more obligations (it creates parental obligations and school obligations to the parents) than if the school just grants the permission.

In general, it is worth noting that Brisk’s Terms of Use “apply to all visitors,...and parents and legal guardians of Students (“Parents”).” It’s not clear if they apply only if the parents give consent or if the terms apply to the parents merely because the students are using it with the permission of their educational institution.

Anyhow, if a school adopts Brisk, their legal department needs to analyze the terms related to age and parental permission very carefully and determine if/what type of parental permission they want to require. They must also determine what Brisk Terms of Use and Privacy Notice provisions they want to share with parents/what they are obligated to share.

^{Legal Compliance}

^“We are signers of the Student Data Privacy Pledge, have earned the Data Privacy Certification and TrustEd Apps Seal from 1EdTech, and comply with all student data privacy laws, including FERPA and COPPA. We have signed the National Data Privacy Agreement with California, Illinois, Massachusetts, Maine, Missouri, New Hampshire, Ohio, Rhode Island, Virginia, Vermont, Washington, and Wisconsin, and are open to signing district-level agreements when state-level Data Privacy Agreements are not possible. “

General Privacy

They do have a privacy policy.

We do hope the companies they work with protect the data: “Data to trusted service providers who have a legitimate need to access such information on our behalf, subject to appropriate contractual terms to protect such data. Our current list of service providers is available upon request.”

Other language that should be reviewed by IT and legal counsel.

“We care about the privacy of our Users. You understand that by using the Services you acknowledge that we will collect, use, and disclose your personally identifiable information as set forth in our Privacy Notice. “

“Brisk owns the account you use to access the Services along with any rights of access or rights to data stored by or on behalf of Brisk on Brisk servers, including but not limited to any data representing any or all of your Brisk Data. Brisk has the right to manage, control and even eliminate Brisk Data.”

“You hereby grant each User a non-exclusive license to access your User Contributions through our Services, and to use, reproduce, distribute, display and perform your User Contributions, which you make available to such User through our Services. You further grant, and represent and warrant that you have all rights necessary to grant, to us an irrevocable, perpetual, transferable, sublicensable (through multiple tiers), fully paid, royalty-free, and worldwide right and license to use, copy, store, modify, distribute and display your User Contributions: (a) to maintain and provide our Services; (b) other User Contributions submitted as personal messaging, to improve our products and services and for our other business purposes, such as data analysis, customer research, developing new products or features, and identifying usage trends; and (c) to perform such other actions as described in the Brisk Privacy Notice or as authorized by you in connection with your use of our Services.”

I’m trying to figure out what this means, but it is not super-encouraging.

General Security & Prevention Malicious Use

Security professionals will need to evaluate this: “Please do not do anything to harm our product or try to hack our Services” and determine if any other language is relevant.

Designing for teaching and learning

Interface Usability.  Interface usability seems reasonably strong.

Advancing equity and providing civil rights

The DOE report makes several recommendations related to equity and civil rights.

Bias reduction. The DOE encourages developers to reduce bias as much as possible. I cannot find documentation related to how Brisk does this and the effectiveness of any efforts. Also, since we don’t know what language model Brisk is prompting, we don’t know what any default bias may be.

Algorithmic discrimination. Again, I cannot find any information related to this.

Access for individuals with disabilities. As a Chrome extension, this likely has the same limits and opportunities in regard to this issue as any web browser.

General equality and equity. Access to the resource and equity will largely be determined by the resources that are invested in it by the district.

Providing evidence for rationale and impact

Evidence of learning

Brisk doesn’t provide any evidence that their app promotes learning, but this is obviously difficult at this stage and a 5+ year longitudinal study of an AI product may be irrelevant in 5 years. I do not hold any such app to a high standard here, but it’s reasonable to assume that an app such as this that helps teachers prepare for class might boost learning. It will be similar to the impact of other LLMs or other wrappers (School AI, Magic School, etc.).

Learning Objectives. Brisk allows teachers to establish learning objectives.

Formative Feedback. Brisk Teaching offers a comprehensive suite of AI-powered formative feedback tools designed to support teachers in providing timely and effective guidance to students. The platform's features include "glow and grow" feedback highlighting strengths and areas for improvement, rubric-aligned comments, and actionable next steps for student development. Importantly, Brisk positions itself as a time-saving assistant rather than a replacement for teacher expertise. Teachers can quickly generate initial feedback using AI, which they can then review, refine, and personalize before sharing with students. The Targeted Feedback Generator, a premium feature, allows for direct comments on Google Docs. By streamlining the feedback process, Brisk enables teachers to provide more frequent and timely input throughout the learning journey, helping to identify knowledge gaps and guide student improvement. The platform also emphasizes educating teachers on effective AI prompt usage, ensuring that the technology enhances rather than replaces human input in the formative assessment process.

When utilizing generative AI systems for formative feedback on writing, teachers must be aware of several critical limitations. Firstly, AI lacks true comprehension of content and context, often identifying only surface-level issues while missing deeper conceptual problems or nuanced arguments. Additionally, AI models can perpetuate biases from their training data and may produce inaccurate or misleading feedback, known as "hallucinations." These systems also struggle to assess creativity and originality, particularly when faced with novel ideas or unconventional approaches that deviate from their training patterns.

Furthermore, AI cannot consider individual student needs, such as personal growth, specific learning goals, or emotional states when providing feedback. There's also a risk of overreliance on AI feedback by both teachers and students, which could potentially stifle critical thinking and writing skill development. AI may lack contextual understanding of assignment goals or the broader context of a piece of writing. Lastly, the use of AI systems for feedback raises privacy concerns regarding student data storage and protection.

Given these limitations, teachers should not rely solely on AI systems for formative feedback. Instead, they should use AI as a complementary tool within a broader assessment strategy that maintains the crucial human element of personalized, context-aware feedback. Teachers should focus on providing meaningful feedback that addresses deeper aspects of writing beyond what AI can evaluate, and help students critically assess AI-generated feedback. By adapting teaching practices, promoting active learning, and addressing ethical concerns, educators can leverage AI as an opportunity to enhance instructional methods while ensuring transparency and considering issues of equity and access.

When teachers use AI for formative feedback, their effectiveness will largely be determined by how much training they receive and how to use the tool appropriately.

Individualized Instruction. There are several types of individualized instruction:

1. Differentiated Instruction: Adapting content, process, or product based on students' readiness, interests, and learning profiles.

2. Personalized Learning: Tailoring the learning experience to individual student goals, interests, and needs.

3. Adaptive Learning: Using technology to adjust the difficulty and content of material based on student performance.

4. Mastery-Based Learning: Allowing students to progress at their own pace, moving on only when they've mastered a concept.

5. Project-Based Learning: Engaging students in complex, real-world projects that cater to their interests and skills.

Brisk Teaching provides several tools that support aspects of individualized instruction:

1. Targeted Feedback: Brisk offers AI-powered tools to generate personalized feedback on student work, including "glow and grow" comments and rubric-aligned feedback.

2. Differentiated Resources: The platform can help create differentiated instructional materials and modify existing content for different learning levels.

3. Personalized Communications: Brisk can generate individualized letters of recommendation and student behavior notes.

4. Integration with Google Tools: This allows for seamless incorporation of individualized instruction strategies within familiar platforms like Google Docs and Classroom.

However, Brisk does not provide:

1. Fully Adaptive Learning Systems: While Brisk offers tools to support individualization, it doesn't provide a comprehensive adaptive learning platform that automatically adjusts content difficulty based on student performance.

2. Standalone Mastery-Based Learning Programs: Brisk's tools can support mastery-based approaches, but it doesn't offer a complete system for implementing this type of instruction.

3. Comprehensive Project-Based Learning Management: While Brisk can assist in creating materials for projects, it doesn't provide a full project-based learning management system.

4. Direct Student Interaction: Brisk is primarily a tool for teachers and doesn't directly interact with or assess students independently.

Hallucination. The DOR report encourages developers to reduce hallucinations.

It is hard to evaluate the hallucination rate because they won’t disclose which LLM they use (see below), but in incidental work, it substantially hallucinated on a simple piece of debate content very severely. Claude got it right, almost perfectly.

Brisk — Completely wrong

Claude

Claude gets it right, except for the fact that the speaker's times for the Summary changed to 3 minutes.

Their take on hallucination is only that they acknowledge it: USE OF THE SERVICES MAY RESULT IN INCORRECT OUTPUT THAT DOES NOT ACCURATELY REFLECT REALITY. YOU MUST EVALUATE THE ACCURACY OF ANY OUTPUT AS APPROPRIATE FOR YOUR USE CASE, INCLUDING BY USING HUMAN REVIEW OF THE OUTPUT.

They do not describe any efforts to reduce hallucinations. And since we don’t know what LLM Brisk prompts, we can’t suggest what the rate may be.

That all said, the second time I asked it, it did do a much better job and generated a pretty good presentation, with some limits.

And it generated it from a pretty simple prompt: “Make me a slide presentation for an introduction to Public Forum debate.”

This time it gets the basic speech order correct, but the flow chart on the left does not match the speech order.

This leaves out the crossfire after the Summary speeches, but it’s otherwise accurate.

The presentation continues, but it generally does a very good job, minus a few incidental hallucinations.

Teachers who are using this tool to develop instructional materials for their content area are going to like it, as they’ll know how to quickly correct the hallucinations. If anyone is using this for something they don’t have expertise in, they may end-up presenting flawed information (which is the case with any generative AI tool). Nonetheless, teachers are likely to enjoy the time saved.

Promoting transparency and earning trust

This is the weakest category of this tool in my opinion.

Model Transparency

Department of Education: “Trust is buoyed by transparency and disclosure” and “Trust is enhanced by demystifying why, how, and what AI is employed”

Brisk does not disclose what model is being used when processing your query/prompt.

 “Brisk Teaching uses a range of Large Language Models to support different use cases. This way, teachers can get the best of all worlds because Brisk Teaching can figure out which AI model will work best for their task.

Since we don’t know where the request is being processed, or even where it may be processed, this makes it difficult to determine if it meets many of the criteria discussed above.

And even if you got them to disclose the models they work with, they claim they alter the model based on the query, so you can’t really know.

Training on User Data

“No, user input is not used to train any AI models.”

This is good, but since we don’t know what underlying LLM they are using, we can’t say how the data is likely being used by that LLM.

A contract one can count on

According to Brisk, the contract can change on a whim, and district officials are supposed to continually look on the website to see if there are any changes, including ones that could leave them out of compliance with their own district policies.

“Because our Services change relatively often, the terms in this Agreement and our Privacy Notice may change too. Upon making changes, we will update the “Effective Date” found at the top of this page. Your continued use of the Services after any changes constitutes your acceptance of the new terms.products and services of Brisk Labs Corp. ("Brisk," "we," "our," or "us") that link to this page, such as Brisk’s websites, online platforms and web browser extensions (collectively, our "Service").”
Note: The typo is theirs.

And if you don’t like it, you can quit: “If you do not agree, do not use the Services.”

Then there is this line: “We may terminate this license at any time for any or no reason.”

Personally, I (Stefan) think this line is disrespectful. Only a company that thinks someone will not read the terms or understand the terms would put this in, as at least I would never agree to any contract with such a line.  Who would?

Notice

“Notice” and how notice is provided are pretty basic elements of a contract. Some might even argue that notice provisions are required for a contract to be valid (check with your lawyer).

Brisk’s take on notice: “We continually update our Services to offer the best possible product. While we plan to continue to offer and improve our Services, we may, *without prior notice,* change, cancel, create usage limits for, or permanently or temporarily stop offering or allowing you access to the Services generally (“Service Changes”)

This said, they say they will provide notice: “If we make material changes to this Agreement, we will provide reasonable notice to you, such as by email to your registered email address, by prominent posting on this website or our online services, or through other appropriate communication channels.”

So, they will provide it, but they don’t have to, and such notice may require you to just check the website regularly to see if you notice any change in the terms somewhere. Perhaps you can have an AI monitor it for you?

Finances

Again, Brisk “may terminate this license at any time for any or no reason.”

And if they do, well…..“In the event that Brisk suspends or terminates your account or this Agreement, you understand and agree that you shall receive no refund or exchange for any unused time on a subscription, any license or subscription fees for any portion of the Services, any content or data associated with your account, or for anything else.”

So, for no reason at all, they can terminate the contract and keep your money.

Protection against dangerous content

They acknowledge the user may be exposed to dangerous content but do not indicate any steps they take to reduce a user’s exposure to such content.

Acknowledge it, do not claim any protection: YOU UNDERSTAND AND AGREE THAT THE OUTPUT MAY CONTAIN “HALLUCINATIONS” AND MAY BE INACCURATE, OBJECTIONABLE, INAPPROPRIATE, OR OTHERWISE UNSUITED TO YOUR PURPOSE, AND YOU AGREE THAT WE SHALL NOT BE LIABLE FOR ANY DAMAGES YOU OR ANY THIRD PARTY ALLEGES TO INCUR AS A RESULT OF OR RELATING TO ANY OUTPUT OR OTHER CONTENT GENERATED BY OR ACCESSED ON OR THROUGH THE SERVICES.”

“We cannot review all User Contributions before they are Shared and cannot ensure prompt removal of objectionable material after it has been posted. We assume no liability for any action or inaction regarding transmissions, communications, or content provided by any User or third party.”

Note: I don’t understand why some things are in caps and others aren’t and if ALL CAPS has any legal significance.  Are they more committed to the contract terms in all caps? Is the user more obligated to follow them.

If anything goes wrong, it’s your fault: “*** You acknowledge that the Services are provided on an “AS IS,” “WITH ALL FAULTS,” and “AS AVAILABLE” basis, including without limitation with respect to any AI Features. The entire risk associated with the quality, performance, accuracy, and effort of the Services lies with you. To the maximum extent permitted by applicable law, the Services are provided without warranties of any kind, whether express or implied, including, but not limited to, warranties or conditions of, and all liability for: (a) merchantability; (b) fitness for a particular purpose; (c) workmanlike effort; (d) title, non-infringement, and breach of third party rights; (e) adequacy, accuracy, timeliness, and completeness of content or results; (f) arising through course of dealing or usage of trade; and (g) uninterrupted or error-free access or use, in respect of the Services and any outputs therefrom.”

Technical errors. After about 20 queries, I only had one technical error (it didn’t produce what I wanted).

Tech support. It’s unclear what technical support is offered.

Accountability. The Terms of Use swear off any accountability for Brisk and require the user to indemnify the company. Since we don’t know what model is being used and there is no evidence of positive academic impact, accountability is difficult.

Brisk requires the user to do nothing more than put a checkbox on the Terms of Use. The user doesn’t even have to open them.

Additional Notes/Criteria

Teachers who require students to use the service have significant responsibilities. “If you are an Educational Institution or Teacher who intends to make the Services available to Students for use in an educational setting, you must contact us to enter into a student data processing agreement before permitting Students to use the Services.”

This is an illustration of why the district must approve apps.

Is it redundant with anything? Before purchasing Brisk, districts should determine what features in Brisk may be redundant with other similar products. Limiting purchases/consolidating apps can save districts money and allow district Coaches to focus. Other than a Chrome extension to make it simple, I don’t see any unique features of this tool.

Teacher support and use. Ultimately, apps will only be successful if teachers want to use them. From what we’ve heard, there is support for Brisk.

Are other schools using it? Widespread use by other schools offers some evidence that others find it useful and safe/compliant.

School leverage. Legal and technical experts should evaluate compliance with relevant laws and any security-related issues with the Chrome extension. Districts should also insist on more reasonable contract terms, both to protect themselves and teachers.