How to Review Labels

Label Review is the process of reviewing and editing the label suggestions created by the Large Language Model (or LLM) to ensure the highest possible accuracy.

When reviewing a label, you have the option to Accept it, Edit it or Remove it. You can also Manually Add a missing label.

To Review Labels:

• From your Project Dashboard, go to Label Review. This is also the default screen for users who don't have the Manage Team or Upload permissions.
• You can review labels a patient at a time, or one variable at a time.
• After clicking on a patient or a variable, you'll see a table of labels for that patient or variable, along with the raw text and actions.
• Click "Accept" to take a label as is.
• Click "Edit" and enter the new value if the evidence is correct but the interpretation is incorrect.
• Click "Remove" if the displayed evidence is irrelevant.

There are two levels of Review. You will likely toggle between them as you review a patient:

• Review in the Label Table
  • When you select a patient, you’ll see a table with all of the labels generated by the LLM, along with the raw text evidence.
  • If this raw text is enough to review the label, you can Accept, Edit, or Remove the label in line.
  • Otherwise, you can click the link next to the text evidence to Review in the Document Context.
• Review in the Document Context
  • Reviewing in the document context allows you to view the full document, along with all of the variables using evidence from that document and the evidence highlighted in-line.
  • You can Accept, Edit, or Remove the label using the left pane.
  • From this view, you can also manually add a label using the link at the top of the left pane.

  • To return to Review in the Label Table, use the X at the top left of the document view.

    
    

Confidence

Brim gives each label a “confidence” rating, which is the LLM’s confidence that the label is correct. 

Higher values indicate a higher likelihood that the label is correct. This value will be blank if the system didn’t find any evidence in the suggested scope.