Benchmark Datasets
There are a multitude of datasets available for testing face recognition accuracy.
One of the datasets that BriefCam used for benchmarking is IJB-A, which is an industry-accepted standard for face recognition benchmarking. The dataset was created by NIST and is based on public data.
The IJB-A dataset is made up of unconstrained faces “in the wild”, including variations in pose, illumination and age.
There are two important terms when looking at the benchmark:
False acceptance rate (FAR), also known as false positives, is when the system decides that the face is a match, when it isn’t.
False rejection rate (FRR), also known as false negatives, is when the system decides that the face is not a match, when it actually is a match.
When you look at a false rejection rate (FRR) you have to take into consideration the false acceptance rate (FAR), because you need both to determine the face recognition performance (you can always select a matching threshold that brings one of these metrics to 100%, but having success in both is challenging). The FAR working point determines the resulting FRR, which means that when the number of false rejections goes down, the number of false acceptances will go up and vice versa. On one hand, if you have a higher FRR, the system is less tolerant and, on the other hand, if you have a higher FAR, the system is less secure.
The testing of BriefCam’s face recognition using the IJB-A 1:N test resulted in the following:
Note
For an FAR of 1%, BriefCam v5.6’s FRR is 3.8%.
What does this mean?
When you are using face recognition for surveillance, for example when you have an FBI watchlist and you are monitoring a street, there is a 96.2% percent chance that you will find a suspect from the watchlist when he/she appears in the video. If there are 1,000 people walking in the protected area, the cost will be 10 false alarms. If this level of false alarms is too much, the FAR working point can be changed to yield a lower number of false alarms. However, this will come at the expense of false negatives, meaning more of the people on the watchlist will not trigger an alert. On the other hand, if a lower rate of false negatives is required and the operator prefers to reduce the probability of missing people on the watchlist, the result will be a higher level of false positives, meaning alerts will be triggered for people that are actually not on the watchlist.
These working point accuracies can be changed by the user by selecting different “tolerances” (i.e. face matching threshold).