Sometimes in your job, your normal way of doing things won't work. Before you spend an inordinate amount of time trying to shove a square block into a round hole, go to problem solving instead!
When you realize the typical way of validating a new algorithm is not possible, it's time to stop trying to get your algorithm to be 100% perfect because there is no way to validate it without labeled data! Critical thinking time! Problem solving time!
Problem 1: You realize that you do not have ground truth to validate a new motion detection algorithm you are developing. What do you do?
Possible Answers:
How much time would it take to get the ground truth? In my project, I could get some periods of motion by watching a video recording for all 160 hours of footage. This would take too much time! And even if I did watch all the footage, I may not be able to distinguish all periods of motion, meaning I still wouldn't have all of the ground truth.
There were alternative ways of validating the algorithm.
I did have a set of times where it was likely that motion was occurring. However, these periods were not perfect. Sometimes it was labeled but there was no motion. Conversely, there were no labels when there definitely was motion.
One way to validate the algorithm is to compare it against another published method using the labels as if they were the absolute ground truth. Does the new algorithm detect more or fewer motion periods? Does it produce more or fewer false alarms?
A second way of doing it is plotting the number of labelled events per study on the x-axis. And then plot the number of detected motion events with your new algorithm on the y-axis. This will show a relationship that says the new algorithm in general detects 30% of the labeled periods as motion across all 20 studies. So even if the algorithm isn't perfect, it shows that it's consistent.
When you realize the typical way of validating a new algorithm is not possible, it's time to stop trying to get your algorithm to be 100% perfect because there is no way to validate it without labeled data! Critical thinking time! Problem solving time!
Problem 1: You realize that you do not have ground truth to validate a new motion detection algorithm you are developing. What do you do?
Possible Answers:
How much time would it take to get the ground truth? In my project, I could get some periods of motion by watching a video recording for all 160 hours of footage. This would take too much time! And even if I did watch all the footage, I may not be able to distinguish all periods of motion, meaning I still wouldn't have all of the ground truth.
There were alternative ways of validating the algorithm.
I did have a set of times where it was likely that motion was occurring. However, these periods were not perfect. Sometimes it was labeled but there was no motion. Conversely, there were no labels when there definitely was motion.
One way to validate the algorithm is to compare it against another published method using the labels as if they were the absolute ground truth. Does the new algorithm detect more or fewer motion periods? Does it produce more or fewer false alarms?
A second way of doing it is plotting the number of labelled events per study on the x-axis. And then plot the number of detected motion events with your new algorithm on the y-axis. This will show a relationship that says the new algorithm in general detects 30% of the labeled periods as motion across all 20 studies. So even if the algorithm isn't perfect, it shows that it's consistent.
Comments
Post a Comment