The fraction of ground truth footprints correctly identified (purple), identified at too low of an IoU score (orange), or missed completely (red) at three different IoU thresholds and stratified by look angle. How would changing the IoU threshold have influenced each competitor’s recall scores? Let’s check three possible thresholds: 0.25, 0.5, and 0.75: As a result, the two had nearly identical recall scores in the very off-nadir imagery. However, almost 30% of XD_XD’s very off-nadir building predictions were not good enough to satisfy the IoU threshold of 0.5, whereas that was only true for 9% of selim_sef’s very off-nadir predictions. By contrast, selim_sef’s algorithm only found a part of about 57% of the buildings. XD_XD’s algorithm correctly identified part of a lot of buildings in the very off-nadir subset - about 70% of the actual buildings in the imagery. At an IoU threshold of 0.5, their recall is more or less identical. selim_sef achieves relatively similar recall values at IoU thresholds ≤0.5, whereas XD_XD’s performance drops precipitously in this range. See the graph below for a schematic of what fraction of the buildings each algorithm identifies correctly as we change this threshold:īuilding footprint recall is plotted against the IoU threshold for selim_sef and XD_XD, stratified by look angle: nadir (≤25 degrees, blue), off-nadir (26–40 degrees, orange), and very off-nadir (>40 degrees, green). We explored how changing this IoU threshold to be larger or smaller than 0.5 would have affected algorithms’ scores and got some intriguing results. It’s important to consider this threshold when evaluating computer vision algorithms for product deployment: how precisely must objects be labeled for the use case? For our competitions we use the threshold of 0.5 to strike a balance between these two extremes. By contrast, a high IoU threshold demands that predictions closely trace the actual contours of a building to be deemed correct. This is closer to object detection methods, which only produce rectangular bounding boxes that mark where target objects exist. A low IoU threshold means that you don’t care how much of a building is labeled correctly, only that some part of it is identified. ![]() The IoU threshold says a lot about what you value in an algorithm’s performance. For more on the metric, see this explainer post. The overlapping area between the manually labeled ground truth (blue) and the predicted building (red) is divided by the combined area covered by both together. We used an intersection-over-union (IoU) threshold of 0.5 to classify predictions as “good enough” to be called correct:Ī schematic representation of the Intersection over Union metric. Competitors were asked to generate polygons that traced the boundary of each building. This challenge’s task is to identify building footprints. Read on for the answers to these questions! How accurately did prize-winning algorithms label each building? How did trees blocking buildings influence building detection?.Did building size influence the likelihood that a building would be identified?.How similar were the predictions from the different algorithms?. ![]()
0 Comments
Leave a Reply. |