"Active Learning tool," a training data selection support tool that accelerates AI development - ushering in an era where data quality is paramount

Basic way of viewing the mapping

• Meaning of the dot:
  Each point on the graph corresponds to a single image file.
• Meaning of colors:
  The data is color-coded according to the type of number. When data with the same number is grouped together, clusters are formed.
• Meaning of distance:
  Points located close together on a 2D plane indicate that the AI has determined the data to be "similar in shape and characteristics."

The "thinking" of AI that can be gleaned from this diagram

• Cluster isolation
  Numbers like 0 and 1, which form distinct blocks separate from other colors, are easily recognizable data for AI because they have clear characteristics.
• Cluster overlap
  Areas where colors are mixed can be seen at the boundaries between 3 and 8, or 4 and 9. These are areas where the AI perceives the shapes as similar and difficult to distinguish. This is a weak point that can easily lead to misidentification.
• Isolated points
  Points that exist separately from larger clusters may be numbers with strong handwriting characteristics or special data containing noise.

Identifying uncertainty

In object detection, the more confused the model is with the data, the less stable the results tend to be when inference is repeated.

• Data in boundary regions where clusters are intermingled on a map tends to have higher uncertainty.
• By prioritizing the extraction of such data and using it for additional training, we can efficiently improve accuracy with a minimal amount of data.

Completing cluster gaps

When the distribution of trained data and the distribution of untrained data are overlaid, "gap zones" become visible between existing clusters.

By intensively training the model with data that fills in these gaps, we can reduce the model's blind spots and move closer to creating a robust AI that is strong in corner cases.

Visualization using t-SNE provides a map-like representation of how AI features are captured, which can often be a black box. Targeting areas of high uncertainty on this map during training is a shortcut to developing highly accurate models.

Calculation of analytical data (quantification of uncertainty)

The trained model is given multiple input data sets to perform inference, and feature data and uncertainty index data are calculated for each data set.

In this tool's object detection process, an uncertainty metric that focuses on the variability of inference results is used to understand the model's indecision.

In this context, uncertainty refers to how stably the model is able to make decisions regarding its input. For example, if you perform multiple inferences with Dropout enabled, the output will tend to be relatively stable for subjects that the trained model understands well. On the other hand, for patterns or corner cases where the model is not sufficiently trained, there may be variability in the output trends for each inference.

This tool utilizes such fluctuations to understand uncertainty. Data with large fluctuations in output may indicate that the model is not sufficiently confident, making it a good candidate for additional training.

Displaying data distribution through dimensionality reduction

By compressing high-dimensional feature data into 2D or 3D dimensions that are easier for humans to understand, tens of thousands of data points can be viewed as points on a map. Dimensionality reduction techniques such as t-SNE and UMAP can be used.

The following screen is an example of a development screen for the Active Learning tool. By clicking on each dot plotted on the 2D plane, you can view the actual image and detailed data. You can also refer to the uncertainty index Unc and the surrounding density Den.

Strategic FILTERS display

It supports highlighting specific labels and extracting images in order of highest uncertainty index, allowing for uses such as "collecting only images that AI struggles with and focusing on strengthening that area."

The following is an example of a development screen for the Active Learning tool. You can FILTERS by image category and highlight the target data.

↑This is what it looks like after FILTERS by the label "car". The distribution appears to be concentrated relatively towards the upper right.

↑This is what it looks like after FILTERS by "person". The distribution appears to be concentrated relatively towards the bottom left.

↑This shows the results after FILTERS by the label `car_person` (images containing both cars and people). The data appears to be distributed fairly evenly overall.

Integrated display with previously trained data

By displaying both previously trained and untrained data simultaneously, it's possible to identify untrained gaps between the trained areas. Prioritizing the reinforcement of these areas reduces the model's blind spots and efficiently improves recognition accuracy.

The following is an example of mapping using t-SNE with COCO data, highlighting each condition in red. This is also a screenshot of the Active Learning tool under development.

• All: All
• Car_Person: A mix of cars and people
• Person: Human only
• Car: Car only

In "All," you can get an overview of the overall distribution trend.
In the Car_Person dataset, points are present across the entire distribution, indicating that data containing both cars and people spans a wide feature region. Furthermore, in the upper left, we can see a distribution resembling a subcluster, slightly distanced from the larger dataset.
Persons tend to be concentrated in the upper right corner.
Cars tend to be concentrated in the lower left corner.

After confirming the distribution in this way, it becomes possible to, for example, focus on images around subclusters, add new data to those regions, or perform augmentation to improve versatility.

OTHERS features

• FILTERS function using uncertainty indicators
• Function to extract candidate training data for the next training period in order of uncertainty index
• Function to extract candidate training data for the next training session in mAP@0.5 / @0.9 order.
• Detailed preview of each data point, manual extraction function.

The following is an example of a development screen for the Active Learning tool.