FTC Programming

Object Detection

Scan the Skystone using Tensorflow and Vuforia

What is the SkyStone Object Detection Objective?

During autonomous, your robot needs to be able to detect the Skystones out of the row of stones. From there, you must bring each Skystone across your bridge to receive points.

How can I do that though?

There are many options to choose from for object detection, but we chose TensorFlow and Vuforia. FIRST gives code for this for your team to start off, and the first step to getting object detection to work is to understand all of the sample code.

Get the sample code

You can download the sample code here or directly from OnBot. In OnBot, press ADD, then choose the sample code that says: ConceptVuforiaObjectDetection, and then either Webcam or not Webcam. The Github also has this in the external.samples folder.

Get your Vuforia License

You also have to sign up at the Vuforia Developer Panel. This is where you can get a code to run all of the Vuforia sample programs.

FIRST: A Vuforia 'Development' license key, can be obtained free of charge from the Vuforia developer website at https://developer.vuforia.com/license-manager

Go into OnBot and create a new file with the template for Concept: TensorFlow Object Detection.

(Use Webcam if you are using a webcam)

Go a few lines down, and add the Vuforia Key you got from above.

private static final String VUFORIA_KEY = " -- YOUR NEW VUFORIA KEY GOES HERE  --- ";

Now, scroll down till you find the for loop that gives telemetry.

In Rover Ruckus, FIRST was kind enough to give the code to find the position of the gold mineral. Now, all we get are the left, top and right, bottom coordinates for each object the camera sees. As of V5.2 of SkyStone, there is a bug in the code. Just before the for loop ends, add I++ into it. This makes sure that the code keeps adding stones to the telemetry.

So without the positioning, there’s a few options:

  1. 1/3 Random Chance
  2. Using last year's code
  3. Use external tools/custom solution
  4. Use the information given and make a solution from that

We are going to be doing option 4. Using the information given and make a solution from that. We will not be giving the entire code, as we want you to figure it out as well, but we will get you started.

What we are going to do is focus on a set of 3 stones. Out of these three, one is a Skystone. If we know the position of this stone, we can determine the position of the other Skystone.

Scroll down in the code until you find the if statement and for loop that looks like this:

if (updatedRecognitions != null) {
  telemetry.addData("# Object Detected", updatedRecognitions.size());

  // step through the list of recognitions and display boundary info.
  int i = 0;
  for (Recognition recognition : updatedRecognitions) {
    telemetry.addData(String.format("label (%d)", i), recognition.getLabel());
    telemetry.addData(String.format("  left,top (%d)", i), "%.03f , %.03f",
                      recognition.getLeft(), recognition.getTop());
    telemetry.addData(String.format("  right,bottom (%d)", i), "%.03f , %.03f",
            recognition.getRight(), recognition.getBottom());
i++; // ADD THIS

There is a bug in the current code. Just before the for loop ends, add i++. This allows the number of objects to increase based on how many the camera sees. This tells you the (left, top) and (right, bottom) coordinates on a (x,y) plane of every object seen.

If you run the code, the camera places boxes around each object it sees, regardless of being a Skystone or regular stone. The way the code creates labels is using recognition.getLabel(). To differentiate the stones, it’s as simple as creating an if conditional checking that the recognition.getLabel() is equal to Skystone.

From here, you can do many things to identify which stone is the Skystone, whether that is based off the coordinates the telemetry gives you or another option.

Determining positioning is the final step. However, this is the hardest, as it depends on the situation you are in. How we approached it was simple trial & error. The camera output is a giant x-y grid. Each pixel has its own coordinate. However, every time you test your code, chances are that you will not have a consistent identification that is also accurate, unless you give your values a range to work in, as well as multiple variables the robot factors in. This is easily done by setting more parameters in the loop, so that the robot better understands what it should be looking for when trying to find the Skystone.

From there, you can add telemetry to show if a position has been found. This is one of the most effective and simpler ways to use object detection.