Treasure hunt: object detection

Introduction

Self-driving cars need to spot people, bikes and traffic lights. Shop cameras count bottles on a shelf. They all use object detection: finding what is in a picture and where it is.

PictoBlox's object detection model already knows about 80 everyday things, from cups and books to bananas and teddy bears. In this lesson you'll build a treasure hunt: the computer asks for an object, and you race around the house to find it and show it to the camera.

Working principle

• Analyse image from (camera) runs the object detection model on the latest picture.
• Get # of objects says how many objects it found.
• (class) of object (1) reports what the first object is. The same block can report its x and y position, width, height and confidence.
• Is (cup) detected? is true if a cup is anywhere in the picture.
• Set detection threshold to (0.5) sets how sure the model must be (0 to 1) before it reports an object. A higher number means fewer mistakes but more misses.
• The model knows a fixed list of objects, including person, cup, bottle, book, cell phone, scissors, teddy bear, banana, apple, chair, clock and laptop. It can't find things that aren't on its list.

What you'll need

• PictoBlox with the Object Detection extension
• A webcam (a laptop you can carry around is easiest)
• A few things from the list: a cup, a book, a bottle, a spoon, a banana or an apple, a teddy bear

Working in real life

1. Add the Object Detection extension.
2. First test the model: build a tiny script that says (class) of object (1) in a loop, and hold up different things. Which ones does it know?
3. Make a list variable called treasures and add the items you have at home: cup, book, bottle, banana, teddy bear.
4. Build the treasure hunt script below. Make a variable called target and one called score.
5. Click the green flag and race!

Code

when green flag clicked
turn [on] video on stage with (0) % transparency
set detection threshold to (0.5)
set [score v] to (0)
reset timer
repeat until <(timer) > (60)>
  set [target v] to (item (pick random (1) to (length of [treasures v])) of [treasures v])
  say (join [Find me a ] (target))
  wait until <([class] of object (1)) = (target)>
  start sound [Collect]
  change [score v] by (1)
  say [Got it!] for (1) seconds
end
say (join [Time's up! You found ] (join (score) [ treasures]))

The game checks whether the first object the model found is the one it asked for, so hold the treasure close to the camera so it fills most of the picture. If the model keeps finding you (a person) first, hold the treasure up in front of your face. The wait until block needs fresh pictures, so add this helper script that keeps analysing the camera in the background:

when green flag clicked
forever
  analyse image from [camera]
end

Try it

• Change the threshold to 0.2 and then 0.9. Does the model make more mistakes, or miss more objects?
• Count objects: hold up two spoons. Can you make Tobi say how many spoons it sees with get number of detected (spoon)?
• Try tricking the model: show it a photo of a cup on a phone. Does it still say cup? Why?

Conclusion

• Object detection finds each object's class, position and confidence.
• The threshold decides how sure the model must be before it reports something.
• A ready-made model only knows the objects it was trained on.

What next?

The computer can find objects. Can it read? Next: Read it out loud: text recognition.