﻿TEACHER NOTES — PHASE 1: BOSS PICTURES (60 min)

GOAL
Students meet an AI that looks perfect on its training data and still fails on realistic variation. They find out WHY (detective work), fix it with more varied data (Round 2) and see what stays hard (Nightmare and Trap pictures).

LESSON FLOW (about 60 min)
  0-5    Mission (video chapter 1-2). Students predict their score.
  5-20   Build classes, upload 1_TRAIN, train (video chapter 3).
  20-30  Boss test (12 pictures), results sheet.
  30-38  Nightmare: human vs AI.   38-42 Traps.
  42-50  Detective: look for what the failed pictures share. Do NOT tell them the answer — let them find it.
  50-60  Round 2 with the Fix Pack, compare. Optional: fourth class "Other".
  Then Phase 2 (Excel): the same numbers become tables and charts.

WHAT MAKES THESE PICTURES HARD (do not tell students until the detective step is done)
  1. Hidden shortcut: in 1_TRAIN each fruit mostly sits on its own table (apples on wood 82%, bananas on white tiles 82%, oranges on check cloth 82%). The AI can learn "table = fruit".
  2. Boss test pictures were chosen where Teachable-Machine-style models struggle: fruit on the OTHER fruit's table, dim light, small fruit, fruit cut by the edge, varieties that are rare in training (yellow/red-yellow apple, green or brown banana, orange cut in half).
  3. Nightmare: blur, noise, very dark, tiny, cropped, pencil sketch, and 3 pictures with more than one fruit (the correct answer is MIXED: the AI cannot say that).
  4. Traps: tomatoes, lemons, a peach, a pumpkin, a ball, sweet corn, a pepper. The AI has no "none" class and answers anyway, usually very confidently.
  The Fix Pack shows every fruit on every table, in dim light, small and cut — it removes the shortcut. OTHER_PACK teaches a fourth class.

WHAT TO EXPECT (our own measurements)
  We trained models like Teachable Machine (a MobileNet image model with a small trained head) on three different MobileNet versions, 3 random seeds each, and also ran the real Teachable Machine website.
  Pictures like the training set: 100%. Boss test: Round 1 62% (range 53%-69% across model versions), Round 2 95% (range 92%-100%). Nightmare (single-fruit pictures): Round 1 33%, Round 2 49%. Trap pictures: the models were more than 80% sure about 84% of them in Round 1 (all of those answers are wrong). Real Teachable Machine (2 runs): Round 1 Boss test 7 of 12, Round 2 10 of 12. Nightmare (single-fruit pictures): Round 1 1 of 9, Round 2 4 and 3 of 9. Trap pictures with more than 80% confidence: Round 1 11 and 10 of 12, Round 2 12 of 12. With the fourth class Other: 11 of 12 traps were called Other and the Boss test scored 9 of 12.
  Real Teachable Machine runs differ from run to run and from our numbers. Treat these as a guide: Round 1 on the Boss test is usually clearly below the training-like accuracy, Round 2 is clearly better, Nightmare stays hard, Traps are nearly always answered confidently.

ANSWER KEY
  TEACHER_KEY.csv lists every test picture, its true answer and why it is hard (tags). Nightmare 'MIXED' = more than one fruit: students get a point for saying Mixed; the AI cannot score on these. Trap pictures have no correct class among Apple/Banana/Orange.
  Tags: bg_swap = on another fruit's table · dim = dark · small · crop = cut by the edge · unseen_variety = rare variety · half = orange cut in half · bunch = several bananas · clutter = busy background · tiny/blur/noise/dark/sketch/crop/tint = nightmare effects.

DISCUSSION
  - Why did the AI look perfect in training? (it was tested on pictures like the ones it learned from)
  - What is a shortcut? Where else can it happen? (a medical AI that learns the hospital logo)
  - Is 90% confidence a promise? (no)
  - What would you add to the training data? Is it ever enough?

NOTES AND HONESTY
  The pictures are computer-drawn, not photographs. The point is the behaviour of the AI, not photo realism. Teachable Machine needs internet; images stay in the browser unless students choose Save to Drive.
