P(init)
← You (0.55)
0
1
Mastery (0.95) →

Now, let's explore BKT more deeply to find out more about its limitations and behavior in different situations. Complete all four scenarios below to proceed.

0 / 4 scenarios complete

How do Incorrect Answers Impact Mastery?
Begin
What Causes Unexpected Model Behavior?
Begin
When Do You Lose Mastery?
Begin
What is the Role of Speed in BKT?
Begin
Next

How do Incorrect Answers Impact Mastery?

Answer the multiple-choice question below. Pay attention to what happens to your P(init) after you submit your answer.

0
P(init): 0.25
1
What is the word being signed?

Hint: There may not be a correct answer.

Sorry, that is incorrect. The correct answer is: dog.

Did you notice that your P(init) increased, even though you got the answer wrong? We promise there really wasn't a correct answer option. This is another interesting characteristic of BKT, but it isn't necessarily a flaw.

Why might it make sense for P(init) to increase regardless of the correctness of your answer? Tell me!

Well, BKT considers every answer, wrong or right, as a learning opportunity that brings you one step closer to mastery. As you might have experienced in real life, reviewing incorrect answers often helps to solidify your knowledge even more and reduce the chance that you'll make the same mistake again.

Do you think answering incorrectly should always increase P(init). Why or why not?

Take a moment to think about this question and when you're ready, click the button below to return to the main scenario menu.

← Go Back

What Causes Unexpected Behavior?

Some parameter values cause BKT to act in ways that don't make a lot of sense. Follow the steps below and keep an eye on how the mastery bars change—in each case, does P(init) increase more for correct or incorrect answers?

Correct Answer:

0
P(init): 0.40
1

Incorrect Answer:

0
P(init): 0.40
1

First, set P(guess) and P(slip) to a value < 0.5. When you're ready, press the simulation button below.

P(slip) / Letter Similarity: 0.5

0

1

P(guess) / Letter Familiarity: 0.5

0

1

Which caused a greater increase in mastery?

Yes, correct answers increase mastery more in this case.

Continue →

Hopefully, you saw that when P(guess) and/or P(slip) was > 0.5, P(init) increased more if you answered incorrectly rather than correctly (and sometimes P(init) might even decrease with correct answers!).

Weird, right? This is an example of model degeneracy, which is when the model does not behave as expected due to inappropriate parameter values.

Let's take a step back and think about why it doesn’t make sense for P(guess) and P(slip) to be too large.

For P(slip), if the probability of slipping is > 0.5, this suggests that a student who knows the skill is more likely to get the question wrong than right, which doesn't make any sense.

Can you apply similar logic to P(guess)? Click to fill in the blanks!

If the probability of guessing is > 0.5, this suggests that a student who doesn't know the skill is more likely to get the question

than

.

You got it! This is why we often set a maximum threshold for P(slip) and P(guess). To be extra conservative, P(slip) is typically bounded at 0.1 and P(guess) is typically bounded at 0.3. This helps us avoid any unexpected behavior (i.e., model degeneracy) from BKT.

When you're ready to continue, press the button below to return to the main scenario menu.

← Go Back

When Do You Lose Mastery?

Read the scenario below and then answer the questions.

Kris is a student who has been learning ASL for 3 months. Today, she tested her knowledge of fingerspelling with BKT and received a P(init) of 0.8. If Kris doesn't review or practice fingerspelling after today, what do you predict her P(init) to be in:

Drag the sliders!

5 days: 0.8

0

1

2 months: 0.8

0

1

1 year: 0.8

0

1

In general, when do you think P(init) starts decreasing? Enter your answer below.

after you stop reviewing the material.

You guessed:

Alright, now that you’ve submitted your guesses, it's time to reveal the answer! BKT will start decreasing your P(init)...

Tell me!

NEVER!

Did we get you? Sorry, that was a trick question :). But don't feel too bad if you got the answer wrong. You were probably thinking about how Kris (and people in general) tend to forget things over time. And you're totally right! But BKT doesn't know this, which is why forgetting is one of the algorithm's key limitations.

Not being able to handle forgetting may impact the reliability of BKT's predictions in certain situations. For example, in a more general educational context, how might summer vacation lead to problematic estimates of P(init) by BKT?

Take a moment to think about this question and when you're ready, click the button below to return to the main scenario menu.

← Go Back

What is the Role of Speed in BKT?

Read the scenario below and then answer the questions.

Mac and Cheese are two students being tested on their ASL knowledge. If both students get the exact same score, but Mac takes 10 minutes to finish the test while Cheese takes 3 hours, who will have the higher P(init) according to BKT?

Well, it turns out that the answer is neither! BKT doesn't have a parameter for time, so the algorithm doesn't care how fast or slow you submit your answer (even if it took you a year to press submit!). All it pays attention to is whether you get it correct or not, so Mac and Cheese would actually have the same P(init) in this case.

However, as you may be thinking, this behavior of BKT isn't the most intuitive and doesn't really reflect how things work in real life. Think about it. The time it takes you to complete something (aka. your speed) is often an indicator of level of mastery. So in real life, would Mac or Cheese have the higher P(init)?

Fill in the blanks below!

According to BKT, Mac's P(init) is

Cheese's P(init). But in real life, Mac's P(init) should likely be

Cheese's P(init).

Great work!

Why is this true? Well, if a student completes a test faster, that may suggest higher recall and fluency (and thus a higher P(init)) with the material than a student who gets the same score but takes longer to finish the same test. In other words, correctness is important, but it's not the only thing that matters when it comes to assessing learning and mastery, and thus, BKT's predictions may not always be reliable.

When you're ready to continue, press the button below to return to the main scenario menu.

← Go Back