Kensa demo: a handbook bot with 3 planted bugs

A small handbook bot with three planted bugs: it answers vacation questions from memory (24 days) instead of the documents (30), it only reads the first 3 documents, and one run in three it adds a sentence the documents don't support. Every input ran 3 times. Card 5 shows the cut-off: shrinking can't remove the 3 documents in front of the parking one, because the bot only fails while that document is 4th. This page is Kensa's real output (python examples/toy/demo.py). Code on GitHub · Nancy Singh

FAIL · 6 violation(s)
3cases
26checks
237app calls
6violations
InvariantViolationsWhat it means
correct2The expected answer is in the documents, but the app didn't give it.
follows_update2The document was edited, and the app kept the old value.
refuses_when_missing1The document with the answer was removed, and the app still answered.
grounded1The answer says things the retrieved context doesn't support.

1. correct

expected '30' in the answer

QuestionHow many days of paid vacation do employees get per year?
Expected: 30
AnswerEmployees get 24 days of paid vacation per year.
Still fails with only these documents (1 of 5, after shrinking)
  1. Employees get 30 days of paid vacation per year, plus public holidays.

2. refuses_when_missingevidence removed

the evidence was removed, but the app answered instead of saying it doesn't know

OriginalQ: How many days of paid vacation do employees get per year?
A: Employees get 24 days of paid vacation per year.
After the changeQ: How many days of paid vacation do employees get per year?
A: Employees get 24 days of paid vacation per year.
Still fails with only these documents (2 of 5, after shrinking)
  1. The office is open from 9 am to 6 pm on weekdays.
  2. Employees get 30 days of paid vacation per year, plus public holidays.

3. follows_updatefact changed: 30 -> 63

the document now says '63'; the answer doesn't

OriginalQ: How many days of paid vacation do employees get per year?
A: Employees get 24 days of paid vacation per year.
After the changeQ: How many days of paid vacation do employees get per year?
A: Employees get 24 days of paid vacation per year.
Still fails with only these documents (1 of 5, after shrinking)
  1. Employees get 30 days of paid vacation per year, plus public holidays.

4. groundedflaky: fails 2/3 runs

1 sentence(s) not supported by the context, e.g. 'This policy was updated by the board in 2019.' (unsupported, support 0.00)

QuestionHow often are laptops replaced by the IT team?
Expected: 3
AnswerLaptops are replaced every 3 years by the IT team. This policy was updated by the board in 2019.
Still fails with only these documents (1 of 5, after shrinking)
  1. Laptops are replaced every 3 years by the IT team.

5. correct

refused, but the answer (120) is in the documents

QuestionHow many spaces does the parking garage have for staff?
Expected: 120
AnswerI don't know based on the handbook.
Still fails with only these documents (4 of 5, after shrinking)
  1. The office is open from 9 am to 6 pm on weekdays.
  2. Employees get 30 days of paid vacation per year, plus public holidays.
  3. Laptops are replaced every 3 years by the IT team.
  4. The parking garage has 120 spaces for staff.

6. follows_updatefact changed: 120 -> 243

the document now says '243', but the app refused

OriginalQ: How many spaces does the parking garage have for staff?
A: I don't know based on the handbook.
After the changeQ: How many spaces does the parking garage have for staff?
A: I don't know based on the handbook.
Still fails with only these documents (4 of 5, after shrinking)
  1. The office is open from 9 am to 6 pm on weekdays.
  2. Employees get 30 days of paid vacation per year, plus public holidays.
  3. Laptops are replaced every 3 years by the IT team.
  4. The parking garage has 120 spaces for staff.