It’s easy to imagine a computer being a lot, lot smarter than us regular humans. They can do certain tasks much, much quicker than our brains can figure things out.

However, that’s not entirely true! Computers are better than us in some ways, yes. For instance, you can type in a complicated equation into a calculator and it’ll spit out an answer much faster than a human can do it.

But that’s the thing; a computer is much better than us at some things. They’re very good at churning through data or doing calculations, but when it needs to conceptualize or imagine things, it struggles to do what a human can do really easily.

Don’t believe us? Here’s an example. How complicated is it to understand this next sentence?

The mother gave the baby a toy, because they were crying.

It’s not too tricky, is it? You can imagine a baby bawling its eyes out, and their mother giving them a toy to calm it down. To a computer, though, deciphering this sentence is really, really hard.

Why? Because a computer doesn’t know what the ‘they’ in that sentence refers to. As far as a computer knows, it could be describing one of these two scenarios:

1. The baby was crying, so the mother gave them a toy.
2. The mother was crying, and because of that, she gave the baby a toy.

Number two sounds ridiculous to our human brains. Why would a mother give a baby a toy because she was crying? To a computer, though, it doesn’t see the problem. That’s because the computer doesn’t know what a ‘mother’ or a ‘baby’ is. It doesn’t know the relationship between a mother and a baby, and how a mother can give a baby a toy to calm it down when it cries.

This is what the Winograd Schema Challenge (WSC) aims to test. The WSC is a series of tests, much like the Turing test, that gives AI systems sentences like the above. The sentence contains a word like “they” or “it,” and the AI then has to identify what that word is referring to. For example, the ‘they’ in the above sentence is referring to the baby, and the AI needs to recognise that.

Here’s where things get tricky. Let’s look at this sentence: The lions ate the zebra, because they are predators.

We can understand this sentence easily, but a computer may assume that the second object, the zebras, are the predators in this sentence. Now we have a weird world where zebras are hunting down lions, and the lions presumably fought back to protect themselves!

Now, let’s assume the computer decides that when a sentence includes lions and zebras, any instances of ‘they’ always refers to the lions. Sure enough, if we kept feeding the computer sentences with the same structure as above, it’d always get right.

However, it’s not a good solution, especially when we pull this doozy: The lions ate the zebra, because they are tasty.

Now the computer assumes that the lions in the above sentence are the tasty ones. Perhaps the zebras were licking the lions a little too much and they decided to get revenge?

Then you have the really, really evil sentences. Take this one: The gardener stopped tending the plants because it started to rain.

Here, the ‘it’ refers to the weather. The clouds began to rain, so the gardener stopped and went inside. However, the sentence never mentions the weather directly, so a computer may assume that the gardener stopped because either they or the plants burst into water droplets!

The Winograd Schema Challenge may just seem like a funny and weird test, but it’s actually really important. These days, we have a lot of stuff we can command with our voices, like Amazon Alexa and Google Assistant. It’s important for these services to identify the “thems” “theys” and “its” in sentences like in the above examples so they can properly understand their human masters.

One way to do it is to ‘teach’ the computer what these objects are and how they work. You code a program that says mothers take care of children, that lions are predators and eat zebras which are prey, and that ‘started to rain’ refers to weather. But this is a really simple way of breaking it down, and programs undertaking the WSC are far more complex.

So if you’re into building AI, how would you go about solving the problems we detailed above? If you can’t solve them, don’t worry; even really smart adults with impressive degrees are struggling to solve them, too!

Learn More

The Winograd Schema

https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html

WinoGrande: An Adversarial Winograd Schema Challenge at Scale (with examples)

https://winogrande.allenai.org/

Winograd Schema Challenge

https://en.wikipedia.org/wiki/Winograd_schema_challenge

What are Winograd Schemas?

https://medium.datadriveninvestor.com/what-are-winograd-schemas-dfe9e51b765a