test_runner: expectFailure should support a reason (like todo/skip) and a specific expected failure

[vassudanagunta]

What is the problem this feature will solve?

The new expectFailure test option should:

1. be consistent with todo and skip test options, supporting a string value that is its reason.

   For example, an expectFailure might have "known issue #347" as its reason.

2. support a way to specify the specific expected failure. If a test failure is not the originally expected failure, then the test should not pass. The simplest and most flexible way to do this is probably by specifying a RegExp that must match the expected failure message.

   For example, the failure might be expected to match /Expected values to be strictly equal:\s+41 !== 42/

In addition, it should support both of the above independently. It should not be either-or.

What is the feature you are proposing to solve the problem?

One way to meet the above requirements is as follows:

test('one', {expectFailure: true}, (t) => ...) //consistent with todo/skip

test('two', {expectFailure: "known issue #347"}, (t) => ...) //reason string, consistent with todo/skip

test('three', {expectFailure: /<err msg pattern>/}, (t) => ...)

test('four', {expectFailure: {reason: "known issue #347", error:/<err msg pattern>/}}, (t) => ...)

Or perhaps an array to specify both:

test('five', {expectFailure: ["known issue #347", /<err msg pattern>/]}, (t) => ...)

Another way is for expectFailure to be exactly like todo and skip, taking only a boolean or string reason, and add a separate option for a specific expected failure:

test('six', {expectFailure: "known issue #347", expectFailureMessage:/<err msg pattern>/}, (t) => ...)

What alternatives have you considered?

No response