Can I disable individual tests in the ledger-api-test-tool?

The ledger-api-test-tool has an --exclude option which allows me to disable a whole set of tests, e.g., --exclude=ContractKeysIT. However, I would like to disable tests at a finer granularity, e.g., when running the ledger-api-test-tool from SDK 1.0.0 against the current sandbox from HEAD, only the following two tests in ContractKeysIT fail:

ContractKeys
- Divulged contracts cannot be fetched or looked up by key by non-stakeholders ... Assertion failed at line 61
  Error message did not contain [InvalidLookup], but was [Some(Disputed: Interpretation error: Error: User abort: lookup matches)].
- Contract Keys should reject fetching an undisclosed contract ... Assertion failed at line 104
  Error message did not contain [InvalidLookup], but was [Some(Disputed: Interpretation error: Error: User abort: lookup matches)].

All the other contract keys tests are fine so I would like to still run them.

Unfortunately this is currently not possible. As you pointed out correctly, only entire test suites can be excluded from the test run.

@stefanobaghino-da: all individual tests already have a short identifier (e.g. CKFetchOrLookup). What do think about something like optionally printing this identifier (to make it more discoverable than looking at the source code) and adding a CLI flag or expanding the --exclude parameter to accept values of the form of ContractKeysIT:CKFetchOrLookup?

The idea makes sense, it’s been asked a few times already but unfortunately we never had time to work on it. It could make sense to drop the “namespacing” initials and assign party names and command identifiers using the suite and test name. It would be nice for both to be rendered in the same name (which means that probably : cannot be used to separate the two).

Looks like : is a valid character for party identifiers:

digital-asset/daml/blob/49c6a20f0a949574e9173d8a8f6aebbfb78a5cf5/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/IdString.scala#L237-L243

/** Party identifiers are non-empty US-ASCII strings built from letters, digits, space, colon, minus and,
  * underscore limited to 255 chars. We use them to represent [Party] literals. In this way, we avoid
  * empty identifiers, escaping problems, and other similar pitfalls.
  */
override type Party = String
override val Party: ConcatenableStringModule[Party, HexString] =
  new ConcatenableMatchingStringModule(":-_ ".contains(_), 255)

Great, thanks!

I have just merged a change to allow this. As of current master, and therefore probably 1.3, the arguments passed to --include and --exclude will be treated as prefixes of the test names, where the test names are constructed by appending a test identifier to the suite name. This means that the change is backwards compatible: an existing option that used to match a test suite will still match all the tests in that suite.

Test result reports have been amended to display the full name for each test, and there is a new --list-all flag to list all test cases (whereas --list will still list only test suites, as it used to).