diff --git a/packages/trilean-combinatorics/README.md b/packages/trilean-combinatorics/README.md index 4c338fc..14fb1ce 100644 --- a/packages/trilean-combinatorics/README.md +++ b/packages/trilean-combinatorics/README.md @@ -56,7 +56,7 @@ const fastNeedsRatio = trileanConstraint( const model = defineModel({ dimensions, constraints: [fastNeedsRatio] }); // 24 valid combinations; every pair of values among them appears in at least one of these rows. -const rows = collect(model, { strategy: coveringArray({ strength: 2, limit: 10_000 }), limit: 10_000 }).map(({ values }) => values); +const rows = collect(model, { strategy: coveringArray({ strength: 2 }), limit: 10_000 }).map(({ values }) => values); isCoveringArray(rows, model, { strength: 2, limit: 10_000 }); // true ``` @@ -89,9 +89,9 @@ When it is defined, the tree is validated with trilean's `PredicateNodeSchema` a A definite true is `valid` and a definite false `invalid`. An indeterminate result is `unknown` while any dependency is unassigned. Once every dependency is assigned, combinatrix's contract forbids `unknown`, so an indeterminate result there throws `UndecidedConstraintError` (a `ConstraintContractError`) carrying trilean's reason. The usual cause is reading a dimension that is unset in some combination without guarding it: `compare(reference, ...)` is indeterminate `not-found` when the dimension is unset, while `and(exists(reference), compare(reference, ...))` is definitely false there, because a definite false absorbs. Comparing values of different kinds (a text dimension with `compare`, say) is indeterminate `wrong-type` and throws the same way. -### `coveringArray({ strength, limit })` +### `coveringArray({ strength, limit? })` -A combinatrix `Strategy` yielding a covering array of the solution set: valid combinations such that every t-way combination of dimension values that occurs in some valid combination occurs in at least one of them. `strength` is t, a positive integer: 1 covers each value, 2 is pairwise, 3 three-wise, and a strength at or above the dimension count yields the whole solution set. `limit` bounds the solution set it reads, throwing combinatrix's `LimitExceededError` past it, because the construction needs the whole solution set before it can choose a row. Either option out of range throws `RangeError` when the strategy is made. +A combinatrix `Strategy` yielding a covering array of the solution set: valid combinations such that every t-way combination of dimension values that occurs in some valid combination occurs in at least one of them. `strength` is t, a positive integer: 1 covers each value, 2 is pairwise, 3 three-wise, and a strength at or above the dimension count yields the whole solution set. `limit`, optional, is the most rows the construction may hold: every row is held until the array is complete, so past it the strategy throws combinatrix's `LimitExceededError` instead of building a larger array. Without it the array has at most one row per coverable t-tuple. Either option out of range throws `RangeError` when the strategy is made. Every dimension is covered: @@ -99,7 +99,11 @@ Every dimension is covered: - a derived dimension's values (first-valid and computed), since a derived value is part of every combination and can decide behaviour as much as a free one. Coverage is measured on the solution set, so a derived value is only ever required where it really occurs; - a dimension collapsed to unset by `activeWhen` is unset in those combinations, and unset counts as a value there. A combination does not record why a dimension is unset, so unset by choice and unset by inactivity are one value. -The construction is greedy and deterministic: it reads the solution set in enumeration order, then repeatedly yields the combination covering the most t-tuples not yet covered, ties going to the earliest, until every coverable tuple is covered. The same model always gives the same rows in the same order. Each row is yielded as it is chosen, so stopping early, or the `limit` on combinatrix's own `combinations`, ends the work there. An empty solution set yields nothing. +The construction is in-parameter-order ([IPOG](https://csrc.nist.gov/pubs/conference/2007/03/29/ipog-a-general-strategy-for-tway-software-testing/final), with constraints handled on partial rows as [IPOG-C](https://www.nist.gov/node/572651) does) and never reads the solution set. It starts one row per coverable t-tuple of the first t dimensions, then places each later dimension in turn: every row takes the value covering the most of that dimension's uncovered t-tuples, and each tuple still uncovered goes into the first row it fits, or into a new row whose other dimensions are left open. Each row is completed to a valid combination at the end, and a row whose every t-tuple another row also holds is dropped. Whether a t-tuple is coverable, and whether a row may take a value, are both asked of the engine as "is there a valid combination agreeing with this partial assignment" through the strategy context's `completions`, stopping at the first combination found, so the constraints' three-valued verdicts on partial assignments prune each question as they prune enumeration. + +So the construction holds at most one row per coverable t-tuple and one dimension's t-tuples at a time, which is polynomial in the dimension count for a fixed strength and independent of how large the solution set is, and its engine queries are polynomial in the t-tuple count. A computed dimension's possible values are found by one query per choice of the values of the dimensions it reads. The same model always gives the same rows in the same order. The whole array is built before the first row is yielded, since a later dimension can still change any row. An empty solution set yields nothing. + +The array is not always smaller than a greedy construction over the whole solution set (repeatedly taking the valid combination covering the most uncovered t-tuples), which is not size-optimal either: it is larger on some models and smaller on others, and the tests check it is no larger in total over the generated models. ### `isCoveringArray(rows, model, { strength, limit })` @@ -113,7 +117,7 @@ Whether `rows` are a covering array of the model's solution set at that strength The unit suite holds the adapter and the strategy to definitions rather than to examples alone. Kleene soundness is checked exhaustively on a small model: for every partial assignment, a verdict that is not `unknown` equals the verdict on every completion of it, for trees covering every connective, `exists`, `coalesce` and `conditional`, and the adapter agrees with trilean's own `evaluatePredicate` on every complete assignment. The evaluator's budgets are checked at their boundaries, a tree exactly at each limit evaluating and one beyond it being refused, which pins the definition-time count to what the evaluator spends. -The strategy is checked against `isCoveringArray` as an oracle over hundreds of seeded random models using every combinatrix feature (unset values, conditional activity, first-valid and computed dimensions, rules and guarded trilean constraints), at strengths 1, 2 and 3: every output is a covering array of valid, distinct combinations, deterministic, and no larger than the solution set. The oracle is itself shown to discriminate: it refuses the strategy's output with its last row removed, and a row outside the solution set. +The strategy is checked against `isCoveringArray` as an oracle over hundreds of seeded random models using every combinatrix feature (unset values, conditional activity, first-valid and computed dimensions, rules and guarded trilean constraints), at strengths 1, 2 and 3: every output is a covering array of valid, distinct combinations, deterministic, no larger than the solution set and with no more rows than coverable t-tuples. The same models compare it with the greedy construction over the whole solution set, kept in the test support as a reference, and its total over the models must not exceed the reference's. A work-count test runs it on a model whose solution set is far larger than its pair count, counting every query it makes of the engine and every combination the engine hands back, and holds both to a bound derived from the pair count, while the reference cannot even start within that many combinations. The oracle is itself shown to discriminate: it refuses the strategy's output with any one row removed, and a row outside the solution set. ## Licence diff --git a/packages/trilean-combinatorics/src/covering.ts b/packages/trilean-combinatorics/src/covering.ts index 23eff29..8f9991c 100644 --- a/packages/trilean-combinatorics/src/covering.ts +++ b/packages/trilean-combinatorics/src/covering.ts @@ -1,23 +1,34 @@ -import type { Strategy, StrategyContext, Value } from "combinatrix"; +import type { + Dimension, + PartialAssignment, + Strategy, + StrategyContext, + Value, +} from "combinatrix"; import { LimitExceededError } from "combinatrix"; +import { productOf, subsetsOf, tupleKey } from "./internal/tuples"; /** Options for `coveringArray`. */ export interface CoveringArrayOptions { /** The strength t, a positive integer: every t-way combination of values that occurs in the solution set is covered. 1 covers each value, 2 is pairwise, 3 three-wise. A strength above the model's dimension count covers every dimension at once, which is the whole solution set. */ readonly strength: number; - /** The largest solution set the strategy reads, a non-negative integer. The construction needs the whole solution set before it can choose a row, so past this many combinations it throws `LimitExceededError` instead. */ - readonly limit: number; + /** The most rows the construction may hold, a non-negative integer. Every row is held until the last dimension has been placed, so this bounds the construction's memory in rows; past it the strategy throws combinatrix's `LimitExceededError` instead of building a larger array. Without it the array is as large as the construction makes it, which is at most the number of coverable t-tuples. */ + readonly limit?: number; } /** - * A combinatrix strategy yielding a covering array of the model's solution set: a subset of the valid combinations such that every t-way combination of dimension values that occurs in at least one valid combination occurs in at least one yielded row. Coverage is defined against the solution set only, so a t-tuple no valid combination contains is not required, and an empty solution set yields nothing. + * A combinatrix strategy yielding a covering array of the model's solution set: valid combinations such that every t-way combination of dimension values that occurs in at least one valid combination occurs in at least one yielded row. Coverage is defined against the solution set only, so a t-tuple no valid combination contains is not required, and an empty solution set yields nothing. * * Every dimension of the model is covered, its values being whatever it takes in the solution set: * - A free dimension's values, unset included when its domain lists `undefined`. * - A derived dimension's values (first-valid and computed) too. A derived value is part of every combination and can decide behaviour as much as a free one, and because coverage is measured on the solution set, a derived value is required only where it really occurs, so including derived dimensions never asks for a combination the model cannot produce. * - A dimension collapsed to unset by `activeWhen` is unset in those combinations, and unset counts as a value like any other there. The combination does not record why a dimension is unset, so unset by choice and unset by inactivity are one value, and covering it with the other dimensions' values is covering the dimension's absence alongside them. * - * The construction is greedy and deterministic: it reads the solution set in enumeration order, then repeatedly yields the combination covering the most t-tuples not yet covered, breaking ties by enumeration order, until none is left. Rows are therefore always yielded in the same order for the same model. It yields each row as it is chosen, so a consumer that stops early, or combinatrix's own `limit` on `combinations`, ends the work there. Throws `RangeError` for a strength that is not a positive integer or a limit that is not a non-negative integer, when the strategy is made. + * The construction is in-parameter-order, IPOG with its constraint handling done by the engine: Lei, Kacker, Kuhn, Okun and Lawrence, "IPOG: A General Strategy for T-Way Software Testing", ECBS 2007 (https://csrc.nist.gov/pubs/conference/2007/03/29/ipog-a-general-strategy-for-tway-software-testing/final), with constraints as in Yu, Lei, Nourozborazjany, Kacker and Kuhn, "An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation", ICST 2013 (https://www.nist.gov/node/572651). Rows are partial assignments over the dimensions in declaration order. The first t dimensions start one row per coverable t-tuple of theirs. Each later dimension is then placed in two phases: horizontal growth gives every row the value covering the most of that dimension's uncovered t-tuples, and vertical growth puts each tuple still uncovered into the first row it fits, or into a new row whose other dimensions are left open. Each row is then completed to a valid combination by the engine, and last, visiting the rows from the last, a row whose every t-tuple some other row still kept also holds is dropped, which removes repeats and rows later placements made redundant without uncovering anything. + * + * It never reads the solution set. Which t-tuples are coverable, and whether a row may take a value, are both asked of the engine as "is there a valid combination agreeing with this partial assignment", the first combination `completions` yields for it, so the constraints' verdicts on partial assignments prune the search exactly as they do for enumeration, and the strategy stops at the first combination found. Every row is kept able to complete, so every yielded row is valid. Its memory is therefore the rows, the current dimension's tuples and, at the end, one count per distinct t-tuple of the rows: at most one row per coverable t-tuple, since every row is started for a tuple no earlier row holds, each over the model's dimensions, and at most the coverable t-tuples themselves. That is polynomial in the dimension count for a fixed strength and independent of the size of the solution set. The engine queries are polynomial in the t-tuple count too, each closing its search at the first combination: one per candidate tuple, at most one per row and value in horizontal growth and one more for a row none of them fits, at most one per row for each tuple vertical growth places, and one per row to complete it, besides a computed dimension's. A computed dimension's possible values are found from the values of the dimensions it reads, by one engine query per choice of those values, so they are its value set and never a list of combinations. + * + * The output is deterministic: the same model gives the same rows in the same order, ties going to the earlier value and the earlier row. The strategy is lazy in that nothing is computed until the first row is asked for, but the whole array is built before the first row is yielded, since a later dimension may still change any row and the redundancy pass needs every row. The array is not always smaller than the greedy construction over the whole solution set, which is not size-optimal either: on some models it is larger, on others smaller. Throws `RangeError` for a strength that is not a positive integer or a limit that is not a non-negative integer, when the strategy is made. */ export function coveringArray(options: CoveringArrayOptions): Strategy { const { strength, limit } = options; @@ -26,7 +37,7 @@ export function coveringArray(options: CoveringArrayOptions): Strategy { `strength must be a positive integer, not ${String(strength)}`, ); } - if (!Number.isSafeInteger(limit) || limit < 0) { + if (limit !== undefined && (!Number.isSafeInteger(limit) || limit < 0)) { throw new RangeError( `limit must be a non-negative integer, not ${String(limit)}`, ); @@ -34,80 +45,340 @@ export function coveringArray(options: CoveringArrayOptions): Strategy { return { name: `covering array of strength ${String(strength)}`, - select: (context) => greedyRows(context, strength, limit), + select: (context) => inParameterOrderRows(context, strength, limit), }; } -/** A combination of the solution set with the keys of every t-tuple it contains. */ -interface Candidate { - readonly row: Readonly>; - readonly tuples: readonly string[]; -} +/** A row under construction: the dimensions placed in it so far, an absent dimension being open and one present with `undefined` being unset. */ +type Row = Map; -/** The greedy construction `coveringArray` describes. */ -function* greedyRows( +/** The construction `coveringArray` describes. */ +function* inParameterOrderRows( context: StrategyContext, strength: number, - limit: number, + limit: number | undefined, ): Generator>> { - const names = context.dimensions.map((dimension) => dimension.name); - const subsets = subsetsOf(names, Math.min(strength, names.length)); - // A Set iterates in insertion order, so the candidates stay in enumeration order as chosen ones are removed. - const unchosen = new Set(); - for (const row of context.completions()) { - if (unchosen.size === limit) throw new LimitExceededError(limit); - - unchosen.add({ - row, - tuples: subsets.map((subset) => tupleKey(subset, row)), - }); + const { dimensions } = context; + const width = Math.min(strength, dimensions.length); + const names = dimensions.map((dimension) => dimension.name); + const values: (readonly Value[])[] = []; + const rows: Row[] = []; + const addRow = (row: Row): void => { + if (rows.length === limit) throw new LimitExceededError(limit); + + rows.push(row); + }; + + for (let index = 0; index < width; index += 1) { + values.push(possibleValues(context, index, values)); + } + const firstNames = names.slice(0, width); + for (const choice of productOf(values)) { + const row: Row = new Map( + firstNames.map((name, index) => [name, choice[index]]), + ); + if (completes(context, row)) addRow(row); } - const uncovered = new Set( - [...unchosen].flatMap((candidate) => candidate.tuples), + + for (let index = width; index < dimensions.length; index += 1) { + values.push(possibleValues(context, index, values)); + const placement: Placement = { context, names, values, index, width }; + const uncovered = coverableTuples(placement); + growHorizontally(placement, rows, uncovered); + growVertically(context, rows, uncovered, addRow); + } + + yield* withoutRedundantRows( + rows.map((row) => { + const completed = firstCompletion(context, row); + if (completed === undefined) { + throw new Error( + "unreachable: a row is only ever given values the engine found a valid combination for", + ); + } + + return completed; + }), + subsetsOf(names, width), ); - while (uncovered.size > 0) { - let best: Candidate | undefined; - let bestGain = 0; - // Only a strictly larger gain replaces the best, so a tie goes to the earliest candidate in enumeration order. - for (const candidate of unchosen) { - const gain = candidate.tuples.filter((tuple) => - uncovered.has(tuple), - ).length; - if (gain > bestGain) { - best = candidate; - bestGain = gain; +} + +/** The complete rows less every row whose t-tuples all occur in rows kept, visiting the rows from the last, so a row that repeats an earlier one, or that later rows made redundant, is dropped and the rest keep their order. Coverage is unchanged, since a row goes only when every tuple it holds is held by another row still kept. Holds one count per distinct t-tuple of the rows. */ +function* withoutRedundantRows( + rows: readonly Readonly>[], + subsets: readonly (readonly string[])[], +): Generator>> { + const tuplesOf = (row: Readonly>): string[] => + subsets.map((subset) => tupleKey(subset.map((name) => [name, row[name]]))); + const holders = new Map(); + for (const row of rows) { + for (const tuple of tuplesOf(row)) { + holders.set(tuple, (holders.get(tuple) ?? 0) + 1); + } + } + const kept = rows.map(() => true); + for (let index = rows.length - 1; index >= 0; index -= 1) { + const tuples = tuplesOf(rowAt(rows, index)); + if (tuples.some((tuple) => holders.get(tuple) === 1)) continue; + + kept[index] = false; + for (const tuple of tuples) { + holders.set(tuple, (holders.get(tuple) ?? 0) - 1); + } + } + for (const [index, row] of rows.entries()) { + if (kept[index] === true) yield row; + } +} + +function rowAt(rows: readonly T[], index: number): T { + const row = rows[index]; + if (row === undefined) { + throw new Error( + `unreachable: position ${String(index)} is within the ${String(rows.length)} rows`, + ); + } + + return row; +} + +/** + * Every value the dimension at `index` can take in the solution set, possibly with some it cannot, in a fixed order: a free dimension's domain, a first-valid dimension's candidates, or a computed dimension's results on every choice of the values of the dimensions it reads that some valid combination agrees with; unset is added when `activeWhen` can collapse the dimension. Values the solution set never has are harmless, since a tuple holding one is never coverable. `known` holds the possible values of every dimension before `index`. + */ +function possibleValues( + context: StrategyContext, + index: number, + known: readonly (readonly Value[])[], +): Value[] { + const dimension = dimensionAt(context.dimensions, index); + const found = new Map(); + const add = (value: Value): void => { + found.set(tupleKey([[dimension.name, value]]), value); + }; + if (dimension.kind === "choice") dimension.values.forEach(add); + if (dimension.kind === "first-valid") dimension.candidates.forEach(add); + if (dimension.kind === "computed") { + const readLists = dimension.reads.map((read) => { + const list = + known[context.dimensions.findIndex(({ name }) => name === read)]; + if (list === undefined) { + throw new Error( + `unreachable: computed dimension "${dimension.name}" reads "${read}", which combinatrix requires to be declared before it`, + ); + } + + return list; + }); + for (const choice of productOf(readLists)) { + const reads = new Map( + dimension.reads.map((read, position) => [read, choice[position]]), + ); + // Only a choice of the read values some valid combination has is computed, so `compute` is called only on inputs the model can produce. + if (completes(context, reads)) { + add(dimension.compute(Object.fromEntries(reads))); + } + } + } + if (dimension.activeWhen !== undefined) add(undefined); + + return [...found.values()]; +} + +/** What placing one dimension after the first t reads: the engine, every dimension's name, the possible values of this dimension and every one before it, this dimension's position, and the tuple width t. */ +interface Placement { + readonly context: StrategyContext; + readonly names: readonly string[]; + readonly values: readonly (readonly Value[])[]; + readonly index: number; + readonly width: number; +} + +/** Every t-tuple holding the dimension at `index` and `width - 1` dimensions before it that some valid combination contains, keyed by `tupleKey` in dimension order, in the order of the earlier dimensions' subsets and then of their values. */ +function coverableTuples(placement: Placement): Map { + const { context, names, values, index, width } = placement; + const coverable = new Map(); + const earlier = Array.from({ length: index }, (_, position) => position); + for (const subset of subsetsOf(earlier, width - 1)) { + const positions = [...subset, index]; + const lists = positions.map((position) => valuesAt(values, position)); + for (const choice of productOf(lists)) { + const pairs = positions.map((position, place): [string, Value] => [ + nameAt(names, position), + choice[place], + ]); + const tuple: Row = new Map(pairs); + if (completes(context, tuple)) coverable.set(tupleKey(pairs), tuple); + } + } + + return coverable; +} + +/** Gives every row a value for the dimension at `index`: of the values that leave the row able to complete, the one covering the most uncovered tuples, the earlier value on a tie. Removes the tuples it covers from `uncovered`. Every row can complete before, so some value always can after. */ +function growHorizontally( + placement: Placement, + rows: readonly Row[], + uncovered: Map, +): void { + const { context, names, index, width } = placement; + const name = nameAt(names, index); + // The values the uncovered tuples hold for this dimension, once each, in the tuples' order. + const distinct = new Map(); + for (const tuple of uncovered.values()) { + const value = tuple.get(name); + const key = tupleKey([[name, value]]); + if (!distinct.has(key)) distinct.set(key, value); + } + const values = [...distinct.values()]; + for (const row of rows) { + const placed = names + .slice(0, index) + .filter((earlierName) => row.has(earlierName)); + const subsets = subsetsOf(placed, width - 1); + let best: { value: Value; covered: string[] } | undefined; + for (const value of candidatesFor(context, row, name, values)) { + const covered = subsets + .map((subset) => + tupleKey([ + ...subset.map((earlierName): [string, Value] => [ + earlierName, + row.get(earlierName), + ]), + [name, value], + ]), + ) + .filter((key) => uncovered.has(key)); + // Only a strictly larger gain replaces the best, so a tie goes to the earlier value. + if (best === undefined || covered.length > best.covered.length) { + best = { value, covered }; } } if (best === undefined) { throw new Error( - "unreachable: every uncovered tuple came from a candidate not yet chosen, since a chosen candidate's tuples are all covered", + `unreachable: a row that can complete can take some value of "${name}"`, ); } - for (const tuple of best.tuples) uncovered.delete(tuple); - unchosen.delete(best); - yield best.row; + row.set(name, best.value); + for (const key of best.covered) uncovered.delete(key); } } -/** Every way to choose `size` of the items, each keeping the items' order, listed lexicographically by position. Choosing none gives the one empty subset; choosing more than there are gives none. */ -function subsetsOf(items: readonly T[], size: number): T[][] { - if (size === 0) return [[]]; +/** The values a row may take for a dimension while staying able to complete, lazily: the values of the dimension's coverable tuples first, in their order, and only if none of them fits, the value of the row's own first completion, which always fits. */ +function* candidatesFor( + context: StrategyContext, + row: Row, + name: string, + coveringValues: readonly Value[], +): Generator { + let any = false; + for (const value of coveringValues) { + if (completes(context, new Map([...row, [name, value]]))) { + any = true; + yield value; + } + } + if (any) return; - return items.flatMap((item, index) => - subsetsOf(items.slice(index + 1), size - 1).map((rest) => [item, ...rest]), - ); + const completed = firstCompletion(context, row); + if (completed !== undefined) yield completed[name]; } -/** The canonical text of a row's values on the given dimensions: `[name]` for unset and `[name, value]` for a set value, so `null` never collides with unset and two tuples are equal exactly when their texts are. Equal values have equal JSON, so `0` and `-0` are one value, as combinatrix treats them. */ -function tupleKey( - subset: readonly string[], - row: Readonly>, -): string { - return JSON.stringify( - subset.map((name) => { - const value = row[name]; +/** Places every tuple still uncovered after horizontal growth: into the first row agreeing with it wherever both are placed that can still complete with it, or else into a new row of its own. A tuple an earlier placement already covered is skipped. */ +function growVertically( + context: StrategyContext, + rows: readonly Row[], + uncovered: ReadonlyMap, + addRow: (row: Row) => void, +): void { + for (const tuple of uncovered.values()) { + if (rows.some((row) => covers(row, tuple))) continue; - return value === undefined ? [name] : [name, value]; - }), + const host = rows.find((row) => { + if (!agrees(row, tuple)) return false; + + return completes(context, new Map([...row, ...tuple])); + }); + if (host === undefined) { + addRow(new Map(tuple)); + continue; + } + for (const [name, value] of tuple) host.set(name, value); + } +} + +/** Whether the row has every value of the tuple. */ +function covers(row: Row, tuple: Row): boolean { + return [...tuple].every( + ([name, value]) => row.has(name) && sameValue(row.get(name), value), ); } + +/** Whether the row and the tuple have the same value on every dimension both place. */ +function agrees(row: Row, tuple: Row): boolean { + return [...tuple].every( + ([name, value]) => !row.has(name) || sameValue(row.get(name), value), + ); +} + +function sameValue(left: Value, right: Value): boolean { + return tupleKey([["", left]]) === tupleKey([["", right]]); +} + +/** Whether some valid combination agrees with the assignment, asked of the engine, which stops at the first one. */ +function completes( + context: StrategyContext, + assignment: PartialAssignment, +): boolean { + return firstCompletion(context, assignment) !== undefined; +} + +/** The first valid combination, in enumeration order, agreeing with the assignment, or `undefined` when there is none. Only the first is searched for: the search is closed once it is found. */ +function firstCompletion( + context: StrategyContext, + assignment: PartialAssignment, +): Readonly> | undefined { + const search = context.completions(assignment); + const first = search.next(); + search.return(undefined); + + return first.done === true ? undefined : first.value; +} + +function dimensionAt( + dimensions: readonly Dimension[], + index: number, +): Dimension { + const dimension = dimensions[index]; + if (dimension === undefined) { + throw new Error( + `unreachable: position ${String(index)} is within the ${String(dimensions.length)} dimensions`, + ); + } + + return dimension; +} + +function nameAt(names: readonly string[], index: number): string { + const name = names[index]; + if (name === undefined) { + throw new Error( + `unreachable: position ${String(index)} is within the ${String(names.length)} names`, + ); + } + + return name; +} + +function valuesAt( + values: readonly (readonly Value[])[], + index: number, +): readonly Value[] { + const list = values[index]; + if (list === undefined) { + throw new Error( + `unreachable: the possible values of position ${String(index)} are found before any later position's`, + ); + } + + return list; +} diff --git a/packages/trilean-combinatorics/src/covering.unit.test.ts b/packages/trilean-combinatorics/src/covering.unit.test.ts index bf5ccda..8c4d9fe 100644 --- a/packages/trilean-combinatorics/src/covering.unit.test.ts +++ b/packages/trilean-combinatorics/src/covering.unit.test.ts @@ -10,6 +10,8 @@ import { } from "combinatrix"; import { describe, expect, it } from "vitest"; import { coveringArray, isCoveringArray } from "./index"; +import { subsetsOf, tupleKey } from "./internal/tuples"; +import { greedyCoveringArray } from "./test-support/greedy-reference"; import { randomModel } from "./test-support/random-model"; /** How many generated models each property is checked against. */ @@ -130,32 +132,125 @@ describe("coveringArray", () => { ); }); - it("refuses a solution set larger than its limit", () => { + it("refuses to hold more rows than its limit, and needs no limit", () => { + // Every pair of two unconstrained dimensions is a whole combination, so the array is the product and needs that many rows. const model = defineModel({ dimensions: [choice("a", SIZES), choice("b", SIZES)], }); + const rows = SIZES.length ** 2; expect(() => - rowsOf( - model, - coveringArray({ strength: 2, limit: SIZES.length ** 2 - 1 }), - ), + rowsOf(model, coveringArray({ strength: 2, limit: rows - 1 })), ).toThrow(LimitExceededError); expect( - rowsOf(model, coveringArray({ strength: 2, limit: SIZES.length ** 2 })), - ).toHaveLength(SIZES.length ** 2); + rowsOf(model, coveringArray({ strength: 2, limit: rows })), + ).toHaveLength(rows); + expect(rowsOf(model, coveringArray({ strength: 2 }))).toHaveLength(rows); }); it("refuses a strength that is not a positive integer and a limit that is not a non-negative integer", () => { - expect(() => coveringArray({ strength: 0, limit: LIMIT })).toThrow( - RangeError, - ); - expect(() => coveringArray({ strength: 1.5, limit: LIMIT })).toThrow( + expect(() => coveringArray({ strength: 0 })).toThrow(RangeError); + expect(() => coveringArray({ strength: 1.5 })).toThrow(RangeError); + expect(() => coveringArray({ strength: 2, limit: -1 })).toThrow(RangeError); + expect(() => coveringArray({ strength: 2, limit: 1.5 })).toThrow( RangeError, ); - expect(() => coveringArray({ strength: 2, limit: -1 })).toThrow(RangeError); + }); + + it("never reads the solution set: the combinations it asks the engine for are bounded by the t-tuple count", () => { + // Unconstrained, so the solution set is the full product of the domains while the pairs grow only with the square of the dimension count. + const dimensionCount = 14; + const domain = [0, 1, 2]; + const strength = 2; + const model = defineModel({ + dimensions: Array.from({ length: dimensionCount }, (_, index) => + choice(`d${String(index)}`, domain), + ), + }); + const solutions = count(model, { limit: Number.MAX_SAFE_INTEGER }); + const tupleCount = + subsetCount(dimensionCount, strength) * domain.length ** strength; + // The construction's engine queries, from its structure: one per candidate tuple when the tuples are listed (the first t dimensions' included), and per later dimension at most one per row and value plus one fallback in horizontal growth and one per row and uncovered tuple in vertical growth, plus one completion per row. Every row holds a tuple no earlier row did, so there are at most tupleCount rows. + const queryBound = + tupleCount + + dimensionCount * tupleCount * (domain.length + 1) + + tupleCount * tupleCount + + tupleCount; + expect(queryBound).toBeLessThan(solutions); + + const work = { queries: 0, combinations: 0 }; + const counted: Strategy = { + name: "counted", + select: (context) => + coveringArray({ strength }).select({ + ...context, + completions: function* (assignment) { + work.queries += 1; + for (const combination of context.completions(assignment)) { + work.combinations += 1; + yield combination; + } + }, + }), + }; + const rows = rowsOf(model, counted); + expect(rows.length).toBeLessThanOrEqual(tupleCount); + expect(work.queries).toBeLessThanOrEqual(queryBound); + // Each query stops at its first combination, so no more combinations pass through than queries are made. + expect(work.combinations).toBeLessThanOrEqual(work.queries); + // The greedy reference, by contrast, holds the whole solution set before its first row, so even as many combinations as there are tuples is too few for it. + expect(() => + rowsOf(model, greedyCoveringArray({ strength, limit: tupleCount })), + ).toThrow(LimitExceededError); }); }); +/** The number of ways to choose `size` of `total` items. */ +function subsetCount(total: number, size: number): number { + let result = 1; + for (let chosen = 0; chosen < size; chosen += 1) { + result = (result * (total - chosen)) / (chosen + 1); + } + + return result; +} + +describe("coveringArray against the greedy reference", () => { + it.each(STRENGTHS)( + "is no larger in total than the greedy reference over the generated models at strength %i, and never more rows than coverable tuples", + (strength) => { + let rowsTotal = 0; + let referenceTotal = 0; + for (const { model, seed } of generated()) { + const rows = rowsOf(model, coveringArray({ strength })); + const reference = rowsOf( + model, + greedyCoveringArray({ strength, limit: LIMIT }), + ); + rowsTotal += rows.length; + referenceTotal += reference.length; + expect(rows.length, `seed ${String(seed)}`).toBeLessThanOrEqual( + coverableTupleCount(model, strength), + ); + } + expect(rowsTotal).toBeLessThanOrEqual(referenceTotal); + }, + ); +}); + +/** The number of distinct t-tuples the model's solution set contains, read from the whole solution set. */ +function coverableTupleCount(model: Model, strength: number): number { + const names = model.dimensions.map((dimension) => dimension.name); + const subsets = subsetsOf(names, Math.min(strength, names.length)); + const tuples = new Set(); + for (const { values } of collect(model, { limit: LIMIT })) { + for (const subset of subsets) { + tuples.add(tupleKey(subset.map((name) => [name, values[name]]))); + } + } + + return tuples.size; +} + describe("coveringArray against the isCoveringArray oracle", () => { it.each(STRENGTHS)( "is a covering array of strength %i on every generated model, of valid, distinct rows", @@ -190,21 +285,24 @@ describe("coveringArray against the isCoveringArray oracle", () => { } }); - it("the oracle refuses the strategy's output with its last row dropped", () => { + it("the oracle refuses the strategy's output with any one row dropped", () => { let checked = 0; - for (const { model, seed } of generated()) { + for (const { model, seed } of generated().slice( + 0, + MODELS / SAMPLE_DIVISOR, + )) { const rows = rowsOf(model, coveringArray({ strength: 2, limit: LIMIT })); - if (rows.length === 0) continue; - - checked += 1; - // The last row chosen covered at least one tuple no earlier row did, so without it that tuple is uncovered. - expect( - isCoveringArray(rows.slice(0, -1), model, { - strength: 2, - limit: LIMIT, - }), - `seed ${String(seed)}`, - ).toBe(false); + // Redundant rows are removed, so every row holds a tuple no other row does, and without it that tuple is uncovered. + for (const [index] of rows.entries()) { + checked += 1; + expect( + isCoveringArray(rows.toSpliced(index, 1), model, { + strength: 2, + limit: LIMIT, + }), + `seed ${String(seed)}, row ${String(index)}`, + ).toBe(false); + } } expect(checked).toBeGreaterThan(0); }); diff --git a/packages/trilean-combinatorics/src/internal/tuples.ts b/packages/trilean-combinatorics/src/internal/tuples.ts new file mode 100644 index 0000000..f3e240a --- /dev/null +++ b/packages/trilean-combinatorics/src/internal/tuples.ts @@ -0,0 +1,34 @@ +import type { Value } from "combinatrix"; + +/** Every way to choose `size` of the items, each keeping the items' order, listed lexicographically by position. Choosing none gives the one empty subset; choosing more than there are gives none. */ +export function subsetsOf(items: readonly T[], size: number): T[][] { + if (size === 0) return [[]]; + + return items.flatMap((item, index) => + subsetsOf(items.slice(index + 1), size - 1).map((rest) => [item, ...rest]), + ); +} + +/** Every choice of one value from each list, lazily and lexicographically by position, the last list varying fastest. No lists give the one empty choice; an empty list gives none. */ +export function* productOf( + lists: readonly (readonly Value[])[], +): Generator { + const [first, ...rest] = lists; + if (first === undefined) { + yield []; + + return; + } + for (const value of first) { + for (const tail of productOf(rest)) yield [value, ...tail]; + } +} + +/** The canonical text of dimension values given as name and value pairs: `[name]` for unset and `[name, value]` for a set value, so `null` never collides with unset. Two lists of pairs naming the same dimensions in the same order have equal texts exactly when their values are equal, and equal values have equal JSON, so `0` and `-0` are one value, as combinatrix treats them. */ +export function tupleKey(pairs: readonly (readonly [string, Value])[]): string { + return JSON.stringify( + pairs.map(([name, value]) => + value === undefined ? [name] : [name, value], + ), + ); +} diff --git a/packages/trilean-combinatorics/src/readme-example.unit.test.ts b/packages/trilean-combinatorics/src/readme-example.unit.test.ts index f833875..8b04ce2 100644 --- a/packages/trilean-combinatorics/src/readme-example.unit.test.ts +++ b/packages/trilean-combinatorics/src/readme-example.unit.test.ts @@ -49,7 +49,7 @@ describe("the README example", () => { expect(count(model, { limit })).toBe(VALID_COMBINATIONS); const rows = collect(model, { - strategy: coveringArray({ strength: 2, limit }), + strategy: coveringArray({ strength: 2 }), limit, }).map(({ values }) => values); expect(isCoveringArray(rows, model, { strength: 2, limit })).toBe(true); diff --git a/packages/trilean-combinatorics/src/test-support/greedy-reference.ts b/packages/trilean-combinatorics/src/test-support/greedy-reference.ts new file mode 100644 index 0000000..11d7365 --- /dev/null +++ b/packages/trilean-combinatorics/src/test-support/greedy-reference.ts @@ -0,0 +1,70 @@ +import type { Strategy, StrategyContext, Value } from "combinatrix"; +import { LimitExceededError } from "combinatrix"; +import { subsetsOf, tupleKey } from "../internal/tuples"; + +/** + * The reference covering-array construction `coveringArray` is compared with in tests: greedy over the whole solution set, so its size is a yardstick and its memory is the solution set. It reads every valid combination in enumeration order, then repeatedly yields the combination covering the most t-tuples not yet covered, ties going to the earliest, until none is left. Past `limit` combinations it throws `LimitExceededError`. Coverage and the strength reading are those of `coveringArray`. + */ +export function greedyCoveringArray(options: { + readonly strength: number; + readonly limit: number; +}): Strategy { + const { strength, limit } = options; + + return { + name: `greedy covering array of strength ${String(strength)}`, + select: (context) => greedyRows(context, strength, limit), + }; +} + +/** A combination of the solution set with the keys of every t-tuple it contains. */ +interface Candidate { + readonly row: Readonly>; + readonly tuples: readonly string[]; +} + +function* greedyRows( + context: StrategyContext, + strength: number, + limit: number, +): Generator>> { + const names = context.dimensions.map((dimension) => dimension.name); + const subsets = subsetsOf(names, Math.min(strength, names.length)); + // A Set iterates in insertion order, so the candidates stay in enumeration order as chosen ones are removed. + const unchosen = new Set(); + for (const row of context.completions()) { + if (unchosen.size === limit) throw new LimitExceededError(limit); + + unchosen.add({ + row, + tuples: subsets.map((subset) => + tupleKey(subset.map((name) => [name, row[name]])), + ), + }); + } + const uncovered = new Set( + [...unchosen].flatMap((candidate) => candidate.tuples), + ); + while (uncovered.size > 0) { + let best: Candidate | undefined; + let bestGain = 0; + // Only a strictly larger gain replaces the best, so a tie goes to the earliest candidate in enumeration order. + for (const candidate of unchosen) { + const gain = candidate.tuples.filter((tuple) => + uncovered.has(tuple), + ).length; + if (gain > bestGain) { + best = candidate; + bestGain = gain; + } + } + if (best === undefined) { + throw new Error( + "unreachable: every uncovered tuple came from a candidate not yet chosen, since a chosen candidate's tuples are all covered", + ); + } + for (const tuple of best.tuples) uncovered.delete(tuple); + unchosen.delete(best); + yield best.row; + } +}