[Lagrang3]

Addresses: #9156
Supersedes: #9539

This PR adds a new constraint into askrene: maxhops, to limit the length of the routes.
Hard constraints of path length is not possible under the current implementation, minimum cost flows
do not allow that. Our approach is to add a constant cost per hop (a similar tweak in renepay #7540 was already deployed)
and scale that cost until the computed routes fit the maxhops constraints.
Fewer hops are expected with a base cost per hops and a getroutes failure if no routing solutions
are found satisfying maxhops constraint.

On xpay we compute the onion budget for each payment as the maximum number of hops we can afford to put inside
an onion message given the hard limit of 1300 bytes for the onion package, payment amount, blockheight and tail
(ie. final payload in the case of bolt11 and blinded path plus final payload in the case of bolt12).
The onion budget is passed down to getroutes so that solutions will not exceed it.
No more "Could not create payment onion: path too long!" errors are expected afterwards, instead we might
get an error from getroutes if no solutions can be found with the maxhops limit.

TODO:

• unit test onion_budget,
• implement the cost based maxhops constraint into askrene,
• integration test for maxhops in askrene,
• integration test for xpay with heavy final payloads bolt11,
• integration test for xpay with heavy final payloads bolt12,
• promote maxhops from a developer option to a user option and add documentation for maxhops,

[Lagrang3]

It was tricky to add a new feature cost (to have maxhops). Some changes were needed in combine_cost_function()
to make the approach extensible to multiple features. 93f6d34

[ksedgwic]

Overall the per-hop penalty and maxhops do what a rebalancer wants: shorter routes and more requests settled in full.

I looked at what happens with the xrebalance plugin using this algorithm. I ran the askrene commits at 67679bd offline against a mainnet gossip snapshot (about 30,000 channels with real capacities), sending 120 getroutes requests of the kind xrebalance makes when rebalancing (multi-part, 50M to 2.4B msat, 300 and 600 ppm fee budgets), and the same 120 against master. Three results:

• askrene's child process aborts on about a fifth of the requests (24 of 120): an assertion in combine_cost_function().
• The fee weighting starts ten times higher than before. That changes fees and success probability on its own, and the changelog does not mention it.
• maxhops works as described, and a limit of 20 costs nothing.

The abort. combine_cost_function() asserts that the four weights (fees, delay, hop count, probability) sum to exactly 1.0, fabs(total_dial - 1.0) < 1e-30, but renormalize_dials() cannot guarantee that in floating point. With the starting weights (0.1, 0.01, 0.01, 0.88) and the fee weight multiplied by the golden ratio on each over-budget retry, the sum is exact after the first three retries and off by 2.2e-16 after the fourth. Any request still over budget after four retries then kills the child with SIGABRT, and the caller gets code -1 "child died with signal 6" instead of a route or PAY_ROUTE_NOT_FOUND. On master the same requests retried up to ten times and ended with a route or 205. A tolerance of about 1e-9, or renormalizing inside combine_cost_function() instead of asserting, fixes it. With that one change all 120 requests complete.

The fee weight. On master the first solve uses mu = 1 of MU_MAX = 100, so fees weigh 1% against probability 99%. linear_routes() now starts mu at 0.1 with probability at 0.88. To tell this apart from the per-hop change I ran the 120 requests three ways with the assertion fixed: master, the series as is, and the series with mu = 0.01 (probability 0.97). The per-hop factor does what the changelog says at either starting point: routes average 5.5 hops instead of 7.1, the longest is 12 or 13 instead of 19, and 99 requests find routes for the full amount instead of 88. The starting mu is a separate effect. At 0.1 the routes are cheaper, 270 ppm of the delivered amount against 321 on master, but less likely to succeed, median probability_ppm 972,997 against 994,735, and the solver finishes in half the time because it needs fewer retries. At 0.01 the fee rate and probability come back near master's (333 ppm, 983,433) and the shorter routes stay. If the new starting point is intended, a changelog line would help callers that tune maxfee around the old behavior. If not, 0.01 reproduces the old weighting.

maxhops. With limits of 20, 12 and 8 the results were identical or nearly identical to no limit: no route was longer than 13 hops, 12 changed one request's routes and 8 changed seven, with no cost in delivered amount or fee rate. An unreachable limit fails with PAY_ROUTE_NOT_FOUND and "excessive hop count", which a caller that already treats 205 as "no route" handles without change.

[ksedgwic]

Both changes since my last comment work; one trade-off for small payments is worth a decision.

I re-ran the test from my earlier comment on this head (c7dcc9a): the askrene code run offline against a copy of mainnet gossip, with 120 getroutes requests of the kind a rebalancer makes, and the same 120 against master. Three results:

• The abort is gone. All 120 requests complete; before, 24 of them killed the child process.
• Fees are back at master's level, and the routes are shorter: 6 hops on average instead of 7, and 98 requests get routes for the full amount instead of 88. Success probability is about the same. The solver takes about twice as long on requests that go over the fee budget.
• maxhops=20 still costs nothing.

The trade-off, which is what the test_real_data failure in CI is showing: for small payments with a loose fee limit, the new code takes the shortest route even when a longer one is much cheaper. In the test, the first attempt pays 2.4 times master's fees. On mainnet gossip with my requests the effect is smaller, 9% overall, but at the smallest amounts it goes both ways by up to 2x in either direction. A caller with a tight maxfee gets master's fee level back through the retries; a caller with a loose maxfee pays the first-attempt price. If that is the intended behavior, the test expectations just need updating.

Details.

The 120 requests: multi-part, 50M to 2.4B msat, fee budgets of 300 and 600 ppm, master 2220786 against c7dcc9a.

	master	c7dcc9a
requests completed	120	120 (96 at 67679bd)
routed for the full amount	88	98
fee rate, ppm of delivered	321	334
median probability_ppm	994,735	990,624
mean / longest hops	7.1 / 19	6.0 / 15
solver time, 120 requests	45 s	88 s

Solver time: the fee dial now starts at 0.01 and climbs by the golden ratio, about ten steps to reach 1 when a request stays over budget; median per request 0.30 s against 0.42 s.

Loose fee limit, first attempt only, 60 requests (5,000 ppm and unlimited give identical results): both versions find routes for 37 of them. On those 37 the fee rate is 694 ppm on master and 755 on c7dcc9a, a median of 5% more per request, 20 costlier and 15 cheaper. At 50M msat the series returns one 3-hop route where master returns four to nine routes of 7 to 11 hops: 5 to 7x cheaper for two destinations, about 2x costlier for four. test_real_biases moves slightly the same way.

maxhops=20: identical routes and fees to no limit on all 120 requests, 89 s against 88 s.

Also: I read the xpay commits; the onion budget arithmetic and the extra hop for the blinded tail check out. The listconfigs example needs regenerating for dev-invoice-blinded-padding. The valgrind test_splice_rbf_htlc_sigs failure is on master too (First Integration Tests 6/6 at 2220786), not this PR.

[ksedgwic]

Approving d7d51fa. I re-ran the 120 rebalancer-style requests from my earlier comments against this head, same gossip snapshot and method.

The doubling step gives the same routes at the same price as the golden-ratio step, faster: 99 of 120 requests settle in full (98 at c7dcc9a, 88 on master), 336 ppm of delivered amount (334, 321), 6.0 hops on average with the longest 14 (6.0 / 15, 7.1 / 19), and solver time 69 s for the 120 against 88 s before and 45 s on master. 23 requests changed paths or fee; 4 settled at a higher rung than before, none lower.

The dial-race guard never fired in 257 getroutes calls. From the starting dials the fee dial reaches 1 in about 7 doublings and the delay and hop dials in about 10, so the guard sits at the edge of the range; every route-not-found on this sample was the ordinary "could not find a usable set of paths".

maxhops=20 is still free: identical routes and fees on all 120, 69.1 s against 69.2 s.

	master	c7dcc9a	d7d51fa
settled in full	88	98	99
fee rate, ppm of delivered	321	334	336
median probability_ppm	994,735	990,624	990,624
mean / longest hops	7.1 / 19	6.0 / 15	6.0 / 14
solver time, 120 requests	45 s	88 s	69 s

[vincenzopalazzo]

Is it possible to backport this? @daywalker90 @Lagrang3

thanks

[Lagrang3]

@vincenzopalazzo, yes.
#9656