April 5, 2025

Knuth's Optimization¶

Knuth's optimization, also known as the Knuth-Yao Speedup, is a special case of dynamic programming on ranges, that can optimize the time complexity of solutions by a linear factor, from $O (n^{3})$ for standard range DP to $O (n^{2})$ .

Conditions¶

The Speedup is applied for transitions of the form

d p (i, j) = min_{i \leq k < j} [d p (i, k) + d p (k + 1, j) + C (i, j)] .

Similar to divide and conquer DP, let $o p t (i, j)$ be the value of $k$ that minimizes the expression in the transition ( $o p t$ is referred to as the "optimal splitting point" further in this article). The optimization requires that the following holds:

o p t (i, j - 1) \leq o p t (i, j) \leq o p t (i + 1, j) .

We can show that it is true when the cost function $C$ satisfies the following conditions for $a \leq b \leq c \leq d$ :

$C (b, c) \leq C (a, d)$ ;
$C (a, c) + C (b, d) \leq C (a, d) + C (b, c)$ (the quadrangle inequality [QI]).

This result is proved further below.

Algorithm¶

Let's process the dp states in such a way that we calculate $d p (i, j - 1)$ and $d p (i + 1, j)$ before $d p (i, j)$ , and in doing so we also calculate $o p t (i, j - 1)$ and $o p t (i + 1, j)$ . Then for calculating $o p t (i, j)$ , instead of testing values of $k$ from $i$ to $j - 1$ , we only need to test from $o p t (i, j - 1)$ to $o p t (i + 1, j)$ . To process $(i, j)$ pairs in this order it is sufficient to use nested for loops in which $i$ goes from the maximum value to the minimum one and $j$ goes from $i + 1$ to the maximum value.

Generic implementation¶

Though implementation varies, here's a fairly generic example. The structure of the code is almost identical to that of Range DP.

int solve() {
    int N;
    ... // read N and input
    int dp[N][N], opt[N][N];

    auto C = [&](int i, int j) {
        ... // Implement cost function C.
    };

    for (int i = 0; i < N; i++) {
        opt[i][i] = i;
        ... // Initialize dp[i][i] according to the problem
    }

    for (int i = N-2; i >= 0; i--) {
        for (int j = i+1; j < N; j++) {
            int mn = INT_MAX;
            int cost = C(i, j);
            for (int k = opt[i][j-1]; k <= min(j-1, opt[i+1][j]); k++) {
                if (mn >= dp[i][k] + dp[k+1][j] + cost) {
                    opt[i][j] = k; 
                    mn = dp[i][k] + dp[k+1][j] + cost; 
                }
            }
            dp[i][j] = mn; 
        }
    }

    return dp[0][N-1];
}

Complexity¶

A complexity of the algorithm can be estimated as the following sum:

\sum_{i = 1}^{N} \sum_{j = i + 1}^{N} [o p t (i + 1, j) - o p t (i, j - 1)] = \sum_{i = 1}^{N} \sum_{j = i}^{N - 1} [o p t (i + 1, j + 1) - o p t (i, j)] .

As you see, most of the terms in this expression cancel each other out, except for positive terms with $j = N - 1$ and negative terms with $i = 1$ . Thus, the whole sum can be estimated as

\sum_{k = 1}^{N} [o p t (k, N) - o p t (1, k)] = O (n^{2}),

rather than $O (n^{3})$ as it would be if we were using a regular range DP.

On practice¶

The most common application of Knuth's optimization is in Range DP, with the given transition. The only difficulty is in proving that the cost function satisfies the given conditions. The simplest case is when the cost function $C (i, j)$ is simply the sum of the elements of the subarray $S [i, i + 1, . . ., j]$ for some array (depending on the question). However, they can be more complicated at times.

Note that more than the conditions on the dp transition and the cost function, the key to this optimization is the inequality on the optimum splitting point. In some problems, such as the optimal binary search tree problem (which is, incidentally, the original problem for which this optimization was developed), the transitions and cost functions will be less obvious, however, one can still prove that $o p t (i, j - 1) \leq o p t (i, j) \leq o p t (i + 1, j)$ , and thus, use this optimization.

Proof of correctness¶

To prove the correctness of this algorithm in terms of $C (i, j)$ conditions, it suffices to prove that

o p t (i, j - 1) \leq o p t (i, j) \leq o p t (i + 1, j)

assuming the given conditions are satisfied.

Lemma

$d p (i, j)$ also satisfies the quadrangle inequality, given the conditions of the problem are satisfied.

Proof

The proof for this lemma uses strong induction. It has been taken from the paper Efficient Dynamic Programming Using Quadrangle Inequalities, authored by F. Frances Yao, which introduced the Knuth-Yao Speedup (this particular statement is Lemma 2.1 in the paper). The idea is to induct on the length $l = d - a$ . The case where $l = 1$ is trivial. For $l > 1$ consider 2 cases:

$b = c$
The inequality reduces to $d p (a, b) + d p (b, d) \leq d p (a, d)$ (This assumes that $d p (i, i) = 0$ for all $i$ , which is the case for all problems using this optimization). Let $o p t (a, d) = z$ .
- If $z < j$ ,
  Note that
  
  $d p (a, b) \leq d p_{z} (a, b) = d p (a, z) + d p (z + 1, b) + C (a, b) .$
  
  Therefore,
  
  $d p (a, b) + d p (b, d) \leq d p (a, z) + d p (z + 1, b) + d p (b, d) + C (a, b)$
  
  From the induction hypothesis, $d p (z + 1, b) + d p (b, d) \leq d p (z + 1, d)$ . Also, it is given that $C (a, b) \leq C (a, d)$ . Combining these 2 facts with above inequality yields the desired result.
- If $z \geq j$ , the proof of this case is symmetric to the previous case.
$b < c$
Let $o p t (b, c) = z$ and $o p t (a, d) = y$ .
- If $z \leq y$ ,
  
  $d p (a, c) + d p (b, d) \leq d p_{z} (a, c) + d p_{y} (b, d)$
  
  where
  
  $d p_{z} (a, c) + d p_{y} (b, d) = C (a, c) + C (b, d) + d p (a, z) + d p (z + 1, c) + d p (b, y) + d p (y + 1, d) .$
  
  Using the QI on $C$ and on the dp state for the indices $z + 1 \leq y + 1 \leq c \leq d$ (from the induction hypothesis) yields the desired result.
- If $z > y$ , the proof of this case is symmetric to the previous case.

This completes the proof of the lemma.

Now, consider the following setup. We have 2 indices $i \leq p \leq q < j$ . Set $d p_{k} = C (i, j) + d p (i, k) + d p (k + 1, j)$ .

Suppose we show that

d p_{p} (i, j - 1) \geq d p_{q} (i, j - 1) ⟹ d p_{p} (i, j) \geq d p_{q} (i, j) .

Setting $q = o p t (i, j - 1)$ , by definition, $d p_{p} (i, j - 1) \geq d p_{q} (i, j - 1)$ . Therefore, applying the inequality to all $i \leq p \leq q$ , we can infer that $o p t (i, j)$ is at least as much as $o p t (i, j - 1)$ , proving the first half of the inequality.

Now, using the QI on some indices $p + 1 \leq q + 1 \leq j - 1 \leq j$ , we get

\begin{aligned} d p (p + 1, j - 1) + d p (q + 1, j) \leq d p (q + 1, j - 1) + d p (p + 1, j) \\ ⟹ & (d p (i, p) + d p (p + 1, j - 1) + C (i, j - 1)) + (d p (i, q) + d p (q + 1, j) + C (i, j)) \\ \leq & (d p (i, q) + d p (q + 1, j - 1) + C (i, j - 1)) + (d p (i, p) + d p (p + 1, j) + C (i, j)) \\ ⟹ & d p_{p} (i, j - 1) + d p_{q} (i, j) \leq d p_{p} (i, j) + d p_{q} (i, j - 1) \\ ⟹ & d p_{p} (i, j - 1) - d p_{q} (i, j - 1) \leq d p_{p} (i, j) - d p_{q} (i, j) \end{aligned}

Finally,

\begin{aligned} d p_{p} (i, j - 1) \geq d p_{q} (i, j - 1) \\ ⟹ 0 \leq d p_{p} (i, j - 1) - d p_{q} (i, j - 1) \leq d p_{p} (i, j) - d p_{q} (i, j) \\ ⟹ d p_{p} (i, j) \geq d p_{q} (i, j) \end{aligned}

This proves the first part of the inequality, i.e., $o p t (i, j - 1) \leq o p t (i, j)$ . The second part $o p t (i, j) \leq o p t (i + 1, j)$ can be shown with the same idea, starting with the inequality $d p (i, p) + d p (i + 1, q) \leq d p (i + 1, p) + d p (i, q)$ .

This completes the proof.

Practice Problems¶

References¶

Contributors:

adamant-pwn (70.77%)
omkarbajaj073 (28.21%)
mhayter (0.51%)
FloppaInspector (0.51%)