Learn · Testing

Why one-moment click tests pass by luck

Most click tests move a knob once and check that no sample jumps too far. We deleted the Time glide from the kit’s feedback delay and ran that test at 238 different moments. The broken code failed at most of them and passed at 50. Whether the test caught the bug depended on when the knob moved.

The test

The delay plays a steady 202 Hz note at level 0.5, with the dry signal and the echo mixed together. A second in, Time jumps from 375 ms to 125 ms. With the 80 ms glide in place, the read position slides and the pitch bends like tape. With the glide deleted, the read position jumps 12,000 samples at once, and that’s a click.

The test measures the biggest jump between two neighbouring samples after the move. Under 0.2 passes. The note on its own steps up to 0.019 per sample.

Same bug, 238 moments

Line chart of the worst per-sample step against the moment Time was moved, across one cycle of a 202 Hz note. With the glide deleted the step swings between 0.019 and 0.614 and dips under the 0.2 pass line twice per cycle. With the 80 ms glide it stays between 0.065 and 0.067. Eight dots mark the moments the fixed test uses; the worst of them is 0.614.
Real C++ renders of the kit’s FeedbackDelay.h at 48 kHz, one render per moment: Time 375 → 125 ms at 1.000 s plus 0 to 237 samples (one cycle of the note). Glide deleted: 0.019 to 0.614. Glide on: 0.065 to 0.067 at every moment.

With the glide deleted, the jump measured anywhere from 0.019 to 0.61 per sample. Move Time at exactly 1.000 s and the test catches the bug. Move it 60 samples later, 1.25 ms, and the same broken code passes:

$ ./one_moment 0     # glide deleted
FAIL time 375 -> 125 ms at 1.00000 s: worst sample step 0.614
$ ./one_moment 60    # glide deleted
PASS time 375 -> 125 ms at 1.00125 s: worst sample step 0.019

Here’s why. An instant jump swaps the sample being read for one from 250 ms earlier, and the click is the difference between those two values. In this test the two read positions sit exactly half a cycle of the note apart, so the difference is largest at the peaks of the wave and drops to nothing where it crosses zero, twice a cycle. At those moments the read position has jumped 12,000 samples and the output barely moves.

The kit’s tremolo click test failed the same way. Its depth jump landed on the LFO peak, where gain is 1 at any depth, so it passed with the smoothing deleted. The testing guide tells that story.

The fix: move it at 8 moments, keep the worst

The fixed test runs the same jump at 8 moments, 128 samples apart, and judges the worst step. We chose 128 because a browser OfflineAudioContext can only pause on 128-sample block boundaries, so the C++ test and the browser test can use the same moments. They’re the dots on the chart. They land at different points of the pattern, and the worst of them sits on the peak.

for (int k = 0; k < 8; ++k) {
    const size_t at = 48000 + (size_t) k * 128;      // 8 moments, 128 samples apart
    FeedbackDelay fd; fd.setParameters (375, 0, 12000, 1); fd.prepare (48000.0);
    // ... render the 202 Hz note, switch Time to 125 ms at sample `at` ...
    worst = std::max (worst, maxStep (y, at - 1, at + 24000));
}
ok (fin && worst < 0.2f, m);

Real output, C++ and then the browser version in headless Chrome:

# as shipped (80 ms glide)
PASS time 375 -> 125 ms mid-note, 8 moments: worst sample step 0.067 (steady note 0.019), no click
# glide deleted
FAIL time 375 -> 125 ms mid-note, 8 moments: worst sample step 0.614 (steady note 0.019), CLICK: the jump is not smoothed

# browser, as shipped
PASS time 375 → 125 ms mid-note, 8 moments: worst sample step 0.067 (steady note 0.019), no click
# browser, glide deleted
FAIL time 375 → 125 ms mid-note, 8 moments: worst sample step 0.572 (steady note 0.019), CLICK: the jump is not smoothed

This 8-moment test and the slow-move check have been in the kit’s feedback-delay tests, in C++ and in the browser, since the Oct 16 update.

Use it on your own tests

  • Move the knob at several moments, spread across one cycle of the slowest thing in the test: the note, the LFO, the delay time.
  • Judge the worst result, not the first one.
  • Then break it on purpose. Delete the smoothing and check that the test goes red. If it stays green, it isn’t checking anything.
  • Know what a step test can’t see. A delay that reads whole samples and skips interpolation never makes a big step on a slow Time move. It makes a buzz instead. The updated delay test checks for that separately: the buzz above 4 kHz stays 98 dB below the note with interpolation and comes up to 58 dB without it.

How we measured: sweep.cpp renders the kit’s FeedbackDelay.h once per moment with c++ -std=c++17 -O2, as shipped and with one line changed (timeCoeff = 1.0f, the glide deleted). The browser numbers come from the same test in headless Chrome.

The feedback delay, its C++ port and its tests are in the $49 Vibepatch kit, along with the tremolo and chorus builds.