Nyrkio/change-detection GitHub Action is released!

Today we are proud to announce the release of the Nyrkio/change-detection GitHub action. Despite the “v2” label, this is our first release of this new product, after months of intense development and testing together with our partners at Turso Database. This GitHub action will make it a lot easier to start using Nyrkiö Change Detection together with you Continuous Benchmarking workflows. Just add a few lines of YAML and that’s all.

Literally, this is all you need to do:

- name: Analyze benchmark results with Nyrkiö
            uses: nyrkio/change-detection@v2
            with:
              tool: 'go'   # 'jmh', 'time', etc...
              output-file-path: output.txt
              # Pick up your token https://nyrkio.com/docs/getting-started
              nyrkio-token: ${{ secrets.NYRKIO_JWT_TOKEN }}

The old way of sending hand crafted json files of course is also available and remains useful to those users not yet using GitHub workflows for CI.

What does it do?

The Nyrkio GitHub Action understands the output of all the major benchmarking frameworks:

cargo bench for Rust projects
go test -bench for Go projects
benchmark.js for JavaScript/TypeScript projects
pytest-benchmark for Python projects with pytest
Google Benchmark Framework for C++ projects
Catch2 for C++ projects
BenchmarkTools.jl for Julia packages
Benchmark.Net for .Net projects
benchmarkluau for Luau projects
JMH for Java projects
time Unix utility to measure execution time of any program
Custom benchmarks where either ‘biggerIsBetter’ or ‘smallerIsBetter’
Nyrkios own JSON format, which is simply passed directly to Nyrkiö API.

Nyrkio/change-detection understands the output of all of the above benchmark frameworks, and converts the benchmark results into a common internal representation. This is then sent to the Nyrkio API for the actual change point detection analysis.

More integrations are already underway (e.g. pgbench) and are easy to add.

Based on 6 years of open source collaboration

Nyrkio/change-detection is based on the popular github-action-benchmark action, by “Linda pp”, Chris Trześniewski and other contributors. Thus the code base is not new rather we are building on a mature and well tested foundation. And it goes without saying Nyrkio is committed to upstreaming new integrations as we add more.

To underscore what Nyrkiö brings to the table here: The original github-action-benchmark uses threshold based alerting and is by default set to alert if a benchmark result is 150% slower than previous results. Nyrkiö Change Detection replaces this with our state of the art change detection algorithm, which is able to detect regressions and improvements as small as 1-2%, even in the presence of random noise higher than that!

Try it!

If you are already running benchmarks in your github workflows, try Getting Started with Nyrkiö in 3 easy steps.

And if you aren’t yet running automated benchmarks… Maybe you should? We can help you.