Apache NuttX RTOS: Notify via Mastodon the results of Rewind Builds

Read the article...

• "Auto-Rewind for Daily Test (Apache NuttX RTOS)"

If the Daily Test fails for Apache NuttX RTOS... Can we Auto-Rewind and discover the Breaking Commit? Let's try this...

1. Every Day at 00:00 UTC: Ubuntu Cron shall trigger a Daily Build and Test of NuttX for QEMU RISC-V (knsh64 / 64-bit Kernel Build)

2. If The Test Fails: Our Machine will Backtrack The Commits, rebuilding and retesting each commit (on QEMU Emulator)

3. When it discovers the Breaking Commit: Our Machine shall post a Mastodon Alert, that includes the (suspicious) Pull Request

4. Bonus: The Machine will draft a Polite Note for our NuttX Colleague to investigate the Pull Request, please

   

Why are we doing this?

If NuttX Fails on QEMU RISC-V: High chance that NuttX will also fail on RISC-V SBCs like Ox64 BL808 and Oz64 SG2000.

Thus it's important to Nip the Bud and Fix the Bug early, before it hurts our RISC-V Devs. (Be Kind, Rewind!)

Find the Breaking Commit

We wrote a script that will Rewind the NuttX Build and discover the Breaking Commit...

## Set the GitLab Token, check that it's OK
## export GITLAB_TOKEN=...
. $HOME/gitlab-token.sh
glab auth status

## Set the GitLab User and Repo for posting GitLab Snippets
export GITLAB_USER=lupyuen
export GITLAB_REPO=nuttx-build-log

## Download the NuttX Rewind Script
git clone https://github.com/lupyuen/nuttx-build-farm
cd nuttx-build-farm

## Find the Breaking Commit for QEMU RISC-V (64-bit Kernel Build)
nuttx_hash=  ## Optional: Begin with this NuttX Hash
apps_hash=   ## Optional: Begin with this Apps Hash
./rewind-build.sh \
  rv-virt:knsh64_test \
  $nuttx_hash \
  $apps_hash

Our Rewind Script runs 20 Iterations of Build + Test...

## Build and Test: Latest NuttX Commit
git reset --hard HEAD
tools/configure.sh rv-virt:knsh64
make -j
qemu-system-riscv64 -kernel nuttx

## Build and Test: Previous NuttX Commit
git reset --hard HEAD~1
tools/configure.sh rv-virt:knsh64
make -j
qemu-system-riscv64 -kernel nuttx
...
## Build and Test: 20th NuttX Commit
git reset --hard HEAD~19
tools/configure.sh rv-virt:knsh64
make -j
qemu-system-riscv64 -kernel nuttx

## Roughly One Hour for 20 Rewinds of Build + Test

Then we run this script to Post the Breaking Commit from Prometheus to Mastodon: run.sh

## Set the GitLab Token, User and Repo for posting GitLab Snippets
## export GITLAB_TOKEN=...
## export GITLAB_USER=lupyuen
## export GITLAB_REPO=nuttx-build-log
. $HOME/gitlab-token.sh

## Set the Mastodon Token
## export MASTODON_TOKEN=...
. $HOME/mastodon-token.sh

## Set the Rust Environment
. $HOME/.cargo/env

## Set the Prometheus Server
export PROMETHEUS_SERVER=luppys-mac-mini.local:9090

## Post the Breaking Commit from Prometheus to Mastodon
cargo run

Build and Test 20 times! Won't it look mighty messy?

Ah that's why we present neatly the 20 Outcomes (Build + Test) as the NuttX Build History, inside our NuttX Dashboard...

Be Kind, Rewind!

1. Wow this looks super complicated. Does it work?

   Dunno, we're still testing? Hopefully the New System will make my Daily Routine a little less painful...

   • Every Morning: I check the NuttX Daily Test

   • Oops Daily Test failed! I run a script to Rewind or Bisect the Daily Build

   • I write a Polite Note (depending on my mood)

   • And post it to the Breaking Pull Request

   That's why we're Fast Tracking the complicated new system: Right now it runs Every Hour (instead of every day)

2. What if it's a smashing success?

   We might extend the Daily Rewind to a Real Board: Oz64 SG2000 RISC-V SBC.

   Or maybe SG2000 Emulator and Ox64 Emulator, since they're quicker and more consistent than Real Hardware. (Though less accurate)

   Plus other QEMU Targets: rv-virt:nsh / nsh64 / knsh

3. Suppose we wish to add Our Own Boards to the System?

   Let's assume we have Automated Board Testing. Then we could upload the NuttX Test Logs (in the prescribed format) to GitLab Snippets or GitHub Gists. They'll appear in NuttX Dashboard and Build History.

   (Rewinding the Build on Our Own Boards? Needs more work)

4. Why Rewind every commit? Isn't Git Bisect quicker?

   Ah remember that we're fixing Runtime Bugs, not Compile Errors. Git Bisect won't work if the Runtime Bug is Not Reliably Reproducible.

   When we Rewind 20 Commits, we'll know if the bug is Reliably Reproducible.

5. Why aren't we using Docker?

   Docker doesn't run OSTest correctly on QEMU RISC-V 64-bit.

6. Any more Grand Plans?

   We might allow a PR Comment to trigger a Build + Test on QEMU. For example, this PR Comment...

   @nuttxpr test rv-virt:knsh64
   

   Will trigger our Test Bot to download the PR Code, and run Build + Test on QEMU RISC-V. Or on Real Hardware...

   @nuttxpr test milkv_duos:nsh
   

   Super helpful for Testing Pull Requests before Merging. But might have Security Implications 🤔