How to efficiently track down a bug in Gutenberg (or other large JavaScript project)?

Okay. This is probably a jumble. It’s also a question disguised as an explanation of my experience. It’s possible working through this all has answered my question, but…

I would really like to know if I am missing something more obvious about juggling npm dependencies and a build process when using git bisect to identify a breaking change. Help?

Some background: I have used git bisect extensively when contributing to the WordPress core project over the years. This has always been generally straight-forward because WordPress existed as a web application that could be run without a build step. All JavaScript ran in the web browser natively and did not need any fancy bundling or transpiling. You could check out a commit and reload a page in your browser to see if the bisect condition was good or bad.

When a build process was officially introduced to WordPress, each commit to the development repository also generated a corresponding commit to the core repository. You could use either to track down the original changeset. I believe it was mostly possible to run src JavaScript in the browser and that the build processes were mostly used for minimizing and linting.

Okay. Now with Gutenberg.

I spent some Time last week figuring out how to track down a change in the Gutenberg repository that was incompatible with some code in my Shortnotes plugin. Quite a bit of that Time was spent battling npm, probably in a non-efficient way.

That said, I’ve turned a day lost to running npm install way too many times into a lesson that I think is accurate, but that I would love to be more confident in:

1. Checkout the project’s most recent commit and run npm install
2. Run the build process (for Gutenberg, npm run dev or npm run build)
3. Checkout an old commit
4. Run the build process
   1. If the build fails, run npm install to resolve dependencies using the old package.json and run npm run dev again.
5. Stash any dirty package-lock.json before attempting to checkout another commit.
6. Repeat in some form or another.

Okay. Whether or not that’s correct (please tell me if not!), I made things more difficult on myself this time around by letting several months pass before trying to fix this bug, so there are a lot of commits to sort through.

For this issue I had to go back to Gutenberg 11.3.0 before my code was no longer broken. I shortcut my normal process by performing my own initial bisect with wp plugin update to determine that a change between Gutenberg 11.3.0 and 11.4.0 caused my issue. Now I only need to look for a change in some 122 commits.

Alas, when I checkout v11.3.0 and run npm run dev, I’m told that a dependency discrepancy exists in Typescript. That’s a good notice because it effectively tells me to run step 4.1 from my flowchart above.

But then! When I run npm install, something fails in the react-native-gesture-handler@1.10.1-wp prepare script that prevents it all from working. I figure I’ve been relying on my initial npm install too much and should probably perform one from scratch. Nope, same error.

Here is where I spent a bunch of time assuming that I was doing something wrong.

I tried npm install with a new cache directory. I tried npm ci, I tried googling some stuff that wasn’t relevant. I probably downloaded a good terabyte of data trying to get npm install to work.

Jeremy, work smarter, not harder.

Finally, I looked up react-native-gesture-handler in the Gutenberg repository. Sure enough an issue was raised in August when npm install failed in the CI configuration. An attempt to fix it was made then, but I think the true fix came in September when all of the WordPress mobile dependencies were switched to tarballs rather than git repositories.

After some poking and prodding, I was able to check out Gutenberg v11.3.0, edit the package.json in Gutenberg’s packages/react-native-editor to use the tarball for react-native-gesture-handler, remove package-lock.json and run npm install successfully. 😅

Now I can run npm run dev, verify the issue did not exist in v11.3.0, check out v11.4.0, run npm run dev, and verify the issue did exist there. Progress!

Okay. So I’m slightly more confident when using git bisect on the Gutenberg repository. I can (hopefully) assume I stumbled on strange problems with npm that won’t always exist. The monolith is mighty, but not untraversable.

The biggest bummer now is that the process of git bisect good, npm run dev, reload the web page, type a character, and save the post to see if an error is shown takes roughly 2.5 minutes, of which almost the entirety is the build process. An issue that could take only a few minutes to identify instead takes at least 15 even if you’re staring at the screen and ready to pounce when the build is done.

I guess this could all be better if I wrote my own test for Shortnotes that checks to see if a post publishes successfully through the browser. Then I could script this all to run npm run build, run my tests by launching a headless browser, then mark the bisect as good or bad depending on the result, but yeesh… that would add another handful of minutes to the process and hobby projects should be fun! 🙂

I have some open items / takeaways / assumptions / maybe still questions:

1. Having automated tests for my plugin would eliminate much of the manual work in many cases.
2. Automated tests would not help when npm configs get all flaky. I would have to catch the error in an automated nightly test and then troubleshoot it while the npm config was still working. If I caught it on the next commit to my plugin, I’d probably go through the same pain.
3. There has to be a good way when running git bisect to watch for changes in package.json so that npm install can be run with more confidence. I need to do more research here.
4. I think it would be useful for WordPress to maintain a repository of built Gutenberg that is updated whenever a pull request is merged to the trunk branch. Testing and troubleshooting would be significantly easier in some cases.

Please: if you have any tips or opinions, share away here or in a post of your own! 🍻