When Unit Tests Go Bad Over Time

Today I had one of those I-didn't-change-anything-why-did-it-stop-working moments. Out of a sudden a unit test broke. I went back to the latest known working state, I checked out an older release from CVS -- it still broke. Something weird was going on: the release process makes sure that every unit test passes before creating a new release. Effectively, the past had changed.

The tested class is a somewhat complex piece of code for managing X.509 certificate chains. It makes sure that the chain is valid, that every single certificate is electronically signed, that the certificates aren't expired. And that's what happened: one of the certificates used to test this class expired.

The worst thing isn't the time wasted while trying to debug the problem; it's that we can't use older releases as-is.

I wonder what I can do in order to avoid it in the future. The nature of the tested code, the test arguments and the environment might make it hard. I can't just create a new X.509 certificate during test setup as it has to be signed by a certificate authority.

Any of you unit test gurus have an idea?

Tue, 22 Mar 2005, 20:54 | Java | PermaLink
« It All Depends On The Perspective | Home | Access COM objects with Java »