Some projects go even further, requiring that a new test accompany every bugfix or new feature. Whether this is a good idea or not depends on many factors: the nature of the software, the makeup of the development team, and the difficulty of writing new tests. The CVS (http://www.cvshome.org/) project has long had such a rule. It is a good policy in theory, since CVS is version control software and therefore very risk-averse about the possibility of munging or mishandling the user’s data. The problem in practice is that CVS’s regression test suite is a single huge shell script (amusingly named sanity.sh), hard to read and hard to modify or extend. The difficulty of adding new tests, combined with the requirement that patches be accompanied by new tests, means that CVS effectively discourages patches. When I used to work on CVS, I sometimes saw people start on and even complete a patch to CVS’s own code, but give up when told of the requirement to add a new test to sanity.sh.

It is normal to spend more time writing a new regression test than on fixing the original bug. But CVS carried this phenomenon to an extreme: one might spend hours trying to design one’s test properly, and still get it wrong, because there are just too many unpredictable complexities involved in changing a 35,000-line Bourne shell script. Even longtime CVS developers often grumbled when they had to add a new test. … It is true that switching to a real test framework—whether custom-built or off-the-shelf—would have been a major effort. But neglecting to do so has cost the project much more, over the years. How many bugfixes and new features are not in CVS today, because of the impediment of an awkward test suite?

— Karl Fogel, Producing Open Source Software