We currently have a tool called reftest, which compares the rendering of HTML testcases against reference files – usually written in HTML, but they could just be a container document enclosing a PNG.
It would be really cool if reftest were a Tinderbox test, with the tinderbox building Firefox with the particular options reftest needs. We could then enhance it to also run various HTML and CSS testsuites on the web (e.g. Hixie’s) and also do Save Page As… on various popular websites and add those files into the mix.
If we could get to a position where our latest rendering engine were being automatically and regularly checked against thousands of testcases and real world pages, we’d be immediately be able to see the effect on the web of any changes we made. We’d be much better placed to catch regressions.
Anyone feel like making this happen?