I'm aiming to post a monthly article on bug trapping. I'll use real world samples of our own development, and any of those submitted by fans of DevPartner.
Bug of the Month. This month’s story is a bug of scalability.
When your code interacts with an existing framework, and that framework goes through a significant revision, you should examine closely the existing APIs you were using to ensure their behavior hasn’t changed, even when signature or calling conventions remain the same.
In our case, we have a series of callbacks made from within Visual Studio 10.0 into our “package”, which is a unit of code that realizes an extension to the Visual Studio environment. We register our package to receive callbacks from many events that fire as the end user takes Visual Studio through the motions of loading solutions, building solutions, enabling and disabling options, etc. We had one customer report that in his environment, intolerable delays were occurring for lots of different mouse events, in addition to those events raised by loading solutions, building solutions, enabling and disabling options.
It turns out that Visual Studio 2008 and Visual Studio 2010 differ in terms of the way they act as frameworks raising events to registered packages. This difference excited our code to repeat lengthy disk I/O operations almost continuously as various mouse events fired.
The scalability aspect of this bug is that even with this new unintended behavior occurring, you wouldn’t normally notice it with a “typical” solution of 5-10 projects. Functionally speaking, we would pass our normal IDE compatibility and source language tests. However, our customer’s solution contained 65 projects, and the protracted, continual disk I/O killed the performance of Visual Studio.
Once we recognized the symptom only reproduced at scale, we devised a set of reproducible tests and fixed the package to respect the new eventing model introduced by Visual Studio 10.0. We’ve proven with our fixes that even if the solution contained hundreds of projects, we can now scale with no extraneous disk I/O.
DevPartner Studio can trap this type of issue with coverage analysis. The technique was to profile and analyze our package by comparing the call counts from callback functions between sessions taken under VS2010 and sessions taken under VS2008. By explicitly measuring the API behavior, rather than just trusting that the callback signature didn’t change significantly, we could quantitatively see how often we were called and from where in the framework.
How has DevPartner helped you? If you can share your bug story, please let me know and I’ll capture and share the essence of how you trapped it.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.