Manually Installing Software Agents

I used to be the lead developer on a product that automatically deployed a software agent to every machine on a user-defined list. One of our customers wanted to deploy agents on computers in both America and Japan from a console in New York, but our agent was too large to send over the low bandwidth connection between their American and Japanese offices. So the customer demanded we add the ability to manually install agents on their Japanese machines.

All this happened while I was on vacation. When I returned, I found the install team assuming we had to add support for manual installation of agents. That was what the customer had specifically asked for, after all. Lucky for me, I'd been out of town when that assumption was made, so I immediately asked, "If our product push-installs to computers in a user-defined list, and if the goal is to avoid push-installing over the slow America-Japan connection, then why can't we just install our console on a machine in Japan as well as in America? Let that Japanese console push-install to the Japanese machines, and then uninstall the Japanese console. The agent would then be installed on every machine in both countries, and we wouldn't have to change a single line of code."

And that's exactly what we did. If only the install team had spent a few minutes to think through their assumption that manual installation was necessary, they would have saved themselves a lot of frustration. These guys weren't stupid. They were just stuck on an assumption; in fact, they weren't even aware they had made an assumption at all. That's the power of assumptions. Once you get them in your head, it's hard to get them out. Don't be like that. Recognize when you're making assumptions.

Finding the Point of Failure

All right! Now we have a goal to shoot for: Find the point of failure. We can just look at each step along the way and try to think of a way to test if that part is working. See Figure 3-2 for a conceptual breakdown of how the phone system works.

Figure 3-2: A conceptual breakdown of the cell phone system

One possibility is that my friend's phone is broken. Maybe her phone is failing to transmit the message to the phone company's voice mail server. That explains why I never get her messages. How reasonable does that theory sound? And if it is true, how would we test it? Take a second to think about it. Testing whether the problem is the outgoing message from her phone is pretty easy. We know my phone can't leave her messages either, and our other friends can't leave either of us messages. It seems unlikely that all our phones could break in exactly the same way and at exactly the same time. We can't completely rule out the possibility, but it looks like a safe assumption that the problem isn't in her phone. And by the same logic, assume the problem isn't in my phone, either.

Could the problem be in the satellites and cell phone tower infrastructure? What would be the consequences if that were the case? Presumably, the infrastructure just transmits data. Presumably, it doesn't know or care whether that data is a phone call or a pager message or a voice mail recording. Since I can make regular phone calls with my phone, and even leave recorded messages on my friend's old-fashioned answering machine, we can probably assume the towers, satellites, wires, etc., are all transmitting data correctly.

Besides, if a satellite had gone dark, or a cell phone tower had failed, you have to think the phone company would have noticed. It's too expensive to monitor every individual phone for failure, but you can bet the phone company has an army of engineers constantly monitoring the performance of the multimillion dollar satellites and towers. If there were a problem in this area, every cell phone in the city would be having trouble, and someone would have noticed and fixed the problem already.

Reviewing the Assumptions

Now hold on a second. Did you notice the number of assumptions we just made? We're assuming the phone company is able to quickly fix problems in its infrastructure. Maybe that's not true. And we assumed that if every cell phone in the city were broken, then the phone company would be flooded with service requests alerting them to the problem. That might not be true, either—maybe nobody noticed the phones were broken because people in my city treat cell phones as mere status symbols. And we even made assumptions about the implementation details of the infrastructure. Surely, that's risky. We can't rule out the possibility that the satellite transmits recorded messages on a different wavelength than voice messages, which could potentially explain the bug. Hey, it's possible!

All our assumptions sound reasonable, but we could be wrong. Is that a problem? Not if we're careful. Assumptions are perfectly fine—we can't get anywhere without them. But just remember that all our logical deductions from this point in the debugging process on are based on premises that aren't 100 percent proven, and if we get stuck, be willing to come back to this point and reexamine our assumptions to see which one no longer fits with the evidence.

Yet for now, let's assume the preceding assumptions are correct. We've ruled out all the major components except the voice mail server. By the "everything is working until WHEN?" approach, we should focus there until we uncover evidence pointing elsewhere. Of course, we should at least consider that the problem might not be in any component. Lots of times, each individual component works correctly but the problem is the interaction between them. This is extremely common in software—two components each work exactly as designed, but the communications interface between them fails to convey some important nuance.

But let's make another assumption and hypothesize that's probably not the case with this bug. The protocols used by cell phones have been fully standardized for two decades, and even if the phone company had changed protocols for some reason, then all kinds of other problematic behaviors would have been reported. Try to think of ways to confirm this theory via testing—I'm sure you can come up with dozens.

Reasoning About Strings

Another bug I once saw involved strange behavior with strings. A coworker reported his mixed COM/.NET program crashed with a MarshalDirectiveException when he ran it. When I ran it, there was no crash, but the output string had a dozen characters of unexplained garbage at the end. I didn't have access to the source code, so I couldn't debug properly. But thanks to my experience with COM programming and my ability to reason about the implementation of the system, I had a hunch about what the problem was.

In.NET, there is only one type of string class, and it's easy to forget legacy C++ has dozens of incompatible string types ( char*, wchar*, TCHAR*, CString, std::string, BSTR, CComBSTR, _bstr_t, and so on). But the BSTR is especially important for COM programming. BSTRs are Unicode strings that are prefixed with the length. This length prefix is very important—otherwise languages like VB6 would be unable to read the string. Unfortunately, however, Microsoft's compiler never provided a warning if a non-length-prefixed wchar* was used where a length-prefixed BSTR was expected or vice versa.

My coworker's legacy COM component returned a BSTR to his.NET component. However, he was returning a non-length-prefixed string rather than a properly prefixed BSTR. (He was hardly the first person to make this mistake— millions of COM programmers have done it.) When.NET marshaled the string from the COM component, it expected a length-prefixed string, so unpredictable behavior occurred depending on what the value of the 2 bytes before the string pointer just happened to be. I showed this to my coworker, and a single line change fixed the problem.

Moral #1: When interacting with (or writing) legacy COM components that deal with BSTRs, make sure you are using the SysAllocString API to create real BSTRs rather than simply allocating a standard string.

Moral #2: Once a day, every day, thank your lucky stars that.NET components have only a single string class.

Building a Conceptual Model

We believe there's a bug in the black box component that we call the voice mail server. We don't know the implementation details, but we do know there are at least three different major functions that the server carries out, so let's conceptually treat those three different functions as different components. The server might not actually be implemented this way; but this is a good way to visualize and reason about it. Once we understand what's probably going on in this component, we'll be in a much better position to think about ways to find and fix the bug. As for me, I thought about the likely implementation of the server and came up with the diagram in Figure 3-3.

Figure 3-3: A conceptual breakdown of the voice mail server

Why do I think the voice mail server looks something like that?

First, the messages obviously have to be stored somewhere. Somewhere in the system, there has to be some kind of database or a tape recorder or a secretary with a notepad or a flock of parrots or something to memorize the messages and store them in a place for later retrieval. We don't know the form of this component, but we know something like it has to exist—otherwise, how could the messages be preserved? This component is still a black box, but at least it's a smaller black box than the entire voice mail server, so we should be able to reason about it. Whatever the form of this component, let's call it the data storage object.

What else must logically exist somewhere in the system? Well, my phone has an icon that changes colors depending on whether I have messages in my voice mail inbox. Some part of the server must be responsible for notifying my phone to change the icon. Who knows what the implementation details are? Maybe my phone pings the server every few minutes to ask if there are any new messages. Or maybe the server sends a message to my phone telling it to switch the color of the icon whenever a new message comes in. Either way, this notification functionality has to be handled somewhere. Let's call the component that does this the notification object.

And finally, there is some kind of interface around the data store component to provide routing, security, database querying, etc. When I leave a message for Sally, the message goes in her mailbox, not Harry's. There's security there, too: Some component of the server ensures that only the person calling from Sally's phone can hear her messages. When Harry calls the server, he automatically gets only the messages in his mailbox. And there's a nice little spoken interface on top of each mailbox—"Press 1 to delete this message, 2 to repeat the message," etc. So we're not dealing with raw SQL queries against a database; there's a layer of abstraction to hide the details. All of that functionality has to be handled somewhere, and let's call it the interface component.

The Limits of a Conceptual Model

Breaking black box components into smaller conceptual components is a powerful trick. But of course there are limits. We could drill down into deeper details for each subcomponent, but at some point, hypothesizing about the details of the black boxes becomes mere guesswork. Already, we've assumed three separate components when maybe two of those components were actually built together in one monolithic database trigger. And even though we've identified three major areas of functionality, that doesn't mean there are just three components—probably the system is far more complex than our diagram shows. So we can't use our logical view of the system to make assumptions about the physical layout.

Still, this not-necessarily-accurate-but-close-enough logical view has already given us something to go on. Now we can come up with theories to test. For instance, is the data store component losing messages? Is the notification component failing to send notifications about new voice mail? Is the interface component sending messages to the wrong mailbox? Supposing that all our assumptions were correct up to this point (always being willing to reconsider this crucial step), then the bug is likely to be found in one of those three areas.

Choosing Which Area to Examine First

Databases are one of the most mission-critical applications in the world, so for that reason alone, assuming a failure in the data store object is a bold claim. I should hope that whoever designed the phone company's system checked, double-checked, and triple-checked the data store to make sure data would never be lost. In fact, considering how crucial this component is, it's likely the phone company probably used an existing, well-tested database system rather than creating their own. For this reason alone, I'd examine the other components first before spending too much time looking for a bug in this area.

That's not airtight logic, of course. Just because this component is crucial doesn't mean there's no bug here. I'm merely saying that since we have an argument for why the bug is slightly less likely to be in this component than the other components, we might consider investigating those other areas first and only coming back to this if necessary. We haven't ruled this area out. We're just going to follow the most promising leads first before falling back to less promising leads.

Suppose we did that, though. Suppose we ruled out the other components or else found evidence pointing back here. What would be the game plan for finding whether there's a bug in the data store object?

First, ask what could possibly be wrong with the data store. Without going into low-level details, the data store could either be failing to save the voice mails, or else it could be failing to retrieve them. Those are the only two possible symptoms. We just need to figure out if either of those symptoms is occurring.

If this were a traditional debugging situation, then we'd have access to the data store object and could run tests against it directly. Most databases are equipped with tools for viewing (or in this case, hearing) the data, so we could check to see if messages were getting correctly stored. There would probably be a query analyzer, too, for seeing the results of data retrievals. Unfortunately, you and I don't have access to the phone company's data store, so we have to be more indirect in our debugging. What could we do instead? Pretend this was your bug. What would you do here?

Some Possible Tests

A few possible tests:

• Even though the voice mail indicator icon says you have no new mail, call in to check anyway. If your mailbox really does have mail in it and if you can successfully retrieve it, then the problem clearly isn't in the data store system.

• If there are no new messages in your mailbox, do you have any old, undeleted messages from before this bug appeared? Call the server to listen to those old messages. If this works, then message retrieval is fine and the problem is that new messages aren't getting stored correctly.

• Presumably there is some size limit to your mailbox. What happens when you exceed that limit? I don't know, but you might consider leaving yourself hundreds of blank messages—if at some point you're unable to leave that 100th message because the voice mail system says something like "This mailbox is full and can accept no more new messages," then that probably indicates that the messages really are getting stored somewhere.

And what if the problem were in the interface component instead? What would you do to test that one? And what about the notification component? Try to reason about those objects and see if you can think of any good, easy tests. The best tests confirm or deny theories as quickly as possible.

When I was thinking through this problem, I noticed one truth—my test cases for all three components included the simple test of ignoring the voice mail notification icon and calling the server to check my voice mail anyway. That one step would provide information about all three components, so that's the first thing I decided to try.