Monday, December 12, 2005

Mad Debugging Skillz

My kung fu was strong today. I managed to slay two really annoying bugs that have been occupying space in my buglist for far too long now. To earn my kill I had to chase both bugs down into the lair of the nefarious MAPI (Messaging Application Programming Interface).

The first bug manifested itself as a lack of any and all address books when you installed GroupWise after Outlook XP. I had recently chased down a similar bug against Outlook 2003 and assumed that the two bugs were related. The previous issue had to do with how the MAPI stub was routing calls to the extended MAPI implementation. I verified that the calls were being routed correctly, but still wasn't getting any address books. I was about to write up an incident with Microsoft when it occurred to me to try running a test app against my MAPI profile to see what address books I got back.

When I ran the test app, I could see in the debugger that I got the correct number of rows back when I queried the root address book's hierarchy table, but none of them were being displayed to the debug output. Upon closer inspection, I realized that there was a bug in my test app. I was specifically looking for the ANSI version of PR_DISPLAY_NAME but I didn't specifically ask for it when querying the rows. Instead, I was getting back the UNICODE version of PR_DISPLAY_NAME.

It turns out that the address book application had the exact same bug. We were getting results back from the MAPI subsystem but since we weren't looking for the UNICODE display name, we ended up skipping over all of the address books because they looked like they were unnamed.

The second problem was that the DLL for our Message Store Provider was failing to register when anything other than Outlook was set as the default mail client. I was getting an error message that no default mail client was installed or that the default mail client wasn't capable of handling the call that I was trying to make... or something like that.

It is precisely the error message that you would expect to get when trying to use extended MAPI against a mail client that didn't provide an extended MAPI implementation. The problem was that registering the Message Store Provider DLL shouldn't have been making any MAPI calls. All it is supposed to do is modify the mapisvc.inf and register a COM server for our free/busy support object. It's basically just registry and file access.

I was able to trace the problem to a line of code in the ATL CComModule class that was calling 'ocslen' to determine the length of a string. When I stepped into the assembly for the call I noticed that ocslen was inlined to call a function called 'MNLS_lstrlen'. This function was resolving to mapi32.dll and was causing the MAPI subsystem to load and generate the error message... a stupid string length call. What the hell?

After some investigation, I discovered that in 'mapinls.h' lstrlen is #defined to MNLS_lstrlen. Bastards! I was able to resolve the problem by making sure that 'atlbase.h' was included before any of the MAPI header files.

Congratulations, if you have actually managed to read this far into this post. Unfortunately you don't win anything except for my respect :) I don't normally post this much detail about work, but I was pretty happy to have solved both of these problems today. What a pain in the ass.

3 comments:

Anonymous said...

I think it is cool that you know what all of that means!

Anonymous said...

Well that was an easy way to earn your respect, but I'm not sure it's fair, b/c even though I read it, I didn't understand it. I was preparing my comment in my head as I read: something like, please blog in English. But that's okay. We can't all be computer geniuses.

TK said...

Gee. Is this comment section only for family, or can I comment here, too? :)