Tuesday, September 06, 2005

CalcWindowRect bug

This bug first appeared during the development of movie export. Moving the export to a separate process made it go away, but now it's back in version 1.0.73. Most likely the AVI file DLL was falsely accused and the movie export could have been part of the main app, but that's a secondary issue.

The symptom is, the app crashes in RowDialogForm's CalcWindowRect during the construction of the Parms dialog. It doesn't crash in debug, only in release, and creating a console window makes it go away (great!). Deleting the unused m_Template member variable from CPersistDlg.h also makes it go away. This last symptom clearly points towards a memory corruption problem of some kind. In the release debugger, the problem begins when CRowDialog::OnInitDialog calls CViewDialog::OnInitDialog. After this call, CRowDialog's "this" and HWND are trashed. Note that neither CViewDialog nor its immediate base class (CPersistDlg) override OnInitDialog, so this is effectively a call to CDialog::OnInitDialog.

Removing the call to MakeVJAccelTable in CMainFrame's ctor also makes it go away.

The really bad news: the app only crashes when you execute it from the IDE; running from Explorer is OK.

Other observations:
in Frame's ctor list, m_ResultsDlg(m_View), is incorrect initialization, m_View isn't the dialog's parent and in any case it's NULL! deleting this line doesn't help though...

could be passing 'this' to dialogs isn't such a good idea after all, seems like CMainFrame's 'this' keeps changing somehow or is that just the debugger being bad in release mode?

The solution: in CalcWindowRect, casting GetParent() down to CRowDialog isn't always correct. Initially, the parent is the dummy frame, not CRowDialog. Use a dynamic downcast to determine the parent.

The moral of the story: Downcasting is dangerous so use dynamic downcast and verify the result!

No comments: