I first recieved this error when placing a webimagebutton inside the window.
I then removed the button and placed a grid inside the window, and every firing of any type of script causes this dialog window to appear.
I debugged the javascript and it looks like the error is a stack overflow, but I am unable to see what the cause of this is.
I have been playing around with the Update Panels on my page, and it appears I may have found the culprit. My webdialogwindow is in a usercontrol and the content page that I was calling the window from had an UpdatePanel which I changed to UpdateMode="Conditional" and now I don't recieve the javascript error.
Can you place a webdialogwindow inside an update panel?