The return value of showModalDialog is available under IE and firefox, but Null is returned under Google. The solution is described below

The parent page:

function openwindows(){
var obj = new Object();

//Opens the modal subform and gets the return value
var retval = window.showModalDialog("ordervideo.jsp?rderIds="+"0010,0020,0030",obj,"dialogWidth=500px;dialogHeight=500px");

//for chrome
if(retval == undefined) {
retval = window.returnValue;
}
alert(retval);
}

Child pages:

function onload(){

//for chrome
if(window.opener != undefined) { //The value of window.opener is not empty under Google browser and undefined under IE/firefox to determine if it is a Google browser
window.opener.returnValue = flag; // Method to assign a return value in the Google browser window.opener.close(); // This must be closed once, otherwise execute the following window.close() Unable to close the popup window because the popup window under the Google browser is new window
}

else {
window.returnValue=flag; //This method of assignment is compatible with Internet explorer/firefox, but does not support the Google browser
}
window.close();

}