Model Performance Measures

Different performance measures [3] can be used when selecting the model. A GMDH algorithm typically selects the model with best prediction based on the least squares estimate e.g. sum squared error

where  denotes the ith  instance of dependent variable from the validation data set and  is its approximation calculated by the corresponding polynomial model using the validation data set. Measures derived from it (by multiplication by a constant) are Root Mean Squares Error, Root Relative Squared Error. Relative Absolute Error is also an option.

Minimum Description Length (MDL) [4] is a well known principle which provides a tradeoff between the accuracy and the complexity of the model. According to [5], the MDL for linear polynomial regression consists of two terms:

where N denotes the number of observations and k is the number of parameters of the model. The first term in the equation can be interpreted as the number of bits necessary to encode the observations given the model while the second term can be understood as the number of bits necessary to encode the model. Optimizing with regard to MDL is in general a safeguard against overfitting, so it could potentially be used with the GMDH using one dataset only, istead of using the training and the validation set.

These error performance measures optimize only the corresponding error performances without regard to the complexity of the model. This can lead to complex models with low approximation error having unacceptably large calculation time. In certain applications the complexity of the model must be limited. In these cases it may be necessary to construct simplified surrogates with as low degradation of the approximation accuracy as possible. A simple two-parameter Compound squared relative Error (CE) measure for model selection has been proposed in [6]

where denotes the execution time (complexity) of the model,   and   represent the corresponding thresholds for the Root Relative Squared Error and the execution time, while   denotes the weighting coefficient. The CE measure consists of two normalized terms representing the error and the execution time (complexity) of the model. The weighting coefficient specifies the contribution of each term. For   the CE measure reduces to error-term and for   to complexity-term only. Unlike MDL and single parameter measures the proposed CE measure may control the way characteristics of selected models approach to specified thresholds, thus increasing the probability of discovering the model that satisfies both constraints.

The target model must satisfy the requirements regarding the accuracy and the complexity i.e. execution time:

The execution time (complexity) of the GMDH model can be estimated in the following way:

where  denotes the total number of basic second-order two-dimensional polynomials in the model,  and  is the corresponding total number of Floating Point (FP) additions and multiplications in the basic polynomial, while ,  and  denote the average execution time of software routines implementing the FP addition and the FP multiplication, respectively. If we rewrite the basic polynomial

its calculation is reduced to  FP additions and  FP multiplications i.e.