wGMDH :: Least-squares Fitting

Least-squares fitting

Consider a second-order polynomial, a nonlinear output of the network node, utilized for least-squares fitting of its input regressors $\mathbf{x_{m}^{t}}$ and $\mathbf{x_{n}^{t}}$ to $\mathbf{y^{t}}$

$\hat{y}{_{i}}^{t}=\mathbf{a}\begin{bmatrix} 1\\ x_{mi}^{t}x_{ni}^{t}\\ x_{ni}^{t}\\ x_{ni}^{t}^{2}\\ x_{mi}^{t}\\ x_{mi}^{t}^{2}\\ \end{bmatrix} = \begin{bmatrix} a_{0} & a_{1} & a_{2} & a_{3} & a_{4} & a_{5} \end{bmatrix} \begin{bmatrix} 1\\ x_{mi}^{t}x_{ni}^{t}\\ x_{ni}^{t}\\ x_{ni}^{t}^{2}\\ x_{mi}^{t}\\ x_{mi}^{t}^{2}\\ \end{bmatrix}$
The error of approximation is given by

$e_{SSE}^{t}=\sum_{i=1}^{M}(\hat{y}_{i}^{t}-y_{i}^{t})^{2}$
The minimal error is obtained for a that is a solution to the system of equations

$\begin{bmatrix} N&\sum x_{mi}x_{ni}&\sum x_{ni}&\sum x_{ni}^{2}& \sum x_{mi}&\sum x_{mi}^{2}\\ \sum x_{mi}x_{ni} & \sum x{_{mi}}^{2}x{_{ni}}^{2}&\sum x_{mi}x{_{ni}}^{2}&\sum x_{mi}x{_{ni}}^{3}&\sum x{_{mi}}^{2}x_{ni} &\sum x{_{mi}}^{3}x_{ni}\\ \sum x_{ni}&\sum x_{mi}x{_{ni}}^{2} & \sum x{_{ni}}^{2} & \sum x{_{ni}}^{3}&\sum x_{mi}x_{ni}&\sum x{_{mi}}^{2}x_{ni}\\ \sum x{_{ni}}^{2}&\sum x_{mi}x{_{ni}}^{3} & \sum x{_{ni}}^{3}&\sum x{_{ni}}^{4}&\sum x_{mi}{x_{ni}}^{2} & \sum x{_{mi}}^{2}x{_{ni}}^{2}\\ \sum x_{mi}&\sum x{_{mi}}^{2}x_{ni} & \sum x_{mi}x_{ni}&\sum x_{mi}{x_{ni}}^{2}&\sum x{_{mi}}^{2}&\sum x{_{mi}}^{3}\\ \sum x{_{mi}}^{2}&\sum x{_{mi}}^{3}x_{ni}&\sum x{_{mi}}^{2}x_{ni}&\sum x{_{mi}}^{2}x{_{ni}}^{2}&\sum x{_{mi}}^{3}&\sum x{_{ni}}^{4} \end{bmatrix} \begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ a_{3}\\ a_{4}\\ a_{5} \end{bmatrix} = \begin{bmatrix} \sum y_{i}\\ \sum y_{i}x_{mi}x_{ni}\\ \sum y_{i}x_{ni}\\ \sum y_{i}x{_{ni}}^{2}\\ \sum y_{i}x_{mi}\\ \sum y_{i}x{_{mi}}^{2} \end{bmatrix}$
where the summations are over all the instances.

The least-squares error can be evaluated on the validation set according to

$\hat{y}{_{i}}^{v}=\mathbf{a}\begin{bmatrix} 1\\ x_{mi}^{v}x_{ni}^{v}\\ x_{ni}^{v}\\ x_{ni}^{v}^{2}\\ x_{mi}^{v}\\ x_{mi}^{v}^{2}\\ \end{bmatrix} = \begin{bmatrix} a_{0} & a_{1} & a_{2} & a_{3} & a_{4} & a_{5} \end{bmatrix} \begin{bmatrix} 1\\ x_{mi}^{v}x_{ni}^{v}\\ x_{ni}^{v}\\ x_{ni}^{v}^{2}\\ x_{mi}^{v}\\ x_{mi}^{v}^{2}\\ \end{bmatrix}$
$e_{SSE}^{v}=\sum_{i=1}^{M}(\hat{y}_{i}^{v}-y_{i}^{v})$