A Combination of FR and HS Coefficient in Conjugate Gradient Method for Unconstrained Optimization

The conjugate gradient (CG) method is one of the most popular methods for solving large-scale problems of unconstrained optimization. In this paper, a new CG method based on combination of two classical CG methods of Fletcher-Reeves (FR), and Hestence-Stiefel (HS) is proposed. This method possess the global convergence properties and the sufficient descent condition. The tests of the new CG method by using MATLAB are measured in terms of central processing unit (CPU) time and iteration numbers with strong Wolfe-Powell inexact line search. Results presented have shown that the new CG method performs better compare to other CG methods.


Introduction
The conjugate gradient CG method is one of the optimization methods that are often used in specific applications.The general unconstrained optimization problem is stated by the following rule, where :   →  is continuously differentiable,   refers to the -dimensional Euclidean space,  ∈   is a vector with independent variables, and () is an objective function.The efficiency of the conjugate gradient method CG is the ability to find the minimum value of a function for unconstrained optimization problem and the low memory storage (Yuan and Sun, 1999) and (Hamoda, Rivaie, Mamat, and Salleh, 2015).It is commonly solved by iterative method which defined as the following form,  +1 =   +     ,  = 1,2, ..
where the current iterative point is   the step size is   > 0 and   is the search direction of conjugate gradient CG method.In order to solve the step size there are two ways which are exact and inexact line search.Recently, researchers tend to work with inexact line search since it converge faster.The inexact line search is powerful in practical computational.The search direction of conjugate gradient CG method   is defined by: where   and   is the conjugate gradient coefficient of () and the gradient respectively at the point   .Which we have   ∈  is a scalar while   = (  ) at the point   .Some strengthknown CG methods and their modifications have been created by many researchers are given as follows and we can find all of the methods through (Hager and Zhang, 2005), (Powell, 1986), (Dolan and Moré, 2002) and (Andrei, 2008).
In this study, we create a new CG method with a formula for   Section 2, is the algorithm with our new CG parameter.The proof of the sufficient descent condition for the new parameter under the strong-Wolfe technique is established in Section 3. Section 4, includes the numerical results, discussion and the set of chosen test functions.Lastly, our conclusion is presented in Section 5.

New CG Coefficient
In this section, a new CG coefficient,   based on combination of FR and HS methods had been proposed.This new CG parameter is known as    , where SM denotes Saleh and Mustafa as below The following is the algorithm of new CG method    used in this study: Step 1: Initialization.Given  0 , set  = 0.
Otherwise, go to Step 4.
Step 4: Compute step size   by using inexact line search based on (11) and (12).
Step 5: Update a new point by using (2) Step 6: Compute the convergence test and the stopping criteria.
Otherwise go to Step 1 with  =  + 1.

Convergent Analysis
A good algorithm should fulfil the sufficient descent and global convergence properties.The sufficient descent properties of this new method will be further studied in this section.From (13), we know that Since the HS method is related to conjugacy condition regardless of the objective function and line search (Salleh and Alhawarat, 2016).
i.e      −     −1 = 0 (15) Then ( 14) becomes We need to show that the method satisfies the descent properties From the search direction   , we have The following theorem would be used to show that the proposed method satisfy (17).
Theorem 1: consider a CG method with search direction   and CG coefficient    , then condition (19) holds for all  ≥ 0.
The following assumptions are always useful for convergence analysis of CG approaches, Assumption 1: The level set Ω = { ∈   |() ≤ ( 0 ) } is bounded, where  0 is the starting point and  is a smooth function in a neighborhood  of the level set Ω.
The following theorems originate from this assumption.
Lemma 1: Assume Assumptions 1 holds, consider any CG techniques of the form ( 2) and ( 3),   obtained by Strong Wolfe line search.Then, the Zoutendijk condition holds For the proof of the above lemma, see (Zoutendijk, 1970).Still base on Lemma 1, we have this convergent theorem.
Theorem 2: Assume that Assumptions 1 hold, then for any CG techniques of the form ( 2) and (3),   obtained by Strong Wolfe line search, and the coefficient   obtained by ( 14).Then Proof: This prove is by contradiction.i.e. let's assume Theorem 2 is not true, then there exist a constant say ∅ such that Equation definition of   can be rewritten as Squaring both of ( 23), we get, dividing through ( 24) by ( +1   +1 ) 2 , we have, Applying condition ( 16), we have Then from ( 26) and ( 22), we have However, this contradicts our assumption in Lemma 1.Thus, this completes the proof.∎

Numerical Results and Discussion
In this section, by using the new algorithm the efficiency of new CG parameter, SM can be proved by comparing with four methods, HS, FR, CD and RMIL.We use the test functions taken from (Andrei, 2008) and (Molga, 2005).The output is based on CPU time and number of iterations.All of these algorithms are considered  = 10 −6 .As the stopping criteria, ‖  ‖ ≤ 10 −6 has been fulfilled, the calculation of all methods are terminated.We used MATLAB R2018a and run using Intel Core i3 with RAM 3GB and Window 7 Operation System.Table 1 below presents a list of problem functions, dimension and the initial points to test the method, SM.For every test functions used, there are four dimensions and four initial points.Where the initial points are chosen starting from a point near to the solution point to a point further away from solution point, so that it can be used to test the global convergence of new CG coefficient (Hillstrom, 1977).

Conclusion
In this paper, a new CG method based on combination of FR method and HS method for solving unconstrained optimization problems using strong Wolfe-Powell inexact line search is proposed.
Based on the result, our parameter satisfied the sufficiently descent and global convergence properties.The results show that SM method gives the fastest performance in terms of CPU time and solves problems with minimum number of iterations.SM method gives the best performance compared to HS, CD, FR and RMIL.Thus, SM is a good method for solving unconstrained optimization problems.
Under a strong Wolfe-Powell inexact line search, Fig.1gives the number of iterations graph and Fig.2presents the graph to show the performance profile in terms of CPU time.Based on performance profile introduced by Dolan and More(Dolan and Moré, 2002), the performance results are shown in Figures1 and 2. The figures clearly represent the performance profile of, SM, HS, CD, FR and RMIL.In both figures, the left side of figure represents the method which is fastest in solving test problems, while the right side represents the test problems that were successfully solved by each method.Obviously, the performance profile of SM is the fastest as it is above the other curves, which follows by HS, RMIL, CD, and lastly by FR methods.

Figure 1 .
Figure 1.Performance Profile with respect to the number of iterations

Figure 2 .
Figure 2. Performance Profile relative to the CPU times

Table 1 .
The serial number of a given test functions, list of named problems, dimensions and initial points of all functions used