New Hybrid Conjugate Gradient Method with Global Convergence Properties for Unconstrained Optimization

Nonlinear conjugate gradient (CG) method holds an important role in solving large-scale unconstrained optimization problems. In this paper, we suggest a new modification of CG coefficient 𝛽 𝑘 that satisfies sufficient descent condition and possesses global convergence property under strong Wolfe line search. The numerical results show that our new method is more efficient compared with other CG formulas tested


Introduction
The general form of an unconstrained optimization problem is defined by min xϵR n f(x) , (1) where :   → is a continuously differentiable function and its gradient  ≡ ∇()is available.The iterative formula of the CG method is given by where   the step-size computed by carrying out strong Wolfe line search procedure, defined as follows (4) where 0 <  <  < 1 The parameter  is the search direction defined by   = { −  ,   = 0 −  +    −1 ,   ≥ 1 (5)

Modified Formula and Algorithm
Recently, Wei et al. (2006) gave a variant of the PRP method which is called the WYL method, written as The WYL method and PRP methods both come with restart properties.Zhang (2009) studied and improved WYL CG method and suggested the NPRP method, formulated as Zhang (2009) proved that the NPRP method satisfies descent condition under strong Wolfe line search.Later, Dai and Wen (2012) proposed a modified NPRP method as follows: Based on the above ideas, we present a new β k known as β k YHM , where YHM denotes Yasir, Hamoda and Mamat.The formula for β k YHM is defined by The following algorithm is a general algorithm for solving optimization by CG methods.

Global Convergence analysis
In this section, we study the global convergence properties of β k YHM , starting with the sufficient descent condition.Firstly, we need to simplify β k YHM so that the proving steps will be easier.From (9), we know that: Otherwise, By Cauchy -Schwartz inequality, it is implied that Hence, we can deduce that for both cases of 0 ≤ g k T g k−1 ≤ ‖g k ‖ 2 and otherwise, β k YHM ≥ 0 . (10)

Sufficient descent condition
The sufficient descent condition is defined by: The following theorem shows that YHM with inexact line search possesses the sufficient descent property.Theorem 1.Suppose that the sequence {g k } and {d k } are generated by Algorithm (2.1) and the step-length α k is determined by strong Wolfe line search.If g k ≠ 0, then the sequence {d k }satisfies the sufficient descent condition for all k ≥ 0. Proof.The proof of the descent property of {d k } is by induction.Firstly, we prove the theorem for the case of k = 0. 11) is fulfilled for k = 0. Now suppose that d i , i = 1,2,3, … , k are all descent directions that isg i T d i < 0. From the strong Wolfe condition, and (10)

Case (2):whenβ
, the proof of this theorem can be seen in (Wei et al., 2006).

Global Convergence Properties
The following assumptions are often used in the studies of the CG method.Assumption 1 A. f(x) is bounded from below on the level set Ω = {x ∈ R n , f(x) ≤ f(x 0 )} wherex 0 is the starting point.
B. In some neighbourhoodNof Ω, the objective function is continuously differentiable and its gradient is Lipchitz continuous, that is, there exists constant L > 0 such that: ‖g(x) − g(y)‖ ≤ L‖x − y‖, ∀x, y ∈ N (17) In 1992, Gilbert and Nocedal introduced property (*) which plays an important role in the studies of CG method.This property means that the following search direction automatically approaches the steepest direction when a small step-length is generated, and the step-length are not produced successively (Zhang et al., 2012).

Property (*)
Consider a CG method of the form (2) and (3).Suppose that for all k ≥ 1, 0 <  ≤ ‖g k ‖ ≤ γ − (18) whereγ and γ − are two positive constants.The method has property (*) if there exist constants b > 1 andλ > 0 such that for all k: The following lemma shows that the new parameter β k YHM possesses property (*).
Lemma1.Consider the method of form ( 2) and ( 3), and suppose that Assumption 1 holds, then the CG method withβ k YHM has property (*).
The proof is complete.
Case 2: ,the proof of this theorem can be seen in (Wei et al., 2006).Lemma 2.Suppose that Assumption 1holds andx k is generated by Algorithm 2.1 whered k satisfies g k T d k < 0 for all k.The step sizeα k is obtained by (SWP) line search ( 4) and ( 5), then, Proof.By Assumption 1 and the strong Wolfe line search, we obtain We combine (20) with ( 12), which then results to The proof is complete.Theorem 2.Consider any CG method of the form (2) and (3) that satisfies the following conditions: (1) β k ≥ 0 (2) The search directions fulfil the sufficient descent condition.
(4) Property(*) holds.If Assumptions1 and2 hold, then the iteration are globally convergent.From equations ( 11), ( 16), and ( 17) and Lemma 2, we found that the YHM method satisfies all four conditions in Theorem 2 under the strong Wolfe line search, so the method is globally convergent.

Numerical results and discussions
In this section, we present the results of the numerical tests conducted on our new parameter.
The test problems used are taken from Andrei (2008), as shown in Table 1.We measure the performance of the proposed method by comparing it with other, well-established CG methods; FR, PRP, WYL and DPRP.A laptop with Intel(R) Core™ i5-M520 (2.40GHz) CPU processor and 4GB RAM in addition to MATLAB software version 8.3.0.532 (R2014a)are used to execute the optimization algorithms.We consider‖  ‖ ≤  as the stopping criteria as suggested by Hillstrom (1977) with  = 10 −6 .The dimensions of the test problems lay in the range of 2 to 10000.For each test function, we use four initial points, starting from a point close to the solution to another point far from it.In some cases, the computation is stopped due to the line search failing to find a positive step-size, thus it is considered a failure.The performance results are shown in Figures 1 and 2, respectively, based on the performance profile introduced by Dolan and More (2002).From Figures 1 and 2, we found that our proposed algorithm solves 100% of the test problems, followed by WYL which solves 99.4% and DPRP with 88.4%of problems solved.Older CG methods like FR and PRP solve about 57% and 49 % of the test functions, respectively.

Conclusion
This paper gives anew  formula for solving unconstrained optimization problems.Under strong Wolfe line search, this new  possesses global convergence properties.Numerical results show that the YHM method is very efficient and has the best performance when compared with other tested CG methods.

Figure 1 .
Figure 1.Performance profile relative to the number of iteration.

Figure 2 .
Figure 2. Performance profile relative to the CPU Time