Another Improved Three Term PRP-CG Method with Global Convergent Properties for Solving Unconstrained Optimization Problems

Recently, several three-term conjugate gradient (CG) methods for solving unconstrained optimization problems have been proposed. Most of the methods centred on improving the convergence of the classical PRP-CG method while retaining its excellent numerical performance. Generally the PRP method is not convergent, because it failed to satisfy the sufficient descent property, especially, under modified Armijo line search or Wolfe line search method. In this paper, we propose an efficient three-term conjugate gradient method by utilizing the modified PRP formula which satisfies both the sufficient descent and the global convergence properties under the Wolfe line search. In particular, a new conjugate parameter is constructed. This parameter retains the numerator of PRP method and constructs an acceleration model in the denominator. The new denominator is designed to enhance the reduction in the number of iteration, CPU time as well as the convergence. Numerical results generated using various standard unconstrained optimization problem shows that the proposed method is promising and demonstrates a better numerical performance in comparison with other well-known CG methods.


Computing and Applied Mathematics
where :   →  is a smooth function and its gradient ∇(  ) =   is available.Equation above has many application in engineering, science, economic and art, particularly, when minimizing the cost or risk or improving the profit.
There are various methods for solving (1), each methods uses different search direction to generate the sequence of iterate{  }, that hopefully converge to the solution of (1).Among all the methods, Conjugate gradient (CG) method is more preferred, because of its simplicity and low memory requirement.This method use the recurrence   =   + (2) t o g e n e r a t e t h e s e q u e n c e {  }.W h e r e   > 0, i s c a l l e d t h e s t e p s i z e u s u a l l y, obtained using a line search procedure.The line search procedure either minimizes the objective function or reduces it sufficiently along the search direction   Rivaie et al . (2012)..There are two type of line search procedure, namely, exact and inexact line search.The Exact line search is too expensive and most of the times not suitable for solving real life problems.Researchers mostl y preferred the inexact line over the exact line search, especially, the Armijo, Wolfe, strong Wolfe or Goldstein method.The Armijo and strong Wolfe line search performance is quit near to the exact line search, that is why some times they appear not to gives any major improvement.Basically, method for solving unconstrained problems whose utilized the Wolfe line search appears to perform better both in theory and numerical computation.This method is defined by Where   = ∇(  ) is the gradient of  at   and   ∈  is a scalar, called the conjugate parameter.There are various conjugate parameters such as: the Hestenes-Stiefel (1952), Fletcher-Reeves (1964), Polak et al. (1969), Rivaie et al. (2012) and Abashar et al. (2015).
All the above parameters are equivalent if the objective function is quadratic and exact line search is used in finding the step size.However, for nonlinear problems each parameter leads to a different CG method.Over the years, the global convergent properties of these parameters have been study by different researchers.For example: Zoutendijk shows that FR method under exact line search is globally convergent for general function.Al-Baali et al. (2015) extends this result using inexact line search.Others researchers like Powell (1984), Hager and Zhang (2006) and Dai and Wen (2012) have also study the FR method.
The good convergence of the FR method is attributed to its ability to satisfy the sufficient descent properties (equation ( 5)), for some c > 0. The PRP CG method which did not satisfy (5) in most cases is not globally convergent Abdelrahman et al. (2015).Researchers, such as Gilbert and Nocedal (1992), Powell (1984), Rivaie et al. (2011) and Abashar et al. (2014) have independently developed a variant of PRP method.Their approaches provide a good convergent and efficient method under different line search procedure.Zhang et al. (2007) proposed a new class of conjugate gradient method called three term CG method via a modified PRP parameter as follows, where   =   −1 ‖  ‖ 2 and   = (  −  −1 ), to enhance the effectiveness of one-term and twoterm CG method.Since exception the three-term conjugate gradient has been widely studied and given much importance.The performance of this class of CG method largely depends on how the scalar parameter is being selected.Recently, Yanlin Wu ( 2017 Motivated by that, in this paper, we are interesting in developing a three-term conjugate gradient method which satisfies the sufficient descent condition, the trust region condition, and the global convergence under Wolfe line search method.We put our attention on the numerator of PRP method and constructing a new denominator which would accelerate the convergence results and lead to better numerical results.The rest of this paper is structured as follows: In section 2, we present our new method via modified PRP parameter.Section 3 deals with global convergence prove.The numerical results and discussion are reported in section 4. Finally, a concluding note is presented in section 5.

New Method
Recently, Kamilu et al. (2015) proposed a new CG formula, which has the same numerator as the PRP, HS and RMIL formula.The numerator was retained to give the formula a restart property.
This formula under certain condition can be reduced to FR or PRP parameter.And, geometrically, the denominator is not far different from the well-known CG denominator.Since   +  −1 forms a vector which is scaled by  −1 .The new vector would be another vector a little bite away from the normal vector produce by    (  −  −1 ) in PRP parameter.This little shift would improve the performance and simplified the convergence prove.
In addition, Z Dai and Wen (2012) used WYL formula To developed a new method which is globally convergent under exact and inexact line search.Inspired by Zhang (2007), Z Dai et al. (2012) and Wu (2017), below we state our new CG method: In this approach we used two restart properties to refresh the iteration process, whenever the distance is too small.The use of the second restart parameter is purposely to ease the theoretical prove.This is a small alteration that will guarantee another encouraging result.Algorithm 2.1; (KMM4) Initialization, given a starting point x0, let  ∈ (0,1) for k = 0 Step 1 Terminate if ‖  ‖ < 10 −6 or  ≥ 1000 Step 2 Find the search direction using ( 9), where   is given by ( 7) Step 3 Calculate the step size using Wolfe line search (3) Step 4 Updated   using (2) Step 5 Set k = k + 1, Go to Step 1.

Convergence Result
We begin by simplifying β KMAR , to makes the latter proof easier, This reduction is based on using the absolute value properties and Cauchy Schwartz inequalities.Now using algorithm 2.1, we first show that, the search direction   satisfied the sufficient descent condition.This condition is the main ingredient, that, guarantee the global convergent of any new method.But let start by making the following basic assumption on the objective function.

(ii)
In some neighborhood  ∈ Ω,  is continuously differentiable and its gradient () is Lipschitz continuous, i.e. ∃  > 0 such that Also, because the objective function is decreasing a s  → ∞ t h e n from (2) we can deduce that, the sequence generated by Algorithm 2.1 would be contained in the bounded region, which in turn would imply convergent.

SUFFICIENT DESCENT
Theorem 3.1 Let the iterate point   be given by Algorithm 2.1, then the search direction   given by (9) satisfies the following two relation for all k ≥ 0.
Note that the two inequalities ( 10) and ( 11) indicate that our new search direction has a sufficient descent and trust region properties.Now, we prove the global convergent of Algorithm 2.1 for both convex and non convex function.
Lemma 3.1.Suppose Assumption 1 holds, consider a three-term CG method in Algorithm 2.1, where   is a descent direction, and the step length   is obtained using the Wolfe condition.Then Lemma 3.1 is the popular Zoutendijk lemma, used in most of the optimization journal.The proof of this lemma can be found in Fletcher (1964).Proof: If  = 0 then ( 16) holds.Suppose ( 16) is not true, then there exist a constant ∈> 0 such that From ( 9), we have Squaring both side and applying descent condition we get Divided both side of ( 18) by (     ) 2 and let  ′ =  −1   ‖ −1 ‖ , we get Since 0 <    <    we get Which contradict (11), hence the proof is complete.The last inequalities is based on the Lipschitz condition.At last we have Therefore by ( 21) and ( 22), we have ( 15).The proof is complete.

Numerical Result
In this section, we present the numerical performance of our new method when compared with FR, PRP, MPRP and RMIL methods.All the algorithms are coded in MATLAB R2015b and tested for some well know benchmark problems from Dolan and More (2002) with various dimension from 2-30,000.For each method, we used   < 10 −6 to stop the iteration whenever the solution is achieved, or when the number of iteration exceed 1000.The name of all the standard test functions used is presented in Table 1.The table also includes the dimension of the functions.For the step size αk we used the following values µ = 0.83 τ = 0.0001.The performance profile introduced by Dolan and More (2002) is used to analyze the complete results.This profile gives the performance of a solver efficiency and probability of success in a generalized way.In Figures 1-2 we compare the performance based on the number of iteration and CPU time.For each method, we plot fraction P Of problems for which the method is within a factor t of the best time.The left-hand side of the figure shows the percentage of the test problem of which method is efficient and the fastest; the right-hand side of the figure represents the percentage of test problems that are solved successfully by each method  The Figure 1 and Figure 2 shows that our new method is efficient with best numerical a result, that is why its curves appear at the top, as expected FR has the lower performance, due to its jamming behavior.The performance of PRP, MPRP, and RMIL method falls between our new method and FR method.Although MPRP method is robust, it cannot solve all the test problems.

Conclusions
In this paper, we present a new algorithm for solving unconstrained optimization problems.It is a three term CG method that utilized the modified PRP formula under Wolfe line search.The idea is to give another variant of PRP method under Wolfe line search method which stratified the sufficient descent, trust region and global convergent properties.Various standard test problems were used to generate the numerical results.The outcome shows that our new method is efficient, reliable and effective.And can be used in place of the existence one.In future we intend to test the new method using different line search procedure.
) improve theZhang (2007) method by constructing a new modified PRP parameter namely MPRP.The performance is quite impressive.The new method in addition to the sufficient descent condition also satisfied the trust region properties, which to some extent help in making the convergence proof easier.Other researchers likeAndrei (2013), Y Narushima et al. (2011), L Zhang et al. (2006), Cheng (2007), M Al-Baali et al. (2015), Babaie-Kafaki and Ghanbari (2014), and Sun and Liu (2015) have also developed a new and effective methods using different approaches.One interesting thing is that, most of these methods convergences under Armijo or modified Armijo line search but failed under Wolfe line search.

Theorem 3. 2 .
Suppose assumption 1 holds, consider Algorithm 2.1 with the step size satisfying the standard Wolfe condition and the function  is uniformly convex, then

Figure 1 .
Figure 1.Performance Profile Based on the Number of Iteration Figure2.Performance profile Based on CPU Time