Line Search Methods

Line Search Methods
Consider the following unconstrained optimization problem
min?? f(x)? (1)
x?R
Any optimization algorithm starts by an initial point x_0 and performs series of iterations to reach the optimal point x^*. At any k_th iteration the next point is given by the sum of old point and the direction in which to search of a next point multiplied by how far to go in that direction. Thus
? x?_(k+1)=x_k+?_k d_k, (2)
where, d_k is the search direction and ?_k is a positive scalar determining how far to go in that direction, it is called the step length.
Additionally, the new point must be such that the function value (the function which we are optimizing) at that point should be less than or equal to the previous point.
This is quite obvious because if we are moving in a different direction where the function value is increasing we are not really moving towards the minimum. Thus