Logistic regression is different from linear regression in the sense that the outcome is discrete values rather than continuous number. Using the prediction of diabetics as an example, there are three measurements – weight, height, blood pressure one may have, and will use these three values to predict if the object is diabetic or not (discrete, binary outcome).

To put in math language, see below, the dependable value is logit of the odds being diagnosed diabetics versus that of not diabetics, it is regressionable to a bunch of factors from X1, X2, … Xj.

To understand Newton’s method for logistic regression, we need to first revisit some basic notions in Differential Calculus. There are algebra meaning and geometry meaning of differentiation: algebra meaning answers how is it computed, it is an average rate change, delta y divided by delta x, if x is time, it is the average speed, and when delta x is extremely tiny, it’s the instantaneous speed at that point of time. The geometry meaning is more pervasively and intuitively accepted as tangible line over a curve, and it’s the slope value.

Taylor’s formuala to find the most approximate polinomial equation for a function f(x):

Gradient is matrix of partial derivatives on all dimensions

Jacobian matrix of 1st order partial derivatives

Hessian is the matrix of 2nd order partial derivatives

Newton-Raphson Method, It is a method for finding successively better approximation to the roots of a real-valued function. It can be used to find the roots, or to find the minima. Code from Author: Daniel Homola, Licence: BSD 3-clause:

# root

from scipy.optimize import newton

from sklearn.utils.testing import assert_almost_equal

def f(x):

return 6*x 5-5*

**x**4-4

*x*

**3+3****x**2

def df(x):

return 30

*x*

**4-20****x**3-12

*x*

*2+6*x

def dx(f, x):

return abs(0-f(x))

def newtons_method(f, df, x0, e, print_res=False):

delta = dx(f, x0)

while delta > e:

x0 = x0 – f(x0)/df(x0)

delta = dx(f, x0)

if print_res:

print ‘Root is at: ‘, x0

print ‘f(x) at root is: ‘, f(x0)

return x0

def test_with_scipy(f, df, x0s, e):

for x0 in x0s:

my_newton = newtons_method(f, df, x0, e)

scipy_newton = newton(f, x0, df, tol=e)

assert_almost_equal(my_newton, scipy_newton, decimal=5)

print ‘Tests passed.’

if **name** == ‘**main**‘:

# run test

x0s = [0, .5, 1]

test_with_scipy(f, df, x0s, 1e-5)

for x0 in x0s: newtons_method(f, df, x0, 1e-10, True)

# optimization

“””

Newton’s method

Author: Daniel Homola

Licence: BSD 3-clause

“””

from scipy.optimize import newton

from sklearn.utils.testing import assert_almost_equal

def f(x):

return 6*x 5-5*

**x**4-4

*x*

**3+3****x**2

def df(x):

return 30

*x*

**4-20****x**3-12

*x*

*2+6*x

def dx(f, x):

return abs(0-f(x))

def newtons_method(f, df, x0, e, print_res=False):

delta = dx(f, x0)

while delta > e:

x0 = x0 – f(x0)/df(x0)

delta = dx(f, x0)

if print_res:

print ‘Root is at: ‘, x0

print ‘f(x) at root is: ‘, f(x0)

return x0

def test_with_scipy(f, df, x0s, e):

for x0 in x0s:

my_newton = newtons_method(f, df, x0, e)

scipy_newton = newton(f, x0, df, tol=e)

assert_almost_equal(my_newton, scipy_newton, decimal=5)

print ‘Tests passed.’

if **name** == ‘**main**‘:

# run test

x0s = [0, .5, 1]

test_with_scipy(f, df, x0s, 1e-5)

for x0 in x0s: newtons_method(f, df, x0, 1e-10, True)