Logistic regression is different from linear regression in the sense that the outcome is discrete values rather than continuous number. Using the prediction of diabetics as an example, there are three measurements – weight, height, blood pressure one may have, and will use these three values to predict if the object is diabetic or not (discrete, binary outcome).
To put in math language, see below, the dependable value is logit of the odds being diagnosed diabetics versus that of not diabetics, it is regressionable to a bunch of factors from X1, X2, … Xj.
To understand Newton’s method for logistic regression, we need to first revisit some basic notions in Differential Calculus. There are algebra meaning and geometry meaning of differentiation: algebra meaning answers how is it computed, it is an average rate change, delta y divided by delta x, if x is time, it is the average speed, and when delta x is extremely tiny, it’s the instantaneous speed at that point of time. The geometry meaning is more pervasively and intuitively accepted as tangible line over a curve, and it’s the slope value.
Taylor’s formuala to find the most approximate polinomial equation for a function f(x):
Gradient is matrix of partial derivatives on all dimensions
Jacobian matrix of 1st order partial derivatives
Hessian is the matrix of 2nd order partial derivatives
Newton-Raphson Method, It is a method for finding successively better approximation to the roots of a real-valued function. It can be used to find the roots, or to find the minima. Code from Author: Daniel Homola, Licence: BSD 3-clause:
root
from scipy.optimize import newton
from sklearn.utils.testing import assert_almost_equal
def f(x):
return 6x5-5x4-4x3+3x2
def df(x):
return 30x4-20x3-12x2+6x
def dx(f, x):
return abs(0-f(x))
def newtons_method(f, df, x0, e, print_res=False):
delta = dx(f, x0)
while delta > e:
x0 = x0 – f(x0)/df(x0)
delta = dx(f, x0)
if print_res:
print ‘Root is at: ‘, x0
print ‘f(x) at root is: ‘, f(x0)
return x0
def test_with_scipy(f, df, x0s, e):
for x0 in x0s:
my_newton = newtons_method(f, df, x0, e)
scipy_newton = newton(f, x0, df, tol=e)
assert_almost_equal(my_newton, scipy_newton, decimal=5)
print ‘Tests passed.’
if name == ‘main‘:
# run test
x0s = [0, .5, 1]
test_with_scipy(f, df, x0s, 1e-5)
for x0 in x0s:
newtons_method(f, df, x0, 1e-10, True)
optimization
“””
Newton’s method
Author: Daniel Homola
Licence: BSD 3-clause
“””
from scipy.optimize import newton
from sklearn.utils.testing import assert_almost_equal
def f(x):
return 6x5-5x4-4x3+3x2
def df(x):
return 30x4-20x3-12x2+6x
def dx(f, x):
return abs(0-f(x))
def newtons_method(f, df, x0, e, print_res=False):
delta = dx(f, x0)
while delta > e:
x0 = x0 – f(x0)/df(x0)
delta = dx(f, x0)
if print_res:
print ‘Root is at: ‘, x0
print ‘f(x) at root is: ‘, f(x0)
return x0
def test_with_scipy(f, df, x0s, e):
for x0 in x0s:
my_newton = newtons_method(f, df, x0, e)
scipy_newton = newton(f, x0, df, tol=e)
assert_almost_equal(my_newton, scipy_newton, decimal=5)
print ‘Tests passed.’
if name == ‘main‘:
# run test
x0s = [0, .5, 1]
test_with_scipy(f, df, x0s, 1e-5)
for x0 in x0s:
newtons_method(f, df, x0, 1e-10, True)
Naixian Zhang 