Logistic Regression

Logistic regression is different from linear regression in the sense that the outcome is discrete values rather than continuous number. Using the prediction of diabetics as an example, there are three measurements – weight, height, blood pressure one may have, and will use these three values to predict if the object is diabetic or not (discrete, binary outcome).

To put in math language, see below, the dependable value is logit of the odds being diagnosed diabetics versus that of not diabetics, it is regressionable to a bunch of factors from X1, X2, … Xj.

To understand Newton’s method for logistic regression, we need to first revisit some basic notions in Differential Calculus. There are algebra meaning and geometry meaning of differentiation: algebra meaning answers how is it computed, it is an average rate change, delta y divided by delta x, if x is time, it is the average speed, and when delta x is extremely tiny, it’s the instantaneous speed at that point of time. The geometry meaning is more pervasively and intuitively accepted as tangible line over a curve, and it’s the slope value.

Taylor’s formuala to find the most approximate polinomial equation for a function f(x):

Gradient is matrix of partial derivatives on all dimensions
Jacobian matrix of 1st order partial derivatives
Hessian is the matrix of 2nd order partial derivatives

Newton-Raphson Method, It is a method for finding successively better approximation to the roots of a real-valued function. It can be used to find the roots, or to find the minima. Code from Author: Daniel Homola, Licence: BSD 3-clause:

root

from scipy.optimize import newton
from sklearn.utils.testing import assert_almost_equal

def f(x):
return 6x5-5x4-4x3+3x2
def df(x):
return 30x4-20x3-12x2+6x

def dx(f, x):
return abs(0-f(x))

def newtons_method(f, df, x0, e, print_res=False):
delta = dx(f, x0)
while delta > e:
x0 = x0 – f(x0)/df(x0)
delta = dx(f, x0)
if print_res:
print ‘Root is at: ‘, x0
print ‘f(x) at root is: ‘, f(x0)
return x0

def test_with_scipy(f, df, x0s, e):
for x0 in x0s:
my_newton = newtons_method(f, df, x0, e)
scipy_newton = newton(f, x0, df, tol=e)
assert_almost_equal(my_newton, scipy_newton, decimal=5)
print ‘Tests passed.’

if name == ‘main‘:
# run test
x0s = [0, .5, 1]
test_with_scipy(f, df, x0s, 1e-5)

for x0 in x0s:
    newtons_method(f, df, x0, 1e-10, True)

optimization

“””
Newton’s method
Author: Daniel Homola
Licence: BSD 3-clause
“””

from scipy.optimize import newton
from sklearn.utils.testing import assert_almost_equal

def f(x):
return 6x5-5x4-4x3+3x2
def df(x):
return 30x4-20x3-12x2+6x

def dx(f, x):
return abs(0-f(x))

def newtons_method(f, df, x0, e, print_res=False):
delta = dx(f, x0)
while delta > e:
x0 = x0 – f(x0)/df(x0)
delta = dx(f, x0)
if print_res:
print ‘Root is at: ‘, x0
print ‘f(x) at root is: ‘, f(x0)
return x0

def test_with_scipy(f, df, x0s, e):
for x0 in x0s:
my_newton = newtons_method(f, df, x0, e)
scipy_newton = newton(f, x0, df, tol=e)
assert_almost_equal(my_newton, scipy_newton, decimal=5)
print ‘Tests passed.’

if name == ‘main‘:
# run test
x0s = [0, .5, 1]
test_with_scipy(f, df, x0s, 1e-5)

for x0 in x0s:
    newtons_method(f, df, x0, 1e-10, True)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.