\documentclass{article}
\usepackage{fullpage, epsf, setspace, latexsym, cite}
\usepackage{xspace, amsmath, amssymb, subfigure, hyperref}
\usepackage{graphicx}

\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{cor}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{ex}{Example}
\newtheorem{defn}{Definition}

\def\QED{\mbox{\rule[0pt]{1.5ex}{1.5ex}}}
\def\endproof{\hspace*{\fill}~\QED\par\endtrivlist\unskip}
\newenvironment{pf}{\begin{verse}{\bf Proof:}\rm}{\endproof \end{verse}}
\newcommand{\1}[1]{\ensuremath{{\bf 1}_{\{#1\}}}}



\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf ECE901 Spring 2007 Statistical Learning Theory \hfill Instructor: R. Nowak} }
       \vspace{6mm}
       \hbox to 6.28in { {\Large \hfill #1  \hfill} }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{#1}{#1}
   \setlength{\headsep}{10mm}
   \vspace*{4mm}
}

%notation

\def\X{\ensuremath{{\mathcal X}}\xspace}
\def\Y{\ensuremath{{\mathcal Y}}\xspace}
\def\F{\ensuremath{{\mathcal F}}\xspace}
\def\R{\ensuremath{{\bf R}}\xspace}
\def\E{{E}}
\def\sign{\ensuremath{\mbox{sign}}}
\def\ie{{\em i.e.,}}
\def\eg{{\em e.g.,}}

\begin{document}

\lecture{Lecture 3: Introduction to Complexity Regularization}

\section{Competing Goals: The Bias-Variance Tradeoff}

We ended the previous lecture with a brief discussion of overfitting.
Recall that, given a set of $n$ data points, $D_n$, and a space of
functions (or {\em models}) \F, our goal in solving the learning from
data problem is to choose a function $\widehat{f}_n \in \F$ which
minimizes the expected risk $\E\left[R(\widehat{f}_n)\right]$, where
the expectation is being taken over the distribution $P_{XY}$ on the
data points $D_n$.  One approach to avoiding overfitting is to
restrict $\F$ to some subset of all measurable function.  To gauge the
performance of a given $f$ in this case, we examine the difference
between the expected risk of $f$ and the Bayes' risk (called the {\em
  excess risk}).
\begin{eqnarray*}
\E\left[R(\widehat{f}_n)\right] - R^* &=& \underbrace{\left(\E[R(\widehat{f}_n)] -
\inf_{f \in \F} R(f) \right)}_{\mbox{estimation error}} +
\underbrace{\left(\inf_{f \in \F} R(f) -
R^*\right)}_{\mbox{approximation error}}
\end{eqnarray*}
The {\it approximation error} term quantifies the performance hit
incurred by imposing restrictions on \F. The {\it estimation error}
term is due to the randomness of the training data, and it expresses
how well the chosen function $\widehat{f}_n$ will perform in relation
to the best possible $f$ in the class $\F$.  This decomposition into
stochastic and approximation errors is similar to the bias-variance
tradeoff which arises in classical estimation theory.  The
approximation error is like a bias squared term, and the estimation
error is like a variance term.  By allowing the space \F to be
large\footnote{When we say \F is large, we mean that $|\F|$, the
  number of elements in \F, is large.} we can make the approximation
error as small as we want at the cost of incurring a large estimation
error.  On the other hand, if \F is very small then the approximation
error will be large, but wthe estimation error may be very small.
This tradeoff is illustrated in Figure~\ref{fig0}. \\ \ \\ \vspace{-2.3in}

\begin{figure}[h]
\centering %
\includegraphics[width=5in]{bv.pdf}
\vspace{-1.6in}
\caption{Illustration of tradeoff between estimation and approximation
errors as a function of the size (complexity) of the \F.}
\label{fig0}
\end{figure}
\newpage

Why is this the case?  We do not know the true distribution $P_{XY}$
on the data, so instead of minimizing the expected risk of we design a
predictor by minimizing the empirical risk:
\begin{eqnarray*}
\widehat{f}_n &=& \arg\min_{f \in \F} \widehat{R}_n(f), \\
\widehat{R}_n(f) &=& \frac{1}{n} \sum_{i=1}^n \ell(f(X_i), Y_i).
\end{eqnarray*}
If \F is very large then $\widehat{R}_n(f)$ can be made
arbitrarily small and the resulting $\widehat{f}_n$ can ``overfit'' to the
data since $\widehat{R}_n(f)$ is not be a good estimator of the true risk
$R(\widehat{f}_n)$.

\begin{figure}[ht]
\centering %
\includegraphics[width=6in]{fig1}
\caption{Illustration of empirical risk and the problem of overfitting to the data.}
\label{fig1}
\end{figure}

The behavior of the true and empirical risks, as a function of the
size (or {\em complexity}) of the space $\F$, is illustrated in
Figure~\ref{fig1}.  Unfortunately, we can't easily determine whether
we are over or underfitting just by looking at the empirical risk.

\section{Strategies To Avoid Overfitting}
Picking
\begin{eqnarray*}
\widehat{f}_n = \arg\min_{f \in \F} \widehat{R}_n(f)
\end{eqnarray*}
is problematic if \F is large.  We will examine two general
approaches to dealing with this problem:
\begin{enumerate}
\item Restrict the size or dimension of \F (e.g., restrict \F to the
  set of all lines, or polynomials with maximum degree $d$).  This
  effectively places an upper bound on the estimation error, but in
  general it also places a lower bound on the approximation error.

\item Modify the empirical risk criterion to include an extra cost
associated with each model (e.g.,  higher cost for more complex
models):
\begin{eqnarray*}
\widehat{f}_n &=& \arg\min_{f \in \F} \left\{\widehat{R}_n(f) +
C(f)\right\}.
\end{eqnarray*}
The cost is designed to mimic the behavior of the estimation error so
that the model selection procedure avoids models with a estimation error.
Roughly this can be interpreted as trying to balance the tradeoff illustrated
in Figure~\ref{fig0}.  Procedures of this type are often called complexity
penalization methods.
\end{enumerate}

\begin{ex}
Revisit the polynomial regression example (Lecture~2, Ex.~4), and
incorporate a penalty term $C(f)$ which is proportional to the
degree of $f$, or the derivative of $f$.  In essence, this
approach penalizes for functions which are too ``wiggly'', with the
intuition being that the true function is probably smooth so a
function which is very wiggly will overfit the data.
\end{ex}

How do we decide how to restrict or penalize the empirical risk
minimization process?  Approaches which have appeared in the
literature include the following.

\subsection{Method of Sieves (Grenander, 1981)}

Perhaps the simplest approach is to try to limit the size of \F
in a way that depends on the number of training data $n$.  The more
data we have, the more complex the space of models we can entertain.
Let the class of candidate functions grow with $n$.  That is, take
$$F_1, \F_2,\dots,\F_n,\dots$$
where $|\F_i|$ grows as $i\rightarrow \infty$.  In other words,
consider a sequence of spaces with increasing complexity or
degrees of freedom depending on the number of training data
samples, $n$.

Given samples $\{X_i, Y_i\}_{i=1}^n$ i.i.d. distributed according
to $P_{XY}$, select $f \in \F_n$ to minimize the empirical risk
$$\widehat{f}_n = \arg\min_{f \in \F_n} \widehat{R}_n(f).$$
In the
next lecture we will consider an example using the method of sieves.
The basic idea is to design the sequence of model spaces in such a way
that the excess risk decays to zero as $n \rightarrow \infty$.  This
sort of idea has been around for decades, but Grenander's method of
sieves is often cited as a nice formalization of the idea: Grenander
(1981), {\em Abstract Inference}, Wiley, New York.

\subsection{Complexity Penalization Methods}
\subsubsection{Bayesian Methods (Bayes, 1764)}

In certain cases, the empirical risk happens to be a (log) likelihood
function, and one can then interpret the cost $C(f)$ as reflecting
prior knowledge about which models are more or less likely.  In this
case, $e^{-C(f)}$ is like a prior probability distribution on the
space \F.  The cost $C(f)$ is large if $f$ is highly improbable, and
$C(f)$ is small if $f$ is highly probable.

Alternatively, if we restrict \F to be small, and denote the space of
all measurable functions as $\mathbb{F} = \F \cup \F^c$, then it is
essentially as if we have placed a uniform prior over all functions in
\F, and zero prior probability on the functions in $\F^c$.

\subsubsection{Description Length Methods (Rissanen, 1978)}

Description length methods represent each $f$ with a string of bits.
More complicated functions require more bits to represent.
Accordingly, we can then set the cost $c(f)$ proportional to the
number of bits needed to describe $f$ (the {\em description length}).
This results in what is known as the minimum description length (MDL)
approach where the minimum description length is given by
\begin{eqnarray*}
\min_{f \in \F} \left\{\widehat{R}_n(f) + C(f)\right\}.
\end{eqnarray*}

In the Bayesian setting, $p(f) \propto e^{-C(f)}$ can be interpreted
as a prior probability density on \F, with more complex models being
less probable and simpler models being more probable.  In that sense,
both the Bayesian and MDL approaches have a similar spirit.

\subsubsection{Vapnik-Cervonenkis Dimension (Vapnik \&
Cervonenkis, 1971)}

The Vapnik-Cervonenkis (VC) dimension measures the complexity of a
class \F relative to a random sample of $n$ training data.  For
example, take \F to be all linear classifiers in 2-dimensional
feature space. Clearly, the space of linear classifiers is
infinite (there are an infinite number of lines which can be drawn
in the plane). However, many of these linear classifiers would assign
the same labels to the training data.  

The number of unique labellings of the training data that can be
achieved with linear classifiers is, in fact, finite.  A line can be
defined by picking {\em any} pair of training points, as illustrated
in Figure~\ref{fig:vc}(a).  Two classifiers can be defined from each
such line: one that outputs a label ``1'' for everything on or above
the line, and another that outputs ``0'' for everything on or above.
There exist $n \choose{2}$ such pairs of training points, and these
define all possible unique labellings of the training data.
Therefore, there are at most $2 {n \choose{2}}$ unique linear
classifiers for any random set of $n$ 2-dimensional features (the
factor of $2$ is due to the fact that for each linear classifier there
are $2$ possible assignments of the labelling).

\begin{figure}[ht]
\centering %
\subfigure[][]{\includegraphics[width=3in]{vc2}} %
\hspace{0.2in}
\subfigure[][]{\includegraphics[width=3in]{vc3}} %
\caption{Fitting a linear classifier to 2-dimensional data.  There
are an infinite number of such classifiers.  (a) We can generate a
linear classifier by choosing two data points, drawing a line with
both points on one side, and declaring all points on or above the
line to be ``$+1$'' (or ``$-1$'') and all points below the line to
be ``$-1$'' (or ``$+1$'').  From this perspective, we see that the
two linear classifiers depicted in (b) are equivalent for this set
of data points, and hence relative to the set of $n$ training data
there are only on the order of $n^2$ unique linear classifiers.}
\label{fig:vc}
\end{figure}

Thus, instead of infinitely many linear classifiers, we realize that
as far as a random sample of $n$ training data is concerned, there are
at most
\begin{eqnarray*}
2 {n \choose{2}} &=& \frac{2
n!}{(n-2)! 2!} \\
&=& n(n-1)
\end{eqnarray*}
unique linear classifiers.  That is, using linear classification
rules, there are at most $n(n-1) \approx n^2$ unique label assignments
for $n$ data points.  If we like, we can encode each possibility with
$\log_2 n(n-1) \approx 2 \log_2 n$ bits.  In $d$ dimensions there are
$2 {n \choose{d}}$ hyperplane classification rules which can be
encoded in roughly $d \log_2 n$ bits.  Roughly speaking, the number of
bits required for encoding each model is the VC dimension.  The
remarkable aspect of the VC dimension is that it is often finite even
when $\F$ is infinite (as in this example).


If \X has $d$ dimensions in total, we might consider linear
classifiers based on $1,2,\dots,d$ features at a time.  Lower
dimensional hyperplanes are less complex than higher dimensional
ones. Suppose we set
\begin{eqnarray*}
\F_1 &=& \mbox{linear classifiers using 1 feature} \\
\F_2 &=& \mbox{linear classifiers using 2 features} \\
\dots & & \mbox{and so on }
\end{eqnarray*}
These spaces have increasing VC dimensions, and we can try to balance
the empirical risk and a cost function depending on the VC dimension.
Such procedures are often referred to as {\em Structural Risk
  Minimization}.  This gives you a glimpse of what the VC dimension is
all about.  In future lectures we will revisit this topic in greater
detail.

\subsection{Hold-out Methods}

The basic idea of ``hold-out'' methods is to split the $n$
samples $D \equiv \{X_i, Y_i\}_{i=1}^n$ into a training set, $D_T$,
and a test set, $D_V$.
\begin{eqnarray*}
D_T = \{X_i, Y_i\}_{i=1}^m, & & D_V = \{X_i, Y_i\}_{i=m+1}^n
\end{eqnarray*}
Now, suppose we have a collection of different model spaces
$\{\F_\lambda\}$ indexed by $\lambda \in \Lambda$ (e.g., $\F_\lambda$
is the set of polynomials of degree $d$, with $\lambda = d$), or
suppose that we have a collection of complexity penalization criteria
$L_\lambda(f)$ indexed by $\lambda$ (\eg let $L_\lambda(f) =
\widehat{R}(f) + \lambda c(f)$, with $\lambda \in \R^+$).  We can
obtain candidate solutions using the training set as follows. Define
\begin{eqnarray*}
\widehat{R}_m(f) &=& \sum_{i=1}^m \ell(f(X_i), Y_i)
\end{eqnarray*}
and take
\begin{eqnarray*}
\hat{f}_\lambda &=& \arg\min_{f \in {\cal F}_\lambda} \widehat{R}_m(f)
\end{eqnarray*}
or
\begin{eqnarray*}
\hat{f}_\lambda &=& \arg\min_{f \in \F} \, \left\{\widehat{R}_m(f) + \lambda
c(f)\right\}
\end{eqnarray*}
This provides us with a set of candidate solutions
$\{\hat{f}_\lambda\}$. Then we can define the hold-out error estimate using
the test set:
\begin{eqnarray*}
\widehat{R}_V(f) &=& \frac{1}{n-m+1} \sum_{i=m+1}^n \ell(f(X_i),
Y_i),
\end{eqnarray*}
and select the ``best'' model to be $\widehat{f} =
\widehat{f}_{\hat{\lambda}}$ where
\begin{eqnarray*}
\widehat{\lambda} &=& \arg\min_{\lambda}
\widehat{R}_V\left(\hat{f}_\lambda\right)
\end{eqnarray*}
This type of procedure has many nice theoretical guarantees, provided
both the training and test set grow  with $n$.

\subsubsection{Leaving-one-out Cross-Validation (Wahba, 1971)}

A very popular hold-out method is the so call ``leaving-one-out
cross-validation'' studied in depth by Grace Wahba (UW-Madison,
Statistics).  For each $\lambda$ we compute
\begin{eqnarray*}
\hat{f}_\lambda^{(k)} &=& \arg\min_{f \in \F} \frac{1}{n}
\sum_{\stackrel{i=1}{i\ne k}}^n \ell(f(X_i), Y_i) + \lambda C(f)
\end{eqnarray*}
or
\begin{eqnarray*}
\hat{f}_\lambda^{(k)} &=& \arg\min_{f \in \F_\lambda} \frac{1}{n}
\sum_{\stackrel{i=1}{i\ne k}}^n \ell(f(X_i), Y_i).
\end{eqnarray*}
Then we have cross-validation function
\begin{eqnarray*}
V(\lambda) &=& \frac{1}{n} \sum_{k=1}^n \ell(f_\lambda^{(k)}(X_k),
Y_k) \\
\lambda^* &=& \arg\min_\lambda V(\lambda).
\end{eqnarray*}

\section{Summary}
To summarize, this lecture gave a brief and incomplete survey of
different methods for dealing with the issues of overfitting and model
selection.  Given a set of training data, $D_n = \{X_i,Y_i\}_{i=1}^n$,
our overall goal is to find
$$f^* = \arg\min_{f \in \F} R(f)$$
from some collection of functions,
\F.  Because we do not know the true distribution $P_{XY}$ underlying
the data points $D_n$, it is difficult to get an exact handle on the
risk, $R(f)$.  If we only focus on minimizing the empirical risk
$\widehat{R}(f)$ we end up overfitting to the training data.  Two
general approaches were presented.
\begin{enumerate}
\item In the first approach we consider an indexed collection of
spaces $\{\F_\lambda\}_{\lambda \in \Lambda}$ such that the
complexity of $\F_\lambda$ increases as $\lambda$ increases, and
$$\lim_{\lambda \rightarrow \infty} \F_\lambda = \F.$$
A solution is given by
\begin{eqnarray*}
\widehat{f}_{\lambda^*} = \arg\min_{f \in \F_{\lambda^*}}
\widehat{R}_n(f)
\end{eqnarray*}
where either $\lambda^*$ is a function which increases
with $n$,
\begin{eqnarray*}
\lambda^* &=& \lambda(n),
\end{eqnarray*}
or $\lambda^*$ is chosen by hold-out validation.

\item The alternative approach is to incorporate a penalty term
into the risk minimization problem formulation.  Here we consider
an indexed collection of penalties $\{C_\lambda\}_{\lambda \in
\Lambda}$ satisfying the following properties:
\begin{enumerate}
\item $C_\lambda : \F \rightarrow \R^+$;

\item For each $f \in \F$ and $\lambda_1 < \lambda_2$ we have
$C_{\lambda_1}(f) \le C_{\lambda_2}(f)$;

\item There exists $\lambda_0 \in \Lambda$ such that
$C_{\lambda_0}(f) = 0$ for all $f \in \F$.
\end{enumerate}
In this formulation we find a solution
\begin{eqnarray*}
\widehat{f}_{\lambda^*} &=& \arg\min_{f \in \F} \widehat{R}_n(f) +
C_{\lambda^*}(f),
\end{eqnarray*}
where either $\lambda^* = \lambda(n)$, a function growing the
number of data samples $n$, or $\lambda^*$ is selected by hold-out
validation.
\end{enumerate}

\section{Consistency}

If an estimator or classifier $\widehat{f}_{\lambda^*}$ satisfies
$$E\left[R(\widehat{f}_{\lambda^*})\right] \rightarrow \inf_{f\in \F}
R(f) \ \ \ \ \ \ \mbox{as $n \rightarrow \infty$},$$
then we say that
$\widehat{f}_{\lambda^*}$ is $\F$-consistent with respect to the risk
$R$.  When the context is clear, we will simply say that $\widehat{f}$
is consistent.

\end{document}
