ECE901 Spring 2007 Statistical Learning Theory Instructor: R. Nowak

Lecture 3: Introduction to Complexity Regularization

We ended the previous lecture with a brief discussion of overfitting. Recall that, given a set of n data points, D_{n}, and a space of functions (or models) F, our goal in solving the learning from data problem is to choose a function f̂_{n} ∈ F which minimizes the expected risk E [R(f̂_{n})], where the expectation is being taken over the distribution P_{XY} on the data points D_{n}. One approach to avoiding overfitting is to restrict F to some subset of all measurable function. To gauge the performance of a given f in this case, we examine the difference between the expected risk of f and the Bayes’ risk (called the excess risk).

E [R(f̂_{n})] − R^{∗} = \underbrace{(E[R(f̂_{n})] − inf_{f∈F} R(f))}_{estimation error} + \underbrace{( inf_{f∈F} R(f) − R^{∗})}_{approximation error}

The approximation error term quantifies the performance hit incurred by imposing restrictions on F. The estimation error term is due to the randomness of the training data, and it expresses how well the chosen function f̂_{n} will perform in relation to the best possible f in the class F. This decomposition into stochastic and approximation errors is similar to the bias-variance tradeoff which arises in classical estimation theory. The approximation error is like a bias squared term, and the estimation error is like a variance term. By allowing the space F to be large^{1} we can make the approximation error as small as we want at the cost of incurring a large estimation error. On the other hand, if F is very small then the approximation error will be large, but wthe estimation error may be very small. This tradeoff is illustrated in Figure 1.

Figure 1: Illustration of tradeoff between estimation and approximation errors as a function of the size (complexity) of the F.

1When we say F is large, we mean that |F|, the number of elements in F, is large.

Why is this the case? We do not know the true distribution P_{XY} on the data, so instead of minimizing the expected risk of we design a predictor by minimizing the empirical risk:

f̂_{n} = arg min_{f∈F} R̂_{n}(f),
R̂_{n}(f) = \frac{1}{n} ∑_{i=1}^{n} ℓ(f(X_{i}), Y_{i}).

If F is very large then R̂_{n}(f) can be made arbitrarily small and the resulting f̂_{n} can “overfit” to the data since R̂_{n}(f) is not be a good estimator of the true risk R(f̂_{n}).

Figure 2: Illustration of empirical risk and the problem of overfitting to the data.

The behavior of the true and empirical risks, as a function of the size (or complexity) of the space F, is illustrated in Figure 2. Unfortunately, we can’t easily determine whether we are over or underfitting just by looking at the empirical risk.

1 Strategies To Avoid Overfitting

Picking

f̂_{n} = arg min_{f∈F} R̂_{n}(f)

is problematic if F is large. We will examine two general approaches to dealing with this problem:

1. Restrict the size or dimension of F (e.g., restrict F to the set of all lines, or polynomials with maximum degree d). This effectively places an upper bound on the estimation error, but in general it also places a lower bound on the approximation error.

2. Modify the empirical risk criterion to include an extra cost associated with each model (e.g., higher cost for more complex models):

f̂_{n} = arg min_{f∈F} {R̂_{n}(f) + C(f)} .

The cost is designed to mimic the behavior of the estimation error so that the model selection procedure avoids models with a estimation error. Roughly this can be interpreted as trying to balance the tradeoff illustrated in Figure 1. Procedures of this type are often called complexity penalization methods.

Example 1 Revisit the polynomial regression example (Lecture 2, Ex. 4), and incorporate a penalty term C(f) which is proportional to the degree of f, or the derivative of f. In essence, this approach penalizes for functions which are too “wiggly”, with the intuition being that the true function is probably smooth so a function which is very wiggly will overfit the data.

How do we decide how to restrict or penalize the empirical risk minimization process? Approaches which have appeared in the literature include the following.

1.1 Method of Sieves (Grenander, 1981)

Perhaps the simplest approach is to try to limit the size of F in a way that depends on the number of training data n. The more data we have, the more complex the space of models we can entertain. Let the class of candidate functions grow with n. That is, take

F_{1}, F_{2}, …, F_{n}, …

where |F_{i}| grows as i → ∞. In other words, consider a sequence of spaces with increasing complexity or degrees of freedom depending on the number of training data samples, n.

Given samples {X_{i}, Y_{i}}_{i=1}^{n} i.i.d. distributed according to P_{XY}, select f ∈ F_{n} to minimize the empirical risk

f̂_{n} = arg min_{f∈F_{n}} R̂_{n}(f).

In the next lecture we will consider an example using the method of sieves. The basic idea is to design the sequence of model spaces in such a way that the excess risk decays to zero as n → ∞. This sort of idea has been around for decades, but Grenander’s method of sieves is often cited as a nice formalization of the idea: Grenander (1981), Abstract Inference, Wiley, New York.

1.2 Complexity Penalization Methods

1.2.1 Bayesian Methods (Bayes, 1764)

In certain cases, the empirical risk happens to be a (log) likelihood function, and one can then interpret the cost C(f) as reflecting prior knowledge about which models are more or less likely. In this case, e^{−C(f)} is like a prior probability distribution on the space F. The cost C(f) is large if f is highly improbable, and C(f) is small if f is highly probable.

Alternatively, if we restrict F to be small, and denote the space of all measurable functions as 𝔽 = F ∪F^{c}, then it is essentially as if we have placed a uniform prior over all functions in F, and zero prior probability on the functions in F^{c}.

1.2.2 Description Length Methods (Rissanen, 1978)

Description length methods represent each f with a string of bits. More complicated functions require more bits to represent. Accordingly, we can then set the cost c(f) proportional to the number of bits needed to describe f (the description length). This results in what is known as the minimum description length (MDL) approach where the minimum description length is given by

min_{f∈F} {R̂_{n}(f) + C(f)} .

In the Bayesian setting, p(f) ∝ e^{−C(f)} can be interpreted as a prior probability density on F, with more complex models being less probable and simpler models being more probable. In that sense, both the Bayesian and MDL approaches have a similar spirit.

1.2.3 Vapnik-Cervonenkis Dimension (Vapnik & Cervonenkis, 1971)

The Vapnik-Cervonenkis (VC) dimension measures the complexity of a class F relative to a random sample of n training data. For example, take F to be all linear classifiers in 2-dimensional feature space. Clearly, the space of linear classifiers is infinite (there are an infinite number of lines which can be drawn in the plane). However, many of these linear classifiers would assign the same labels to the training data.

The number of unique labellings of the training data that can be achieved with linear classifiers is, in fact, finite. A line can be defined by picking any pair of training points, as illustrated in Figure 3(a). Two classifiers can be defined from each such line: one that outputs a label “1” for everything on or above the line, and another that outputs “0” for everything on or above. There exist \binom{n}{2} such pairs of training points, and these define all possible unique labellings of the training data. Therefore, there are at most 2\binom{n}{2} unique linear classifiers for any random set of n 2-dimensional features (the factor of 2 is due to the fact that for each linear classifier there are 2 possible assignments of the labelling).

(a)  (b)

Figure 3: Fitting a linear classifier to 2-dimensional data. There are an infinite number of such classifiers.

(a) We can generate a linear classifier by choosing two data points, drawing a line with both points on one side, and declaring all points on or above the line to be “+1” (or “−1”) and all points below the line to be “−1” (or “+1”). From this perspective, we see that the two linear classifiers depicted in (b) are equivalent for this set of data points, and hence relative to the set of n training data there are only on the order of n^{2} unique linear classifiers.

Thus, instead of infinitely many linear classifiers, we realize that as far as a random sample of n training data is concerned, there are at most

2\binom{n}{2} = \frac{2n!}{(n − 2)!2!}
= n(n − 1)

unique linear classifiers. That is, using linear classification rules, there are at most n(n − 1) ≈ n^{2} unique label assignments for n data points. If we like, we can encode each possibility with log_{2} n(n − 1) ≈ 2 log_{2} n bits. In d dimensions there are 2\binom{n}{d} hyperplane classification rules which can be encoded in roughly d log_{2} n bits. Roughly speaking, the number of bits required for encoding each model is the VC dimension. The remarkable aspect of the VC dimension is that it is often finite even when F is infinite (as in this example).

If X has d dimensions in total, we might consider linear classifiers based on 1, 2, …, d features at a time. Lower dimensional hyperplanes are less complex than higher dimensional ones. Suppose we set

F_{1} = linear classifiers using 1 feature
F_{2} = linear classifiers using 2 features
…  and so on

These spaces have increasing VC dimensions, and we can try to balance the empirical risk and a cost function depending on the VC dimension. Such procedures are often referred to as Structural Risk Minimization. This gives you a glimpse of what the VC dimension is all about. In future lectures we will revisit this topic in greater detail.

1.3 Hold-out Methods

The basic idea of “hold-out” methods is to split the n samples D ≡ {X_{i}, Y_{i}}_{i=1}^{n} into a training set, D_{T}, and a test set, D_{V}.

D_{T} = {X_{i}, Y_{i}}_{i=1}^{m},  D_{V} = {X_{i}, Y_{i}}_{i=m+1}^{n}

Now, suppose we have a collection of different model spaces {F_{λ}} indexed by λ ∈ Λ (e.g., F_{λ} is the set of polynomials of degree d, with λ = d), or suppose that we have a collection of complexity penalization criteria L_{λ}(f) indexed by λ (e.g.,let L_{λ}(f) = R̂(f) + λc(f), with λ ∈ R^{+}). We can obtain candidate solutions using the training set as follows. Define

R̂_{m}(f) = ∑_{i=1}^{m} ℓ(f(X_{i}), Y_{i})

and take

f̂_{λ} = arg min_{f∈F_{λ}} R̂_{m}(f)

or

f̂_{λ} = arg min_{f∈F} {R̂_{m}(f) + λc(f)}

This provides us with a set of candidate solutions {f̂_{λ}}. Then we can define the hold-out error estimate using the test set:

R̂_{V} (f) = \frac{1}{n − m + 1} ∑_{i=m+1}^{n} ℓ(f(X_{i}), Y_{i}),

and select the “best” model to be f̂ = f̂λ̂ where

λ̂ = arg min_{λ} R̂_{V} \binom{}{f̂_{λ}}

This type of procedure has many nice theoretical guarantees, provided both the training and test set grow with n.

1.3.1 Leaving-one-out Cross-Validation (Wahba, 1971)

A very popular hold-out method is the so call “leaving-one-out cross-validation” studied in depth by Grace Wahba (UW-Madison, Statistics). For each λ we compute

f̂_{λ}^{(k)} = arg min_{f∈F} \frac{1}{n} ∑_{i≠k^{i=1}}^{n} ℓ(f(X_{i}), Y_{i}) + λC(f)

or

f̂_{λ}^{(k)} = arg min_{f∈F_{λ}} \frac{1}{n} ∑_{i≠k^{i=1}}^{n} ℓ(f(X_{i}), Y_{i}).

Then we have cross-validation function

V (λ) = \frac{1}{n} ∑_{k=1}^{n} ℓ(f_{λ}^{(k)}(X_{k}), Y_{k})
λ^{∗} = arg min_{λ} V (λ).

2 Summary

To summarize, this lecture gave a brief and incomplete survey of different methods for dealing with the issues of overfitting and model selection. Given a set of training data, D_{n} = {X_{i}, Y_{i}}_{i=1}^{n}, our overall goal is to find

f^{∗} = arg min_{f∈F} R(f)

from some collection of functions, F. Because we do not know the true distribution P_{XY} underlying the data points D_{n}, it is difficult to get an exact handle on the risk, R(f). If we only focus on minimizing the empirical risk R̂(f) we end up overfitting to the training data. Two general approaches were presented.

1. In the first approach we consider an indexed collection of spaces {F_{λ}}_{λ∈Λ} such that the complexity of

F_{λ} increases as λ increases, and

lim_{λ→∞} F_{λ} = F.

A solution is given by

f̂_{λ^{∗}} = arg min_{f∈F_{λ^{∗}}} R̂_{n}(f)

where either λ^{∗} is a function which increases with n,

λ^{∗} = λ(n),

or λ^{∗} is chosen by hold-out validation.

2. The alternative approach is to incorporate a penalty term into the risk minimization problem formulation. Here we consider an indexed collection of penalties {C_{λ}}_{λ∈Λ} satisfying the following properties:

(a) C_{λ} : F → R^{+};

(b) For each f ∈ F and λ_{1} < λ_{2} we have C_{λ_{1}}(f) ≤ C_{λ_{2}}(f);

(c) There exists λ_{0} ∈ Λ such that C_{λ_{0}}(f) = 0 for all f ∈ F.

In this formulation we find a solution

f̂_{λ^{∗}} = arg min_{f∈F} R̂_{n}(f) + C_{λ^{∗}}(f),

where either λ^{∗} = λ(n), a function growing the number of data samples n, or λ^{∗} is selected by hold-out validation.

Lecture 3: Introduction to Complexity Regularization 7

3 Consistency

If an estimator or classifier f̂_{λ^{∗}} satisfies

E [R(f̂_{λ^{∗}})] → inf_{f∈F} R(f) as n → ∞,

then we say that f̂_{λ^{∗}} is F-consistent with respect to the risk R. When the context is clear, we will simply say that f̂ is consistent.