ECE 901: Statistical Learning Theory Prerequisites:Background in graduate level applied mathematics, probability, and statistics
<>Instructor:
Robert Nowak
E-mail: nowak@engr.wisc.edu
Web: http://www.ece.wisc.edu/~nowak/
Phone: 608 265 3914
3627 Engineering Hall
Office Hours: TBA
Lectures:
Spring 2006
Time/Place: 11:00-12:15pm Tuesday and Thursday
Course Format:
The course will consist of 15-20 introductory lectures,
followed by readings and discussion of recent developments
Lectures:
Lecture 1 (pdf) Elements of Statistical Data Analysis
Lecture 2 (pdf) Introduction to Classification and Regression
Lecture 3 (pdf) Complexity and Regularization
Lecture 4 (pdf) Denoising in Smooth Function Spaces
Lecture 5 (pdf) The Histogram Classification Rule
Lecture 6 (pdf) PAC Learning
Lecture 7 (pdf) Agnostic Learning and Hoeffding's Inequality
Lecture 8 (pdf) Error Bounds for Classification
Lecture 9 (pdf) Error Bounds for Countably Infinite Classes
Lecture 10 (pdf) Complexity Regularization
Lecture 11 (pdf) Classification Trees
Lecture 12 (pdf) Complexity Regularization for Regression
Lecture 13 (pdf) Maximum Likelihood Estimation
Lecture 14 (pdf) Maximum Likelihood Estimation and Complexity Regularization
Lecture 15 (pdf) Denoising II: Adapting to Unknown Smoothness
Lecture 16 (pdf) Nonlinear Approximation and Wavelet Analysis
Lecture 17 (pdf) Denoising III: Spatial Adaptivity
Lecture 18 (pdf) Introduction to Vapnik-Chevronenkis Theory
Lecture 19 (pdf) The Vapnik-Chevronenkis Inequality
Lecture 20 (pdf) Applications of the Vapnik-Chevronenkis Inequality
Homework Problems:
Homework 1 (pdf), rob.mat (right-click browser and "Save as"
Homework 2 (pdf)
Homework 3 (pdf)
Homework 4 (pdf)
Homework 5 (pdf)
Homework 6 (pdf)
Readings:
A Tutorial on Support Vector Machines for Pattern Recognition(pdf) by C. Burges
An Introduction to Kernel Based Learning Algorithms(pdf) by K-R. Muller, S. Mika, G. Ratsch, K. Tsdua, and B. Scholkopf
The Boosting Approach to Machine Learning: An Overview(ps) by R. Schapire
Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods(ps) by R. Schapire, Y. Freund, P. Bartlett, and W. Lee
Convexity, Classification, and Risk Bounds(pdf) by P. Bartlett, M. Jordan, and J. McAuliffe
Model Selection and Error Estimation(ps) by P. Bartlett, S. Boucheron, and G. Lugosi
Stability and Generalization(pdf) by O. Bousquet and A. Elisseeff
Textbooks and References:
A textbook will not be followed in this course. A collection of
notes, relevant papers and materials will be prepared and distributed.
Textbooks recommended for further reading are listed below.
A probabilistic theory of pattern recognition, Devroye, Gyorfi, Lugosi, Springer
Nonparameteric Estimation Theory, Iain Johnstone, unpublished monograph
The Elements of Statistical Learning, Hastie, et al, Springer
An introduction to support vector machines, Cristianini and Shawe-Taylor, Cambridge Press
Combinatorial methods in density estimation, Devroye and Lugosi, Springer
Statistical Learning Theory, Vapnik, Wiley
An Introduction to Computational Learning Theory, Kearns and Vazirani, MIT Press
Empirical Processes in M-Estimation, van de Geer, Cambridge PressGrading and Evaluation:
Grades will be based on course participation, projects, and paper presentations.