Understanding classifier performance

Dear Dr Ballings,

In Model Evaluation of the book, in order to compute bias and variance of the model, you showed the steps in order to compute equation "xi"
(Sorry, you did not provide equation number). I have couple of questions about the first three steps.

1. Make K bootstrap samples (keep some data out-of-bag, i.e., set some data aside, e.g., 20% )
We are choosing a bootstrap samples with replacement, from 1 to K.

2. Learn a model on each bootstrap sample
We are running an algorithm on EACH of these full bootstrap samples (not on 80% of the bootstrap sample, since we keep some data out-of-bag,
i.e., set some data aside, e.g., 20% ) and having K different hypotheses.

3. For each model predict \hat{y} on out-of-bag data. We now have, for each instance, one y and \hat{y}_{1}, \hat{y}_{2}, ..., \hat{y}_{K}
I especially did not understand that part. What I understood, if we have one y of an instance and many predictions of that instance, that
means that we need to assure that this instance should be kept in out-of-bag of all the bootstrap samples (from 1 to K). is this right or

4. Compute \bar{y} = \sum_{j=1}^{K} \hat{y}_{i}
I guess there is a typo here, the indice 'i' should be 'j', right?


Answers and follow-up questions

Answer or follow-up question 1

Dear Student,

Thanks for pointing out that there are no equation numbers. I'll add these going forward.

1. Correct: we are making K bootstrap samples. Bootstrap samples are always sampled with replacement. Before we start sampling we set some
data apart as out-of-bag (OOB). Say we keep 20% as OOB. This OOB will be the same for all bootstraps. Specifically: first split the data up
in train (80%) and test (20%). Second, create K bootstrap samples based on train.

2. We are estimating K models (=hypotheses) (one model on each bootstrap sample).

3. Right. See 1.

4. Yes and no. I will clarify this in the notes. This is indeed not as clear as it could be.

Michel Ballings

Sign in to be able to add an answer or mark this question as resolved.