```{r setup, include=FALSE}
  knitr::opts_chunk$set(echo = TRUE)
  knitr::opts_chunk$set(dev = 'pdf')
  def.chunk.hook  <- knitr::knit_hooks$get("chunk")
  knitr::knit_hooks$set(chunk = function(x, options) {
    x <- def.chunk.hook(x, options)
    ifelse(options$size != "normalsize", paste0("\n \\", options$size,"\n\n", x, "\n\n \\normalsize"), x)
  })
```




## Last time {.t}

 - We saw that running a t-test on a small sample from a non-normal distribution may suffer in terms of its Type I Error rate
 
\vspace{2mm}

 - We learned how to obtain simulated estimates of Type I Error rates

And we said:

### Things still to come:
\vspace{-2mm}
 - \underline{Non-parametric} tests (e.g., but not limited to, what you've learned in DSC 10 and 80 where you get a p-value via simulating the null distribution) are less dependent on conditions to be valid.

\vspace{4mm}

 - What about statistical power?

 
## Non-parametric one-sample tests {.t}

::: {.block}
### Recall...
\vspace{-2mm}
The t-Test relies on $\overline{x}$ following a normal distribution in order for its test statistic $t_s$ to follow the t-Distribution under $H_0$. 
:::

Question: how might we perform a non-parametric test for $H_0: \mu = \mu_0$?

\vspace{3mm}

The main thing is that it needs to require no conditions on the distribution of values (e.g., in the UCSD students' sleep example from last time, a non-parametric test would presume that the distribution on the number of hours of sleep per night could be anything)


## Non-parametric one-sample tests {.t}
Consider:

 - In some sense, the most true that $H_0$ could possibly be is if every data value equaled $\mu_0$ (in this case, 6 hours)

\vspace{3mm}

 - But, how else might the data look and still be in line with $H_0$?

\vspace{4mm}

pollev.com/chi


## The Sign Test {.t}
One common example of a one-sample non-parametric test is known as the \underline{sign test}. How does this work?

::: {.block}
### Sign test

 - Take each data value and subtract $\mu_0^{***}$
   - If the difference is positive, assign a "+" value
   - If the difference is negative, assign a "-" value
   - If the difference is 0, ignore that value
   
\vspace{4mm}

 - Under $H_0$, there should be $50\%$ "+" and $50\%$ "-". We then calculate the probability of observing at least as many "+" or "-" as what was actually observed, under a Binomial distribution with $p=0.5$.
:::

$^{***}$Almost. See next slide for correction.


## The Sign Test {.t}
Question: Why is the sign test \underline{not} actually a test for $\mu$?

::: {.block}
### Answer: 
\vspace{-2mm}
It works by assuming that, under $H_0$: 

 - 50\% of all values are above the null value 
 - and 50\% of all values are below the null value.

But, a mean does not always actually have this property. What is the name of the thing that does?? 
:::

\vspace{5mm}
\pause
So instead of $\mu_0$, we will designate the null hypothesis value as $\tilde{\mu}_0$ to represent the null hypothesis value of the \underline{median}.

## The Sign Test {.t}
Example: recall the UCSD sleep data:

$$
3, 7, 1, 2, 2
$$


and $\tilde{\mu}_0 = 6$. So, our data for the sign test become:

$$
-, +, -, -, -
$$

pollev.com/chi

\vspace{2mm}

(Don't look at the next slide yet!)


## The Sign Test {.t}
The p-value for a sign test is thus:

$$
P(X=4) + P(X=5) + P(X=0) + P(X=1)
$$

where $X \sim Binomial(n=5, p=0.5)$. In R, this is:

```{r, size="footnotesize"}
dbinom(x=4, size=5, prob=0.5) + dbinom(x=5, size=5, prob=0.5) + 
  dbinom(x=0, size=5, prob=0.5) + dbinom(x=1, size=5, prob=0.5)
```


or equivalently,
```{r, size="footnotesize"}
pbinom(1, size=5, prob=0.5)*2 # Wait why?
```

So now we would fail to reject $H_0$. And furthermore, note that this p-value is way bigger than what we found via the t-Test last time!

## The Sign Test {.t}
Question: what is the Type I Error rate of the sign test under a variety of situations?


::: {.block}
### Last time, with the t-test at $\alpha=0.05$ and $n=5$, we saw that...
\vspace{-2mm}
 - If the data follow a Gamma distribution, the Type I Error rate was approximately 0.10 - 0.11

\vspace{2mm}

 - If the data follow a uniform distribution, the Type I Error rate was approximately 0.06 - 0.07

\vspace{2mm}

 - If the data follow a normal distribution, the Type I Error rate was approximately the nominal 0.05 level
:::

How does the sign test do in these scenarios?


## The Sign Test {.t}
\label{why}
One problem: with $n=5$, the sign test cannot actually do a 0.05-level test.

\vspace{3mm}

### Daily Check Question (answer to be written in your Rmd file):
\vspace{-2mm}
Why can't it?

## Your Turn \#1 {.t}
\label{sign1}
First, in your R Markdown file, write a function to perform the sign test for a sample of size $n=5$ and $\alpha=0.0625 + \epsilon$. 

 - The function should take a vector of 5 values as its input, and return the (two-sided) p-value according to the sign test. 
 
\vspace{2mm}

 - You may hardcode the null hypothesis value of 6. 
 
\vspace{2mm}

 - Your function may ignore ties (since we will be simulating from continuous distributions, there is a probability of 0 that any simulated value will exactly equal 6).

\vspace{2mm}

 - Using the `pbinom` function in some manner will likely be the easiest way to do it.

## Your Turn \#1 {.t}
Now, here is the code from last time to estimate the Type I Error rate of the t-Test with Gamma-distributed data:
```{r, eval=FALSE, size="small"}
count <- 0

for(i in 1:10000){
  gam_data <- rgamma(n=5, shape=1.2, scale=(6/1.2))
  p.val <- t.test(gam_data, mu=6)$p.value
  if(p.val < 0.05){
    count <- count + 1
  }
}

TypeI <- count / 10000
TypeI
```

## Your Turn \#1 {.t}
\label{sign2}

 - Edit the code on the previous slide to produce a simulated estimate of the Type I Error rate of the sign test for Gamma-distributed data with $n=5$ at $\alpha=0.0625 + \epsilon$.
   - Note: to obtain a median of 6, we actually need to edit the `scale` value to be equal to 6.757 (the details of why are way beyond the scope of this course)
 
\vspace{2mm}

 - Then, repeat with normal distribution with $\mu=6$ and $\sigma=1$ like last time.

\vspace{2mm}

 - Repeat again with the uniform distribution from 2 to 10 (also from last time)
 
Comment briefly on what you observe, specifically on whether each estimate of the Type I Error rate is as expected or not. 

## Bootstrapped Confidence Intervals {.t}
In DSC 10, we learned how to construct bootstrap confidence intervals, with Python code looking like this:

```{r, echo=FALSE, warning=FALSE}
library(reticulate)
```


```{python, echo=FALSE}
import numpy as np
np.set_printoptions(legacy='1.25')
```

```{python, size="footnotesize"}
boot_means = np.array([])

sleep = [3, 7, 1, 2, 2]

for i in range(10000):
    
    # Resample from my_sample WITH REPLACEMENT and compute the mean.
    mean = np.random.choice(sleep, size=5, replace=True).mean()

    # Store it in our array of means
    boot_means = np.append(boot_means, mean)
```

## Bootstrapped Confidence Intervals {.t}

and then the 95\% bootstrap confidence interval is:
```{python}
left = np.percentile(boot_means, 2.5)
right = np.percentile(boot_means, 97.5)
[left, right]
```

 
## Bootstrapped Confidence Intervals {.t}
Here is R code to do the same thing:
```{r}
sleep <- c(3, 7, 1, 2, 2)
boot_means <- NA

for(i in 1:1000){
  boot_means[i] <- mean(sample(sleep, replace=TRUE))
}

quantile(boot_means, probs=c(0.025, 0.975))
```

(there is randomness, so the confidence intervals from R and Python may or may not match, but they are in principle from exactly the same procedure)

## Bootstrapped Confidence Intervals {.t}
We also learned in DSC 10 that a confidence interval can be used to perform a hypothesis test:

::: {.block}
### Inverting the confidence interval
\vspace{-2mm}
 - CI does not contain $\mu_0 \Leftrightarrow$ reject $H_0$

\vspace{3mm}

 - CI contains $\mu_0 \Leftrightarrow$ fail to reject $H_0$
:::

While we don't exactly get a p-value from this procedure, we do get the decision of whether to reject or fail to reject $H_0$ at any $\alpha$ level (e.g. a 95\% confidence interval corresponds to $\alpha=0.05$).

## Now, what is the Type I Error rate of this procedure? {.t}
Like before, let us estimate the Type I Error rate under three scenarios:

 - Gamma-distributed data (the original one that had a mean of 6)

\vspace{3mm}

 - Normally distributed data with $\mu=6$ and $\sigma=1$.

\vspace{3mm}

 - Uniformly distributed data from 2 to 10 (also from last time)


In each case, we will use $n=5$ again. And here, we can actually go back to doing a test of the mean $\mu$ (instead of the median like the sign test does). 

## Your Turn \#2 {.t}
\label{boot1}
Let's do it for the Gamma-distributed data together. Here's some partial code:

```{r, size="scriptsize", eval=FALSE}
count <- 0

# The outer loop is so that it simulates 1000 random sets of data
for(j in 1:1000){
  gam_data <- _______
  boot_means <- NA
  
  # This is the bootstrap on each dataset
  for(i in 1:1000){
    boot_means[i] <- ______
  }
  
  ci <- _______
  if(______){
    count <- count + 1
  }
}

TypeI <- _______
```


## Your Turn \#2 {.t}
\label{boot2}
Then, on your own, modify the code to estimate the Type I Error rate under the normal and uniform cases.



## Recap {.t}

 - We evaluated the Type I Error rates of the sign test and the bootstrap hypothesis test via simulation
   - The sign test always gives a proper $\alpha$-level test without any distributional conditions necessary, but:
     - It is a test of the median (not the mean)
     - Not every desired $\alpha$-level is possible due to the discrete nature of the rejection regions
   - The bootstrap hypothesis test is (surprisingly) not good!
     - We will investigate this further in Lab 2. 


\vspace{4mm}

### Still to come
\vspace{-2mm}

 - Statistical power!
 
\vspace{3mm}

 - More complex statistical models

## Daily check for today {.t}
Upload your R Markdown pdf output file to Gradescope, consisting of:

1. The answer to the question on Slide \ref{why}: Why can't a sign test do a 0.05-level test with $n=5$?

\vspace{4mm}

2. Your Turn \#1 from Slides \ref{sign1} - \ref{sign2} consisting of the simulations to estimate Type I Error rate from the sign test for the three cases (Gamma, normal, uniform).

\vspace{4mm}

3. Your Turn \#2 from Slides \ref{boot1} - \ref{boot2} consisting of the simulations to estimate Type I Error rate from the bootstrap hypothesis test for the three cases (Gamma, normal, uniform).
