Skip to content
Snippets Groups Projects
Commit 258abbf2 authored by Jäkel, Frank's avatar Jäkel, Frank
Browse files

Clarified the other exercises

parent f6c701b2
No related branches found
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
signal_detection_homework.ipynb
+ 4
− 4
View file @ 258abbf2
%% Cell type:markdown id: tags:
# Signal Detection Theory: Homework
You've collected data in a yes-no experiment. Now we will analyze these data using signal detection theory. In our course we use the following text:
Wickens, T.D. (2002). *Elementary Signal Detection Theory*. Oxford University Press.
For this homework the most relevant sections are 1.1-1.3 (p. 3-15), 2.1-2.3 (p. 17-26) and sections 3.1 (p. 39-42) and 3.3 (45-48). But you can probably use any introduction to signal detection theory instead as long as it covers the equal-variance Gaussian model and receiver operating characteristics (and they all do). Let's first establish some common notation (following Wickens) and show you how to make a nice plot that illustrates the equal-variance Gaussian model.
%% Cell type:code id: tags:
``` python
from signal_detection import *
# the standard normal density
def phi(x):
return norm.pdf(x,0,1)
# the standard normal cumulative distribution function
def Phi(x):
return norm.cdf(x,0,1)
# the inverse of Phi
def Z(x):
return norm.ppf(x,0,1)
```
%% Cell type:markdown id: tags:
Here's the typical plot that you see in all the introductions to signal detection theory.
%% Cell type:code id: tags:
``` python
# we abbreviate lambda to lam because lambda is a keyword in python
d_prime = 3
lam = 1
plot_sdt(d_prime,lam,p=None)
```
%% Cell type:markdown id: tags:
The red curve is for the probability distribution for the noise-only trials. The green curve is the probability distribution for the signal trials. Here, both are Gaussians with equal variance (this is just the simplest variant of signal detection theory). The red area gives you the probability for a false alarm and the green hatched area gives you the probability for a hit. As a warm-up for the exercises and to remind you of the basic logic of signal detection theory: Play with $d'$ and $\lambda$ and observe how the hit rate and false alarm rate change.
%% Cell type:markdown id: tags:
Now let's load your data.
%% Cell type:code id: tags:
``` python
subject = 'roc_xy' # change xy to your initials
intensity = 20 # put in the 75%-threshold that you used in the roc experiment
data = load_data(subject)
df = summarize(data,p=None,intensity=intensity,mode='all')
df = df.droplevel('intensity') # intensity is always the same anyway
df
```
%% Cell type:code id: tags:
``` python
# we need the hits h and the false alarms f
f = df['f']
h = df['h']
# and the values of the prior probabilities in each condition
p = df.index.values
# later we'll also need the performance in each block
pc = df['pc']
```
%% Cell type:markdown id: tags:
## Exercise 1
Compute your sensitivity $d'$ and your response bias $\lambda$ for each condition. Use `plot_sdt` to make a plot for each condition. Check that the hit rates and false alarm rates match the empirical data. What happens to the criterion $\lambda$ as the probability *p* for the signal trials increases and why is this reasonable?
### Answer
%% Cell type:code id: tags:
``` python
# compute d_prime_hat
d_prime_hat = ...
```
%% Cell type:code id: tags:
``` python
# compute lambda_hat
lam_hat = ...
```
%% Cell type:code id: tags:
``` python
# make plots
...
```
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 2
Write a function `plot_roc_z` that produces a z-transformed ROC-plot for your data. Base your function on the function `plot_roc` in `signal_detection.py` (but leave out the confidence intervals). What's a good limit for the axes? Place the original ROC plot and the new z-transformed ROC plot next to each other using `subplot`.
### Answer
%% Cell type:code id: tags:
``` python
def plot_roc_z(df, labels=True, color='tab:blue'):
...
```
%% Cell type:code id: tags:
``` python
subplot(1,2,1)
plot_roc(df)
subplot(1,2,2)
plot_roc_z(df)
```
%% Cell type:markdown id: tags:
## Exercise 3
Assuming that your sensitivity is constant in all conditions, we can compute the average $\hat{d'}$ across all conditions as our estimate for your sensitivity. Use this estimate to draw out the corresponding ROC-curves in both plots from the previous exercise. Are there any data points that the ROC-curve for the equal-variance Gaussian model fits better than the ROC-line that results from the assumption that the proportion of correct responses stays constant? Are the data better described by the assumption that *pc* stays constant when *p* changes (see ROC-line in `signal_detection_data_collection.ipynb`) or by the assumption that $\hat{d'}$ is constant? How do you judge this from the two plots?
### Answer
%% Cell type:code id: tags:
``` python
# compute average over all d_prime_hat in all conditions
...
```
%% Cell type:code id: tags:
``` python
# make plots
...
```
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 4
From now on, we take as our "best" estimate for your sensitivity the average $\hat{d'}$ over all conditions -- as in the last exercise. Make a plot with `plot_sdt` with this estimate for an unbiased subject. How do you have to choose $\lambda$ if you want to be unbiased? Give a formula to compute the unbiased $\lambda$. Were you unbiased in the $p=0.5$ condition (go back to Exercise 1 to check)? How much does your $\lambda$ differ from the unbiased $\lambda$? Why do the hit rate and the false alarm rate add to one for an unbiased subject? Where on the ROC-curve is the point where you're unbiased? Plot your equal-variance-Gaussian-ROC-curve again, mark this point and draw the line where hit rate and false alarm rate add to one.
From now on, we take as our "best" estimate for your sensitivity the average $\hat{d'}$ over all conditions -- as in the last exercise. Make a plot with `plot_sdt` with this estimate for an unbiased subject. How do you have to choose $\lambda$ if you want to be unbiased? Give a formula to compute the unbiased $\lambda$. Were you unbiased in the $p=0.5$ condition (go back to Exercise 1 to check)? How much does your $\lambda$ differ from the unbiased $\lambda$? Why do the hit rate and the false alarm rate add to one for an unbiased subject? Where on the ROC-curve is the point where you're unbiased? Plot your equal-variance-Gaussian-ROC-curve again, mark this point, and draw the line where hit rate and false alarm rate add to one.
### Answer
%% Cell type:code id: tags:
``` python
...
```
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 5
How biased should you have been in each of the other conditions, where $p\neq0.5$? Given your estimate for your sensitivity $d'$, what should your criterion $\lambda$ have been? In order to derive and compute the optimal $\lambda$ for a given $d'$ and $p$ you will have to use Bayes' theorem. Let $X$ be the random variable based on which you decide and $S$ is the stimulus in a trial. A useful variant of Bayes' theorem is this:
$$
\frac{p(S=\text{signal}\mid X=x)}{p(S=\text{noise}\mid X=x)} = \frac{p(X=x\mid S=\text{signal})}{p(X=x\mid S=\text{noise})}\cdot\frac{p(S=\text{signal})}{p(S=\text{noise})}
$$
The term on the left is called the posterior odds, the second the term is the likelihood ratio and the last term gives the prior odds. The task for this exercise is: Derive this equation from the traditional form of Bayes' theorem.
### Answer
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 6
Obviously, you should say that you saw the stimulus whenver the posterior odds are greater than 1 because this means that it is more probable that there was a signal than that there was not. Hence, given an observation $X=x$ you should ask yourself:
$$
\frac{p(X=x\mid S=\text{signal})}{p(X=x\mid S=\text{noise})} > \frac{p(S=\text{noise})}{p(S=\text{signal})} ?
$$
Intuitively speaking, this inequality asks whether there is enough evidence of a signal in the observation $X=x$ to overrule the prior odds for there being no signal. In the equal-variance Gaussian model we have
\begin{align*}
p(X=x\mid S=\text{noise}) &= \phi(x) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^2} \\
p(X=x\mid S=\text{signal}) &= \phi(x-d') = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(x-d')^2}
\end{align*}
and just as before we will use the shorthand $p(S=\text{signal})=p$ and correspondingly $p(S=\text{noise})=1-p$. Plug all of these into the above equation and solve for $x$, i.e. answer the question: for which values of $x$ should you say that there was a signal? What is the optimal criterion $\lambda$ for a given $d'$ and $p$. According to your derivation, what is the optimal criterion in the condition $p=0.5$ when you should be unbiased?
and just as before we will use the shorthand $p(S=\text{signal})=p$ and correspondingly $p(S=\text{noise})=1-p$. Plug all of these into the above equation and solve for $x$, i.e. answer the question: for which values of $x$ should you say that there was a signal? What is the optimal criterion $\lambda$ for a given $d'$ and $p$. According to your derivation, what is the optimal criterion in the condition $p=0.5$ (where you should be unbiased)? What happens relative to this unbiased criterion when $p>0.5$ or $p<0.5$, respectively? Is this optimal behavior qualitatively consistent with your observed behavior?
### Answer
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 7
How does the proportion of correct responses change with the criterion? For your estimate of your sensitivity make a plot of the proportion of correct responses in each condition as a function of the criterion $\lambda$. Mark zero and $d'$ on the x-axis. Also mark the optimal $\lambda$ on each curve as computed in Exercise 6. For $\lambda$ on the far left and the far right, where do the error curves saturate and why? In your plots you should be able to see that the optimal $\lambda$ really maximizes the proportion of of correct responses. How does your actual proportion of correct responses compare to the maximum proportion of correct responses we would expect based your estimated sensitivity?
How does the proportion of correct responses change with the criterion? For your estimate of your sensitivity make a plot of the proportion of correct responses in each condition as a function of the criterion $\lambda$. Mark zero and $d'$ on the x-axis. Also mark the optimal $\lambda$ on each curve as computed in Exercise 6. For $\lambda$ on the far left and the far right, where do the error curves saturate and why? In your plots you should be able to see that the optimal $\lambda$ really maximizes the proportion of correct responses. How does your actual proportion of correct responses compare to the maximum proportion of correct responses we would expect based your estimated sensitivity?
### Answer
%% Cell type:code id: tags:
``` python
# optimal lambda
lam_opt = ...
```
%% Cell type:code id: tags:
``` python
# make plots
...
```
%% Cell type:code id: tags:
``` python
# optimal proportion of correct responses
```
%% Cell type:code id: tags:
``` python
# observed proportion of correct responses
```
%% Cell type:markdown id: tags:
...
%% Cell type:markdown id: tags:
## Exercise 8
Your performance was probably a little worse than it could have been. This means that you haven't set your criterion optimally. How does the measured criterion compare to the optimal criterion in each condition? Do you see systematic deviations? How would you characterize them?
Your performance was probably a little worse than it could have been. This means that you haven't set your criterion optimally. How does the measured criterion compare to the optimal criterion in each condition? Do you see systematic deviations? How would you characterize them? Even if your behavior is not completely optimal, would you still describe your behavior as rational? Why or why not (you might want to refer back to your answer in Exercise 7 and general considerations about rationality form )?
### Answer
%% Cell type:code id: tags:
``` python
...
```
%% Cell type:markdown id: tags:
...
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment