1
00:00:00,330 --> 00:00:02,030
Welcome to this bonus lecture.

2
00:00:02,070 --> 00:00:05,970
I was recently asked: what do you do if you have null results?

3
00:00:05,970 --> 00:00:07,830
And I thought it's a very good question.

4
00:00:08,070 --> 00:00:12,420
And so I decided to add this bonus lecture in which I will do my very best to give you some pointers

5
00:00:12,420 --> 00:00:14,480
to deal with this problem.

6
00:00:14,520 --> 00:00:20,550
So by no results we mean that for example if we are comparing a treatment and a placebo group that there

7
00:00:20,550 --> 00:00:22,890
is no difference between the two groups.

8
00:00:23,070 --> 00:00:29,090
And if you're using inferential statistics then it means that your p value is larger than point 0 5.

9
00:00:29,280 --> 00:00:33,840
And this tends to be a relatively tricky situation because it's quite difficult to sell null results

10
00:00:34,320 --> 00:00:35,600
in your writing.

11
00:00:35,610 --> 00:00:40,320
So in this lecture I will give you some pointers how you can get the most out of your null results.

12
00:00:40,320 --> 00:00:44,410
So first of all why are null results problematic in the first place?

13
00:00:44,500 --> 00:00:49,470
And there are two main problems with null results and the first problem is that conclusions from null

14
00:00:49,470 --> 00:00:52,390
results are often not very convincing.

15
00:00:52,410 --> 00:00:57,660
Remember that the two qualities of an effective answer to your research question is that the question

16
00:00:57,720 --> 00:01:01,220
is convincing and valuable and a null result.

17
00:01:01,230 --> 00:01:04,640
In most cases will make your results less convincing.

18
00:01:04,680 --> 00:01:05,520
So why is that.

19
00:01:05,550 --> 00:01:10,930
Well one reason is that your results can simply reflect methodological issues.

20
00:01:10,980 --> 00:01:16,830
So in the Depression study it could be that the treatment wasn't executed correctly or it could be that

21
00:01:16,830 --> 00:01:23,040
the measure of depression was unreliable and both of these issues can lead to null results even if the

22
00:01:23,040 --> 00:01:24,780
treatment is effective.

23
00:01:24,780 --> 00:01:25,980
And that's the whole problem here.

24
00:01:25,980 --> 00:01:30,990
It's very hard to conclude from a null result that there isn't anything going on for example that the

25
00:01:30,990 --> 00:01:31,950
treatment doesn't work.

26
00:01:32,850 --> 00:01:38,130
Another reason for the same problem is that your results may reflect statistical issues.

27
00:01:38,190 --> 00:01:41,610
Most of the time that will mean that your sample size is too low.

28
00:01:41,610 --> 00:01:45,790
Maybe if you just had more participants we would actually see that the treatment works.

29
00:01:46,520 --> 00:01:51,240
And so for all of these reasons it's hard to draw a convincing conclusion from no results.

30
00:01:51,240 --> 00:01:52,640
So that is the first problem.

31
00:01:52,680 --> 00:01:58,170
And the second problem is that conclusions from no results often also do not seem valuable.

32
00:01:58,170 --> 00:02:03,150
So for example even if you can make the case that there are no methodological issues in your study and

33
00:02:03,150 --> 00:02:09,210
no statistical issues that you can really say with quite high certainty that the treatment didn't work

34
00:02:09,870 --> 00:02:13,890
then the problem is who really cares about an ineffective treatment.

35
00:02:13,890 --> 00:02:18,020
And so to deal with null results we need to fix all of these problems that you can see here.

36
00:02:18,060 --> 00:02:23,180
We need to make the conclusion convincing by ruling out methodological and statistical issues.

37
00:02:23,400 --> 00:02:27,090
And we need to make the case that our results are still valuable.

38
00:02:27,090 --> 00:02:30,820
Okay so the first step will be to deal with methodological issues.

39
00:02:31,010 --> 00:02:36,540
And one of the best solutions is to use a manipulation check if you have a manipulation check.

40
00:02:36,540 --> 00:02:37,550
In your study.

41
00:02:37,860 --> 00:02:45,450
So for example if you show that although the refocus treatment does not reduce depression it does cause

42
00:02:45,450 --> 00:02:51,270
people to focus less on the negative than that may rule out the concern that you just didn't execute

43
00:02:51,270 --> 00:02:52,630
the treatment correctly.

44
00:02:52,890 --> 00:02:55,200
Because it obviously did what it was supposed to do.

45
00:02:55,200 --> 00:02:57,890
It reduced the focus on the negative.

46
00:02:57,900 --> 00:02:59,280
It just didn't reduce depression.

47
00:03:00,270 --> 00:03:00,560
Okay.

48
00:03:00,570 --> 00:03:05,970
If you don't have a manipulation check then another solution you can use is to look at other people's

49
00:03:05,970 --> 00:03:07,230
research.

50
00:03:07,290 --> 00:03:12,030
So the goal here would be to show that your method has been established by previous research so that

51
00:03:12,030 --> 00:03:14,600
there cannot be much doubt about your method.

52
00:03:14,610 --> 00:03:20,280
So for example if you followed the same treatment protocol as another study and if you use the same

53
00:03:20,280 --> 00:03:26,160
measurement of depression as another study and both seem to be finding the other study then there isn't

54
00:03:26,160 --> 00:03:30,030
really much reason to believe that there are methodological issues in your study.

55
00:03:30,030 --> 00:03:35,280
And so by comparing yourself to similar research you can make the case that there were no methodological

56
00:03:35,280 --> 00:03:36,280
issues in your study.

57
00:03:36,990 --> 00:03:38,670
Okay so that is step one.

58
00:03:38,670 --> 00:03:42,030
Now the next step is to rule out statistical issues.

59
00:03:42,150 --> 00:03:44,260
And again there are several things you can do.

60
00:03:44,280 --> 00:03:50,940
One thing you can do is to use a power analysis a power analysis can show that your sample size would

61
00:03:50,940 --> 00:03:54,780
be sufficient to detect a very small effect.

62
00:03:54,810 --> 00:03:59,970
In other words where the power analysis you can make the case that even if there is a treatment effect

63
00:04:00,360 --> 00:04:02,400
it will be very very small.

64
00:04:02,400 --> 00:04:05,940
Another solution is again to use other people's research.

65
00:04:05,940 --> 00:04:11,280
So what you can do is you can show that similar research found significant results with the same or

66
00:04:11,280 --> 00:04:13,830
even a lower sample size than yours.

67
00:04:13,830 --> 00:04:17,690
And I've seen successful examples of both of these solutions.

68
00:04:17,710 --> 00:04:19,130
OK so that's step two.

69
00:04:19,280 --> 00:04:24,540
Now the last step will be to get the reader interested in your null result and usually the way you do

70
00:04:24,540 --> 00:04:31,220
that is by embedding your research into a theoretical context in which the net result is interesting.

71
00:04:31,260 --> 00:04:32,570
Okay that sounds very abstract.

72
00:04:32,580 --> 00:04:33,930
So let me give you an example.

73
00:04:34,680 --> 00:04:41,530
So in the fictional depression study the fact that the treatment is ineffective may not be very interesting.

74
00:04:41,610 --> 00:04:47,070
However this study was not just about the question whether the treatment is effective but it was also

75
00:04:47,070 --> 00:04:53,880
about testing different explanations of depression is depression caused by an unprocessed trauma or

76
00:04:53,880 --> 00:05:00,190
is it caused by a certain information processing style where people overly focus on the negative so

77
00:05:00,190 --> 00:05:05,230
in this case you could make the null result more interesting by focusing more on the implication that

78
00:05:05,230 --> 00:05:07,510
the cognitive explanation of depression.

79
00:05:07,690 --> 00:05:10,990
The one about the negative focus is false.

80
00:05:11,290 --> 00:05:16,420
And the reason why this is more interesting is because this is not just about a specific type of treatment

81
00:05:16,870 --> 00:05:18,970
but it's about a whole class of treatments.

82
00:05:19,390 --> 00:05:23,760
So there's a much bigger implication and therefore a lot more interesting.

83
00:05:23,800 --> 00:05:24,020
OK.

84
00:05:24,040 --> 00:05:28,090
Once you made it through all of these steps then the next thing you will have to do is to write this

85
00:05:28,090 --> 00:05:33,160
down and here's really important to use very cautious language.

86
00:05:33,160 --> 00:05:36,840
Remember what I discussed with you about writing about limitations.

87
00:05:36,850 --> 00:05:41,830
You always want to admit the limitation first so that you don't give the impression that you're just

88
00:05:41,860 --> 00:05:44,030
trying to argue your way out of it.

89
00:05:44,080 --> 00:05:50,410
So you want to write something like given that the results were not significant conclusions must be

90
00:05:50,410 --> 00:05:52,060
drawn with caution.

91
00:05:52,090 --> 00:06:01,000
Nevertheless the data may allow for a few general conclusions when taking all results into consideration

92
00:06:01,750 --> 00:06:06,180
and then you could write down your solutions to all the three steps that we just discussed.

93
00:06:06,280 --> 00:06:11,560
And with such a careful and skeptical introduction your reader will be much more likely to go along

94
00:06:11,560 --> 00:06:16,370
with your thinking rather than thinking that you're just trying to argue your way out of it.

95
00:06:16,420 --> 00:06:18,640
Okay so that's how we can deal with null results.

96
00:06:18,640 --> 00:06:19,900
I hope it's helpful.

97
00:06:19,990 --> 00:06:25,180
And if you haven't given a rating for this cause yet you could do me and future students a huge favor

98
00:06:25,240 --> 00:06:27,640
by just quickly leaving a rating for this course.

99
00:06:28,140 --> 00:06:28,370
All right.

100
00:06:28,390 --> 00:06:31,110
Thank you for listening and let me know if you have more questions.