1
00:00:00,630 --> 00:00:08,100
Hi, welcome back. In this video, you will learn how to download historical stock data via Python.

2
00:00:09,620 --> 00:00:17,870
Here on my screen, you can see an example of a CSV file which contains historical stock data, these

3
00:00:17,870 --> 00:00:21,170
data are for "Apple", the company.

4
00:00:21,710 --> 00:00:27,770
So some companies allow their stocks to be sold to everyone in the world.

5
00:00:28,370 --> 00:00:31,070
So basically, you can buy shares of a company.

6
00:00:31,340 --> 00:00:35,450
And so every day there is a price of the share.

7
00:00:36,050 --> 00:00:41,930
It opens at a certain price in dollars or euros or other currencies.

8
00:00:42,620 --> 00:00:48,800
It reaches a peak the price every day, and then it reaches perhaps a low.

9
00:00:49,340 --> 00:00:59,270
And then when the day closes at 4pm or 5pm, the closing price is also registered and also the volume

10
00:00:59,270 --> 00:01:01,430
how many shares were sold that day?

11
00:01:01,670 --> 00:01:07,260
So these are data for every day for a particular company.

12
00:01:07,280 --> 00:01:08,420
These are for Apple.

13
00:01:08,720 --> 00:01:17,750
So you can download any data for any company which is public, which means they allow you to buy their

14
00:01:17,750 --> 00:01:18,380
shares.

15
00:01:18,830 --> 00:01:25,250
And the website that has these data is finance.yahoo.com.

16
00:01:25,460 --> 00:01:33,200
So you need to go to finance.yahoo.com and then understand how the data are made available.

17
00:01:33,560 --> 00:01:35,450
So this is how it works.

18
00:01:36,560 --> 00:01:43,880
Each company has a ticker symbol, so basically a code that identifies the company in the stock markets.

19
00:01:44,120 --> 00:01:45,140
For Apple,

20
00:01:45,410 --> 00:01:48,260
that is AAPL.

21
00:01:49,440 --> 00:01:57,690
For Microsoft, for example, it's MSFT, and so you can find these ticker symbols

22
00:01:57,870 --> 00:02:02,340
somewhere on the internet to know what company you want.

23
00:02:02,610 --> 00:02:17,130
So for Apple, you'd want to enter AAPL and then search, that button, then you'd end up in the Apple Web

24
00:02:17,130 --> 00:02:19,890
page for Yahoo Finance.

25
00:02:20,940 --> 00:02:22,740
So this is how you do it manually.

26
00:02:23,070 --> 00:02:29,550
Then you'd go to Historical Dot so you can do all these clicks manually, but also with selenium.

27
00:02:30,840 --> 00:02:34,190
And then this is the historical data web page.

28
00:02:34,760 --> 00:02:43,760
But then you can click here and then click on Max, so Max means you get all the data since the company

29
00:02:44,090 --> 00:02:46,130
started to sell stocks.

30
00:02:47,240 --> 00:02:50,090
So by clicking that and clicking apply.

31
00:02:51,240 --> 00:02:52,490
You should get data from...

32
00:02:54,790 --> 00:03:00,530
In the case of Apple, that would be from 1980.

33
00:03:00,770 --> 00:03:03,100
So 40 years ago.

34
00:03:03,790 --> 00:03:04,970
That's a lot of data.

35
00:03:05,560 --> 00:03:07,330
And you can see the data here.

36
00:03:07,330 --> 00:03:11,490
You can scrape them from this web page.

37
00:03:11,500 --> 00:03:15,430
Or even better, you can use this download button.

38
00:03:15,760 --> 00:03:18,940
So you can click with selenium in the download button.

39
00:03:18,940 --> 00:03:23,680
And that will give us a CSV file.

40
00:03:23,980 --> 00:03:24,880
You see here.

41
00:03:25,930 --> 00:03:26,890
It's downloaded.

42
00:03:26,890 --> 00:03:29,610
And yeah, those are the data.

43
00:03:30,160 --> 00:03:38,020
So from 1980, up to row 10361.

44
00:03:39,100 --> 00:03:41,260
So that's 2022.

45
00:03:43,300 --> 00:03:44,800
January 12.

46
00:03:46,190 --> 00:03:46,490
Now.

47
00:03:48,850 --> 00:03:50,350
This is how you do it manually.

48
00:03:50,890 --> 00:03:57,550
How do we do it with the Selenium now? Well, you can do it by repeating those clicks with

49
00:03:57,550 --> 00:03:59,690
Selenium as you learned previously.

50
00:03:59,710 --> 00:04:03,930
So you find the element, you click it using the dot click method.

51
00:04:04,000 --> 00:04:13,870
And so one step by step, or you can be smarter and not use selenium at all, because what happens

52
00:04:13,870 --> 00:04:21,070
is that if you click this download button, what's happening is that you are clicking a link.

53
00:04:22,270 --> 00:04:29,050
You can get that link by right clicking on the download button and then go to copy, link, address

54
00:04:29,050 --> 00:04:32,590
or copy link, depending on your browser.

55
00:04:32,590 --> 00:04:37,840
I'm on Chrome, so I have this copy link address. With that copy link address.

56
00:04:38,500 --> 00:04:43,690
You can go to Python and paste that link there.

57
00:04:44,530 --> 00:04:51,460
So that link is the link that if visited, it downloads a CSV file.

58
00:04:51,730 --> 00:04:58,990
So instead of using Selenium, we can use this link because this link has embedded the period.

59
00:04:59,350 --> 00:05:01,120
So from that period,

60
00:05:02,160 --> 00:05:11,910
that's in POSIX time, we will convert it, so that first period is the first date in the year 1980 when

61
00:05:11,910 --> 00:05:13,590
the data became available.

62
00:05:13,750 --> 00:05:16,370
And then the second period is the last day.

63
00:05:16,380 --> 00:05:23,040
So yesterday. Because the stock today is not open yet, so that is from yesterday.

64
00:05:24,240 --> 00:05:25,740
So those are the periods.

65
00:05:25,950 --> 00:05:35,190
So this is basically a query, and we can modify this URL by inputting other periods here to get any

66
00:05:35,190 --> 00:05:37,200
period that we want with Data.

67
00:05:37,620 --> 00:05:43,340
Then we have the interval "1d", so we want data for every day.

68
00:05:43,350 --> 00:05:49,020
One day the interval, and then we have the ticker symbol.

69
00:05:49,020 --> 00:05:53,610
So the codes of the company, we want data from Apple in this case.

70
00:05:54,180 --> 00:06:03,060
So we write Apple there, after the download, so download, slash Apple, or MSFT  for Microsoft or other ticker

71
00:06:03,060 --> 00:06:04,350
symbols for other companies.

72
00:06:04,650 --> 00:06:11,490
So there's a pattern that can be followed, and you can download any CSV data, any stock data

73
00:06:11,760 --> 00:06:12,600
with Python.

74
00:06:13,500 --> 00:06:14,310
So let's do this.

75
00:06:14,670 --> 00:06:19,080
To download files with Python, you can use requests.

76
00:06:22,520 --> 00:06:32,170
Then once you have a URL as a string, so let's say you URL is equal to that string, so enclose the

77
00:06:32,180 --> 00:06:43,670
URL in quotes and then content is equal to requests.get. Get is a method which expects a URL.

78
00:06:44,810 --> 00:06:46,640
So the URL is that one.

79
00:06:48,010 --> 00:06:54,020
And then you say dot content. Content is a property.

80
00:06:54,670 --> 00:07:00,490
So that is the requests objects and then you grab the content of that request object.

81
00:07:00,790 --> 00:07:02,290
Let's see what content is.

82
00:07:04,880 --> 00:07:06,060
By printing it out.

83
00:07:06,950 --> 00:07:09,320
So it's a bite object.

84
00:07:09,620 --> 00:07:12,380
And currently it's forbidden.

85
00:07:12,860 --> 00:07:16,010
That is because we need the headers.

86
00:07:16,190 --> 00:07:23,900
So, the web page, the Yahoo Finance web page doesn't allow scripts to download files.

87
00:07:24,740 --> 00:07:28,610
You can fix that by writing some headers.

88
00:07:29,630 --> 00:07:30,580
So that's a variable.

89
00:07:30,590 --> 00:07:31,880
It has this dictionary.

90
00:07:32,060 --> 00:07:36,890
You can find this attached in this lecture, in the lecture resources.

91
00:07:37,640 --> 00:07:45,920
So this will impersonate a browser, so the scripts will behave like a browser, and so it will be allowed

92
00:07:45,920 --> 00:07:47,510
to download the file.

93
00:07:48,050 --> 00:07:53,720
But this header symbol now you have to provide it to the get method as a second argument.

94
00:07:54,440 --> 00:07:57,890
So headears is equal to to headers.

95
00:07:58,370 --> 00:08:00,680
The headers arguments is equal to that variable.

96
00:08:01,660 --> 00:08:02,500
Let's try again.

97
00:08:08,480 --> 00:08:11,120
And this time it seems to be working.

98
00:08:13,340 --> 00:08:15,050
So you see some dates here.

99
00:08:16,560 --> 00:08:22,200
With the price, the opening price, the closing price and so on.

100
00:08:22,710 --> 00:08:28,140
So those are the data, but it's just a byte object.

101
00:08:29,180 --> 00:08:36,950
What we need to do now is to create an empty file and then write those bytes in the file so that we

102
00:08:36,950 --> 00:08:41,570
have an actual file on disk. To create a file we use with open.

103
00:08:44,720 --> 00:08:51,800
And set a name for the file such as data.csv. Open it in write byte mode.

104
00:08:52,700 --> 00:09:01,040
You can also use, for CSV files you can also use write, and then change that content to text to deal

105
00:09:01,040 --> 00:09:02,450
with simple text.

106
00:09:03,070 --> 00:09:10,910
But when you use content here and 'wb' here, this is a more generalized approach.

107
00:09:11,120 --> 00:09:20,870
You can download binary files as well, such as audio files, MP3, MP4 files, zip files.

108
00:09:20,870 --> 00:09:23,690
So any binary files which are not text files.

109
00:09:24,530 --> 00:09:28,610
So this method includes everything non-binary and binary files.

110
00:09:29,000 --> 00:09:38,290
So I'll use this method. With open data.csv open as binary in write mode, as file.

111
00:09:38,700 --> 00:09:45,950
So in this file mode, which opened in write mode, we can write what's the content.

112
00:09:47,210 --> 00:09:55,130
So that's content now is written in this empty file, and so we will end up with a file on disk.

113
00:09:55,900 --> 00:10:01,980
First, it's printed out from this print content, you can remove that. And then,

114
00:10:02,060 --> 00:10:04,130
this code is executed.

115
00:10:04,370 --> 00:10:07,610
So we we end up with this data.csv file.

116
00:10:08,480 --> 00:10:10,450
Yep, that's the file. You can download it.

117
00:10:10,460 --> 00:10:17,690
by going to here and you can open it with Excel or any other spreadsheet programs.

118
00:10:17,930 --> 00:10:25,820
So we have this header, the date, the open, the high, the low, the close, the adjusted close and the volume

119
00:10:26,960 --> 00:10:34,340
and it runs up to yesterday, that date, right?

120
00:10:40,150 --> 00:10:48,520
Now, let's further improve our scripts by letting the user choose whatever period they want and whatever

121
00:10:48,790 --> 00:10:51,400
ticker symbol they want data for.

122
00:10:52,270 --> 00:10:58,660
For that, I would like to implement an input function. Actually two input functions.

123
00:10:58,930 --> 00:11:08,440
The first would be from date eq,ual to input and enter a message for the users something like:

124
00:11:08,830 --> 00:11:10,750
"Enter start date".

125
00:11:16,520 --> 00:11:17,540
In this format.

126
00:11:21,120 --> 00:11:30,150
So the user is supposed to answer the start date from when they want data for, and that should be in

127
00:11:30,270 --> 00:11:40,620
year slash month slash day, that format. And then we will parse that format and and convert it into

128
00:11:40,620 --> 00:11:44,610
a timestamp into this POSIX time, EPOCH time.

129
00:11:44,880 --> 00:11:46,740
So this here is seconds.

130
00:11:46,770 --> 00:11:48,810
And this is a time system.

131
00:11:49,090 --> 00:11:55,890
It is seconds from the year 1970, 1st of January of that year.

132
00:11:56,700 --> 00:12:01,920
And so we can convert that into human readable date time.

133
00:12:02,490 --> 00:12:09,480
Actually, we get the human readable date time in this format, and then we convert it to this timestamp

134
00:12:09,720 --> 00:12:17,370
so that we can supply this to this URL because this works with this kind of format.

135
00:12:18,780 --> 00:12:26,970
So I would like to duplicate this input function and renamed the variable to _date and end_date

136
00:12:27,660 --> 00:12:29,190
in that format again.

137
00:12:30,600 --> 00:12:39,720
We're going to use a chain of methods to convert these data type formats into Epoch time format.

138
00:12:40,170 --> 00:12:40,770
It's simple

139
00:12:40,770 --> 00:12:43,020
actually. I'll show you how I can do it.

140
00:12:43,680 --> 00:12:47,640
I'll show you first in the interactive shell and then implement it here.

141
00:12:47,910 --> 00:12:59,310
So let's suppose that from date given by the user is 2022, one for January and 10 for the date.

142
00:12:59,310 --> 00:13:02,760
So 10th of January 2022.

143
00:13:03,540 --> 00:13:08,730
So that's a strength that the user will enter and that is stored in this variable.

144
00:13:08,940 --> 00:13:12,240
Now, how can we convert it into this format?

145
00:13:13,500 --> 00:13:19,380
Well, first we need to convert it from a string into a date time object.

146
00:13:19,650 --> 00:13:24,390
For that, we need from datetime, import datetime.

147
00:13:25,500 --> 00:13:28,200
So we're going to need to import that as well in here.

148
00:13:29,460 --> 00:13:32,490
So past it there so that we can have it for later.

149
00:13:33,420 --> 00:13:36,210
And then, you use this date time here.

150
00:13:38,920 --> 00:13:47,290
And use its strptime method, which gets two arguments, the first argument is the string.

151
00:13:49,290 --> 00:13:50,800
So that is the string.

152
00:13:50,850 --> 00:13:55,770
And the second argument is the format that this string contains.

153
00:13:55,950 --> 00:13:59,610
So this will give us a specialized datetime object.

154
00:13:59,620 --> 00:14:00,780
You will see what that is.

155
00:14:01,200 --> 00:14:05,730
So the format of this string is percentage Y.

156
00:14:05,760 --> 00:14:09,770
So that means the year is the first one and then there is the slash.

157
00:14:09,780 --> 00:14:16,890
And then there is the month the percentage lowercase m, and then a slash again.

158
00:14:16,890 --> 00:14:24,570
You see, I'm following this, and then percentage d. So these are format codes, percentage, capital Y.

159
00:14:24,570 --> 00:14:36,060
Percentage lowercase m, and percentage d. So I'm going to store this in a variable D and then D is

160
00:14:36,060 --> 00:14:37,680
this datetime format.

161
00:14:37,680 --> 00:14:41,250
So it's the same date, 10th of January 2022.

162
00:14:41,520 --> 00:14:46,230
Now, having this means we can convert it into epoch time.

163
00:14:47,250 --> 00:14:49,970
For that, we need the time module.

164
00:14:49,980 --> 00:14:56,520
So inport time, and then epoch would be time.mktime.

165
00:14:58,050 --> 00:15:05,130
And then you point to the d variable, and then you point to the time tuple variable which converts

166
00:15:05,130 --> 00:15:09,720
these datetime object into yet another similar object.

167
00:15:09,930 --> 00:15:22,440
So this mktime needs that other object so execute, and epoch finally is in that format similar

168
00:15:22,440 --> 00:15:22,860
to that and that.

169
00:15:22,860 --> 00:15:37,110
So let's copy this code we get from time we convert it into d. Let's see from datetime.

170
00:15:38,670 --> 00:15:50,760
Same, to datetime, so we converted the end date as well into this, so paste it there, change these too, to_date.

171
00:15:51,090 --> 00:15:59,490
So that string goes here and that goes in here of and then we need to convert into epoch, so import time.

172
00:16:00,120 --> 00:16:04,800
And write, creates a variable from_epoch is equal to

173
00:16:06,310 --> 00:16:08,860
time.mktime().

174
00:16:10,050 --> 00:16:21,120
And this is from_datetime.timetuple() that time to pull so from_epoch, and then let's duplicate this.

175
00:16:21,780 --> 00:16:23,610
This is to_datetime.

176
00:16:24,890 --> 00:16:25,700
To_epoch.

177
00:16:27,000 --> 00:16:35,020
And that's it, so once we have this, we can inject them into the URL, how can we do that?

178
00:16:35,040 --> 00:16:36,570
Well, using an f string.

179
00:16:36,570 --> 00:16:43,860
So add F in front of the url string, and then instead of period here,

180
00:16:43,890 --> 00:16:53,940
instead of this number, period is equal to curly brackets and then goes from epoch, that variable.

181
00:16:54,240 --> 00:16:56,610
So that number will be placed in here.

182
00:16:57,090 --> 00:16:58,460
So careful here.

183
00:16:58,470 --> 00:17:02,040
After that comes this and operator.

184
00:17:02,340 --> 00:17:05,220
Then we have period_to and then we have the other

185
00:17:05,240 --> 00:17:09,869
number, it's to_periods.

186
00:17:10,470 --> 00:17:15,690
And of course, you can also make the ticker symbol dynamic.

187
00:17:16,290 --> 00:17:19,349
For example, ticker is equal to input.

188
00:17:20,970 --> 00:17:24,300
Enter the ticker symbol.

189
00:17:24,950 --> 00:17:30,870
So, for example, the user will enter AAPL for Apple, et cetera.

190
00:17:31,230 --> 00:17:34,620
Therefore, we need to replace AAPL to

191
00:17:36,410 --> 00:17:40,190
these curly brackets and then enter ticker there, that variable.

192
00:17:41,180 --> 00:17:50,570
So, for example, this time I'll try MSFT for Microsoft's, so let's run the code, enter the ticker

193
00:17:50,570 --> 00:18:01,790
symbol, it was MSFT, FT, enter the start date, 2021.

194
00:18:02,940 --> 00:18:11,550
Let's say, November for, 11 and then the 20th of November.

195
00:18:14,310 --> 00:18:24,710
And then, the end date, 2021, December, the 20th, enter. I've got an error, to _period is not defined.

196
00:18:24,720 --> 00:18:30,180
Obviously, I placed here to_period, but it should be to_epoch.

197
00:18:31,710 --> 00:18:32,580
So let's try again.

198
00:18:35,180 --> 00:18:36,500
MSFT.

199
00:18:37,890 --> 00:18:45,710
2021, 11, 20, 12, 20, enter.

200
00:18:47,410 --> 00:18:52,990
And so we have another error, but this time is the error is not from the Python interpreter.

201
00:18:53,350 --> 00:18:55,660
It is from the server.

202
00:18:56,080 --> 00:18:56,710
Why?

203
00:18:56,740 --> 00:19:02,380
Well, because this print function is printing out the content and this is the content we got.

204
00:19:02,800 --> 00:19:09,970
And in the content, which is written by the server, we get this message that the parameter period

205
00:19:09,970 --> 00:19:12,250
is of an invalid type.

206
00:19:13,910 --> 00:19:24,530
That is because this is tricky, but if you remember our from_epoch and the to_epoch values were in

207
00:19:24,890 --> 00:19:34,670
in a float format, so dot something dot zero or something else and this URL does not accept this

208
00:19:35,120 --> 00:19:36,560
kind of decimal point

209
00:19:36,570 --> 00:19:37,040
number.

210
00:19:37,250 --> 00:19:43,040
So we need to convert it into an integer and we can do that just here.

211
00:19:44,150 --> 00:19:51,050
So we create from_epoch and then we convert it into an int and we convert to_epoch into an int

212
00:19:51,050 --> 00:19:51,410
as well.

213
00:19:52,010 --> 00:19:53,750
And that should do it, so run again.

214
00:19:55,090 --> 00:19:55,600
MSFT.

215
00:19:55,820 --> 00:20:03,470
2021, 11, 20th and until December.

216
00:20:03,560 --> 00:20:06,320
Enter and yeah, those are the data.

217
00:20:06,330 --> 00:20:06,800
Finally.

218
00:20:07,280 --> 00:20:10,220
And you can also see them in here.

219
00:20:11,150 --> 00:20:13,370
So that's one month of data.

220
00:20:14,030 --> 00:20:17,840
It's 19 rows in December.

221
00:20:18,500 --> 00:20:23,300
So that excludes holidays and Saturdays and Sundays.

222
00:20:23,630 --> 00:20:31,280
That's why we don't have thirty one rows, but we have only nineteen, so lots of holidays in December.

223
00:20:32,030 --> 00:20:41,240
And so it starts from the 22nd of November, obviously the 20th and the 21st of November,

224
00:20:41,480 --> 00:20:44,000
where a Saturday and Sundays, probably.

225
00:20:44,390 --> 00:20:46,610
And so we don't have those data there.

226
00:20:48,270 --> 00:20:52,030
And that's the final code, so I thank you, and I'll see you later.