1
00:00:00,660 --> 00:00:07,380
Hi, welcome to a new lecture here you learn how to use beautiful soup, a very important web scraping

2
00:00:07,380 --> 00:00:08,520
of library for Python.

3
00:00:08,760 --> 00:00:11,370
No, we use selenium through this section.

4
00:00:12,150 --> 00:00:14,760
And why are we using beautiful soup?

5
00:00:15,000 --> 00:00:17,610
And what's the difference between selenium and beautiful soup?

6
00:00:17,970 --> 00:00:27,450
Well, selenium is more targets for browser automations, which means you can use selenium to automate

7
00:00:27,450 --> 00:00:34,530
actions on a browser, such as feeling text boxes, clicking on a button, logging in and out, and

8
00:00:34,530 --> 00:00:35,160
so on.

9
00:00:35,520 --> 00:00:42,480
So you can use browser automation to scrape values than or log in and into actions.

10
00:00:42,930 --> 00:00:51,450
But if you want to just extract dots from a website, you don't have to open the browser like you do

11
00:00:51,450 --> 00:00:52,320
with selenium.

12
00:00:52,560 --> 00:01:00,420
So for that purpose, we can use beautiful soup, which is a more straightforward approach to web scraping.

13
00:01:01,200 --> 00:01:06,750
So you can think of selenium as a hybrid between browser automation and web scraping.

14
00:01:06,900 --> 00:01:13,860
So it does both and think of beautiful soup as a library exclusively for web scraping.

15
00:01:14,790 --> 00:01:21,630
So for this lecture, we're going to extract, we're going to scrape currency rates, the real time

16
00:01:21,630 --> 00:01:24,840
currency rate between any two currencies.

17
00:01:25,620 --> 00:01:30,420
For that, we're going to use excretes dot com extension rates dot com.

18
00:01:31,230 --> 00:01:32,550
So that's a web page.

19
00:01:32,910 --> 00:01:39,360
You can go to currency calculator and then you will see the current currency rate between these two

20
00:01:39,360 --> 00:01:43,560
current currencies, which is US dollars and Indian rupee.

21
00:01:44,850 --> 00:01:50,790
So the first thing you do is you want to specify manually on the browser what currency rates you want.

22
00:01:53,360 --> 00:01:58,940
So currently, this is us three Indian rupees, if you want to change these to, for example, the Euros

23
00:01:58,940 --> 00:01:59,900
to Indian rupees.

24
00:02:00,350 --> 00:02:06,830
The page will refresh and you will get the rates for euros to Indian rupees.

25
00:02:07,100 --> 00:02:11,630
Now, the browser address bar also changed to another URL.

26
00:02:11,960 --> 00:02:21,080
So now it's as you can see from Euro to INR, which is the currency code for Indian rupees.

27
00:02:21,290 --> 00:02:29,030
So any time you change this, let's say from Euro to USD, the browser with points to another URL,

28
00:02:29,030 --> 00:02:37,790
so any currency rate has its own URL, and we can make use of this to access the value for any euro.

29
00:02:38,150 --> 00:02:46,190
So for any conversion between any currencies, all we have to do is just have a string in Python or

30
00:02:46,190 --> 00:02:58,700
this URL and then change, for example, EU or to USD and the two part to I and our or other exchange

31
00:02:58,700 --> 00:02:59,390
currencies.

32
00:02:59,810 --> 00:03:06,080
And so we get the value for any currency and then we extract that value from the webpage.

33
00:03:06,200 --> 00:03:07,100
So let's do that.

34
00:03:08,430 --> 00:03:15,420
I'm going to create a new empty report, you can do this on your local I.D..

35
00:03:20,550 --> 00:03:29,910
Script currency with beautiful soup, so beautiful soup is the name of the library.

36
00:03:35,270 --> 00:03:39,090
Let's go ahead and imported from B.C. for import.

37
00:03:39,410 --> 00:03:47,730
Now best for is how you how you use beautiful soup, but the way you install it is, you go to the terminal

38
00:03:48,150 --> 00:03:55,860
or to the shell on Ripple and you see paper or paper three point nine or three points in whatever you're

39
00:03:55,860 --> 00:03:56,230
using.

40
00:03:56,700 --> 00:03:57,390
Install.

41
00:03:58,140 --> 00:04:05,430
View tea for soup for beauty, I hear.

42
00:04:08,080 --> 00:04:08,590
Execute.

43
00:04:11,530 --> 00:04:22,150
Once you have installed, it's a van you imported as from the US for import, a beautiful subclass event,

44
00:04:22,150 --> 00:04:29,890
the way I approach this is I will write a function defined gets currency and this function is going

45
00:04:29,890 --> 00:04:35,410
to get inputs, currency and an ultimate currency.

46
00:04:36,160 --> 00:04:37,090
So two arguments.

47
00:04:38,620 --> 00:04:44,370
So this will be the currency code, so you as the EU or I and cetera.

48
00:04:45,040 --> 00:04:52,840
And then inside this function, I'm going to construct a euro, which is going to be have strings.

49
00:04:54,100 --> 00:05:04,420
So basically, I'm going to copy this euro and paste this in there and then instead of you or I will

50
00:05:04,420 --> 00:05:06,660
have a placeholder.

51
00:05:06,700 --> 00:05:10,930
So let me expand this like that.

52
00:05:12,600 --> 00:05:19,080
So inside there we'll go the input currency entered by the user.

53
00:05:20,430 --> 00:05:28,230
So from that currency to instead of USD, I'm going to have alt currency, so two variables.

54
00:05:30,480 --> 00:05:31,650
So let's test that.

55
00:05:32,070 --> 00:05:41,340
I'm going to bring the URL and just call the function here, gets currency, enter, for example, I

56
00:05:41,610 --> 00:05:49,410
are here and you you as the and then execute the Python file.

57
00:05:50,810 --> 00:05:55,670
To see if the URL is being constructed correctly.

58
00:05:59,480 --> 00:06:01,160
So that's the URL we get.

59
00:06:01,970 --> 00:06:02,960
If I click that.

60
00:06:05,380 --> 00:06:06,860
We're going to get the rates here.

61
00:06:06,880 --> 00:06:11,170
Silver Page from Indian rupees to USD.

62
00:06:11,680 --> 00:06:17,770
So from Indian rupees to the US, the and that was the rate right to know that we have, you know,

63
00:06:18,790 --> 00:06:21,160
correctly constructed.

64
00:06:21,160 --> 00:06:24,900
Then we can go ahead and get the content.

65
00:06:24,900 --> 00:06:29,950
And the source code of this web page for source codes is on a web browser.

66
00:06:29,950 --> 00:06:36,340
You can find the source codes by Rudds clicking over the page and then go to show page source.

67
00:06:38,050 --> 00:06:40,390
And I need to expand this.

68
00:06:42,800 --> 00:06:47,100
So that's the source code that's constructs this Web page.

69
00:06:47,120 --> 00:06:50,870
It's a combination of HTML and CSS and JavaScript.

70
00:06:53,910 --> 00:07:02,580
So you want to access all that's called from Python, and we can do that using requests, so you need

71
00:07:02,580 --> 00:07:05,940
to import requests.

72
00:07:07,550 --> 00:07:09,920
And also install it if.

73
00:07:12,960 --> 00:07:21,000
With PIP install requests, so with requests, we're going to get access to the page.

74
00:07:23,870 --> 00:07:31,630
So the contents of the page, the source code is equal to requests that get you answer the other box

75
00:07:31,940 --> 00:07:38,090
text and if you print out content and run the script.

76
00:07:39,290 --> 00:07:41,930
We have this name error.

77
00:07:42,140 --> 00:07:49,490
I forgot the type of year you are L instead of you are so that local variable.

78
00:07:50,090 --> 00:07:55,220
And that's the source code that sees the same source code that we so hear on the browser.

79
00:07:55,520 --> 00:07:55,880
No.

80
00:07:57,710 --> 00:08:01,310
Out of that source codes we only want.

81
00:08:04,150 --> 00:08:13,720
This number, so that is hidden somewhere in the source code, if I search for that's using this button

82
00:08:13,720 --> 00:08:14,050
here.

83
00:08:15,190 --> 00:08:18,430
Zero point zero one three three.

84
00:08:20,110 --> 00:08:22,180
You see, it's somewhere in here.

85
00:08:23,230 --> 00:08:27,190
We want to get that number only from that entire source codes.

86
00:08:27,730 --> 00:08:30,130
So that number shows up when I search.

87
00:08:30,410 --> 00:08:36,400
But if you want to go directly to the number going to right click here and go to inspect element.

88
00:08:38,679 --> 00:08:40,630
Again, that would point to that.

89
00:08:42,970 --> 00:08:46,930
You may need to expand this by clicking the triangle button.

90
00:08:47,450 --> 00:08:48,640
And so that's a no.

91
00:08:48,640 --> 00:08:54,760
Again, it's hidden in between these spam elements, HMO elements.

92
00:08:56,910 --> 00:09:00,800
So I'll keep this winter open here and go back to my code.

93
00:09:01,770 --> 00:09:10,200
No, the idea is that we need to pass that source code and this is where beautiful soup comes in.

94
00:09:10,530 --> 00:09:16,440
So beautiful soup is designed to understand HCM mouth structure like this.

95
00:09:17,250 --> 00:09:22,920
It knows it has knowledge of both elements and how a natural page is constructed.

96
00:09:23,130 --> 00:09:24,090
You know, that's it.

97
00:09:24,090 --> 00:09:30,870
Has this div tags and these diff tags may have other elements, insides and so on.

98
00:09:31,110 --> 00:09:38,490
So the next step is then to have a variable.

99
00:09:38,490 --> 00:09:40,140
Usually it's called soup.

100
00:09:40,290 --> 00:09:45,600
You can have the rights, you can call it whatever you like, but that is equal to a beautiful soup.

101
00:09:45,720 --> 00:09:46,560
That's class.

102
00:09:48,120 --> 00:09:57,210
So we're making a soup here, basically, by using the contents of that, we got via the request library

103
00:09:57,210 --> 00:09:57,540
here.

104
00:09:57,840 --> 00:10:03,670
The content goes there and you specify that we're talking about an HMO.

105
00:10:04,170 --> 00:10:08,260
That's a parser that is the parser that's beautiful.

106
00:10:08,260 --> 00:10:14,430
The soup is going to use because there are other languages such as example, if a page is based on that

107
00:10:14,440 --> 00:10:21,180
language, but usually it's A. M. Parsing what we're doing, so simple parser goes in there.

108
00:10:21,930 --> 00:10:27,740
And now if you print out soup just to see what it is.

109
00:10:29,760 --> 00:10:30,180
Ron?

110
00:10:32,130 --> 00:10:34,500
So this time the function will bring soup.

111
00:10:36,090 --> 00:10:37,620
It's basically the same thing.

112
00:10:41,560 --> 00:10:44,380
But this is a more specific object.

113
00:10:44,410 --> 00:10:52,720
It's a beautiful soup object, so now internally, it has knowledge about what each of these elements

114
00:10:52,720 --> 00:10:53,290
are.

115
00:10:53,770 --> 00:10:55,000
So then you can do.

116
00:10:56,750 --> 00:11:04,130
Soup, don't find and you specify what do you want to find?

117
00:11:06,470 --> 00:11:10,400
So what we want to find is a spam element.

118
00:11:13,480 --> 00:11:14,830
That's the Spanish element.

119
00:11:15,190 --> 00:11:22,840
So it's inside this text, and no, there are different there are various Spanish elements in the source

120
00:11:22,840 --> 00:11:23,110
code.

121
00:11:23,860 --> 00:11:30,100
So we want to be more specific and say that we want the span element with a class of this value.

122
00:11:30,490 --> 00:11:36,430
So I'm copying this with Control C and going in here.

123
00:11:36,700 --> 00:11:45,310
And then you say class with underscore equal to that.

124
00:11:46,680 --> 00:11:48,000
Collapse value.

125
00:11:49,930 --> 00:11:54,370
So if we print out this ultimate, which I'm going to.

126
00:11:59,630 --> 00:12:04,820
Puts it in a very both currency and then I print out of currency and we see what we get.

127
00:12:06,260 --> 00:12:13,610
So you see that with that, we got the entire elements, including the class of the elements and the

128
00:12:13,610 --> 00:12:19,910
value that we won't know that this value is changing every certain time.

129
00:12:19,910 --> 00:12:24,980
So you may not see the same value as you saw in here a minute ago.

130
00:12:25,220 --> 00:12:26,150
So it's changing.

131
00:12:26,150 --> 00:12:26,990
It's real time.

132
00:12:28,270 --> 00:12:32,990
And so out of this, now we want to extract using the get tax methods.

133
00:12:33,260 --> 00:12:36,680
We want to extract only the text only.

134
00:12:36,680 --> 00:12:37,670
That's part.

135
00:12:38,150 --> 00:12:38,960
So Ron.

136
00:12:41,060 --> 00:12:45,680
And we get this and that is the current steroids.

137
00:12:45,950 --> 00:12:47,460
No, this is a string.

138
00:12:47,480 --> 00:12:56,930
As you may notice, it has this USB-C port, so it's good that we get this as a float without the USB

139
00:12:57,080 --> 00:12:57,710
port.

140
00:12:58,880 --> 00:13:06,650
But before we do the stripping of that string, let's first make sure that this function is working

141
00:13:06,650 --> 00:13:13,070
by trying some other values here for the currencies euros to something else.

142
00:13:13,520 --> 00:13:21,050
Let's see what we have a D for Australian dollars to run.

143
00:13:23,220 --> 00:13:26,520
Yeah, that's the current rates for that exchange.

144
00:13:27,880 --> 00:13:30,040
It seems to be working now.

145
00:13:30,070 --> 00:13:31,600
Let's clean out that.

146
00:13:32,770 --> 00:13:35,590
Strained and gets only the flow, the number.

147
00:13:36,650 --> 00:13:44,240
What we actually want is we want to remove the last four characters of the shrink, so one character

148
00:13:44,240 --> 00:13:51,560
to three and the space between the number and the currency codes for characters.

149
00:13:53,080 --> 00:13:53,680
To do that.

150
00:13:55,750 --> 00:14:03,550
What I will do is say currency or I will rename this variable, actually.

151
00:14:05,090 --> 00:14:10,130
Two reads, it makes more sense that is the rate, that's the rate.

152
00:14:11,120 --> 00:14:14,510
So the rate is going to be and the current rate.

153
00:14:15,650 --> 00:14:20,540
So we want from zero to.

154
00:14:21,660 --> 00:14:28,750
So we're splitting the string from zero means we want the character with index zero, that is the character

155
00:14:28,750 --> 00:14:37,710
who is index zero and then we want the Dot also and all the numbers, but we don't want the characters

156
00:14:37,710 --> 00:14:39,180
from minus four.

157
00:14:39,900 --> 00:14:42,780
So minus four means the space here.

158
00:14:43,050 --> 00:14:52,380
So we have minus one for D minus two for you, minus three for eight and minus four is the index of

159
00:14:52,380 --> 00:14:53,010
the space.

160
00:14:53,370 --> 00:14:59,460
So from that to that, we don't want and we don't need a zero, this will do it.

161
00:15:00,090 --> 00:15:00,780
So that's one.

162
00:15:01,960 --> 00:15:09,040
If I run this this time, we're going to get the number only, but this is Typekit string.

163
00:15:09,070 --> 00:15:18,160
If you print out the type of that's value you're going to see, it's on a C or so.

164
00:15:18,160 --> 00:15:22,060
What we want to do also is convert it into a float.

165
00:15:23,380 --> 00:15:27,080
And then you run it and then it's a float.

166
00:15:27,160 --> 00:15:34,270
Now we can do math operations with that number, so it makes sense to return the correct to the delta

167
00:15:34,270 --> 00:15:38,770
type because it's a no, it's not a string, actually.

168
00:15:39,670 --> 00:15:44,500
Um, so finally, this function should return the rate.

169
00:15:45,760 --> 00:15:52,780
That's and so if I run this, nothing is going to be printed out because we don't have a print function

170
00:15:52,780 --> 00:15:53,650
anymore here.

171
00:15:53,920 --> 00:16:01,210
So if you want to free notes something now, you don't put print functions in the function in your custom

172
00:16:01,210 --> 00:16:01,740
functions.

173
00:16:01,750 --> 00:16:03,280
That's not a good practice.

174
00:16:04,090 --> 00:16:10,060
We did that because we were developing the function and we were looking at the intermediate results

175
00:16:10,060 --> 00:16:12,580
we were getting, but now we want to return to that.

176
00:16:13,240 --> 00:16:16,180
And then if you want to print, you say prints here.

177
00:16:17,080 --> 00:16:19,300
So print the value of that function.

178
00:16:20,770 --> 00:16:22,000
And to get that value.

179
00:16:22,090 --> 00:16:24,250
Alternatively, you can see

180
00:16:27,490 --> 00:16:36,490
current rates, a variable that is equal to vets, so current rates is now a variable that has the actual

181
00:16:36,940 --> 00:16:40,210
clothes, the actual rates of the currency.

182
00:16:41,380 --> 00:16:43,240
You can print outs, current rates.

183
00:16:45,040 --> 00:16:46,510
And you get the same output.

184
00:16:46,960 --> 00:16:48,280
So that's the final codes.

185
00:16:49,540 --> 00:16:56,440
Now later in the course, we will build a desktop application with a graphical user interface where

186
00:16:56,440 --> 00:16:59,320
the user can choose from.

187
00:16:59,320 --> 00:17:06,140
A dropdown list can choose the currency rates, the source currency rate and the target currency rates.

188
00:17:06,160 --> 00:17:07,930
Also in another dropdown list.

189
00:17:08,290 --> 00:17:16,300
And then they can enter a value and then click a button and then get the converted rate in the target's

190
00:17:16,300 --> 00:17:18,760
its currency in the user interface.

191
00:17:18,970 --> 00:17:21,609
So it's a real desktop program.

192
00:17:22,150 --> 00:17:29,650
We will build this ain't the gooey section where we built applications for the down in the course.

193
00:17:30,940 --> 00:17:33,700
So that's something exciting that I wanted you to know.

194
00:17:34,030 --> 00:17:35,990
And I'll talk to you later.

195
00:17:36,010 --> 00:17:36,390
Thank you.