1
00:00:00,150 --> 00:00:00,870
In this video

2
00:00:00,900 --> 00:00:10,020
we're going to scrape this text from this website, automated.pythonanywhere.com.

3
00:00:10,470 --> 00:00:18,960
So that's the current webpage is the homepage and we will be using selenium for this.

4
00:00:19,050 --> 00:00:28,410
So selenium is a Python library, a third party python library that is used to automate browser actions.

5
00:00:28,920 --> 00:00:35,640
That includes things like clicking on a link and then on the next page doing something

6
00:00:35,640 --> 00:00:38,130
else, such as copying some text.

7
00:00:38,430 --> 00:00:40,630
Or perhaps you want to log in.

8
00:00:40,650 --> 00:00:43,000
You're going to learn that's doing the next videos.

9
00:00:43,500 --> 00:00:50,400
So you can write a Python script to log into this website, enter the username and password

10
00:00:50,430 --> 00:00:52,470
of course, everything is automated.

11
00:00:52,770 --> 00:00:58,890
Selenium and Python will do that through that Python script and then do something else on the next web

12
00:00:58,890 --> 00:01:00,840
page after the login and so on.

13
00:01:01,410 --> 00:01:05,209
But for this particular video, we're going to do something simple.

14
00:01:05,220 --> 00:01:09,270
So just scrape this text with selenium.

15
00:01:09,390 --> 00:01:15,690
So which means that we're going to get the text and load it in our Python program and maybe print it

16
00:01:15,690 --> 00:01:19,200
out, or you can do whatever you want then with that text.

17
00:01:19,860 --> 00:01:21,570
So let's go to Repl.

18
00:01:22,170 --> 00:01:28,410
And here you can see that I have created an empty folder for this section.

19
00:01:28,410 --> 00:01:30,900
So browser automation and web scraping.

20
00:01:31,260 --> 00:01:41,730
And then I'm going to create a new Repl in this folder using Python and let's name it Scrape Simple

21
00:01:42,300 --> 00:01:43,380
Text with...

22
00:01:45,150 --> 00:01:45,510
With

23
00:01:47,040 --> 00:01:49,620
Selenium, create Repl.

24
00:01:52,760 --> 00:01:55,190
And so this is our empty Repl.

25
00:01:55,190 --> 00:01:58,070
It's a good idea to install selenium beforehand.

26
00:01:59,360 --> 00:02:00,510
Selenium,

27
00:02:01,040 --> 00:02:07,500
like that, press the plus icon and wait until the installation completes.

28
00:02:07,520 --> 00:02:15,740
Sometimes though the installation can hang, so you'll see that these messages will not disappear.

29
00:02:15,860 --> 00:02:21,710
If that's the case, you might want to reload the browser and see here if selenium has been installed,

30
00:02:21,710 --> 00:02:26,600
if it's listed here, if not, try again until it is installed.

31
00:02:26,990 --> 00:02:33,680
And always you can also just import selenium here and when you run the script it will install automatically.

32
00:02:33,680 --> 00:02:34,940
So you have these two options.

33
00:02:34,940 --> 00:02:38,090
Use the packages menu or directly import.

34
00:02:39,170 --> 00:02:41,720
My installation was completed.

35
00:02:43,970 --> 00:02:50,240
And now before we import Selenium and start writing the code, we need to do another configuration.

36
00:02:50,590 --> 00:02:56,090
But you need to do this configuration only if you created a new Repl.

37
00:02:56,240 --> 00:03:01,700
If you forked my Repl, which is attached in the lecture resources as a link,

38
00:03:02,150 --> 00:03:04,910
you don't need to do this step.

39
00:03:05,240 --> 00:03:07,580
So for those who created a new Repl,

40
00:03:07,940 --> 00:03:13,880
you can go to these three dots and go to show hidden files and you'll see these two files.

41
00:03:14,570 --> 00:03:19,100
If you fork my Repl, you will not see this files when you show the hidden files.

42
00:03:19,760 --> 00:03:24,440
So this is only for those people, again, who created a new Repl.

43
00:03:25,070 --> 00:03:26,690
So you need to go to 

44
00:03:26,690 --> 00:03:27,530
replit.nix.

45
00:03:30,100 --> 00:03:41,740
And then you need to go in here, make a new line and add pkgs.chromium and then

46
00:03:41,740 --> 00:03:46,300
again pkgs.chromedriver.

47
00:03:47,020 --> 00:03:53,800
So we need these to packages to be able to scrape data with selenium.

48
00:03:55,150 --> 00:03:56,290
So close that file.

49
00:03:57,500 --> 00:04:00,340
You can then hide those files again.

50
00:04:00,350 --> 00:04:04,340
So you'll see only those working files of your projects.

51
00:04:05,690 --> 00:04:08,960
And now we can go ahead and import selenium here.

52
00:04:09,440 --> 00:04:15,770
But what I'll do is I'll just import from selenium

53
00:04:17,300 --> 00:04:18,290
the webdriver.

54
00:04:19,070 --> 00:04:28,520
Web driver is basically a tool which allows us to instruct the behavior of the web browser.

55
00:04:29,330 --> 00:04:36,980
So the web driver makes everything possible, and we have access to that web driver from Python through

56
00:04:37,010 --> 00:04:37,970
selenium.

57
00:04:38,540 --> 00:04:45,650
And the first thing, the first logical thing to do is to create a driver variable and use webdriver.

58
00:04:45,860 --> 00:04:47,930
Actually, we don't use the webdriver.

59
00:04:47,930 --> 00:04:53,120
We use the Chrome class of the webdriver.

60
00:04:53,810 --> 00:05:01,970
Normally, Chrome, this class expects the executable path of a chrome driver, which you can download

61
00:05:01,970 --> 00:05:03,530
and install from the web.

62
00:05:04,490 --> 00:05:09,530
So that's the chrome, the driver for the Google Chrome Web browser.

63
00:05:09,770 --> 00:05:12,710
Every Web browser have a different driver.

64
00:05:13,310 --> 00:05:22,850
However, here on Repl, you don't have to provide that path to an executable because it's automatically

65
00:05:22,850 --> 00:05:31,280
configured, which makes our programming job easier so we don't have to go through downloads and configuration

66
00:05:31,790 --> 00:05:32,450
actions.

67
00:05:32,780 --> 00:05:33,890
So just like that.

68
00:05:35,990 --> 00:05:44,660
And while this is the minimum thing you can do to configure a driver, there are other things, other

69
00:05:44,660 --> 00:05:47,090
options that you add to your driver.

70
00:05:47,540 --> 00:05:49,250
So this is what I'm going to do now.

71
00:05:50,000 --> 00:05:57,200
I'm going to create some options using webdriver.ChromeOptions.

72
00:05:57,650 --> 00:06:01,330
So this can be an empty class call.

73
00:06:01,670 --> 00:06:04,490
So we call the class, but no need to provide any arguments.

74
00:06:05,090 --> 00:06:07,670
And so this is an instance of this.

75
00:06:07,970 --> 00:06:12,800
Now it's the ChromeOptions object, and in that options

76
00:06:13,130 --> 00:06:15,800
we add arguments.

77
00:06:19,040 --> 00:06:24,680
These are basically flags and these are what I was referring as the options.

78
00:06:25,190 --> 00:06:30,470
The first flag we want to add is disable-infobars.

79
00:06:32,410 --> 00:06:34,330
So disable dash infobars.

80
00:06:34,690 --> 00:06:42,550
What this will do is on a browser sometimes we have these bars here which display some information

81
00:06:42,550 --> 00:06:47,260
such as of a memory is low or things like that.

82
00:06:47,650 --> 00:06:56,050
Now those bars may interfere with the script accessing, taking actions on the browser.

83
00:06:56,050 --> 00:07:04,060
So the script may want to click somewhere, but that popup, that info bar could confuse the cursor

84
00:07:04,060 --> 00:07:04,870
and so on.

85
00:07:05,290 --> 00:07:07,270
So we add that's to disable them.

86
00:07:07,780 --> 00:07:12,280
Other options that is good to have is to add arguments.

87
00:07:13,600 --> 00:07:17,190
The next one is start

88
00:07:19,120 --> 00:07:19,930
maximized.

89
00:07:21,040 --> 00:07:25,270
So that will start the browser as maximized.

90
00:07:26,200 --> 00:07:33,490
That's also a good thing because some web pages, for example, may change the contents when you resize

91
00:07:33,490 --> 00:07:35,110
when you make them smaller.

92
00:07:35,830 --> 00:07:40,000
So we want to access the maximized version of the browser.

93
00:07:41,500 --> 00:07:42,550
Another one is.

94
00:07:46,950 --> 00:07:47,850
Disable-dev-shm

95
00:07:50,610 --> 00:07:54,390
-usage.

96
00:07:56,040 --> 00:08:04,140
This particular option is to avoid some issues that occur when you interact with a browser on a Linux

97
00:08:04,140 --> 00:08:04,590
computer.

98
00:08:04,590 --> 00:08:11,400
And this here or Repl is a Linux machine, so it's good to have that to avoid some particular issues.

99
00:08:13,440 --> 00:08:18,540
The next one is a let me be consistent here with the quotes.

100
00:08:22,950 --> 00:08:24,030
Just for consistency.

101
00:08:24,900 --> 00:08:28,920
And next, no-sandbox.

102
00:08:30,390 --> 00:08:32,850
Now browsers for security.

103
00:08:32,850 --> 00:08:35,820
They use something called sandboxing.

104
00:08:36,240 --> 00:08:45,180
And if we want our program, our scripts to have greater privileges on that particular web page that

105
00:08:45,180 --> 00:08:51,690
the script is going to access and scrape, we want to add this option so that sandboxing is disabled.

106
00:08:54,720 --> 00:09:03,510
Next, we have some experimental options which are new and we have exclude.

107
00:09:06,190 --> 00:09:10,120
Switches and in a lists

108
00:09:11,560 --> 00:09:12,850
comes enable

109
00:09:14,500 --> 00:09:15,710
automation.

110
00:09:16,970 --> 00:09:20,810
Now, this option and another one that I'm going to add here.

111
00:09:23,640 --> 00:09:30,030
So these experimental options are options that help selenium to avoid detection from the browser.

112
00:09:30,090 --> 00:09:34,020
So some browsers, web pages, they don't like scripts.

113
00:09:34,020 --> 00:09:38,970
So we enable our scripts to access those browsers.

114
00:09:40,110 --> 00:09:46,920
So if the other one is not actually that. It's an add-argument option, it's not an experimental one.

115
00:09:47,490 --> 00:09:50,790
So this is disable-blink-

116
00:09:52,570 --> 00:09:53,860
features=.

117
00:09:55,700 --> 00:10:01,430
Equal to out to AutomationControlled.

118
00:10:04,400 --> 00:10:05,350
And that's it.

119
00:10:05,360 --> 00:10:07,370
So these are the options.

120
00:10:09,200 --> 00:10:14,090
Set options to make browsing easier.

121
00:10:14,760 --> 00:10:15,110
Right.

122
00:10:15,380 --> 00:10:21,380
And then we need to provide that options variable in here.

123
00:10:21,860 --> 00:10:25,860
So options now has all these arguments.

124
00:10:25,880 --> 00:10:28,130
This variable contains all those arguments.

125
00:10:29,870 --> 00:10:40,520
Now, what I would prefer to do here is to just select all these, indent them with a tab and then put

126
00:10:40,520 --> 00:10:45,260
them inside a function which I'd like to name get_driver.

127
00:10:45,860 --> 00:10:49,370
So I'm trying to keep things organized here.

128
00:10:49,820 --> 00:10:56,930
So this function is responsible to create a driver and return that driver.

129
00:10:58,280 --> 00:10:58,820
That's good.

130
00:10:58,940 --> 00:11:01,160
However, the driver is currently empty.

131
00:11:01,160 --> 00:11:03,170
It's not connected to a webpage.

132
00:11:03,380 --> 00:11:07,190
So we want to connect it to a web page just inside the function.

133
00:11:07,520 --> 00:11:15,680
So driver.get and here this gets the URL, so the URL is this web page, right?

134
00:11:15,680 --> 00:11:20,360
So we said that we want to scrape this text and that text is in this web page.

135
00:11:20,660 --> 00:11:23,630
Therefore, we provide that web page as a string in here.

136
00:11:24,110 --> 00:11:26,180
So, it starts with HTTP and so on.

137
00:11:27,380 --> 00:11:31,970
Now we return the driver for this particular web page.

138
00:11:33,230 --> 00:11:37,610
And then I would like to create another function.

139
00:11:39,110 --> 00:11:40,910
So let's name this main.

140
00:11:42,680 --> 00:11:46,880
And here I'm going to get the driver from get_driver.

141
00:11:47,870 --> 00:11:49,490
So we call that function.

142
00:11:51,250 --> 00:11:58,450
Therefore this variable now holds this driver because that function returns that.

143
00:11:58,450 --> 00:11:59,620
So it makes sense.

144
00:12:00,580 --> 00:12:06,250
And with that driver, now we extract an element from that driver.

145
00:12:06,260 --> 00:12:07,150
So driver dot find

146
00:12:08,510 --> 00:12:13,970
element by X path which is done here.

147
00:12:14,750 --> 00:12:15,170
Yes.

148
00:12:16,650 --> 00:12:20,220
And this gets the X path as a string.

149
00:12:20,700 --> 00:12:22,020
What is the x path?

150
00:12:22,620 --> 00:12:23,700
Well, let's find that out.

151
00:12:24,330 --> 00:12:27,030
So we want the x path of this text, right?

152
00:12:27,690 --> 00:12:32,580
Therefore, we select somewhere over here and go to inspect element.

153
00:12:38,480 --> 00:12:46,850
You may need to enable your browser developer options if this inspect element options doesn't show up.

154
00:12:46,970 --> 00:12:48,860
So you have to maybe google something.

155
00:12:48,860 --> 00:12:57,230
How to show inspect menu on your particular browser, which could be Firefox, Chrome or Safari, etc..

156
00:12:58,010 --> 00:13:02,180
Anyway, so a web page is made of HTML elements.

157
00:13:02,180 --> 00:13:06,500
For example, this  element that you see here.

158
00:13:10,050 --> 00:13:12,900
This creates this text.

159
00:13:13,480 --> 00:13:18,630
 means the text is quite big, the font.

160
00:13:19,050 --> 00:13:26,100
And if I click over here, you see that the text is a inside this  tag.

161
00:13:26,250 --> 00:13:30,150
You see  here, slash  here.

162
00:13:30,540 --> 00:13:32,760
So these tags, html tags

163
00:13:33,240 --> 00:13:42,970
hold this text and if I click over this tag and I go to copy and I go to X path, I just copied the

164
00:13:42,970 --> 00:13:46,650
Xpath of that text and if I pasted there,

165
00:13:49,270 --> 00:13:53,800
you see, we're going to get this string.

166
00:13:53,800 --> 00:13:55,560
So that's what I copied from there.

167
00:13:56,970 --> 00:13:57,270
Now,

168
00:14:00,140 --> 00:14:05,450
let's return to this element and see what it is made of, how it looks like.

169
00:14:05,960 --> 00:14:11,480
And therefore I'm going to print the call of the main function.

170
00:14:12,260 --> 00:14:13,610
So I'm calling main here.

171
00:14:14,000 --> 00:14:14,330
Right.

172
00:14:14,480 --> 00:14:19,550
Print main with a print to print out that element and run.

173
00:14:22,370 --> 00:14:24,010
It seems that we have an error.

174
00:14:24,020 --> 00:14:26,690
So let's look at it. Experienced string,

175
00:14:26,690 --> 00:14:29,900
bytes or path like object, not options.

176
00:14:30,890 --> 00:14:38,570
So the error started in line 22 when we called the function and it changed back to line 18 when we get

177
00:14:38,570 --> 00:14:42,800
the driver, which goes back to line 13, options.

178
00:14:42,800 --> 00:14:46,670
And that's where the roots of error is in our script.

179
00:14:47,120 --> 00:14:48,710
Then we have library errors here.

180
00:14:49,400 --> 00:14:52,790
So line 13 is these options?

181
00:14:54,020 --> 00:14:58,880
So it seems that options is actually a positional argument.

182
00:14:58,890 --> 00:15:03,320
So we have to mention it's name options equals two options.

183
00:15:03,530 --> 00:15:05,780
So the name of the arguments and the variable,

184
00:15:06,320 --> 00:15:07,100
this one in here.

185
00:15:07,850 --> 00:15:08,060
Right.

186
00:15:08,090 --> 00:15:08,990
So let's run again.

187
00:15:09,380 --> 00:15:10,760
And this is the output.

188
00:15:10,970 --> 00:15:12,830
So that's the element.

189
00:15:14,190 --> 00:15:14,670
This one.

190
00:15:15,930 --> 00:15:17,070
And we also have.

191
00:15:18,090 --> 00:15:27,030
A warning, deprecation warning which means this find element by, so this methods has been

192
00:15:28,140 --> 00:15:36,540
deprecated, which means it's a good idea to remove it because in the later versions of selenium,

193
00:15:36,540 --> 00:15:38,490
it's going to be removed as a method.

194
00:15:38,880 --> 00:15:44,340
So we can take measures now and use what this suggests, find element.

195
00:15:44,760 --> 00:15:50,130
So let's do find elements which gets this two arguments by and value.

196
00:15:50,520 --> 00:15:56,490
So I think without even looking at the documentation of this value, I think what they want us to

197
00:15:56,490 --> 00:15:57,570
do is say

198
00:16:00,560 --> 00:16:01,100
by xPath.

199
00:16:02,180 --> 00:16:10,100
So we declare that we are accessing this elements using the X path options and not other options

200
00:16:11,630 --> 00:16:15,290
such as fines, element by text or by class, etc..

201
00:16:16,830 --> 00:16:21,330
So by xpath is good, and this would be value.

202
00:16:22,860 --> 00:16:25,500
So yeah, that should do it, run again.

203
00:16:27,800 --> 00:16:30,230
And here is a browser also.

204
00:16:31,370 --> 00:16:32,630
It's being access.

205
00:16:32,630 --> 00:16:36,170
We don't see anything now because this is just a simple operation.

206
00:16:36,950 --> 00:16:38,600
So that's the element.

207
00:16:39,590 --> 00:16:39,920
Right.

208
00:16:40,460 --> 00:16:45,620
And to give the actual texts, you can just do element.text here in the return and run.

209
00:16:50,240 --> 00:16:50,400
Yeah.

210
00:16:50,920 --> 00:16:52,600
Here is the text.

211
00:16:53,880 --> 00:16:55,560
So that's one in there.

212
00:16:56,970 --> 00:17:02,130
And this concludes this video, so you learn how to set up selenium.

213
00:17:03,150 --> 00:17:07,200
It was a long one, probably because we have to configure lots of options.

214
00:17:07,410 --> 00:17:17,339
But the good news is that you going to use this function just as it is in future videos of this section.

215
00:17:17,730 --> 00:17:19,230
So I'll talk to you in the next video.

216
00:17:19,410 --> 00:17:19,740
See you.