1
00:00:01,380 --> 00:00:05,250
In the previous video, we scraped this text with selenium.

2
00:00:05,580 --> 00:00:10,760
In this video, we're going to scrape this value here, which is dynamic.

3
00:00:10,770 --> 00:00:13,570
You can see it's changing as I'm talking.

4
00:00:14,700 --> 00:00:19,710
So this value is just a random number that is being generated by this website.

5
00:00:20,070 --> 00:00:29,010
So it's not the actual average world temperature right now, but for education purposes, we can suppose

6
00:00:29,010 --> 00:00:38,690
that we can pretend that this is some real world value, right? And we want to scrape that. Later on,

7
00:00:38,710 --> 00:00:45,240
we're going to scrape that every five seconds or so and store that value in a text file.

8
00:00:45,480 --> 00:00:48,240
So each value is going to be inside a text file.

9
00:00:48,570 --> 00:00:55,680
Or you could also add it's in the same text file, so create a table, the row containing each value.

10
00:00:56,430 --> 00:01:02,010
But for now, we're just going to scrape one single value one time.

11
00:01:02,340 --> 00:01:03,390
So let's do that.

12
00:01:04,680 --> 00:01:06,740
I'm going to fork this repl.

13
00:01:06,780 --> 00:01:12,540
So to duplicate it, the previous one, since most of the code is the same.

14
00:01:16,820 --> 00:01:18,890
So what do we have to change here?

15
00:01:19,250 --> 00:01:24,050
Well, I think we have to change the value, so the x-path.

16
00:01:26,080 --> 00:01:27,280
Therefore, let's go here.

17
00:01:27,560 --> 00:01:30,310
Right click over this, go to inspect element.

18
00:01:31,720 --> 00:01:42,640
And this should be hidden somewhere so you can scroll aren't to see that this part here is highlighted,

19
00:01:43,750 --> 00:01:44,890
so it's somewhere here.

20
00:01:46,090 --> 00:01:47,470
If we expand that.

21
00:01:49,070 --> 00:01:56,320
So that is h1, the text, and the next one is h, the next one is this one in here.

22
00:01:56,840 --> 00:02:04,790
So if you expand that, you will see average world temperature now 26, and it changes to 20, 21.

23
00:02:05,450 --> 00:02:11,930
And so one, every 30 seconds or so, you want to click over that div, right?

24
00:02:12,770 --> 00:02:15,140
Go to copy X path.

25
00:02:16,740 --> 00:02:19,230
And then paste it in here.

26
00:02:19,830 --> 00:02:23,400
So as the value of the value argument.

27
00:02:23,760 --> 00:02:29,970
No, this doesn't look right to me because it's just slash div.

28
00:02:30,300 --> 00:02:38,550
What we had before was this slash html slash body slash div one index, div h one index one.

29
00:02:38,550 --> 00:02:43,830
And that was the X path for the text, for this text.

30
00:02:44,310 --> 00:02:49,380
So it seems to me like things are not working this time as they are supposed to.

31
00:02:49,950 --> 00:03:00,360
So it's not always as straightforward as just clicking over an element and getting the X path as it

32
00:03:00,360 --> 00:03:02,730
was in our previous video.

33
00:03:03,630 --> 00:03:11,850
Sometimes you will not get the X path because the HTML structure could be more complicated as it is in

34
00:03:11,850 --> 00:03:12,360
this case.

35
00:03:13,200 --> 00:03:19,950
So in that case, we want to use our logic, our knowledge about what this x-path is.

36
00:03:20,400 --> 00:03:24,690
So let's look at this previous X path.

37
00:03:25,800 --> 00:03:26,130
Right.

38
00:03:26,700 --> 00:03:33,990
So I'm going to keep it somewhere in here and explain what this means.

39
00:03:34,860 --> 00:03:40,170
So you see slash html. Slash html means this document.

40
00:03:40,530 --> 00:03:40,830
Right?

41
00:03:40,920 --> 00:03:48,090
So this is like a hierarchy slash email slash body. Slash body is this main tag here.

42
00:03:49,410 --> 00:03:55,280
Now this tag, this element or tag whatever you would like to call it, contains other tags.

43
00:03:55,290 --> 00:03:57,600
It contains a div tag.

44
00:03:57,990 --> 00:04:03,060
You see here this div? But we have this div.

45
00:04:05,180 --> 00:04:06,140
Here, right?

46
00:04:07,890 --> 00:04:15,090
And we also have another dive in this level, so under body, we have one div, we have two divs.

47
00:04:16,570 --> 00:04:27,580
Therefore, this one here means we are getting the first div, so that one, and not that one.

48
00:04:28,400 --> 00:04:31,210
Then if we expand the first div.

49
00:04:34,350 --> 00:04:37,380
You see that we have another div inside, right?

50
00:04:37,710 --> 00:04:38,490
There's one in here.

51
00:04:39,780 --> 00:04:42,370
So html, body?

52
00:04:43,470 --> 00:04:51,630
This is the body, div one, this div one, not the other one which was under here somewhere.

53
00:04:52,140 --> 00:04:54,450
And then we have this div.

54
00:04:54,450 --> 00:04:55,770
So that one.

55
00:04:56,250 --> 00:04:58,410
And then we have h1.

56
00:04:58,890 --> 00:05:01,710
So this h1, the first one.

57
00:05:02,730 --> 00:05:05,370
Not this second h1.

58
00:05:06,090 --> 00:05:09,780
Now this was the case for that text, right?

59
00:05:10,680 --> 00:05:11,160
That one.

60
00:05:12,920 --> 00:05:19,580
For this thing here, for this value, we want to get the second h2, therefore we want to change this

61
00:05:20,120 --> 00:05:20,910
to two.

62
00:05:21,650 --> 00:05:23,330
That's all we have to do.

63
00:05:24,530 --> 00:05:27,450
So just change that to two.

64
00:05:28,450 --> 00:05:30,010
And run,

65
00:05:33,430 --> 00:05:37,960
and as we wait for the execution to finish, I I'd like to give you a tip.

66
00:05:38,410 --> 00:05:47,170
So if you don't want to use this hard way or finding out the x-path of an element as we did here,

67
00:05:47,170 --> 00:05:52,330
so we tried to use some logic, some thinking of the hierarchy, how it works.

68
00:05:52,600 --> 00:05:57,640
So if you want an easier way, you should use Google Chrome, Google Chrome browser.

69
00:05:57,880 --> 00:06:03,280
So here I am on Google Chrome and if I right click over here and go to inspect.

70
00:06:04,300 --> 00:06:09,800
And so it should show you the elements for that.

71
00:06:09,820 --> 00:06:19,390
But if it doesn't try to figure out where it is by expanding these elements, and then so this is the

72
00:06:19,390 --> 00:06:19,940
one, right?

73
00:06:19,960 --> 00:06:21,700
H1, the second h1.

74
00:06:21,880 --> 00:06:28,930
And then you want to right-click and go to copy and then go to copy full x-path, not copy x-path.

75
00:06:28,930 --> 00:06:30,730
So copy full x-path.

76
00:06:31,390 --> 00:06:35,470
And let's see now what I have on my clipboard.

77
00:06:36,550 --> 00:06:41,560
You see, it's the same x path that we constructed

78
00:06:41,560 --> 00:06:43,060
manually ourselves.

79
00:06:43,660 --> 00:06:45,520
So Google Chrome is an easier option.

80
00:06:45,980 --> 00:06:51,100
Let's go back to repl now to see if the script has finished running.

81
00:06:57,660 --> 00:07:05,930
Now we see the browser, so selenium is accessing it, and it got the average world temperature now.

82
00:07:05,940 --> 00:07:13,830
The text, but not the value. It's didn't get the value because see what's happened when we just load

83
00:07:13,830 --> 00:07:14,460
the browser

84
00:07:14,820 --> 00:07:20,730
after about two seconds, the value appears, so it's not appearing immediately.

85
00:07:21,090 --> 00:07:31,020
In those cases, what you want to do is you want to use the time module and before you extract the elements,

86
00:07:31,020 --> 00:07:35,270
you want to stop the script for two seconds maybe.

87
00:07:35,310 --> 00:07:45,030
So time.sleep, so we get the driver right for this particular web page, and that means

88
00:07:45,030 --> 00:07:47,310
we are loading the page.

89
00:07:47,640 --> 00:07:53,040
Here we have opened the page at this point, but then we stay on the page for two seconds.

90
00:07:53,740 --> 00:07:59,340
Then we look at this particular element, right?

91
00:07:59,610 --> 00:08:00,210
Let's try.

92
00:08:05,020 --> 00:08:12,430
You see the script waited a bit, and then it extracted that text together with the value.

93
00:08:12,700 --> 00:08:15,850
Now you say, Oh, I just want the value, 22.

94
00:08:16,090 --> 00:08:17,770
In this case, I don't want the entire text.

95
00:08:18,130 --> 00:08:22,900
For that, we need to do some manipulation of the text.

96
00:08:23,800 --> 00:08:33,010
What I'd do is not work on this function, but create a new function instead, which is only responsible

97
00:08:33,010 --> 00:08:34,809
for cleaning out that text.

98
00:08:35,830 --> 00:08:38,679
So you could sort of write that function somewhere in here.

99
00:08:39,100 --> 00:08:43,030
Define clean_text, perhaps.

100
00:08:43,690 --> 00:08:53,170
And so this gets as inputs, maybe a text parameter, and let's give it a doc string.

101
00:08:53,190 --> 00:09:04,060
So, an explanation what this function does. Extract only the temperature from text.

102
00:09:05,910 --> 00:09:07,140
With thriple quotes.

103
00:09:07,950 --> 00:09:13,050
And then the output is going to be text dot.

104
00:09:13,500 --> 00:09:14,840
So what can we do?

105
00:09:14,850 --> 00:09:15,960
How can we split this?

106
00:09:16,320 --> 00:09:20,400
Yeah, we can split it using a split method.

107
00:09:20,880 --> 00:09:28,200
So text strings have a split method, which could split a string into two parts.

108
00:09:28,860 --> 00:09:30,000
We want to split it

109
00:09:31,200 --> 00:09:31,530
at

110
00:09:32,490 --> 00:09:39,120
this character, so there is a column and there is a space.

111
00:09:40,830 --> 00:09:48,540
Now this will give us a list because, you know, if you split some text.

112
00:09:48,720 --> 00:09:50,070
So let's get this text.

113
00:09:50,850 --> 00:09:54,210
Put it there and apply split.

114
00:09:56,010 --> 00:09:56,730
Like that?

115
00:09:57,090 --> 00:09:58,440
Yes, that's what we're doing.

116
00:09:59,070 --> 00:10:03,360
You get the list of two items, the first item is the first part.

117
00:10:03,750 --> 00:10:10,590
The second is the second part and you don't get this column and the space in any of these because that's

118
00:10:10,620 --> 00:10:10,860
the

119
00:10:12,090 --> 00:10:18,390
split character, so out of that list, we want the second item, which has an index of one.

120
00:10:18,660 --> 00:10:20,610
So this one has an index of one.

121
00:10:21,000 --> 00:10:25,170
And you see this is string, so I'm going to convert it into a float.

122
00:10:27,240 --> 00:10:34,080
Like that, you could also use an integer here, so instead of float, but this is more inclusive.

123
00:10:34,080 --> 00:10:42,300
So if you if this website decides to show decimal points numbers instead of introducing the future,

124
00:10:42,720 --> 00:10:45,330
then your script, it would still work.

125
00:10:46,620 --> 00:10:51,420
So let's convert it into a float and then this is the function.

126
00:10:51,810 --> 00:11:01,720
Now here, instead of returning element.text, we return the cleaned version of that element

127
00:11:01,740 --> 00:11:02,190
dot text.

128
00:11:03,890 --> 00:11:05,510
So that makes sense.

129
00:11:05,710 --> 00:11:07,610
Oh, don't forget to return output.

130
00:11:08,780 --> 00:11:09,140
Right?

131
00:11:10,370 --> 00:11:19,970
So what's happening here is that this function is called in here and element.text is going to provide

132
00:11:20,110 --> 00:11:23,630
this argument with input.

133
00:11:23,930 --> 00:11:27,470
So elemental text goes there, the entire text and then.

134
00:11:28,520 --> 00:11:34,340
That text is split into a list and converted into a float, stored in this variable, this variable is

135
00:11:34,340 --> 00:11:35,180
returned in here.

136
00:11:35,570 --> 00:11:36,230
We have an error.

137
00:11:37,070 --> 00:11:39,560
It should be output, right?

138
00:11:40,920 --> 00:11:43,560
And so output then is returned in here.

139
00:11:43,890 --> 00:11:53,160
So, this entire thing is equal to that output run and see what we will have as output.

140
00:11:57,480 --> 00:12:02,250
Wow, good, we made it, so that's the complete script.

141
00:12:02,940 --> 00:12:07,670
Thank you so much for following, and I'm happy to talk to you in the next videos.

142
00:12:07,890 --> 00:12:08,460
See you there!