问题:如何在 Julia 中进行网页抓取?

我想从这个站点中提取大学及其网站的名称到列表中。

在 Python 中,我使用 BeautifulSoup v4 做到了:

import requests
from bs4 import BeautifulSoup
import pandas as pd

page = requests.get('https://thebestschools.org/features/best-computer-science-programs-in-the-world/')
content = BeautifulSoup(page.text, 'html.parser')

college_name = []
college_link = []
college_name_list = content.find_all('h3',class_='college')
for college in college_name_list:
    if college.find('a'):
        college_name.append(college.find('a').text)
        college_link.append(college.find('a')['href'])

我真的很喜欢用 Julia 编程,因为它与 Python 非常相似,所以我想知道我是否也可以在 Julia 中进行网络抓取。任何帮助,将不胜感激。

解答

你的 python 代码不太好用。我猜这个网站最近更新了。据我所知,他们已经删除了链接。这是使用Gumbo.jl和Cascadia.jl的类似示例。

我正在使用内置的download命令下载网页。它将它写入临时文件中的磁盘,然后我将其读入字符串。使用HTTP.jl可能会更干净,它可以直接将其读入字符串。但是对于这个简单的例子来说很好

using Gumbo
using Cascadia

url = "https://thebestschools.org/features/best-computer-science-programs-in-the-world/"

page = parsehtml(read(download(url), String))


college_name = String[]
college_location = String[]


sections = eachmatch(sel"section", page.root)
for section in sections
    maybe_col_heading = eachmatch(sel"h3.college", section)
    if length(maybe_col_heading) == 0
        continue
    end
    col_heading = first(maybe_col_heading)

    name = strip(text(last(col_heading.children)))
    push!(college_name, name)

    loc = first(eachmatch(sel".school-location", section))
    push!(college_location, text(loc[1]))
end


[college_name college_location]

输出

julia> [college_name college_location]
51×2 Array{String,2}:
 "Massachusetts Institute of Technology (MIT)"  "Cambridge, Massachusetts"
 "Massachusetts Institute of Technology (MIT)"  "Cambridge, Massachusetts"
 "Stanford University"                          "Stanford, California"
 "Carnegie Mellon University"                   "Pittsburgh, Pennsylvania"
 ⋮

 "Shanghai Jiao Tong University"                "Shanghai, China"
 "Lomonosov Moscow State University"            "Moscow, Russia"
 "City University of Hong Kong"                 "Hong Kong"

似乎它两次列出了麻省理工学院。可能我演示中的过滤代码并不安静。但是 :shrug: 我听说 MIT 是一所很棒的大学。 Julia 是在那里发明的 :joy: