问题:使用带有 rvest 的“加载更多”按钮发出抓取页面

我想获取此页面上列出的自动取款机的链接:https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/

我需要对页面底部的“加载更多”按钮做些什么吗?

我一直在使用可以为 chrome 下载的选择器工具来选择 CSS 路径。

我编写了下面的代码块,它似乎只检索前十个链接。

library(rvest)

base <- "https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/"
base_read <- read_html(base)
atm_urls <- html_nodes(base_read, ".place > a")
all_urls_final <- html_attr(atm_urls, "href" )
print(all_urls_final)

我希望能够检索到该区域中列出的 atms 的所有链接,但我的 R 代码没有这样做。

任何帮助都会很棒。对不起,如果这是一个非常简单的问题。

解答

你应该试试RSelenium。我可以使用以下代码获取链接:

# install.packages("RSelenium")
library(RSelenium)
library(rvest)

# Download binaries, start driver, and get client object.
rd <- rsDriver(browser = "firefox", port = 4444L)
ffd <- rd$client

# Navigate to page.
ffd$navigate("https://coinatmradar.com/city/345/bitcoin-atm-birmingham-uk/")

# Find the load button and assign, then send click event.
load_btn <- ffd$findElement(using = "css selector", ".load-more .btn")
load_btn$clickElement()

# Wait for elements to load.
Sys.sleep(2)

# Get HTML data and parse
html_data <- ffd$getPageSource()[[1]]
html_data %>% 
    read_html() %>% 
    html_nodes(".place a:not(.operator-link)") %>% 
    html_attr("href")

#### OUTPUT ####

#  [1] "/bitcoin_atm/5969/bitcoin-atm-shitcoins-club-birmingham-uk-bitcoin-embassy/"                   
#  [2] "/bitcoin_atm/7105/bitcoin-atm-general-bytes-northampton-costcutter/"                           
#  [3] "/bitcoin_atm/4759/bitcoin-atm-general-bytes-birmingham-uk-costcutter/"                         
#  [4] "/bitcoin_atm/2533/bitcoin-atm-general-bytes-birmingham-uk-londis-# convenience/"                 
#  [5] "/bitcoin_atm/5458/bitcoin-atm-general-bytes-coventry-agg-african-restaurant/"                  
#  [6] "/bitcoin_atm/711/bitcoin-atm-general-bytes-coventry-bigs-barbers/"                             
#  [7] "/bitcoin_atm/5830/bitcoin-atm-general-bytes-telford-bpred-lion-service-station/"               
#  [8] "/bitcoin_atm/5466/bitcoin-atm-general-bytes-nottingham-24-express-off-licence/"                
#  [9] "/bitcoin_atm/4615/bitcoin-atm-general-bytes-northampton-costcutter/"                           
# [10] "/bitcoin_atm/4841/bitcoin-atm-lamassu-worcester-computer-house/"                               
# [11] "/bitcoin_atm/3150/bitcoin-atm-bitxatm-leicester-keshs-wines-and-newsagents-braustone/"         
# [12] "/bitcoin_atm/2948/bitcoin-atm-bitxatm-coventry-nisa-local/"                                    
# [13] "/bitcoin_atm/4742/bitcoin-atm-bitxatm-birmingham-uk-custcutter-coventry-road-hay-mills/"       
# [14] "/bitcoin_atm/4741/bitcoin-atm-bitxatm-derby-michaels-drink-store-alvaston/"                    
# [15] "/bitcoin_atm/4740/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-crabtree-# hockley/"              
# [16] "/bitcoin_atm/4739/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-subway-boldmere/"               
# [17] "/bitcoin_atm/4738/bitcoin-atm-bitxatm-birmingham-uk-ashtree-convenience-store/"                
# [18] "/bitcoin_atm/4737/bitcoin-atm-bitxatm-birmingham-uk-nisa-local-finnemore-road-bordesley-green/"
# [19] "/bitcoin_atm/3160/bitcoin-atm-bitxatm-birmingham-uk-costcutter/"