如何从 Google 地图中网络抓取评论

Andrei Ogiolan on Apr 21 2023

导言

谷歌地图是世界上使用最广泛的地图和导航服务之一，为用户提供了查找和探索地点、企业和兴趣点的简便方法。谷歌地图的主要功能之一是搜索地点并查看其详细信息，如位置、评论、照片等。

Scraping this data from Google Maps can be useful for businesses to track and analyze the performance of their locations, for researchers to study patterns in consumer behavior and for individuals to find and explore new places.

The purpose of this article is to provide a step-by-step guide on how to scrape Google Maps Reviews with our API using Node.js. We will cover everything from setting up the development environment to extracting relevant data and discussing potential issues. By the end of this article, you will have the knowledge and tools you need to scrape Google Maps place results on your own.

为什么要使用专业铲运机而不是自制铲运机？

出于以下几个原因，使用专业搜索器可能比创建自己的搜索器更好。首先，专业搜刮工具是为处理各种搜刮任务而设计的，并针对性能、可靠性和可扩展性进行了优化。它们专为处理大量数据而设计，可以处理各种类型的网站和网络技术。这意味着，专业的刮擦工具通常能比定制的刮擦工具更快、更准确地提取数据。

此外，专业的搜索工具通常具有验证码解锁、IP 轮换和错误处理等内置功能，可以使搜索过程更高效、更不易出错。它们还提供支持和文档，在您遇到任何问题时都会有所帮助。

另一个重要方面是，专业的搜刮工具提供商必须遵守其搜刮网站的搜刮政策，并能合法使用数据，这一点在搜刮数据时必须牢记。

Finally, in our particular case, in order to scrape Google Maps Reviews, for best results, you need to pass a data_id parameter to your Google URL. This parameter usually looks something like this : 0x87c0ef253b04093f:0xafdfd6dc1d3a2b4e. I know this may sound intimidating at first as you may have no idea how to get the data_id property for a specific place and you are right, because Google hides this information and it is not visible on the page when you are searching for a place in Google Maps. But, fortunately, using a professional scraper like ours takes care of that by finding this data for you. We will talk in the later sections about how to get the data_id and how to scrape Google Maps reviews using our API.

确定我们的目标

What are Google Maps reviews?

Google Maps reviews are the ratings and comments left by users on Google Maps about a specific place. These reviews include information such as the user's name, the date the review was left, the rating given, and the review text.

Scraping Google Maps reviews can be useful for businesses who want to track and analyze the performance of their locations, researchers who want to study patterns in consumer behavior, and individuals who want to find and explore new places. By extracting the reviews data, businesses can identify the strengths and weaknesses of their locations, and make improvements accordingly. Researchers can study the sentiment of the reviews and find patterns in consumer behavior. Individuals can also use this information to make decisions about where to go and what to do.

我们的目标是什么样子的？

设置

Before beginning to scrape Google Maps reviews, it's important to have the necessary tools in place. The primary requirement is Node.js, a JavaScript runtime that enables the execution of JavaScript on the server-side, which can be downloaded from their official website. Additionally, an API KEY is required, which can be obtained by creating an account here and activating the SERP service.

设置好 Node.js 并获得 API KEY 后，下一步就是创建 Node.js 脚本文件。运行以下命令即可完成：

$ touch scraper.js 

然后将下面一行粘贴到文件中：

console.log("Hello World!")

然后运行以下命令

$ node scraper.js

If you see the message "Hello World!" displayed on the terminal, it indicates that Node.js has been successfully installed and you are ready to proceed to the final step. This final step is to obtain the Place ID of the place you are interested in scraping reviews. This is where our API comes in handy, it's easy to use and doesn't require any additional libraries to be installed.

首先，您需要在 js 文件中导入 Node.js `https` 内置模块，以便能够向我们的 API 发送请求。具体方法如下：

const https = require("https")；

其次，您需要指定您的 API 密钥、搜索词和您感兴趣的地点的坐标：

const API_KEY = "<YOUR-API-KEY-HERE>" // You can get by creating an account - https://app.webscrapingapi.com/register

const query = "Waldo%20Pizza"

const coords = "@38.99313451901278,-94.59368586441806"

Tip: This is how you get the coordinates for a place on Google Maps:

The next step is to include the obtained Place ID in an options object, to let our API know which location's reviews you want to scrape:

const options = {

  "method": "GET",

  "hostname": "serpapi.webscrapingapi.com",

  "port": null,

  "path": `/v1?engine=google_maps&api_key=${API_KEY}&type=search&q=${query}&ll=${coords}`,

  "headers": {}

};

接下来，您需要将所有这些信息调用到我们的应用程序接口：

const req = https.request(options, function (res) {

 const chunks = [];

 res.on("data", function (chunk) {

   chunks.push(chunk);

 });

 res.on("end", function () {

   const body = Buffer.concat(chunks);

   const response = JSON.parse(body.toString());

   const data_id = response.place_results.data_id;

   if (data_id) {

       console.log(data_id);

   }

   else {

       console.log('We could not find a data_id property for your query. Please try using another query')

   }

 });

});

req.end();

Lastly, you can execute the script you have just created and wait for the results to be returned:

$ node scraper.js

然后，屏幕上就会打印出 data_id 属性：

$ ​​0x87c0ef253b04093f:0xafdfd6dc1d3a2b4es

That concludes the setup process, with the data_id property, now you have all the necessary information to create a scraper for Google Maps reviews using our API using Node.js.

Let’s start scraping Google Reviews

With the environment set up, you are ready to begin scraping Google Maps Reviews with our API. To proceed, you need to set up the data parameter as previously mentioned. With all the necessary information available, you can set up the data_id parameter as follows:

const data_id = "0x87c0ef253b04093f:0xafdfd6dc1d3a2b4e" // the data_id we retrieved earlier

Now, the only thing left to do is to modify the options object, thus telling our API that you would like to scrape reviews from Google Maps:

const options = {

  "method": "GET",

  "hostname": "serpapi.webscrapingapi.com",

  "port": null,

  "path": `/v1?engine=google_maps_reviews&api_key=${API_KEY}&data_id=${data_id}`, // there is no need in having a query anymore, data_id is enough to identify a place

  "headers": {}

};

And this is everything you need to do. Your script should now look like this:

const http = require("https");

const API_KEY = "<YOUR-API-KEY-HERE>"

const data_id = "0x87c0ef253b04093f:0xafdfd6dc1d3a2b4e" // the data_id we retrieved earlier

const options = {

  "method": "GET",

  "hostname": "serpapi.webscrapingapi.com",

  "port": null,

  "path": `/v1?engine=google_maps_reviews&api_key=${API_KEY}&data_id=${data_id}`, // there is no need in having a query anymore, data_id is enough to identify a place

  "headers": {}

};

const req = http.request(options, function (res) {

  const chunks = [];

  res.on("data", function (chunk) {

    chunks.push(chunk);

  });

  res.on("end", function () {

    const body = Buffer.concat(chunks);

    const response = JSON.parse(body.toString())

    console.log(response);

  });

});

req.end();

After executing this script, you should receive a response that appears similar to this:

reviews: [

    {

      link: 'https://www.google.com/maps/reviews/data=!4m8!14m7!1m6!2m5!1sChZDSUhNMG9nS0VJQ0FnSUMyem9pOEdBEAE!2m1!1s0x0:0xafdfd6dc1d3a2b4e!3m1!1s2@1:CIHM0ogKEICAgIC2zoi8GA%7CCgwI1vuBkwYQiKeWyQE%7C?hl=en-US',

      date: '8 months ago',

      rating: 5,

      snippet: 'Wow, if you have dietary restrictions this is absolutely the place to go! Both for the variety of restrictions they cater to as well as the taste of the dishes.The good: great tasting food. Very conscious of dietary restrictions which include multiple types of vegan cheeses as well as gluten free. Decent drink selection.The meh: service is nice but a touch slow. Maybe understaffed? Prices are average for pizzas.The bad: noneFeatures: Did not see any masks on anyone inside. Unsure of cleaning practices so I cannot speak to that.Dine in: Yes\n' +

        'Takeout: Yes\n' +

        'Curbside pickup: YesWow, if you have dietary restrictions this is absolutely the place to go! Both for the variety of restrictions they cater to as well as the taste of the dishes. ...More',

      likes: 3,

      user: [Object],

      images: [Array]

    },

    {

      link: 'https://www.google.com/maps/reviews/data=!4m8!14m7!1m6!2m5!1sChZDSUhNMG9nS0VJQ0FnSURXOUxHSUl3EAE!2m1!1s0x0:0xafdfd6dc1d3a2b4e!3m1!1s2@1:CIHM0ogKEICAgIDW9LGIIw%7CCgwI3OnIkQYQwLGL1gM%7C?hl=en-US',

      date: '9 months ago',

      rating: 5,

      snippet: "We love Waldo Pizza! We have dairy allergies and Waldo offers a wide range of vegan cheeses as well as a ton of different toppings. The vegan dessert here is always excellent as well, super rich in flavor. Of course the traditional pizza, pasta and dessert are also amazing! It's great to have both options under one roof!Dine in: Yes\n" +

        'Outdoor seating: No ...More',

      likes: 1,

      user: [Object],

      images: [Array]

    }

    . . . 

]    

And that's it! You have successfully scraped Google Maps reviews using our API and you can now use the obtained data for various purposes such as data analysis, business analysis, machine learning and more. For further reference and code samples in other 6 programming languages, you can check out our Google Maps reviews documentation.

Limitations of Google Maps Reviews

Even though using a professional scraper to extract Google Maps reviews can be more efficient and accurate than building your own scraper, there are still some limitations to keep in mind. One limitation is that some professional scrapers may have usage limits, which means that you can only scrape a certain number of reviews per day or per month. Another limitation is that some professional scrapers may not be able to bypass IP blocks or CAPTCHAs, which can make it difficult to extract large amounts of data without encountering errors. Luckily, at WebScrapingAPI we have residential proxies which rotate the IP addresses, thus getting you covered and eliminating the worry of being banned or rate limited. One thing you should keep in mind is that Google Maps reviews are usually in natural language, which can make them difficult to analyze and interpret without the use of natural language processing techniques.

结论

In conclusion, scraping Google Maps reviews can be a valuable tool for businesses, researchers, and individuals. It allows you to gather data on a large scale and analyze it for various purposes. However, it's important to keep in mind that there are limitations to scraping Google Maps reviews, including usage limits, CAPTCHAs and IP blocks and natural language processing. Using a professional scraper can make the process more efficient and accurate and can get you rid of some of the limitations.Overall, scraping Google Maps reviews can provide useful information, but it's important to approach it with caution and care.