How to scrape a website using Node.js and Puppeteer

JUNE 26, 2023

In a world where data has become the "new oil," web scraping and automation tools like web scrapers and crawlers are at the forefront of unlocking this valuable resource. Harnessing the power of these tools, you can extract vital information for market research, content aggregation, price comparison, or creating machine learning datasets.

This blog post dives into the specifics of these tools, highlighting the Puppeteer library's unique features - a potent Node.js library for web automation. Furthermore, we will guide you on web scraping implementation using Puppeteer and unveil how its integration with Multilogin can supercharge your browser automation tasks.

What is a web scraper?

A web scraper is a software tool or program that extracts website data. It automates retrieving specific information from web pages, including text, images, links, and other structured data. Web scrapers simulate human browsing behavior to navigate web pages, access desired content, and extract relevant data.

Web scrapers can be programmed to target specific websites or follow predefined patterns to scrape information from multiple sites.

They can explore websites, click on links, complete forms, and interact with elements to find hidden or changing information.

The extracted data from web scraping can be used for various purposes, such as:

market research
data analysis
content aggregation
price comparison
monitoring website changes
creating datasets for machine learning models.

What is a web crawler?

A web crawler, also called an automaton or arachnid bot, is an online automaton that autonomously explores and surveys web pages.

Crawlers are frequently aided by exploration tools (such as Google or Bing) to amass all the data from a webpage and categorize it.

Web crawlers assist in gathering data from openly accessible web pages, discovering facts, and cataloging online documents. Furthermore, scurriers scrutinize the connections among web URLs to ascertain how these documents interrelate.

Crawling - utilized when we desire to seek out information on the internet.
Extracting - utilized when we wish to procure that information from the internet.

Features of Puppeteer

Puppeteer, a powerful Node.js library, offers an extensive range of web automation and scraping features.

Here's a proper list with explanations of Puppeteer's features for web automation and scraping:

Programmable Web Browser Control: Puppeteer allows you to control web browsers programmatically. This means you can automate tasks like generating screenshots, creating PDFs of web pages, and even submitting forms automatically.
Robust API: Puppeteer provides a powerful API that grants you access to manipulate web pages. You can interact with elements, modify content, and navigate different pages.
Headless Browser Support: Puppeteer supports various headless browsers, including Chromium. The headless mode enables you to simulate browser behavior without a graphical interface, making it ideal for efficient web scraping and automation tasks.
Intercepting Network Requests: Puppeteer offers advanced features such as intercepting and modifying network requests. This lets you capture and manipulate HTTP requests and responses, opening up possibilities for dynamic content extraction and handling.
Authentication Handling: Puppeteer simplifies the process of handling authentication on websites. You can log in to restricted areas, manage cookies, and maintain sessions as part of your web scraping or automation workflows.
JavaScript Execution: Puppeteer enables you to execute custom JavaScript code within web pages. This capability allows you to interact with the DOM, manipulate elements, and extract data that may require client-side rendering or user interaction.
Flexibility and Ease of Use: Puppeteer is known for its flexibility and user-friendly nature. Its intuitive API design makes it easy to start with web scraping and automation tasks, even for developers with minimal experience in these areas.
Comprehensive Feature Set: Puppeteer encompasses many features for effective web scraping and automation. It provides the tools to navigate complex websites, handle dynamic content, and extract structured data efficiently.

Web Scraping in Node.js using Puppeteer

In this focused section, we dive into the world of web scraping in Node.js using the formidable Puppeteer library. With a step-by-step approach, we will explore the seamless integration of Node.js and Puppeteer to unleash the power of automated data extraction.

Step 1: Setting Up Your Environment

Before we begin, ensure that Node.js is installed on your system. Node.js, created by Ryan Dahl, is a JavaScript runtime built on Chrome's V8 JavaScript engine that allows you to run JavaScript on your server. It's event-driven, single-threaded, and perfect for real-time applications.

Once Node.js is installed, you can install Puppeteer using npm (Node Package Manager). Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium browsers over the DevTools Protocol. Run the following command in your terminal:

1npm install puppeteer

Step 2: Creating a New Puppeteer Project

After installing Puppeteer, create a new project directory and initialize it with npm:

1 mkdir puppeteer-project && cd puppeteer-project

This will create a new package.json file in your project directory, setting up the server-side of your web application.

Step 3: Writing Your First Puppeteer Script

Now, let's write our first Puppeteer script. Create a new file named scrape.js and open it in your favorite code editor. Import the Puppeteer library at the top of your file:

1const puppeteer = require('puppeteer');

Next, we'll write a simple script that opens a webpage and takes a screenshot:

1 (async () => { 
2
3 const browser = await puppeteer.launch(); 
4
5 const page = await browser.newPage(); 
6
7 await page.goto('https://example.com'); 
8
9 await page.screenshot({ path: 'example.png' }); 
10
11 
12
13 
14
15 await browser.close(); 
16
17})();

This script launches a new headless browser instance, opens a new page, navigates to https://example.com, takes a screenshot, and saves it as example.png.

Step 4: Running Your Puppeteer Script

To run your Puppeteer script, use the following command in your terminal:

1 node scrape.js

If everything is set up correctly, you should see a new file named example.png in your project directory.

Step 5: Scraping Data with Puppeteer

Now that we've covered the basics let's move on to the main topic: web scraping. With Puppeteer, you can easily select and extract data from web pages. Here's a simple example:

1(async () => { 
2
3 const browser = await puppeteer.launch(); 
4
5 const page = await browser.newPage(); 
6
7 await page.goto('https://example.com'); 
8
9   
10
11 const data = await page.evaluate(() => { 
12
13  const title = document.querySelector('h1').innerText; 
14
15  return title; 
16
17 }); 
18
19 
20
21 
22
23 console.log(data); 
24
25 await browser.close(); 
26
27})();

This script navigates to https://example.com, selects the first h1 element on the page and logs its text content to the console.

Puppeteer browser automation with Multilogin

Multilogin can help you simplify your browser tasks using Puppeteer, an API that automates Chromium-based browsers. We understand the value of automation, so we've made it easy for you to integrate Puppeteer with our platform.

Our solution allows you to create web crawlers that search and collect data using our Mimic browser. What's unique about this? Our Mimic browser has masked fingerprints so that you can collect data more efficiently and securely.

Setting up Puppeteer with Multilogin is a breeze. All you need to do is predefine the application port in the app.properties file. Now you can refer to the Multilogin application through this port, and you're all set to automate your browser tasks with Puppeteer!

But that's not all. By combining Multilogin and Puppeteer, you can automate a wide range of tasks, from simple data collection to complex web interactions. And the best part? You get to do all this while enjoying the anonymity and security that our advanced browser fingerprint offers.

Want to learn more? Check out our detailed guide on how to use Multilogin with Puppeteer for browser automation here. We're here to make your browser automation journey smoother and more efficient!