{"id":1629,"date":"2019-03-01T04:53:13","date_gmt":"2019-03-01T04:53:13","guid":{"rendered":"https:\/\/blog.hassler.ec\/wp\/?p=1629"},"modified":"2019-02-28T02:59:01","modified_gmt":"2019-02-28T02:59:01","slug":"web-scraping-for-web-developers-a-concise-summary","status":"publish","type":"post","link":"https:\/\/blog.hassler.ec\/wp\/2019\/03\/01\/web-scraping-for-web-developers-a-concise-summary\/","title":{"rendered":"Web scraping for web developers: a concise\u00a0summary"},"content":{"rendered":"
\n
\n
<\/div>\n<\/div>
Photo by rawpixel<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n

Knowing one approach to web scraping may solve your problem in the short term, but all methods have their own strengths and weaknesses. Being aware of this can save you time and help you to solve a task more efficiently.<\/p>\n

Numerous resources exist, which will show you a single technique for extracting data from a web page. The reality is that multiple solutions and tools can be used for that.<\/p>\n

What are your options to programmatically extract data from a web page?<\/p>\n

What are the pros and cons of each approach?<\/p>\n

How to use cloud services to increase the degree of automation?<\/p>\n

This guide meant to answer these questions.<\/strong><\/p>\n

I assume you have a basic understanding of browsers in general, HTTP<\/strong>requests, the DOM<\/strong> (Document Object Model), HTML<\/strong>, CSS selectors<\/strong>, and Async JavaScript<\/strong>.<\/p>\n

If these phrases sound unfamiliar, I suggest checking out those topics before continue reading. Examples are implemented in Node.js, but hopefully you can transfer the theory into other languages if needed.<\/p>\n

Static content<\/h3>\n

HTML source<\/h4>\n

Let\u2019s start with the simplest approach.<\/p>\n

If you are planning to scrape a web page, this is the first method to try. It requires a negligible amount of computing power and the least time to implement.<\/p>\n

However, it only works if the HTML source code contains the data<\/strong> you are targeting. To check that in Chrome, right-click the page and choose View page source<\/em>. Now you should see the HTML source code.<\/p>\n

It\u2019s important to note here, that you won\u2019t see the same code by using Chrome\u2019s inspect tool, because it shows the HTML structure related to the current state of the page, which is not necessarily the same as the source HTML document that you can get from the server.<\/p>\n

Once you find the data here, write a CSS selector<\/a> belonging to the wrapping element, to have a reference later on.<\/p>\n

To implement, you can send an HTTP GET request to the URL of the page and will get back the HTML source code.<\/p>\n

In Node<\/strong>, you can use a tool called CheerioJS<\/a> to parse this raw HTML and extract the data using a selector. The code looks something like this:<\/p>\n

\n
\n
\n
const fetch = require('node-fetch');\nconst cheerio = require('cheerio');\n\nconst url = 'https:\/\/example.com\/';\nconst selector = '.example';\n\nfetch(url)\n  .then(res => res.text())\n  .then(html => {\n    const $ = cheerio.load(html);\n    const data = $(selector);\n    console.log(data.text());\n  });<\/pre>\n<\/div>\n<\/div>\n<\/figure>\n
Dynamic content<\/h3>\n
In many cases, you can\u2019t access the information from the raw HTML code, because the DOM was manipulated by some JavaScript, executed in the background. A typical example of that is a SPA (Single Page Application), where the HTML document contains a minimal amount of information, and the JavaScript populates it at runtime.<\/p>\n
In this situation, a solution is to build the DOM and execute the scripts located in the HTML source code, just like a browser does. After that, the data can be extracted from this object with selectors.<\/p>\n
Headless browsers<\/h4>\n
This can be achieved by using a headless browser. A headless browser is almost the same thing as the normal one you are probably using every day but without a user interface. It\u2019s running in the background and you can programmatically control it instead of clicking with your mouse and typing with a keyboard.<\/p>\n
A popular choice for a headless browser is Puppeteer<\/a>. It is an easy to use Node library which provides a high-level API to control Chrome in headless mode. It can be configured to run non-headless, which comes in handy during development. The following code does the same thing as before, but it will work with dynamic pages as well:<\/p>\n
\n
\n
\n
const puppeteer = require('puppeteer');\n\nasync function getData(url, selector){\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.goto(url);\n  const data = await page.evaluate(selector => {\n    return document.querySelector(selector).innerText;\n  }, selector);\n  await browser.close();\n  return data;\n}\n\nconst url = 'https:\/\/example.com';\nconst selector = '.example';\ngetData(url,selector)\n  .then(result => console.log(result));<\/pre>\n<\/div>\n<\/div>\n<\/figure>\n
Of course, you can do more interesting things with Puppeteer, so it is worth checking out the documentation<\/a>. Here is a code snippet which navigates to a URL, takes a screenshot and saves it:<\/p>\n
\n
\n
\n
const puppeteer = require('puppeteer');\n\nasync function takeScreenshot(url,path){\n  const browser = await puppeteer.launch();\n  const page = await browser.newPage();\n  await page.goto(url);\n  await page.screenshot({path: path});\n  await browser.close();\n}\n\nconst url = 'https:\/\/example.com';\nconst path = 'example.png';\ntakeScreenshot(url, path);<\/pre>\n<\/div>\n<\/div>\n<\/figure>\n
As you can imagine, running a browser requires much more computing power than sending a simple GET request and parsing the response. Therefore execution is relatively costly and slow. Not only that but including a browser as a dependency makes the deployment package massive.<\/p>\n
On the upside, this method is highly flexible. You can use it for navigating around pages, simulating clicks, mouse moves, and keyboard events, filling out forms, taking screenshots or generating PDFs of pages, executing commands in the console, selecting elements to extract its text content. Basically, everything can be done that is possible manually in a browser.<\/p>\n
Building just the DOM<\/h4>\n
You may think it\u2019s a little bit of overkill to simulate a whole browser just for building a DOM. Actually, it is, at least under certain circumstances.<\/p>\n
There is a Node library, called Jsdom<\/a>, which will parse the HTML you pass it, just like a browser does. However, it isn\u2019t a browser, but a tool for building a DOM from a given HTML source code<\/strong>, while also executing the JavaScript code within that HTML.<\/p>\n
Thanks to this abstraction, Jsdom is able to run faster than a headless browser. If it\u2019s faster, why don\u2019t use it instead of headless browsers all the time?<\/p>\n
Quote from the documentation:<\/p>\n
People often have trouble with asynchronous script loading when using jsdom. Many pages load scripts asynchronously, but there is no way to tell when they\u2019re done doing so, and thus when it\u2019s a good time to run your code and inspect the resulting DOM structure. This is a fundamental limitation.<\/p><\/blockquote>\n
\u2026 This can be worked around by polling for the presence of a specific element.<\/p><\/blockquote>\n
This solution is shown in the example. It checks every 100 ms if the element either appeared or timed out (after 2 seconds).<\/p>\n
It also often throws nasty error messages when some browser feature in the page is not implemented by Jsdom, such as: \u201cError: Not implemented: window.alert\u2026\u201d or \u201cError: Not implemented: window.scrollTo\u2026\u201d.<\/em> This issue also can be solved with some workarounds (virtual consoles<\/a>).<\/p>\n
Generally, it\u2019s a lower level API than Puppeteer, so you need to implement certain things yourself.<\/p>\n
These things make it a little messier to use, as you will see in the example. Puppeteer solves all these things for you behind the scenes and makes it extremely easy to use. Jsdom for this extra work will offer a fast and lean solution.<\/p>\n
Let\u2019s see the same example as previously, but with Jsdom:<\/p>\n
\n
\n
\n
const jsdom = require(\"jsdom\");\nconst { JSDOM } = jsdom;\n\nasync function getData(url,selector,timeout) {\n  const virtualConsole = new jsdom.VirtualConsole();\n  virtualConsole.sendTo(console, { omitJSDOMErrors: true });\n  const dom = await JSDOM.fromURL(url, {\n    runScripts: \"dangerously\",\n    resources: \"usable\",\n    virtualConsole\n  });\n  const data = await new Promise((res,rej)=>{\n    const started = Date.now();\n    const timer = setInterval(() => {\n      const element = dom.window.document.querySelector(selector)\n      if (element) {\n        res(element.textContent);\n        clearInterval(timer);\n      }\n      else if(Date.now()-started > timeout){\n        rej(\"Timed out\");\n        clearInterval(timer);\n      }\n    }, 100);\n  });\n  dom.window.close();\n  return data;\n}\n\nconst url = \"https:\/\/example.com\/\";\nconst selector = \".example\";\ngetData(url,selector,2000).then(result => console.log(result));<\/pre>\n<\/div>\n<\/div>\n<\/figure>\n
Reverse engineering<\/h4>\n
Jsdom is a fast and lightweight solution, but it\u2019s possible even further to simplify things.<\/p>\n
Do we even need to simulate the DOM?<\/p>\n
Generally speaking, the webpage that you want to scrape consists of the same HTML, same JavaScript, same technologies you\u2019ve already know. So, if you<\/strong>find that piece of code from where the targeted data was derived, you can repeat the same operation in order to get the same result.<\/strong><\/p>\n
If we oversimplify<\/strong> things, the data you\u2019re looking for can be:<\/p>\n