Here’s why web scraping will become huge for FS next year

By Puja Sharma

Four in five financial service organisations expect to shift their focus more towards web scraping in the coming 12 months, according to the findings from Oxylabs’ white paper, Alternative Data Unlocks Key Decisions in the UK & US Finance Industries. Among the main reasons given for such a shift were enriching the data they have available, making extraction more efficient, and getting access to a larger volume of information through web scraping.

The findings highlight the important role data collection plays in helping financial organisations gain valuable insight into their performance, as well as key industry trends. As a result, web scraping providers should look to accommodate the increasing volume of financial services companies who want to put further emphasis on the process, while acting quickly to alleviate their lingering concerns.

“It seems like over the last 12 months the attitude towards web scraping has changed drastically, and while it is not surprising, it is incredibly telling. Companies in the finance sector are seeing others gain an informational advantage through new data acquisition methods. To keep up, companies need to make the shift themselves.

“The results also highlight a clear distinction between the US and UK, with companies in the US more enthusiastic about the shift (86% vs 74%) than UK businesses. Such a wide margin of difference is hard to explain purely from a technological adoption perspective. It may be influenced, however, by online data being primarily US-driven, making web scraping slightly more useful for information gathering purposes,” Gediminas Rickevičius, VP of Global Partnerships at Oxylabs, said.

While web scraping is on the rise, several businesses still associate it with risk. Potential legal issues have previously been one of the primary concerns. This is still the case, with legal complications remaining the number one issue (38%) for those who are yet to implement web scraping. Complicated technology (36%) and budget constraints follow suit (36%) with a lack of technical expertise (34%) also figuring prominently.

Vaidotas Šedys, Head of Risk Management at Oxylabs, said: “Risk is inherent to all new technologies, as pioneering a field means there are no footsteps to follow. However, any web scraping risks can be greatly mitigated by applying data collection and management best practices, and by employing legal and risk professionals. These practices may take some time to get used to, but they will ensure the organisation derives consistent ROI from web scraping.”

US companies are more concerned by legal complications than their UK counterparts (43% vs 34%). This could be due to organisations perceiving legal processes as more stringent and demanding in the US, causing companies to be extra wary about implementing such practices.

“The finance sector has always used data as a key tool for making important decisions. However, the findings in our research have revealed that even more emphasis will be put on data collection over the coming months. It’s no secret that the web scraping industry is still fraught with numerous legal and technical challenges, but the organisations that can overcome these hurdles will be the ones reaping the benefits in the years to come,” added Šedys.

Key highlights

• Around  80% of financial services businesses expect to shift more towards web scraping in the coming year.
• The trend rises to 85% among US organisations but drops to 74% among UK businesses.
• However, concerns remain, with the risk of legal complications being the primary reason organisations do not use web scraping (37%).
• Complicated technology was another reason (35%) along with budget constraints (35%).