Crawling the Web for Supply Chain Data

by Source Intelligence

on October 21, 2019

October brings harvest festivals, spiced lattes and a seasonal favorite here at Source Intelligence - Halloween! Halloween, of course, makes me think of spiders, which makes me think of webs, which makes me think of how successful we have been here at Source Intelligence in using web crawling to find compliance documents.

The web houses vast amounts of information, which makes finding exactly what you’re looking for a bit tricky sometimes. Web crawling or data aggregation allows us to navigate through this huge amount of information and extract documents from suppliers that are already publicly available. It precludes the need for bugging a supplier for the information right off the bat.


Publicly Available Compliance Data


We recently published, “A Professional’s Guide To Supply Chain Program Success,” where we outlined best practices for companies at all levels of the supply chain to implement successful supply chain initiatives. One of the tips we shared was making your compliance data publicly available.



In today's landscape, a company's website is its "home base" for information. Having publicly available compliance data on a company's website is a standard for most large-scale component and finished-goods manufacturers. Instead of responding to one-off requests, companies who have their information publicly available can point all customers to their website and alleviate the majority of work behind customer supply chain data requests.


Download White Paper


AI Powered Compliance


Source Intelligence is now beginning to use AI to make this extraction more efficient and precise. The way it works is that collected documents are analyzed for relevance and compliance value. Once highly relevant documents are detected they are passed into our usual due diligence process. As we continue this process, we analyze more documents and continue training our system, our AI gets "smarter" and is able to find more relevant documents, faster and more efficiently. 

Click here to learn more about our application of AI and machine learning to the supply chain compliance process.


Web crawling for supply chain data drives strong program results


The result of using web crawling technology for supplier data aggregation is that we can contribute to a program’s completion and success even before the campaign is launched. Let us make your document search a little less scary/frightening this year. Talk to the Source team about web data aggregation for your compliance program. 

Contact Us


Read More of What You Like.

Are You Ready to Trace Cobalt in Your Supply Chain?

Conflict Minerals Reporting Best Practices

Best Practices for End-to-end Supply Chain Due Diligence in the EU