WHY THIS MATTERS IN BRIEF
- The deep web is home to both legal and illegal institutions and activities that have always traditionally been hard to search, crawl and analyse but now a new upgrade from DARPA promises to change all that
A lot of people have never heard of the US Governments “Memex” program. Memex is a deep web search engine that was first developed by a Stanford graduate working for Rescue Forensics on behalf of the Defense Advanced Research Projects Agency (DARPA) – the agency that created the original ARPANET, which then went on to form the foundation of the internet.
According to reports, because no one really knows for sure, the deep web is about 20 to a hundred times larger than the surface web – the part of the web that we all regularly search and trawl through and in order to access it, let alone search it, you need special tools. As a consequence Memex doesn’t work in the same way that Google or Bing do.
DARPA describes Memex as a set of search tools that government users can use to perform domain specific searches that display as rich, sophisticated infographics and apparently Memex can crawl over 95% of the deep web.
Memex’s original purpose in life was to try to help government agencies surface human trafficking, slavery and terrorist related content in an effort to help them try to stem the tides – in 2016 alone it’s thought that the deep web helped these organisations traffic over 21 million people and move over $500 Billion in funds, and as a result of its use it’s helped secure thousands of convictions.
Rescue Forensics was recently acquired by IST Research, another company who also works with DARPA but this time on a platform called “Pulse”, a deep web search platform that works by looking at phone addresses and E-Mails as well as commonalities such as the spacing used in deep web adverts and aggregation sites, and that specialises in crawling deep web nets in parts of the world where security and accessibility aren’t great. Now the two are hooking up.
In the next year DARPA, IST Research and Research Forensics will join together and integrate Memex and Pulse to create a new, better platform and according to a report from Sophos the technology is becoming so good at what it does that if it keeps on being successful, we might need to rename what is currently the deep web.