DARPA is giving its deep web search tech a huge upgrade
The deep web is home to both legal and illegal institutions and activities that have always traditionally been hard to search, crawl and analyse but now a new upgrade from DARPA promises to change all that
Cite or link to this article
Griffin, M. (2016) 'DARPA is giving its deep web search tech a huge upgrade', 311 Institute, 13 December. Available at: https://www.311institute.com/darpas-new-deep-web-search-tech-is-so-good-we-might-have-to-rename-the-deep-web/ (Accessed: 1 October 2026).
A lot of people have never heard of the US Governments "Memex" program. Memex is a deep web search engine that was first developed by a Stanford graduate working for Rescue Forensics on behalf of the Defense Advanced Research Projects Agency (DARPA) - the agency that created the original ARPANET, which then went on to form the foundation of the internet.
According to reports, because no one really knows for sure, the deep web is about 20 to a hundred times larger than the surface web - the part of the web that we all regularly search and trawl through and in order to access it, let alone search it, you need special tools. As a consequence Memex doesn't work in the same way that Google or Bing do.
DARPA describes Memex as a set of search tools that government users can use to perform domain specific searches that display as rich, sophisticated infographics and apparently Memex can crawl over 95% of the deep web.
Memex's original purpose in life was to try to help government agencies surface human trafficking, slavery and terrorist related content in an effort to help them try to stem the tides - in 2016 alone it's thought that the deep web helped these organisations traffic over 21 million people and move over $500 Billion in funds, and as a result of its use it's helped secure thousands of convictions.
Rescue Forensics was recently acquired by IST Research, another company who also works with DARPA but this time on a platform called "Pulse", a deep web search platform that works by looking at phone addresses and E-Mails as well as commonalities such as the spacing used in deep web adverts and aggregation sites, and that specialises in crawling deep web nets in parts of the world where security and accessibility aren't great. Now the two are hooking up.
In the next year DARPA, IST Research and Research Forensics will join together and integrate Memex and Pulse to create a new, better platform and according to a report from Sophos the technology is becoming so good at what it does that if it keeps on being successful, we might need to rename what is currently the deep web.
FAQ
Why does this matter?
The deep web is home to both legal and illegal institutions and activities that have always traditionally been hard to search, crawl and analyse but now a new upgrade from DARPA promises to change all that
About the author
Matthew Griffin Founder, 311 Institute
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath."
Read full bio
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath." 15-time best selling author of the "Codex of the Future" series, Matthew is the Founder and Futurist in Chief of the 311 Institute, a global Futures and Deep Futures advisory firm working with royal households, world leaders, G7, G20, and G77 governments, NGOs, and multi-national mid and mega cap firms to help them explore, shape, and lead the next 50 years of business and society.
An award-winning YouTube creator with over a million followers, with an unrivalled global reach and impact, Matthew is a highly sought-after international keynote speaker, lecturer, and mentor who collaborates with global leaders through the United Nations Alliance of Civilizations (UNAOC) and United Nations General Assembly (UNGA) to shape pivotal initiatives such as the UN’s AI for Humanity program, the United Nations Conference of the Parties (UN COP), and the World Economic Forum in Davos.
As the former Global Head of Cloud, National Security, and Enterprise Sales for companies including Atos, Dell-EMC, and IBM, Matthew has a proven track record of building multi-billion dollar business units and turning failing divisions into market leaders. His ability to identify, analyse, and communicate the implications of hundreds of emerging technologies and trends is unparalleled, and his insights are trusted by many of the world’s most respected organisations, including ABB, Accenture, Adidas, AON, ARM, BCG, Centrica, Citi, Coca-Cola, Dentons, Deloitte, Dow Jones, EY, Google, KPMG, Lego, Legal & General, LinkedIn, Microsoft, PepsiCo, Qualcomm, RWE, Samsung, Siemens AG and Siemens Energy, T-Mobile, UBS, VISA, Walmart, Workday, Worldpay and many others.
Regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, Matthews mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.
What future do you need to see?
Choose one to get started on security and privacy and the future of your organisation.
Sources and further reading
- Rescue Forensics rescueforensics.com
- set of search tools darpa.mil
- IST Research istresearch.com
Source: first published by the 311 Institute on 13 December 2016. Cite as: Griffin, M. (2016). DARPA is giving its deep web search tech a huge upgrade. 311 Institute. https://www.311institute.com/darpas-new-deep-web-search-tech-is-so-good-we-might-have-to-rename-the-deep-web/
You are welcome to quote this article with credit and a link to the original.