Questions Leak Search Engine

[mungo]

There are so many search engines around like intelx, stealseek etc all ingesting over billions of records. I want to know how do they ingest it where do they store the data for such fast retrieval. Even when locally going through logs files with maybe 700 million records it takes a good while using tools like ripgrep. What do you guys use for searching through the log data locally?
Also if someone understands the tech that goes behind these leak search engines please tell I would love to learn. From what I have seen they must be using some kind of chunking with inverted indexes for parameters like domain, email, ip etc. IntelX seems to divide files into smaller chunks and store each one into a bucket based on the network traffic when you lookup a domain.
Using something like rust we can parse through large logs under a minute with tokenization but ingestion is painfully slow on even a single txt log from Alien.

 

[eliantis]

this tread is very inerresting, if you have an information, please share it

 

[leaklalala]

Write a parser to feed into DB. Then build indexes on the data columns you need. Indexing does all the magic, but indexing 100 GB will require 1 TB space minimum. Also, searching requires memory minimum 32 GB RAM. Simple version of indexing is COMB script, they separated the files based on the alphabets serially and it is a very simple script to search. This version is not scalable.

Leak monitoring services spend a lot on R&D. and computing power. You need to have overall budget in millions to perform lookups at their scale. They use big data processing like Apache Hadoop.