Questions How To Parse Extremely Large Data
Leftmyworl
User
- Thread starter
- Thread Author
- #1
how to parse extremely large data, whats the best method? share thanks
Questions How To Parse Extremely Large Data
- Thread starter Leftmyworl
- Start date
I assume you are talking about txt files, so I suppose the quickest way for you would be the bash "grep" command for this. If you use Windows, you can download git, which installs a bash emulator.
grep -i "[some text]" file.txt
or
grep -i "[something]" .
for scan all files in the directory.
I suggest splitting large files into smaller parts with the split command
split largefile.txt -b 1024m
where "1024m" is the desired size, in megabytes. Change this as needed.
What I posted are just simple examples, research what these commands are capable of doing on your own.
grep -i "[some text]" file.txt
or
grep -i "[something]" .
for scan all files in the directory.
I suggest splitting large files into smaller parts with the split command
split largefile.txt -b 1024m
where "1024m" is the desired size, in megabytes. Change this as needed.
What I posted are just simple examples, research what these commands are capable of doing on your own.
