[Rebenga]

"I do not own 10tb of data"


But if i was to own it, how can i manage it eg (create a search engine) locally.
Whats the best practice for this, frameworks to use etc..
what specs will i need ?


Note:
I need something good, as fast as mongo but with less ram it's ok if its a bit slower something that would mainly rely on the SSDs Nvme

[eVee]

If you have a small machine. just stick to grep and patience
Otherwise mongo or casandra.
Ideally Redis for lightning fast responses but requires a shitload of ram

[Rebenga]

If you have a small machine. just stick to grep and patience
Otherwise mongo or casandra.
Ideally Redis for lightning fast responses but requires a shitload of ram

Yeah man those are just too much ram, my budget just cant handle it.
im trying to make this work on 64gb ram 4tb ssd if i can somehow compress/index the data

[eVee]

If you have a small machine. just stick to grep and patience
Otherwise mongo or casandra.
Ideally Redis for lightning fast responses but requires a shitload of ram

Yeah man those are just too much ram, my budget just cant handle it.
im trying to make this work on 64gb ram 4tb ssd if i can somehow compress/index the data

Any form of compression will slow it down.
To make it work, you can look at casandra with LZ4 compression enabled. not sure if 4tb will be quite enough.  I think lz4 will compress it down to about 6 or 7tb

[NaN485]

Don't forget that in order to have redundancy, if you want to store 10TB of data you'll need 20TB of storage. Look into RAID technologies to see which one is the most apropriate for your use case.