idiosyncrasies that I wanted to bring to the attention of anyone thinking of using it.īefore you index large datasets, I would highly recommend splittingup large files to chucks no bigger than 1GB in size. While indexing large amounts of data I figured out that Filelocator pro has a few…. I try to stick to free resources whenever I can and FilelocatorPro has a $60 cost but it seemed to be the easiest and most affordable methodof accomplishing what I was going for without the need to massage a lot ofdata. As I started searching foroptions to index this data I realized that the same company that made AgentRansack made a professional version called “Filelocator Pro” which has indexingcapabilities.
Years ago in a forensics class I learned of a free toolcalled “Agent Ransack” ( )which made searching drives for information easier. I felt the results were well worth it since now my searches took2 minutes instead of 50. I started up ajob to index the data which took two full days to run and an extra 76 GB instorage space. I had a dataset of breach data that was 126 GB in size.Searching that data for an email address took about 50 minutes. To understand the tradeoffs and advantages, here’s a real world example. Ironically indexing isn’t nearly as common as it used to be in forensics but the technique works very well for breach data. You could let the drive process over the weekend and Monday morning quickly view the results and perform your searches. Imagine getting a hard drive in to examine on a Friday. You basically trade effort and extra storage space now for much quicker search results in the future. Indexing has been used in forensics for years. Similar to a recent blog post I wrote where I used a forensics tool called bulk extractor to help quickly acquire selectors (emails, phone numbers etc) from a large dataset, I decided to use a common forensics technique of indexing for this problem.
With that idea in mind, I wanted to find a way to make large breach datasets searchable without the need to maintain huge databases, normalize hundreds (or more) of disparate datasets etc. When I have given talks on memory forensics, I have always used the Windows standalone version of Volatility instead of Linux for my demos so attendees who were not really comfortable with Linux wouldn’t feel like they couldn’t try the techniques. Whenever I give conference talks I try to remove or reduce any barriers to entry.
How some organizations are using breach data to improve their security posture.How to make large data sets searchable in a reasonable amount of time.Tomorrow I’ll be giving a talk on breach data including:
0 kommentar(er)
