datahoarder

Here is the download link for a text file that has all the original URL's https://wormhole.app/PpjJ3P#SFfAOKm1bnCyi-h2YroRyA The link will only last for 24 hours.

Has anyone made a Dataset 9 and 10 torrent file without the files in it that the NYT reported as potentially CSAM?

Do people here have the partial dataset 9? or are you all missing the entire set? There is a magnet link floating around for ~100GB of it, the one removed in the OP

I am trying to figure out exactly how many files dataset 9 is supposed to have in it. Before the zip file went dark, I was able to download about 2GB of it. This was today, maybe not the original zip file from jan 30th In the head of the zip file is an index file, VOL00009.OPT, you don't need the full download in order to read this index file. The index file says there are 531,307 pdfs the 100GB torrent has 531,256, it's missing 51 pdfs. I checked the 51 file names and they no longer exist as individual files on the DOJ website either. I'm assuming these are the CSAM.

note that the 3M number of released documents != 3M pdfs. each pdf page is counted as a "document". dataset 9 contains 1,223,757 documents, and according to the index, we are missing only 51 documents, they are not multipage. In total, I have 2,731,789 documents from datasets 1-12, short of the 3M number. the index I got also was not missing document ranges

it's curious that the zip file had an extra 80GB when only 51 documents are missing. I'm currently scraping links from the DOJ webpage to double check the filenames

The BBC is now reporting that "thousands" of documents have been removed because the DOJ improperly redacted information that can be used to identify the victims: https://www.bbc.com/news/articles/cn0k65pnxjxo

@wild_cow_5769:matrix.org If someone has a group working on finding the dataset.

There are billions of people on earth. Someone downloaded dataset 9 before the link was taken down. We just have to find them :)

Holy shit

The entire Court Records and FOIA page is completely gone too! Fuckers!

Someone mentioned a matrix group. Can they DM and invite me. I want to help. Thx

In regard to Dataset 9, it's currently being shared on Dread (forum).

I have no idea if it's legit or not, and Idc to find out after reading about what's in it from NYT.

While I feel hopeful that we will be able to reconstruct the archive and create some sort of baseline that can be put back out there, I also cant stop thinking about the "and then what" aspect here. We've see our elected officials do nothing with this info over and over again and I'm worried this is going to repeat itself.

I'm fully open to input on this, but I think having a group path forward is useful here. These are the things I believe we can do to move the needle.

Right Now:

1. Create a clean Data Archive for each of the known datasets (01-12). Something that is actually organized and accessible.
2. Create a working Archive Directory containing an "itemized" reference list (SQL DB?) the full Data Archive, with each document's listed as a row with certain metadata. Imagining a Github repo that we can all contribute to as we work. -- File number -- Dir. Location -- File type (image, legal record, flight log, email, video, etc.) -- File Status (Redacted bool, Missing bool, Flagged bool
3. Infill any MISSING records where possible.
4. Extract images away from .pdf format, Breakout the "Multi-File" pdfs, renaming images/docs by file number. (I made a quick script that does this reliably well.)
5. Determine which files were left as CSAM and "redact" them ourselves, removing any liability on our part.

What's Next: Once we have the Archive and Archive Directory. We can begin safely and confidently walking through the Directory as a group effort and fill in as many files/blanks as possible.

1. Identify and dedact all documents with garbage redactions, (remember the copy/paste DOJ blunders from December) & Identify poorly positioned redaction bars to uncover obfuscated names
2. LABELING! If we could start adding labels to each document in the form of tags that contain individuals, emails, locations, businesses - This would make it MUCH easier for people to "connect the dots"
3. Event Timeline... This will be hard, but if we can apply a timeline ID to each document, we can put the archive in order of events
4. Create some method for visualizing the timeline, searching, or making connection with labels.

We may not be detectives, legislators, or law men, but we are sleuth nerds, and the best thing we can do is get this data in a place that can allow others to push for justice and put an end to this crap once and for all. Its lofty, I know, but enough is enough. ...Thoughts?