datahoarder

10554 readers
1 users here now

Who are we?

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

We are one. We are legion. And we're trying really hard not to forget.

-- 5-4-3-2-1-bang from this thread

founded 6 years ago
MODERATORS
1
 
 

@ray@lemmy.ml Got it done, I'm first of the mods here and will be learning a little Lemmy over the next few weeks.

While everything is up in the air with the reddit changes I'll be very busy working on replacing the historical pushshift API without reddits bastardizations should a PS version come back.

In the mean time you should all mirror this data ensuring its survival, do what you do best and HOARD!!

https://the-eye.eu/redarcs/

2
 
 

cross-posted from: https://lemmy.ml/post/49741495

After the recent post about thousands of chiptune music from cracking teams in mp3 format, I wanted to share an already well known collection of Commodore 64 music in their original tracking format. To play these SID format music files, an audio player that supports the format is required.

You only need to download a single complete package, that is in size 80 MB for 7z format or 97 MB for RAR.


61157 SIDS

About HVSC

The High Voltage SID Collection (HVSC) is a freeware hobby project which organises Commodore 64 music (also known as SID music) into an archive for both musicians and fans alike. The work on the collection is done completely in the Team and contributors' spare time and is proudly one of the largest and most accurate computer music collections known.

3
5
2.5 vs 3.5 HDD (sh.itjust.works)
submitted 3 weeks ago* (last edited 3 weeks ago) by pastermil@sh.itjust.works to c/datahoarder@lemmy.ml
 
 

I'm looking to get a new internal hard drive for a specific use case:

  • it's for a desktop, meaning I can fit either
  • i need 2TB, no more and no less, also meaning I can get either
  • it's for my media collection, mostly movies
  • it's not gonna be read-only, but I wouldn't write too often (I would torrent, but all the program's metadata would be on another drive)
  • most likely gonna be formatted with btrfs (subject to change, open to suggestion)

Should I get the 2.5 inch or the 3.5 one?

Thanks in advance.

4
 
 

Collection of pre-patched Super Mario World Romhacks and Mods from SWMDB.me 202107 By Storage Manager (Repack)

Uploaded by Thingsiplay

Originally uploaded by Storage Manager at https://archive.org/details/super-mario-world-hacks (defunced now). This is a repackage of the same files. The original collection contains over 29 thousand files, only compressed as individual files. This means that Internet Archive's internal "on-the-fly-zip" doesn't work and each file has to be downloaded individually. I downloaded all files and packaged them up into multiple archives based on their category directory they were sorted into. Also the original source no longer exists, therefore this is a reupload for archival purpose.

Legacy information from original upload:

Complete: Entire collection of original upload. Each category is archived separately for easy download. The original 14 GiB full set of 29k compressed Roms. Every single Rom file is compressed as 7z.

Reduced: A script clean_smwdb.sh tries to detect multiple versions of a Rom and keeps only the newest entry of it. Furthermore bad files and unneeded categories (such as Jokes and Trash) are deleted. This reduces the total number of files from 29k to "only" 8k uncompressed Roms. Also while each category is still packaged for batch downloads, every single Rom file in it are already extracted (unlike the Complete packages).

5
 
 

YouTube is still one of the major points of centralization on the internet, so I've been brainstorming ways around the problem.

From the readme:


Torrent-Tube is a set of tools to help decentralize YouTube videos, by moving them to torrents, which can be shared by many people. It includes:

  • A Torrent-Tube search site which searches the Torrents-csv search engine to see if the given YouTube video already exists, and is being seeded.
    • It does this by extracting the YouTube [VIDEO_ID] from a link, which you can also do manually if you like (IE, the text after watch?v=...).
  • A script to download, and create torrent files from YouTube videos, with a uniform naming style and format, taken from TheFrenchGhosty's YouTube-DL-Scripts.
  • You will need to upload these torrent files yourself to a service (details below), and seed them.

Torrent-Tube Search

In the future, it may be possible to create a browser plugin that checks a video link that you're currently watching for existing torrents.

Create torrent script

Requirements

Instructions

Copy a YouTube video URL.

# Clone this repo
git clone https://github.com/dessalines/torrent-tube

# Run the script
./create_torrent.sh [YOUTUBE_URL]

The video will download, and is saved in the videos folder. The torrent file is saved in the torrents folder.

Add the torrent to your torrent app, such as qbittorrent.

6
 
 

geteilt von: https://feddit.org/post/31252728

Dear Fedi friends, I need your help.

If by any chance any of you have saved the video of #WSocial CEO Anna Zeiter's presentation at the Monaco AI conference (uploaded to YouTube on May 21) I would be ETERNALLY grateful.

OG URL: https://youtu.be/xDNGHaISqs4

The 35 minute video included priceless quotes like: "Yes we have a business model, it's not a philanthropy, we are not a B corp, we actually want to make money."

Following the publication of my exposé, the video has been removed from YouTube!

7
 
 

I have found a few on 4chan. A rotating realistic chicken burger with a transparent background for example.

It seems to be mostly the same 20 or so GIFs in regular circulation.

Ideally I'd like to avoid individual downloads or signing up for an account. I thought someone here may have stumbled across a page on the Internet Archive or something.

8
2
submitted 2 months ago* (last edited 2 months ago) by tdTrX@lemmy.ml to c/datahoarder@lemmy.ml
 
 

1. Purpose

  1. Internal Disk - OpenZFS/ext4/NTFS ? for all OS (Win,Lin,mac)
  2. External Disk - Is exFAT OK ? Else OpenZFS/ext4 ? for all OS
  3. Backup Disk - guy says exFAT is bad for backup as it corrupts. so ***ext4/***OpenZFS ?
  4. SOHO Server/NAS - ZFS

2. OpenZFS vs BtrFS?

3.1 Is ZFS on Win/mac mature enough ?

3.2 If not Is Ext4 on Win/mac good ?

9
 
 

https://archive.org/details/nes-romhack-collection_202103-repacked

Total size of all 24.258 .nes Roms (plus a few other files) unpacked: 5.5gb


A huge collection of NES Rom Hacks.

Originally uploaded by MiroMirian at https://archive.org/details/nes-romhack-collection_202103 . This is a repackage of the same files. The original collection contains over 24 thousand files, without compression. This means that Internet Archive's internal "on-the-fly-zip" doesn't work and each file has to be downloaded individually. I downloaded all files and packaged them up in .7z archives. This reduces the download size drastically and also makes it possible to download all of them in one go.

Complete: Entire collection in a single archive. The original full set.

Split: Divided the collection by letter groups into multiple archives. Also removed all non .nes Rom files for additional cleanup. Otherwise contains the same Roms found in the Complete package.

10
 
 

cross-posted from: https://lemmy.sdf.org/post/40623875

Hello. Me and a few friends are attempting to backup every files from AndroidFileHost, and we need some help in doing so.

For those who haven't heard of it, AndroidFileHost is a website that hosts various Android related files. It's one of the last surviving large Android related file serving sites, and holds a LOT of rare files especially for older android devices. (rip d-h.st) Despite being such a valuable site, it hasn't been well maintained for the past few years. Their Xitter account's last update is from around 2022, and the owner isn't replying to any e-mails. The site has been extremely unstable with various issues, most recently no file could ever be downloaded from it for about a month. Luckily, it has been (kind of) solved for now, and most (not all, about 20% files are still gone) files are back online now. However, it's clear this site needs a backup.

I have scraped their website which gives us the unique ID and MD5 hash for every files available on the site. Now, using this ID we can automate the process of requesting mirror links, downloading them and checking for integrity. (Please check an example file to understand how their system works -- https://androidfilehost.com/?fid=745425885120701975 )

The sum of every file sizes we know is roughly 180TB. It's impossible to download this on a single machine, so I've developed a "tracker" system to concurrently download multiple files using different machines. The tracker server keeps a list of every known file IDs (btw, it's 256,640 files which is a bit less than 277,467 displayed on their main page. I believe it includes deleted files as well but not sure atm), assign it to each clients that request and appropriately mark the file as downloaded. The system is pretty robust now, so our plan is working great. Except that our internet is pretty slow and we can't afford 180TB instantly.

By talking to friends and their friends, we've got quite a few people willing to help a bit here. Unfortunately many of them lack storage space, so they need to keep downloading from AFH and uploading to my server. This works for a few clients, but not for many. The "my server" here every client uploads to have 500Mbps internet, and it gets terribly slow pretty quick. Plus, 180TB of storage isn't really cheap and easy to afford.

Ideally, we need to get people with faster internet speeds (I'm in asia, so not the best place to fetch files from AFH servers mostly around Europe and America) and more storage space. If you have some bandwidth or storage to share, it would greatly help us.

I'm sorry if a post like this isn't welcomed here, if so please feel free to remove it. Thanks for reading this post.

P.S. Also worth checking out - related XDA thread https://xdaforums.com/t/did-anyone-else-notice-signs-of-androidfilehost-com-being-abandoned.4578561/ (I'm LegendOcta)

11
 
 

I am upgrading the HDDs on my QNAP TS-432X-eU rack mount NAS. The NAS is connected to a UPS via a USB cable and is set to turn off after 5 minutes if it senses a power loss. What would happen if I were to lose power while resilvering the array? Would it suspend the resilvering, turn off, then resume when power is restored? Or would the array be corrupted?

12
 
 

I'm looking to spec out a new NAS. I have a relatively small media collection, that I hope to grow as I digitize more family VHS tapes etc. Right now I have around 4 TB of data, shared across an external drive and my internal ssd.

Whats the best path forward on drives in this new NAS? I've heard advice for buying one big 20TB drive over multiple smaller drives. What's best for mitigation of drive failure? Is that even a concern? If I do multiple drives, should I use RAID?

I'm a little new to this. If you have resources for learning some best practices I'm all ears.

13
 
 

Anyone used this successfully in their setup?

Garage is an S3-compatible distributed object storage service designed for self-hosting at a small-to-medium scale.

Garage is designed for storage clusters composed of nodes running at different physical locations, in order to easily provide a storage service that replicates data at these different locations and stays available even when some servers are unreachable. Garage also focuses on being lightweight, easy to operate, and highly resilient to machine failures.

Garage is built by Deuxfleurs, an experimental small-scale self hosted service provider, which has been using it in production since its first release in 2020.

14
 
 

Looking to build a collection that I just outright own, so any streaming platform that doesn't allow me to download the raw files is a no go. Other than the big players (Amazon, Walmart, etc.) what are some good sources for buying?

15
 
 

Looking to upgrade my NAS hard drives. Currently have two 4TB WD Red Plus hard drives but I wanted to get some large capacity drives. Was looking into getting 16 or 18TB drives. My current drives are basically whisper quiet and have been running great since 2019 but I feel like it's time to upgrade the capacity.

The NAS is currently on a desk beside my computer. I don't have any cabinets to place it in and would prefer not to connect to it through Wi-Fi. Hence why I'd like for the drives to be as quiet as possible.

I was considering getting a Seagate Exos or Ironwolf (and buying used for the great price) but I've read users online saying they regret buying those models because of their noise. I was also looking at the WD Red Pro but WD's own website only rates them at 3.6/5 with most of the negative complaints about dead on arrival drives. Additionally 25% of all reviews are 1 star; both of which don't fill me with much confidence.

TLDR: What's a quiet and reliable hard drive recommendation for a NAS?

Would it be better just to go with the WD Red Plus at a lower capacity?

16
 
 

Okay, so now that my little experiment with a bunch of scam nvme drive from amazon is done and over with and I got my money back from amazon. Where do I look for some cheap and semi decent 4tb nvmes? Adata used to by my goto budget flash memory, never had any problems with any of their drives. But they're not so inexpensive anymore... Team group seems like they have good prices but how reliable are they?

Is prime day or boxing day even a good time to buy drives?

Is there any 4tb nvme under $300(CAD) even worth looking at?

Again, I'm just farting around and experimenting but any suggestion will be greatly appreciated and win you imaginary internet points from a stranger sitting on a porcelain thrown as he writes this.

17
 
 

I'm trying to digitize some VHS tapes (presumably recorded as NTSC), but I have some questions that I've yet to find answers for. My current process/setup is as follows:

  • VHS tapes are played in a PV-D4745S-K VCR
  • The VCR's composite output is captured using a generic EasyCap capture card.
  • The captured output is fed into OBS Studio with the following settings:
    • A source with it's device set to the capture card, the video format is set to YUYV 4:2:2, the resolution is set to 720x480, the frame rate is currently set to Leave Unchanged (more on this later).
    • Under Settings>Video I have set Common FPS Values with 29.97.
    • I also have set my encoding options under Settings>Output, as well as audio settings under Settings>Audio, but the details of that aren't relevant in this context.
    • I also have deinterlacing disabled by right clicking on the scene and selecting Deinterlacing>Disable.

With this, I seem to be able to capture VHS tapes with decent quality, but I have some nagging questions:

  1. How do I verify if OBS has indeed captured interlaced? I'm trying to capture both fields, but I'm unsure if that's actually happening, and I'm not sure how to go about verifying it.
  2. Should I capture at 29.97 FPS or 59.94 FPS? My thinking is that, given that I'm capturing interlaced, I would think I would multiply the number of captured frames by 2 as, if I understand correctly, each captured frame contains 2 fields, and each would be captured sequentially, so if I want to capture at 29.97 FPS interlaced, I would need to capture at 59.94 FPS. I'm not sure if I'm right about that though.
  3. I mentioned above that the framerate under the source properties is set to Leave Unchanged. The reason for why I chose that option is because the only other options that it offers for framerates are 30.00, 20.00, 10.00, and 5.00 — ie there is no option for 29.97, nor 59.94 — so I'm using Leave Unchanged in the hopes that it's autodetecting the proper frame rate, but that's mostly an assumption on my part. The closest to NTSC's 29.97 would be 30.00, but I'm not sure if this is an issue. And what's confusing me more is that I have 29.97 FPS set under Settings>Video with Common FPS Values and 29.97 set. If I set to source framerate to 30 with OBS at 29.97, will that lead to syncing issues? Is there a way to force the source to use 29.97 to match OBS? What's confusing me further about this is that if I list the formats for the capture device with
    v4l2-ctl --device=/dev/video2 --list-formats-ext
    
    I get the following output (I have truncated it to only list what's relevant, as the full output is long and contains unnecessary information):
    […]
    [0]: 'YUYV' (YUYV 4:2:2)
        size: Discrete 720x480
    […]
         Interval: Discrete 0.033s (30.000 fps)
         Interval: Discrete 0.050s (20.000 fps)
         Interval: Discrete 0.100s (10.000 fps)
         Interval: Discrete 0.200s (5.000 fps)
    […]
    
    There is no option for 29.97 FPS, and, as can be seen by the output, it matches what OBS sees. Is this an issue? It seems, to me, that the capture card isn't capable of proper NTSC framerates, and can only capture at 30 FPS as the closest value.
18
 
 

So i'm testing one of the drives I got on amazon using an old computer. It started off promising, didn't get any errors when formating the drive, write speed using a usb adapter at 25mb/s... Then it dropped off to 7mb/s and the expected time shot up from 40hours to 156h🤣

I'm going to let it run over night and see what happens in the morning. If I can get it to show 4tb without error I might still keep them to test out my geekworm pi nas, otherwise back to amazon you go!

19
 
 

I've never transferred Pokemon between gens and I've never used Pokemon Home, but it seems wild to me to be so invested into such a fickle storage system. Thoughts and prayers for the guy affected

20
 
 

I stumbled upon that new use of mp4 format. Interesting.

21
 
 

I'm sure some of you already using it like this but if not, this could be useful for you.

It creates a directory with the channel's name, create sub-directories with the playlist name, it gives them a number and put them in an order, it can continue to download if you have to cancel it midway.

You can modify it to your needs.

Add this to your ~/.bashrc or your favourite shell config.

alias yt='yt-dlp --yes-playlist --no-overwrites --download-archive ~/Downloads/yt-dlp/archive.txt -f "bestvideo[height<=1080]+bestaudio/best[height<=1080]" -o "~/Downloads/yt-dlp/%(uploader)s/%(playlist_title,single_playlist)s/%(playlist_index,00)s - %(title)s - [%(id)s].%(ext)s"'

You can even limit the download speed by adding this parameter: --limit-rate 640K This example is for 5 Mb/s.

22
 
 

Harvard made available some politics courses entirely for free available on edX which made me nosy to check out other courses in the platform and I'm finding plenty of very interesting courses in there, is there a way to bulk download the courses?

23
 
 

Scouring through amazon for random stuff yesterday, saw these "generic brand" nvmes for $65 a pop. Figured I's give it a shot for my little geekworm pie nas. 4 for the raid and 1 for backup if something goes boogers up. 20tb for $325 was too good to pass up, worse case scenario they are either 1tb each or they fail after a few months. We'll see whats up when they get here in 2 weeks.

24
 
 

Hi, I'm having a bit of information overload. I'm trying to digitize my VHS collection as I know they'll stop working one day, and I'd like to keep their contents around after their analog form has parished. Ive tried startpaging this issue, but I'm honestly overwhelmed. The biggest take-away I have is avoid cheap USB capture devices (EasyCaps and generic variants.) What is the best option for an American on a fixed budget who wants to digitize their collection in the highest quality possible? Ive seen people recommending: TV Tuner Cards RF Video capture via MISRC and Domescape? i think were the names Using a combo DVD/VHS player to copy the VHS to DVD and then ripping the DVD with MakeMKV or handbrake Probably more, I can't recall at the moment but, I'm sure someone will.

Basically, if I dont have to spend money, I would prefer not to, but I do want a quality recording/rip. I know for a fact anything over $200 USD is going to be out if my budget. I have sufficient Linux knowledge and soldering knowledge and I have seen some people those are needed for the RF tools, but I also see some of those options are pricy.

Lemmings, what are your opinions on my situation? Is there anything short of an easycap in my budget that I should be looking into? What are your experiences? Thanks.

25
 
 

SnapRAID doesn't compute the parity in real time, so there's this window between making a change to the data and syncing where your data isn't protected. The docs say

Here’s an example, you acquire a file and save it to disk called ‘BestMovieEver.mkv’. This file sits on disk and is immediately available as usual but until you run the parity sync the file is unprotected. This means if in between your download and a parity sync and you were to experience a drive failure, that file would be unrecoverable.

Which implies that the only data at risk is the data that's been changed, but that doesn't line up with my understanding of how parity works.

Say we have three disks that store 1 bit of information and a parity drive: 101 parity 0. If we modify the data in the first disk (data 001 parity 0), then the data is out of sync. Say we now lose disk 2 (data 0?1 parity 0). How does it then recover that data? We're in an inconsistent state where the remaining data tells us that drive 2 used to hold 0^1^0=1 when it actually held a 0. So doesn't that mean that between any modifications and a sync operation, all your data in that disk region is now at risk? Does SnapRAID do anything special to handle this?

view more: next ›