this post was submitted on 02 Jul 2026
8 points (100.0% liked)

Technology

42853 readers
79 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.

Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.

Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 7 years ago
MODERATORS
MinutePhrase@lemmy.ml
8
How to deduplicate sentences in large .odt and .txt files? (lemmy.ml)
submitted 22 hours ago by zdhzm2pgp@lemmy.ml to c/technology@lemmy.ml
11 comments fedilink hide all child comments
 

Over the years, I've downloaded a lot of old emails to my laptop, which I saved as .eml files and then converted to .docx, .odt, and .txt files (mostly the last two).

The unfortunate habit of almost everyone (including myself, but no longer!) of quoting the original message as part of an email exchange has left me with text file(s) full of repeated sentences and paragraphs. What I've tended to do is to dump all the text from a one on one correspondence into a single file ("Dad-Erinaceus Complete Correspondence.odt," for example) and then try cleaning it up and re-ordering the messages by date.

Apart from the emails, I have I guess what you'd call a "journal" which is a very long .odt file that runs to about 300 pages or so. Much of this has the same sentences and paragraphs over and over again, but sometimes with slight variations that I would like to keep. So far, in either .odt or .txt files, I've started by searching for the first sentence, deleting subsequent appearances of it, and then going on to the next sentence, and so forth. Very time consuming! Is there a faster (and safe) way to do this?

There is quite possibly a very simple solution to this that I haven't thought of, but I'd be much obliged for any suggestions.

top 11 comments
sorted by: hot top controversial new old
[–] supersquirrel@sopuli.xyz 3 points 19 hours ago* (last edited 18 hours ago) (1 children)

Looks like uniq might be helpful, it appears built precisely like what you want.

https://www.ibm.com/docs/en/aix/7.2.0?topic=u-uniq-command

https://www.geeksforgeeks.org/linux-unix/uniq-command-in-linux-with-examples/

emacs

https://www.gnu.org/software/emacs/

https://susam.net/cc/mastering-emacs/ch05.html

  • M-x delete-duplicate-lines RET: Delete all but one copy of duplicate lines in region. When executed on the whole of the example buffer presented above, it leaves us with three non-empty lines and one blank line. When duplicate lines are encountered, the first instance of each line is kept intact and the others are deleted.

https://rants.org/2023/01/count-fold-lines

https://emacsredux.com/blog/2014/03/01/a-peek-at-emacs-24-dot-4-delete-duplicate-lines/

[–] zdhzm2pgp@lemmy.ml 1 points 19 hours ago

What is different between uniq and awk?

[–] pheonixdown@sh.itjust.works 5 points 21 hours ago (1 children)

Python should be able to handle this for you with 100% accuracy at removing duplicated sentences (and headers), with added complexity you could have it even make result files based on unique emails you corresponded with, and it can handle all the various file types and even nested folders if that's involved.

Probably could do this in 50-100 lines of code depending on complexity. Runtime should be minutes at most.

Learning python to do this would probably take a couple to a few hours if you have any familiarity with coding, bit more if you don't.

[–] zdhzm2pgp@lemmy.ml 1 points 21 hours ago (2 children)

Hi @pheonixdown@sh.itjust.works, thanks for your response. Unfortunately I have no knowledge of python and my coding abilities are essentially nonexistent 🙁 . . .

[–] pheonixdown@sh.itjust.works 1 points 19 hours ago (1 children)

I doubt you'll find anything for this bespoke purpose built as a utility online, so if you want it you'll need to make it (or I guess hire someone).

Python is fairly approachable to learn, the 2nd best time to plant a tree is today. Personally, I'd recommend installing Anaconda and using a Jupyter Notebook. For this process, you'd need to learn about:

  1. "pip" to install any libraries/packages you need.

  2. Importing packages

  3. Working with strings

  4. Getting a list of all files (maybe only those of certain types) in a given directory.

  5. For loops

  6. some way to read each file type (pandas can handle txt, not sure about the others but you should be able to literally just Google "Python import " to find a package. Import them as strings and concatenate a huge list of them in a for loop.

  7. Pandas dataframes, Parsing strings into new columns, including into distinct lists of strings. The complexity here varies a lot based on what you specifically want to get out of the process.

  8. Exporting dataframe rows to files (better imho if you did it as tabs in a workbook, but it is trickier) in a for loop.

Don't overwrite objects as you go and keep different logical chunks in their own cells. That'll make it easier to troubleshoot. Just Google any errors you don't immediately understand, you'll find lots of advice online.

Don't have whatever you make delete any of your original files.

Depending on your feelings about AI, you could probably get one to make something that'll work after a few tries and troubleshooting by giving it error messages, but aside from the ethical problems, you'd never really know if it got it right, even if it seemed to work, so I would recommend against it.

[–] zdhzm2pgp@lemmy.ml 1 points 19 hours ago

You are clearly on a higher level of consciousness than I will ever be (plus I'm old 👴).

[–] Zarobi@aussie.zone 1 points 20 hours ago (1 children)

I have good news for you, you get to spend days learning an entire skillset to solve 1 specific problem!

In all seriousness though, there's actually a big problem with email "quoting" as you say, that each email client will re create the entire email chain in it's own format, leading to a complete mess.

If you still have access to that email client, it might be easier to see if there's an export feature to export them as actual email files, which would be much easier to process than a word document.

Another option, if you are ok with "close enough" and some inaccuracies, you could ask an A.I. like Claude to process these files for you.

Otherwise you're probably stuck with manual tedious processing. Personally I would just start again and ignore the file you have. Create an actual indexed word document with a table of contents so you can actually find anything. Add a header per email topic, with a sub header per message / reply. Manually copy paste, being careful not to include the quote this time.

[–] zdhzm2pgp@lemmy.ml 1 points 19 hours ago (1 children)

Text, as far as I know (?) doesn't take up that much space memory-wise, but on the other hand can you imagine the amount of server space that gets taken up by redundant text from people's emails (just for starters)?

Some of my approaches to the apparently much-hated "Inbox Zero" are to be diligent about emptying the trash, not keeping copies of emails in my sent folder, and most importantly, to download messages, personal or professional ones (mostly the former) that I find personally valuable, and then delete the original email so it isn't floating around forever in cyberspace. This leaves me with lots of txt files, which, as I said above, I then merge all together into one file and then edit. You'd think that there'd be a tool that cleans up the contents of email messages; there do appear to be some online ones, but I don't trust them. While researching this problem, I found out about "(g)awk" and am now fiddling around with it to see if I can successfully clean up my files without completely messing them up, as in

awk '!seen[$0]++' input.txt > output.txt

making sure that I experiment on a duplicated file and not on the original. I barely know what I'm doing here, but I'll see what I can come up with . . .

[–] Zarobi@aussie.zone 2 points 19 hours ago* (last edited 19 hours ago) (1 children)

Inbox zero doesn't mean deleting all your emails lol. It just means getting them out of your inbox. I do this myself. For each email in my inbox I "do something" with it instead of just leaving it there. Put it in a folder, tag it, spam, delete, archive, anything. Personally I use 3 zones:

  • inbox: currently to-do. Things I need to reply to or action I "flag" so they're at the top. Everything else is moved either to
  • archive: things I might need in the future. Any important correspondence. Bills. Etc. The important part is it is "out of sight". I use labels to tag everything, I find this is easier to search and navigate. Plus if you put a label on the email, you know why you kept it, it forces you to quantify that reason.
  • trash. Auto deleted after 30 days in case I change my mind and actually need it. Most emails go to trash. Aggressively unsubscribe from newsletters.
[–] zdhzm2pgp@lemmy.ml 1 points 19 hours ago

🙂👍

[–] adarza@piefed.ca 1 points 19 hours ago

it would compress well for an archival backup, so that's what i'd do for the 'originals'.

if your long message chains look anything like mine, there's far more quoted material overall than new text in most mails; and not all new text would be relevant to whatever is being saved.. so it'd be quicker to do it the other way--copy and paste what you did want to new documents instead of trying to clean up these long compilations by deleting what you don't.