this post was submitted on 06 Mar 2026
18 points (80.0% liked)

Sysadmin

13471 readers
9 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

founded 2 years ago
MODERATORS
 

I work on an HPC and often I have to share files with other users. The most approachable solution is to have an external cloud storage and recline back and forth. However there's some projects that are quite heavy (several TB) and that is unfeasible. We do not have a shared group. The following is the only solution I found which is not to just set al permissions to 777, and I still don't like it.

Create a directory and set ACL to give access to the selected users. This works fine if the users create new files in there, but it does not work if they copy from somewhere else as default umask is 022. Thus the only appropriate solution is to change default umask to 002, which however affects file creation system wide. The alternative is to change permissions every time you copy something, but you all know very well that is not going to happen.

Does it really have to be such a pain in the ass?

top 50 comments
sorted by: hot top controversial new old
[–] chris@l.roofo.cc 4 points 2 days ago

You can set acls on directories that get applied recursively. This makes ist possible to have all files be the correct permission. I am on the go right now but you should look into setfacl. It's been a while but I am pretty sure that worked. That way you should even be able to say which groups or users can do what with granularity.

[–] bjoern_tantau@swg-empire.de 17 points 3 days ago (7 children)

Uh, why not create the shared group? That's more or less exactly the purpose of their existence.

load more comments (7 replies)
[–] warmaster@lemmy.world 13 points 3 days ago (17 children)

I'm no sysadmin, I just run my homelab. Let me get this straight... You want to bypass system level access level restrictions with some form of control but not go through your company's standard method of doing so because of bureaucracy?

If that's the case: why not put something in front Like opencloud for example?

I mean, maybe OC is not what you need, but conceptually... would a middleman solution work for you? If so, you could go with a thousand different alternatives depending on your needs.

load more comments (17 replies)
[–] blackbirdbiryani@lemmy.world 3 points 2 days ago* (last edited 2 days ago) (2 children)

I'm in a similar position as you. Our lab has a partition on HPC but i need a way to quasi-administrate other lab members without truly having root access. What I found works is to have a shared bashrc script (which also contains useful common aliases and env variables) and get all your users to source it (in their own bashrc files). Set the umask within the shared bashrc file. Set certain folders to read only (for common references, e.g. genomes) if you don't want people messing with shares resources. However, I've found that it's only worth trying to admin shared resources and large datasets, otherwise let everyone junk their home folder with their own analyses. If the home folder is size limited, create a user's folder in the scratch partition and let people store their junk there however they want. Just routinely check that nobody is abusing your storage quota.

EDIT: absolutely under no circumstances give people write access to raw shared data on hpc. I guarantee some idiot will edit it and mess it up for everyone. If people need to rename files they can learn how to symlink them.

[–] biber@feddit.org 2 points 2 days ago (1 children)

This is a pretty good idea!

In addition, I recommend having all data e.g. as a (private)datalad archive synchronized to Dataverse, osf, figshare or wherever - edits are versioned then

[–] ranzispa@mander.xyz 1 points 2 days ago (1 children)

I am generally using DVC to version data, are those better options?

[–] biber@feddit.org 1 points 2 days ago

I don't know, seems to be quite similar :)

[–] ranzispa@mander.xyz 1 points 2 days ago (1 children)

Thanks, this is a great idea! I can see you have been doing this for a long time and you're talking from experience. Regarding shared data: I use this more as a way to give raw data to other people and collect results from them. I use it mostly as a temporary directory used to transfer data, anything significant will get copied over to my share and backed up.

I can see how you could be worried about storage quota, luckily I don't have that many people to worry about. But it is funny you mention it as I could really see someone stashing a few conda environments in there just because they finished their inside quota...

[–] blackbirdbiryani@lemmy.world 1 points 2 days ago (1 children)

If you're not that worried about storage then you can just make copies if necessary, then you don't really have to worry about permissions (apart from read, which is typically default for the same group). But yea if there's any chance more than 1 person might work off the same copy of data on HPC, make it read only for the peace of mind. Regarding conda envs, yea I have a few common read only conda environments so that scripts can be used by multiple users without the hassle of ensuring everyone has the same env. Quite useful.

[–] ranzispa@mander.xyz 1 points 1 day ago

The shared environment thing seems like a very cool idea! I'll try to set it up.

[–] poinck@lemmy.world 1 points 2 days ago (1 children)

I have a similar need and I am curious whether my current solution is any good:

The data of interest is on a server which can only be accessed with ssh inside the institution. I've setup a read-only nfs share to a server which has a webserver (https enabled). There, I set up a temporary webdav share to the read-only nfs mount point and protected with htpasswd, hence external institution members do not have accounts at our institution.

As soon as the transfer is complete I remove all the shares (nfs, webdav).

[–] ranzispa@mander.xyz 2 points 2 days ago

This is a good idea and something I may setup once we setup our own compute server. However at that point wouldn't a synced directory be a better fit for the purpose? Such as you define a directory on the external server to be used to share data and every user syncs it to their own share on the main server to get all the shared data through rsync or unison.

Just throwing it out there, I'm not sure if that fits your use case.

[–] twack@lemmy.world 3 points 3 days ago* (last edited 3 days ago)

Here's someone that solved this by monitoring the directory using inotifywait, but based on the restrictions you already mentioned I'm assuming you can't install packages or set up root daemons, correct?

https://bbs.archlinux.org/viewtopic.php?id=280937

Edit: CallMeAI beat me with this exact same answer by 15 minutes.

[–] linuxguy@piefed.ca 2 points 3 days ago (5 children)
load more comments (5 replies)
[–] frongt@lemmy.zip 2 points 3 days ago (14 children)

A dedicated file sharing application.

load more comments (14 replies)
load more comments
view more: next ›