Looks like you're on Fedora Silverblue (or other Atomic version). This is happening because the system groups are in /usr/lib/group rather than /etc/group and this causes the issue you're seeing here. You can work around it by getting into a root shell with something like

sudo -i

and then getting the group added to /etc/group with

grep -E '^dialout' /usr/lib/group >> /etc/group

after that, you'll be able to add your user to the group with

usermod -aG dialout pipe