• Re: innd: SERVER cant open /var/www/inn/inn_status.html: Permission den

    From Russ Allbery@21:1/5 to Jeffery Small on Tue Oct 4 08:27:45 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    I recently upgraded from Xubuntu 20.04 to 22.04.1. The upgrade was very rocky, but no changes were made (by me!) to the inn2 installation. Inn
    is the standard repository package version 2.6.4-2build4. The news
    system is working generally. Ever since the upgrade, I'm getting a
    steady stream of the error message (see subject line) but I cannot
    determine why. The file exists where it always has. (Note that
    /var/www is a symlink and the actual path is
    /x/u/www/inn/inn_status.html)

    % ls -l /var/www/inn/inn_status.html
    -rw-rw-r-- 1 news news 1057 Sep 2 15:25 /var/www/inn/inn_status.html

    Check the permissions on the parent directories, maybe? I'm wondering if something changed the permissions in /var/www (/x/u/www) such that the
    news user can't traverse it.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to All on Tue Oct 4 09:46:34 2022
    BTW, my home directory is /u/jeff and not located under /home. It's been this way for 40 years. I did make a symlink from /home to /u. I only mention this because snap packages totally break for "non-standard" $HOME and maybe the
    same is now true for other things*. Nevertheless, I can't see how this has any bearing on programs being run by root or news. I just mention it here in case it
    sparks a thought.
    --
    * Every day UNIX gets destroyed a little more. :-(

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Tue Oct 4 09:39:10 2022
    On Tuesday, October 4, 2022 at 8:27:47 AM UTC-7, Russ Allbery wrote:
    Jeffery Small writes:

    I recently upgraded from Xubuntu 20.04 to 22.04.1. The upgrade was very rocky, but no changes were made (by me!) to the inn2 installation. Inn
    is the standard repository package version 2.6.4-2build4. The news
    system is working generally. Ever since the upgrade, I'm getting a
    steady stream of the error message (see subject line) but I cannot determine why. The file exists where it always has. (Note that
    /var/www is a symlink and the actual path is
    /x/u/www/inn/inn_status.html)

    % ls -l /var/www/inn/inn_status.html
    -rw-rw-r-- 1 news news 1057 Sep 2 15:25 /var/www/inn/inn_status.html
    Check the permissions on the parent directories, maybe? I'm wondering if something changed the permissions in /var/www (/x/u/www) such that the
    news user can't traverse it.

    Yes that was my first thought. I checked all the paths and everything is kosher. However I just sued into news and walked to the directory without problem. But when I tried to manually edit the inn_status.html file I get the message:

    "Authorization requires, but no authorization protocol specified"

    Even if I su to root, I get the same message!

    I have a ~/.Xauthority file and (forever) I have had /.Xauthority as a symlink to this
    file. It looks like there have been some serious changes to the auth program that I'm
    unaware of and this may be an underlying problem here, but I really don't understand
    what is going on and how this affects daemon and cron programs, or why only this
    particular directory is affected. When I try to edit it myself, I don't get this message,
    but (of course) can only open it RO.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Tue Oct 4 10:17:14 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    Yes that was my first thought. I checked all the paths and everything
    is kosher. However I just sued into news and walked to the directory
    without problem. But when I tried to manually edit the inn_status.html
    file I get the message:

    "Authorization requires, but no authorization protocol specified"

    This is probably unrelated. What editor are you using? It sounds like
    it's trying to open an X connection. (If Emacs, use emacs -nw instead.)

    Is this file system mounted over the network, possibly?

    Another possibility is that you have a corrupt file system.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Tue Oct 4 12:48:40 2022
    On Tuesday, October 4, 2022 at 10:17:15 AM UTC-7, Russ Allbery wrote:
    This is probably unrelated. What editor are you using? It sounds like
    it's trying to open an X connection. (If Emacs, use emacs -nw instead.)

    Is this file system mounted over the network, possibly?

    Another possibility is that you have a corrupt file system.

    I was just using the vim (non-graphical) text editor inside a terminal window. Root is on a mirrored SSD and the file itself is on a separate ext4 hard disk (actually two mirrored disks) mounted a /x, so no funny business there and
    the mirror reports no problems.

    I tried other editors such as mousepad and gedit and get the same results
    as with vim -- except that these others open a new window while vim
    stays withing the current terminal..

    I believe you are correct that this is an unrelated issue, and I will look into it further. That brings us back to the original problem....

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Tue Oct 4 13:24:00 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    I believe you are correct that this is an unrelated issue, and I will
    look into it further. That brings us back to the original problem....

    Oh, what does /lib/systemd/system/inn2.service look like? I bet that innd
    is now started with systemd following your upgrade and probably has
    various security features enabled, and you're getting bitten by the fact
    that /var/www is a symlink off to some other file system and that file
    system probably isn't set up to be writable by the innd process.

    If something like ProtectSystem=strict is set in that file, you may need
    to create a directory named /etc/systemd/system/inn2.service.d and drop a
    file in there named something like www.conf and containing:

    [Service]
    ReadWritePaths=/var/www/inn

    to whitelist an additional writable path. I think it will automatically resolve symlinks; if not, you may need:

    [Service]
    ReadWritePaths=/x/u/www

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Wed Oct 5 20:09:23 2022
    On Tuesday, October 4, 2022 at 1:24:01 PM UTC-7, Russ Allbery wrote:
    I believe you are correct that this is an unrelated issue, and I will
    look into it further. That brings us back to the original problem....
    Oh, what does /lib/systemd/system/inn2.service look like? I bet that innd
    is now started with systemd following your upgrade and probably has
    various security features enabled, and you're getting bitten by the fact that /var/www is a symlink off to some other file system and that file system probably isn't set up to be writable by the innd process.

    If something like ProtectSystem=strict is set in that file, you may need
    to create a directory named /etc/systemd/system/inn2.service.d and drop a file in there named something like www.conf and containing:

    [Service]
    ReadWritePaths=/var/www/inn

    to whitelist an additional writable path. I think it will automatically resolve symlinks; if not, you may need:

    [Service]
    ReadWritePaths=/x/u/www

    Russ:

    Just a quick follow-up. I tried your suggestions but unfortunately, it didn't work. Below I'll list the primary systemd inn2.service file as well as my addition. I did add multiple directory paths and reloaded the daemon and restarted inn2. Still I
    get a stream of errors. ProtectSystem is set to full. I'm new to systemd and have been attacking it piecemeal. I'm going to read up on the whole thing so I have a better understanding of what's going on. In the meantime, any further suggestions are
    always welcome.

    % cat /usr/lib/systemd/system/inn2.service
    [Unit]
    Description=InterNetNews
    Documentation=man:innd(8)
    After=network-online.target
    Wants=network-online.target

    [Service]
    Type=notify
    Restart=on-abort
    ExecStart=/usr/lib/news/bin/rc.news
    ExecStop=/usr/lib/news/bin/rc.news stop
    ExecReload=/usr/sbin/ctlinnd -t 20 reload all 'by systemd'
    User=news
    Group=news
    ConfigurationDirectory=news
    LogsDirectory=news
    LogsDirectoryMode=775
    RuntimeDirectory=news
    StateDirectory=news
    StateDirectoryMode=775
    ReadWritePaths=/var/spool/news/
    ProtectSystem=full
    ProtectControlGroups=yes
    ProtectHome=yes
    # These directives are not compatible with innbind (or postdrop from Postfix)
    # because they automatically enable NoNewPrivileges:
    # PrivateDevices=yes
    # ProtectClock=yes
    # ProtectHostname=yes
    # ProtectKernelLogs=yes
    # ProtectKernelModules=yes
    # ProtectKernelTunables=yes
    # RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
    # RestrictNamespaces=yes
    # RestrictRealtime=yes
    # RestrictSUIDSGID=yes
    # LockPersonality=yes
    # MemoryDenyWriteExecute=yes
    # SystemCallArchitectures=native
    # SystemCallErrorNumber=EPERM
    # SystemCallFilter=@system-service
    LimitNOFILE=infinity

    [Install]
    WantedBy=multi-user.target

    $ cat /etc/systemd/system/inn2.service.d/www.conf
    [Service]
    ReadWritePaths=\
    /var/www/ \
    /var/www/inn/ \
    /x/u/www/ \
    /x/u/www/inn/

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Wed Oct 5 20:36:04 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    Just a quick follow-up. I tried your suggestions but unfortunately, it didn't work. Below I'll list the primary systemd inn2.service file as
    well as my addition. I did add multiple directory paths and reloaded
    the daemon and restarted inn2. Still I get a stream of errors.
    ProtectSystem is set to full.

    Ah, that's not it, then. ProtectSystem=full doesn't do anything with /var
    or other file systems. It just does /usr, /boot, /efi, and /etc.

    I'm still pretty confused! Definitely worth trying to write to the file
    after su to the news user, just to rule out all the normal UNIX permission things.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Wed Oct 5 21:02:37 2022
    On Wednesday, October 5, 2022 at 8:36:06 PM UTC-7, Russ Allbery wrote:
    Ah, that's not it, then. ProtectSystem=full doesn't do anything with /var
    or other file systems. It just does /usr, /boot, /efi, and /etc.

    I'm still pretty confused! Definitely worth trying to write to the file after su to the news user, just to rule out all the normal UNIX permission things.

    Just happened to still be at my desk. I did su to news and I am able to create a new file in the /var/www/inn directory and append to the inn_status.html file, so there is no Linux permission problem. If I can do that, why can't the service programs?

    I still have the xauth problem and, as news, cannot edit the file with vim. I fixed the problem for root by removing the existing .Xauthority file in my home directory and at /. I see root now has a new home directory! I also found an .Xauthority file
    in the news home directory (/var/spool/news) and eliminated that, but that didn't fix the problem. Still looking.....

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Wed Oct 5 21:14:55 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    Just happened to still be at my desk. I did su to news and I am able to create a new file in the /var/www/inn directory and append to the inn_status.html file, so there is no Linux permission problem. If I can
    do that, why can't the service programs?

    Okay, then it's some sort of security control related to the service
    startup. It's not immediately jumping out to me in the service unit,
    though. Do you have SELinux or AppArmor or anything like that enabled? (Ubuntu, so AppArmor is probably more likely. I don't know if INN has
    AppArmor rules, though.)

    For the record, the reason for all this is that Linux now supports
    limiting what a service can access so that if there's a security
    vulnerability in the service, it doesn't compromise your whole system.
    It's in general a very good thing (particularly for fairly old C code like
    INN) and it makes Linux systems way more secure, but like every security measure there's a usability trade-off and some amount of faffing about
    trying to figure out where the security rule blocking the thing that the service needs to do is.

    In this case, I'm pretty sure the symlink to a different file system is
    the root of the problem, I'm just not sure how.

    I still have the xauth problem and, as news, cannot edit the file with
    vim. I fixed the problem for root by removing the existing .Xauthority
    file in my home directory and at /. I see root now has a new home
    directory! I also found an .Xauthority file in the news home directory (/var/spool/news) and eliminated that, but that didn't fix the problem.
    Still looking.....

    For some reason all the editors you're using including vim are trying to connect to your X display, I think. One thing to try would be to unset
    the DISPLAY environment variable before trying to edit the file, which
    should hopefully clue in editors that support either using X or not that there's no X display they have access to.

    It's possible that your vim actually ended up being vim-gtk3 or something during the upgrade.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Thu Oct 6 16:43:11 2022
    On Wednesday, October 5, 2022 at 9:14:57 PM UTC-7, Russ Allbery wrote:
    Okay, then it's some sort of security control related to the service
    startup. It's not immediately jumping out to me in the service unit,
    though. Do you have SELinux or AppArmor or anything like that enabled? (Ubuntu, so AppArmor is probably more likely. I don't know if INN has AppArmor rules, though.)

    AppArmor is installed and the service is running. I have never touched
    this and there is no inn2 profile, so I don't think it is the problem.

    In this case, I'm pretty sure the symlink to a different file system is
    the root of the problem, I'm just not sure how.

    I see that apache2 doesn't have any access problems. I looked in the /etc/apache2/apache2.conf file and see the following lines that have
    been there for many years:

    <Directory /u/www/>
    AddHandler cgi-script .cgi
    Options Includes Indexes FollowSymLinks ExecCGI MultiViews
    AllowOverride All
    Require all granted
    </Directory>

    <Directory /var/www/>
    AddHandler cgi-script .cgi
    Options Includes Indexes FollowSymLinks ExecCGI MultiViews
    AllowOverride All
    Require all granted
    </Directory>

    So, at some point, I apparently added a duplicate entry for /u/www
    which seems strange because the disk is actually mounted on /x
    and /u is a symlink pointing to /x/u just as /var/www is a symlink
    to /u/www -- which is actually a path with two symlinks.

    So, regarding apache2, there doesn't seem to be a restriction regarding symlinks.

    In /etc/news/inn.conf file there is only one entry that points to a symlink path:

    pathhttp: /var/www/inn

    I guess there are two things I could do. 1) change the path to eliminate symlinks:

    pathhttp: /x/u/www/inn

    or 2) completely change the location of the directory to somewhere on the
    root drive. For example:

    pathhttp: /var/inn2

    The only thing under the current /var/www/inn directory is the sole inn_status.html file. However, I'm not sure what making a change line
    this would do to the overall inn2 operations.

    Suggestions?

    For some reason all the editors you're using including vim are trying to connect to your X display, I think. One thing to try would be to unset
    the DISPLAY environment variable before trying to edit the file, which
    should hopefully clue in editors that support either using X or not that there's no X display they have access to.

    It's possible that your vim actually ended up being vim-gtk3 or something during the upgrade.

    You are correct. Unsetting the DISPLAY variable allowed the news user to
    edit the file. I am going to have to look into what new things auth is doing under the hood. One thought I had was that user news had a home directory
    of /var/spool/news, but cannot login since the shell is set to "nologin". Other
    users get a ~/.Xauthority file created when they first login. (Copying my file to that location doesn't work.) Is there a way to force non-login users to get an .Xauthority file? If not, maybe I need to add a shell, login to create the auth file and then reset it to nologin.

    Thoughts?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Thu Oct 6 17:34:22 2022
    Jeffery Small <jefferysmall@gmail.com> writes:
    On Wednesday, October 5, 2022 at 9:14:57 PM UTC-7, Russ Allbery wrote:

    In this case, I'm pretty sure the symlink to a different file system is
    the root of the problem, I'm just not sure how.

    I see that apache2 doesn't have any access problems.

    There's definitely some sort of security configuration at play that's
    specific to innd. (Either that or innd isn't actually running as the user news, but presumably you've checked that with ps already.)

    I am suspicious that it's something that mounts /x read-only for innd,
    which is definitely a thing that one can do with systemd security configuration, but I don't know why it's happening here because your
    systemd configuration does not seem to be doing that.

    In /etc/news/inn.conf file there is only one entry that points to a symlink path:

    pathhttp: /var/www/inn

    I guess there are two things I could do. 1) change the path to eliminate symlinks:

    pathhttp: /x/u/www/inn

    or 2) completely change the location of the directory to somewhere on the root drive. For example:

    pathhttp: /var/inn2

    The only thing under the current /var/www/inn directory is the sole inn_status.html file. However, I'm not sure what making a change line
    this would do to the overall inn2 operations.

    This is the only thing about innd behavior that setting controls. But I
    didn't mean that the fact that path points to a symlink is inherently a problem.

    I'm trying to figure out what's different on your system than on other
    Ubuntu systems where the inn2 package presumably works (or there would be
    a lot of other bug reports). The one thing that we've found so far that
    seems to be different on your system is that your /var/www is a symlink to
    a different file system, and I know that some systemd security
    configuration for daemons mounts different file systems read-only for
    security protection. That's consistent with the symptoms that you're
    seeing where innd cannot write to that file with a permission denied
    error, but when you su to the news user (which doesn't involve any of the systemd configuration) you can write to the file just fine.

    You are correct. Unsetting the DISPLAY variable allowed the news user
    to edit the file. I am going to have to look into what new things auth
    is doing under the hood. One thought I had was that user news had a
    home directory of /var/spool/news, but cannot login since the shell is
    set to "nologin". Other users get a ~/.Xauthority file created when
    they first login. (Copying my file to that location doesn't work.) Is
    there a way to force non-login users to get an .Xauthority file? If
    not, maybe I need to add a shell, login to create the auth file and then reset it to nologin.

    In general, you no longer have X access to spawn new X clients after
    running su or sudo or the like. I know there's something you can do to
    restore access, but I never do this and therefore don't remember what it
    is. (Also, I'm not sure if you're running Wayland rather than X; if
    that's the case, it has a whole different permissions model and I have not
    yet learned how it works.)

    I don't think logging on as the user will do it since the web session is
    still under your normal user, not under that different user.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to All on Sat Oct 8 07:04:54 2022
    Well, I still don't have a systemd solution or answer to the problem,
    but I changed the /etc/news/inn.conf line to:

    pathhttp: /x/u/www/inn

    which is the original location on the mounted disk, but bypassing
    the symlink. I restarted the inn2 service, but this also fails.

    I then created a new directory, /var/inn, which is on the primary
    drive and placed the inn_status.html file there. I changed the /etc/news/inn.conf line to:

    pathhttp: /var/inn

    and restarted the inn2 service. No more error messages and the
    file is now updating. So the problem is with the file being on the
    mounted disk and doesn't seem to be related to the symlink.
    Does this provide a clue? I'm not seeing any problems elsewhere.
    The /u/www (/x/u/www) directory contains all the html files used
    by apache2 and /u (/x/u) contains home directories, the location
    of my KVM images and other misc. things. VM, apache2 and other
    programs that access music (e.g., clementine) or picture (e.g., gimp)
    files read/write without problem.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Jeffery Small on Sat Oct 8 08:58:57 2022
    Jeffery Small <jefferysmall@gmail.com> writes:

    Well, I still don't have a systemd solution or answer to the problem,
    but I changed the /etc/news/inn.conf line to:

    pathhttp: /x/u/www/inn

    which is the original location on the mounted disk, but bypassing
    the symlink. I restarted the inn2 service, but this also fails.

    I then created a new directory, /var/inn, which is on the primary
    drive and placed the inn_status.html file there. I changed the /etc/news/inn.conf line to:

    pathhttp: /var/inn

    and restarted the inn2 service. No more error messages and the
    file is now updating. So the problem is with the file being on the
    mounted disk and doesn't seem to be related to the symlink.

    Yup, that's what I thought.

    Something is making that file system read-only for innd. I'm not sure
    what. I still suspect systemd security controls somewhere, something
    about either ProtectHome=true or ProtectSystem=full that I'm not
    understanding and that isn't overridden by ReadWritePaths for some reason.

    One thing you could do to work around the problem, although unsatisfying
    since it doesn't explain why, is symlink the other direction: have INN
    write to /var/inn or whatever, and make a symlink from /x/u/www/inn to
    that directory so that your web server can still serve it. The size of
    files that INN will write are quite small, so it shouldn't be a problem
    with available space.

    --
    Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

    Please post questions rather than mailing me directly.
    <https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeffery Small@21:1/5 to Russ Allbery on Sat Oct 8 17:00:35 2022
    On Saturday, October 8, 2022 at 8:58:58 AM UTC-7, Russ Allbery wrote:
    One thing you could do to work around the problem, although unsatisfying since it doesn't explain why, is symlink the other direction

    Done! Russ, thanks for the help and let me know if I can do anything to further help nail down the real issue.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)