• New mini project: mnpgr.

    From Kaz Kylheku@21:1/5 to All on Sun Oct 15 00:46:43 2023
    Hi All,

    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    This is glue code, written in TXR Lisp which turns Vim into a
    page for reading man pages.

    Plus, a small Vim syntax filee.

    https://www.kylheku.com/cgit/mnpgr/about/

    The benefits are

    - Remembers your last position in any man page. This is useful for
    working with long man pages.

    - Leverages your Vim skills and environment.

    - Has better rendering of backspace overstrikes than the "less" utility.
    For instance, it correctly renders the French accented characters
    exemplified in the GNU grep man page.
    - bold+italic is recognized and assigned to its own highlight
    category.

    The program translates man's overstrikes into its own markup notation;
    for instance, bold text is {B{bold text}B}. The Vim syntax file hides
    the markup and colorizes the payload.

    --
    TXR Programming Language: http://nongnu.org/txr
    Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
    Mastodon: @Kazinator@mstdn.ca
    NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Kaz Kylheku on Sun Oct 15 17:56:22 2023
    On 15.10.2023 02:46, Kaz Kylheku wrote:
    Hi All,

    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    This is glue code, written in TXR Lisp which turns Vim into a
    page for reading man pages.

    Plus, a small Vim syntax filee.

    https://www.kylheku.com/cgit/mnpgr/about/

    The benefits are

    - Remembers your last position in any man page. This is useful for
    working with long man pages.

    - Leverages your Vim skills and environment.

    - Has better rendering of backspace overstrikes than the "less" utility.
    For instance, it correctly renders the French accented characters
    exemplified in the GNU grep man page.
    - bold+italic is recognized and assigned to its own highlight
    category.

    The program translates man's overstrikes into its own markup notation;
    for instance, bold text is {B{bold text}B}. The Vim syntax file hides
    the markup and colorizes the payload.

    I don't think I'll use 'txr', but your post made me think about the
    task...

    If 'vim' is your pager you could do

    man ... | vim -

    :set syntax=man will also do some (predefined) highlighting. While in
    Vim you can of course not only remember the "last position" (there's
    the default mark, a single quote, for that) but you can also define
    arbitrary marks and, of course, randomly jump between them. Or split
    the screen to see more than one man page context. You can also (e.g.)
    extract portions of the man page. Do whatever an editor supports.

    A reason against using an editor is probably the size consideration.

    What doesn't seem to work nicely is setting the pager to vim

    PAGER='vim -' man ...

    that will (as opposed to the pipe above) show all the overstrikes you
    mention above. Simply adding a pipe for col(1) (to remove that junk)
    isn't straightforward it seems.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Janis Papanagnou on Sun Oct 15 17:18:24 2023
    On 2023-10-15, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:
    On 15.10.2023 02:46, Kaz Kylheku wrote:
    Hi All,

    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    This is glue code, written in TXR Lisp which turns Vim into a
    page for reading man pages.

    Plus, a small Vim syntax filee.

    https://www.kylheku.com/cgit/mnpgr/about/

    The benefits are

    - Remembers your last position in any man page. This is useful for
    working with long man pages.

    - Leverages your Vim skills and environment.

    - Has better rendering of backspace overstrikes than the "less" utility.
    For instance, it correctly renders the French accented characters
    exemplified in the GNU grep man page.
    - bold+italic is recognized and assigned to its own highlight
    category.

    The program translates man's overstrikes into its own markup notation;
    for instance, bold text is {B{bold text}B}. The Vim syntax file hides
    the markup and colorizes the payload.

    I don't think I'll use 'txr', but your post made me think about the
    task...

    If 'vim' is your pager you could do

    man ... | vim -

    :set syntax=man will also do some (predefined) highlighting. While in

    The problem is that vim will not remember the position in a buffer that
    had been read from a pipe. This whole thing was prompted by someone
    trying to solve that problem. They wanted to remember the position in
    each individual man page. That resonated with me because there have been
    times when I wanted to revisit something I recently read in a man page.

    (One such situation is when I'm working *on* a man page, making
    adjustments to some section and wanting to see their effect!)

    Secondly, the "man" syntax highlighting in Vim is a poor hack. It only highlights some salient features like headings. It has no idea where the keywords are that were supposed to be bolded and italicized.

    Vim you can of course not only remember the "last position" (there's
    the default mark, a single quote, for that) but you can also define
    arbitrary marks and, of course, randomly jump between them. Or split
    the screen to see more than one man page context. You can also (e.g.)
    extract portions of the man page. Do whatever an editor supports.

    A reason against using an editor is probably the size consideration.

    I tried it on a 13-year-old Core i5, slow machine, on the txr man page,
    which is 65000 lines. There is a bit of a delay, but otherwise it's
    fine.

    I don't know of a larger man page; it makes a PDF document over 930
    pages with the default troff margins.

    (That man page also uses text that is both bold and italic, which
    provides a test case for the third color.)

    What doesn't seem to work nicely is setting the pager to vim

    PAGER='vim -' man ...

    For that you want:

    MANPAGER='vim +MANPAGER' man

    (The MANPAGER variable is a red herring here; I'm using it in
    this example instead of PAGER because PAGER will have no effect on man
    if MANPAGER is already defined.)

    The :MANPAGER vim command referenced by +MANPAGER brings in some
    packaged support for reading man pages, and does render the overstrikes.

    But:

    1. I was not able to use +MANPAGER as the basis of a solution that
    can remember the last position.

    Here is why, I think: vim +MANPAGER doesn't actually read the man page
    input; it seems to work by analyzing the MAN_PN environment variable and
    itself fetching the man page.

    2. For that reason, it also doesn't work with "man -l <file>",
    which is a dealbreaker for people who work on man pages or view
    man pages inside source trees, not only installed pages.

    3. While the overstrike codes are rendered, the underlying data
    has the interspersed backspaces. This means you cannot search for
    those words. If foo is highlighted, /foo won't find it because it's
    actually f^Hfo^Hoo^Ho.

    So I gave up on the vim +MANPAGER approach.

    --
    TXR Programming Language: http://nongnu.org/txr
    Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
    Mastodon: @Kazinator@mstdn.ca
    NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli the Bearded@21:1/5 to 864-117-4973@kylheku.com on Sun Oct 15 23:41:22 2023
    In comp.unix.shell, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    mnpgr: 8270e33c0a820ccb14c1cc6b97460db0a5748f0a
    txr: bf5a4cbe020b6726e8f9cef4cd935861442541cf

    $ mnpgr perl
    /home/username/bin/mnpgr:72: syntax error

    This is glue code, written in TXR Lisp which turns Vim into a
    page for reading man pages.

    Turns out txr is a lot more work to install than mnpgr.

    - Leverages your Vim skills and environment.

    - Has better rendering of backspace overstrikes than the "less" utility.

    These two features sound compelling, hence giving it a chance.

    Elijah
    ------
    is not about to try to debug lisp

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Eli the Bearded on Mon Oct 16 00:54:50 2023
    On 2023-10-15, Eli the Bearded <*@eli.users.panix.com> wrote:
    In comp.unix.shell, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    mnpgr: 8270e33c0a820ccb14c1cc6b97460db0a5748f0a
    txr: bf5a4cbe020b6726e8f9cef4cd935861442541cf

    $ mnpgr perl
    /home/username/bin/mnpgr:72: syntax error

    If you remove the .tl suffix, you have to use "txr --lisp" on it,
    or else the "txrlisp" alternative executable name (hard link).
    Otherwise it's treated as different language.

    The mnpgr.tl program takes no arguments and is not meant to be invoked
    directly by the user (so you don't achieve any ergonomics by dropping
    the suffix).

    It is invoked by man, and expects to find the MAN_PN environment
    variable set up by man.

    See README.md file.

    $ export MANPAGER=/path/to/mnpgr.tl

    or

    $ txr --compile=/path/to/mnpgr.tl
    $ export MANPAGER=/path/to/mnpgr.tlo

    Then just

    $ man perl

    It only works particular man implementation used on GNU/Linux systems;
    the one maintained by Colin Watson.

    This is glue code, written in TXR Lisp which turns Vim into a
    page for reading man pages.

    Turns out txr is a lot more work to install than mnpgr.

    If you're building from source, it should be just:

    ./configure && make && make install # "make tests" is recommended.

    The dependencies are low and optional. If the configure script deosn't
    detect header files for libz and libffi, you get a build without them.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli the Bearded@21:1/5 to 864-117-4973@kylheku.com on Mon Oct 16 02:21:56 2023
    In comp.unix.shell, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    On 2023-10-15, Eli the Bearded <*@eli.users.panix.com> wrote:
    $ mnpgr perl
    /home/username/bin/mnpgr:72: syntax error
    If you remove the .tl suffix, you have to use "txr --lisp" on it,
    or else the "txrlisp" alternative executable name (hard link).
    Otherwise it's treated as different language.

    Hmmm. Okay.

    I changed the #! line and:

    $ MANPAGER=mnpgr man man
    MAN(1) Manual pager utils
    MAN(1)

    {B{NAME}B}
    man - an interface to the system reference manuals

    {B{SYNOPSIS}B}
    {B{man}B} [{I{man options}I}] [[{I{section}I}] {I{page}I} ...] ...

    $

    No highlighting, just your markup. Vim version 8.1.3741, vimrc:
    $ grep -v '^"' ~/.vimrc
    set mouse=
    set exrc
    set nohlsearch
    set noincsearch
    map * "yyy@y
    map S F r
    map = y p
    set t_Co=
    set sw=2 ai ruler
    set shortmess=filnxtToOs
    set nobackup
    set backspace=indent,eol
    set pastetoggle=<Insert>
    map #F /^sub /<CR>w"pywf{%"ppF}a # end &<Esc>
    $ find $HOME/.vim
    /home/username/.vim
    /home/username/.vim/syntax
    /home/username/.vim/syntax/mnpgr.vim
    $

    Also:

    $ MANPAGER=mnpgr man ./txr.1

    Hangs for ~30 seconds before showing me the 62828L manpage.

    The mnpgr.tl program takes no arguments and is not meant to be invoked directly by the user (so you don't achieve any ergonomics by dropping
    the suffix).

    It is invoked by man, and expects to find the MAN_PN environment
    variable set up by man.

    Also: MAN_PN is a poorly documented feature that exists only in some implementations of 'man'. I don't get it set on NetBSD, for example.

    It only works particular man implementation used on GNU/Linux systems;
    the one maintained by Colin Watson.

    Yes, apparently.

    ./configure && make && make install # "make tests" is recommended.

    You forgot the 25M git clone. Also I set a prefix for the install. It
    took a while to build, I didn't time out, and make install errored out (harmlessly it seems):

    $ make install
    INSTALL txr -> /home/username/usr/txr/bin
    HARDLINK /home/username/usr/txr/bin/txrlisp -> /home/username/usr/txr/bin/txr HARDLINK /home/username/usr/txr/bin/txrvm -> /home/username/usr/txr/bin/txr INSTALL LICENSE -> /home/username/usr/txr/share/txr
    INSTALL METALICENSE -> /home/username/usr/txr/share/txr
    INSTALL txr.1 -> /home/username/usr/txr/share/man/man1
    INSTALL stdlib/*.txr stdlib/*.tl stdlib/*.tlo -> /home/username/usr/txr/share/txr/stdlib
    cp: cannot stat 'stdlib/*.tlo': No such file or directory
    make: failing command:
    cp -f stdlib/*.txr stdlib/*.tl stdlib/*.tlo /home/username/usr/txr/share/txr/stdlib
    make: *** [Makefile:536: install] Error 1
    $

    The dependencies are low and optional. If the configure script deosn't
    detect header files for libz and libffi, you get a build without them.

    I had them installed, didn't even notice the dependency there.

    Elijah
    ------
    contemplating attempting to rewrite the .tl file in perl

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Eli the Bearded on Mon Oct 16 04:16:42 2023
    On 2023-10-16, Eli the Bearded <*@eli.users.panix.com> wrote:
    {B{SYNOPSIS}B}
    {B{man}B} [{I{man options}I}] [[{I{section}I}] {I{page}I} ...] ...

    OK, so that part of it is working fine. We know man is invoking
    mnpgr, which is doing the filtering and running Vim

    If you type ":syntax on" does anything happen, or still no?

    Do you uget syntax highlighting automatically when opening files?

    I tried installing nothing but your configuration as my ~/.vimrc.

    I still get the syntax highlighting.

    (The background is white---my eyes!---and word wrap is on, which
    screws up some lines: :set nowrap is needed.)

    $ MANPAGER=mnpgr man ./txr.1

    Hangs for ~30 seconds before showing me the 62828L manpage.

    Compiling mnpgr.tl makes a big difference.

    Another possible contributing factor to this is that the the compiled
    stdlib didn't get built (see below).

    For me, it comes up in about 7 seconds on a really slow, low-end Core
    i5-3570 machine from 2012.

    On the same machine:

    $ time txr --compile=mnpgr.tl

    real 0m0.192s
    user 0m0.184s
    sys 0m0.009s

    If the compiler isn't compiled, that will be a lot slower.

    Also: MAN_PN is a poorly documented feature that exists only in some implementations of 'man'. I don't get it set on NetBSD, for example.

    Yep. This manpgr thing is not a portable project. At least not in
    its present state.

    (Not to mention that it won't work with NetBSD's vi implementation, ha!)

    You forgot the 25M git clone.

    I think that could be smaller if you don't need the history going back
    to 2009, using "git clone --depth <number-of-commits>". There is a way
    to pull a tarball also from the cgit.

    (That goes for any project out there.)

    Also I set a prefix for the install. It
    took a while to build, I didn't time out, and make install errored out (harmlessly it seems):

    $ make install

    Unfortunately, "make install" does not build everything. You have to run
    "make" to build the default target.

    cp: cannot stat 'stdlib/*.tlo': No such file or directory

    That is not exactly harmless; it means that the compiled stdlib didn't
    get installed (since the .tlo files weren't built). It will fall back on
    the interpreted one. Programs that rely on that library will run
    slower. Possibly a lot slower.

    contemplating attempting to rewrite the .tl file in perl

    Great; if you get stuck on anything, let me know.

    --
    TXR Programming Language: http://nongnu.org/txr
    Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
    Mastodon: @Kazinator@mstdn.ca
    NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Eli the Bearded on Mon Oct 16 06:09:12 2023
    On 2023-10-16, Eli the Bearded <*@eli.users.panix.com> wrote:
    In comp.unix.shell, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    If you type ":syntax on" does anything happen, or still no?

    E484: Can't open file /usr/local/share/vim/syntax/syntax.vim

    That is odd. Is that because it doesn't exist, or the perms
    are wrong?

    /usr/local: you built Vim yourself?

    It's interesting that there is no Vim version in the path, similar to /usr/local/share/vim/vim82/syntax/syntax.vim.

    I see that version-specific directory in distro Vim installations
    like Ubuntu. (Is that a Vim thing, or courtesy of the distro, I
    wonder.)

    The 'configure' tool told me to "make tests and "make install" so that's
    what I did. I thought that the tests target would include all of
    components of the default target. Is that not so?

    It actually says the following, which lends itself to unwanted
    interpretations:

    The next step is to build the program with make.

    If that is successful, please follow the INSTALL guide.

    Usually, most users just need to "make tests" and "make install",
    possibly switching to superuser for "make install" if the prefix
    points to a privileged location like /usr/local/.

    I will revise it.


    --
    TXR Programming Language: http://nongnu.org/txr
    Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
    Mastodon: @Kazinator@mstdn.ca
    NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli the Bearded@21:1/5 to 864-117-4973@kylheku.com on Mon Oct 16 05:47:17 2023
    In comp.unix.shell, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    On 2023-10-16, Eli the Bearded <*@eli.users.panix.com> wrote:
    {B{SYNOPSIS}B}
    {B{man}B} [{I{man options}I}] [[{I{section}I}] {I{page}I} ...] ...

    OK, so that part of it is working fine. We know man is invoking
    mnpgr, which is doing the filtering and running Vim

    If you type ":syntax on" does anything happen, or still no?

    E484: Can't open file /usr/local/share/vim/syntax/syntax.vim

    Weird thing is, some files show syntax, eg, the mnpgr.vim file and
    mnpgr.tl. Using ":set syntax?" I see the "mnpgr" value.

    Do you uget syntax highlighting automatically when opening files?

    Usually I run ":syntax off" but I removed that line from my vimrc for
    these experiments.

    (The background is white---my eyes!---and word wrap is on, which

    All my xterms are white text on black.

    screws up some lines: :set nowrap is needed.)

    I use that setting sparingly.

    For me, it comes up in about 7 seconds on a really slow, low-end Core
    i5-3570 machine from 2012.

    I'm (presently) testing this on my Surface Go 2.

    (Not to mention that it won't work with NetBSD's vi implementation, ha!)

    You won't get nvi with this:

    90 (sh `vim +'@{vim-commands}' '@{rendered-file}' < /dev/tty`))))

    You forgot the 25M git clone.
    I think that could be smaller if you don't need the history going back
    to 2009, using "git clone --depth <number-of-commits>".

    That's true, and maybe I should default to that, but for other things I
    do want history.

    There is a way to pull a tarball also from the cgit.

    Github tarballs do not give last commit hash that I have noticed, which
    makes them useful as point in time snapshots, but harder to report
    issues with. I typically avoid the tarballs for that reason.

    $ make install

    Unfortunately, "make install" does not build everything. You have to run "make" to build the default target.

    The 'configure' tool told me to "make tests and "make install" so that's
    what I did. I thought that the tests target would include all of
    components of the default target. Is that not so?

    I ran "make -k install" after the first install failed, then ran "make
    install" again to capture the output for the post.

    Elijah
    ------
    wrote a much cruder "man in vi" in the 1990s

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Kaz Kylheku on Mon Oct 16 12:34:21 2023
    On 15.10.2023 19:18, Kaz Kylheku wrote:
    On 2023-10-15, Janis Papanagnou <janis_papanagnou+ng@hotmail.com> wrote:

    If 'vim' is your pager you could do

    man ... | vim -

    :set syntax=man will also do some (predefined) highlighting. While in

    The problem is that vim will not remember the position in a buffer that
    had been read from a pipe. This whole thing was prompted by someone
    trying to solve that problem. They wanted to remember the position in
    each individual man page. That resonated with me because there have been times when I wanted to revisit something I recently read in a man page.

    Ah, I see; you meant memorizing marks across sessions.


    (One such situation is when I'm working *on* a man page, making
    adjustments to some section and wanting to see their effect!)

    Isn't that then a _new_ (manpage-) document? - I wouldn't expect that
    a mark from a previous version would find some identical match in the
    new version; should that be by line number or by pattern match? - I'd
    say we can't tell generally.

    But I think it's also about how we work in such cases (and with Vim).

    What one can do is searching the section you are working on, say
    /DESCRIPTION
    And if you leave and restart the Vim-session - also a piped one! -,
    you'd just type
    n
    to get again to the DESCRIPTION in your document.

    Though usually I work with two windows, one for the editing and one
    to see and test the results. Another method is shell escaping (:sh)
    from the editor to run and test a change. (It depends.)


    A reason against using an editor is probably the size consideration.

    I tried it on a 13-year-old Core i5, slow machine, on the txr man page,
    which is 65000 lines. There is a bit of a delay, but otherwise it's
    fine.

    Oh, I meant size of executables (vim: ~2400 kB vs. less: ~150 kB);
    i.e. what the reason is to *re-implement* a very small subset of
    editor functions (search, highlight, ...) in the pager software
    (instead of just using an editor). Executable size matters maybe.

    But (in case of Vim) data sizes matter as well, I observed. - But
    (despite insanely large man pages in some cases) probably not for
    man pages of reasonable size.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Kaz Kylheku@21:1/5 to Kaz Kylheku on Sun Oct 22 19:43:02 2023
    On 2023-10-15, Kaz Kylheku <864-117-4973@kylheku.com> wrote:
    Hi All,

    Prompted by a Unix StackExchange question, I started a mini project
    called mnpgr a few days ago. It's in a very usable state now.

    I just discovered and addressed the following issue.

    If you view the same man page using different widths (number of
    columns) the posititions remembered by Vim don't make sense.

    The solution is to determine what the width is by examining
    the COLUMNS and MANWIDTH variables, falling back on 80.
    Then this column value is interpolated into the name of the
    temporary file.

    Thus Vim remembers the position separately for different
    width renderings of a man page.

    --
    TXR Programming Language: http://nongnu.org/txr
    Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
    Mastodon: @Kazinator@mstdn.ca
    NOTE: If you use Google Groups, I don't see you, unless you're whitelisted.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)