XPost: news.software.readers

Hi all,

The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

However, RFC 3977 also states in Section 8.4.2 that "for compatibility
with existing implementations, the last two lines MAY instead be:

Bytes:
Lines:

even though they refer to metadata, not headers."



Are you aware of news readers that send LIST OVERVIEW.FMT and would
break if they encountered ":bytes" or ":lines"? I'm wondering whether
it is safe nowadays to advertise the expected strings.


Also, are there useful other header fields that news readers are already searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news
servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

--
Julien ÉLIE

« Tant qu'il y a des marmites, il y a de l'espoir ! » (Astérix)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien wrote:

The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

However, RFC 3977 also states in Section 8.4.2 that "for compatibility
with existing implementations, the last two lines MAY instead be:

Bytes:
Lines:

even though they refer to metadata, not headers."

Are you aware of news readers that send LIST OVERVIEW.FMT and would
break if they encountered ":bytes" or ":lines"? I'm wondering whether
it is safe nowadays to advertise the expected strings.

Also, are there useful other header fields that news readers are already >searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news >servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would make my life easier, certainly. I also have kill file
entries based on Path.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Hi Adam,

Also, are there useful other header fields that news readers are already
searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news
servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would make my life easier, certainly. I also have kill file entries based on Path.

I see that you're using trn. The question would then be: were the
Newsgroups and Path header fields be present in (X)OVER responses, would
trn use them directly? (without sending (X)HDR or HEAD commands to
retrieve these header fields)
It is the whole point of having them in the overview database. Yet, I
don't know whether current versions of news clients would directly grab
them from there. (It may be an interesting feature to implement in news clients if they aren't already doing that.)

--
Julien ÉLIE

« Just don't create a file called -rf. » (Larry Wall)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Adam,

Also, are there useful other header fields that news readers are already >>>searching for in their (X)OVER requests and then would benefit from them >>>if present in the overview? The idea would then be to suggest to news >>>admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news >>>servers already provide the Xref header field, but one cannot take for >>>sure that an article is crossposted with that mere field.)

Newsgroups would make my life easier, certainly. I also have kill file >>entries based on Path.

I see that you're using trn. The question would then be: were the
Newsgroups and Path header fields be present in (X)OVER responses, would
trn use them directly? (without sending (X)HDR or HEAD commands to
retrieve these header fields)

Perhaps Wayne Davidson is still reading news.software.readers on
occassion. Eli the Bearded had contributed some trn patches a few years
ago; maybe he could glance at the code.

It is the whole point of having them in the overview database. Yet, I
don't know whether current versions of news clients would directly grab
them from there. (It may be an interesting feature to implement in news >clients if they aren't already doing that.)

Well, yes, since which headers are being downloaded is configurable at
the server level, it would be ideal if the newsreader could determine
which additional headers beyond the standard headers had been downloaded.

You're absolutely right. That should be a newsreader feature.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:

It is the whole point of having them in the overview database. Yet, I
don't know whether current versions of news clients would directly grab
them from there. (It may be an interesting feature to implement in news clients if they aren't already doing that.)

JFTR, tin should (for Path)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien ÉLIE wrote:

[...]
Also, are there useful other header fields that news readers are already searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would be nice to have for scoring.

Currently the OVER command must be disabled for this to work in flnews.
Quoted from man page:
|
| o nntp_no_over (Integer) Disable usage of overview (OVER command) for
| NNTP protocol
|
| Setting this to a nonzero value disables usage of overview and makes
| the whole newsgroup list usable for scoring rules.
| [...]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien ÉLIE wrote:

[...]
The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

RFC 3977 says in Section 8.4.2:
|
| This command MAY generate different results if it is used more than
| once in a session.

A newsreader needs to know when to refresh the information in this case.
Or is it intended to be repeated before every OVER command?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Michael Bäuerle <michael.baeuerle@stz-e.de> writes:

Julien ÉLIE wrote:

[...]
The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

RFC 3977 says in Section 8.4.2:

| This command MAY generate different results if it is used more than
| once in a session.

A newsreader needs to know when to refresh the information in this case.
Or is it intended to be repeated before every OVER command?

That is just a cover-your-ass provision for the news server that allows
the admin to reconfigure overview storage dynamically without having to
take special precautions to prevent the list from changing. I wouldn't
expect a newsreader to refresh its understanding of the valid fields
except when it reconnects.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Hi all,

RFC 3977 says in Section 8.4.2:
|
| This command MAY generate different results if it is used more than
| once in a session.

Speaking of "session", I'm intrigued by:

12.6. Caching of Capability Lists

In most situations, the capabilities list in a given server state
will not change from session to session; for example, a given
extension will be installed permanently on a server. Some clients
may therefore wish to remember which extensions a server supports to
avoid the delay of an additional command and response, particularly
if they open multiple connections in the same session.


There is a notion of "multiple connections in the same session". Yet,
it seems that it is the only place in RFC 3977 where multiple
connections are mentioned. I would have thought a connection and a
session were synonyms but I'm maybe wrong.
Does a session have the meaning of a period of time during which a
client has at least 1 connection opened? (The term is not really
defined, and if it has that meaning, a few uses of it are ambiguous
compared to a connection.)

--
Julien ÉLIE

« Petite annonce : Artificier cherche femme canon. »

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Hi Michael,

Also, are there useful other header fields that news readers are already
searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news
servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Newsgroups would be nice to have for scoring.

Sure.

Currently the OVER command must be disabled for this to work in flnews. Quoted from man page:
|
| o nntp_no_over (Integer) Disable usage of overview (OVER command) for
| NNTP protocol
|
| Setting this to a nonzero value disables usage of overview and makes
| the whole newsgroup list usable for scoring rules.
| [...]

Why not automatically disable OVER if there exists a scoring rule using
the Newsgroups header field? (and keep using OVER otherwise)

In case you wish to test a possible use of the Newsgroups field if
present in overview data, I've added it in my news server (news.trigofacile.com):

LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
.

GROUP news.software.nntp
211 10940 1 15264 news.software.nntp

OVER 15262
224 Overview information for 15262 follows
15262 Re: Advertising metadata and other fields in overview Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@stz-e.de> Mon, 1 Aug 2022 12:55:15 +0200 (CEST) <AABi57ETSaEAAAJK.A3.flnews@WStation5.stz-e.de> <tc6fmu$2q025$1@news.trigofacile.com> 1585 21 Xref: news.trigofacile.com news.software.nntp:15262 Newsgroups: news.software.nntp,news.software.readers Path: news.trigofacile.com!weretis.net!feeder8.news.weretis.net!lilly.ping.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
.



I've also added Path, as Urs has a special handling of it in tin.
And switched to the "preferred" naming of ":bytes" and ":lines" (feel
free to test that everything is still OK with these names).

--
Julien ÉLIE

« Just don't create a file called -rf. » (Larry Wall)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

In news.software.readers, Adam H. Kerman <ahk@chinet.com> wrote:

I see that you're using trn. The question would then be: were the
Newsgroups and Path header fields be present in (X)OVER responses, would
trn use them directly? (without sending (X)HDR or HEAD commands to
retrieve these header fields)

Perhaps Wayne Davidson is still reading news.software.readers on
occassion. Eli the Bearded had contributed some trn patches a few years
ago; maybe he could glance at the code.

As far as I know, Wayne Davidson no longer has an interest in trn or
netnewd. I do have an interest, and I keep trn source handy.

I'm not quite sure if the overview code in rt-ov.h properly deals with
extra content in overview. My experience suggests yes, but it could
have been masked by fast XHDR action. To use overview files for
searching (including KILL commands), you need to use the Hheadername
flag, like so:

/trn/Hsubject:+

The article search code in artsrch.c for ARTSCOPE_ONEHDR calls
fetchlines() in head.c to get the line, that uses fetchcache() from
cache.c to check cached headers (eg found in overview) and falls back to
XHDR if needed.

Scoring calls fetchcache() as well, but I didn't review that code.

Well, yes, since which headers are being downloaded is configurable at
the server level, it would be ideal if the newsreader could determine
which additional headers beyond the standard headers had been
downloaded.

Well trn (in rt-ov.c), for one, is clearly checking the overview.fmt
file for what overview has.

Elijah
------
the provided support/Score.pl script does not check overview.fmt

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Eli the Bearded <*@eli.users.panix.com> writes:

I'm not quite sure if the overview code in rt-ov.h properly deals with
extra content in overview. My experience suggests yes, but it could have
been masked by fast XHDR action.

INN will automatically use overview to satisfy XHDR, XPAT, and PAT
commands if possible, FYI, although with those commands it will fall back
to opening each article individually if the header requested is not in overview, which obviously can be quite slow.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien ÉLIE wrote:

Michael Bäuerle wrote:

[...]
Currently the OVER command must be disabled for this to work in flnews. Quoted from man page:
|
| o nntp_no_over (Integer) Disable usage of overview (OVER command) for
| NNTP protocol
|
| Setting this to a nonzero value disables usage of overview and makes
| the whole newsgroup list usable for scoring rules.
| [...]

Why not automatically disable OVER if there exists a scoring rule using
the Newsgroups header field? (and keep using OVER otherwise)

Currently scoring rules are silently skipped, if they cannot be applied.
This can happen in other cases too, e.g. if a rule contains text that
cannot be converted to the charset of the locale (flnews uses Unicode internally, but explicitly supports operating systems without Unicode
locales).

But yes, this is likely not what a user would expect.

In case you wish to test a possible use of the Newsgroups field if
present in overview data, I've added it in my news server (news.trigofacile.com):

LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
.

GROUP news.software.nntp
211 10940 1 15264 news.software.nntp

OVER 15262
224 Overview information for 15262 follows
15262 Re: Advertising metadata and other fields in overview Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@stz-e.de> Mon, 1 Aug 2022 12:55:15 +0200 (CEST) <AABi57ETSaEAAAJK.A3.flnews@WStation5.stz-e.de> <tc6fmu$2q025$1@news.trigofacile.com> 1585 21 Xref: news.trigofacile.com
news.software.nntp:15262 Newsgroups: news.software.nntp,news.software.readers Path: news.trigofacile.com!weretis.net!feeder8.news.weretis.net!lilly.ping.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
.

Thank you. I will try to implement support for LIST OVERVIEW.FMT and
Newsgroups field from overview.

This should be sufficient to feed the corresponding scoring rules
without the automatic OVER disabling logic discussed above.

I've also added Path, as Urs has a special handling of it in tin.
And switched to the "preferred" naming of ":bytes" and ":lines" (feel
free to test that everything is still OK with these names).

My plan is to simply skip the first 7 fields of the response.
Different order or meaning is not allowed by RFC 3977. And this
information was already used before (implicitly).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Urs Janßen wrote:

Michael Bäuerle <michael.baeuerle@stz-e.de> wrote:

[...]
My plan is to simply skip the first 7 fields of the response.
Different order or meaning is not allowed by RFC 3977. And this

newsoverview(5) from Geoff Collyer (some time in 1992) did specify an order (but did't mention LIST OVERVIEW.FMT as that was invented AFAIK ~1 year later).

RFC 2980 (from 10.2000) didn't specify an order.

flnews has never supported RFC 2980 (XOVER command).

RFC 3977 (from 10.2006) does specify an order.

IIRC I've seen non RFC 3977 compliant setups in the wild (reversed order, missing mandatory fields, duplicate fields, ...).

Hopefully such setups have not advertised OVER via CAPABILITIES
command.

Currently flnews uses overview only with OVER command, and only if the
server advertises OVER capability. If the overview is broken, it can be disabled with the "nntp_no_over" option.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Michael Bäuerle <michael.baeuerle@stz-e.de> wrote:

I've also added Path, as Urs has a special handling of it in tin.

JFTR: works like expected in tin; no extra HDR, XHDR or XPAT cmds. issued
when Path data is available in NOV and used for scoring.

My plan is to simply skip the first 7 fields of the response.
Different order or meaning is not allowed by RFC 3977. And this

newsoverview(5) from Geoff Collyer (some time in 1992) did specify an order (but did't mention LIST OVERVIEW.FMT as that was invented AFAIK ~1 year
later).

RFC 2980 (from 10.2000) didn't specify an order.

RFC 3977 (from 10.2006) does specify an order.

IIRC I've seen non RFC 3977 compliant setups in the wild (reversed order, missing mandatory fields, duplicate fields, ...).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Julien ÉLIE wrote:

[...]
In case you wish to test a possible use of the Newsgroups field if
present in overview data, I've added it in my news server (news.trigofacile.com):

LIST OVERVIEW.FMT
215 Order of fields in overview database
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
.
[...]

I have implemented support for LIST OVERVIEW.FMT in the snapshot flnews 1.2.0pre1. According to the comment from Russ, the command is executed
only once per session.

Protocol log: ------------------------------------------------------------------------
[Using NNTP protocol driver]

Connect to news.trigofacile.com:nntps]

[Established encrypted connection using TLSv1.3 protocol with cipher suite TLS_AES_256_GCM_SHA384]
[<= Expect X509 certificate]
Certificate:
[...]
[Server certificate verification successful]
[<=] 200 news.trigofacile.com InterNetNews NNRP server INN 2.8.0 (20220801 prerelease) ready (no posting)

] CAPABILITIES

[<=] 101 Capability list:
[<= Expect multiline data block]
VERSION 2
IMPLEMENTATION INN 2.8.0 (20220801 prerelease)
AUTHINFO USER SASL
COMPRESS DEFLATE
HDR
LIST ACTIVE ACTIVE.TIMES COUNTS DISTRIB.PATS DISTRIBUTIONS HEADERS MODERATORS MOTD NEWSGROUPS OVERVIEW.FMT SUBSCRIPTIONS
OVER
READER
SASL GS2-IAKERB GS2-KRB5 SCRAM-SHA-1 SCRAM-SHA-256 GSS-SPNEGO GSSAPI DIGEST-MD5 OTP NTLM CRAM-MD5 LOGIN PLAIN
XPAT
[Switch to NNTP V2 (RFC 3977) mode]

[Client authentication]
[Compression]

] LIST OVERVIEW.FMT

[<=] 215 Order of fields in overview database
[<= Expect multiline data block]
Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines
Xref:full
Newsgroups:full
Path:full
[Newsgroups header field available from overview]
[...]
------------------------------------------------------------------------

Urs noted that some server implementations are broken.
The mandatory fields are now checked for presence and order.

The Newsgroups header field, if present, is copied from the overview.
Scoring rules of type "group" are applied to all groups in this case.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Hi Michael,

I have implemented support for LIST OVERVIEW.FMT in the snapshot flnews 1.2.0pre1.

Good news!

[Compression]

I hope more news clients and servers get to implement the COMPRESS
command...

[Newsgroups header field available from overview]

Urs noted that some server implementations are broken.

The mandatory fields are now checked for presence and order.

The Newsgroups header field, if present, is copied from the overview.
Scoring rules of type "group" are applied to all groups in this case.

Sounds great! Thanks for it.

--
Julien ÉLIE

« Il y a un proverbe serbe qui dit ceci : « Notre passé est sinistre,
notre présent est invivable, heureusement que nous n'avons pas
d'avenir ! » (Philippe Geluck)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

XPost: news.software.readers

Den 2022-07-31 skrev Julien ÉLIE <iulius@nom-de-mon-site.com.invalid>:

Hi all,

Hello Julien,

The LIST OVERVIEW.FMT command MUST return these 7 following
case-insensitive strings:

Subject:
From:
Date:
Message-ID:
References:
:bytes
:lines

However, RFC 3977 also states in Section 8.4.2 that "for compatibility
with existing implementations, the last two lines MAY instead be:

Bytes:
Lines:

even though they refer to metadata, not headers."

Are you aware of news readers that send LIST OVERVIEW.FMT and would
break if they encountered ":bytes" or ":lines"? I'm wondering whether
it is safe nowadays to advertise the expected strings.

As far as I can tell by looking at the source of slrn, which I use, it
expects Bytes and Lines to be present for overview support to be
enabled, but it would not break if :bytes and :lines are present in
addition to the two aforementioned fields.

Someone, please correct me if I'm wrong.

Also, are there useful other header fields that news readers are already searching for in their (X)OVER requests and then would benefit from them
if present in the overview? The idea would then be to suggest to news
admins adding such fields in their overview data.
Like Newsgroups to know whether the article is crossposted? (Some news servers already provide the Xref header field, but one cannot take for
sure that an article is crossposted with that mere field.)

Personally, I would find it great to have easier filtering on Path,
although I don't think slrn supports it in the overview at the moment.
For some groups, Google's spam is horrendous and it would be nice to
be able to filter it.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)