On Sun, Mar 16, 2025 at 12:12:08PM +0000, Pranav P wrote:

I am still continuing my search on the issue.
It seems that the issue is rising from pylsqpack.
When the value field in the packet header for HTTP3 GET request contains
long strings there are problems while encoding (Only in s390x).
Due to this one of the GET parameters gets jumbled and this results in a bad request.
I am not able to see the same issue on ls-qpack though.
I will update any new findings.

Yes, I was just going through this today (I hadn't noticed your emails
until after I'd spent some time on it) and I found much the same thing.
I reduced it to the following more manageable test case:

# amd64
>>> import pylsqpack
>>> encoder = pylsqpack.Encoder()
>>> decoder = pylsqpack.Decoder(4096, 16)
>>> _, frame = encoder.encode(0, [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])
>>> decoder.feed_header(0, frame)
(b'', [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])

# s390x
>>> import pylsqpack
>>> encoder = pylsqpack.Encoder()
>>> decoder = pylsqpack.Decoder(4096, 16)
>>> _, frame = encoder.encode(0, [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])
>>> decoder.feed_header(0, frame)
(b'', [(b':path', b'd/snq-euyrd?snA=AAABAAABAAAAAAAAAAABAAABAAAAAAAAR2cuZwbn92bnUGAAAEAAQ')])

--
Colin Watson (he/him) [cjwatson@debian.org]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Control: reassign -1 python3-pylsqpack 0.3.18-1
Control: affects -1 src:dnspython

On Mon, Mar 17, 2025 at 05:03:56PM +0000, Colin Watson wrote:

On Sun, Mar 16, 2025 at 12:12:08PM +0000, Pranav P wrote:

I am still continuing my search on the issue.
It seems that the issue is rising from pylsqpack.
When the value field in the packet header for HTTP3 GET request contains >>long strings there are problems while encoding (Only in s390x).
Due to this one of the GET parameters gets jumbled and this results in a bad request.
I am not able to see the same issue on ls-qpack though.
I will update any new findings.

Yes, I was just going through this today (I hadn't noticed your emails
until after I'd spent some time on it) and I found much the same
thing. I reduced it to the following more manageable test case:

# amd64

import pylsqpack
encoder = pylsqpack.Encoder()
decoder = pylsqpack.Decoder(4096, 16)
_, frame = encoder.encode(0, [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])
decoder.feed_header(0, frame)

(b'', [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])

# s390x

import pylsqpack
encoder = pylsqpack.Encoder()
decoder = pylsqpack.Decoder(4096, 16)
_, frame = encoder.encode(0, [(b':path', b'/dns-query?dns=AAABAAABAAAAAAAAAAABAAABAAAAAAAAA2RucwZnb29nbGUAAAEAAQ')])
decoder.feed_header(0, frame)

(b'', [(b':path', b'd/snq-euyrd?snA=AAABAAABAAAAAAAAAAABAAABAAAAAAAAR2cuZwbn92bnUGAAAEAAQ')])

How does the attached patch look? The basic problem is that the Huffman encoder was assuming little-endian when reading from the source buffer.

I realize this is against vendored code, but upstream ls-qpack seems to
have pretty much the same code in this area, so if this looks good I'll
tidy it up and submit it there.

Thanks,

--
Colin Watson (he/him) [cjwatson@debian.org]

diff --git a/vendor/ls-qpack/lsqpack.c b/vendor/ls-qpack/lsqpack.c
index de125e0..6b0c65f 100644
--- a/vendor/ls-qpack/lsqpack.c
+++ b/vendor/ls-qpack/lsqpack.c
@@ -5188,7 +5188,7 @@ qenc_huffman_enc (const unsigned char *src, const unsigned char *const src_end,

while (src + sizeof(bits) * 8 / SHORTEST_CODE + sizeof(idx) < src_end)
{
- memcpy(&idx, src, 2);
+ idx = (uint16_t) src[0] | (((uint16_t) src[1]) << 8);
henc = &hencs[idx];
src += 2;
while (bits_used + henc->lens < sizeof(bits) * 8)
@@ -5196,7 +5196,7 @@ qenc_huffman_enc (const unsigned char *src, const unsigned char *const src_end,
bits <<= henc->lens;
bits |= henc->code;
bits_used += henc->lens;
- memcpy(&idx, src, 2);
+ idx = (uint16_t) src[0] | (((uint16_t) src[1]) << 8);
henc = &hencs[idx];
src += 2;
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Mar 19, 2025 at 08:57:34AM +0000, Pranav P wrote:

In a file called huff-tables.h in ls-qpack, some calculations where happening based on the following code:
#if __BYTE_ORDER == __LITTLE_ENDIAN
#define I(i,j) ((j<<8)|i)
#else
#define I(i,j) ((i<<8)|j)
#endif
Surprisingly when compiled with the c99 standards the #else part was getting executed while for gnu99 #if part was getting triggered.
Writing some #ifdef statements it was found that __BYTE_ORDER and __LITTLE_ENDIAN were not defined when compiled with c99 and hence the macro evaluated to
#if NULL == NULL
which caused the little endian logic of the code to get executed.
In little endian systems, this would not be an issue as either ways only little endian part of the logic will get executed.
__BYTE_ORDER and __LITTLE_ENDIAN are defined in the header file endian.h in glibc.
This header file is automatically included when compiling with gnu99 and not when compiling with c99(This was tested by preprocessing with both c99 and gnu99 with the -E flag passed to gcc).
In the setup.py file of pylsqpack it is mentioned to adhere to c99 standards. >So, adding <endian.h> header file to lsqpack.c file seems to fix the issue.

Ah yes, of course. In the circumstances I think that's better than my
patch, which I think would have broken (due to overcorrection) if
<endian.h> happened to be included by something else.

Please see if the attached patch is fine. I also am not aware of how to submit a patch. So, if the attached patch is all good, can you please guide me on how to submit the patch.

I think the best way to go is to send this as a pull request to https://github.com/litespeedtech/ls-qpack, and then figure out how to
get pylsqpack upgraded upstream (though I can apply the patch to the
Debian package, at least).

If you can make a pull request yourself, then that would be best, but I
can send it upstream on your behalf if you aren't set up for that.
Could you at least write a short commit message for your patch, and
reference https://bugs.debian.org/1099935 somewhere in it for tracking purposes?

Thanks,

--
Colin Watson (he/him) [cjwatson@debian.org]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Mar 20, 2025 at 07:17:10AM +0000, Pranav P wrote:

Thanks a lot. I raised an issue with a suggested fix on pylsqpack's github repo

https://github.com/aiortc/pylsqpack/issues/38, for reference.

and has
asked them how they want to integrate the changes. I will also raise a PR to ls-qpack's
repo. Meanwhile, I am attaching the patch with a commit message generated via quilt.
I am new to the community so please feel free to share if something is to be modified.

Thanks
Pranav

[https://res.public.onecdn.static.microsoft/assets/mail/file-icon/png/generic_16x16.png]0001-Fix-enddianness-bug-in-pylsqpack-Encoder.encode.patch<https://ibm-my.sharepoint.com/:u:/p/pranav_p7/EcY4vFeM7QpCnoEzfnuMXr0BYOU-u3vndDV3ESeVodixIA>

This attachment wasn't an email attachment - it was some kind of
Sharepoint link that I can't access. Could you please resend your
attachment as an actual email attachment, and then I can get it
into the Debian pylsqpack package?

Alternatively, pointing me at your ls-qpack PR once it exists would be
fine too - I don't see anything there yet.

Thanks,

--
Colin Watson (he/him) [cjwatson@debian.org]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)