1. Forum
  2. Usenet
  3. LINUX.DEBIAN.DEVEL

  • Bug#1108022: ITP: librust-spm-precompiled-dev -- emulate https://github

    From Kohei Sendai@1:229/2 to All on Thu Jun 19 07:00:01 2025
    XPost: linux.debian.bugs.dist
    From: kouhei.sendai@gmail.com

    Package: wnpp
    Severity: wishlist
    Owner: Kohei Sendai <kouhei.sendai@gmail.com>
    X-Debbugs-Cc: debian-devel@lists.debian.org, kouhei.sendai@gmail.com

    * Package name : librust-spm-precompiled-dev
    Version : 0.1.4
    Upstream Contact: Nicolas Patry <patry.nicolas@protomail.com>
    * URL : https://github.com/huggingface/spm_precompiled
    * License : Apache-2.0
    Programming Lang: Rust
    Description : emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer

    This crate aims to emulate https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer. It's main intent is to be used with tokenizers that is a Rust library that aims to provide facilities to tokenize string for use with
    HuggingFace's transformers library

    This crate is highly specialized and not intended for general use.

    The core of the algorithm is to read spm's binary precompiled_charsmap

    - This is highly related to tokenizers which is really important for
    packaging vllm. Now, I'm planning to packaging vllm.

    --- SoupGate-Win32 v1.05
    * Origin: you cannot sedate... all the things you hate (1:229/2)