1. Forum
  2. Usenet
  3. COMP.LANG.SMALLTALK.DOLPH

  • Improving performance of stdout for interprocess communication

    From Joe Betz@21:1/5 to All on Wed Feb 15 06:48:16 2023
    I want to communicate with another Windows process using pipes but am getting really poor performance when reading from its stdout pipe. Specifically, calling `outputPipe readStream nextLine` seems to always take at least 100ms to return and <1MB
    payloads take seconds. By comparison, if I switch the interface to using http via IWinHttpRequest, everything is fast, <10ms.

    The only way I've found to improve performance of the pipes interface by increasing the block size in StdioTextFileStream>>nextLine. But even that doesn't get it to the right order of magnitude.

    Anyone know a solution to his, or have a better definition of the problem? I would just switch to Http, but I need bidirectional communication which would require WebSockets. And between implementing WebSockets in Dolphin or fixing ExternalPipe, the
    latter seems more worthwhile.

    I'm using the example found in `ExternalPipe>>example1` as a template and didn't make any significant changes to it. Here is the relevant code:

    start
    outputPipe := ExternalPipe new.
    inputPipe := ExternalPipe new.
    process := ExternalProcess new.
    process
    commandLine: 'ensemble.exe --interface pipes';
    stdoutPipe: outputPipe;
    stdinPipe: inputPipe.
    [process executeSync] fork.
    messageHandler :=
    [[outputPipe readStream atEnd] whileFalse:
    [| messageString |
    messageString := outputPipe readStream nextLine.
    self handleMessage: (STONJSON fromString: messageString)]]
    fork.
    [process isAlive] whileFalse

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zenchess@21:1/5 to All on Wed Feb 15 20:48:42 2023
    I ran into this issue when I was making a chess GUI and communicating with stockfish. I don't remember exactly how I solved it, but I did some searching through the source and noticed I have a new method on
    StdioFileStream

    upToEnd
    ^self next: self size

    I believe I had to make this change so that it wouldn't freeze when trying to get the rest of the input. Let me know if that helps, if not, I'll do a deeper dive into the difference between my chess gui source and the regular dolphin source

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zenchess@21:1/5 to All on Wed Feb 15 21:07:35 2023
    Just in case it helps I'll dump the method I had in which I was using pipes as I noticed you did it a little differently. This method of using them this way may have come from some old externalpipes package source code.

    visualize
    |output readStream numCommands|
    "
    self exampleChessEngine
    "

    "Create external proccess"

    numCommands := 0.
    output := ''.
    chessBoardPresenter view shell: self.
    engineIsRunning ifTrue: [self closeEngine. ^self].
    engineIsRunning := true.
    engineOutputCollection := OrderedCollection new.
    engineOutput := ReadWriteStream on: String new.
    process := ExternalProcess new.

    process commandLine: 'cmd'.
    "Query Pipes"
    outputPipe := process stdoutPipe.
    inputPipe := process stdinPipe.

    "load the chess engine"
    (inputPipe writeStream)
    nextPutAll: 'cd ' , FileLocator default basePath;
    cr;
    flush.
    (inputPipe writeStream)
    nextPutAll: 'stockfish_10_x32.exe';
    cr;
    flush.

    "Execute exeternal proccess in different smalltalk process"
    process executeAsync.

    "Start Display to transcript proccess"
    Transcript show;clear.
    readStream := outputPipe readStream.
    myProcess := [ [ |previousCommands currentcommand|

    previousCommands := numCommands.
    readStream size > 0 ifTrue: [ output := output, readStream upToEnd].
    numCommands := output occurrencesOf: Character nl.
    numCommands > previousCommands ifTrue: [ |substrings|
    substrings := output subStrings: 'info'.
    substrings size > 1 ifTrue: [
    currentcommand := substrings at: (substrings size - 1).
    Transcript nextPutAll: currentcommand;
    cr;
    flush.
    self processVisualizeOutput: currentcommand.].
    ].

    Processor sleep: 100.
    ] repeat ] fork.
    "Wait untíl external process is alive"
    [process isAlive] whileFalse.
    "Ask for input as long as the process is running."
    "(inputPipe writeStream)
    nextPutAll: 'uci';
    cr;
    flush.
    "
    (inputPipe writeStream)
    nextPutAll: 'go infinite';
    cr;
    flush.
    Transcript show: 'go infinite'; cr.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Joe Betz@21:1/5 to All on Sat Feb 25 21:09:18 2023
    Thanks for the snippet Zenchess, but I wasn't able to get things to work any better with your implementation.

    upToEnd
    ^self next: self size

    I need each line to be processed separately, so even if this did speed things up, I don't think it would have the correct behavior.

    Anyways, shortly after writing the original message I came across https://github.com/rko281/Swazoo, a WebSockets implementation for Dolphin, so I'm going to try that next.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Joe Betz@21:1/5 to All on Sat Feb 25 22:03:32 2023
    Anyways, shortly after writing the original message I came across https://github.com/rko281/Swazoo, a WebSockets implementation for Dolphin, so I'm going to try that next.

    Scratch that. Swazoo, or at least version of it, doesn't have a WebSockets implementation.

    I did find WebSocket functions in the WinHTTP API, however, so that looks promising.

    https://learn.microsoft.com/en-us/windows/win32/api/websocket/nf-websocket-websocketsend

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Zenchess@21:1/5 to All on Mon Feb 27 02:50:41 2023
    I actually made a websocket package a while back when trying to make a multiplayer game. I didn't perfectly implement the protocol, although most of the work is done and it worked for my scenario. I tried loading it in dolphin 8 and it said it required
    a prequisite package "Sockets Connection", but I think it will load in Dolphin 7. You could try it and if you have any issues I could fix it up. I'm hosting it here:

    http://zenchess.com/dolphin/WebSocket.pac

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Joe Betz@21:1/5 to Zenchess on Tue Feb 28 04:29:35 2023
    On Monday, February 27, 2023 at 11:50:42 AM UTC+1, Zenchess wrote:
    I actually made a websocket package a while back when trying to make a multiplayer game. I didn't perfectly implement the protocol, although most of the work is done and it worked for my scenario. I tried loading it in dolphin 8 and it said it required
    a prequisite package "Sockets Connection", but I think it will load in Dolphin 7. You could try it and if you have any issues I could fix it up. I'm hosting it here:

    http://zenchess.com/dolphin/WebSocket.pac

    I actually need a client, not a server. :}

    And implementing it with WinHTTP did work out in the end. About a day of implementation and a couple days of debugging.

    Two things I got stuck on:
    - Specifying *dword for an argument that was specified in Win32 docs as DWORD_PTR. Switching to dword fixed it, though I'm still not sure that's technically correct.
    - Figuring out to set up the receive function so it doesn't block literally everything while waiting for a new message. Learned about overlapping calls and that worked perfectly.

    I will probably publish it to my Github (JBetz) after I've cleaned it up, though I can email it to you or anyone else if interested. Currently it's littered with debug logs.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)