Oh. In that case I'd say it's more acceptable since there's no purity argument against it. fork(2) is lightweight on probably every OS where your program sees meaningful usage, you don't do much in the descendants so memory impact is minimal, you don't triple-fork in a loop. I'd say it's fine. Lots of crappy scripts fork way more than that. (Hell, lots of people fork ~30 times to generate their shell prompt. Not kidding: https://archive.zhimingwang.org/blog/2015-05-03-why-oh-my-zsh-is-completely-broken.html#incredibly-poor-code-quality Holy crap, five years after I wrote that blog post it hasn't improved a bit.)
I don’t think you need to touch spawn. You just need to spawn in a child process that literally serves one purpose: open the write end of the pipe before spawn, and close it after spawn returns. This ensures the read end of the pipe receives an EOF even if the spawned process never open the pipe for writing. (Of course, you need to live with the double fork.)
I already know the basic flow of your implementation, but not the nitty gritty details. The problem is, you don’t start reading until the command finishes, but the command can’t finish because it’s blocking on write to the pipe. Whereas I read form the pipe simultaneously as the command runs. (The double fork of course isn’t pretty, but it works. Maybe there’s a way to cut to a single fork.)
Yeah, I checked, no problem at all?
I created 10k files in `/tmp/lotsoffiles`:
```sh
for i in {1..10000}; do touch /tmp/lotsoffiles/$i; done
```
Then I ran my program with this script:
```sh
#!/bin/sh
find /tmp/lotsoffiles -print0 >$FIFO_PATH
```
Works just fine:
```console
$ ./pipe ./test
...
read 4096 bytes from /tmp/fifo_demo
read 4096 bytes from /tmp/fifo_demo
read 1823 bytes from /tmp/fifo_demo
total bytes read from /tmp/fifo_demo: 218911
```
I mean, if
```sh
find /tmp/lotsoffiles -print0 | xargs -0 echo
```
works, there's no reason this wouldn't work...
> You example script doesn't get input from the user if I'm not missing something, mimelist needs to get input from the user.
Reading input is completely orthogonal.
```sh
#!/bin/sh
echo -n 'number of blocks: '
read count
dd bs=1024 count=$count if=/dev/zero of=$FIFO_PATH
```
```console
$ ./pipe ./test
number of blocks: 256
read 1024 bytes from /tmp/fifo_demo
...
read 1024 bytes from /tmp/fifo_demo
256+0 records in
256+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00116431 s, 225 MB/s
total bytes read from /tmp/fifo_demo: 262144
```
256 is read from stdin. No problem.
> We are trying to do exactly what you are suggesting. Spawn the script, get user input, run the rest of the script in the background, read the pipe from nnn.
The difference is my approach doesn't require any work on the script's end (running in background). You can use any blocking script, as I've demonstrated.
> if you have 10K files in a directory and you do a find . -print0 it writes much more than 64K.
Will check, it's rather unexpected that it doesn't write in chunks.
> Plugins can be independent or write back. Currently we wait for all plugins. (Not just for processing to complete, nnn runs in ncurses mode, plugins may write to the terminal too leading to a broken terminal).
I still can't tell why the standard flow is not feasible.
There appears to be a lot of code to read, so I didn't bother implementing the flow into nnn, but here's a simple PoC:
```c
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
static const char FIFO_PATH[] = "/tmp/fifo_demo";
int main(int argc, const char* argv[])
{
char buf[4096];
int bytes_read, total_bytes_read = 0;
int ret = 0;
if (argc == 1) {
fprintf(stderr, "Usage: %s command [arg ...]\n", argv[0]);
exit(1);
}
if (mkfifo(FIFO_PATH, 0600) == -1) {
perror("fifo");
exit(1);
}
if (fork() == 0) {
int wfd = open(FIFO_PATH, O_WRONLY | O_NONBLOCK);
char fifo_path_env[1024];
char* child_argv[argc];
int pid, status;
snprintf(fifo_path_env, sizeof(fifo_path_env), "FIFO_PATH=%s", FIFO_PATH);
putenv(fifo_path_env);
for (int i = 1; i < argc; i++) {
child_argv[i - 1] = (char*)argv[i];
}
child_argv[argc - 1] = NULL;
if ((pid = fork()) == 0) {
execvp(child_argv[0], child_argv);
}
if (waitpid(pid, &status, 0) == -1) {
perror("waitpid");
}
close(wfd); // EOF to pipe
return 0;
}
int rfd = open(FIFO_PATH, O_RDONLY);
while ((bytes_read = read(rfd, buf, sizeof(buf))) > 0) {
total_bytes_read += bytes_read;
printf("read %d bytes from %s\n", bytes_read, FIFO_PATH);
}
printf("total bytes read from %s: %d\n", FIFO_PATH, total_bytes_read);
if (bytes_read == -1) {
perror("read");
ret = 1;
goto cleanup;
}
close(rfd);
cleanup:
if (unlink(FIFO_PATH) == -1) {
perror("unlink");
exit(1);
}
return ret;
}
```
You can execute anything, whether it writes to the pipe or not, and however much it writes.
For instance, something that doesn't touch the pipe:
```console
$ ./pipe ls /tmp
fifo_demo snap.rocketchat-server systemd-private-fcd842cdeee94fbba18b123f1dd8849a-systemd-logind.service-cfsvdi
fork ssh-mS5TQUcxfw systemd-private-fcd842cdeee94fbba18b123f1dd8849a-systemd-resolved.service-88lhhf
snap.lxd stress systemd-private-fcd842cdeee94fbba18b123f1dd8849a-systemd-timesyncd.service-9RanVe
total bytes read from /tmp/fifo_demo: 0
```
A script that does touch the pipe:
```sh
#!/bin/sh
dd bs=1024 count=256 if=/dev/zero of=$FIFO_PATH
```
```
$ ./pipe ./test
read 1024 bytes from /tmp/fifo_demo
...
read 1024 bytes from /tmp/fifo_demo
256+0 records in
256+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.001237 s, 212 MB/s
total bytes read from /tmp/fifo_demo: 262144
```
Maybe I'm missing something.
I don't understand what makes the classical pipe workflow unsuitable?
```
process
|
pipe(pipefd)
|
fork
| \
| \
| \
| write to pipefd[1]
|
read from pipefd[0] in a loop,
until EOF
```
doesn't really involve stdin in the subprocess.
Hmm, ten results won’t necessarily fit onto two pages either, and even if they do, if you need to open anything hidden from view you need to scroll back regardless, so I don’t think “avoid scrolling back at all costs” is what users want, nor is it what they’ll get with an r command.
Meanwhile, at least one user expressed wish for a permanent reversal.
Anyway, I think we’re disagreeing on what’s a less bad compromise on a problem neither of us are particularly concerned about ourselves? I guess my preferred solution then is not choose a compromise (unless you want to use the command yourself.)
Oh, I guess my suggestion was confusing. What I was saying: pipe should be read while the plugin is being executed, not after the plugin finishes running. I didn’t check where you’re reading into, though.
Sure, order doesn’t actually matter on pages other than the first, which means reverse is just as good as default. But having to run a separate command to reverse the order of the first page every time is still about as much trouble as scrolling back. “I'd expect people who like this behavior to permanently have it enabled.” still stands.
I don't think you can set pipe size on BSD variants.
- macOS: https://opensource.apple.com/source/xnu/xnu-6153.61.1/bsd/kern/sys_pipe.c.auto.html
Look for `choose_pipespace` calls.
- FreeBSD: https://github.com/freebsd/freebsd/blob/cdc7401798469a746434f71eb6359c915bcf43d8/sys/kern/sys_pipe.c
Look for `pipespace` calls.
- OpenBSD: https://github.com/openbsd/src/blob/7752f9fda66205f947a2171e176537a136444c36/sys/kern/sys_pipe.c
Look for `pipe_buffer_realloc` calls.
Basically, always limited to 64k.
Probably want to use the pipe correctly instead of reading in one fell swoop.
Kinda confused here. Reversing the order is for people with short terminals and want to see the more relevant results without scrolling. With that in mind, why would they use a separate command to only temporarily reverse the order? Seems about as much trouble as scrolling back. I'd expect people who like this behavior to permanently have it enabled.