GHC 2018-12-07

5 comments.

, https://git.io/fpSsr in SNH48Live/KVM48
Oh what a coincidence. In fact, only after I posted the comment above did I notice the "I was trying to invoke KVM48 from Node.js" part... And I mentioned Node community as the worst offender. So yeah, Node people handle standard streams wrong all the time; if that's what you've mostly been exposed to, you might need to spend more time with more native command line tools. (Please don't take this personally.)

, https://git.io/fpSso in SNH48Live/KVM48
Hmm, you might have the wrong understanding of stdout and stderr.

Standard output is for *output*, which is generally somewhat structured to be parseable by other programs (so you could easily pipe and process — that's the third principle of the Unix philosophy). In the case of KVM48, output contains lines of tab delimited URLs and (relative) download paths, and optionally an asterisk as the new VOD indicator.

On the other hand, standard error, despite the name, is not only meant for errors. All messages and logs should go to stderr, regardless of level. Informational logs are generally of level INFO (sometimes DEBUG), and they are treated the same as WARNING, ERROR, and CRITICAL. If messages and logs are written to stdout, stdout becomes useless and you might as well just use one stream.

To show what I said is the norm, I'll observe two things (appealing to authority here): (1) Python's `logging.StreamHandler` logs to stderr by default (regardless of level, of course); (2) On most *nix operating systems, by default stdout is buffered, while stderr is unbuffered; of course informational messages go to the unbuffered stream since you want to see progress and stuff in real time.

Now, when I first wrote KVM48, instead of setting up logging proper, I cheaped out and littered `sys.stderr.write` everywhere. (See [caterpillar](https://github.com/zmwangx/caterpillar/), the dependency for m3u8 downloads, where I didn't cheap out and did the right thing.) That's unfortunate, but I'm always careful with writing to the correct stream.

I know many people these days get the streams wrong; I find this particularly prevalent among command line tools written in Node by web developers, where they dump all the crap to stdout, possibly because `console.error` sounds prohibitive. But I've lived and breathed Unix shells for long enough to know the right way, or at least form a strong opinion.

, https://git.io/fpSkQ in SNH48Live/KVM48
Oh okay, good to hear.

, https://git.io/fpSk7 in SNH48Live/KVM48
Still not sure what's causing the encoding issue with your `/usr/bin/python3`, but the output from `/usr/local/bin/python3` looks good. Could you please reinstall KVM48 with `/usr/local/bin/pip3` and see if the problem goes away?

, https://git.io/fpSk5 in SNH48Live/KVM48
Your locale probably isn't what you think it is. Also, the Python in use is clearly Python 3.5. What's the output of the following command?

```sh
python3 -c 'import locale, os, sys; print("executable:", sys.executable); print("version:", sys.version_info); print("default locale:", locale.getdefaultlocale()); print("current locale:", locale.getlocale()); print("locale preferred encoding:", locale.getpreferredencoding()); print("filesystem encoding:", sys.getfilesystemencoding()); print("env:"); print("\n".join("%s: %s" % (k, v) for k, v in sorted(os.environ.items())));'
```

The output should be look somewhat like this:

```
executable: /Users/zmwang/.pyenv/versions/3.7.1/bin/python3
version: sys.version_info(major=3, minor=7, micro=1, releaselevel='final', serial=0)
default locale: ('en_US', 'UTF-8')
current locale: ('en_US', 'UTF-8')
locale preferred encoding: UTF-8
filesystem encoding: utf-8
env:
...
```

If you have sensitive information in environment variables, please scrub them.