Merge #13448: Add linter: Make sure we explicitly open all text files using UTF-8 encoding in Python

c8176b3cc7556d7bcec39a55ae4d6ba16453baaa Add linter: Make sure we explicitly open all text files using UTF-8 or ASCII encoding in Python (practicalswift)
634bd970013eca90f4b4c1f9044eec8c97ba62c2 Explicitly specify encoding when opening text files in Python code (practicalswift)

Pull request description:

  Add linter: Make sure we explicitly open all text files using UTF-8 encoding in Python.

  As requested by @laanwj in #13440.

Tree-SHA512: 1651c00fe220ceb273324abd6703aee504029b96c7ef0e3029145901762c733c9b9d24927da281394fd4681a5bff774336c04eed01fafea997bb32192c334c06

2 files changed, 21 insertions, 2 deletions

diff --git a/test/lint/check-rpc-mappings.py b/test/lint/check-rpc-mappings.py
index 7e96852c5c..c3cdeef580 100755
--- a/test/lint/check-rpc-mappings.py
+++ b/test/lint/check-rpc-mappings.py
@@ -44,7 +44,7 @@ def process_commands(fname):
     """Find and parse dispatch table in implementation file `fname`."""
     cmds = []
     in_rpcs = False
-    with open(fname, "r") as f:
+    with open(fname, "r", encoding="utf8") as f:
         for line in f:
             line = line.rstrip()
             if not in_rpcs:
@@ -70,7 +70,7 @@ def process_mapping(fname):
     """Find and parse conversion table in implementation file `fname`."""
     cmds = []
     in_rpcs = False
-    with open(fname, "r") as f:
+    with open(fname, "r", encoding="utf8") as f:
         for line in f:
             line = line.rstrip()
             if not in_rpcs:
diff --git a/test/lint/lint-python-utf8-encoding.sh b/test/lint/lint-python-utf8-encoding.sh
new file mode 100755
index 0000000000..ce973e710a
--- /dev/null
+++ b/test/lint/lint-python-utf8-encoding.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# Copyright (c) 2018 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+#
+# Make sure we explicitly open all text files using UTF-8 (or ASCII) encoding to
+# avoid potential issues on the BSDs where the locale is not always set.
+
+EXIT_CODE=0
+OUTPUT=$(git grep " open(" -- "*.py" | grep -vE "encoding=.(ascii|utf8|utf-8)." | grep -vE "open\([^,]*, ['\"][^'\"]*b[^'\"]*['\"]")
+if [[ ${OUTPUT} != "" ]]; then
+    echo "Python's open(...) seems to be used to open text files without explicitly"
+    echo "specifying encoding=\"utf8\":"
+    echo
+    echo "${OUTPUT}"
+    EXIT_CODE=1
+fi
+exit ${EXIT_CODE}