diff options
Diffstat (limited to 'bip-0380.mediawiki')
| -rw-r--r-- | bip-0380.mediawiki | 335 |
1 files changed, 335 insertions, 0 deletions
diff --git a/bip-0380.mediawiki b/bip-0380.mediawiki new file mode 100644 index 0000000..27b7908 --- /dev/null +++ b/bip-0380.mediawiki @@ -0,0 +1,335 @@ +<pre> + BIP: 380 + Layer: Applications + Title: Output Script Descriptors General Operation + Author: Pieter Wuille <pieter@wuille.net> + Ava Chow <me@achow101.com> + Comments-Summary: No comments yet. + Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0380 + Status: Draft + Type: Informational + Created: 2021-06-27 + License: BSD-2-Clause +</pre> + +==Abstract== + +Output Script Descriptors are a simple language which can be used to describe collections of output scripts. +There can be many different descriptor fragments and functions. +This document describes the general syntax for descriptors, descriptor checksums, and common expressions. + +==Copyright== + +This BIP is licensed under the BSD 2-clause license. + +==Motivation== + +Bitcoin wallets traditionally have stored a set of keys which are later serialized and mutated to produce the output scripts that the wallet watches and the addresses it provides to users. +Typically backups have consisted of solely the private keys, nowadays primarily in the form of BIP 39 mnemonics. +However this backup solution is insufficient, especially since the introduction of Segregated Witness which added new output types. +Given just the private keys, it is not possible for restored wallets to know which kinds of output scripts and addresses to produce. +This has lead to incompatibilities between wallets when restoring a backup or exporting data for a watch only wallet. + +Further complicating matters are BIP 32 derivation paths. +Although BIPs 44, 49, and 84 have specified standard BIP 32 derivation paths for different output scripts and addresses, not all wallets support them nor use those derivation paths. +The lack of derivation path information in these backups and exports leads to further incompatibilities between wallets. + +Current solutions to these issues have not been generic and can be viewed as being layer violations. +Solutions such as introducing different version bytes for extended key serialization both are a layer violation (key derivation should be separate from script type meaning) and specific only to a particular derivation path and script type. + +Output Script Descriptors introduces a generic solution to these issues. +Script types are specified explicitly through the use of Script Expressions. +Key derivation paths are specified explicitly in Key Expressions. +These allow for creating wallet backups and exports which specify the exact scripts, subscripts (redeemScript, witnessScript, etc.), and keys to produce. +With the general structure specified in this BIP, new Script Expressions can be introduced as new script types are added. +Lastly, the use of common terminology and existing standards allow for Output Script Descriptors to be engineer readable so that the results can be understood at a glance. + +==Specification== + +Descriptors consist of several types of expressions. +The top level expression is a <tt>SCRIPT</tt>. +This expression may be followed by <tt>#CHECKSUM</tt>, where <tt>CHECKSUM</tt> is an 8 character alphanumeric descriptor checksum. +Although the checksum is optional for parsing, applications may choose to reject descriptors that do not contain a checksum. + +===Script Expressions=== + +Script Expressions (denoted <tt>SCRIPT</tt>) are expressions which correspond directly with a Bitcoin script. +These expressions are written as functions and take arguments. +Such expressions have a script template which is filled with the arguments correspondingly. +Expressions are written with a human readable identifier string with the arguments enclosed with parentheses. +The identifier string should be alphanumeric and may include underscores. + +The arguments to a script expression are defined by that expression itself. +They could be a script expression, a key expression, or some other expression entirely. + +===Key Expressions=== + +A common expression used as an argument to script expressions are key expressions (denoted <tt>KEY</tt>). +These represent a public or private key and, optionally, information about the origin of that key. +Key expressions can only be used as arguments to script expressions. + +Key expressions consist of: +* Optionally, key origin information, consisting of: +** An open bracket <tt>[</tt> +** Exactly 8 hex characters for the fingerprint of the key where the derivation starts (see BIP 32 for details) +** Followed by zero or more <tt>/NUM</tt> or <tt>/NUMh</tt> path elements to indicate the unhardened or hardened derivation steps between the fingerprint and the key that follows. +** A closing bracket <tt>]</tt> +* Followed by the actual key, which is either: +** A hex encoded public key, which depending the script expression, may be either: +*** 66 hex character string beginning with <tt>02</tt> or <tt>03</tt> representing a compressed public key +*** 130 hex character string beginning with <tt>04</tt> representing an uncompressed public key +** A [[https://en.bitcoin.it/wiki/Wallet_import_format|WIF]] encoded private key +** <tt>xpub</tt> encoded extended public key or <tt>xprv</tt> encoded extended private key (as defined in BIP 32) +*** Followed by zero or more <tt>/NUM</tt> or <tt>/NUMh</tt> path elements indicating BIP 32 derivation steps to be taken after the given extended key. +*** Optionally followed by a single <tt>/*</tt> or <tt>/*h</tt> final step to denote all direct unhardened or hardened children. + +If the <tt>KEY</tt> is a BIP 32 extended key, before output scripts can be created, child keys must be derived using the derivation information that follows the extended key. +When the final step is <tt>/*</tt> or <tt>/*'</tt>, an output script will be produced for every child key index. +The derived key must be not be serialized as an uncompressed public key. +Script Expressions may have further requirements on how derived public keys are serialized for script creation. + +In the above specification, the hardened indicator <tt>h</tt> may be replaced with alternative hardened indicators of <tt>H</tt> or <tt>'</tt>. + +====Normalization of Key Expressions with Hardened Derivation==== + +When a descriptor is exported without private keys, it is necessary to do additional derivation to remove any intermediate hardened derivation steps for the exported descriptor to be useful. +The exporter should derive the extended public key at the last hardened derivation step and use that extended public key as the key in the descriptor. +The derivation steps that were taken to get to that key must be added to the previous key origin information. +If there is no key origin information, then one must be added for the newly derived extended public key. +If the final derivation is hardened, then it is not necessary to do additional derivation. + +===Character Set=== + +The expressions used in descriptors must only contain characters within this character set so that the descriptor checksum will work. + +The allowed characters are: +<pre> +0123456789()[],'/*abcdefgh@:$%{} +IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ +ijklmnopqrstuvwxyzABCDEFGH`#"\<space> +</pre> +Note that <tt><space></tt> on the last line is a space character. + +This character set is written as 3 groups of 32 characters in this specific order so that the checksum below can identify more errors. +The first group are the most common "unprotected" characters (i.e. things such as hex and keypaths that do not already have their own checksums). +Case errors cause an offset that is a multiple of 32 while as many alphabetic characters are in the same group while following the previous restrictions. + +===Checksum=== + +Following the top level script expression is a single octothorpe (<tt>#</tt>) followed by the 8 character checksum. +The checksum is an error correcting checksum similar to bech32. + +The checksum has the following properties: +* Mistakes in a descriptor string are measured in "symbol errors". The higher the number of symbol errors, the harder it is to detect: +** An error substituting a character from <tt>0123456789()[],'/*abcdefgh@:$%{}</tt> for another in that set always counts as 1 symbol error. +*** Note that hex encoded keys are covered by these characters. Extended keys (<tt>xpub</tt> and <tt>xprv</tt>) use other characters too, but also have their own checksum mechanism. +*** <tt>SCRIPT</tt> expression function names use other characters, but mistakes in these would generally result in an unparsable descriptor. +** A case error always counts as 1 symbol error. +** Any other 1 character substitution error counts as 1 or 2 symbol errors. +* Any 1 symbol error is always detected. +* Any 2 or 3 symbol error in a descriptor of up to 49154 characters is always detected. +* Any 4 symbol error in a descriptor of up to 507 characters is always detected. +* Any 5 symbol error in a descriptor of up to 77 characters is always detected. +* Is optimized to minimize the chance of a 5 symbol error in a descriptor up to 387 characters is undetected +* Random errors have a chance of 1 in 2<super>40</super> of being undetected. + +The checksum itself uses the same character set as bech32: <tt>qpzry9x8gf2tvdw0s3jn54khce6mua7l</tt> + +Valid descriptor strings with a checksum must pass the criteria for validity specified by the Python3 code snippet below. +The function <tt>descsum_check</tt> must return true when its argument <tt>s</tt> is a descriptor consisting in the form <tt>SCRIPT#CHECKSUM</tt>. + +<pre> +INPUT_CHARSET = "0123456789()[],'/*abcdefgh@:$%{}IJKLMNOPQRSTUVWXYZ&+-.;<=>?!^_|~ijklmnopqrstuvwxyzABCDEFGH`#\"\\ " +CHECKSUM_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l" +GENERATOR = [0xf5dee51989, 0xa9fdca3312, 0x1bab10e32d, 0x3706b1677a, 0x644d626ffd] + +def descsum_polymod(symbols): + """Internal function that computes the descriptor checksum.""" + chk = 1 + for value in symbols: + top = chk >> 35 + chk = (chk & 0x7ffffffff) << 5 ^ value + for i in range(5): + chk ^= GENERATOR[i] if ((top >> i) & 1) else 0 + return chk + +def descsum_expand(s): + """Internal function that does the character to symbol expansion""" + groups = [] + symbols = [] + for c in s: + if not c in INPUT_CHARSET: + return None + v = INPUT_CHARSET.find(c) + symbols.append(v & 31) + groups.append(v >> 5) + if len(groups) == 3: + symbols.append(groups[0] * 9 + groups[1] * 3 + groups[2]) + groups = [] + if len(groups) == 1: + symbols.append(groups[0]) + elif len(groups) == 2: + symbols.append(groups[0] * 3 + groups[1]) + return symbols + +def descsum_check(s): + """Verify that the checksum is correct in a descriptor""" + if s[-9] != '#': + return False + if not all(x in CHECKSUM_CHARSET for x in s[-8:]): + return False + symbols = descsum_expand(s[:-9]) + [CHECKSUM_CHARSET.find(x) for x in s[-8:]] + return descsum_polymod(symbols) == 1 +</pre> + +This implements a BCH code that has the properties described above. +The entire descriptor string is first processed into an array of symbols. +The symbol for each character is its position within its group. +After every 3rd symbol, a 4th symbol is inserted which represents the group numbers combined together. +This means that a change that only affects the position within a group, or only a group number change, will only affect a single symbol. + +To construct a valid checksum given a script expression, the code below can be used: + +<pre> +def descsum_create(s): + """Add a checksum to a descriptor without""" + symbols = descsum_expand(s) + [0, 0, 0, 0, 0, 0, 0, 0] + checksum = descsum_polymod(symbols) ^ 1 + return s + '#' + ''.join(CHECKSUM_CHARSET[(checksum >> (5 * (7 - i))) & 31] for i in range(8)) + +</pre> + +==Test Vectors== + +The following tests cover the checksum and character set: + +* Valid checksum: <tt>raw(deadbeef)#89f8spxm</tt> +* No checksum: <tt>raw(deadbeef)</tt> +* Missing checksum: <tt>raw(deadbeef)#</tt> +* Too long checksum (9 chars): <tt>raw(deadbeef)#89f8spxmx</tt> +* Too short checksum (7 chars): <tt>raw(deadbeef)#89f8spx</tt> +* Error in payload: <tt>raw(deedbeef)#89f8spxm</tt> +* Error in checksum: <tt>raw(deedbeef)##9f8spxm</tt> +* Invalid characters in payload: <tt>raw(Ü)#00000000</tt> + +The following tests cover key expressions: + +Valid expressions: + +* Compressed public key: <tt>0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Uncompressed public key: <tt>04a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd5b8dec5235a0fa8722476c7709c02559e3aa73aa03918ba2d492eea75abea235</tt> +* Public key with key origin: <tt>[deadbeef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Public key with key origin (<tt>'</tt> as hardened indicator): <tt>[deadbeef/0'/0'/0']0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Public key with key origin (mixed hardened indicator): <tt>[deadbeef/0'/0h/0']0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* WIF uncompressed private key <tt>5KYZdUEo39z3FPrtuX2QbbwGnNP5zTd7yyr2SC1j299sBCnWjss</tt> +* WIF compressed private key <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1</tt> +* Extended public key: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL</tt> +* Extended public key with key origin: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL</tt> +* Extended public key with derivation: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3/4/5</tt> +* Extended public key with derivation and children: <tt>[deadbeef/0h/1h/2h]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3/4/5/*</tt> +* Extended public key with hardened derivation and unhardened children: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*</tt> +* Extended public key with hardened derivation and children: <tt>xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*h</tt> +* Extended public key with key origin, hardened derivation and children: <tt>[deadbeef/0h/1h/2]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/3h/4h/5h/*h</tt> +* Extended private key: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc</tt> +* Extended private key with key origin: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc</tt> +* Extended private key with derivation: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3/4/5</tt> +* Extended private key with derivation and children: <tt>[deadbeef/0h/1h/2h]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3/4/5/*</tt> +* Extended private key with hardened derivation and unhardened children: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*</tt> +* Extended private key with hardened derivation and children: <tt>xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*h</tt> +* Extended private key with key origin, hardened derivation and children: <tt>[deadbeef/0h/1h/2]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc/3h/4h/5h/*h</tt> + +Invalid expression: + +* Children indicator in key origin: <tt>[deadbeef/0h/0h/0h/*]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Trailing slash in key origin: <tt>[deadbeef/0h/0h/0h/]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Too short fingerprint: <tt>[deadbef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Too long fingerprint: <tt>[deadbeeef/0h/0h/0h]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Invalid hardened indicators: <tt>[deadbeef/0f/0f/0f]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Invalid hardened indicators: <tt>[deadbeef/0H/0H/0H]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Invalid hardened indicators: <tt>[deadbeef/-0/-0/-0]0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600</tt> +* Private key with derivation: <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1/0</tt> +* Private key with derivation children: <tt>L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1/*</tt> +* Derivation index out of range: <tt>xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483648</tt> +* Invalid derivation index: <tt>xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/1aa</tt> +* Multiple key origins: <tt>[aaaaaaaa][aaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt> +* Missing key origin start: <tt>aaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt> +* Non hex fingerprint: <tt>[gaaaaaaa]xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U/2147483647'/0</tt> +* Key origin with no public key: <tt>[deadbeef]</tt> + +==Backwards Compatibility== + +Output script descriptors are an entirely new language which is not compatible with any existing software. +However many components of the expressions reuse encodings and serializations defined by previous BIPs. + +Output script descriptors are designed for future extension with further fragment types and new script expressions. +These will be specified in additional BIPs. + +==Reference Implementation== + +Descriptors have been implemented in Bitcoin Core since version 0.17. + +==Appendix A: Index of Expressions== + +Future BIPs may specify additional types of expressions. +All available expression types are listed in this table. + +{| +! Name +! Denoted As +! BIP +|- +| Script +| <tt>SCRIPT</tt> +| 380 +|- +| Key +| <tt>KEY</tt> +| 380 +|- +| Tree +| <tt>TREE</tt> +| [[bip-0386.mediawiki|386]] +|} + +==Appendix B: Index of Script Expressions== + +Script expressions will be specified in additional BIPs. +This Table lists all available Script expressions and the BIPs specifying them. + +{| +! Expression +! BIP +|- +| <tt>pk(KEY)</tt> +| [[bip-0381.mediawiki|381]] +|- +| <tt>pkh(KEY)</tt> +| [[bip-0381.mediawiki|381]] +|- +| <tt>sh(SCRIPT)</tt> +| [[bip-0381.mediawiki|381]] +|- +| <tt>wpkh(KEY)</tt> +| [[bip-0382.mediawiki|382]] +|- +| <tt>wsh(SCRIPT)</tt> +| [[bip-0382.mediawiki|382]] +|- +| <tt>multi(NUM, KEY, ..., KEY)</tt> +| [[bip-0383.mediawiki|383]] +|- +| <tt>sortedmulti(NUM, KEY, ..., KEY)</tt> +| [[bip-0383.mediawiki|383]] +|- +| <tt>combo(KEY)</tt> +| [[bip-0384.mediawiki|384]] +|- +| <tt>raw(HEX)</tt> +| [[bip-0385.mediawiki|385]] +|- +| <tt>addr(ADDR)</tt> +| [[bip-0385.mediawiki|385]] +|- +| <tt>tr(KEY)</tt>, <tt>tr(KEY, TREE)</tt> +| [[bip-0386.mediawiki|386]] +|} |