diff options
| author | Volker Rümelin <vr_qemu@t-online.de> | 2020-03-08 20:33:20 +0100 |
|---|---|---|
| committer | Gerd Hoffmann <kraxel@redhat.com> | 2020-03-16 10:18:07 +0100 |
| commit | 194bdf50697689768335096b6c01c5b010f023ca (patch) | |
| tree | 6cb6fbe1a1df715f0ee6b14f71e2e092731c0444 /audio/mixeng_template.h | |
| parent | 4218fdd77f1c8ab4dab5ced30c3a0db946a6f04c (diff) | |
audio: fix saturation nonlinearity in clip_* functions
The current positive limit for the saturation nonlinearity is
only correct if the type of the result has 8 bits or less.
Signed-off-by: Volker Rümelin <vr_qemu@t-online.de>
Message-id: 20200308193321.20668-5-vr_qemu@t-online.de
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Diffstat (limited to 'audio/mixeng_template.h')
| -rw-r--r-- | audio/mixeng_template.h | 5 |
1 files changed, 2 insertions, 3 deletions
diff --git a/audio/mixeng_template.h b/audio/mixeng_template.h index fc8e1d4d9e..bc8509e423 100644 --- a/audio/mixeng_template.h +++ b/audio/mixeng_template.h @@ -83,10 +83,9 @@ static inline int64_t glue (conv_, ET) (IN_T v) static inline IN_T glue (clip_, ET) (int64_t v) { - if (v >= 0x7f000000) { + if (v >= 0x7fffffffLL) { return IN_MAX; - } - else if (v < -2147483648LL) { + } else if (v < -2147483648LL) { return IN_MIN; } |