diff options
| author | Li Qiang <liq3ea@163.com> | 2019-08-31 08:39:22 -0700 |
|---|---|---|
| committer | Gerd Hoffmann <kraxel@redhat.com> | 2019-09-17 13:45:10 +0200 |
| commit | 6bf21f3d83e95bcc4ba35a7a07cc6655e8b010b0 (patch) | |
| tree | 659fa371214fe7d4b736d3940004ac4aeff1a5c7 /ui/vnc-enc-zlib.c | |
| parent | 6105683da35babad9ae168a72d1e89e63e9d6974 (diff) | |
vnc: fix memory leak when vnc disconnect
Currently when qemu receives a vnc connect, it creates a 'VncState' to
represent this connection. In 'vnc_worker_thread_loop' it creates a
local 'VncState'. The connection 'VcnState' and local 'VncState' exchange
data in 'vnc_async_encoding_start' and 'vnc_async_encoding_end'.
In 'zrle_compress_data' it calls 'deflateInit2' to allocate the libz library
opaque data. The 'VncState' used in 'zrle_compress_data' is the local
'VncState'. In 'vnc_zrle_clear' it calls 'deflateEnd' to free the libz
library opaque data. The 'VncState' used in 'vnc_zrle_clear' is the connection
'VncState'. In currently implementation there will be a memory leak when the
vnc disconnect. Following is the asan output backtrack:
Direct leak of 29760 byte(s) in 5 object(s) allocated from:
0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3)
1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb)
2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7)
3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87
4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344
5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919
6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271
7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340
8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502
9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb)
10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb)
This is because the opaque allocated in 'deflateInit2' is not freed in
'deflateEnd'. The reason is that the 'deflateEnd' calls 'deflateStateCheck'
and in the latter will check whether 's->strm != strm'(libz's data structure).
This check will be true so in 'deflateEnd' it just return 'Z_STREAM_ERROR' and
not free the data allocated in 'deflateInit2'.
The reason this happens is that the 'VncState' contains the whole 'VncZrle',
so when calling 'deflateInit2', the 's->strm' will be the local address.
So 's->strm != strm' will be true.
To fix this issue, we need to make 'zrle' of 'VncState' to be a pointer.
Then the connection 'VncState' and local 'VncState' exchange mechanism will
work as expection. The 'tight' of 'VncState' has the same issue, let's also turn
it to a pointer.
Reported-by: Ying Fang <fangying1@huawei.com>
Signed-off-by: Li Qiang <liq3ea@163.com>
Message-id: 20190831153922.121308-1-liq3ea@163.com
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Diffstat (limited to 'ui/vnc-enc-zlib.c')
| -rw-r--r-- | ui/vnc-enc-zlib.c | 11 |
1 files changed, 6 insertions, 5 deletions
diff --git a/ui/vnc-enc-zlib.c b/ui/vnc-enc-zlib.c index 33e9df2f6a..900ae5b30f 100644 --- a/ui/vnc-enc-zlib.c +++ b/ui/vnc-enc-zlib.c @@ -76,7 +76,8 @@ static int vnc_zlib_stop(VncState *vs) zstream->zalloc = vnc_zlib_zalloc; zstream->zfree = vnc_zlib_zfree; - err = deflateInit2(zstream, vs->tight.compression, Z_DEFLATED, MAX_WBITS, + err = deflateInit2(zstream, vs->tight->compression, Z_DEFLATED, + MAX_WBITS, MAX_MEM_LEVEL, Z_DEFAULT_STRATEGY); if (err != Z_OK) { @@ -84,16 +85,16 @@ static int vnc_zlib_stop(VncState *vs) return -1; } - vs->zlib.level = vs->tight.compression; + vs->zlib.level = vs->tight->compression; zstream->opaque = vs; } - if (vs->tight.compression != vs->zlib.level) { - if (deflateParams(zstream, vs->tight.compression, + if (vs->tight->compression != vs->zlib.level) { + if (deflateParams(zstream, vs->tight->compression, Z_DEFAULT_STRATEGY) != Z_OK) { return -1; } - vs->zlib.level = vs->tight.compression; + vs->zlib.level = vs->tight->compression; } // reserve memory in output buffer |