aboutsummaryrefslogtreecommitdiff
path: root/ui/vnc-enc-zrle.c
AgeCommit message (Collapse)Author
2019-09-17vnc: fix memory leak when vnc disconnectLi Qiang
Currently when qemu receives a vnc connect, it creates a 'VncState' to represent this connection. In 'vnc_worker_thread_loop' it creates a local 'VncState'. The connection 'VcnState' and local 'VncState' exchange data in 'vnc_async_encoding_start' and 'vnc_async_encoding_end'. In 'zrle_compress_data' it calls 'deflateInit2' to allocate the libz library opaque data. The 'VncState' used in 'zrle_compress_data' is the local 'VncState'. In 'vnc_zrle_clear' it calls 'deflateEnd' to free the libz library opaque data. The 'VncState' used in 'vnc_zrle_clear' is the connection 'VncState'. In currently implementation there will be a memory leak when the vnc disconnect. Following is the asan output backtrack: Direct leak of 29760 byte(s) in 5 object(s) allocated from: 0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3) 1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb) 2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7) 3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87 4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344 5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919 6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271 7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340 8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502 9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb) 10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb) This is because the opaque allocated in 'deflateInit2' is not freed in 'deflateEnd'. The reason is that the 'deflateEnd' calls 'deflateStateCheck' and in the latter will check whether 's->strm != strm'(libz's data structure). This check will be true so in 'deflateEnd' it just return 'Z_STREAM_ERROR' and not free the data allocated in 'deflateInit2'. The reason this happens is that the 'VncState' contains the whole 'VncZrle', so when calling 'deflateInit2', the 's->strm' will be the local address. So 's->strm != strm' will be true. To fix this issue, we need to make 'zrle' of 'VncState' to be a pointer. Then the connection 'VncState' and local 'VncState' exchange mechanism will work as expection. The 'tight' of 'VncState' has the same issue, let's also turn it to a pointer. Reported-by: Ying Fang <fangying1@huawei.com> Signed-off-by: Li Qiang <liq3ea@163.com> Message-id: 20190831153922.121308-1-liq3ea@163.com Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-05-11rename included C files to foo.inc.c, remove osdep.hPaolo Bonzini
osdep.h is only needed for files that are compiled directly. Remove it from included C source files, and rename them to *.inc.c so that scripts/clean-includes knows to skip them. Cc: Eric Blake <eblake@redhat.com> Cc: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-05-12vnc: simple clean upWei Qi
It is unnecessary to assign 'packed_bytes' to 'estimated_bytes', because 'estimated_bytes' unused after assignment. Signed-off-by: Wei Qi <weiqi4@huawei.com> Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2016-02-04ui: Clean up includesPeter Maydell
Clean up includes so that osdep.h is included first and headers which it implies are not included manually. This commit was created with scripts/clean-includes. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 1454089805-5470-2-git-send-email-peter.maydell@linaro.org
2012-11-01pixman/vnc: use pixman images in vnc.Gerd Hoffmann
The vnc code uses *three* DisplaySurfaces: First is the surface of the actual QemuConsole, usually the guest screen, but could also be a text console (monitor/serial reachable via Ctrl-Alt-<nr> keys). This is left as-is. Second is the current server's view of the screen content. The vnc code uses this to figure which parts of the guest screen did _really_ change to reduce the amount of updates sent to the vnc clients. It is also used as data source when sending out the updates to the clients. This surface gets replaced by a pixman image. The format changes too, instead of using the guest screen format we'll use fixed 32bit rgb framebuffer and convert the pixels on the fly when comparing and updating the server framebuffer. Third surface carries the format expected by the vnc client. That isn't used to store image data. This surface is switched to PixelFormat and a boolean for bigendian byte order. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2011-02-23vnc: fix uint8_t comparisons with negative valuesCorentin Chary
Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-02-23vnc: Add ZRLE and ZYWRLE encodings.Corentin Chary
Add ZRLE [1] and ZYWRLE [2] encodings. The code is inspire^W stolen from libvncserver (again), but have been rewriten to match QEMU coding style. [1] http://www.realvnc.com/docs/rfbproto.pdf [2] http://micro-vnc.jp/research/remote_desktop_ng/ZYWRLE/publications/ Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>