aboutsummaryrefslogtreecommitdiff
path: root/include/exec/memory-internal.h
AgeCommit message (Collapse)Author
2019-09-25cputlb: Merge and move memory_notdirty_write_{prepare,complete}Richard Henderson
Since 9458a9a1df1a, all readers of the dirty bitmaps wait for the rcu lock, which means that they wait until the end of any executing TranslationBlock. As a consequence, there is no need for the actual access to happen in between the _prepare and _complete. Therefore, we can improve things by merging the two functions into notdirty_write and dropping the NotDirtyInfo structure. In addition, the only users of notdirty_write are in cputlb.c, so move the merged function there. Pass in the CPUIOTLBEntry from which the ram_addr_t may be computed. Reviewed-by: David Hildenbrand <david@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2019-08-16include: Make headers more self-containedMarkus Armbruster
Back in 2016, we discussed[1] rules for headers, and these were generally liked: 1. Have a carefully curated header that's included everywhere first. We got that already thanks to Peter: osdep.h. 2. Headers should normally include everything they need beyond osdep.h. If exceptions are needed for some reason, they must be documented in the header. If all that's needed from a header is typedefs, put those into qemu/typedefs.h instead of including the header. 3. Cyclic inclusion is forbidden. This patch gets include/ closer to obeying 2. It's actually extracted from my "[RFC] Baby steps towards saner headers" series[2], which demonstrates a possible path towards checking 2 automatically. It passes the RFC test there. [1] Message-ID: <87h9g8j57d.fsf@blackfin.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg03345.html [2] Message-Id: <20190711122827.18970-1-armbru@redhat.com> https://lists.nongnu.org/archive/html/qemu-devel/2019-07/msg02715.html Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Message-Id: <20190812052359.30071-2-armbru@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2019-04-18memory: Clean up how mtree_info() printsMarkus Armbruster
mtree_info() takes an fprintf()-like callback and a FILE * to pass to it, and so do its helper functions. Passing around callback and argument is rather tiresome. Its only caller hmp_info_mtree() passes monitor_printf() cast to fprintf_function and the current monitor cast to FILE *. The type-punning is technically undefined behaviour, but works in practice. Clean up: drop the callback, and call qemu_printf() instead. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Message-Id: <20190417191805.28198-9-armbru@redhat.com>
2018-06-15tcg: remove tb_lockEmilio G. Cota
Use mmap_lock in user-mode to protect TCG state and the page descriptors. In !user-mode, each vCPU has its own TCG state, so no locks needed. Per-page locks are used to protect the page descriptors. Per-TB locks are used in both modes to protect TB jumps. Some notes: - tb_lock is removed from notdirty_mem_write by passing a locked page_collection to tb_invalidate_phys_page_fast. - tcg_tb_lookup/remove/insert/etc have their own internal lock(s), so there is no need to further serialize access to them. - do_tb_flush is run in a safe async context, meaning no other vCPU threads are running. Therefore acquiring mmap_lock there is just to please tools such as thread sanitizer. - Not visible in the diff, but tb_invalidate_phys_page already has an assert_memory_lock. - cpu_io_recompile is !user-only, so no mmap_lock there. - Added mmap_unlock()'s before all siglongjmp's that could be called in user-mode while mmap_lock is held. + Added an assert for !have_mmap_lock() after returning from the longjmp in cpu_exec, just like we do in cpu_exec_step_atomic. Performance numbers before/after: Host: AMD Opteron(tm) Processor 6376 ubuntu 17.04 ppc64 bootup+shutdown time 700 +-+--+----+------+------------+-----------+------------*--+-+ | + + + + + *B | | before ***B*** ** * | |tb lock removal ###D### *** | 600 +-+ *** +-+ | ** # | | *B* #D | | *** * ## | 500 +-+ *** ### +-+ | * *** ### | | *B* # ## | | ** * #D# | 400 +-+ ** ## +-+ | ** ### | | ** ## | | ** # ## | 300 +-+ * B* #D# +-+ | B *** ### | | * ** #### | | * *** ### | 200 +-+ B *B #D# +-+ | #B* * ## # | | #* ## | | + D##D# + + + + | 100 +-+--+----+------+------------+-----------+------------+--+-+ 1 8 16 Guest CPUs 48 64 png: https://imgur.com/HwmBHXe debian jessie aarch64 bootup+shutdown time 90 +-+--+-----+-----+------------+------------+------------+--+-+ | + + + + + + | | before ***B*** B | 80 +tb lock removal ###D### **D +-+ | **### | | **## | 70 +-+ ** # +-+ | ** ## | | ** # | 60 +-+ *B ## +-+ | ** ## | | *** #D | 50 +-+ *** ## +-+ | * ** ### | | **B* ### | 40 +-+ **** # ## +-+ | **** #D# | | ***B** ### | 30 +-+ B***B** #### +-+ | B * * # ### | | B ###D# | 20 +-+ D ##D## +-+ | D# | | + + + + + + | 10 +-+--+-----+-----+------------+------------+------------+--+-+ 1 8 16 Guest CPUs 48 64 png: https://imgur.com/iGpGFtv The gains are high for 4-8 CPUs. Beyond that point, however, unrelated lock contention significantly hurts scalability. Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Emilio G. Cota <cota@braap.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2018-05-31Make memory_region_access_valid() take a MemTxAttrs argumentPeter Maydell
As part of plumbing MemTxAttrs down to the IOMMU translate method, add MemTxAttrs as an argument to memory_region_access_valid(). Its callers either have an attrs value to hand, or don't care and can use MEMTXATTRS_UNSPECIFIED. The callsite in flatview_access_valid() is part of a recursive loop flatview_access_valid() -> memory_region_access_valid() -> subpage_accepts() -> flatview_access_valid(); we make it pass MEMTXATTRS_UNSPECIFIED for now, until the next several commits have plumbed an attrs parameter through the rest of the loop and we can add an attrs parameter to flatview_access_valid(). Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20180521140402.23318-8-peter.maydell@linaro.org
2018-05-09exec: reintroduce MemoryRegion cachingPaolo Bonzini
MemoryRegionCache was reverted to "normal" address_space_* operations for 2.9, due to lack of support for IOMMUs. Reinstate the optimizations, caching only the IOMMU translation at address_cache_init but not the IOMMU lookup and target AddressSpace translation are not cached; now that MemoryRegionCache supports IOMMUs, it becomes more widely applicable too. The inlined fast path is defined in memory_ldst_cached.inc.h, while the slow path uses memory_ldst.inc.c as before. The smaller fast path causes a little code size reduction in MemoryRegionCache users: hw/virtio/virtio.o text size before: 32373 hw/virtio/virtio.o text size after: 31941 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-03-06memory: inline some performance-sensitive accessorsPaolo Bonzini
These accessors are called from inlined functions, and the call sequence is much more expensive than just inlining the access. Move the struct declaration to memory-internal.h so that exec.c and memory.c can both use an inline function. Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2018-02-05memory-internal.h: Remove obsolete claim that header is obsoletePeter Maydell
The memory-internal.h header claims that it is for "obsolete exec.c functions" which "will be removed soon". This statement was added in 2011, six years ago, but the header is still here. (Admittedly none of the prototypes added in commit 67d95c153bef55f6 are still in the header.) It's convenient to have a place to put prototypes for functions which are used internally to the various .c files of the memory system or by the accel/tcg code, which is inevitably fairly closely coupled. So keep the header but update the comments to reflect what we're actually using it for. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-Id: <1511276888-17834-1-git-send-email-peter.maydell@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-11-21exec.c: Factor out before/after actions for notdirty memory writesPeter Maydell
The function notdirty_mem_write() has a sequence of actions it has to do before and after the actual business of writing data to host RAM to ensure that dirty flags are correctly updated and we flush any TCG translations for the region. We need to do this also in other places that write directly to host RAM, most notably the TCG atomic helper functions. Pull out the before and after pieces into their own functions. We use an API where the prepare function stashes the various bits of information about the write into a struct for the complete function to use, because in the calls for the atomic helpers the place where the complete function will be called doesn't have the information to hand. Cc: qemu-stable@nongnu.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-id: 1511201308-23580-2-git-send-email-peter.maydell@linaro.org
2017-09-21memory: Rework "info mtree" to print flat views and dispatch treesAlexey Kardashevskiy
This adds a new "-d" switch to "info mtree" to print dispatch tree internals. This changes the way "-f" is handled - it prints now flat views and associated address spaces. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Message-Id: <20170921085110.25598-15-aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-09-21memory: Rename mem_begin/mem_commit/mem_add helpersAlexey Kardashevskiy
This renames some helpers to reflect better what they do. This should cause no behavioural change. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Message-Id: <20170921085110.25598-9-aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-09-21memory: Switch memory from using AddressSpace to FlatViewAlexey Kardashevskiy
FlatView's will be shared between AddressSpace's and subpage_t and MemoryRegionSection cannot store AS anymore, hence this change. In particular, for: typedef struct subpage_t { MemoryRegion iomem; - AddressSpace *as; + FlatView *fv; hwaddr base; uint16_t sub_section[]; } subpage_t; struct MemoryRegionSection { MemoryRegion *mr; - AddressSpace *address_space; + FlatView *fv; hwaddr offset_within_region; Int128 size; hwaddr offset_within_address_space; bool readonly; }; This should cause no behavioural change. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Message-Id: <20170921085110.25598-7-aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-09-21memory: Move AddressSpaceDispatch from AddressSpace to FlatViewAlexey Kardashevskiy
As we are going to share FlatView's between AddressSpace's, and AddressSpaceDispatch is a structure to perform quick lookup in FlatView, this moves ASD to FlatView. After previosly open coded ASD rendering, we can also remove as->next_dispatch as the new FlatView pointer is stored on a stack and set to an AS atomically. flatview_destroy() is executed under RCU instead of address_space_dispatch_free() now. This makes mem_begin/mem_commit to work with ASD and mem_add with FV as later on mem_add will be taking FV as an argument anyway. This should cause no behavioural change. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Message-Id: <20170921085110.25598-5-aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2017-09-21memory: Open code FlatView renderingAlexey Kardashevskiy
We are going to share FlatView's between AddressSpace's and per-AS memory listeners won't suit the purpose anymore so open code the dispatch tree rendering. Since there is a good chance that dispatch_listener was the only listener, this avoids address_space_update_topology_pass() if there is no registered listeners; this should improve starting time. This should cause no behavioural change. Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Message-Id: <20170921085110.25598-3-aik@ozlabs.ru> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2015-02-10memory: unregister AddressSpace MemoryListener within BQLPaolo Bonzini
address_space_destroy_dispatch is called from an RCU callback and hence outside the iothread mutex (BQL). However, after address_space_destroy no new accesses can hit the destroyed AddressSpace so it is not necessary to observe changes to the memory map. Move the memory_listener_unregister call earlier, to make it thread-safe again. Reported-by: Alex Williamson <alex.williamson@redhat.com> Fixes: 374f2981d1f10bc4307f250f24b2a7ddb9b14be0 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
2014-01-13memory: split cpu_physical_memory_* functions to its own includeJuan Quintela
All the functions that use ram_addr_t should be here. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: make cpu_physical_memory_reset_dirty() take a length parameterJuan Quintela
We have an end parameter in all the callers, and this make it coherent with the rest of cpu_physical_memory_* functions, that also take a length parameter. Once here, move the start/end calculation to tlb_reset_dirty_range_all() as we don't need it here anymore. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: s/dirty/clean/ in cpu_physical_memory_is_dirty()Juan Quintela
All uses except one really want the other meaning. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_clear_dirty_range() now uses bitmap operationsJuan Quintela
We were clearing a range of bits, so use bitmap_clear(). Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_set_dirty_range() now uses bitmap operationsJuan Quintela
We were setting a range of bits, so use bitmap_set(). Note: xen has always been wrong, and should have used start instead of addr from the beginning. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: use find_next_bit() to find dirty bitsJuan Quintela
This operation is way faster than doing it bit by bit. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: s/mask/clear/ cpu_physical_memory_mask_dirty_rangeJuan Quintela
Now all functions use the same wording that bitops/bitmap operations Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_get_dirty() is used as returning a boolJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: make cpu_physical_memory_get_dirty() the main functionJuan Quintela
And make cpu_physical_memory_get_dirty_flag() to use it. It used to be the other way around. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: unfold cpu_physical_memory_set_dirty_flag()Juan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: unfold cpu_physical_memory_set_dirty() in its only userJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: unfold cpu_physical_memory_clear_dirty_flag() in its only userJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: split dirty bitmap into threeJuan Quintela
After all the previous patches, spliting the bitmap gets direct. Note: For some reason, I have to move DIRTY_MEMORY_* definitions to the beginning of memory.h to make compilation work. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_clear_dirty_flag() result is never usedJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: make sure that client is always inside rangeJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_mask_dirty_range() always clears a single flagJuan Quintela
Document it Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: cpu_physical_memory_set_dirty_range() always dirty all flagsJuan Quintela
So remove the flag argument and do it directly. After this change, there is nothing else using cpu_physical_memory_set_dirty_flags() so remove it. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: set single dirty flags when possibleJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: all users of cpu_physical_memory_get_dirty used only one flagJuan Quintela
So cpu_physical_memory_get_dirty_flags is not needed anymore Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: make cpu_physical_memory_is_dirty return boolJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13exec: create function to get a single dirty bitJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2014-01-13memory: create function to set a single dirty bitJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
2014-01-13memory: cpu_physical_memory_set_dirty_flags() result is never usedJuan Quintela
So return void. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
2013-07-04ioport: Switch dispatching to memory core layerJan Kiszka
The current ioport dispatcher is a complex beast, mostly due to the need to deal with old portio interface users. But we can overcome it without converting all portio users by embedding the required base address of a MemoryRegionPortio access into that data structure. That removes the need to have the additional MemoryRegionIORange structure in the loop on every access. To handle old portio memory ops, we simply install dispatching handlers for portio memory regions when registering them with the memory core. This removes the need for the old_portio field. We can drop the additional aliasing of ioport regions and also the special address space listener. cpu_in and cpu_out now simply call address_space_read/write. And we can concentrate portio handling in a single source file. Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-06-20memory: move private types to exec.cPaolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-05-29memory: export memory_region_access_valid to exec.cPaolo Bonzini
We'll use it to implement address_space_access_valid. Reviewed-by: Richard Henderson <rth@twiddle.net> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-05-29memory: move unassigned_mem_ops to memory.cPaolo Bonzini
reservation_ops is already doing the same thing. Reviewed-by: Richard Henderson <rth@twiddle.net> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-05-24exec: make qemu_get_ram_ptr privatePaolo Bonzini
It is a private interface between exec.c and memory.c. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-04-15exec: remove useless declarations from memory-internal.hPaolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2013-04-08hw: move headers to include/Paolo Bonzini
Many of these should be cleaned up with proper qdev-/QOM-ification. Right now there are many catch-all headers in include/hw/ARCH depending on cpu.h, and this makes it necessary to compile these files per-target. However, fixing this does not belong in these patches. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2012-12-19exec: move include files to include/exec/Paolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
generated by cgit v1.2.3 (git 2.26.2) at 2024-12-16 15:17:50 +0000