The NVIDIA Persistence Daemon is a userspace daemon on Linux to support
persistence of driver state across Cuda job runs.  The daemon targets
all current Tesla, Quadro and GRID GPUs.  For further information, see

https://docs.nvidia.com/deploy/driver-persistence/index.html

The driver now includes rc.nvidia-persistenced as an option.
While it can be run as root, it is more secure to run it as an
unpriveleged user, so we run it as user nvidia.

To install it, Pass PERSISTENCED=yes to the script.  It will create the
user and group nvidia, and install rc.nvidia-persistenced to /etc/rc.d.

To activate the daemon, add the following to /etc/rc.d/local:

# Start the nvidia-persistenced daemon:
if  [ -x /etc/rc.d/rc.nvidia-persistenced ]; then
    /etc/rc.d/rc.nvidia-persistenced start
fi

and then 'chmod +x /etc/rc.d/rc.nvidia-persistenced' to make it
executable.

Add the following to /etc/rc.d/rc.local_shutdown:

# Stop the nvidia-persistenced daemon:
if  [ -x /etc/rc.d/rc.nvidia-persistenced ]; then
    /etc/rc.d/rc.nvidia-persistenced stop
fi

NOTE: This may not work on all systems.  If you encounter problems
with it, just 'chmod -x /etc/rc.d/rc.persistenced' and it will no
longer load.  YMMV.