##############################################################################
@RELEASE: 7.0-2c
This maintenance release includes a few key fixes to the supervisor, core, and
worker, and is a recommended update to all 7.0-x customers.
Specific to Windows, it also includes a change where all modules and binary
files have been rebuilt against a version of Perl that has been built in-house
from source.
IMPORTANT: A note about Perl versions on Windows workers
--------------------------------------------------------
Perl v5.26 is the only supported version on Windows, and must be installed on
all workers. All other versions of Perl are unsupported.
==== CL 20342 ====
@CHANGE: made sure installed Perl versions other than 5.26 are detected as "unsupported" and properly rejected as such (Windows).
==== CL 20338 ====
@FIX: Proxy program was responding to orders (such as migrate/interrupt/preempt) prematurely, before the operations had completed,
causing race coditions in some (rare) cases, resulting in issues such as agenda-based job instances going into an infinite "wait" loop.
ZD: 19066
==== CL 20303 ====
@FIX: bug in a couple of SQL queries causing issues, such as tons of unnecessary extra auto-complete operations.
==== CL 20302 ====
@FIX: qblogin password registriation failing with "ERROR: passwords don't match" on Windows 10
ZD: 18963
JIRA: QUBE-3439
##############################################################################
@RELEASE: 7.0-2b
This is a supervisor-only release that fixes a critical bug introduced in
7.0-2a. Recommended for any site currently using a 7.0-x supervisor.
==== CL 20255 ====
@FIX: issue introduced in 7.0-2a supe where ageda-based jobs will not dispatch to instances.
==== CL 20252 ====
@NEW: add README-DB-VERSIONING.txt file, for informational purposes only.
JIRA: QUBE-3190
##############################################################################
@RELEASE: 7.0-2a
##############################################################################
This is a supervisor-only release that includes a few key fixes, and is
recommended for any site currently using a 7.0-x supervisor.
==== CL 20233 ====
@FIX: timeout value set via the API routine qbsettimeout() now respected more accurately
ZD: 18524
JIRA: QUBE-3181
==== CL 20223 ====
@FIX: race-condition can dispatch the same agenda item to multiple instances
ZD: 18980
==== CL 20222 ====
@FIX: "running_monitor" background thread/routine would sometimes try to verify instance-worker combinations that don't exist in reality (and sometimes even non-existent instances), then put those instances back to "pending".
ZD: 18960
==== CL 20218 ====
@FIX: supervisor install fails on Windows where .py files are not associated with the python interpreter
==== CL 20215 ====
@FIX: issue where Supe fails to register dependencies of some jobs at submission, due to DB ERROR: duplicate key value
JIRA: QUBE-3427
ZD: 18957
##############################################################################
@RELEASE: 7.0-2
##############################################################################
==== CL 20172 ====
@FIX: Multiple false matches for 'regex_outputPaths' when an instance is preempted
==== CL 20151 ====
@FIX: qbhosts, qbjobs, qblock, and qbmodify commands' "-group" option not working as expected.
Note: required a change to both the client programs as well as the supervisor (had a incompatible SQL statement).
JIRA: QUBE-3413
==== CL 20131 ====
@CHANGE: add code to check supervisor_max_threads relative to the DB server's max_connections, and adjust it if it's too large.
"Too Large" is: supervisor_max_threads > max_connections - 25
JIRA: QUBE-3396
==== CL 20115 ====
@FIX: Calling qbstdout or qbstderr API (c++ or Python API) with "pos=-N" on a file of N - 1 bytes or smaller can crash supervisor process
JIRA: QUBE-3145
==== CL 20113 ====
@FIX: qbjobs should be case-insensitive for user name; qbhosts should be case-insensitive for host names
JIRA: QUBE-3320, QUBE-3386
==== CL 20108 ====
@FIX: fix issue where Qube 7 Perl API module doesn't load on Linux, causing Perl based jobs such as the Maya jobtype job not be able to run.
Error messages:
Can't load '/usr/local/pfx/qube//api/perl/qb.so' for module qb: libQt5Core.so.5: cannot open shared object file: No such file or directory at /usr/lib64/perl5/DynaLoader.pm line 190.
Added /etc/ld.so.cof.d/qube-x86_64.conf file to the qube-core package, to add the path /usr/local/pfx/qube/lib/Qt to the runtime library search path
JIRA: QUBE-3403
ZD: 18920
==== CL 20102 ====
@FIX: 'DB ERROR: relation "joblog" does not exit' message in supelog
Fixed issue where ERROR messages like the following would show in the supelog:
QbDatabasePostgreSQL::_raw() DB ERROR: lastError=ERROR: relation "joblog" does not exist
LINE 1: SELECT id, jobid, subid, type, data, host FROM joblog WHERE...
JIRA: QUBE-3186
ZD: 18899
==== CL 20094 ====
@FIX: Qube API module for python 2.6 (_qb26.pyd) not loading
JIRA: QUBE-3387
==== CL 20081 ====
@FIX: On the Mac, the "pfx" user is created every time the supervisor package is installed or upgraded (add_pfx_account_osx.sh script).
##############################################################################
@RELEASE: 7.0-1c
##############################################################################
------------------------------------------------------------------
Added support for CentOS-7.4 and CentOS-7.5 in this release
------------------------------------------------------------------
==== CL 20071 ====
@FIX: agenda items failing due to timeout won't auto-retry
ZD: 18874
==== CL 20070 ====
@FIX: 7.0-1 supe installer won't work on platforms with python 2.6 (issue with init_supe_db.py script)
JIRA: QUBE-3379
==== CL 20028 ====
@FIX: querying supervisor config never returns database_port unless it's explicitly defined
JIRA: QUBE-3357
==== CL 20025 ====
@FIX: extremely inaccurate cumulative cpu time for agenda items
JIRA: QUBE-3375
ZD: 18841
##############################################################################
@RELEASE: 7.0-1a
##############################################################################
==== CL 20002 ====
@NEW: add new filtering options to Python API's qb.jobinfo():
submittedBeforesubmittedAfterupdatedBeforeupdatedAfter
Args specifying time may be in seconds since Unix epoch, or a datetime.date or datetime.datetime object.
Examples:
import qb
qb.jobinfo(updatedAfter=1530680480, updatedBefore=1530680700)
import datetime
weekago = datetime.date.today() - datetime.timedelta(days=7)
qb.jobinfo(submittedAfter=weekago)
JIRA: QUBE-3353
==== CL 19999 ====
@NEW: add new filtering options to qbjobs command: submittedBefore, submittedAfter, updatedBefore, updatedAfter. Option arg specifying time needs to be in seconds since Unix epoch.
JIRA: QUBE-3353
##############################################################################
@RELEASE: 7.0-1
##############################################################################
This release includes key fixes to the supervisor, core and WV, among other fixes, and is
strongly recommended for any site currently using 7.0-0 or above.
==== CL 19967 ====
@FIX: include support for earlier versions of Perl. Now we support Perl 5.14 thru 5.26.
==== CL 19964 ====
@CHANGE: adjust (Windows) perl support to versions 5.14 thru 5.26
JIRA: QUBE-3368
==== CL 19957 ====
@FIX: update perl API's copy of IPRC::Run module to properly support Perl 5.26
==== CL 19938 ====
@FIX: Python 2.6 module (_qb26.pyd) was not included. Python modules were named with a .dll extension, instead of the more preferred .pyd
JIRA: QUBE-3358
==== CL 19937 ====
@FIX: Python API .py files missing and/or incorrectly located w/o proper directory structure under QBDIR/api/lib/python
JIRA: QUBE-3358
==== CL 19935 ====
@CHANGE: ignore if database_port is set to 3300 or 3306 and revert to the default, 50055.
JIRA: QUBE-3359
==== CL 19934 ====
@FIX: init_supe_db.py: add version-aware execution of qubedb_xxxx.sql files
JIRA: QUBE-3336
==== CL 19933 ====
@FIX: add important missing INDEXes to the qube.subjob DB table for performance boost.
==== CL 19931 ====
@FIX:Fix relative movie paths in images_to_move.py
==== CL 19915 ====
@CHANGE: remove direct SQL access from job cleanup script
@CHANGE: find_corrupt_jobs.py removed from qube installation, it checked for the existence of tables which are no longer in the Qube 7 schema
==== CL 19897 ====
@FIX: auto_remove worker flag missing from worker config dialogs
==== CL 19888 ====
@FIX: issue where Windows password couldn't be updated.
==== CL 19849 ====
@CHANGE: add the postgres-based database checks back into the supervisor installers
==== CL 19836 ====
@NEW: convert database_checks cmdline utility from MySQL to PostgreSQL
##############################################################################
@RELEASE: 7.0-0a
##############################################################################
==== CL 19831 ====
@FIX: jobs submitted that reserves a global resource never runs
Bug introduced at 7.0-0, where jobs specifying any global resource reservation would be stuck indefinitely in "pending" state.
JIRA: QUBE-3328
##############################################################################
@RELEASE: 7.0-0
##############################################################################
==== CL 19673 ====
@CHANGE: allow Metered Licensing (ML) with just a valid, unexpired supe license, and no worker licenese
JIRA: QUBE-2823
==== CL 19637 ====
@FIX: Perl API: added proper qb::version() support
==== CL 19636 ====
@NEW: add support for Perl 5.18, 20, 22, 24, and 26 on Windows.
JIRA: QUBE-749
==== CL 19529 ====
@NEW: add paexec.exe and ntrights.exe to aid with proper installation of postgresql server
==== CL 19507 ====
@NEW: add com.pipelinefx.postgresql.plist file to enable launchd support of PostgreSQL DB server on macOS
JIRA: QUBE-3100
==== CL 19504 ====
@NEW: add_pfx_account_osx.sh script that creates the "pfx" account that runs the PostgreSQL DB server.
The script is run from the "postinstall" process of the pkg installer.
JIRA: QUBE-3100
==== CL 19478 ====
@FIX: workers are always "auto-remove"d, even if "auto_remove" is not set in worker_flags.
ZD: 18512
JIRA: QUBE-3174
==== CL 19475 ====
@FIX: issue where instances would be stuck in "QB_PREEMPT_MODE_FAIL", causing the supervisor to tell instances to "wait and retry later" in response to retryWork() indefinitely.
Issue was caused when the preemptJobNetwork() routine determines that the
instance has started but has NOT yet started working on an agenda item, in
which case it would mark the QB_PREEMPT_MODE_FAIL in order to interrupt
(i.e. aggressively preempt) the instance; However, the interrupt was not
being triggered properly.
Issue was apparently introduced in CL19126.
==== CL 19462 ====
@FIX: issue where some daemon (supe/worker) threads exit early, after processing less client requests than specified via max_clients (e.g. 65, not 256).
Early exits should now only happen when "max threads" happened earlier.
==== CL 19457 ====
@TWEAK: add a couple of useful supelog lines to pring in assignjob(), regarding result of calling converseWorker() for dispatch
==== CL 19454 ====
@FIX: add call to sendHostReport() so that a statusHost message is sent to the supe when the worker "received kill order for unassigned job". This should eliminate some of the jobs that stay in "dying"(or allow "kill" of jobs that are stuck in "dying")
==== CL 19443 ====
@NEW:Add KeyShot commandline render script for batch rendering
==== CL 19437 ====
@FIX: workid is not duplicated by QbHistory copy constructor
==== CL 19436 ====
@FIX: "down" workers not always detected properly
JIRA: QUBE-3155
ZD: 18425
==== CL 19425 ====
@FIX: issue when supe thread doesn't hear back from worker during a dispatch. Related to CL19243.
Also fixed an issue (probably harmless) where an extra call to queue.releaseJob() was sometimes made in the findSubjobAndReserveJob() method.
==== CL 19415 ====
@CHANGE: add qbsub support for jobtypes "pyCmdrange" and "pyCmdline".
==== CL 19263 ====
@FIX: log directories for jobs submitted after the utility has been started but before the orphaned log removal is begun are erroneously removed
==== CL 19258 ====
@FIX: not running --use-frm when first-pass repair fails when message has different line-endings than OS X
==== CL 19243 ====
@FIX: add code to avoid mixed-up job instance status when worker-supervisor communications are dropped during job dispatch on an intermittently unreliable network
It was found that network hiccups can cause a worker to not respond to the
supervisor during the dispatch of a job instance, but still start running
the instance anyway. The worker would send the "running" instance report to
the supervisor, which is processed by a separate thread, which updates the
DB, causing a status mix-up.
Added code to detect such situations, and allowed the system to let the job
run (instead of force-removing it from the duty table) on the worker in
question.
Also added error-checking code on the worker side-- if worker detects that
it couldn't respond to the supe for a dispatch order, it will give up on
that job and release resources that it had just reserved for it.
ZD: 17868
==== CL 19236 ====
@FIX: jobs submitted by non-admin user without a specified priority attempt to submit at priority -1
JIRA: QUBE-3015
==== CL 19209 ====
@FIX: "down" workers would not be detected properly by the supervisor even when the supervisor_heartbeat_timeout expired.
ZD: 18057
JIRA: QUBE-3018
==== CL 19178 ====
@FIX: timing issue causing workers to get stuck with job instances.
Issue was seen on a very busy farm with intermittently drops in network
communications, when many supe threads would try to dispatch a single
instance at the same time.
ZD: 17868
==== CL 19164 ====
@CHANGE: On Unix, by default, supe uses a Unix domain socket to connect to the PostgreSQL server, unless the "database_host" parameter is set.
The default value of database_host is "" on Unix (Linux/macOSX), and "localhost" on Windows.
==== CL 19163 ====
@FIX: fix an issue where a worker can sometimes get stuck with a job instance that it's not running any longer
* Issue was seen when job instances are migrated and there are intermittent
networking issues between the supe and worker causing job updates to NO
come thur in an expected, orderly fashion.
ZD: 17868
==== CL 19126 ====
@FIX: on a network with intermittent worker-supe commnuication issues, bad timing can cause job instances to get stuck in "running" state
* In a bunch of routines that handle job-command executions (i.e., migrate,
kill, etc.) in QbSupervisorCommand, add code to do one last check when a
worker is unreachable, to see if the instance still belongs to the worker
before updating the instance on DB. It was found that, since a thread
dealing with down workers can spend quite a long time, sometimes
instances that a worker was processing can be moved off of it and the DB
updated by another thread (for example, assigned and running on another
worker)-- the check is designed to prevent our thread from overwriting
such updates.
ZD: 17868
==== CL 19121 ====
@FIX: job instances cane get into an odd state when dispatch routine doesn't hear back from the worker ("found dead").
Networking hiccups can cause this communication drop, which in turn may
cause job instances to be "stuck" in the running state on a worker, and be
unkillable.
ZD: 17868
==== CL 19118 ====
@FIX: Systemctl unit files for worker and supervisor not installed into correct location
==== CL 19109 ====
@FIX: optimize job cleanup script
@CHANGE: only scan log directories if log removal necessary
@CHANGE: removal of large number of orphaned log directories does not require skipping sanity checks
==== CL 18985 ====
@FIX: 'No database selected' MySQL error when removing ghost jobs
ZD: 17882
==== CL 18911 ====
@TWEAK: add workerlog to show the host's available properties when inspecting a newly dispatched job (when "checking job requirements").
==== CL 18910 ====
@INTERNAL FIX: supervisor patches to help cut down on the number of threads, and reduce chances of repeated worker rejections on some farms due to race-conditions/timing issues.
ZD17713
==== CL 18831 ====
@NEW: add support for retrieval of only a specified range of jobs (IDs, date, N most recent, etc) in the qbjobinfo() API
Changed the "sign" field of QbFilter class to be a QbString rather than a char, to support SQL operators that are longer than a single character, such as ">=", "<>" or "!=".
Added a "limit" and "order_by" fields to the QbQuery class, so that any query can limit the number of jobs returned, and specify the sort order.
Made change to db-support code (QbDatbase.cpp) and supervisor code (QbSupervisorQuery.cpp and Queue) to take advantage of the above changes and implement the desierd range-specific queries.
JIRA: QUBE-2658
==== CL 18822 ====
@FIX: a bug in the startHost() dispatch routine causing the supervisor NOT to always dispatch jobs to workers when they became available.
@INTERNAL: QbServer::printMemUsage() modified to only kick in if QB_DEBUG_SERVER_MEM_USAGE is defined
ZD: 17713
==== CL 18802 ====
@FIX:Correct 'restrictions' variable name and 'Restrictions' label
==== CL 18717 ====
@FIX: Job instances can become unkill-able with QB_PREEMPT_MODE_FAIL internal status
JIRA: QUBE-2819
==== CL 18680 ====
@FIX: supervisor rpm uninstall leaves the mysql/mariadb service in a stopped state instead of restarting it