##############################################################################
@RELEASE: 6.3.0
##############################################################################
==== CL 9013 ====
@NEW: added description of supervisor_job_flags in the qb.conf.template file
==== CL 9010 ====
@FIX: fixed memory bloat issue in supervisor threads on start up, on farms with many jobs.
In some cases, it had been reported that each supe thread was taking up 500+ MB.
==== CL 8975 ====
@NEW: add section (8.7) for "externally updatable worker resources and properties" to Administration.doc
==== CL 8957 ====
@NEW: add user name to print to supelog when a worker lock is updated
BUGZID: 63661
ZD: 4860
==== CL 8949 ====
@FIX: fix datawarehouse crontab so that 7-day tables are rebuilt twice a day
==== CL 8948 ====
@NEW: add global_resource tracking to the datawarehouse
==== CL 8935 ====
@FIX: update qb.conf templates to show the correct default value for supervisor_default_security
@INTERNAL: previous setting was an hex value, which seems to be unsupported now.
==== CL 8910 ====
@NEW: add C++ examples for using the qbupdateworkerresource(), qbupdateworkerproperties(), qbdeleteworkerresources(), and qbdeleteworkerproperties() routines
==== CL 8909 ====
@NEW: add Perl API routines for externally updated worker resources/properties
* add bindings to perl
** add qb::updateworkerresources() and updateworkerproperties() to perl api
qb::updateworkerresources("shinyambp.local", "host.ooga=2/3,host.extern=0/10")
qb::updateworkerrproperties("shinyambp.local", "host.oogaprop=3,host.oogaextprop2=11")
** add deleteworkerresources() and deleteworkerproperties() to perl
qb::deleteworkerresources($host, @resources);
qb::deleteworkerresources("shinyambp.local", "host.extenres", "host.ooga");
==== CL 8901 ====
@FIX: fixed bug where subjobs will be retried indefinitely when retrysubjob is set.
BUGZID: 63517
ZD: 2950 4661
==== CL 8889 ====
@FIX: fixed issue where the supervisor kept adding duplicate auto-wrangling and mail callbacks when jobs are resubmitted
BUGZID: 63655
ZD: 4661
==== CL 8886 ====
@INTEG: rel-6.2 -> main
----
@FIX: properly remove datawarehouse scheduled tasks for round-robin tables
==== CL 8885 ====
@FIX: properly remove datawarehouse scheduled tasks for round-robin tables
==== CL 8872 ====
@FIX: issue introduced in 6.2.1 that broke callbacks (not being triggered)
==== CL 8859 ====
@FIX: add bookmarks (TOC) to Admin docs, update section for qblock to refer to "Users guide" instead of non-existent "Command Reference"
==== CL 8857 ====
@NEW: add externally-updatable worker resources and properties
BUGZID:
==== CL 8847 ====
@CHANGE: upgrade_config tool no longer comments out some of the customized paths in qb.conf
ZD: 4470
==== CL 8846 ====
@FIX: supe and worker RPMs now correctly "require" specific qube-core version (like "6.2-1")
BUGZID: 63644
ZD: 4470
==== CL 8841 ====
@FIX: issue with supervisor threads stalling, waiting for NFS I/O on the "mounted" job logs, when NFS latency is large.
==== CL 8840 ====
@UPDATE: "Use" doc with p-agenda documentation
@UPDATE: also added/updated some qbsub examples
BUGZID: 63636
==== CL 8837 ====
@NEW: add example scripts to demonstrate submission of p-agenda jobs in perl and python
BUGZID: 63636
==== CL 8836 ====
@NEW: adding docs for retryworkdelay (qbsub option)
==== CL 8811 ====
@FIX: fixed worker installer to start the worker service iff the system has not already turned it OFF via chkconfig.
ZD: 4286
==== CL 8798 ====
@NEW: optimization when submitting big groups of jobs via qbsubmit() loaded with callbacks and dependencies
Fixed reported issue where submission performance will degrade linearly proportional to the number of jobs in the queue.
==== CL 8795 ====
@UPDATE: added descriptions of new/missing qb.conf parameters to the qb.conf.template file, which is used to build the default qb.conf.
* added p-agenda params (supe and client)
* added auto-wrangling params (supe)
* added per-user/pgrp subjob limit params (supe)
* added mail setup params (supe)
* added database setup params (supe)
==== CL 8794 ====
@NEW: add p-agenda submission options to qbsub (p_agenda, p_priority, and p_cpus), and updated online help text.
==== CL 8790 ====
@CHANGE: Python API qb.reportjob() now takes a subjob object (dict). It can still take just the status (string).
This should enable the custom jobtype back-end programmer to pass back subjob-level "resultpackage" data to the supe, for example.
==== CL 8783 ====
@NEW: add supervisor_p_agenda_max qb.conf parameter, for the site-admin to control the maximum number of p-agenda any job can have.
==== CL 8782 ====
@NEW: add p_agenda_cpus to enable control of the number of "cpus" used for the p-agenda jobs. Defaults to number of p-agenda items.
@CHANGE: removed code that automatically makes a job to become a p-agenda job when
p_agenda_priority() is set. The "p_agenda" list or the "p_agenda" job flag must be explicitly
set for a job to be a p-agenda job.
==== CL 8781 ====
@CHANGE: if an agenda-based job specifies the p_agenda_priority, then automatically add the p_agenda flag.
@CHANGE: added code to check that the job being submitted is an agenda-based one, before doing the p-agenda magic
==== CL 8775 ====
@UPDATE: doc update w/ "qbhash" and encrypted DB password descriptions
@UPDATE: Added section for qbhash, and updated section for qblogin.
@UPDATE: section for database_password
BUGZID: 63383 63628 39741
==== CL 8769 ====
@NEW: add "qbhash" tool, used to generate/display encrypted passwords
@NEW: add "-password" option to qblogin, to specify password in a command-line option instead of on the stdin
BUGZD: 63383
==== CL 8767 ====
@FIX: install datawarehouse plists on OSX (missing from installer package)
==== CL 8764 ====
@NEW: add p-agenda (p-frames, "p" stands for Priority/Preview/Poster) support, where a select few agenda items of a job can be sent at a higher priority for quicker turn around for previewing purposes.
To use in API: set the "p_agenda" job flag when submitting an agenda-based job.
Optionally attach a list, job['p_agenda'] in python API, to the job on submission to explicitly specify the p-agenda items. If not set explicitly, the system will automatically choose the 1st, last, and middle items to be rendered at a higher priority.
The priority of the p-agenda items may also be specified on submission, by setting the job's p_agenda_priority parameter.
p-agenda job support for the standard submission tools (GUI, qbsub) coming shortly.
@NEW: qb.conf parameters: client_p_agenda_priority, supervisor_default_p_agenda_priority (default 1)
==== CL 8760 ====
@UPDATE: Administration.doc with details about the new worker_boot_diagnostic_retries and worker_boot_diagnostics_retry_interval parameters
BUGZID: 63600
==== CL 8755 ====
@FIX: Added worker_boot_diagnostics_retries and worker_boot_diagnostics_retry_interval
These new configuration parameters tell the worker to automatically retry the boot-time
diagnostic routines for "worker_boot_diagnostics_retries" times, with
"worker_boot_diagnostics_retry_interval" seconds of sleep time inbetween the retries.
By default, they are set to 1 and 30 (seconds) respectively. These values may be
set in the local qb.conf file, or in the qbwrk.conf file.
@FIX: issue where worker will "panic" when proxy settings are set in the remote qbwrk.conf file.
BUGZID: 63600 63422 63407
ZD: 3650 1638 2035
==== CL 8743 ====
@NEW: add qb.frontend package, will serve as base class for constructing jobs for new python jobtypes
==== CL 8727 ====
@CHANGE: database_password is now expected to be encrypted.
Plain text password still works, but if a password has been set up to access the MySQL db, site administrators are
recommended, but not required, that they use "qblogin -display" to generate the encrypted password, and set
database_password in qb.conf to the encrypted string for more security.
BUGZID: 63628
==== CL 8722 ====
@NEW: add optional artificial delay before auto-retry of agenda items via "retrywork"
When a failed frame is automatically retried via "retrywork", an artificial delay may be inserted before the subjob starts processing it.
Requested by customers to work around issues with, for example, application license contentions.
Submission APIs (C++, Perl, Python) and clients (qbsub, QubeGUI) modified to allow specifying "retrywork_delay" when submitting jobs.
==== CL 8717 ====
@FIX: logs written into a "hidden" file, in "log/user/.hst", which grows very large
Actions initiated by the supe (as opposed to a particular user), such as
"starting a subjob on worker", were logged into this hidden ".hst"
file. Fixed it so the file has a special folder/name,
"__QUBE_SYSTEM__/__QUBE_SYSTEM__.hst".
Also modified code so that if the "user" flag was ommitted from the
"supervisor_log_flags", then this user action logging is disabled
altogether.
BUGZID: 62030
==== CL 8713 ====
@FIX: turned off worker debug-level logging that accidentally made it into the 6.2.0 release.
==== CL 8712 ====
@FIX: issue where worker processes will stall when a config field, such as "worker_description" has quotes in them.
==== CL 8704 ====
@FIX: support bash exported function definitions, which are saved as multi-line environment variable values
BUGZID: 63624
ZD: 4100
==== CL 8702 ====
@NEW: add perl 5.12 and 5.14 support for windows x64 and 32-bit.
BUGZID: 63631
==== CL 8695 ====
@FIX: export_environment now works properly with built-in cmd* jobtypes
@FIX: cmd* jobtype backends will run jobs in a non-login shell if
export_environment flag is set on the job, to avoid overriding of
environment variables set by the job's submission environment.
@NEW: QbApi::qbsystem() now optionally takes a boolean to specify
commands to be run in a login shell.
@CHANGE: By default now, QbApi::qbsystem() will run the given command in a
non-login shell.
@NEW: added optional "shell" parameter to QbEnv::setToEnv(user, [shell])
method, so the user environment for a non-default shell can be fetched.
This new method is called from QbWorker::QbUnix.cpp now.
BUGZID: 63625
ZD: 4100
==== CL 8693 ====
@FIX: fix "ERROR 1290 (HY000) at line 31 in file: '.\create_stored_programs.sql'" on new Windows installations
@FIX: fix Windows 5.15-beta version specific SQL syntax error (does not exhibit in later versions of MySQL)
==== CL 8676 ====
@FIX: add code to license check routine to validate hostid against all mac addresses on the host, as opposed to just the primary one.
Note: this involves changes to the base library (utils/QbList, utils/QbServer)
BUGZID: 63621
--
@CHANGE: modify license verification code to only run when the license file had been changed, or a new day has arrived, or on boot.
The code still checks to modification time of the license file everytime that a license access is required but most of the logic is now short-circuited, if no mod was made to the file.
It turns out to be rather tricky to, say, add a "reread" option to "qbadmin" to only read the license on demand, since all supe thread must be told to read the file (for quick access, license data is kept in memory of each thread/proc), and such "broadcast" type of instruction to go out to all threads is not supported at the moment.
The optimization being checked in, however, should significantly reduce the overhead in license-checking nontheless, especially with the new code where each license key's hostid is checked against all mac addresses for validation.
BUGZID: 63622
==== CL 8668 ====
@FIX: fix "/etc/rc.d/init.d/supervisor: line 139: [: /var/spool/qube/user/jburk/jburk.hst: binary operator expected" error message in supervisor startup
BUGZID:63618
==== CL 8662 ====
@UPDATE: update doc with mail_from parameter description.
BUGZID: 63591
==== CL 8654 ====
@FIX: Made the qube-core RPM "obsolete" the "qube" package, to
accomodate the change in RPM package name.
BUGZID: 63611
ZD: 3950
==== CL 8641 ====
@FIX: added more details to default qb.conf template's description of proxy_nice_value, and also included explanation for Windows.
Also corrected the commented-out default proxy_account to "qubeproxy" (from "proxyuser") in the same qb.conf.template.
@DOC: update proxy_nice_value doc accordingly.
==== CL 8610 ====
@FIX: issue where supe will install but not run, due to missing python25.dll file.
==== CL 8606 ====
@FIX: The "Start Time" parameter for SCHTASKS.EXE (/ST option) must be in hh:mm:ss format for earlier versions of Windows (notably winxp 32).
==== CL 8598 ====
@NEW: add sample perl-based submit script that submits jobs with per-work email notification callbacks.
ZD: 3854
==== CL 8550 ====
@FIX: rolling back to linking supe against python 2.5 for its embedded interpreter instead of 2.7 to avoid runtime linkage issues with 2.7
BUGZID:
==== CL 8547 ====
@FIX: added "post" as possible supervisor_language_flags
@FIX: default for supervisor_manifest_flags should be empty
==== CL 8535 ====
@CHANGE:Enhanced shotgun integration in job submission.
==== CL 8503 ====
@CHANGE: grant access to the PFX_*QBTIME* functions to MySQL user "qube_readonly"
@CHANGE: grant the pfx_dw user all rights to the pfx_stats DB
==== CL 8483 ====
@CHANGE: add support for non-cmdrange type backends, don't require qbTokens
==== CL 8482 ====
@NEW: a framework for python-based jobtype backends, as well as a base class for jobtypes which use an application's embedded python terminal prompt
* for use by the Nuke python jobtype (dynamic allocation)
* used by the intra-frame progress in pycmdrange
* can be used for Houdini jobytpe dynamic allocation (not mantra cmd-line renderer though)
==== CL 8480 ====
@FIX: fixed issue with perl API where the system won't respect the "retrywork" specified in jobs processed with a perl-based custom jobtype back-end.
@CHANGE: added some useful logging message to print to supelog when retrywork is being considered
==== CL 8472 ====
@FIX: issue where perl-based custom policy didn't work on some systems.
Embedded perl interpreter had to be initialized much earlier than it was, before the supervisor goes into multi-proc, and
before initializing customizable modules (algorithm, policy) that rely on it.
ZD: 3718
BUGZID: 63603
==== CL 8468 ====
@FIX: (Windows) modified worker memory tracking to store values in KB instead of bytes, to avoid buffer overflow.
ZD: 3308
==== CL 8466 ====
@FIX: (OSX) modified worker memory tracking to store values in KB instead of bytes, to avoid buffer overflow.
ZD: 3308
==== CL 8465 ====
@FIX: (Linux) modified worker memory tracking to store values in KB instead of bytes, to avoid buffer overflow.
BUGZID: 3308
ZD:
==== CL 8458 ====
@NEW: add documentation for the "Get Next n Jobs" jobList pagination, document the new behavior of the User filterCtrl, since it now serves as both a display and request filter.
==== CL 8453 ====
@NEW: implement the ability to parse the logs on the fly to determine intra-chunk progress
@INTERNAL: clean up backend base class and backendUtils in preparation of more wide-spread use
==== CL 8449 ====
@NEW: a pure-python implementation of the cmdrange jobtype; it implements intra-chunk progress by parsing the output stream from the command as it's being written to disk during the course of the job, not after the job completes. Progess calculation works on both single- and multiple-item agenda jobs.
==== CL 8436 ====
@NEW: add doc for per-user/pgrp subjobs limits