TOLERANT Match Release 9.0

2018/12/03

Notices

  • A version v1 was added to the package name of the Java API classes. If you are using the Java API for your integration of Tolerant products, please adjust the package names accordingly.
  • Request ID parameter for bulk requests will now be read as string. Since the request Id is stored in the database, you are advised to deleted the bulk database (TLDATA/bulk) and restart the service, so the new database table structure could be generated.
  • Tolerant Match now uses a newer version of the MongoDB driver (3.6.3) which can be used to connect with MongoDB Atlas.
  • The synchronization protocol for cluster data was extended to enable the support of original data. For the synchronization of original data to work, all cluster installations should have version 8.1 or a later version.
  • The question, whether batch should be installed, is not relevant for the installation and will be skipped from the installer script.
  • The default value of ekPartitionPrefixLength is now set to 10, which should improve the perfomance of wildcard operations.
  • Match C-API has a new function (matchSetConnectionTimeout) to set the connection timeout.
  • Improved and more strict validation of lines in csv-files. Characters after the „enclose character“ are now interpreted as errors, which could lead to possible validation errors in configured normalizer- or synonym-files.
  • Updated English Manuals

New Product Features

Configuration and Administration GUI

  • GUI fields for all boolean values are now shown as drop boxes.
  • The abort button for batch process in the Admin GUI will stay activated even if the stopPort is not set in the configuration. Using the abort function without setting the stopPort results in an error message.
  • Overwriting and adding new fields in the GUI input view now supports auto-mapping of the added fields.
  • License file status and content can be viewed in the Admin GUI.

Batch

  • The question, whether batch should be installed, is not relevant for the installation and will be skipped from the installer script.

Service

  • A version v1 was added to the package name of the Java API classes. After this change, the usage of different API versions of different Tolerant products is possible.
  • Reconfiguring Match service checks now whether changes in phonetic usage are admissible.
  • The response of delete request with wildcard contains a boolean field ‚resultOverflow‘, determining whether more keys were matched but not deleted.
  • The database table for the request logging can now be automatically created using the number of columns configured in the runtime attribute reqLogDbNrCols.
  • The synchronization protocol for cluster data was extended to enable the support of original data.
  • Start, Stop and reconfigure of a project does not require setting the service in maintenance mode.
  • Following changes were made to simplify the usage of the java API:
    • In case of an exception, the exception will be attached as cause to the APIException
    • API retries can be configured using the java property: com.tolerantsoftware.api.maxRetries
    • The check for stale connections and the period of this check can be configured using the java property: com.tolerantsoftware.api.staleCheck
    • The error message: Response contains failure was extended to include the return code from the service response and the causing error message.
  • Request-id can now be set using a new constructor for the connection object.
  • The default value of ekPartitionPrefixLength is now set to 10, which should improve the perfomance of wildcard operations.
  • Storing original data in SQL databases was optimized to use fewer statements.
  • The size of Match index is stored in the counter with the Id Kernel..indexSize which can be read either by using jconsol (JMX beans) or using the Admin request: matchAdmin –local –cmd „get metric Kernel..indexSize“.
  • Bulk search requests are now processed simultaneously to better utilize cpu’s available cores.
  • The JVM trust store is now created under TLCONFIG/truststore/cacerts
  • Installer creates the keystore for jetty in the pkcs#12-format.

General

  • A micro version was added to all Tolerant products. The micro version is set to the patch number.
  • The configuration converter now ignores match.InternalData and match.OriginalData when converting a service configuration to a batch configuration.
  • Integration of a new web application to enable the configuration and execution of batch runs without extended knowledge about TOLERANT Match.
  • Tolerant Match now uses a newer version of the MongoDB driver (3.6.3) which can be used to connect with MongoDB Atlas.
  • The bundled JRE was upgraded to the latest Java security patch (Java 8 Update 191)
  • License tool returns a description text for the available modules.
  • The command REMOVE_LOCK of the CLI Tool matchAdjustPagingDatabase now removes all lock entries from the database.
  • Following measures were implemented to optimize the performance of wildcard operations:
    • The memory footprint of wildcard delete operations was reduced
    • The character ‚-‚ will be recongnized as part of the key prefix
  • Updated English Manuals
  • Tolerant Match under Solaris Sparc now uses a different memory allocator, which has shown a better results in our performance tests.
  • Signal handling was improved to protocol the code address of the problem in the stack trace.

API

  • The C-API example was extended so that it uses per default the local service URL entered during the installation.
  • Match C-API has a new function (matchSetConnectionTimeout) to set the connection timeout. The timeout is entered in milliseconds. The default connection timeout is 10000 milliseconds.

Fixed Bugs

Configuration and Administration GUI

  • TPR-4890: Environment variable substituation in the Admin GUI can now be used for all integer fields.
  • TPR-5149: Reflection problem is fixed, which has resulted in a misleading exception in the GUI trace file.
  • TPR-5167: The security right translitrules.check was missing from the default ADMIN role.
  • TPR-5169: The automatic codeset detection will no be triggered when a new synonym file is selected.
  • TPR-5244: The validation of the codesets in the Admin GUI was corrected, so that codesets with ‚_‘ can be selected.
  • TPR-5245: A new check to prevent errors from occurring when adding service input fields and setting length handling to none
  • TPR-5249: Codeset detection for input files will not be trigged automatically if a codeset was already selected.
  • TPR-5410: The Admin GUI now reports an error when the same character is used as separator, escape and quoting character.
  • TPR-5479: File perview now shows fields of type date (D) correctly.
  • TPR-5884: Extra check to prevent request redirect if a link points to a non-existing file.
  • TPR-6281: The Admin GUI resolve paths relative to Tolerant paths (TLCONFIG, TLDATA ..ect)
  • TPR-6361: Adding new rules after chaning a field type from STOREONLY to an index-relevant type now doesn’t result in incorrect configuration file.

Batch

  • TPR-4700: Match batch now shows the right number of lines read.
  • TPR-4808: Batch summary no longer generates invalid json protocols.
  • TPR-5557: Internal structure used to store the results of different rules is now better initialized to avoid false-positive hits.
  • TPR-5805: For batch process a default logger will be added for missing log elements.
  • TPR-6265: Database errors occurring during a Batch process are now written in the log files and shown in the statistic output.

Service

  • TPR-4606: A change was made to the service start script so that a root user cannot start the service.
  • TPR-4699: Request ID parameter for bulk requests will now be read as string.
  • TPR-4724: Paths in the configuration are now resolved relative to Tolerant paths environment variables for both service and initial-load.
  • TPR-5003: Deprecated attribute ‚autoLoad‘ was used in cluster implementation
  • TPR-5089: A service inactive project can now be reactivated and started from the administrative GUI.
  • TPR-5143: resultMode (5) can be now used even if the original data is not configured.
  • TPR-5272: The stop script of Match service now terminates with RC=0 even if the paging database was not configured.
  • TPR-5295: Internal check was added to insure closing database connection proxy of type SQLite after the initial load process.
  • TPR-5403: It is now allowed to left a service input field of type ‚D‘ empty.
  • TPR-5419: The state of the local cluster member will be evaluted when configuring the service runtime and the cluster projects.
  • TPR-5480: The results of matchInspectPagingDatabase for the command BINARY_OBJECT and objectId ALL are now interpreted correctly.
  • TPR-5493: The cluster synchronization protocol now checks if a save point is being created on a remote node to avoid synchronizing wrong or incomplete data.
  • TPR-5496: A debug message stating the number of values in match fields was outputtted only for the first fields group.
  • TPR-5514: Tolerant Service now stops all its cron jobs, so that it can be deployed in tomcat without thread leaking problems.
  • TPR-5545: An small refactoring was carried out to minimize the chance of exceptions in the finalize phase of the initial load process.
  • TPR-5563: When starting Match service, only the database used from the service will be checked. Database connection used e.g. for the initial load input, will be ignored in the service.
  • TPR-5592: Default values were not taken into consideration when building the json response.<
  • TPR-5619: It is now possible to use X-HTTP-Method-Override to send all parameters in the request body
  • TPR-5624: The requirement of msvcr100.dll being installed into the os for registering the product as a windows service has been dropped as we have included this into the installation.
  • TPR-5657: Match project didn’t load its index if it was stopped and then started using admin commands.
  • TPR-5665: Mismatching service configurations in cluster don’t lead to exceptions being logged in the trace file.
  • TPR-5703: Extra check was implemented to cover the case, that an admin command could be called for an unknown project.
  • TPR-6005: Number of records in case of search by key with wildcards will now be calculated correctly to avoid unneccessary exceptions.
  • TPR-6022: Handling of unexpected cluster errors was improved.
  • TPR-6224: Statistics output after the initial-load is calculated correctly regardless of the configured thread count.
  • TPR-6251: Temporary jetty files are execluded from the installation consistency check.

General

  • TPR-4921: Upgrade process now doesn’t overwrite the hostname in tolerantEnv that was entered during the installation.
  • TPR-4953: MULTISTRING field was optimized, so that the score calculation now takes into account the possibility of multiple occurrences of the same or similar token in the field value.
  • TPR-5085: Silent installation now checks whether the backup-path can be created, otherwise the installation breaks with suitable error message.
  • TPR-5140: The uniqueness of profile name and rule name will now be validated correctly.
  • TPR-5154: The rolloverMode attribute of the log-element now overrides the value set in the logging-element.
  • TPR-5212: Useful messages logged from different CLI tools were not outputted to the console
  • TPR-5224: Frequency metric is now better dimensioned to get as few false positives as possible.
  • TPR-5281: A problem was fixed, which could lead to inconsistent data being stored in the original database for transliterated and/or normalized match fields.
  • TPR-5438: Synonym substitution and transliterations can now be used for String output fields.
  • TPR-5448: Extra check was implemented to prevent empty fragments in an internal mapping structure in Match kernel from causing a memory access violation error.
  • TPR-5500: Error handling was missing in few method calls in Match kernel.
  • TPR-5593: Pattern for IO fields of type Date (D) are now mandatory.
  • TPR-5596: Improved and more strict validation of lines in csv-files. Characters after the „enclose character“ are now interpreted as errors, which could lead to possible validation errors in configured normalizer- or synonym-files.
  • TPR-5639: Exception that could occur during connection rollback will now be protocolled in the trace file.
  • TPR-5816: Potentially stalled threads will now be woken up after error in page-in operation
  • TPR-5902: The following metrics are now project-specific:
    • Pipeline.Time.
    • Pipeline.RemainingQueueCapacity.
    • Pipeline.ActivityMessages.
    • Pipeline.FailureMessages.
    • Pipeline.WarningMessages.
  • TPR-5972: Adjusting the calculated page size now takes place only if the actual change in the page was successful.
  • TPR-6083: Fixed Memory Leak after Error
  • TPR-6241: The installer now logs only the components that were installed.
  • TPR-6248: Match fields of type BITARRAY can now return their call trace information (with resultMode=5)
  • TPR-6264: Match configuration check reports mismatching synonym usage between service and the initial-load as warnings.
  • TPR-6267: Writing a full save-point now deletes all incremental save points written after the last full save-point regardless of the configured value for nrFullsavepoints.
  • TPR-6350: lock entry will now be generated with the correct port-number for the running service.

API

  • TPR-5540: Code changes were made to fix a memory leak, that could occur while parsing the json response from service.

Known Bugs

  • Currently no known bugs.