The LightWave Server™ processes (SERVER & SWORKER) can be configured to collect and forward web service call data and statistics to a customer provided collector process. The information delivered to the collector process may include:
- Request / response inter-process messages (IPM).
- HTTP protocol information
- HTTP request / response bodies
- User data
- Request metadata
The collector process may use this information for any purpose, including but not limited to:
- Implement custom request logging
- Accumulate and report processing statistics
- Store message traffic for compliance purposes
Selected information for each request is sent to a specified collector target process as an inter-process message. Configuration details for the collector process, and request selection criteria in a Message Logging Configuration File.
Collector processes must be Pathway serverclasses. Standalone processes are not supported.
The following sections describe how to configure and use Message Logging.
Hereinafter, the sending process refers to either the SERVER or SWORKER process, depending on which process is handling the client request.
The Message Log Request IPM
The sending process sends request information to the collector process using structures defined in the LWMLDDL DDL file. The request IPM consists of a standard header, a message data map containing offset and length information for the elements in the message data buffer, and a buffer containing the message data itself. The collector process decomposes the message data buffer using the information in the map. In the event that the Log Request IPM exceeds the maximum inter-process I/O size for the collector process, the request is streamed to the collector in multiple sequential requests.
Request Field Definitions
|The standard request code for the Message Log request. This field will have the value LW-ML-RQ-REQUEST
|The Message Log request version.
|The length in bytes of the entire Message Log request.
|A 128-bit timestamp returned from the TS_UNIQUE_CREATE_ system procedure call that is unique to the system
it is generated on and any system in the same EXPAND network
Message Data Map Field Types
The message map contains a list of fields describing data in the message data area. A map field my be one of the following types:
|This data type consists of a fixed length buffer containing binary data. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the buffer offset from the start of the request IPM and the -LEN element indicates the length of the buffer in bytes. Fixed length buffer fields are guaranteed to be 4 byte aligned with the start of the IPM.
|This data type consists of a NULL terminated character string. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the offset of the string from the start of the request IPM and the -LEN element indicates the length of the string including the NULL terminator. String fields are byte aligned with the start of the IPM.
|This data type consists of a collection of name / value pairs consisting of String data. Map fields for this data type include <field-name>-OFFSET, <field-name>-LEN, and <field-name>-COUNT. The -OFFSET element indicates the offset of the name / value collection from the start of the request IPM and the -LEN element indicates the length of the collection data including all NULL terminators. The -COUNT element indicates the number of name / value pairs in the collection.
|A 32 or 64 bit integer value.
|A 64 bit JULIANTIMESTAMP
|A 32 bit unsigned integer containing a time interval in microseconds.
Message Data Map Fields
|The request start time.
|The request end time.
|Total request time in microseconds
|The TLS handshake time.
|The request send time.
|The response time which consists of the service processing time and the response network transfer time.
|Request serialization time.
|The response deserialization time.
|The process or serverclass I/O time.
|The application request IPM.
|The HTTP request line.
|The request method, e.g, "POST".
|The request URI.
|The request query string parameters.
|The HTTP protocol version, e.g. "1.1".
|The HTTP request headers.
|The HTTP request body
|The application reply IPM
|The HTTP response status.
|The HTTP response line.
The HTTP response headers.
|The HTTP response body
|Any fields designated in the request IPM as User Data. See Logging Request User Data.
|Any fields selected from the available request metadata. See Logging Request Metadata.
Log Request Streaming
The sending process writes the Message Log request buffer to the collector process using the largest inter-process message size available. This may be anywhere from 32K to 2MB, depending on the system configuration. In the event that the Message Log request is larger than the maximum size allowed, it will be sent to the collector in multiple consecutive requests within a Pathway Dialog. The collector process should issue a sufficient number of $RECEIVE read operations in order to read the entire request.
Logging User Data
Application request IPM data may be included in the Message Log request by indicating which fields to include in the API definition type schema. Fields with the msgLogUserData schema property set to true will be included in the USER-DATA collection. Note that:
- Only primitive data types (string, integer, etc.) can be logged. Object and array types cannot be logged.
- All data regardless of IPM type is converted to string for inclusion in the USER-DATA collection.
The following schema fragment illustrates how to include application request data in the log request.
<element name="requestData" type="string" size="32" />
<element name="requestId" type="string" size="32" msgLogUserData="1" />
Logging Request Metadata
Message Logging may be configured to include additional request metadata. The metadata includes information such as sending process information and API request information. The following metadata is available.
|The API name.
|The API file name.
|The API operation method.
|The API operation path.
|The API base url.
|The API operation alias
|An item for each of the configuration options for the collector, e.g, config-collector-name, config-collector-pathmon, etc. Also the name of the filter selected, as config-filter-name.
|The local IP address of the socket connection.
|The local port of the socket connection.
|The remote IP address of the socket connection.
|The remote port of the socket connection.
|The TCP/IP process in use.
|The process name of the sending process.
|The sending process program file name.
|The process name of the sending process ancestor.
|The program file name of the sending process ancestor.
|The TLS protocol version.
|The TLS cipher
Sensitive Data Masking
The Sensitive Data Masking feature allows message fields to be designated as holding sensitive data by adding the attribute "sensitive" : true to the element describing the field. Fields so designated are masked in the Message Log request. The feature can be disable by starting the SERVER process with the --disable-sensitive-data-masking option. For more information see Sensitive Data Masking.
Configuring Message Logging
Message Logging is configured using a process configuration file. The configuration is activated by specifying the configuration options in an EDIT file and providing the file location with the --msg-log startup option. Change monitoring for the configuration file may be configured using the --monitor option, for example:
tacl> run SERVER --msg-log +$vol.subvol.mlogcfg [ --monitor msg-log ]
The configuration is specified using YAML 1.1 format. Multiple collector processes and message selection rules may be configured, for example:
# Log HTTP info for successful requests
# Log all info for failed requests
# Illustrate catch all rule that sends to both collectors.
Checking Configuration File Syntax
During the process of creating or updating a configuration file, the configuration may be validated using the LWSCOM CHECK command.
tacl> run lwscom
LightWave Server COM 1.1.0 - \NODE.$X123
Copyright (c) 2020 NuWave Technologies, Inc. All rights reserved.
LWSCOM 1-> check config mlogcfg
Checking CONFIG file MLOGCFG.
The file contains a valid 'msg-log' configuration.
The CONFIG file is valid.
The basic structure of the configuration file is
# A YAML document begins with '---' and end with '...'
# The msg-log configuration begins with a 'msg-log' element.
# The 'collectors' element contains a collection of named collector definitions.
# A collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among collectors.
# Collector elements
# The 'filters' element contains a collection of named filter definitions. Filters are checked in order until a match is found.
# A filter name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among filters.
filter-name: # Collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9.
# Filter elements
|A boolean such as "true" or "false", indicating if the collector is enabled. Disabled collectors may be referenced by filters but messages will not be sent. Required.
The type of collector, which must be "serverclass". If omitted, the default value is "serverclass".
|The process name of the serverclass PATHMON process, which may be specified as a DEFINE. Required if type is "serverclass".
|The serverclass name. Required if type is "serverclass".
A comma separated list of one or more of the following content selectors:
The default value is 'all'. May be negated, see 51151454.
The number of seconds to wait before retrying a failed I/O to the collector target. The value must be in the range 5 - 30 seconds.
|The number of times to retry a failed I/O to the collector target. The value must be in the range 1 - 10 retries.
|The maximum size of any I/O to the collector target. The sending process always tries to use the maximum I/O size possible for the target collector. This parameter may be used to reduce the I/O size to any limit imposed by the target process. Note that this parameter may also be used to artificially limit the I/O size of a Pathsend serverclass collector, in order to test the servers ability to handle streaming requests using Pathway dialogs.
|The IPM request code to use for the message logging request. By default, the the value of LW-ML-RQ-MSG-LOG as the IPM request code. This option may be used to override that value and allow each collector configuration to use a unique request code.
|A boolean such as "true" or "false", indicating if the filter is enabled. Disabled filters are ignored.
A boolean such as "true" or "false", indicating if filter processing should continue if the current filter matches. By default, when a filter matches, no further filters are checked. If omitted, the default value is "false".
|The match-all element contains a collection of filter rules. In order for the filter to match, all rules must match. Optional.
|The match-any element contains a collection of filter rules. In order for the filter to match, any of the rules must match. Optional.
A boolean such as "true" or "false", indicating that the filter always matches. Optional. If present, the filter matches and any match-all or match-any collections are ignored. This option can be used to create a "catch all" filter.
|A single collector name or a sequence of collector names. If the filter rules match, a logging request will be sent to each collector.
Note that a filter may contain both the match-all and match-any options. When both are present, they must both match in order for the filter to match (they are logically ANDed).
Filter Rule Elements
Enclosing rule element values in double quotes is strongly recommended.
In the following examples, one or more required filter elements may have been omitted for brevity.
A single pattern or sequence of patterns used to select or ignore an operation path. See 51151454. May be negated, see 51151454. Note that the path used for matching is the actual path used to invoke the operation, not the path specified in the API definition.
|rq-header / rp-header
Boolean true of false may also be specified to match when the header is or is not present.
A pattern may be a literal string, a simple pattern, or a Perl compatible regular expression.
- literal string - A string that exactly matches the element value. The string comparison is case insensitive.
- simple pattern - A string that may contain the question mark ('?') character to match any single character or the asterisk ('*') to match multiple characters. The string comparison is case insensitive.
- regular expression - A string prefixed with "regex:" which contains a Perl compatible regular expression used to match the element value. See Perl Compatible Regular Expressions.
For example, the following rules all match the path "/acme/account/9001"
path: "/acme/account/9001" # literal pattern
path: "/acme/account/*" # match any account
path: "regex:(?i)^\/acme\/account\/(9001|9002)$" # match account 9001 or 9002
A range of values is specified as a comma separated list of single integers, or two integers separated by a colon ':' indicating an inclusive range. For example, the range "1,2,5:10" matches the values 1, 2, and 5-10 inclusive.
Where noted, patterns and ranges may be negated in order to indicate that the rule should match when the pattern or range does NOT match. Negation is indicated by prefixing the pattern or range with the exclamation mark ('!'). Note that because the exclamation mark is a special character in YAML, patterns or ranges specified with negation must be enclosed in double quotes. For example:
path: "!/acme/account/9001" # match when this is NOT the path.
http-status: !200:299 # match when the HTTP status is NOT in the 200-299 range
content-type: "!application/json" # match when the content-type reply header is NOT "application/json"
Sample collectors processes in C and COBOL can be found on Github.