Page tree
Skip to end of metadata
Go to start of metadata

The LightWave Server processes (SERVER & SWORKER) can be configured to collect and forward web service call data and statistics to a customer provided collector process. The information delivered to the collector process may include:

  • Request / response inter-process messages (IPM).
  • HTTP protocol information
  • HTTP request / response bodies
  • User data
  • Request metadata

The collector process may use this information for any purpose, including but not limited to:

  • Implement custom request logging
  • Accumulate and report processing statistics
  • Store message traffic for compliance purposes

Selected information for each request is sent to a specified collector target process as an inter-process message. Configuration details for the collector process, and request selection criteria in a Message Logging Configuration File.

Collector processes must be Pathway serverclasses. Standalone processes are not supported.

The following sections describe how to configure and use Message Logging.


Hereinafter, the sending process refers to either the SERVER or SWORKER process, depending on which process is handling the client request.

The Message Log Request IPM

The sending process sends request information to the collector process using structures defined in the LWMLDDL DDL file. The request IPM consists of a standard header, a message data map containing offset and length information for the elements in the message data buffer, and a buffer containing the message data itself. The collector process decomposes the message data buffer using the information in the map.  In the event that the Log Request IPM exceeds the maximum inter-process I/O size for the collector process, the request is streamed to the collector in multiple sequential requests.

Request Field Definitions

Header Fields

Field NameDescription
RQ-CODEThe standard request code for the Message Log request. This field will have the value LW-ML-RQ-REQUEST
RQ-VERSIONThe Message Log request version.
RQ-LENGTHThe length in bytes of the entire Message Log request.
RQ-TS-UNIQUEA 128-bit timestamp returned from the TS_UNIQUE_CREATE_ system procedure call that is unique to the system

it is generated on and any system in the same EXPAND network

Message Data Map Field Types

The message map contains a list of fields describing data in the message data area. A map field my be one of the following types:

Data TypeDescription
BufferThis data type consists of a fixed length buffer containing binary data. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the buffer offset from the start of the request IPM and the -LEN element indicates the length of the buffer in bytes. Fixed length buffer fields are guaranteed to be 4 byte aligned with the start of the IPM.
StringThis data type consists of a NULL terminated character string. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the offset of the string from the start of the request IPM and the -LEN element indicates the length of the string including the NULL terminator. String fields are byte aligned with the start of the IPM.
CollectionThis data type consists of a collection of name / value pairs consisting of String data. Map fields for this data type include <field-name>-OFFSET, <field-name>-LEN, and <field-name>-COUNT. The -OFFSET element indicates the offset of the name / value collection from the start of the request IPM and the -LEN element indicates the length of the collection data including all NULL terminators. The -COUNT element indicates the number of name / value pairs in the collection.
IntegerA 32 or 64 bit integer value.
TimestampA 64 bit JULIANTIMESTAMP
IntervalA 32 bit unsigned integer containing a time interval in microseconds.

Message Data Map Fields

Field NameData TypeDescription
START-TIMETimestampThe request start time.
END-TIMETimestampThe request end time.
TOTAL-TIMEIntervalTotal request time in microseconds
CONNECT-TIMEIntervalAlways 0.
CONNECT-HS-TIMEIntervalThe TLS handshake time.
REQUEST-TIMEIntervalThe request send time.
RESPONSE-TIMEIntervalThe response time which consists of the service processing time and the response network transfer time.
SERIALIZE-TIMEIntervalRequest serialization time.
DESERIALIZE-TIMEIntervalThe response deserialization time.
SERVER-IO-TIMEIntervalThe process or serverclass I/O time.
RQ-IPMBufferThe application request IPM.
RQ-REQUEST-LINEStringThe HTTP request line.
RQ-METHODStringThe request method, e.g, "POST".
RQ-URIStringThe request URI.
RQ-PARAMSCollectionThe request query string parameters.
RQ-HTTP-VERStringThe HTTP protocol version, e.g. "1.1".
RQ-HEADERSCollectionThe HTTP request headers.
RQ-BODYBufferThe HTTP request body
RP-IPMBufferThe application reply IPM
RP-STATUSIntegerThe HTTP response status.
RP-STATUS-LINEStringThe HTTP response line.
RP-HEADERSCollection

The HTTP response headers.

RP-BODYBufferThe HTTP response body
USER-DATACollectionAny fields designated in the request IPM as User Data. See Logging Request User Data.
METADATACollectionAny fields selected from the available request metadata. See Logging Request Metadata.

Log Request Streaming

The sending process writes the Message Log request buffer to the collector process using the largest inter-process message size available. This may be anywhere from 32K to 2MB, depending on the system configuration. In the event that the Message Log request is larger than the maximum size allowed, it will be sent to the collector in multiple consecutive requests within a Pathway Dialog. The collector process should issue a sufficient number of $RECEIVE read operations in order to read the entire request. 

Logging User Data

Application request IPM data may be included in the Message Log request by indicating which fields to include in the API definition type schema. Fields with the msgLogUserData schema property set to true will be included in the USER-DATA collection. Note that:

  • Only primitive data types (string, integer, etc.) can be logged. Object and array types cannot be logged.
  • All data regardless of IPM type is converted to string for inclusion in the USER-DATA collection.

The following schema fragment illustrates how to include application request data in the log request.

<type name="RequestMessage>
  <element name="requestData" type="string" size="32" />
  <element name="requestId" type="string" size="32" msgLogUserData="1" />
</type>

Logging Request Metadata

Message Logging may be configured to include additional request metadata. The metadata includes information such as sending process information and API request information. The following metadata is available.


NameValue
process-nameThe process name of the sending process.
process-program-fileThe sending process program file name.
process-ancestor-nameThe process name of the sending process ancestor.
process-ancestor-programThe program file name of the sending process ancestor.
api-nameThe API name.
api-file-nameThe API file name.
api-methodThe API operation method.
api-pathThe API operation path.
api-base-urlThe API base url.
api-aliasThe API operation alias
tls-versionThe TLS protocol version.
tls-cipherThe TLS cipher
net-local-ipThe local IP address of the socket connection.
net-local-portThe local port of the socket connection.
net-remote-ipThe remote IP address of the socket connection.
net-remote-portThe remote port of the socket connection.
net-processThe TCP/IP process in use.
config-*An item for each of the configuration options for the collector, e.g, config-collector-name, config-collector-pathmon, etc. Also the name of the filter selected, as config-filter-name.

Sensitive Data Masking

The Sensitive Data Masking feature allows message fields to be designated as holding sensitive data by adding the attribute "sensitive" : true to the element describing the field. Fields so designated are masked in the Message Log request. The feature can be disable by starting the SERVER process with the --disable-sensitive-data-masking option. For more information see Sensitive Data Masking.

Configuring Message Logging

Message Logging is configured using a process configuration file. The configuration is activated by specifying the configuration options in an EDIT file and providing the file location with the --msg-log startup option. Change monitoring for the configuration file may be configured using the --monitor option, for example:

tacl> run SERVER --msg-log +$vol.subvol.mlogcfg [ --monitor msg-log ]

The configuration is specified using YAML 1.1 format. Multiple collector processes and message selection rules may be configured, for example:

---
msg-log:
  collectors:
    collector-all-content:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-all
      content: all
    collector-http-only:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-http
      content: rq-http,rp-http
  filters:
    successful-request:
	  # Log HTTP info for successful requests
      enabled: true
      match-all:  
        http-status: 200:299
        reply-code: 0
      collectors: collector-http-only
    failed-request:
      # Log all info for failed requests
      enabled: true
      match-any:  
        http-status: 300:600
        reply-code: 1,1
      collectors: collector-all-content
    catch-all:
      # Illustrate catch all rule that sends to both collectors.
      enabled: true
      match-always: true
      collectors: 
	    - collector-http-only
        - collector-all-content
...

More information on YAML can be found on the official YAML web site and Wikipedia.

Checking Configuration File Syntax

During the process of creating or updating a configuration file, the configuration may be validated using the LWSCOM CHECK command. 

tacl> run lwScom
LightWave Server COM 1.1.0 - \NODE.$X123
Copyright (c) 2020 NuWave Technologies, Inc. All rights reserved.
LWSCOM 1-> check config mlogcfg
Checking CONFIG file MLOGCFG.
The file contains a valid 'msg-log' configuration.
The CONFIG file is valid.
LWSCOM 2->


Configuration Reference

The basic structure of the configuration file is

---
# A YAML document begins with '---' and end with '...'
# The msg-log configuration begins with a 'msg-log' element.
msg-log:
  collectors:
	# The 'collectors' element contains a collection of named collector definitions.
    # A collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among collectors.
	collector-name: 
      # Collector elements
  filters:
	# The 'filters' element contains a collection of named filter definitions. Filters are checked in order until a match is found. 
    # A filter name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among filters.
	filter-name: # Collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9.
	  # Filter elements
...

Collector Elements

OptionDescription
enabledA boolean such as "true" or "false", indicating if the collector is enabled. Disabled collectors may be referenced by filters but messages will not be sent. Required.
type

The type of collector, which must be "serverclass". If omitted, the default value is "serverclass".

pathmonThe process name of the serverclass PATHMON process, which may be specified as a DEFINE. Required if type is "serverclass".
serverclassThe serverclass name. Required if type is "serverclass".
content

A comma separated list of one or more of the following content selectors:

  • rq-ipm - The request IPM
  • rq-http - The request HTTP protocol information (request line, headers, etc.)
  • rq-body - The request body
  • rp-ipm - The reply IPM
  • rp-http - The reply HTTP protocol information (request line, headers, etc).
  • rp-body- The reply body
  • user - The user data.
  • all - all possible content.

The default value is 'all'. May be negated, see Using Negation.

io-retry-interval

The number of seconds to wait before retrying a failed I/O to the collector target. The value must be in the range 5 - 30 seconds.

io-retry-limitThe number of times to retry a failed I/O to the collector target. The value must be in the range 1 - 10 retries.
io-max-lengthThe maximum size of any I/O to the collector target. The sending process always tries to use the maximum I/O size possible for the target collector. This parameter may be used to reduce the I/O size to any limit imposed by the target process. Note that this parameter may also be used to artificially limit the I/O size of a Pathsend serverclass collector, in order to test the servers ability to handle streaming requests using Pathway dialogs.
request-codeThe IPM request code to use for the message logging request. By default, the the value of LW-ML-RQ-MSG-LOG as the IPM request code. This option may be used to override that value and allow each collector configuration to use a unique request code.

Filter Elements

OptionDescription
enabledA boolean such as "true" or "false", indicating if the filter is enabled. Disabled filters are ignored.
continue-on-match

A boolean such as "true" or "false", indicating if filter processing should continue if the current filter matches. By default, when a filter matches, no further filters are checked. If omitted, the default value is "false".

match-allThe match-all element contains a collection of filter rules. In order for the filter to match, all rules must match. Optional.
match-anyThe match-any element contains a collection of filter rules. In order for the filter to match, any of the rules must match. Optional.
match-always

A boolean such as "true" or "false", indicating that the filter always matches. Optional. If present, the filter matches and any match-all or match-any collections are ignored. This option can be used to create a "catch all" filter.

collectorsA single collector name or a sequence of collector names. If the filter rules match, a logging request will be sent to each collector.

Note that a filter may contain both the match-all and match-any options. When both are present, they must both match in order for the filter to match (they are logically ANDed).

Filter Rule Elements

Enclosing rule element values in double quotes is strongly recommended.

In the following examples, one or more required filter elements may have been omitted for brevity.


OptionDescription
alias

A single pattern or a sequence of patterns used to select or ignore an operation alias. See Using Patterns. May be negated, see Using Negation.


Example
filters:
  match-when-alias-starts-with-put:
	match-all:
      alias: "put*"
  match-when-alias-starts-with-post-or-get:
    match-all:
      alias:
        - "post*"
        - "get*"
http-status

A single range or a sequence of ranges used to select or ignore http-status values. See Using Ranges. May be negated, see Using Negation.


Example
filters:
  match-when-in-200-range:
    match-all:
      http-status: "200:299"
  match-when-in-200-range-but-not-222:
    match-all:
      http-status:
        - "200:299"
        - "!222"
method

A comma separated list or pattern used to select or ignore HTTP methods. See Using Patterns. May be negated, see Using Negation.


Example
filters:
  match-when-post-or-put:
    match-all:
      method: "post,put"
  match-when-not-get:
    match-all:
      method: "!get"
path

A single pattern or sequence of patterns used to select or ignore an operation path. See Using Patterns. May be negated, see Using Negation. Note that the path used for matching is the actual path used to invoke the operation, not the path specified in the API definition.

Example
filters:
  match-when-account-request:
    match-all:
      path: "/api/1/account/*"
  match-when-account-or-customer-request:
    match-all:
      path:
        - "/api/v1/account/*"
        - "/api/v1/customer/*"    
reply-code

A single range or sequence of ranges used to select or ignore reply-code values. See Using Ranges. May be negated, see Using Negation.


Example
filters:
  match-when-0:
    match-all:
      reply-code: "0"
  match-when-10-20-but-not-15:
    match-all:
      reply-code:
        - "!15"
        - "10:20"
rq-header / rp-header

A collection of HTTP request or response header name value pairs. The name matches the HTTP header name. The value may be a pattern. See Using Patterns. May be negated, see Using Negation.

Example
filters:
  match-when-json-reply:
    match-all:
      rp-header:
        content-type: application/json

Using Patterns

A pattern may be a literal string, a simple pattern, or a Perl compatible regular expression.

  • literal string - A string that exactly matches the element value. The string comparison is case insensitive.
  • simple pattern - A string that may contain the question mark ('?') character to match any single character or the asterisk ('*') to match multiple characters. The string comparison is case insensitive.
  • regular expression - A string prefixed with "regex:" which contains a Perl compatible regular expression used to match the element value. See Perl Compatible Regular Expressions.

For example, the following rules all match the path "/acme/account/9001"

path: "/acme/account/9001"  # literal pattern
path: "/acme/account/*"     # match any account
path: "regex:(?i)^\/acme\/account\/(9001|9002)$" # match account 9001 or 9002

Using Ranges

A range of values is specified as a comma separated list of single integers, or two integers separated by a colon ':' indicating an inclusive range. For example, the range "1,2,5:10" matches the values 1, 2, and 5-10 inclusive.

Using Negation

Where noted, patterns and ranges may be negated in order to indicate that the rule should match when the pattern or range does NOT match. Negation is indicated by prefixing the pattern or range with the exclamation mark ('!'). Note that because the exclamation mark is a special character in YAML, patterns or ranges specified with negation must be enclosed in double quotes. For example:

path: "!/acme/account/9001"  # match when this is NOT the path.
http-status: !200:299        # match when the HTTP status is NOT in the 200-299 range
rp-header:
  content-type: "!application/json"   # match when the content-type reply header is NOT "application/json"

Sample Collectors

Sample collectors processes in C and COBOL can be found on Github.


  • No labels


LightWave Server 1.1.0-beta.1 - Pre-release, unsupported software