Message Logging

The LightWave Client™ CLIENT process can be configured to collect and forward web service call data and statistics to a customer provided collector process. The information delivered to the collector process may include:

• Request / response inter-process messages (IPM).
• HTTP protocol information
• HTTP request / response bodies
• User data
• Request metadata

The collector process may use this information for any purpose, including but not limited to:

• Implement custom request logging
• Accumulate and report processing statistics
• Store message traffic for compliance purposes

Selected information for each request is forwarded to a specified collector target process as an inter-process message. Configuration details for the collector process and the desired request information is provided in a CLIENT process Configuration File.

Collector processes must be Pathway serverclasses. Standalone processes are not supported.

The following sections describe how to configure and use Message Logging.

Hereinafter, the  sending  process refers to the CLIENT process.

The Message Log Request IPM

The sending process forwards request information to the collector process using structures defined in the LWMLDDL DDL file. The request IPM consists of a standard header, a message data map containing offset and length information for the elements in the message data buffer, and a buffer containing the message data itself. The collector process decomposes the message data buffer using the information in the map.  In the event that the Log Request IPM exceeds the maximum inter-process I/O size for the collector process, the request is streamed to the collector in multiple sequential requests.

Request Field Definitions

Header Fields

Message Data Map Field Types

The message map contains a list of fields describing data in the message data area. A map field my be one of the following types:

Message Data Map Fields

Log Request Streaming

The sending process writes the Message Log request buffer to the collector process using the largest inter-process message size available. This may be anywhere from 32K to 2MB, depending on the system configuration. In the event that the Message Log request is larger than the maximum size allowed, it will be sent to the collector in multiple consecutive requests. The collector process should issue a sufficient number of $RECEIVE read operations in order to read the entire request. Pathway process collectors must use Pathway dialogs to process the request stream.

Logging User Data

Application request IPM data may be included in the Message Log request by indicating which fields to include in the API definition type schema. Fields with the msgLogUserData schema property set to true will be included in the USER-DATA collection. Note that:

• Only primitive data types (string, integer, etc.) can be logged. Object and array types cannot be logged.
• All data regardless of IPM type is converted to string for inclusion in the USER-DATA collection.

The following schema fragment illustrates how to include application request data in the log request.

{
  "RequestMessage": {
    "anonymous": false,
    "elements": [
      {
        "name": "requestData",
        "type": "string",
        "size": 32
      },
      {
        "name": "requestId",
        "type": "string",
        "size": 32,
		"msgLogUserData" : true
      }
    ]
  }
}

Logging Request Metadata

Message Logging may be configured to include additional request metadata. The metadata includes information such as sending process information and API request information. The following metadata is available.

Sensitive Data Masking

The Sensitive Data Masking feature allows message fields to be designated as holding sensitive data by adding the attribute "sensitive" : true to the element describing the field. Fields so designated are masked in the Message Log request. The feature can be disabled by starting the sending process with the --disable-sensitive-data-masking option. For more information see Sensitive Data Masking.

Configuring Message Logging

Message Logging is configured using a process configuration file. The configuration is activated by specifying the configuration options in an EDIT file and providing the file location with the --msg-log startup option. Change monitoring for the configuration file may be configured using the --monitor option, for example:

tacl> run CLIENT --msg-log +$vol.subvol.mlogcfg [ --monitor msg-log ]

The configuration is specified using YAML 1.1 format. Multiple collector processes and message selection rules may be configured, for example:

---
msg-log:
  collectors:
    collector-all-content:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-all
      content: all
    collector-http-only:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-http
      content: rq-http,rp-http
  filters:
    successful-request:
	  # Log HTTP info for successful requests
      enabled: true
      match-all:  
        http-status: 200:299
        reply-code: 0
      collectors: collector-http-only
    failed-request:
      # Log all info for failed requests
      enabled: true
      match-any:  
        http-status: 300:600
        reply-code: 1,1
      collectors: collector-all-content
    catch-all:
      # Illustrate catch all rule that sends to both collectors.
      enabled: true
      match-always: true
      collectors: 
		- collector-http-only
        - collector-all-content
...

More information on YAML can be found on the official YAML web site and Wikipedia.

Checking Configuration File Syntax

During the process of creating or updating a configuration file, the configuration may be validated using the LWCCOM CHECK command. 

tacl> run lwccom
LightWave Client COM 1.2.0 - \NODE.$X123
Copyright (c) 2020 NuWave Technologies, Inc. All rights reserved.
LWCCOM 1-> check config mlogcfg
Checking CONFIG file MLOGCFG.
The file contains a valid 'msg-log' configuration.
The CONFIG file is valid.
LWCCOM 2->

Configuration Reference

The basic structure of the configuration file is

---
# A YAML document begins with '---' and end with '...'
# The msg-log configuration begins with a 'msg-log' element.
msg-log:
  collectors:
	# The 'collectors' element contains a collection of named collector definitions.
    # A collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among collectors.
	collector-name: 
      # Collector elements
  filters:
	# The 'filters' element contains a collection of named filter definitions. Filters are checked in order until a match is found. 
    # A filter name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among filters.
	filter-name: # Collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9.
	  # Filter elements
...

Collector Elements

Filter Elements

Note that a filter may contain both the match-all and match-any options. When both are present, they must both match in order for the filter to match (they are logically ANDed).

Filter Rule Elements

Enclosing rule element values in double quotes is strongly recommended.

In the following examples, one or more required filter elements may have been omitted for brevity.

The examples use the match-all element but all options may also be used with match-any.

filters:
  match-when-alias-starts-with-put:
	match-all:
      alias: "put*"
  match-when-alias-starts-with-post-or-get:
    match-all:
      alias:
        - "post*"
        - "get*"

filters:
  match-when-in-200-range:
    match-all:
      http-status: "200:299"
  match-when-in-200-range-but-not-222:
    match-all:
      http-status:
        - "200:299"
        - "!222"

filters:
  match-when-post-or-put:
    match-all:
      method: "post,put"
  match-when-not-get:
    match-all:
      method: "!get"

filters:
  match-when-account-request:
    match-all:
      path: "/api/1/account/*"
  match-when-account-or-customer-request:
    match-all:
      path:
        - "/api/v1/account/*"
        - "/api/v1/customer/*"    

filters:
  match-when-0:
    match-all:
      reply-code: "0"
  match-when-10-20-but-not-15:
    match-all:
      reply-code:
        - "!15"
        - "10:20"

filters:
  match-when-json-reply:
    match-all:
      rp-header:
        content-type: application/json
        header-1: false # match if header-1 is not present
        header-2: true # match if header-2 is present, regardless of value

filters:
  match-when-less-than-2-seconds:
    match-all:
      total-time-exceeds: "!20000000"
  match-when-exceeds-2-seconds:
    match-all:
      total-time-exceeds: "20000000" 

Using Patterns

A pattern may be a literal string, a simple pattern, or a Perl compatible regular expression.

• literal string - A string that exactly matches the element value. The string comparison is case insensitive.
• simple pattern - A string that may contain the question mark ('?') character to match any single character or the asterisk ('*') to match multiple characters. The string comparison is case insensitive.
• regular expression - A string prefixed with "regex:" which contains a Perl compatible regular expression used to match the element value. See Perl Compatible Regular Expressions.

For example, the following rules all match the path "/acme/account/9001"

path: "/acme/account/9001"  # literal pattern
path: "/acme/account/*"     # match any account
path: "regex:(?i)^\/acme\/account\/(9001|9002)$" # match account 9001 or 9002

Using Ranges

A range of values is specified as a comma separated list of single integers, or two integers separated by a colon ':' indicating an inclusive range. For example, the range "1,2,5:10" matches the values 1, 2, and 5-10 inclusive.

Using Negation

Where noted, patterns and ranges may be negated in order to indicate that the rule should match when the pattern or range does NOT match. Negation is indicated by prefixing the pattern or range with the exclamation mark ('!'). Note that because the exclamation mark is a special character in YAML, patterns or ranges specified with negation must be enclosed in double quotes. For example:

path: "!/acme/account/9001"  # match when this is NOT the path.
http-status: !200:299        # match when the HTTP status is NOT in the 200-299 range
rp-header:
  content-type: "!application/json"   # match when the content-type reply header is NOT "application/json"

Sample Collectors

Sample collectors processes in C and COBOL can be found on Github.