Skip to main content
Skip table of contents

Message Logging

The LightWave Server processes (SERVER & SWORKER) can be configured to collect and forward web service call data and statistics to a customer provided collector process. The information delivered to the collector process may include:

  • Request / response inter-process messages (IPM).

  • HTTP protocol information

  • HTTP request / response bodies

  • User data

  • Request metadata

The collector process may use this information for any purpose, including but not limited to:

  • Implement custom request logging

  • Accumulate and report processing statistics

  • Store message traffic for compliance purposes

Selected information for each request is sent to a specified collector target process as an inter-process message. Configuration details for the collector process, and request selection criteria in a Message Logging Configuration File.

Collector processes must be Pathway serverclasses. Standalone processes are not supported.

The following sections describe how to configure and use Message Logging.

Hereinafter, the sending  process refers to either the SERVER or SWORKER process, depending on which process is handling the client request.

The Message Log Request IPM

The sending process sends request information to the collector process using structures defined in the LWMLDDL DDL file. The request IPM consists of a standard header, a message data map containing offset and length information for the elements in the message data buffer, and a buffer containing the message data itself. The collector process decomposes the message data buffer using the information in the map.  In the event that the Log Request IPM exceeds the maximum inter-process I/O size for the collector process, the request is streamed to the collector in multiple sequential requests.

Request Field Definitions

Header Fields

Field Name

Description

RQ-CODE

The standard request code for the Message Log request. This field will have the value LW-ML-RQ-REQUEST

RQ-VERSION

The Message Log request version.

RQ-LENGTH

The length in bytes of the entire Message Log request.

RQ-TS-UNIQUE

A 128-bit timestamp returned from the TS_UNIQUE_CREATE_ system procedure call that is unique to the system

it is generated on and any system in the same EXPAND network

Message Data Map Field Types

The message map contains a list of fields describing data in the message data area. A map field may be one of the following types:

Data Type

Description

Buffer

This data type consists of a fixed length buffer containing binary data. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the buffer offset from the start of the request IPM and the -LEN element indicates the length of the buffer in bytes. Fixed length buffer fields are guaranteed to be 4 byte aligned with the start of the IPM.

String

This data type consists of a NULL terminated character string. Map fields for this data type include <field-name>-OFFSET and <field-name>-LEN. The -OFFSET element indicates the offset of the string from the start of the request IPM and the -LEN element indicates the length of the string including the NULL terminator. String fields are byte aligned with the start of the IPM.

Collection

This data type consists of a collection of name / value pairs consisting of String data. Map fields for this data type include <field-name>-OFFSET, <field-name>-LEN, and <field-name>-COUNT. The -OFFSET element indicates the offset of the name / value collection from the start of the request IPM and the -LEN element indicates the length of the collection data including all NULL terminators. The -COUNT element indicates the number of name / value pairs in the collection.

Integer

A 32 or 64 bit integer value.

Timestamp

A 64 bit JULIANTIMESTAMP

Interval

A 32 bit unsigned integer containing a time interval in microseconds.

Message Data Map Fields

Field Name

Data Type

Description

START-TIME

Timestamp

The request start time.

END-TIME

Timestamp

The request end time.

TOTAL-TIME

Interval

Total request time in microseconds

CONNECT-TIME

Interval

Always 0.

CONNECT-HS-TIME

Interval

The TLS handshake time.

REQUEST-TIME

Interval

The request send time.

RESPONSE-TIME

Interval

The response time which consists of the service processing time and the response network transfer time.

SERIALIZE-TIME

Interval

Request serialization time.

DESERIALIZE-TIME

Interval

The response deserialization time.

SERVER-IO-TIME

Interval

The process or serverclass I/O time.

RQ-IPM

Buffer

The application request IPM.

RQ-REQUEST-LINE

String

The HTTP request line.

RQ-METHOD

String

The request method, e.g, "POST".

RQ-URI

String

The request URI.

RQ-PARAMS

Collection

The request query string parameters.

RQ-HTTP-VER

String

The HTTP protocol version, e.g. "1.1".

RQ-HEADERS

Collection

The HTTP request headers.

RQ-BODY

Buffer

The HTTP request body

RP-IPM

Buffer

The application reply IPM

RP-STATUS

Integer

The HTTP response status.

RP-STATUS-LINE

String

The HTTP response line.

RP-HEADERS

Collection

The HTTP response headers.

RP-BODY

Buffer

The HTTP response body

USER-DATA

Collection

Any fields designated in the request IPM as User Data. See Logging User Data.

METADATA

Collection

Any fields selected from the available request metadata. See Logging Request Metadata.

Log Request Streaming

The sending process writes the Message Log request buffer to the collector process using the largest inter-process message size available. This may be anywhere from 32K to 2MB, depending on the system configuration. In the event that the Message Log request is larger than the maximum size allowed, it will be sent to the collector in multiple consecutive requests within a Pathway Dialog. The collector process should issue a sufficient number of $RECEIVE read operations in order to read the entire request. 

Logging User Data

Application request IPM data may be included in the Message Log request by indicating which fields to include in the API definition type schema. Fields with the msgLogUserData schema property set to true will be included in the USER-DATA collection. Note that:

  • Only primitive data types (string, integer, etc.) can be logged. Object and array types cannot be logged.

  • All data regardless of IPM type is converted to string for inclusion in the USER-DATA collection.

The following schema fragment illustrates how to include application request data in the log request.

XML
<type name="RequestMessage>
  <element name="requestData" type="string" size="32" />
  <element name="requestId" type="string" size="32" msgLogUserData="1" />
</type>

Logging Request Metadata

Message Logging may be configured to include additional request metadata. The metadata includes information such as sending process information and API request information. The following metadata is available.

Name

Value

api-name

The API name.

api-file-name

The API file name.

api-method

The API operation method.

api-path

The API operation path.

api-base-url

The API base url.

api-alias

The API operation alias

config-*

An item for each of the configuration options for the collector, e.g, config-collector-name, config-collector-pathmon, etc. Also the name of the filter selected, as config-filter-name.

net-local-ip

The local IP address of the socket connection.

net-local-port

The local port of the socket connection.

net-remote-ip

The remote IP address of the socket connection.

net-remote-port

The remote port of the socket connection.

net-process

The TCP/IP process in use.

process-name

The process name of the sending process.

process-program-file

The sending process program file name.

process-ancestor-name

The process name of the sending process ancestor.

process-ancestor-program

The program file name of the sending process ancestor.

tls-version

The TLS protocol version.

tls-cipher

The TLS cipher

Sensitive Data Masking

The Sensitive Data Masking feature allows message fields to be designated as holding sensitive data by adding the attribute "sensitive" : true to the element describing the field. Fields so designated are masked in the Message Log request. The feature can be disable by starting the SERVER process with the --disable-sensitive-data-masking option. For more information see Sensitive Data Masking.

Configuring Message Logging

Message Logging is configured using a process configuration file. The configuration is activated by specifying the configuration options in an EDIT file and providing the file location with the --msg-log startup option. Change monitoring for the configuration file may be configured using the --monitor option, for example:

CODE
tacl> run SERVER --msg-log +$vol.subvol.mlogcfg [ --monitor msg-log ]

The configuration is specified using YAML 1.1 format. Multiple collector processes and message selection rules may be configured, for example:

YML
---
msg-log:
  collectors:
    collector-all-content:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-all
      content: all
    collector-http-only:
      enabled: true
      type: serverclass
      pathmon: $lwml
      serverclass: collect-http
      content: rq-http,rp-http
  filters:
    successful-request:
	  # Log HTTP info for successful requests
      enabled: true
      match-all:  
        http-status: 200:299
        reply-code: 0
      collectors: collector-http-only
    failed-request:
      # Log all info for failed requests
      enabled: true
      match-any:  
        http-status: 300:600
        reply-code: 1,1
      collectors: collector-all-content
    catch-all:
      # Illustrate catch all rule that sends to both collectors.
      enabled: true
      match-always: true
      collectors: 
	    - collector-http-only
        - collector-all-content
...

More information on YAML can be found on the official YAML web site and Wikipedia.

Checking Configuration File Syntax

During the process of creating or updating a configuration file, the configuration may be validated using the LWSCOM CHECK command. 

TEXT
tacl> run lwscom
LightWave Server COM 1.1.0 - \NODE.$X123
Copyright (c) 2020 NuWave Technologies, Inc. All rights reserved.
LWSCOM 1-> check config mlogcfg
Checking CONFIG file MLOGCFG.
The file contains a valid 'msg-log' configuration.
The CONFIG file is valid.
LWSCOM 2->

Configuration Reference

The basic structure of the configuration file is

YML
---
# A YAML document begins with '---' and end with '...'
# The msg-log configuration begins with a 'msg-log' element.
msg-log:
  collectors:
	# The 'collectors' element contains a collection of named collector definitions.
    # A collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among collectors.
	collector-name: 
      # Collector elements
  filters:
	# The 'filters' element contains a collection of named filter definitions. Filters are checked in order until a match is found. 
    # A filter name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9, and must be unique among filters.
	filter-name: # Collector name is 1-80 characters, begins with a-z, may contain a-z, 0-9, '-', '_', '.', must end with a-z or 0-9.
	  # Filter elements
...

Collector Elements

Option

Description

enabled

A boolean such as "true" or "false", indicating if the collector is enabled. Disabled collectors may be referenced by filters but messages will not be sent. Required.

type

The type of collector, which must be "serverclass". If omitted, the default value is "serverclass".

pathmon

The process name of the serverclass PATHMON process, which may be specified as a DEFINE. Required if type is "serverclass".

serverclass

The serverclass name. Required if type is "serverclass".

content

A comma separated list of one or more of the following content selectors:

  • api - API metadata

  • net - Network metadata

  • process - Process metadata

  • rq-ipm - The request IPM

  • rq-http - The request HTTP protocol information (request line, headers, etc.)

  • rq-body - The request body

  • rp-ipm - The reply IPM

  • rp-http - The reply HTTP protocol information (request line, headers, etc).

  • rp-body- The reply body

  • tls - TLS metadata

  • user - The user data.

  • all - all possible content.

The default value is 'all'. May be negated, see Using Negation.

io-retry-interval

The number of seconds to wait before retrying a failed I/O to the collector target. The value must be in the range 5 - 30 seconds.

io-retry-limit

The number of times to retry a failed I/O to the collector target. The value must be in the range 1 - 10 retries.

io-max-length

The maximum size of any I/O to the collector target. The sending process always tries to use the maximum I/O size possible for the target collector. This parameter may be used to reduce the I/O size to any limit imposed by the target process. Note that this parameter may also be used to artificially limit the I/O size of a Pathsend serverclass collector, in order to test the servers ability to handle streaming requests using Pathway dialogs.

request-code

The IPM request code to use for the message logging request. By default, the the value of LW-ML-RQ-MSG-LOG as the IPM request code. This option may be used to override that value and allow each collector configuration to use a unique request code.

Filter Elements

Option

Description

enabled

A boolean such as "true" or "false", indicating if the filter is enabled. Disabled filters are ignored.

continue-on-match

A boolean such as "true" or "false", indicating if filter processing should continue if the current filter matches. By default, when a filter matches, no further filters are checked. If omitted, the default value is "false".

match-all

The match-all element contains a collection of filter rules. In order for the filter to match, all rules must match. Optional.

match-any

The match-any element contains a collection of filter rules. In order for the filter to match, any of the rules must match. Optional.

match-always

A boolean such as "true" or "false", indicating that the filter always matches. Optional. If present, the filter matches and any match-all or match-any collections are ignored. This option can be used to create a "catch all" filter.

collectors

A single collector name or a sequence of collector names. If the filter rules match, a logging request will be sent to each collector.

Note that a filter may contain both the match-all and match-any options. When both are present, they must both match in order for the filter to match (they are logically ANDed).

Filter Rule Elements

Enclosing rule element values in double quotes is strongly recommended.

In the following examples, one or more required filter elements may have been omitted for brevity.


Option

Description

alias

A single pattern or a sequence of patterns used to select or ignore an operation alias. See Using Patterns. May be negated, see Using Negation.

Example

YML
filters:
  match-when-alias-starts-with-put:
	match-all:
      alias: "put*"
  match-when-alias-starts-with-post-or-get:
    match-all:
      alias:
        - "post*"
        - "get*"


http-status

A single range or a sequence of ranges used to select or ignore http-status values. See Using Patterns. May be negated, see Using Negation.

Example

YML
filters:
  match-when-in-200-range:
    match-all:
      http-status: "200:299"
  match-when-in-200-range-but-not-222:
    match-all:
      http-status:
        - "200:299"
        - "!222"


method

A comma separated list or pattern used to select or ignore HTTP methods. See Using Patterns. May be negated, see Using Negation.

Example

YML
filters:
  match-when-post-or-put:
    match-all:
      method: "post,put"
  match-when-not-get:
    match-all:
      method: "!get"


path

A single pattern or sequence of patterns used to select or ignore an operation path. See Using Patterns. May be negated, see Using Negation. Note that the path used for matching is the actual path used to invoke the operation, not the path specified in the API definition.

Example

YML
filters:
  match-when-account-request:
    match-all:
      path: "/api/1/account/*"
  match-when-account-or-customer-request:
    match-all:
      path:
        - "/api/v1/account/*"
        - "/api/v1/customer/*"    


reply-code

A single range or sequence of ranges used to select or ignore reply-code values. See Using Patterns. May be negated, see Using Negation.

Example

YML
filters:
  match-when-0:
    match-all:
      reply-code: "0"
  match-when-10-20-but-not-15:
    match-all:
      reply-code:
        - "!15"
        - "10:20"


rq-header / rp-header

A collection of HTTP request or response header name value pairs. The name matches the HTTP header name. The value may be a pattern. See Using Patterns. May be negated, see Using Negation.

Boolean true of false may also be specified to match when the header is or is not present.

Example

YML
filters:
  match-when-json-reply:
    match-all:
      rp-header:
        content-type: application/json
        header-1: false  # match if header-1 is not present
        header-2: true   # match if header-2 is present, regardless of value


Using Patterns

A pattern may be a literal string, a simple pattern, or a Perl compatible regular expression.

  • literal string - A string that exactly matches the element value. The string comparison is case insensitive.

  • simple pattern - A string that may contain the question mark ('?') character to match any single character or the asterisk ('*') to match multiple characters. The string comparison is case insensitive.

  • regular expression - A string prefixed with "regex:" which contains a Perl compatible regular expression used to match the element value. See Perl Compatible Regular Expressions.

For example, the following rules all match the path "/acme/account/9001"

YML
path: "/acme/account/9001"  # literal pattern
path: "/acme/account/*"     # match any account
path: "regex:(?i)^\/acme\/account\/(9001|9002)$" # match account 9001 or 9002

Using Ranges

A range of values is specified as a comma separated list of single integers, or two integers separated by a colon ':' indicating an inclusive range. For example, the range "1,2,5:10" matches the values 1, 2, and 5-10 inclusive.

Using Negation

Where noted, patterns and ranges may be negated in order to indicate that the rule should match when the pattern or range does NOT match. Negation is indicated by prefixing the pattern or range with the exclamation mark ('!'). Note that because the exclamation mark is a special character in YAML, patterns or ranges specified with negation must be enclosed in double quotes. For example:

YML
path: "!/acme/account/9001"  # match when this is NOT the path.
http-status: !200:299        # match when the HTTP status is NOT in the 200-299 range
rp-header:
  content-type: "!application/json"   # match when the content-type reply header is NOT "application/json"

Sample Collectors

Sample collectors processes in C and COBOL can be found on Github.


JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.