Add new ssu_tokens_.pl1 module to bound_ssu_, ssu_.alm, ssu_.info, pl1.dcl

[Gary Dixon]

A new routine for identifying tokens (lexemes) in a character string has been designed for ssu_ subsystem use. The entry points its added should be included in the ssu_ documentation and user interface.
This ticket recommends those additions to the ssu_ subsystem.

The new entry points are: ssu_$display_tokens, ssu_$find_token_with_id,
ssu_$get_token_area, ssu_$get_tokens, ssu_$token_id

2025-05-31 ssu_$get_token_area

Function: This subroutine obtains an area with characteristics ideal
for tokens created by the ssu_$get_tokens subroutine.  It calls the
ssu_$get_area subroutine to obtain an extensible, no-freeing area in a
temporary segment managed by ssu_.  The ssu_ request or standalone
command may explicitly release the area by calling ssu_$release_area;
or it will be released automatically when the subsystem invocation is
destroyed.

Syntax:
declare ssu_$get_token_area entry (ptr, char(*)) returns (ptr);

token_area_ptr = ssu_$get_token_area (sci_ptr, comment);

2025-06-11  ssu_$get_tokens


Function:  This entry is called to find lexemes (meaningful substrings)
within an input string.  A token descriptor is allocated to describe
each lexeme found in the input_string.  Descriptors are linked in a
list in the order of appearance of their lexeme within the input
string.  See "Notes on lexemes" for rules on how lexemes are identified
in the input string.

You may use the ssu_$display_tokens subroutine to display the tokens
returned by ssu_$get_tokens.


Syntax:
declare ssu_$get_tokens entry(ptr, ptr,
     char(*), fixed bin(21), char(32) var,
     ptr, bit(1) aligned,
     char(4) var,  char(*) var,
     ptr, fixed bin, fixed bin(35));

call ssu_$get_tokens(sci_ptr, token_area_ptr,
     input_string, ignored_inputL, input_string_name,
     caller_data_ptr, list_rescan_sw,
     statement_delimiter, extra_break_chars,
     first_token_ptr, token_count, code);

2025-05-31  ssu_$display_tokens


Function:  This entry displays token descriptors found by the
ssu_$get_tokens subroutine.  See "Notes on token display" for details
on how a token descriptor is displayed.


Syntax:
declare ssu_$display_tokens entry(ptr, char(32) var, ptr,
     bit(1) aligned, bit(1) aligned);

call ssu_$display_tokens(sci_ptr, input_string_name, first_token_ptr,
     list_display_sw, statement_display_sw);

2025-05-31  ssu_$find_token_with_id


Function:  Given a pointer to the first token in a token tree returned
by ssu_$get_tokens and a token ID locating some token within that tree,
this subroutine returns a pointer to the token having that ID; or
returns a null pointer with non-zero status code if no token matches
that ID.


Syntax:
declare ssu_$find_token_with_id (ptr, ptr, char(52) var, ptr,
                                 fixed bin(35));

call ssu_$find_token_with_id (sci_ptr, first_token_ptr, token_id,
                              found_token_ptr, code);

2025-05-31  ssu_$token_id


Function:  Given a pointer to a token within a token tree returned by
the ssu_$get_tokens subroutine, this function returns the token's
identifier locating the token's position within the tree.


Syntax:
declare ssu_$token_id entry (ptr, ptr, ptr) returns (char(52) var);

token_id = ssu_$token_id (sci_ptr, token_ptr, parent_token_ptr);

An MCR will be written to describe these routines in more detail with a justification for their addition to bound_ssu_.

[Eric Swenson]

​https://s3.amazonaws.com/eswenson-multics/public/mcrs/MCR10162.pdf

[Eric Swenson]

Installed as MR12.9-1037.

[Eric Swenson]

Installed as MR12.9-1037.