This page is a replication of the passage Functions of the Fix Readme.md. Status: Fix Release 1.1.2

Functions

Script-level functions

include

Includes a Fix file and executes it as if its statements were written in place of the function call.

Parameters:

• path (required): Path to Fix file (if the path starts with a ., it is resolved relative to the including file’s directory; otherwise, it is resolved relative to the current working directory).

Options:

• All options are made available as “dynamic” local variables in the included Fix file.

include("<path>"[, <dynamicLocalVariables>...])

Java Code

nothing

Does nothing. It is used for benchmarking in Catmandu.

nothing()

Example in Playground

Java Code

put_filemap

Defines an external map for lookup from a file or a URL. Maps with more than 2 columns are supported but are reduced to a defined key and a value column.

put_filemap("<sourceFile>", "<mapName>", sep_char: "\t")

Example in Playground

The separator (sep_char) will vary depending on the source file, e.g.:

Type	Separator
CSV	, or ;
TSV	\t

Options:

• allow_empty_values: Sets whether to allow empty values in the filemap or to ignore these entries. (Default: false)
• compression: Sets the compression of the file.
• decompress_concatenated: Flags whether to use decompress concatenated file compression.
• encoding: Sets the encoding used to open the resource.
• expected_columns: Sets number of expected columns; lines with different number of columns are ignored. Set to -1 to disable the check and allow arbitrary number of columns. (Default: 2)
• key_column: Defines the column to be used for keys. Uses zero index. (Default: 0)
• value_column: Defines the column to be used for values. Uses zero index. (Default: 1)

Java Code

put_map

Defines an internal map for lookup from key/value pairs.

put_map("<mapName>",
  "dog": "mammal",
  "parrot": "bird",
  "shark": "fish"
)

Example in Playground

Java Code

put_rdfmap

Defines an external RDF map for lookup from a file or an HTTP(S) resource. As the RDF map is reducing RDF triples to a key/value map it is mandatory to set the target. The targeted RDF property can optionally be bound by an RDF language tag.

put_rdfmap("<rdfResource>", "<rdfMapName>", target: "<rdfProperty>")
put_rdfmap("<rdfResource>", "<rdfMapName>", target: "<rdfProperty>", select_language: "<rdfLanguageTag>")

Example in Playground

Java Code

put_var

Defines a single global variable that can be referenced with $[<variableName>].

put_var("<variableName>", "<variableValue>")

Example in Playground

Java Code

put_vars

Defines multiple global variables that can be referenced with $[<variableName>].

put_vars(
  "<variableName_1>": "<variableValue_1>",
  "<variableName_2>": "<variableValue_2>"
)

Example in Playground

Java Code

Record-level functions

add_field

Creates a field with a defined value.

add_field("<targetFieldName>", "<fieldValue>")

Example in Playground

Java Code

array

Converts a hash/object into an array.

array("<sourceField>")

E.g.:

array("foo")
# {"name":"value"} => ["name", "value"]

Example in Playground

Java Code

call_macro

Calls a named macro, i.e. a list of statements that have been previously defined with the do put_macro bind.

Parameters:

• name (required): Unique name of the macro.

Options:

• All options are made available as “dynamic” local variables in the macro.

do put_macro("<macroName>"[, <staticLocalVariables>...])
  ...
end
call_macro("<macroName>"[, <dynamicLocalVariables>...])

Example in Playground

Java Code

copy_field

Copies a field from an existing field.

copy_field("<sourceField>", "<targetField>")

Example in Playground

Java Code

format

Replaces the value with a formatted (sprintf-like) version.

—- TODO: THIS NEEDS MORE CONTENT —–

format("<sourceField>", "<formatString>")

Java Code

hash

Converts an array into a hash/object.

hash("<sourceField>")

E.g.:

hash("foo")
# ["name", "value"] => {"name":"value"}

Example in Playground

Java Code

move_field

Moves a field from an existing field. Can be used to rename a field.

move_field("<sourceField>", "<targetField>")

Example in Playground

Java Code

parse_text

Parses a text into an array or hash of values.

—- TODO: THIS NEEDS MORE CONTENT —–

parse_text("<sourceField>", "<parsePattern>")

Example in Playground

Java Code

paste

Joins multiple field values into a new field. Can be combined with additional literal strings.

The default join_char is a single space. Literal strings have to start with ~.

paste("<targetField>", "<sourceField_1>"[, ...][, "join_char": ", "])

E.g.:

# a: eeny
# b: meeny
# c: miny
# d: moe
paste("my.string", "~Hi", "a", "~how are you?")
# "my.string": "Hi eeny how are you?"

Example in Playground

Java Code

print_record

Prints the current record as JSON either to standard output or to a file.

Parameters:

• prefix (optional): Prefix to print before the record; may include format directives for counter and record ID (in that order). (Default: Empty string)

Options:

• append: Whether to open files in append mode if they exist. (Default: false)
• compression (file output only): Compression mode. (Default: auto)
• destination: Destination to write the record to; may include format directives for counter and record ID (in that order). (Default: stdout)
• encoding (file output only): Encoding used by the underlying writer. (Default: UTF-8)
• footer: Footer which is written at the end of the output. (Default: \n)
• header: Header which is written at the beginning of the output. (Default: Empty string)
• id: Field name which contains the record ID; if found, will be available for inclusion in prefix and destination. (Default: _id)
• internal: Whether to print the record’s internal representation instead of JSON. (Default: false)
• pretty: Whether to use pretty printing. (Default: false)
• separator: Separator which is written after the record. (Default: \n)

print_record(["<prefix>"][, <options>...])

E.g.:

print_record("%d) Before transformation: ")
print_record(destination: "record-%2$s.json", id: "001", pretty: "true")
print_record(destination: "record-%03d.json.gz", header: "After transformation: ")

Java Code

random

Creates (or replaces) a field with a random number (less than the specified maximum).

random("<targetField>", "<maximum>")

Example in Playground

Java Code

remove_field

Removes a field.

remove_field("<sourceField>")

Example in Playground

Java Code

rename

Replaces a regular expression pattern in subfield names of a field. Does not change the name of the source field itself.

rename("<sourceField>", "<regexp>", "<replacement>")

Example in Playground

Java Code

retain

Deletes all fields except the ones listed (incl. subfields).

retain("<sourceField_1>"[, ...])

Example in Playground

Java Code

set_array

Creates a new array (with optional values).

set_array("<targetFieldName>")
set_array("<targetFieldName>", "<value_1>"[, ...])

Example in Playground

Java Code

set_field

Creates (or replaces) a field with a defined value.

set_field("<targetFieldName>", "<fieldValue>")

Java Code

set_hash

Creates a new hash (with optional values).

set_hash("<targetFieldName>")
set_hash("<targetFieldName>", "subfieldName": "<subfieldValue>"[, ...])

Example in Playground

Java Code

timestamp

Creates (or replaces) a field with the current timestamp.

Options:

• format: Date and time pattern as in java.text.SimpleDateFormat. (Default: timestamp)
• timezone: Time zone as in java.util.TimeZone. (Default: UTC)
• language: Language tag as in java.util.Locale. (Default: The locale of the host system)

timestamp("<targetField>"[, format: "<formatPattern>"][, timezone: "<timezoneCode>"][, language: "<languageCode>"])

Example in Playground

Java Code

vacuum

Deletes empty fields, arrays and objects.

vacuum()

Example in Playground

Java Code

Field-level functions

append

Adds a string at the end of a field value.

append("<sourceField>", "<appendString>")

Example in Playground

Java Code

capitalize

Upcases the first character in a field value.

capitalize("<sourceField>")

Example in Playground

Java Code

count

Counts the number of elements in an array or a hash and replaces the field value with this number.

count("<sourceField>")

Example in Playground

Java Code

downcase

Downcases all characters in a field value.

downcase("<sourceField>")

Example in Playground

Java Code

filter

Only keeps field values that match the regular expression pattern. Works only with array of strings/repeated fields.

filter("<sourceField>", "<regexp>")

Example in Playground

Java Code

flatten

Flattens a nested array field.

flatten("<sourceField>")

Example in Playground

Java Code

from_json

Replaces the string with its JSON deserialization.

Options:

• error_string: Error message as a placeholder if the JSON couldn’t be parsed. (Default: null)

from_json("<sourceField>"[, error_string: "<errorValue>"])

Java Code

index

Returns the index position of a substring in a field and replaces the field value with this number.

index("<sourceField>", "<substring>")

Example in Playground

Java Code

isbn

Extracts an ISBN and replaces the field value with the normalized ISBN; optionally converts and/or validates the ISBN.

Options:

• to: ISBN format to convert to (either ISBN10 or ISBN13). (Default: Only normalize ISBN)
• verify_check_digit: Whether the check digit should be verified. (Default: false)
• error_string: Error message as a placeholder if the ISBN couldn’t be validated. (Default: null)

isbn("<sourceField>"[, to: "<isbnFormat>"][, verify_check_digit: "<boolean>"][, error_string: "<errorValue>"])

Example in Playground

Java Code

join_field

Joins an array of strings into a single string.

join_field("<sourceField>", "<separator>")

Example in Playground

Java Code

lookup

Looks up matching values in a map and replaces the field value with this match. External files, internal maps as well as RDF resources can be used.

Parameters:

• path (required): Field path to look up.
• map (optional): Name or path of the map in which to look up values.

Options:

• __default: Default value to use for unknown values. (Default: Old value)
• delete: Whether to delete unknown values. (Default: false)
• print_unknown: Whether to print unknown values. (Default: false)

Additional options when printing unknown values:

• append: Whether to open files in append mode if they exist. (Default: true)
• compression (file output only): Compression mode. (Default: auto)
• destination: Destination to write unknown values to; may include format directives for counter and record ID (in that order). (Default: stdout)
• encoding (file output only): Encoding used by the underlying writer. (Default: UTF-8)
• footer: Footer which is written at the end of the output. (Default: \n)
• header: Header which is written at the beginning of the output. (Default: Empty string)
• id: Field name which contains the record ID; if found, will be available for inclusion in prefix and destination. (Default: _id)
• prefix: Prefix to print before the unknown value; may include format directives for counter and record ID (in that order). (Default: Empty string)
• separator: Separator which is written after the unknown value. (Default: \n)

lookup("<sourceField>"[, <mapName>][, <options>...])

E.g.:

# local (unnamed) map
lookup("path.to.field", key_1: "value_1", ...)

# internal (named) map
put_map("internal-map", key_1: "value_1", ...)
lookup("path.to.field", "internal-map")

# external file map (implicit)
lookup("path.to.field", "path/to/file", sep_char: ";")

# external file map (explicit)
put_filemap("path/to/file", "file-map", sep_char: ";")
lookup("path.to.field", "file-map")

# RDF map (explicit)
put_rdfmap("path/to/file", "rdf-map", target: "<rdfProperty>")
lookup("path.to.field", "rdf-map")

# with default value
lookup("path.to.field", "map-name", __default: "NA")

# with printing unknown values to a file
lookup("path.to.field", "map-name", print_unknown: "true", destination: "unknown.txt")

Java Code

prepend

Adds a string at the beginning of a field value.

prepend("<sourceField>", "<prependString>")

Example in Playground

Java Code

replace_all

Replaces a regular expression pattern in field values with a replacement string. Regexp capturing is possible; refer to capturing groups by number ($<number>) or name (${<name>}).

replace_all("<sourceField>", "<regexp>", "<replacement>")

Example in Playground

Java Code

reverse

Reverses the character order of a string or the element order of an array.

reverse("<sourceField>")

Example in Playground

Java Code

sort_field

Sorts strings in an array. Alphabetically and A-Z by default. Optional numerical and reverse sorting.

sort_field("<sourceField>")
sort_field("<sourceField>", reverse: "true")
sort_field("<sourceField>", numeric: "true")

Example in Playground

Java Code

split_field

Splits a string into an array and replaces the field value with this array.

split_field("<sourceField>", "<separator>")

Example in Playground

Java Code

substring

Replaces a string with its substring as defined by the start position (offset) and length.

substring("<sourceField>", "<startPosition>", "<length>")

Example in Playground

Java Code

sum

Sums numbers in an array and replaces the field value with this number.

sum("<sourceField>")

Example in Playground

Java Code

to_json

Replaces the value with its JSON serialization.

Options:

• error_string: Error message as a placeholder if the JSON couldn’t be generated. (Default: null)
• pretty: Whether to use pretty printing. (Default: false)

to_json("<sourceField>"[, pretty: "<boolean>"][, error_string: "<errorValue>"])

Example in Playground

Java Code

to_base64

Replaces the value with its Base64 encoding.

to_base64("<sourceField>")

Example in Playground

Java Code

trim

Deletes whitespace at the beginning and the end of a field value.

trim("<sourceField>")

Example in Playground

Java Code

uniq

Deletes duplicate values in an array.

uniq("<sourceField>")

Example in Playground

Java Code

upcase

Upcases all characters in a field value.

upcase("<sourceField>")

Example in Playground

Java Code

uri_encode

Encodes a field value as URI. Aka percent-encoding.

Options:

• plus_for_space: Sets whether “space” ( ) will be substituted by a “plus” (+) or be percent escaped (%20). (Default: true)
• safe_chars: Sets characters that won’t be escaped. Safe characters are the ranges 0..9, a..z and A..Z. These are always safe and should not be specified. (Default: .-*_)

uri_encode("<sourceField>"[, <options>...])

E.g.:

uri_encode("path.to.field", plus_for_space:"false", safe_chars:"")

Example in Playground

Java Code

Selectors

reject

Ignores records that match a condition.

if <condition>
  reject()
end

Example in Playground

Java Code

Binds

do list

Iterates over each element of an array. In contrast to Catmandu, it can also iterate over a single object or string.

Java Code

do list(path: "<sourceField>")
  ...
end

Example in Playground

Only the current element is accessible in this case (as the root element).

When specifying a variable name for the current element, the record remains accessible as the root element and the current element is accessible through the variable name:

do list(path: "<sourceField>", "var": "<variableName>")
  ...
end

Example in Playground

do list_as

Iterates over each named element of an array (like do list with a variable name). If multiple arrays are given, iterates over the corresponding elements from each array (i.e., all elements with the same array index, skipping elements whose arrays have already been exhausted).

Example in Playground

Java Code

do list_as(element_1: "<sourceField_1>"[, ...])
  ...
end

E.g.:

# "ccm:university":["https://ror.org/0304hq317"]
# "ccm:university_DISPLAYNAME":["Gottfried Wilhelm Leibniz Universität Hannover"]
set_array("sourceOrga[]")
do list_as(orgId: "ccm:university[]", orgName: "ccm:university_DISPLAYNAME[]")
  copy_field(orgId, "sourceOrga[].$append.id")
  copy_field(orgName, "sourceOrga[].$last.name")
end
# {"sourceOrga":[{"id":"https://ror.org/0304hq317","name":"Gottfried Wilhelm Leibniz Universität Hannover"}]}

do once

Executes the statements only once (when the bind is first encountered), not repeatedly for each record.

do once()
  ...
end

Example in Playground

Java Code

In order to execute multiple blocks only once, tag them with unique identifiers:

do once("maps setup")
  ...
end
do once("vars setup")
  ...
end

do put_macro

Defines a named macro, i.e. a list of statements that can be executed later with the call_macro function.

Variables can be referenced with $[<variableName>], in the following order of precedence:

1. “dynamic” local variables, passed as options to the call_macro function;
2. “static” local variables, passed as options to the do put_macro bind;
3. global variables, defined via put_var/put_vars.

Parameters:

• name (required): Unique name of the macro.

Options:

• All options are made available as “static” local variables in the macro.

do put_macro("<macroName>"[, <staticLocalVariables>...])
  ...
end
call_macro("<macroName>"[, <dynamicLocalVariables>...])

Example in Playground

Java Code

Conditionals

Conditionals start with if in case of affirming the condition or unless rejecting the condition.

Conditionals require a final end.

Additional conditionals can be set with elsif and else.

if <condition(params, ...)>
  ...
end

unless <condition(params, ...)>
  ...
end

if <condition(params, ...)>
  ...
elsif
  ...
else
  ...
end

contain

all_contain

Executes the functions if/unless the field contains the value. If it is an array or a hash all field values must contain the string.

Example in Playground

Java Code

any_contain

Executes the functions if/unless the field contains the value. If it is an array or a hash one or more field values must contain the string.

Example in Playground

Java Code

none_contain

Executes the functions if/unless the field does not contain the value. If it is an array or a hash none of the field values may contain the string.

Example in Playground

Java Code

str_contain

Executes the functions if/unless the first string contains the second string.

Example in Playground

Java Code

equal

all_equal

Executes the functions if/unless the field value equals the string. If it is an array or a hash all field values must equal the string.

Example in Playground

Java Code

any_equal

Executes the functions if/unless the field value equals the string. If it is an array or a hash one or more field values must equal the string.

Example in Playground

Java Code

none_equal

Executes the functions if/unless the field value does not equal the string. If it is an array or a hash none of the field values may equal the string.

Example in Playground

Java Code

str_equal

Executes the functions if/unless the first string equals the second string.

Example in Playground

Java Code

exists

Executes the functions if/unless the field exists.

if exists("<sourceField>")

Example in Playground

Java Code

in

Executes the functions if/unless the field value is contained in the value of the other field.

Also aliased as is_contained_in.

Example in Playground

Java Code

is_contained_in

Alias for in.

is_array

Executes the functions if/unless the field value is an array.

Example in Playground

Java Code

is_empty

Executes the functions if/unless the field value is empty.

Example in Playground

Java Code

is_false

Executes the functions if/unless the field value equals false or 0.

Example in Playground

Java Code

is_hash

Alias for is_object.

Example in Playground

Java Code

is_number

Executes the functions if/unless the field value is a number.

Example in Playground

Java Code

is_object

Executes the functions if/unless the field value is a hash (object).

Also aliased as is_hash.

is_string

Executes the functions if/unless the field value is a string (and not a number).

Example in Playground

Java Code

is_true

Executes the functions if/unless the field value equals true or 1.

Example in Playground

Java Code

match

all_match

Executes the functions if/unless the field value matches the regular expression pattern. If it is an array or a hash all field values must match the regular expression pattern.

Example in Playground

Java Code

any_match

Executes the functions if/unless the field value matches the regular expression pattern. If it is an array or a hash one or more field values must match the regular expression pattern.

Example in Playground

Java Code

none_match

Executes the functions if/unless the field value does not match the regular expression pattern. If it is an array or a hash none of the field values may match the regular expression pattern.

Example in Playground

Java Code

str_match

Executes the functions if/unless the string matches the regular expression pattern.

Example in Playground

Java Code