Docs Connect Bloblang Methods Bloblang Methods Page options Copy as Markdown Copied! View as plain text Ask AI about this topic Add MCP server to VS Code Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods): root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}} Methods support both named and nameless style arguments: root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} General apply Apply a declared mapping to a target value. Parameters Name Type Description mapping string The mapping to apply. Examples map thing { root.inner = this.first } root.foo = this.doc.apply("thing") # In: {"doc":{"first":"hello world"}} # Out: {"foo":{"inner":"hello world"}} map create_foo { root.name = "a foo" root.purpose = "to be a foo" } root = this root.foo = null.apply("create_foo") # In: {"id":"1234"} # Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"} catch If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead. Parameters Name Type Description fallback query expression A value to yield, or query to execute, if the target query fails. Examples root.doc.id = this.thing.id.string().catch(uuid_v4()) The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back: root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url}) # In: {"url":"invalid %&# url"} # Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}} When the input document is not structured attempting to reference structured fields with this will result in an error. Therefore, a convenient way to delete non-structured data is with a catch: root = this.catch(deleted()) # In: {"doc":{"foo":"bar"}} # Out: {"doc":{"foo":"bar"}} # In: not structured data # Out: <Message deleted> from Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are content, json and meta. Parameters Name Type Description index integer The message index to use as a perspective. Examples For example, the following map extracts the contents of the JSON field foo specifically from message index 1 of a batch, effectively overriding the field foo for all messages of a batch to that of message 1: root = this root.foo = json("foo").from(1) from_all Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are content, json and meta. Examples root = this root.foo_summed = json("foo").from_all().sum() map Executes a query on the target value, allowing you to transform or extract data from the current context. Parameters Name Type Description query query expression A query to execute on the target. not Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true. or If the result of the target query fails or resolves to null, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator |. Parameters Name Type Description fallback query expression A value to yield, or query to execute, if the target query fails or resolves to null. Examples root.doc.id = this.thing.id.or(uuid_v4()) Encoding and encryption compress Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage. Parameters Name Type Description algorithm string The compression algorithm: flate, gzip, pgzip (parallel gzip), lz4, snappy, zlib, or zstd. level integer Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. Examples Compress and encode for safe transmission: root.compressed = content().bytes().compress("gzip").encode("base64") # In: {"message":"hello world I love space"} # Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"} Compare compression ratios across algorithms: root.original_size = content().length() root.gzip_size = content().compress("gzip").length() root.lz4_size = content().compress("lz4").length() # In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. # Out: {"gzip_size":114,"lz4_size":85,"original_size":89} decode Decodes an encoded string according to a chosen scheme. Parameters Name Type Description scheme string The decoding scheme to use. Examples root.decoded = this.value.decode("hex").string() # In: {"value":"68656c6c6f20776f726c64"} # Out: {"decoded":"hello world"} root = this.encoded.decode("ascii85") # In: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} # Out: this is totally unstructured data decompress Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm. Parameters Name Type Description algorithm string The decompression algorithm: gzip, pgzip (parallel gzip), zlib, bzip2, flate, snappy, lz4, or zstd. Examples Decompress base64-encoded compressed data: root = this.compressed.decode("base64").decompress("gzip") # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: hello world I love space Convert decompressed bytes to string for JSON output: root.message = this.compressed.decode("base64").decompress("gzip").string() # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: {"message":"hello world I love space"} decrypt_aes Decrypts an AES-encrypted string or byte array. Parameters Name Type Description scheme string The scheme to use for decryption, one of ctr, gcm, ofb, cbc. key string A key to decrypt with. iv string An initialization vector / nonce. Examples let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.decrypted = this.value.decode("hex").decrypt_aes("ctr", $key, $vector).string() # In: {"value":"84e9b31ff7400bdf80be7254"} # Out: {"decrypted":"hello world!"} encode Encodes a string or byte array according to a chosen scheme. Parameters Name Type Description scheme string The encoding scheme to use. Examples root.encoded = this.value.encode("hex") # In: {"value":"hello world"} # Out: {"encoded":"68656c6c6f20776f726c64"} root.encoded = content().encode("ascii85") # In: this is totally unstructured data # Out: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} encrypt_aes Encrypts a string or byte array using AES encryption. Parameters Name Type Description scheme string The scheme to use for encryption, one of ctr, gcm, ofb, cbc. key string A key to encrypt with. iv string An initialization vector / nonce. Examples let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.encrypted = this.value.encrypt_aes("ctr", $key, $vector).encode("hex") # In: {"value":"hello world!"} # Out: {"encrypted":"84e9b31ff7400bdf80be7254"} hash Hashes a string or byte array using a specified algorithm. Parameters Name Type Description algorithm string The hashing algorithm to use. key (optional) string An optional key to use. polynomial string An optional polynomial key to use when selecting the crc32 algorithm, otherwise ignored. Options are IEEE (default), Castagnoli and Koopman Examples root.h1 = this.value.hash("sha1").encode("hex") root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"} The crc32 algorithm supports options for the polynomial: root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex") root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"c99465aa","h2":"df373d3c"} uuid_v5 Generates a version 5 UUID from a namespace and name. Parameters Name Type Description ns (optional) string An optional namespace name or UUID. It supports the dns, url, oid and x500 predefined namespaces and any valid RFC-9562 UUID. If empty, the nil UUID will be used. Examples root.id = "example".uuid_v5() root.id = "example".uuid_v5("x500") root.id = "example".uuid_v5("77f836b7-9f61-46c0-851e-9b6ca3535e69") GeoIP geoip_anonymous_ip Looks up an IP address against a MaxMind database file and, if found, returns an object describing the anonymous IP associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_asn Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ASN associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_city Looks up an IP address against a MaxMind database file and, if found, returns an object describing the city associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_connection_type Looks up an IP address against a MaxMind database file and, if found, returns an object describing the connection type associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_country Looks up an IP address against a MaxMind database file and, if found, returns an object describing the country associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_domain Looks up an IP address against a MaxMind database file and, if found, returns an object describing the domain associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_enterprise Looks up an IP address against a MaxMind database file and, if found, returns an object describing the enterprise associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. geoip_isp Looks up an IP address against a MaxMind database file and, if found, returns an object describing the ISP associated with it. Parameters Name Type Description path string A path to an mmdb (maxmind) file. JSON web tokens parse_jwt_es256 Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The ES256 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w== -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_es384 Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The ES384 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY----- MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU 8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA 2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_es512 Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The ES512 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY----- MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh 9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9 UeYyTt05zRRWuD+p5bY= -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_hs256 Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The HS256 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_hs384 Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The HS384 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_hs512 Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The HS512 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_rs256 Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The RS256 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_rs384 Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The RS384 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} parse_jwt_rs512 Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims. Parameters Name Type Description signing_secret string The RS512 secret that was used for signing the token. Examples root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} sign_jwt_es256 Hash and sign an object representing JSON Web Token (JWT) claims using ES256. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.-8LrOdkEiv_44ADWW08lpbq41ZmHCel58NMORPq1q4Dyw0zFhqDVLrRoSvCvuyyvgXAFb9IHfR-9MlJ_2ShA9A"} root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_es384 Hash and sign an object representing JSON Web Token (JWT) claims using ES384. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.8FmTKH08dl7dyxrNu0rmvhegiIBCy-O9cddGco2e9lpZtgv5mS5qHgPkgBC5eRw1d7SRJsHwHZeehzdqT5Ba7aZJIhz9ds0sn37YQ60L7jT0j2gxCzccrt4kECHnUnLw"} root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_es512 Hash and sign an object representing JSON Web Token (JWT) claims using ES512. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.AQbEWymoRZxDJEJtKSFFG2k2VbDCTYSuBwAZyMqexCspr3If8aERTVGif8HXG3S7TzMBCCzxkcKr3eIU441l3DlpAMNfQbkcOlBqMvNBn-CX481WyKf3K5rFHQ-6wRonz05aIsWAxCDvAozI_9J0OWllxdQ2MBAuTPbPJ38OqXsYkCQs"} root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_hs256 Hash and sign an object representing JSON Web Token (JWT) claims using HS256. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.hUl-nngPMY_3h9vveWJUPsCcO5PeL6k9hWLnMYeFbFQ"} root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_hs384 Hash and sign an object representing JSON Web Token (JWT) claims using HS384. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zGYLr83aToon1efUNq-hw7XgT20lPvZb8sYei8x6S6mpHwb433SJdXJXx0Oio8AZ"} root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_hs512 Hash and sign an object representing JSON Web Token (JWT) claims using HS512. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zBNR9o_6EDwXXKkpKLNJhG26j8Dc-mV-YahBwmEdCrmiWt5les8I9rgmNlWIowpq6Yxs4kLNAdFhqoRz3NXT3w"} root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_rs256 Hash and sign an object representing JSON Web Token (JWT) claims using RS256. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_rs384 Hash and sign an object representing JSON Web Token (JWT) claims using RS384. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} sign_jwt_rs512 Hash and sign an object representing JSON Web Token (JWT) claims using RS512. Parameters Name Type Description signing_secret string The secret to use for signing the token. headers (optional) unknown Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. Examples root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":"<signed JWT token>"} Number manipulation abs Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value. Examples root.outs = this.ins.map_each(ele -> ele.abs()) # In: {"ins":[9,-18,1.23,-4.56]} # Out: {"outs":[9,18,1.23,4.56]} bitwise_and Performs a bitwise AND operation between the integer and the specified value. Parameters Name Type Description value integer The value to AND with Examples root.new_value = this.value.bitwise_and(6) # In: {"value":12} # Out: {"new_value":4} root.masked = this.flags.bitwise_and(15) # In: {"flags":127} # Out: {"masked":15} bitwise_or Performs a bitwise OR operation between the integer and the specified value. Parameters Name Type Description value integer The value to OR with Examples root.new_value = this.value.bitwise_or(6) # In: {"value":12} # Out: {"new_value":14} root.combined = this.flags.bitwise_or(8) # In: {"flags":4} # Out: {"combined":12} bitwise_xor Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value. Parameters Name Type Description value integer The value to XOR with Examples root.new_value = this.value.bitwise_xor(6) # In: {"value":12} # Out: {"new_value":10} root.toggled = this.flags.bitwise_xor(5) # In: {"flags":3} # Out: {"toggled":6} ceil Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. Examples root.new_value = this.value.ceil() # In: {"value":5.3} # Out: {"new_value":6} # In: {"value":-5.9} # Out: {"new_value":-5} root.result = this.price.ceil() # In: {"price":19.99} # Out: {"result":20} cos Calculates the cosine of a given angle specified in radians. Examples root.new_value = (this.value * (pi() / 180)).cos() # In: {"value":45} # Out: {"new_value":0.7071067811865476} # In: {"value":0} # Out: {"new_value":1} # In: {"value":180} # Out: {"new_value":-1} float32 Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats. Examples root.out = this.in.float32() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674283e-11} float64 Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the strconv.ParseFloat documentation for details regarding the supported formats. Examples root.out = this.in.float64() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674282313423544e-11} floor Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. Examples root.new_value = this.value.floor() # In: {"value":5.7} # Out: {"new_value":5} # In: {"value":-3.2} # Out: {"new_value":-4} root.whole_seconds = this.duration_seconds.floor() # In: {"duration_seconds":12.345} # Out: {"whole_seconds":12} int16 Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.int16() root.b = this.b.round().int16() root.c = this.c.int16() root.d = this.d.int16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} root = this.int16() # In: "0xDE" # Out: 222 int32 Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.int32() root.b = this.b.round().int32() root.c = this.c.int32() root.d = this.d.int32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} root = this.int32() # In: "0xDEAD" # Out: 57005 int64 Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.int64() root.b = this.b.round().int64() root.c = this.c.int64() root.d = this.d.int64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} root = this.int64() # In: "0xDEADBEEF" # Out: 3735928559 int8 Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.int8() root.b = this.b.round().int8() root.c = this.c.int8() root.d = this.d.int8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} root = this.int8() # In: "0xD" # Out: 13 log Calculates the natural logarithm (base e) of a number. Examples root.new_value = this.value.log().round() # In: {"value":1} # Out: {"new_value":0} # In: {"value":2.7183} # Out: {"new_value":1} root.ln_result = this.number.log() # In: {"number":10} # Out: {"ln_result":2.302585092994046} log10 Calculates the base-10 logarithm of a number. Examples root.new_value = this.value.log10() # In: {"value":100} # Out: {"new_value":2} # In: {"value":1000} # Out: {"new_value":3} root.log_value = this.magnitude.log10() # In: {"magnitude":10000} # Out: {"log_value":4} max Returns the largest number from an array. All elements must be numbers and the array cannot be empty. Examples root.biggest = this.values.max() # In: {"values":[0,3,2.5,7,5]} # Out: {"biggest":7} root.highest_temp = this.temperatures.max() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"highest_temp":23.4} min Returns the smallest number from an array. All elements must be numbers and the array cannot be empty. Examples root.smallest = this.values.min() # In: {"values":[0,3,-2.5,7,5]} # Out: {"smallest":-2.5} root.lowest_temp = this.temperatures.min() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"lowest_temp":19.8} pow Returns the number raised to the specified exponent. Parameters Name Type Description exponent float The exponent you want to raise to the power of. Examples root.new_value = this.value * 10.pow(-2) # In: {"value":2} # Out: {"new_value":0.02} root.new_value = this.value.pow(-2) # In: {"value":2} # Out: {"new_value":0.25} round Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float. Examples root.new_value = this.value.round() # In: {"value":5.3} # Out: {"new_value":5} # In: {"value":5.9} # Out: {"new_value":6} root.rounded = this.score.round() # In: {"score":87.5} # Out: {"rounded":88} sin Calculates the sine of a given angle specified in radians. Examples root.new_value = (this.value * (pi() / 180)).sin() # In: {"value":45} # Out: {"new_value":0.7071067811865475} # In: {"value":0} # Out: {"new_value":0} # In: {"value":90} # Out: {"new_value":1} tan Calculates the tangent of a given angle specified in radians. Examples root.new_value = "%f".format((this.value * (pi() / 180)).tan()) # In: {"value":0} # Out: {"new_value":"0.000000"} # In: {"value":45} # Out: {"new_value":"1.000000"} # In: {"value":180} # Out: {"new_value":"-0.000000"} uint16 Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.uint16() root.b = this.b.round().uint16() root.c = this.c.uint16() root.d = this.d.uint16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} root = this.uint16() # In: "0xDE" # Out: 222 uint32 Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.uint32() root.b = this.b.round().uint32() root.c = this.c.uint32() root.d = this.d.uint32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} root = this.uint32() # In: "0xDEAD" # Out: 57005 uint64 Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.uint64() root.b = this.b.round().uint64() root.c = this.c.uint64() root.d = this.d.uint64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} root = this.uint64() # In: "0xDEADBEEF" # Out: 3735928559 uint8 Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use .round() on the value. Please refer to the strconv.ParseInt documentation for details regarding the supported formats. Examples root.a = this.a.uint8() root.b = this.b.round().uint8() root.c = this.c.uint8() root.d = this.d.uint8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} root = this.uint8() # In: "0xD" # Out: 13 Object & array manipulation all Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays. Parameters Name Type Description test query expression A test query to apply to each element. Examples root.all_over_21 = this.patrons.all(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"all_over_21":false} # In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]} # Out: {"all_over_21":true} root.all_positive = this.values.all(v -> v > 0) # In: {"values":[1,2,3,4,5]} # Out: {"all_positive":true} # In: {"values":[1,-2,3,4,5]} # Out: {"all_positive":false} any Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays. Parameters Name Type Description test query expression A test query to apply to each element. Examples root.any_over_21 = this.patrons.any(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"any_over_21":true} # In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]} # Out: {"any_over_21":false} root.has_errors = this.results.any(r -> r.status == "error") # In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]} # Out: {"has_errors":true} # In: {"results":[{"status":"ok"},{"status":"ok"}]} # Out: {"has_errors":false} append Adds one or more elements to the end of an array and returns the new array. The original array is not modified. Examples root.foo = this.foo.append("and", "this") # In: {"foo":["bar","baz"]} # Out: {"foo":["bar","baz","and","this"]} root.combined = this.items.append(this.new_item) # In: {"items":["apple","banana"],"new_item":"orange"} # Out: {"combined":["apple","banana","orange"]} assign Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead. Parameters Name Type Description with unknown A value to merge the target value with. Examples root = this.foo.assign(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":"foos","second_name":"barer"} Override defaults with user settings: root.config = this.defaults.assign(this.user_settings) # In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}} # Out: {"config":{"retries":3,"timeout":60}} collapse Flattens a nested structure into a flat object with dot-notation keys. Parameters Name Type Description include_empty bool Whether to include empty objects and arrays in the resulting object. Examples root.result = this.collapse() # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}} Set include_empty to true to preserve empty objects and arrays in the output: root.result = this.collapse(include_empty: true) # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}} concat Concatenates an array value with one or more argument arrays. Examples root.foo = this.foo.concat(this.bar, this.baz) # In: {"foo":["a","b"],"bar":["c"],"baz":["d","e","f"]} # Out: {"foo":["a","b","c","d","e","f"]} contains Tests if an array or object contains a value. Parameters Name Type Description value unknown A value to test against elements of the target. Examples root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} diff Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems. Parameters Name Type Description other unknown The value to compare against the current value. Can be any structured data (object or array). Examples Compare two objects to track field changes: root.changes = this.before.diff(this.after) # In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}} # Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]} Detect deletions in configuration changes: root.changelog = this.old_config.diff(this.new_config) # In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}} # Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]} enumerated Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element. Examples root.foo = this.foo.enumerated() # In: {"foo":["bar","baz"]} # Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]} Useful for filtering by index position: root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value) # In: {"items":["a","b","c","d"]} # Out: {"first_two":["a","b"]} exists Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise. Parameters Name Type Description path string A dot path to a field. Examples root.result = this.foo.exists("bar.baz") # In: {"foo":{"bar":{"baz":"yep, I exist"}}} # Out: {"result":true} # In: {"foo":{"bar":{}}} # Out: {"result":false} # In: {"foo":{}} # Out: {"result":false} Also returns true for null values if the field exists: root.has_field = this.data.exists("optional_field") # In: {"data":{"optional_field":null}} # Out: {"has_field":true} # In: {"data":{}} # Out: {"has_field":false} explode Expands a nested field into multiple documents. Parameters Name Type Description path string A dot path to a field to explode. Examples On arrays When exploding an array, each element becomes a separate document with the array element replacing the original field: root = this.explode("value") # In: {"id":1,"value":["foo","bar","baz"]} # Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}] On objects When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value: root = this.explode("value") # In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}} # Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}} filter Filters array or object elements based on a condition. Parameters Name Type Description test query expression A query to apply to each element, if this query resolves to any value other than a boolean true the element will be removed from the result. Examples root.new_nums = this.nums.filter(num -> num > 10) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[11,17]} On objects When filtering objects, the query receives a context with key and value fields for each entry: root.new_dict = this.dict.filter(item -> item.value.contains("foo")) # In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}} # Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}} find Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation. Parameters Name Type Description value unknown A value to find. Examples root.index = this.find("bar") # In: ["foo", "bar", "baz"] # Out: {"index":1} root.index = this.things.find(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz"]} # Out: {"index":1} find_all Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation. Parameters Name Type Description value unknown A value to find. Examples root.index = this.find_all("bar") # In: ["foo", "bar", "baz", "bar"] # Out: {"index":[1,3]} root.indexes = this.things.find_all(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz", "bar", "buz"]} # Out: {"indexes":[1,3]} find_all_by Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match. Parameters Name Type Description query query expression A query to execute for each element. Examples root.index = this.find_all_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":[0,2]} Find all indexes matching criteria: root.error_indexes = this.logs.find_all_by(log -> log.level == "error") # In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]} # Out: {"error_indexes":[1,3]} find_by Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query. Parameters Name Type Description query query expression A query to execute for each element. Examples root.index = this.find_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":0} Find first object matching criteria: root.first_adult = this.users.find_by(u -> u.age >= 18) # In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]} # Out: {"first_adult":1} flatten Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed. Examples root.result = this.flatten() # In: ["foo",["bar","baz"],"buz"] # Out: {"result":["foo","bar","baz","buz"]} Deeper nesting requires multiple flatten calls: root.result = this.data.flatten() # In: {"data":["a",["b",["c","d"]],"e"]} # Out: {"result":["a","b",["c","d"],"e"]} fold Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration. Parameters Name Type Description initial unknown The initial value to start the fold with. For example, an empty object {}, a zero count 0, or an empty string "". query query expression A query to apply for each element. The query is provided an object with two fields; tally containing the current tally, and value containing the value of the current element. The query should result in a new tally to be passed to the next element query. Examples Sum numbers in an array: root.sum = this.foo.fold(0, item -> item.tally + item.value) # In: {"foo":[3,8,11]} # Out: {"sum":22} Concatenate strings: root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value)) # In: {"foo":["hello ", "world"]} # Out: {"result":"hello world"} Merge an array of objects into a single object: root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value)) # In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]} # Out: {"smoothie":{"apple":5,"banana":3,"orange":8}} get Extract a field value, identified via a dot path, from an object. Parameters Name Type Description path string A dot path identifying a field to obtain. Examples root.result = this.foo.get(this.target) # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"bar"} # Out: {"result":"from bar"} # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"baz"} # Out: {"result":"from baz"} index Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on. Parameters Name Type Description index integer The index to obtain from an array. Examples root.last_name = this.names.index(-1) # In: {"names":["rachel","stevens"]} # Out: {"last_name":"stevens"} It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer: root.last_byte = this.name.bytes().index(-1) # In: {"name":"foobar bazson"} # Out: {"last_byte":110} join Joins an array of strings with an optional delimiter. Parameters Name Type Description delimiter (optional) string An optional delimiter to add between each string. Examples root.joined_words = this.words.join() root.joined_numbers = this.numbers.map_each(this.string()).join(",") # In: {"words":["hello","world"],"numbers":[3,8,11]} # Out: {"joined_numbers":"3,8,11","joined_words":"helloworld"} json_path Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at https://goessner.net/articles/JsonPath/. For more complex logic, you can use Gval expressions (https://github.com/PaesslerAG/gval). Parameters Name Type Description expression string The JSONPath expression to execute. Examples root.all_names = this.json_path("$..name") # In: {"name":"alice","foo":{"name":"bob"}} # Out: {"all_names":["alice","bob"]} # In: {"thing":["this","bar",{"name":"alice"}]} # Out: {"all_names":["alice"]} root.text_objects = this.json_path("$.body[?(@.type=='text')]") # In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]} # Out: {"text_objects":[{"id":"bar","type":"text"}]} json_schema Checks a JSON schema against a value and returns the value if it matches or throws and error if it does not. Parameters Name Type Description schema string The schema to check values against. Examples root = this.json_schema("""{ "type":"object", "properties":{ "foo":{ "type":"string" } } }""") # In: {"foo":"bar"} # Out: {"foo":"bar"} # In: {"foo":5} # Out: Error("failed assignment (line 1): field `this`: foo invalid type. expected: string, given: integer") In order to load a schema from a file use the file function: root = this.json_schema(file(env("BENTHOS_TEST_BLOBLANG_SCHEMA_FILE"))) key_values Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted. Examples root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key) # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]} Filter object entries by value: root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key) # In: {"items":{"a":5,"b":15,"c":20,"d":3}} # Out: {"large_items":["c"]} keys Extracts all keys from an object and returns them as a sorted array. Examples root.foo_keys = this.foo.keys() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_keys":["bar","baz"]} Check if specific keys exist: root.has_id = this.data.keys().contains("id") # In: {"data":{"id":123,"name":"test"}} # Out: {"has_id":true} length Returns the length of an array, object, or string. Examples root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} map_each Applies a mapping to each element of an array or object. Parameters Name Type Description query query expression A query that will be used to map each element. Examples On arrays Transforms each array element using a query. Return deleted() to remove an element, or the new value to replace it: root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} On objects Transforms each object value using a query. The query receives an object with 'key' and 'value' fields for each entry: root.new_dict = this.dict.map_each(item -> item.value.uppercase()) # In: {"dict":{"foo":"hello","bar":"world"}} # Out: {"new_dict":{"bar":"WORLD","foo":"HELLO"}} map_each_key Transforms object keys using a mapping query. Parameters Name Type Description query query expression A query that will be used to map each key. Examples root.new_dict = this.dict.map_each_key(key -> key.uppercase()) # In: {"dict":{"keya":"hello","keyb":"world"}} # Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}} Conditionally transform keys: root = this.map_each_key(key -> if key.contains("kafka") { "_" + key }) # In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"} # Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"} merge Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method. Parameters Name Type Description with unknown A value to merge the target value with. Examples root = this.foo.merge(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"} Merge arrays: root.combined = this.list1.merge(this.list2) # In: {"list1":["a","b"],"list2":["c","d"]} # Out: {"combined":["a","b","c","d"]} patch Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state. Parameters Name Type Description changelog unknown The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. Examples Apply recorded changes to update an object: root.updated = this.current.patch(this.changelog) # In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]} # Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}} Restore previous state by applying inverse changes: root.restored = this.modified.patch(this.reverse_changelog) # In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]} # Out: {"restored":{"debug":true,"timeout":30}} slice Extracts a portion of an array or string. Parameters Name Type Description low integer The low bound, which is the first element of the selection, or if negative selects from the end. high (optional) integer An optional high bound. Examples root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} sort Sorts array elements in ascending order. Parameters Name Type Description compare (optional) query expression An optional query that should explicitly compare elements left and right and provide a boolean result. Examples root.sorted = this.foo.sort() # In: {"foo":["bbb","ccc","aaa"]} # Out: {"sorted":["aaa","bbb","ccc"]} Custom comparison for complex objects - return true if left < right: root.sorted = this.foo.sort(item -> item.left.v < item.right.v) # In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]} # Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]} sort_by Sorts array elements by a specified field or expression. Parameters Name Type Description query query expression A query to apply to each element that yields a value used for sorting. Examples root.sorted = this.foo.sort_by(ele -> ele.id) # In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]} # Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]} Sort by numeric field: root.sorted = this.items.sort_by(item -> item.priority) # In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]} # Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]} squash Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the .merge() method). Examples root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash() # In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]} # Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}} sum Returns the sum of numeric values in an array. Examples root.sum = this.foo.sum() # In: {"foo":[3,8,4]} # Out: {"sum":15} Works with decimals: root.total = this.prices.sum() # In: {"prices":[10.5,20.25,5.00]} # Out: {"total":35.75} unique Returns an array with duplicate elements removed. Parameters Name Type Description emit (optional) query expression An optional query that can be used in order to yield a value for each element to determine uniqueness. Examples root.uniques = this.foo.unique() # In: {"foo":["a","b","a","c"]} # Out: {"uniques":["a","b","c"]} Use a query to determine uniqueness by a field: root.unique_users = this.users.unique(u -> u.id) # In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]} # Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]} values Returns an array of all values from an object. Examples root.foo_vals = this.foo.values().sort() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_vals":[1,2]} Find max value in object: root.max = this.scores.values().sort().index(-1) # In: {"scores":{"player1":85,"player2":92,"player3":78}} # Out: {"max":92} with Returns an object where all but one or more field path arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields. If a key within a nested path does not exist then it is ignored. Examples root = this.with("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"d":"fourth","inner":{"a":"first","c":"third"}} without Returns an object with specified keys removed. Examples root = this.without("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"e":"fifth","inner":{"b":"second"}} Remove sensitive fields: root = this.without("password","ssn","creditCard") # In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"} # Out: {"email":"alice@example.com","username":"alice"} zip Zip an array value with one or more argument arrays. Each array must match in length. Examples root.foo = this.foo.zip(this.bar, this.baz) # In: {"foo":["a","b","c"],"bar":[1,2,3],"baz":[4,5,6]} # Out: {"foo":[["a",1,4],["b",2,5],["c",3,6]]} Parsing bloblang Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as file and env, or that access message information directly, such as content or json, are not enabled for dynamic Bloblang mappings. Parameters Name Type Description mapping string The mapping to execute. Examples root.body = this.body.bloblang(this.mapping) # In: {"body":{"foo":"hello world"},"mapping":"root.foo = this.foo.uppercase()"} # Out: {"body":{"foo":"HELLO WORLD"}} # In: {"body":{"foo":"hello world 2"},"mapping":"root.foo = this.foo.capitalize()"} # Out: {"body":{"foo":"Hello World 2"}} format_json Formats a value as a JSON string. Parameters Name Type Description indent string Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. no_indent bool Disable indentation. escape_html bool Escape problematic HTML characters. Examples root = this.doc.format_json() # In: {"doc":{"foo":"bar"}} # Out: { "foo": "bar" } Pass a string to the indent parameter in order to customise the indentation: root = this.format_json(" ") # In: {"doc":{"foo":"bar"}} # Out: { "doc": { "foo": "bar" } } Use the .string() method in order to coerce the result into a string: root.doc = this.doc.format_json().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"{\n \"foo\": \"bar\"\n}"} Set the no_indent parameter to true to disable indentation. The result is equivalent to calling bytes(): root = this.doc.format_json(no_indent: true) # In: {"doc":{"foo":"bar"}} # Out: {"foo":"bar"} Escapes problematic HTML characters: root = this.doc.format_json() # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo\u0026bar@benthos.dev", "name": "foo\u003ebar" } Set the escape_html parameter to false to disable escaping of problematic HTML characters: root = this.doc.format_json(escape_html: false) # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo&bar@benthos.dev", "name": "foo>bar" } format_msgpack Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed. Examples Serialize object to MessagePack and encode as hex for transmission: root = this.format_msgpack().encode("hex") # In: {"foo":"bar"} # Out: 81a3666f6fa3626172 Serialize data to MessagePack and base64 encode for embedding in JSON: root.msgpack_payload = this.data.format_msgpack().encode("base64") # In: {"data":{"foo":"bar"}} # Out: {"msgpack_payload":"gaNmb2+jYmFy"} format_xml Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array. Parameters Name Type Description indent string String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. no_indent bool Disable indentation and newlines to produce compact XML on a single line. root_tag (optional) string Custom name for the root XML element. By default, the root element name is derived from the first key in the object. Examples Serialize object to pretty-printed XML with default indentation: root = this.format_xml() # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: <foo> <bar> <baz>foo bar baz</baz> </bar> </foo> Create compact XML without indentation for smaller message size: root = this.format_xml(no_indent: true) # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: <foo><bar><baz>foo bar baz</baz></bar></foo> format_yaml Formats a value as a YAML string. Examples root = this.doc.format_yaml() # In: {"doc":{"foo":"bar"}} # Out: foo: bar Use the .string() method in order to coerce the result into a string: root.doc = this.doc.format_yaml().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"foo: bar\n"} infer_schema Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. parse_csv Parses CSV data into an array. Parameters Name Type Description parse_header_row bool Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. delimiter string The delimiter to use for splitting values in each record. It must be a single character. lazy_quotes bool If set to true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. Examples Parses CSV data with a header row: root.orders = this.orders.parse_csv() # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} Parses CSV data without a header row: root.orders = this.orders.parse_csv(false) # In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]} Parses CSV data delimited by dots: root.orders = this.orders.parse_csv(delimiter:".") # In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} Parses CSV data containing a quote in an unquoted field: root.orders = this.orders.parse_csv(lazy_quotes:true) # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]} parse_form_url_encoded Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result. Examples root.values = this.body.parse_form_url_encoded() # In: {"body":"noise=meow&animal=cat&fur=orange&fur=fluffy"} # Out: {"values":{"animal":"cat","fur":["orange","fluffy"],"noise":"meow"}} parse_json Parses a JSON string into a structured value. Parameters Name Type Description use_number (optional) bool An optional flag that when set makes parsing numbers as json.Number instead of the default float64. Examples root.doc = this.doc.parse_json() # In: {"doc":"{\"foo\":\"bar\"}"} # Out: {"doc":{"foo":"bar"}} root.doc = this.doc.parse_json(use_number: true) # In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"} # Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}} parse_logfmt Parses logfmt formatted data into an object. Examples root = this.msg.parse_logfmt() # In: {"msg":"level=info msg=\"hello world\" dur=1.5s"} # Out: {"dur":"1.5s","level":"info","msg":"hello world"} parse_msgpack Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices. Examples Parse MessagePack data from hex-encoded content: root = content().decode("hex").parse_msgpack() # In: 81a3666f6fa3626172 # Out: {"foo":"bar"} Parse MessagePack from base64-encoded field: root.decoded = this.msgpack_data.decode("base64").parse_msgpack() # In: {"msgpack_data":"gaNmb2+jYmFy"} # Out: {"decoded":{"foo":"bar"}} parse_parquet Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array. Parameters Name Type Description byte_array_as_string bool Deprecated: This parameter is no longer used. Examples Parse Parquet file data into structured objects: root.records = content().parse_parquet() Process Parquet data from a field and extract specific columns: root.users = this.parquet_data.parse_parquet().map_each(row -> {"name": row.name, "email": row.email}) parse_url Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow https://pkg.go.dev/net/url#URL, and may be expanded in future in order to present more information. Examples root.foo_url = this.foo_url.parse_url() # In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"} # Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}} root.username = this.url.parse_url().user.name | "unknown" # In: {"url":"amqp://foo:bar@127.0.0.1:5672/"} # Out: {"username":"foo"} # In: {"url":"redis://localhost:6379"} # Out: {"username":"unknown"} parse_xml Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules: Element attributes are prefixed with a hyphen (e.g., -id for an id attribute) Elements with both attributes and text content store the text in a #text field Repeated elements become arrays XML comments, directives, and processing instructions are ignored Optionally cast numeric and boolean strings to their proper types. Parameters Name Type Description cast (optional) bool Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. Examples Parse XML document into object structure: root.doc = this.doc.parse_xml() # In: {"doc":"<root><title>This is a title</title><content>This is some content</content></root>"} # Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}} Parse XML with type casting enabled to convert strings to numbers and booleans: root.doc = this.doc.parse_xml(cast: true) # In: {"doc":"<root><title>This is a title</title><number id=\"99\">123</number><bool>True</bool></root>"} # Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}} parse_yaml Parses a YAML string into a structured value. Examples root.doc = this.doc.parse_yaml() # In: {"doc":"foo: bar"} # Out: {"doc":{"foo":"bar"}} Regular expressions re_find_all Finds all matches of a regular expression in a string. Parameters Name Type Description pattern string The pattern to match against. Examples root.matches = this.value.re_find_all("a.") # In: {"value":"paranormal"} # Out: {"matches":["ar","an","al"]} root.numbers = this.text.re_find_all("[0-9]+") # In: {"text":"I have 2 apples and 15 oranges"} # Out: {"numbers":["2","15"]} re_find_all_object Finds all regex matches as objects with named groups. Parameters Name Type Description pattern string The pattern to match against. Examples root.matches = this.value.re_find_all_object("a(?P<foo>x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]} root.matches = this.value.re_find_all_object("(?m)(?P<key>\\w+):\\s+(?P<value>\\w+)$") # In: {"value":"option1: value1\noption2: value2\noption3: value3"} # Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]} re_find_all_submatch Finds all regex matches with capture groups. Parameters Name Type Description pattern string The pattern to match against. Examples root.matches = this.value.re_find_all_submatch("a(x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[["axxb","xx"],["ab",""]]} root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)") # In: {"text":"Contact: alice@example.com or bob@test.org"} # Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]} re_find_object Finds the first regex match as an object with named groups. Parameters Name Type Description pattern string The pattern to match against. Examples root.matches = this.value.re_find_object("a(?P<foo>x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":{"0":"axxb","foo":"xx"}} root.matches = this.value.re_find_object("(?P<key>\\w+):\\s+(?P<value>\\w+)") # In: {"value":"option1: value1"} # Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}} re_match Tests if a string matches a regular expression. Parameters Name Type Description pattern string The pattern to match against. Examples root.matches = this.value.re_match("[0-9]") # In: {"value":"there are 10 puppies"} # Out: {"matches":true} # In: {"value":"there are ten puppies"} # Out: {"matches":false} re_replace Replaces all regex matches with a replacement string that can reference capture groups using $1, $2, etc. Use for pattern-based transformations or data reformatting. Parameters Name Type Description pattern string The pattern to match against. value string The value to replace with. re_replace_all Replaces all regex matches with a replacement string. Parameters Name Type Description pattern string The pattern to match against. value string The value to replace with. Examples root.new_value = this.value.re_replace_all("ADD ([0-9]+)","+($1)") # In: {"value":"foo ADD 70"} # Out: {"new_value":"foo +(70)"} root.masked = this.email.re_replace_all("(\\w{2})\\w+@", "$1***@") # In: {"email":"alice@example.com"} # Out: {"masked":"al***@example.com"} SQL vector Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database. Examples Convert embeddings array to vector for pgvector storage: root.embedding = this.embeddings.vector() root.text = this.text Process ML model output into database-ready vector format: root.doc_id = this.id root.vector_embedding = this.model_output.map_each(num -> num.number()).vector() String manipulation capitalize Converts a string to title case with Unicode letter mapping. Examples root.title = this.title.capitalize() # In: {"title":"the foo bar"} # Out: {"title":"The Foo Bar"} root.name = this.name.capitalize() # In: {"name":"alice smith"} # Out: {"name":"Alice Smith"} compare_argon2 Checks whether a string matches a hashed secret using Argon2. Parameters Name Type Description hashed_secret string The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. Examples root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} compare_bcrypt Checks whether a string matches a hashed secret using bcrypt. Parameters Name Type Description hashed_secret string The hashed secret value to compare with the input. Examples root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} contains Tests if an array or object contains a value. Parameters Name Type Description value unknown A value to test against elements of the target. Examples root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} escape_html Escapes HTML special characters. Examples root.escaped = this.value.escape_html() # In: {"value":"foo & bar"} # Out: {"escaped":"foo & bar"} root.safe_html = this.user_input.escape_html() # In: {"user_input":"<script>alert('xss')</script>"} # Out: {"safe_html":"<script>alert('xss')</script>"} escape_url_path Escapes a string for use in URL paths. Examples root.escaped = this.value.escape_url_path() # In: {"value":"foo & bar"} # Out: {"escaped":"foo%20&%20bar"} root.url = "https://example.com/docs/" + this.path.escape_url_path() # In: {"path":"my document.pdf"} # Out: {"url":"https://example.com/docs/my%20document.pdf"} escape_url_query Escapes a string for use in URL query parameters. Examples root.escaped = this.value.escape_url_query() # In: {"value":"foo & bar"} # Out: {"escaped":"foo+%26+bar"} root.url = "https://example.com?search=" + this.query.escape_url_query() # In: {"query":"hello world!"} # Out: {"url":"https://example.com?search=hello+world%21"} filepath_join Joins filepath components into a single path. Examples root.path = this.path_elements.filepath_join() # In: {"path_elements":["/foo/","bar.txt"]} # Out: {"path":"/foo/bar.txt"} filepath_split Splits a filepath into directory and filename components. Examples root.path_sep = this.path.filepath_split() # In: {"path":"/foo/bar.txt"} # Out: {"path_sep":["/foo/","bar.txt"]} # In: {"path":"baz.txt"} # Out: {"path_sep":["","baz.txt"]} format Formats a value using a specified format string. Examples root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers) # In: {"name":"lance","age":37,"fingers":13} # Out: {"foo":"lance(37): 13"} root.message = "User %s has %v points".format(this.username, this.score) # In: {"username":"alice","score":100} # Out: {"message":"User alice has 100 points"} has_prefix Tests if a string starts with a specified prefix. Parameters Name Type Description value string The string to test. Examples root.t1 = this.v1.has_prefix("foo") root.t2 = this.v2.has_prefix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":true,"t2":false} has_suffix Tests if a string ends with a specified suffix. Parameters Name Type Description value string The string to test. Examples root.t1 = this.v1.has_suffix("foo") root.t2 = this.v2.has_suffix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":false,"t2":true} index_of Returns the index of the first occurrence of a substring. Parameters Name Type Description value string A string to search for. Examples root.index = this.thing.index_of("bar") # In: {"thing":"foobar"} # Out: {"index":3} root.index = content().index_of("meow") # In: the cat meowed, the dog woofed # Out: {"index":8} length Returns the length of an array, object, or string. Examples root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} lowercase Converts all letters in a string to lowercase. Examples root.foo = this.foo.lowercase() # In: {"foo":"HELLO WORLD"} # Out: {"foo":"hello world"} root.email = this.user_email.lowercase() # In: {"user_email":"User@Example.COM"} # Out: {"email":"user@example.com"} quote Wraps a string in double quotes and escapes special characters. Examples root.quoted = this.thing.quote() # In: {"thing":"foo\nbar"} # Out: {"quoted":"\"foo\\nbar\""} root.literal = this.text.quote() # In: {"text":"hello\tworld"} # Out: {"literal":"\"hello\\tworld\""} repeat Creates a string by repeating the input a specified number of times. Parameters Name Type Description count integer The number of times to repeat the string. Examples root.repeated = this.name.repeat(3) root.not_repeated = this.name.repeat(0) # In: {"name":"bob"} # Out: {"not_repeated":"","repeated":"bobbobbob"} root.separator = "-".repeat(10) # In: {} # Out: {"separator":"----------"} replace Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings. Parameters Name Type Description old string A string to match against. new string A string to replace with. replace_all Replaces all occurrences of a substring with another. Parameters Name Type Description old string A string to match against. new string A string to replace with. Examples root.new_value = this.value.replace_all("foo","dog") # In: {"value":"The foo ate my homework"} # Out: {"new_value":"The dog ate my homework"} root.clean = this.text.replace_all(" ", " ") # In: {"text":"hello world foo"} # Out: {"clean":"hello world foo"} replace_all_many Performs multiple find-and-replace operations in sequence. Parameters Name Type Description values array An array of values, each even value will be replaced with the following odd value. Examples root.new_value = this.value.replace_all_many([ "<b>", "<b>", "</b>", "</b>", "<i>", "<i>", "</i>", "</i>", ]) # In: {"value":"<i>Hello</i> <b>World</b>"} # Out: {"new_value":"<i>Hello</i> <b>World</b>"} replace_many Performs multiple find-and-replace operations in sequence using an array of [old, new] pairs. More efficient than chaining multiple replace_all calls. Use for bulk text transformations. Parameters Name Type Description values array An array of values, each even value will be replaced with the following odd value. reverse Reverses the order of characters in a string. Examples root.reversed = this.thing.reverse() # In: {"thing":"backwards"} # Out: {"reversed":"sdrawkcab"} root = content().reverse() # In: {"thing":"backwards"} # Out: }"sdrawkcab":"gniht"{ slice Extracts a portion of an array or string. Parameters Name Type Description low integer The low bound, which is the first element of the selection, or if negative selects from the end. high (optional) integer An optional high bound. Examples root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} slug Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters. Parameters Name Type Description lang (optional) string Examples Create a URL-friendly slug from a string with special characters: root.slug = this.title.slug() # In: {"title":"Hello World! Welcome to Redpanda Connect"} # Out: {"slug":"hello-world-welcome-to-redpanda-connect"} Create a slug preserving French language rules: root.slug = this.title.slug("fr") # In: {"title":"Café & Restaurant"} # Out: {"slug":"cafe-et-restaurant"} split Splits a string into an array of substrings. Parameters Name Type Description delimiter string The delimiter to split with. empty_as_null bool To treat empty substrings as null values Examples root.new_value = this.value.split(",") # In: {"value":"foo,bar,baz"} # Out: {"new_value":["foo","bar","baz"]} root.new_value = this.value.split(",", true) # In: {"value":"foo,,qux"} # Out: {"new_value":["foo",null,"qux"]} root.words = this.sentence.split(" ") # In: {"sentence":"hello world from bloblang"} # Out: {"words":["hello","world","from","bloblang"]} strip_html Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others. Parameters Name Type Description preserve (optional) unknown Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. Examples Extract plain text from HTML content: root.plain_text = this.html_content.strip_html() # In: {"html_content":"<p>Welcome to <strong>Redpanda Connect</strong>!</p>"} # Out: {"plain_text":"Welcome to Redpanda Connect!"} Preserve specific HTML elements while removing others: root.sanitized = this.html.strip_html(["strong", "em"]) # In: {"html":"<div><p>Some <strong>bold</strong> and <em>italic</em> text with a <script>alert('xss')</script></p></div>"} # Out: {"sanitized":"Some <strong>bold</strong> and <em>italic</em> text with a "} trim Removes leading and trailing characters from a string. Parameters Name Type Description cutset (optional) string An optional string of characters to trim from the target value. Examples root.title = this.title.trim("!?") root.description = this.description.trim() # In: {"description":" something happened and its amazing! ","title":"!!!watch out!?"} # Out: {"description":"something happened and its amazing!","title":"watch out"} trim_prefix Removes a specified prefix from the beginning of a string. Parameters Name Type Description prefix string The leading prefix substring to trim from the string. Examples root.name = this.name.trim_prefix("foobar_") root.description = this.description.trim_prefix("foobar_") # In: {"description":"unchanged","name":"foobar_blobton"} # Out: {"description":"unchanged","name":"blobton"} trim_suffix Removes a specified suffix from the end of a string. Parameters Name Type Description suffix string The trailing suffix substring to trim from the string. Examples root.name = this.name.trim_suffix("_foobar") root.description = this.description.trim_suffix("_foobar") # In: {"description":"unchanged","name":"blobton_foobar"} # Out: {"description":"unchanged","name":"blobton"} unescape_html Converts HTML entities back to their original characters. Examples root.unescaped = this.value.unescape_html() # In: {"value":"foo & bar"} # Out: {"unescaped":"foo & bar"} root.text = this.html.unescape_html() # In: {"html":"<p>Hello & goodbye</p>"} # Out: {"text":"<p>Hello & goodbye</p>"} unescape_url_path Unescapes URL path encoding. Examples root.unescaped = this.value.unescape_url_path() # In: {"value":"foo%20&%20bar"} # Out: {"unescaped":"foo & bar"} root.filename = this.path.unescape_url_path() # In: {"path":"my%20document.pdf"} # Out: {"filename":"my document.pdf"} unescape_url_query Unescapes URL query parameter encoding. Examples root.unescaped = this.value.unescape_url_query() # In: {"value":"foo+%26+bar"} # Out: {"unescaped":"foo & bar"} root.search = this.param.unescape_url_query() # In: {"param":"hello+world%21"} # Out: {"search":"hello world!"} unicode_segments Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners. Parameters Name Type Description segmentation_type string Type of segmentation: "grapheme", "word", or "sentence" Examples Split text into sentences (preserves trailing spaces): root.sentences = this.text.unicode_segments("sentence") # In: {"text":"Hello world. How are you?"} # Out: {"sentences":["Hello world. ","How are you?"]} Split text into grapheme clusters (handles complex emoji correctly): root.graphemes = this.emoji.unicode_segments("grapheme") # In: {"emoji":"👨👩👧👦❤️"} # Out: {"graphemes":["👨👩👧👦","❤️"]} unquote Removes surrounding quotes and interprets escape sequences. Examples root.unquoted = this.thing.unquote() # In: {"thing":"\"foo\\nbar\""} # Out: {"unquoted":"foo\nbar"} root.text = this.literal.unquote() # In: {"literal":"\"hello\\tworld\""} # Out: {"text":"hello\tworld"} uppercase Converts all letters in a string to uppercase. Examples root.foo = this.foo.uppercase() # In: {"foo":"hello world"} # Out: {"foo":"HELLO WORLD"} root.code = this.product_code.uppercase() # In: {"product_code":"abc-123"} # Out: {"code":"ABC-123"} Timestamp manipulation parse_duration Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h". Examples Parse microseconds to nanoseconds: root.delay_for_ns = this.delay_for.parse_duration() # In: {"delay_for":"50us"} # Out: {"delay_for_ns":50000} Parse hours to seconds: root.delay_for_s = this.delay_for.parse_duration() / 1000000000 # In: {"delay_for":"2h"} # Out: {"delay_for_s":7200} parse_duration_iso8601 Parses an ISO 8601 duration string into nanoseconds. Format: "P[n]Y[n]M[n]DT[n]H[n]M[n]S" or "P[n]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place). Examples Parse complex ISO 8601 duration to nanoseconds: root.delay_for_ns = this.delay_for.parse_duration_iso8601() # In: {"delay_for":"P3Y6M4DT12H30M5S"} # Out: {"delay_for_ns":110839937000000000} Parse hours to seconds: root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000 # In: {"delay_for":"PT2H"} # Out: {"delay_for_s":7200} ts_add_iso8601 Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years. Parameters Name Type Description duration string Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) Examples Add one year to a timestamp: root.next_year = this.created_at.ts_add_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"next_year":"2021-08-14T05:54:23Z"} Add a complex duration with multiple units: root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2020-01-01T00:00:00Z"} # Out: {"future_date":"2021-03-04T04:05:06Z"} ts_format Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats. Parameters Name Type Description format string The output format using Go’s reference time. tz (optional) string Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. Examples Format timestamp with custom format: root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} Format unix timestamp with timezone specification: root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC") # In: {"created_at":1597405526} # Out: {"something_at":"2020-Aug-14 11:45:26"} ts_parse Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead. Parameters Name Type Description format string The format of the input string using Go’s reference time. Examples Parse a date with abbreviated month name: root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} Parse a custom datetime format: root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)") # In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"} # Out: {"parsed":"2020-08-14T05:54:00Z"} ts_round Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. Parameters Name Type Description duration integer A duration measured in nanoseconds to round by. Examples Round timestamp to the nearest hour: root.created_at_hour = this.created_at.ts_round("1h".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_hour":"2020-08-14T06:00:00Z"} Round timestamp to the nearest minute: root.created_at_minute = this.created_at.ts_round("1m".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_minute":"2020-08-14T05:54:00Z"} ts_strftime Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats. Parameters Name Type Description format string The output format using strptime specifiers. tz (optional) string Optional timezone. Defaults to input timezone or local time for unix timestamps. Examples Format timestamp with strftime specifiers: root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} Format with microseconds using %f directive: root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26.371000"} ts_strptime Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts_parse for Go-style reference time formats instead. Parameters Name Type Description format string The format string using strptime specifiers (e.g., %Y-%m-%d). Examples Parse date with abbreviated month using strptime format: root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} Parse datetime with microseconds using %f directive: root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f") # In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}} # Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}} ts_sub Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration. Parameters Name Type Description t2 timestamp The timestamp to subtract from the target timestamp. Examples Calculate absolute duration between two timestamps: root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs() # In: {"started_at":"2020-08-13T05:54:23Z"} # Out: {"between":86400000000000} Calculate signed duration (can be negative): root.duration_ns = this.end_time.ts_sub(this.start_time) # In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"} # Out: {"duration_ns":5400000000000} ts_sub_iso8601 Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years. Parameters Name Type Description duration string Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) Examples Subtract one year from a timestamp: root.last_year = this.created_at.ts_sub_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"last_year":"2019-08-14T05:54:23Z"} Subtract a complex duration with multiple units: root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2021-03-04T04:05:06Z"} # Out: {"past_date":"2020-01-01T00:00:00Z"} ts_tz Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. Parameters Name Type Description tz string The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". Examples Convert timestamp to UTC timezone: root.created_at_utc = this.created_at.ts_tz("UTC") # In: {"created_at":"2021-02-03T17:05:06+01:00"} # Out: {"created_at_utc":"2021-02-03T16:05:06Z"} Convert timestamp to a specific timezone: root.created_at_ny = this.created_at.ts_tz("America/New_York") # In: {"created_at":"2021-02-03T16:05:06Z"} # Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"} ts_unix Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. Examples Convert RFC 3339 timestamp to unix seconds: root.created_at_unix = this.created_at.ts_unix() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000} Unix timestamp passthrough returns same value: root.timestamp = this.ts.ts_unix() # In: {"ts":1257894000} # Out: {"timestamp":1257894000} ts_unix_micro Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. Examples Convert timestamp to microseconds since epoch: root.created_at_unix = this.created_at.ts_unix_micro() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000} Preserve microsecond precision from timestamp: root.precise_time = this.timestamp.ts_unix_micro() # In: {"timestamp":"2020-08-14T11:45:26.123456Z"} # Out: {"precise_time":1597405526123456} ts_unix_milli Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. Examples Convert timestamp to milliseconds since epoch: root.created_at_unix = this.created_at.ts_unix_milli() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000} Useful for JavaScript timestamp compatibility: root.js_timestamp = this.event_time.ts_unix_milli() # In: {"event_time":"2020-08-14T11:45:26.123Z"} # Out: {"js_timestamp":1597405526123} ts_unix_nano Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. Examples Convert timestamp to nanoseconds since epoch: root.created_at_unix = this.created_at.ts_unix_nano() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000000} Preserve full nanosecond precision: root.precise_time = this.timestamp.ts_unix_nano() # In: {"timestamp":"2020-08-14T11:45:26.123456789Z"} # Out: {"precise_time":1597405526123456789} Type coercion array Converts a value to an array. Examples root.my_array = this.name.array() # In: {"name":"foobar bazson"} # Out: {"my_array":["foobar bazson"]} bool Converts a value to a boolean with optional fallback. Parameters Name Type Description default (optional) bool An optional value to yield if the target cannot be parsed as a boolean. Examples root.foo = this.thing.bool() root.bar = this.thing.bool(true) bytes Marshals a value into a byte array. Examples root.first_byte = this.name.bytes().index(0) # In: {"name":"foobar bazson"} # Out: {"first_byte":102} not_empty Ensures a value is not empty. Examples root.a = this.a.not_empty() # In: {"a":"foo"} # Out: {"a":"foo"} # In: {"a":""} # Out: Error("failed assignment (line 1): field `this.a`: string value is empty") # In: {"a":["foo","bar"]} # Out: {"a":["foo","bar"]} # In: {"a":[]} # Out: Error("failed assignment (line 1): field `this.a`: array value is empty") # In: {"a":{"b":"foo","c":"bar"}} # Out: {"a":{"b":"foo","c":"bar"}} # In: {"a":{}} # Out: Error("failed assignment (line 1): field `this.a`: object value is empty") not_null Ensures a value is not null. Examples root.a = this.a.not_null() # In: {"a":"foobar","b":"barbaz"} # Out: {"a":"foobar"} # In: {"b":"barbaz"} # Out: Error("failed assignment (line 1): field `this.a`: value is null") number Converts a value to a number with optional fallback. Parameters Name Type Description default (optional) float An optional value to yield if the target cannot be parsed as a number. Examples root.foo = this.thing.number() + 10 root.bar = this.thing.number(5) * 10 string Converts a value to a string representation. Examples root.nested_json = this.string() # In: {"foo":"bar"} # Out: {"nested_json":"{\"foo\":\"bar\"}"} root.id = this.id.string() # In: {"id":228930314431312345} # Out: {"id":"228930314431312345"} timestamp Converts a value to a timestamp with optional fallback. Parameters Name Type Description default (optional) timestamp An optional value to yield if the target cannot be parsed as a timestamp. Examples root.foo = this.ts.timestamp() root.bar = this.none.timestamp(1234567890.timestamp()) type Returns the type of a value as a string. Examples root.bar_type = this.bar.type() root.foo_type = this.foo.type() # In: {"bar":10,"foo":"is a string"} # Out: {"bar_type":"number","foo_type":"string"} root.type = this.type() # In: "foobar" # Out: {"type":"string"} # In: 666 # Out: {"type":"number"} # In: false # Out: {"type":"bool"} # In: ["foo", "bar"] # Out: {"type":"array"} # In: {"foo": "bar"} # Out: {"type":"object"} # In: null # Out: {"type":"null"} root.type = content().type() # In: foobar # Out: {"type":"bytes"} root.type = this.ts_parse("2006-01-02").type() # In: "2022-06-06" # Out: {"type":"timestamp"} Deprecated format_timestamp This method is deprecated and will be removed in a future version. Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts_strftime for strftime-style formats. Parameters Name Type Description format string The output format using Go’s reference time. tz (optional) string Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. format_timestamp_strftime This method is deprecated and will be removed in a future version. Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts_format for Go-style reference time formats. Parameters Name Type Description format string The output format using strptime specifiers. tz (optional) string Optional timezone. Defaults to input timezone or local time for unix timestamps. format_timestamp_unix This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. format_timestamp_unix_micro This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. format_timestamp_unix_milli This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. format_timestamp_unix_nano This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. parse_timestamp This method is deprecated and will be removed in a future version. Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts_strptime for strftime-style formats instead. Parameters Name Type Description format string The format of the input string using Go’s reference time. parse_timestamp_strptime This method is deprecated and will be removed in a future version. Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts_parse for Go-style reference time formats instead. Parameters Name Type Description format string The format string using strptime specifiers (e.g., %Y-%m-%d). Back to top × Simple online edits For simple changes, such as fixing a typo, you can edit the content directly on GitHub. Edit on GitHub Or, open an issue to let us know about something that you want us to change. Open an issue Contribution guide For extensive content updates, or if you prefer to work locally, read our contribution guide . Was this helpful? thumb_up thumb_down group Ask in the community mail Share your feedback group_add Make a contribution 🎉 Thanks for your feedback! Functions Arithmetic