Benjamin KAHANE
- Updated
Flex Query Language helps end users to perform customised searches within both user interaces (e.g. FlexMAM) and other Dalet Flex microservices which requires search service. FQL enables you to carry out complex structured searches in Dalet Flex.
In Dalet Flex, you can carry out simple lexical text based searches by simply entering a word or phrase. You can also use the Query Language to help you carry out more complex searches. The syntax that can be used is as follows:
| Operators | Description | |
| Logical Operators | AND OR ! | Logical operators are used to search for multiple terms simultaneously. Examples: You can search for two different terms using the OR / AND operator such as: movie OR film, movie AND comedy. |
| Quotes | "" | Quotes are used to search for an exact word or phrase. You wrap the word or phrase in quotation marks. Example: description="city". |
| Tilde | ~ | The tilde character supports a "contains" search. This uses phrase matching to search in elastic search. For example, the search query q=description~"live at midnight" returns all values that include the exact phrase "live at midnight". |
| Comparison Operators | >= <= < > | Equal or greater, equal or lesser, lesser and greater operators are supported. |
| Boosting | ^ | You can boost a term using a caret (cursor) symbol. Example: description^5=city. In this example all the assets which have the word city in the description field will be more considered relevant in the search. If a SORT BY clause is provided, boosting has no effect. |
| Brackets | ( ) | Examples: description=(European City), and (pollen=yellow AND flower=red) OR flower=pink vs pollen=yellow AND (flower=red OR flower=pink) The first example searches the for the specific terms "European" and "City" within any description fields. The second pair of examples show how bracket precedence can be used in boolean queries as per normal mathematical expectations. |
| Sorting | SORT BY ASC/DESC | Enables you to sort by a given field. This includes sorting in ascending and descending order. If you do not specify an order, it will sort in ascending by default. |
Dalet Flex supports performing a semantic search on assets. To enhance content retrieval, semantic search uses queries that are similar in meaning to the actual content of the Asset Name, Description and other string metadata fields (as configured), images and keyframes rather than just matching keywords. Semantic search understands synonym recognition, phrase interpretation, entity recognition etc, allowing you to search using a more general query to find useful results.
Semantic search uses the FQL operator ai-search
ai-search(Pope Francis) performs a semantic search for Pope Francis
This operator can be used with all existing FQL operators.
ai-search(Pope Francis) AND fileAssetType=Image performs a semantic search for the phrase Pope Francis together with a filter to return only images.
(ai-search(Pope Francis) AND (Pope)) AND fileAssetType=Image returns results that semantic match the phrase "Pope Francis" and contain the term "Pope" in any attribute or searchable metadata field, along with a filter to fetch only the images.
All queries can be combined with boolean logic using AND and OR, using brackets where necessary to indicate precedence. In the absence of brackets, boolean operations are left-to-right associative, without preferring either AND or OR, i.e.:
If you want to match a given field for a range of possibilities (like an IN operation in SQL) then an operation like field=value1 OR field=value2 OR field=value3 will not generate the most efficient Elasticsearch query under the hood.
Instead, we recommend this syntax: field=(value1 OR value2 OR value3).
| FQL | Semantic meaning |
| Dalet Flex | Matches any objects with 'Dalet' and 'Flex' in any order. |
| "Dalet Flex" | Since this is quoted, the order does matter. Only objects with the words "dalet flex" adjacent and in that order will be returned. |
| Dalet Fl?x MAM | Matches any objects with 'Dalet', 'Fl?x' ('?' matches any single character) and 'mam', in any order. |
| Dalet F*x MAM | Same as above however, * matches 0 to many characters. |
| "Dalet Flex asset manage?ent" | Since this is quoted, the order matters. Only objects matching this exact order will be returned. ? matches a single character. |
| "Dalet ? asset management" | A ? on its own is used to represent a single word. Only objects containing 'Dalet' exactly one word apart from 'asset management' will be returned. |
| "Dalet * management" | A * on its own is used to represent many words. If 'Dalet' appears before 'management', the object will be returned. |
| FQL | Semantic meaning |
| "name" = "panther" | The name field is exactly "panther" |
| !("name" = "panther") | The name field is not "panther" |
| name = "panther" | Any field called name (regardless of path - could be a metadata field, for example) is exactly "panther" |
| "id" = 12345 | The id field matches 12345 |
| "created" = "2020-10-01" | The created field matches any UTC time midnight-midnight on the 1st October 2020 |
| "created" = "2020-10-01 10:59:05" | The created field matches the exact UTC time 10:59:05 on the 1st October 2020 |
| "created" = 1601536528000 | The created field matches the exact UTC time of 1601536528000 milliseconds (i.e. 7:15:28 on on the 1st October 2020) |
| "taxonomy.10027681" = 10125635/6124797/6124805 | The Location taxonomy matches the India / Gujarat / Ahmedabad hierarchy (using the ids corresponding to the taxonomy and the taxons for the query) |
| "metadata-1370.new-text-1" = "fred" | The text metadata field identified by new-text-1 - in MD definition with entity ID 1370 - is exactly "fred" |
| "new-text-1" = "fred" | The text metadata field identified by new-text-1 - in any metadata definition - is exactly "fred" |
| "new-md-field" = "fred" | The metadata field identified by new-md-field - of any metadata field type, and in any metadata definition - is exactly "fred" |
| metadata-1.drill-down = ("project1" AND "project2") | The metadata drill-down field is a list containing both "project1" AND "project2" |
| metadata-1.drill-down = ("project1" OR "project2") | The metadata drill-down contains "project1" OR "project2" |
| !(metadata-1.drill-down = ("project1" AND "project2")) | The metadata drill-down is neither "project1" OR "project2" |
When searching date fields, the date should be written in the format yyyy-MM-dd HH:mm:ss or yyyy-MM-dd surrounded by double quotes. E.g. lastModified="2017-06-22 10:24:05".
Note that there is no logical difference between quoting the value part of a Contains search, except that quoting it may help if your query contains other FQL operators, e.g. "sort by", "or", "and", "=", "(", ")", "!", "^", ">", "<", "~".
| FQL | Semantic meaning |
| "name" ~ mario | The name field contains "mario" |
| "name" ~ m*o | The name field contains any string starting with "m" and ending in "o" |
| "name" ~ "mario sort by id" | The name field contains "mario sort by id" |
| "name" ~ mario sort by id | The name field contains "mario"; results are sorted by ID, ascending |
You can boost results by using the carat character ^, which boosts any matching objects in the results. For example, in the example below you would get all objects whose name contains "film from partner" or is an image asset, but the image asset objects will appear higher in the search results.
name~"film from partner" OR objectType.name^2=image-asset.
You can search on example values using NULL.
title=NULL
Searching on the name of a metadata field will match objects referencing any metadata definition with a field of that name. e.g. If 2 metadata definitions exist, both with a field called "string-field", the FQL string-field=somevalue would match objects using either of those definitions.
In order to target a specific metadata definition in this case, you need to know the ID of the metadata definition, and then perform a search using FQL like "metadata-<definitionid>.string-field"=somevalue.
| Example | Notes |
| New | All objects that have "New" in any field |
| assetOrigin=New | All assets with an origin of "New" |
| assetOrigin=New AND objectType.name=media-asset | All media assets with an origin of "New" |
| created > "2019-08-22" | Any object created after 22nd August 2019 |
| !ownerId=5 | Any object not owned by user with ID 5 |
| (name~film OR name~placeholder) AND objectType.name=media-asset SORT BY created DESC | Any media assets whose name contains "film" or "placeholder", ordered in descending created date order |
| FQL | Semantic meaning |
| "name" = "*ther" | The name field ends with "ther" |
| "name" = "p*r" | The name field starts with "p" and ends with "r" |
| "name" = "b* p?n*" | would match e.g. "black panther" or "big panda" |
https://bitbucket.org/ooyalaflex/flex-query-language-library/src/master/README.md:
Flex Query Language is a (SQL Like) Query DSL to help end users to perform customised search. This is used by both UI (MAM) and other microservices which requires search service. Below are the sample queries that you can perform on flex resources.
FQL is best demonstrated with examples. Note that all queries can be combined with boolean logic using AND and OR, with brackets where necessary to show precedence. In the absence of brackets, boolean operations are left-to-right associative, without preferring either AND or OR, i.e.:
A OR B AND C === (A OR B) AND C
A AND B OR C === (A AND B) OR C
If you want to match a given field for a range of possibilities (like an "IN" operation in SQL) then an operation like
is not the most efficient. Instead we recommend you to use this syntax:
| FQL | Semantic meaning |
|---|---|
The Avengers |
Matches any objects with 'the' and 'avengers' in any order. |
"The Avengers" |
Since this is quoted, the order does matter. Only objects with "the avengers" will be returned. |
Avengers Ultr?n age |
Matches any objects with 'Avengers', 'Ultr?n' ('?' matches a single character) and 'age', in any order. |
Avengers U*n age |
Same as above however, '*' matches 0 to many characters. |
"Avengers age of Ult?on" |
Since this is quoted, the order does matter. Only objects matching this exact order will be returned. '?' matches a single character. |
"Avengers ? of Ultron" |
A '?' on its own is used to represent a single word. Only objects containing 'Avengers' exactly one word apart from 'of Ultron' will be returned. |
"Avengers * Ultron" |
A '*' on its own is used to represent many words. If 'Avengers' appears BEFORE 'Ultron', the object will be returned. |
Note that if the flex/flex-searchelastic-service/featureToggles/wrapFreeTextQueriesWithWildcards Consul KV is not set to false, then all freetext searches will become an ORed combination of: - the old freetext searching behaviour - the old freetext searching behaviour, but with a * character automatically prepended and appended to turn it into a wildcard query.
Customers wishing to explicitly revert to the old behavior may do so be explicitly setting the feature toggle to false.
Unless you deliberately want to match any possible field with a given name, we strongly recommend quoting the field you are targeting, e.g. "name" ~ blah instead of name ~ blah. The first query would only match the top-level object name; the latter query could also return results if there are any object sub-fields or metadata fields called name matching the criteria.
| FQL | Semantic meaning |
|---|---|
!("objectType.id"=1) |
Return all objects which aren't media assets (i.e. their object type is not 1) |
"metadataFullText"=NULL |
The "metadataFullText" field is absent (i.e. the object has no metadata) |
!("metadataFullText"=NULL) |
The "metadataFullText" field is present (i.e. the object has some metadata) |
!(metadata-1.drill-down=NULL) |
The metadata drill-down field is present and not null |
metadata-1.drill-down=NULL |
The metadata drill-down field is either absent, null or [] |
Note that if the flex/flex-searchelastic-service/featureToggles/nullMatchesEmptyStrings Consul KV is set to true, then =NULL FQL clauses will match not only missing fields, but also empty strings (in the case of textual fields).
| FQL | Semantic meaning |
|---|---|
"name" = "panther" |
The name field is exactly "panther" |
name = "panther" |
Any field called name (regardless of path - could be a metadata field, for example) is exactly "panther" |
"name" = "*ther" |
(wildcarding) The name field ends with "ther" |
"name" = "p*r" |
(wildcarding) The name field starts with "p" and ends with "r" |
"name" = "b* p?n*" |
(wildcarding) would match e.g. "black panther" or "big panda" |
"id" = 12345 |
The id field matches 12345 |
"created" = "2020-10-01" |
The created field matches any UTC time midnight-midnight on the 1st October 2020 |
"created" = "2020-10-01 10:59:05" |
The created field matches the exact UTC time 10:59:05 on the 1st October 2020 |
"created" = 1601536528000 |
The created field matches the exact UTC time of 1601536528000 milliseconds (i.e. 7:15:28 on on the 1st October 2020) |
"taxonomy.10027681" = 10125635/6124797/6124805 |
The Location taxonomy matches the India / Gujarat / Ahmedabad hierarchy (using the ids corresponding to the taxonomy and the taxons for the query) |
"metadata-1370.var-text-new-text-1" = "fred" |
The text metadata field identified by new-text-1 - in MD definition with entity ID 1370 - is exactly "fred" |
"var-text-new-text-1" = "fred" |
The text metadata field identified by new-text-1 - in any metadata definition - is exactly "fred" |
"new-md-field" = "fred" |
The metadata field identified by new-md-field - of any metadata field type, and in any metadata definition - is exactly "fred" |
metadata-1.drill-down = ("project1" AND "project2") |
The metadata drill-down field is a list containing both "project1" AND "project2" |
metadata-1.drill-down = ("project1" OR "project2") |
The metadata drill-down contains "project1" OR "project2" |
!(metadata-1.drill-down = ("project1" AND "project2")) |
The metadata drill-down is neither "project1" OR "project2" |
I've included various sorting examples here, for good measure.
| FQL | Semantic meaning |
|---|---|
"created">="2020-10-01" |
The created field matches any UTC time from midnight on the morning of the 1st October 2020 onwards |
"created">="2020-10-01 00:00:00" AND "created"<"2020-10-01 12:00:00" |
The created field matches any UTC time in the morning of the 1st October 2020 |
"id" < 1000 |
The id field is less than 1000 |
"id" < 100 SORT BY name |
The id field is less than 100. Results are sorted by name (ascending by default) |
"id" < 100 SORT BY name ASC |
The id field is less than 100. Results are sorted by name, ascending |
"id" < 100 SORT BY name DESC |
The id field is less than 100. Results are sorted by name, descending |
"id" > 1000 SORT BY accountId DESC, name ASC |
The id field is greater than 1000. Results are sorted by accountId, descending, and then name, ascending |
"id" < 100 SORT BY name ASC MISSING FIRST |
The id field is less than 100. Results are sorted by name, ascending, with results that have no value for this field coming first |
"id" < 100 SORT BY name ASC MISSING LAST |
The id field is less than 100. Results are sorted by name, ascending, with results that have no value for this field coming last |
"id" < 100 SORT BY name ASC MISSING FIRST, id DESC MISSING LAST |
The id field is less than 100. Results are sorted by name, ascending, with results that have no value for this field coming first and then sorted by id, descending, with results that have no value for this field coming last |
Note that there is no logical difference between quoting the value part of a Contains search, except that quoting it may help if your query contains other FQL operators, e.g. "sort by", "or", "and", "=", "(", ")", "!", "^", ">", "<", "~".
| FQL | Semantic meaning |
|---|---|
"name" ~ mario |
The name field contains "mario" |
"name" ~ m*o |
The name field contains any string starting with "m" and ending in "o". |
"name" ~ "mario sort by id" |
The name field contains "mario sort by id". |
"name" ~ mario sort by id |
The name field contains "mario". Results are sorted by id, ascending |
| FQL | Semantic meaning |
|---|---|
ancestors(ancestors.levelsBelowRoot = 1 AND ancestors.objectType.id = 10001) |
The ancestors nested entry where any of subfield levelsBelowRoot is "1" and objectType.id is "10001". |
ancestors(ancestors.name = "Tarak mehta ka ulta chasma" AND ancestors.levelsBelowRoot = 1 AND ancestors.objectType.id = 10001) |
The ancestors nested entry where any of subfield name is "Tarak mehta ka ulta chasma" and levelsBelowRoot is "1" and objectType.id is "10001". |
ancestors(ancestors.name = Tarak AND ancestors.levelsBelowRoot = 1 AND ancestors.objectType.id = 10001) |
The ancestors nested entry where any of subfield name contains "Tarak" and levelsBelowRoot is "1" and objectType.id is "10001". |
ancestors(name = Tarak AND displayName = Tarak) |
The ancestors nested entry where any of subfield name contains "Tarak" and displayName contains "Tarak". |
ancestors("name" ~ tarak AND "displayName" = Tarak) |
The ancestors nested entry where immediate subfield name contains "tarak" and displayName contains "Tarak". |
The following pieces of functionality are not yet implemented, but they are under consideration for future development.
| Semantic meaning | FQL notes | Field types | Ticket |
|---|---|---|---|
The id field is between 10 and 20 |
"id" >= 10 AND "id" <= 20 already works. Enhancement considered a "nice to have". |
numeric, date | FLEX-51883 |
For more examples, please refer the following test files.
FieldMappingTest.javaMappingServiceTest.javaFieldMappingsComparatorTest.java
Comments
0 comments
Please sign in to leave a comment.