Definition of tokenize in US English:


(British tokenise)


[with object]
  • 1Computing
    Substitute a randomly generated identifier for (a sensitive piece of data) in order to prevent unauthorized access.

    ‘sensitive data has been tokenized or strongly encrypted’
    ‘tokenized payment systems’
  • 2Linguistics
    Break (text) into individual linguistic units.

    ‘our text gets tokenized into terms’
    ‘check if words are tokenized and normalized correctly’
  • 3Treat (a member of a minority group) as if they have been chosen by way of tokenism.

    ‘there was immense diversity and no one was tokenized’