Stages Documentation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
712:configure_stages [2026/03/05 09:59] – Prinz, Patrick712:configure_stages [2026/09/23 08:23] (current) – [AI Feature Tokenizer Configuration] Prinz, Patrick
Line 625: Line 625:
   ai.model.url = [azure_ai_ressource_url]   ai.model.url = [azure_ai_ressource_url]
   ai.model.key = [azure_ai_ressource_key]   ai.model.key = [azure_ai_ressource_key]
-  ai.security.user = [custom_name_to_identify_stages_to_azure]+  ai.security.user = [arbitrary_custom_name_to_identify_stages_to_azure]
  
   ai.translateservice.url = [azure_translate_service_url]   ai.translateservice.url = [azure_translate_service_url]
Line 693: Line 693:
 ===== AI Feature Tokenizer Configuration ===== ===== AI Feature Tokenizer Configuration =====
  
-LLM that are NOT using the standard byte-pair-encoding algorithm CANNOT be used with the Azure tokenizer. Stages implements the HuggingFaceTokenizer from deep java library to support customized tokenizer.+>Attention - Introduced with 7.12.8.1: \\ If openAiCompatible Adapter is used and no Tokenizer is provided via the conf folder, Stages falls back to a generic o200k based Tokenizer that is suitable for most of the late openAi compatible models. 
 + 
 +LLM that are NOT using the standard byte-pair-encoding algorithm or are provided via the openAICompatible adapter CANNOT be used with the Azure tokenizer. Stages implements the HuggingFaceTokenizer from deep java library to support customized tokenizer.
  
 Add the property “tokenizerFolder” to the cg-host section Add the property “tokenizerFolder” to the cg-host section
Line 702: Line 704:
   </cg-host>   </cg-host>
  
- In methodpark\stages\conf+In methodpark\stages\conf
  
 Create a folder tokenizer\custom-model-v1 ← make sure the folder name exactly matches the value in the tokenizerFolder property Create a folder tokenizer\custom-model-v1 ← make sure the folder name exactly matches the value in the tokenizerFolder property