Differences

This shows you the differences between two versions of the page.

--- 712:configure_stages [2026/02/24 11:22] – [AI Feature Configuration: How to create Azure OpenAI services] Doczkal, Tristan
+++ 712:configure_stages [2026/03/25 09:44] (current) – [AI Feature Tokenizer Configuration] Prinz, Patrick
@@ Line 415: / Line 415: @@
   * Default file path on linux server: /opt/stages/conf/config.xml
-AI Chatbot Assistant feature:
+**AI Chatbot Assistant feature:**
 <code xml>
@@ Line 485: / Line 485: @@
 </code>
-AI Content Generation feature:
+**AI Content Generation feature (Azure):**
 <code xml>
@@ Line 527: / Line 527: @@
 </code>
-AI translate feature configuration:
+**When using models newer than gpt4o (like gpt5)**
+>The temperature has to be adjusted to the default value of 1. Add <cg-property name="temperature" value="1" /> to the relevant <cg-hosts>
+>The <chatbot-property name="maxTokens" value="500"></chatbot-property> has to be replaced with <chatbot-property name="maxCompletionTokens" value="500"></chatbot-property>
+**AI content generation feature (OpenAI compatible):**
+Stages 7.12.4 and later can be configured to use a LLM and embedding model that are hosted outside of Azure, e. g. by the customers themselves. We currently support installations that use an API that is compatible to that of OpenAI. The configuration is similar, but differs in which properties are available.
+Important differences to Azure:
+  * the cg-type.name “openAICompatible”
+  * A custom tokenizer configuration is required
+  * If run on consumer hardware, it will probably be necessary to increase the default timeout.
+<code ->
+<cg>
+    <cg-type name="openAICompatible">
+        <cg-host ident="local-llm-small" url="${ai.model.url}" displayName="Small self Hosted LLM">
+            <cg-property name="deployment_name" value="self-hosted-small-llm-name"/>
+            <cg-property name="key" value="api-key"/>
+            <cg-property name="tokenizerFolder" value="small-tokenizer-name"/>
+            <cg-property name="maxTokens" value="10000"/>
+            <cg-property name="timeoutInMinutes" value="10"/>
+        </cg-host>
+        <cg-host ident="local-llm" url="${ai.model.url}" displayName="Large self Hosted LLM">
+            <cg-property name="deployment_name" value="self-hosted-large-llm-name"/>
+            <cg-property name="key" value="api-key"/>
+            <cg-property name="tokenizerFolder" value="large-tokenizer-name"/>
+            <cg-property name="maxTokens" value="40000"/>
+            <cg-property name="timeoutInMinutes" value="10"/>
+        </cg-host>
+        <cg-host ident="local-embedding" url="${ai.model.url}" displayName="Self Hosted Embedding">
+            <cg-property name="deployment_name" value="self-hosted-embedding-model-name"/>
+            <cg-property name="key" value="api-key"/>
+            <cg-property name="isEmbeddingModel" value="true"/>
+            <cg-property name="timeoutInMinutes" value="10"/>
+        </cg-host>
+    </cg-type>
+</cg>
+</code>
+>The ai.model.url typically ends in “/v1”, like this: ''[[http://123.45.67.89:12345/v1]]''
+**AI translate feature configuration:**
 **Version**
@@ Line 537: / Line 582: @@
 The request type differs depending on the type of connection you’re using to the service.
-__For globally accessible services the configuration is:__ ''<cg-property name="request_type" value="translate"/>''
+For globally accessible services the configuration is:
-__For virtual network access (mind the v in front of 3.0):__ ''<cg-property name="request_type" value="translator/text/v3.0/translate"/>''
+<code ->
+<cg-property name="request_type" value="translate"/>
+</code>
+For virtual network access (mind the v in front of 3.0):
+<code ->
+<cg-property name="request_type" value="translator/text/v3.0/translate"/>
+</code>
 **Region**
@@ Line 577: / Line 630: @@
   ai.translateservice.key = [azure_translate_service_key]
 A / might be needed after the translate service URL while the AI model URL does not need one.
 If you use Stages as managed service and you don't have an own AI subscription, please contact UL for an subscription offer.
@@ Line 583: / Line 636: @@
 ==== Restart Stages service ====
 Please stop the Stages service after saving the changes, run the update.bat or .sh script located under /methodpark/stages/bin/ and restart Stages.
 ===== AI Feature Configuration: How to create Azure OpenAI services =====
-. Login to Azure-Management-Portal and navigate to "Azure OpenAI" service:\\ \\  {{ :712:create_azure_openai_services_01.jpg?400&direct }}\\  2. Create a new service within the OpenAI service:\\ \\  {{ :712:create_azure_openai_services_02.jpg?400&direct }}\\  3. Define basic information for the resource - e.g. select region:\\ \\  {{ :712:create_azure_openai_services_03.jpg?400&direct }}\\  4. Create resource: click "Create":\\ \\  {{ :712:create_azure_openai_services_04.jpg?400&direct }}\\  5. Select resource and navigate to "AI Foundry portal". Required AI models need to be configured here:\\ \\  {{ :712:create_azure_openai_services_05.jpg?400&direct }}\\  6. Within "AI foundry portal" these three models need to be deployed:\\ \\  {{ :712:create_azure_openai_services_06.jpg?400&direct }}\\  {{ :712:create_azure_openai_services_07.jpg?400&direct }}\\  7. Within the resource navigate to "Keys and Endpoints":\\ \\  {{ :712:create_azure_openai_services_08.jpg?200&direct }}\\  8. Here the required access data for Stages can be exported (URL, Key):\\ \\  {{ :712:create_azure_openai_services_09.jpg?400&direct }}\\  9. Afterwards create another resource for the translator-services:\\ \\  {{ :712:create_azure_openai_services_10.jpg?400&direct }}\\  {{ :712:create_azure_openai_services_11.jpg?400&direct }}\\  10. Define name and region:\\ \\  {{ :712:create_azure_openai_services_12.jpg?400&direct }}\\  11. Leave system-identitiy-switch turned "off":\\ \\  {{ :712:create_azure_openai_services_13.jpg?400&direct }}\\  12. Create resource: click "Create":\\ \\  {{ :712:create_azure_openai_services_14.jpg?400&direct }}\\  13. Export Stages access data (Key, Endpoint):\\ \\  {{ :712:create_azure_openai_services_15.jpg?400&direct }}\\  How to test if given credentials work? Please execute the command below in a command shell on the server that Stages is installed on:
+. Login to Azure-Management-Portal and navigate to "Azure OpenAI" service:\\ \\  {{ :712:create_azure_openai_services_01.jpg?400&direct }}\\  2. Create a new service within the OpenAI service:\\ \\  {{ :712:create_azure_openai_services_02.jpg?400&direct }}\\  3. Define basic information for the resource - e.g. select region:\\ \\  {{ :712:create_azure_openai_services_03.jpg?400&direct }}\\  4. Create resource: click "Create":\\ \\  {{ :712:create_azure_openai_services_04.jpg?400&direct }}\\  5. Select resource and navigate to "AI Foundry portal". Required AI models need to be configured here:\\ \\  {{ :712:create_azure_openai_services_05.jpg?400&direct }}\\  6. Within "AI foundry portal" these three models need to be deployed:\\ \\  {{ :712:create_azure_openai_services_06.jpg?400&direct }}\\  {{ :712:create_azure_openai_services_07.jpg?400&direct }}\\  7. Within the resource navigate to "Keys and Endpoints":\\ \\  {{ :712:create_azure_openai_services_08.jpg?200&direct }}\\  8. Here the required access data for Stages can be exported (URL, Key):\\ \\  {{ :712:create_azure_openai_services_09.jpg?400&direct }}\\  9. Afterwards create another resource for the translator-services:\\ \\  {{ :712:create_azure_openai_services_10.jpg?400&direct }}\\  {{ :712:create_azure_openai_services_11.jpg?400&direct }}\\  10. Define name and region:\\ \\  {{ :712:create_azure_openai_services_12.jpg?400&direct }}\\  11. Leave system-identitiy-switch turned "off":\\ \\  {{ :712:create_azure_openai_services_13.jpg?400&direct }}\\  12. Create resource: click "Create":\\ \\  {{ :712:create_azure_openai_services_14.jpg?400&direct }}\\  13. Export Stages access data (Key, Endpoint):\\ \\  {{ :712:create_azure_openai_services_15.jpg?400&direct }}\\  If you want to check if Stages can reach your AI resources please execute the command below in a command shell on the server that Stages is installed on:
+Linux:
-Linux
   curl -X POST 'https://openai-methodpark-prod-msc-plc.openai.azure.com/openai/deployments/gpt-4o-mini/chat/completions?api-version=2025-01-01-preview' -H 'Content-Type: application/json' -H 'api-key: xxxxxxx' -d '{ "messages": [ { "role": "user", "content": "Hello" } ], "max_tokens": 1 }'
-Windows
+Windows:
   curl -X POST "https://openai-methodpark-prod-msc-plc.openai.azure.com/openai/deployments/gpt-4o-mini/chat/completions?api-version=2025-01-01-preview" -H "Content-Type: application/json" -H "api-key: xxxxxxx" -d "{ \"messages\": [ { \"role\": \"user\", \"content\": \"Hello\" } ], \"max_tokens\": 1 }"
+If a Proxy is being used add the following parameters to the curl command:
+<code ->
+--proxy "http://ROXYSERVER:PORT"
+--proxy-user "USERNAME:PASSWORD"
+</code>
 ===== AI Feature Proxy Configuration: How to use HTTP proxies with AI Ressources =====
@@ Line 601: / Line 663: @@
 >__! This has been introduced with Stages 7.12.4.1 !__
-<code ->
+<code xml>
 <proxies>
   <proxy-host ident="<proxy_ident>" hostName="<proxy_host>" port="<the_port_the_proxy_listens_on>">
@@ Line 612: / Line 674: @@
 To use the proxy configuration with the Stages AI configuration the proxy property has to be added to ALL the affected cg-hosts:
-<code ->
+<code xml>
 <cg-property name="proxy" value="<proxy_ident>" />
 </code>
@@ Line 618: / Line 681: @@
 All affected cg-host sections should look like this afterwards:
-<code ->
+<code xml>
 <cg-host ident="chatModel" url="${ai.model.url}" displayName="dummy_display_name">
   <cg-property name="user" ... />
@@ Line 626: / Line 689: @@
   <cg-property name="proxy" value="<proxy_ident>" />
 <cg-host/>
+</code>
+===== AI Feature Tokenizer Configuration =====
+LLM that are NOT using the standard byte-pair-encoding algorithm or are provided via the openAICompatible adapter CANNOT be used with the Azure tokenizer. Stages implements the HuggingFaceTokenizer from deep java library to support customized tokenizer.
+Add the property “tokenizerFolder” to the cg-host section
+   <cg-host ident="custom-model-ident" url="${ai.model.url}" displayName="custom-model">
+    <cg-property name=".... />
+    <cg-property name="tokenizerFolder" value="custom-model-v1"/>
+  </cg-host>
+ In methodpark\stages\conf
+Create a folder tokenizer\custom-model-v1 ← make sure the folder name exactly matches the value in the tokenizerFolder property
+Add tokenizer.json and tokenizer_config.json
+**Example for using DeepSeek V3.2**
+. cg-host configuration in config.xml
+<code ->
+<cg-type name="openAICompatible">
+ <cg-host ident="training-model" url="${ai.model.url}" displayName="DeepSeek V3.2 for Chat">
+  <cg-property name="deployment_name" value="DeepSeek_V3_2"/>
+  <cg-property name="tokenizerFolder" value="DeepSeek_V3_2"/>
+  ...
+</cg-host>
+<cg-type />
+</code>
+. chatbot configuration in config.xml
+>You need to change all chatbot steps that shall use the configured LLM
+<code ->
+default:
+<chatbot-step ident="summaries" type="azureOpenAI" host="gpt-4o-mini">
+custom configuration:
+<chatbot-step ident="summaries" type="openAICompatible" host="training-model">
+</code>
+. Add tokenizer configuration:
+As the custom models are not processed by the standard azure tokenizer, there has to be a custom configuration added and put into a folder in conf/tokenizer that matches the config.xml (e.g. conf/tokenizer/DeepSeek_V3_2). The custom tokenizer can use tokenizer.json and tokenizer_config.json files.
+For lots of models various configurations can be downloaded from Huggingface.co, e.g.:
+  * [[https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/tokenizer.json]]
+  * [[https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/tokenizer_config.json]]
+To make the tokenizer_config.json work with Stages a few small adjustments have to be made
+>In tokenizer_config.json look up objects (e.g. bos_token) and just use the value from the “content” attribute value instead of the whole object. See the following file examples:
+**Original tokenizer_config.json :**
+<code ->
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": true,
+  "model_max_length": 16384,
+  "pad_token": {
+    "__type": "AddedToken",
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sp_model_kwargs": {},
+  "unk_token": null,
+  "tokenizer_class": "LlamaTokenizerFast",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\\n\\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' in message %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls'] %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{'<｜Assistant｜>' + message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' not in message %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜><think>\\n'}}{% endif %}"
+}
+</code>
+**Updated tokenizer_config.json**
+<code ->
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "legacy": true,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "unk_token": null,
+  "tokenizer_class": "LlamaTokenizerFast",
+  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\\n\\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' in message %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls'] %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{'<｜Assistant｜>' + message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' not in message %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜><think>\\n'}}{% endif %}"
+}
 </code>