Pastein: root@mos:/home/vic/lmd# venv/bin/lmdeploy serve api_server cyankiwi/Ministral-3-14B-Instruct-2512-AWQ-4bit --server-port 23333 --backend turbomind --model-format awq --quant-policy 8 --sessi

Загрузка данных
root@mos:/home/vic/lmd# venv/bin/lmdeploy serve api_server cyankiwi/Ministral-3-14B-Instruct-2512-AWQ-4bit   --server-port 23333   --backend turbomind   --model-format awq   --quant-policy 8   --session-len 32768   --cache-max-entry-count 0.6
Fetching 18 files: 100%|█████████████████████████████████████████████| 18/18 [13:51<00:00, 46.20s/it]
Download complete: ██████████████████████████████████████████████████████████████| 9.66GB, 7.87MB/s  /home/vic/lmd/venv/lib/python3.14/site-packages/torch/cuda/__init__.py:384: UserWarning: Found GPU0 Tesla V100-SXM2-16GB which is of compute capability (CC) 7.0.
The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
- 7.5 which supports hardware CC >=7.5,<8.0
- 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
- 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
- 9.0 which supports hardware CC >=9.0,<10.0
- 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
- 12.0 which supports hardware CC >=12.0,<13.0
Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6
  _warn_unsupported_code(d, device_cc, code_ccs)
Download complete: ██████████████████████████████████████████████████████████████| 9.66GB, 7.87MB/s
Reconstruction complete: 100%|██████████████████████████████████████████| 10.4GB / 10.4GB, 26.8MB/s
2026-09-20 16:21:02,570 - lmdeploy - WARNING - archs.py:41 - Fallback to pytorch engine because '/root/.cache/huggingface/hub/models--cyankiwi--Ministral-3-14B-Instruct-2512-AWQ-4bit/snapshots/d4401438aba6c4e523b929c1a8878d3fe8cc0bca' not supported by turbomind engine.
[transformers] The tokenizer you are loading from '/root/.cache/huggingface/hub/models--cyankiwi--Ministral-3-14B-Instruct-2512-AWQ-4bit/snapshots/d4401438aba6c4e523b929c1a8878d3fe8cc0bca' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[transformers] The tokenizer you are loading from '/root/.cache/huggingface/hub/models--cyankiwi--Ministral-3-14B-Instruct-2512-AWQ-4bit/snapshots/d4401438aba6c4e523b929c1a8878d3fe8cc0bca' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
[transformers] The tokenizer you are loading from '/root/.cache/huggingface/hub/models--cyankiwi--Ministral-3-14B-Instruct-2512-AWQ-4bit/snapshots/d4401438aba6c4e523b929c1a8878d3fe8cc0bca' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
/home/vic/lmd/venv/lib/python3.14/site-packages/torch/cuda/__init__.py:384: UserWarning: Found GPU0 Tesla V100-SXM2-16GB which is of compute capability (CC) 7.0.
The following list shows the CCs this version of PyTorch was built for and the hardware CCs it supports:
- 7.5 which supports hardware CC >=7.5,<8.0
- 8.0 which supports hardware CC >=8.0,<9.0 except {8.7}
- 8.6 which supports hardware CC >=8.6,<9.0 except {8.7}
- 9.0 which supports hardware CC >=9.0,<10.0
- 10.0 which supports hardware CC >=10.0,<11.0 except {10.1}
- 12.0 which supports hardware CC >=12.0,<13.0
Please follow the instructions at https://pytorch.org/get-started/locally/ to install a PyTorch release that supports one of these CUDA versions: 12.6
  _warn_unsupported_code(d, device_cc, code_ccs)
2026-09-20 16:21:14,986 - lmdeploy - ERROR - base.py:54 - AcceleratorError: CUDA error: no kernel image is available for execution on the device
Search for `cudaErrorNoKernelImageForDevice' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

2026-09-20 16:21:14,987 - lmdeploy - ERROR - base.py:55 - <PyTorch> check failed!
PyTorch is not available.
2026-09-20 16:21:18,208 - lmdeploy - ERROR - zmq_rpc.py:62 - PyTorch ZMQ engine process is not alive.
Traceback (most recent call last):
  File "/home/vic/lmd/venv/bin/lmdeploy", line 8, in <module>
    sys.exit(run())
             ~~~^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/cli/entrypoint.py", line 39, in run
    args.run(args)
    ~~~~~~~~^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/cli/serve.py", line 318, in api_server
    run_api_server(
    ~~~~~~~~~~~~~~^
        args.model_path,
        ^^^^^^^^^^^^^^^^
    ...<25 lines>...
        generation_config=args.generation_config,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/serve/openai/api_server.py", line 371, in serve
    server_context.async_engine = pipeline_class(
                                  ~~~~~~~~~~~~~~^
        model_path=model_path,
        ^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
        allowed_media_domains=allowed_media_domains,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        **kwargs)
        ^^^^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/serve/core/async_engine.py", line 154, in __init__
    self.engine = self._build_pytorch(model_path=model_path,
                  ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                                      backend_config=backend_config,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      trust_remote_code=trust_remote_code,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      speculative_config=speculative_config,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                      **kwargs)
                                      ^^^^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/serve/core/async_engine.py", line 219, in _build_pytorch
    return Engine.from_pretrained(model_path,
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
                                  engine_config=backend_config,
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                  speculative_config=speculative_config,
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                  trust_remote_code=trust_remote_code,
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                  **kwargs)
                                  ^^^^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/engine.py", line 281, in from_pretrained
    return build_mp_engine(
        backend=backend,
    ...<3 lines>...
        trust_remote_code=trust_remote_code
    )
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/__init__.py", line 10, in build_mp_engine
    return ZMQMPEngine(model_path, engine_config=engine_config, trust_remote_code=trust_remote_code, **kwargs)
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_engine.py", line 57, in __init__
    super().__init__()
    ~~~~~~~~~~~~~~~~^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/base.py", line 33, in __init__
    self.engine_config = self._collective_rpc('get_engine_config')
                         ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_engine.py", line 199, in _collective_rpc
    return self.rpc_client.call(func, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py", line 446, in call
    reply = self._poll_recv()
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py", line 428, in _poll_recv
    self._check_server_alive()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py", line 392, in _check_server_alive
    self._server_liveness.check_alive()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/vic/lmd/venv/lib/python3.14/site-packages/lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py", line 86, in check_alive
    raise self.new_dead_error()
lmdeploy.pytorch.engine.mp_engine.zmq_rpc.RPCServerDeadError: PyTorch ZMQ engine process is not alive.
Больше возможностей при регистрации:
