fasternet-exp1.log 17 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889
  1. nohup: ignoring input
  2. [W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
  3. [04 07:01:36 <frozen super_pulsar.proto.configuration_super_pulsar_manip>:39] WRN migrate all stand-alone args into a single task
  4. [04 07:01:36 <frozen super_pulsar.proto.configuration_super_pulsar_manip>:159] set task task_0's 0th input model path as /root/axera/axera-quan-hjj/model/fasternet_size-256.onnx
  5. [04 07:01:36 <frozen super_pulsar.proto.configuration_super_pulsar_manip>:178] set task task_0's 0th output model path as /root/axera/axera-quan-hjj/joint/fasternet-handpose.joint
  6. [04 07:01:36 <frozen super_pulsar.proto.configuration_super_pulsar_manip>:297] set task task_0's pulsar_conf.output_dir as /root/axera/axera-quan-hjj
  7. /opt/venv/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  8. return torch._C._cuda_getDeviceCount() > 0
  9. [04 07:01:37 <frozen super_pulsar.func_wrappers.wrapper_pulsar_build>:17] planning task task_0
  10. [04 07:01:37 <frozen super_pulsar.func_wrappers.pulsar_build.neuwizard_step>:459] WRN affine_preprocess at QAT model compiling is deprecated, insert enforce_integers at front, please use scale_to_integers instead.
  11. [04 07:01:37 <frozen super_pulsar.func_wrappers.wrapper_pulsar_build>:340] ################## Running task task_0 ##################
  12. [04 07:01:37 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:30] python3 /root/python_modules/super_pulsar/super_pulsar/toolchain_wrappers/wrapper_neuwizard.py --config /tmp/tmp5q69qwrl.prototxt
  13. [04 07:01:38 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] [W Context.cpp:69] Warning: torch.set_deterministic is in beta, and its design and functionality may change in the future. (function operator())
  14. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] /opt/venv/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  15. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] return torch._C._cuda_getDeviceCount() > 0
  16. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] ONNX Model Version 12 for "/root/axera/axera-quan-hjj/model/fasternet_size-256.onnx"
  17. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning load step finished; elapsed time: 0.01s
  18. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step finished; elapsed time: 0.00s
  19. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step finished; elapsed time: 0.00s
  20. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step finished; elapsed time: 0.00s
  21. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "native" finished; elapsed time: 0.07s
  22. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "native_no_bn" finished; elapsed time: 0.00s
  23. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "pretransformed" finished; elapsed time: 0.60s
  24. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning calibrate step finished; elapsed time: 0.00s
  25. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "transformed" finished; elapsed time: 32.91s
  26. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "posttransformed" finished; elapsed time: 4.36s
  27. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "magma" finished; elapsed time: 0.39s
  28. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "magma_validified" finished; elapsed time: 0.10s
  29. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning transform step to "lava_with_rtv" finished; elapsed time: 1.84s
  30. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning dump_joint_model step finished; elapsed time: 0.00s
  31. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning evaluate step finished; elapsed time: 0.00s
  32. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning ir_bit_macs step finished; elapsed time: 0.00s
  33. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning ir_bit_float_params step finished; elapsed time: 0.00s
  34. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Planning ir_bit_quantized_params step finished; elapsed time: 0.00s
  35. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Loading model finished; elapsed time: 0.00s
  36. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "onnx_step_1" finished; elapsed time: 0.00s
  37. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "onnx_step_2" finished; elapsed time: 0.00s
  38. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "onnx" finished; elapsed time: 0.00s
  39. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "native" finished; elapsed time: 0.03s
  40. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "native_no_bn" finished; elapsed time: 0.00s
  41. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Transforming to "pretransformed" finished; elapsed time: 0.15s
  42. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Calibrating finished; elapsed time: 0.02s
  43. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] Traceback (most recent call last):
  44. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "</root/python_modules/super_pulsar/super_pulsar/toolchain_wrappers/wrapper_neuwizard.py>", line 3, in <module>
  45. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>", line 98, in <module>
  46. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>", line 58, in convert
  47. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen schedule.base>", line 40, in run
  48. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen schedule.steps.calibrate>", line 47, in run
  49. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen calibration.calibrate>", line 127, in calibrate
  50. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "<frozen backends.nn>", line 45, in __init__
  51. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] File "/opt/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
  52. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] self.__class__.__name__, "\n\t".join(error_msgs)))
  53. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] RuntimeError: Error(s) in loading state_dict for NeuwizardModule:
  54. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_29.weight: copying a param with shape torch.Size([10, 10, 3, 3]) from checkpoint, the shape in current model is torch.Size([10, 20, 3, 3]).
  55. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_27.weight: copying a param with shape torch.Size([80, 40, 1, 1]) from checkpoint, the shape in current model is torch.Size([80, 30, 1, 1]).
  56. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_33.weight: copying a param with shape torch.Size([20, 20, 3, 3]) from checkpoint, the shape in current model is torch.Size([20, 40, 3, 3]).
  57. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_31.weight: copying a param with shape torch.Size([160, 80, 1, 1]) from checkpoint, the shape in current model is torch.Size([160, 60, 1, 1]).
  58. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_36.weight: copying a param with shape torch.Size([20, 20, 3, 3]) from checkpoint, the shape in current model is torch.Size([20, 40, 3, 3]).
  59. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_34.weight: copying a param with shape torch.Size([160, 80, 1, 1]) from checkpoint, the shape in current model is torch.Size([160, 60, 1, 1]).
  60. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_40.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  61. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_38.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  62. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_43.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  63. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_41.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  64. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_46.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  65. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_44.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  66. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_49.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  67. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_47.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  68. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_52.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  69. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_50.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  70. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_55.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  71. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_53.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  72. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_58.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  73. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_56.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  74. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_61.weight: copying a param with shape torch.Size([40, 40, 3, 3]) from checkpoint, the shape in current model is torch.Size([40, 80, 3, 3]).
  75. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_59.weight: copying a param with shape torch.Size([320, 160, 1, 1]) from checkpoint, the shape in current model is torch.Size([320, 120, 1, 1]).
  76. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_362:conv1.weight: copying a param with shape torch.Size([80, 0, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 80, 3, 3]).
  77. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_63.weight: copying a param with shape torch.Size([640, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([640, 240, 1, 1]).
  78. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_363:conv1.weight: copying a param with shape torch.Size([80, 0, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 80, 3, 3]).
  79. [04 07:02:19 <frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>:36] DBG [neuwizard] size mismatch for op_66.weight: copying a param with shape torch.Size([640, 320, 1, 1]) from checkpoint, the shape in current model is torch.Size([640, 240, 1, 1]).
  80. Traceback (most recent call last):
  81. File "/root/python_modules/super_pulsar/pulsar", line 27, in <module>
  82. main()
  83. File "/root/python_modules/super_pulsar/pulsar", line 21, in main
  84. args.func(args)
  85. File "<frozen super_pulsar.func_wrappers.wrapper_pulsar_build>", line 410, in <lambda>
  86. File "<frozen super_pulsar.func_wrappers.wrapper_pulsar_build>", line 343, in main_build
  87. File "<frozen super_pulsar.func_wrappers.pulsar_build.neuwizard_step>", line 539, in run
  88. File "<frozen super_pulsar.toolchain_wrappers.wrapper_neuwizard>", line 43, in convert_cli
  89. RuntimeError: neuwizard failed