Trouble Shooting

ValueError: last step must be > 0. It is 0

현상: fpenet train ~실행했을 때 발생한 에러. 경로 제대로 맞춰주었음
원인: gt 경로가 잘못되어서 data_convert 과정에서 data.tfrecord 파일이 제대로 저장이 안되었음. gt 파일은 이미지 폴더 안에 있어야 함

에러

INFO    2022-08-16 07:16:15,401| driveix.fpenet.models.fpenet_basemodel: Loading weights from pretrained model file. /workspace/tao-experiments/fpenet/pretrained_models/fpenet_vtrainable_v1.0/model.tlt
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py", line 141, in <module>
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py", line 130, in main
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/trainers/fpenet_trainer.py", line 150, in build
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/optimizers/adam_optimizer.py", line 45, in build
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/learning_rate_schedules/learning_rate_schedule.py", line 60, in learning_rate_tensor
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/learning_rate_schedules/softstart_annealing_schedule.py", line 107, in get_tensor
ValueError: last step must be > 0. It is 0

AttributeError: 'str' object has no attribute 'decode’

현상: fpenet train ~실행했을 때 발생한 에러. 경로 제대로 맞춰주었음
해결 → 참고
```
pip install h5py==2.10.0
```

에러

ommon/utilities/tlt_utils.py", line 150, in decode_to_keras
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 224, in _deserialize_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'
Traceback (most recent call last):
  File "/usr/local/bin/fpenet", line 8, in <module>
    sys.exit(main())

TypeError: load() missing 1 required positional argument: 'Loader’

현상: fpenet dataset_convert -e dataset_config.yaml실행했을 때 발생한 에러. dataset_config.yaml는 경로 제대로 맞춰주었음
원인: pyyaml 버전 문제
해결: pyyaml의 버전을 6.0에서 5.4.1로 낮춤

ModuleNotFoundError: No module named 'third_party’

현상: fpenet dataset_convert -e dataset_config.yaml실행했을 때 발생한 에러. dataset_config.yaml는 경로 제대로 맞춰주었음

해결: /workspace는 fpenet 명령어를 사용할 작업 경로

export PYTHONPATH=/workspace:/opt/nvidia/third_party/nvml:/opt/nvidia:/workspace

계속 저장하고 싶을 경우

echo "export PYTHONPATH=\"/workspace:/opt/nvidia/third_party/nvml:/opt/nvidia:/workspace\"" >> ~/.bashrc && source ~/.bashrc

에러

  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/hooks/utils.py", line 11, in <module>
  File "/usr/local/lib/python3.6/dist-packages/modulus/distribution/__init__.py", line 8, in <module>
    from modulus.distribution.distribution import Distributor, HorovodDistributor, hvd
  File "/root/.cache/bazel/_bazel_root/b770f990bb7b9e2db5771981fb3a38b4/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/distribution/distribution.py", line 18, in <module>
  File "/usr/local/lib/python3.6/dist-packages/keras/__init__.py", line 28, in <module>
    import third_party.keras.mixed_precision
ModuleNotFoundError: No module named 'third_party'