38 réponses à ce sujet

[vintage4681]

This is my first time trying this and it is brutal. I've been grinding through it with Gemini, but the toolchain keeps breaking, sometimes throwing IR/Opset version mismatches, and other times giving me: 'The onnx model is invalid (shape_inference or model_check failed)'. I even used onnxsim to strip and optimize the model officially, but it's still putting up a fight. It's a huge pain, but I'm learning a lot and hopefully I'll crack it soon

[shockley6668]

Hi vintage, quantizing and deploying custom models can definitely be a bit challenging, especially the first time, so don't worry-it's a common experience. For the RDK X5, opset 11 is recommended. If MiDaS v2.1 contains operators that are not supported by the BPU, you'll need to replace or redesign those parts of the model, as unsupported operators cannot be compiled to the BPU. Also, for quantization-sensitive layers, consider using mixed precision instead of forcing everything to INT8.

[shockley6668]

You can also equip your agent with the OE toolchain to improve its efficiency:https://github.com/HorizonRobotics/OE-Skills/blob/main/README.en.md

[vintage4681]

yup!!
Im trying to get some MiDaS model working

[vintage4681]

Thanks Shockley!!

[vintage4681]

Also the documentation is helping me alot!!

[vintage4681]

I got :"$ docker pull registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_cpu:v1.2.8
Error response from daemon: unknown: failed to resolve reference "registry.d-robotics.cc/deliver/ai_toolchain_ubuntu_20_x5_cpu:v1.2.8": unexpected status from HEAD request to https://registry.d-r...nifests/v1.2.8: 401 Unauthorized
"

[vintage4681]

I used directly horizion to pull

[vintage4681]

to get my toolchain

[vintage4681]

..

[shockley6668]

Hi, @vintage Thanks for sharing your Stage 2 submission! I've reviewed it - great work overall. Looking forward to seeing your 3d cad demo!

[vintage4681]

Yay!!
It worked finally!!
midas worked!
 image.png   1,68 Mo   0 téléchargement(s)

[vintage4681]

Hii Shockely, I need some help. I'm trying to run a YOLO-World + MiDaS pipeline concurrently on the RDK X5 using `pyeasy_dnn` in a ROS 2 (TROS) node, and I am absolutely cooked. Time is running out for my project.

I was using Gemini to help write the tensor decoding logic for the YOLO-World output, and it completely failed me. We went through like five iterations of math, and my bounding boxes are still completely broken.

Here is the exact situation:

1. The Setup

Hardware: D-Robotics RDK X5 (Sunrise 5 BPU).
Framework:ROS 2 (Humble/TROS) using `hobot_dnn.pyeasy_dnn`.
Model:The official `yolo_world.bin` from `/opt/tros/humble/lib/hobot_yolo_world/config/`.

2. The Output Tensors

The BPU inference (`yolo_model[0].forward()`) is giving me two output buffers with these exact shapes:

* `outputs[0].buffer.shape`: **`(1, 8400, 31, 1)`** (Assuming Scores/Classes)
* `outputs[1].buffer.shape`: **`(1, 8400, 4, 1)`** (Assuming Boxes)

3. What Gemini Tried (And How It Failed)

Gemini completely hallucinated how to parse these specific Horizon BPU tensors:

1. The Anchor Grid Fail: It told me to apply a Sigmoid function to the scores and mathematically map the boxes to a generated 640x640 anchor grid. This just shoved 8,400 green boxes into a tiny cluster in the top-left corner.

[vintage4681]

2. The Stride Math Fail: Next, it assumed they were YOLOv8 feature map distances (`l, t, r, b`) and tried multiplying them by strides (8, 16, 32). This literally just drew the entire feature grid across my screen.
3. The "Raw Box" Fail: Finally, it guessed that the BPU *already* did all the math and outputted perfect `[x1, y1, x2, y2]` coordinates and probabilities. We stripped out all the math, and it immediately drew 8,400 boxes over every single piece of empty background in the room.

I have no idea if these tensors are raw logits, stride-multiplied distances, or something proprietary to the BPU compiler.

How exactly am I supposed to parse and decode `outputs[0]` and `outputs[1]` from this specific `.bin` file in Python? Any snippet or guidance would be a lifesaver.

[vintage4681]

...

[shockley6668]

Hi! @vintage Please refer to the official YOLO-World example in the RDK Model Zoo. It includes the correct preprocessing and output post-processing logic for the RDK X5:
https://github.com/D...ision/yoloworld
You can adapt its decoding and NMS implementation to your pyeasy_dnn ROS 2 node.

[shockley6668]

One more note: please don't use hobot_dnn.pyeasy_dnn, as it is the older interface. The current RDK X5 API has been updated to use HBM Runtime.
Please refer to the latest AI Python API documentation:
https://d-robotics.g...5/ai-python-api
The official YOLO-World example using the current runtime is here:
https://github.com/D...ision/yoloworld

[vintage4681]

Thanks Shockley

[shockley6668]

@vintage Hi! Please update PR #13 with the final demo video, correct architecture document link, and complete two-node launch instructions, then ping us for re-review-thanks!
 image.png   198,75 Ko   0 téléchargement(s)