Obstacle location

The carla simulator gives us the possibility to work with many more sensors than just a camera feed. We can emulate an LIDAR, IMU, Depth sensor, Segmentation sensor...

Let's use the LIDAR sensor to locate the exact position of the obstacle that has been located by yolov5.

The lidar point cloud is an array of x, y, z, intensity points.

The coordinates are based on Unreal Engine coordinate system which is:

• z is up
• x is forward
• y is right

More info: https://www.techarthub.com/a-practical-guide-to-unreal-engine-4s-coordinate-system/

and within carla documentation: https://carla.readthedocs.io/en/latest/ref_sensors/#lidar-sensor

You can also check velodyne reference: https://github.com/ros-drivers/velodyne/blob/master/velodyne_pcl/README.md

To get the obstacle location, we are going to compute the angle of every points in the point cloud. We can then map the angle of each pixel of the bounding box to a real point and therefore infere its location. We then transform the coordinate from the relative lIDAR coordinate system into a global coordinate system by adding the current position of the LIDAR sensor. The code can be found here: operators/obstacle_location_op.py.

To use the obstacle location, just add it to the graph with:

nodes:
  - id: oasis_agent
    custom:
      inputs:
        tick: dora/timer/millis/400
      outputs:
        - position
        - speed
        - image
        - objective_waypoints
        - lidar_pc
        - opendrive
      source: shell
      # With Carla_source_node
      args: python3 ../../carla/carla_source_node.py
      #
      # Or with the OASIS AGENT
      #
      # args: >
        # python3 $SIMULATE --output 
        # --oasJson --criteriaConfig $CRITERIA
        # --openscenario $XOSC
        # --agent $TEAM_AGENT
        # --agentConfig $TEAM_AGENT_CONF
        # --destination $DESTINATION
  
  - id: yolov5
    operator: 
      outputs:
        - bbox
      inputs:
        image: oasis_agent/image
      python: ../../operators/yolov5_op.py

  - id: obstacle_location_op
    operator: 
      outputs:
        - obstacles
      inputs:
        lidar_pc: oasis_agent/lidar_pc
        obstacles_bbox: yolov5/bbox
        position: oasis_agent/position
      python: ../../operators/obstacle_location_op.py

  - id: plot
    operator:
      python: ../../operators/plot.py
      inputs:
        image: oasis_agent/image
        obstacles_bbox: yolov5/bbox
        position: oasis_agent/position
        obstacles: obstacle_location_op/obstacles

To run:

dora up
dora start graphs/oasis/oasis_agent_obstacle_location.yaml --attach

You should be able to see a dot within the bounding box representing the estimated location in global coordinate of the obstacle.

For more information on obstacle_location, go on our obstacle_location detail page