When you send the ICMP ping to an IP address belonging to the switch itself(MGMT plane), it will be processed by a slow path (CPU), which needs to wait for the Switch OS scheduler to allocate the CPU/memory resources to handle the packet and send the reply. If the CPU is busy doing other higher priority tasks or memory is low, the ICMP reply will be delayed or even lost.
The best practice for testing switch links is to pass the traffic between workstations, so the traffic can take the fast path. Here is an example:

The intermittent high Latency/RTT for ICMP ping to switch IP is an expected behavior & to understand this behavior, we first need to know about the Logical planes & their functionality.
The various functions of network devices can be logically divided up (categorized) into planes: Logical 'planes'

Data Plane
- This includes functions like adding or removing 802.Iq VLAN tags.
Control Plane
- routing table, MAC address table, ARP table, STP, etc.
For Example:
Logical Planes Operation
• Using a switch as an example:
- When a frame is received, the ASIC (not the CPU) is responsible for the switching logic.
- The MAC address table is stored in a kind of memory called TCAM (Ternary Content-Addressable Memory).
*Another common name for the MAC address table is CAM table*
-> The ASIC feeds the destination MAC address of the frame into the TCAM, which returns the matching MAC address table entry.
-> The frame is then forwarded out of the appropriate interface.
Note: The best & effective way to test the Latency via Switch is to send traffic through the switch "Data Plane" instead of "Control" or "Management Plane"
A simple summary: