The characteristic effect of nanoparticles, boron nitride nanotubes in the nanofluid with and without surfactant, were investigated using numerical CFD. A simplified microchannel heat sink model was created and discretized for numerical analysis. The numerical prediction was validated with previous experiment data for promising numerical agreement. Then, the effect of different Triton X-100 surfactant volume fractions and nanotube mass fractions in the base fluid were carried out on thermal and hydraulic performance. The significant finding revealed that the thermal resistance was reduced by as much as 90% compared to pure water with a surfactant concentration of 0.35 vol.% and adding 0.02 wt.% at the Reynold number (Re) of 400. However, the Nusselt number (Nu) increased twice from the pure water with an additional surfactant of 0.35 vol.% after the Re of 400. Despite improving the thermal performance, the pressure drop seems to be a drawback for the nanotube with surfactant implementation. The present study resulted in a greater comprehension of the nanofluid flow with surfactant effect in the nanofluid in the microchannel heat sink, allowing for better design decisions to be made for the improvement of this application for various purposes. © 2022, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.