The Libyan coastguard says it has rescued 168 migrants whose boat had run into trouble off Tripoli but was unable to help a second vessel nearby.