When handleNewUDPFlow in NETransparentProxyProvider is used to handle UDP data from port 53,
at the same time, run the script continuously to execute nslookup or dig, about tens of thousands of times later,
the nslookup shows the error "isc_socket_bind: address not available".
So I check the system port status, and find all of the ports from 49152 to 65535 are occupied. The number of net.inet.udp.pcbcount is also very high.
net.inet.udp.pcbcount: 91433
Then I made the following attempts:
handleNewUDPFlow function return false directly, the nslookup script runs with no problems.
I write a simple network extension that use handleNewUDPFlow to reply the mock data directly, and only hijack the UDP data from my test program (HelloWorld-5555).
My network exntension code:
override func handleNewUDPFlow(_ flow: NEAppProxyUDPFlow, initialRemoteEndpoint remoteEndpoint: NWEndpoint) -> Bool {
guard let tokenData = flow.metaData.sourceAppAuditToken, tokenData.count == MemoryLayout<audit_token_t>.size else { return false }
let audit_token = tokenData.withUnsafeBytes { buf in
buf.baseAddress?.assumingMemoryBound(to: audit_token_t.self).pointee
}
let pid = audit_token_to_pid(audit_token ?? audit_token_t())
if (!flow.metaData.sourceAppSigningIdentifier.starts(with: "HelloWorld-5555")) {
return false
}
Logger.statistics.log("handleNewUDPFlow \(remoteEndpoint.debugDescription, privacy: .public) \(flow.hash), pid:\(pid), \(flow.metaData.sourceAppSigningIdentifier, privacy: .public)")
flow.open(withLocalEndpoint: nil) { error in
if let error {
os_log("flow open error: %@", error.localizedDescription)
return
}
flow.readDatagrams { data_grams, remote_endpoints, read_err in
guard let read_data_grams = data_grams,
let read_endpoints = remote_endpoints,
read_err == nil else {
os_log("readDatagrams failed")
flow.closeReadWithError(nil)
flow.closeWriteWithError(nil)
return
}
let mockData = Data([0x01,0x02,0x03])
let datagrams = [ mockData ]
guard let remoteEnd = remoteEndpoint as? NWHostEndpoint else {
os_log("Not the NWHostENdpoint")
flow.closeReadWithError(nil)
flow.closeWriteWithError(nil)
return
}
let endpoints = [ NWHostEndpoint(hostname: remoteEnd.hostname, port: remoteEnd.port) ]
flow.writeDatagrams(datagrams, sentBy: endpoints) { error in
if let error {
os_log("writeDatagrams error: %@", error.localizedDescription)
}
os_log("writeDatagrams close")
flow.closeReadWithError(nil)
flow.closeWriteWithError(nil)
}
}
}
return true
}
My test program code:
void send_udp() {
int sockfd;
struct sockaddr_in server_addr;
char buffer[BUFFER_SIZE];
int bytes_sent;
// create socket
if ((sockfd = socket(AF_INET, SOCK_DGRAM, 0)) < 0) {
perror("socket create failed");
exit(EXIT_FAILURE);
}
struct sockaddr_in local_addr;
memset(&local_addr, 0, sizeof(local_addr));
local_addr.sin_family = AF_INET;
local_addr.sin_addr.s_addr = htonl(INADDR_ANY);
local_addr.sin_port = htonl(0);
// bind
if (bind(sockfd, (struct sockaddr*)&local_addr, sizeof(local_addr)) < 0) {
printf("IPV4 bind errno:%d\n", errno);
close(sockfd);
return;
}
// server addr
memset(&server_addr, 0, sizeof(server_addr));
server_addr.sin_family = AF_INET;
server_addr.sin_port = htons(SERVER_PORT);
server_addr.sin_addr.s_addr = inet_addr(SERVER_IP);
// send & recv
strcpy(buffer, "Hello, UDP server!");
bytes_sent = sendto(sockfd, buffer, strlen(buffer), 0,
(struct sockaddr *)&server_addr, sizeof(server_addr));
if (bytes_sent < 0) {
perror("sendto failed");
close(sockfd);
exit(EXIT_FAILURE);
}
printf("sendto ok\n");
char recvbuf[128] = {0};
socklen_t len = sizeof(server_addr);
int sz = recvfrom(sockfd, recvbuf, sizeof(recvbuf), MSG_WAITALL, (struct sockaddr *) &server_addr, &len);
printf("recv sz: %d\n", sz);
close(sockfd);
return;
}
int main() {
send_udp();
return 0;
}
2.1 When I use bind in my program, after the program running tens of thousands of times, the ports are exhausted, and nslookup return the error "isc_socket_bind: address not available". The case looks like running the nslookup script, because the nslookup will call the bind.
2.2 When I remove the bind from my program, all the tests are go.
I have made the above experiments on different systems: 13.x, 14.x, 15.x, and read the kernel source code about bind and port assignment,
bsd/netinet/in_pcb.c
bsd/netinet/udp_usrreq.c
and find kernel will do different action for network extension by call necp_socket_should_use_flow_divert
I have checked my network extension process by lsof and netstat, its sockets or flows are all closed properly.
I don't know how I can avoid this problem to ensure my network extension to work long time properly. Apparently, the port exhaustion is related to the use of bind function and network extension. I doubt there is a port leak problem in system when use network extension.
Hope for your help.
