libutils: Improve performance of utf8_to_utf16/utf16_to_utf8
This CL improves the performance of below functions in helping with conversion between utf8/utf16 with libutils: - utf8_to_utf16_length - utf8_to_utf16 - utf16_to_utf8_length - utf16_to_utf The basic idea is to keep the loop as tight as possible for the most common cases, e.g. in UTF16-->UTF8 case, the most common case is when the character is < 0x80 (ASCII), next is when it's < 0x0800 ( most Latin), and so on. This version of implementation reduces the number of instructions needed for every incoming utf-8 bytes in the original implementation where: 1) calculating how many bytes needed given a leading UTF-8 byte in utf8_codepoint_len(), it's a very clever way but involves multiple instructions to calculate regardless 2) and an intermediate conversion to utf32, and then to utf16 utf8_to_utf32_codepoint() The end result is about ~1.5x throughput improvement. Benchmark results on redfin (64bit) before the change: utf8_to_utf16_length: bytes_per_second=307.556M/s utf8_to_utf16: bytes_per_second=246.664M/s utf16_to_utf8_length: bytes_per_second=482.241M/s utf16_to_utf8: bytes_per_second=351.376M/s After the change: utf8_to_utf16_length: bytes_per_second=544.022M/s utf8_to_utf16: bytes_per_second=471.135M/s utf16_to_utf8_length: bytes_per_second=685.381M/s utf16_to_utf8: bytes_per_second=580.004M/s Ideas for future improvement could include alignment handling and loop unrolling to increase throughput more. This CL also fixes issues below: 1. utf16_to_utf8_length() should return 0 when the source string has length of 0, the original code returns -1 as below: ssize_t utf16_to_utf8_length(const char16_t *src, size_t src_len) { if (src == nullptr || src_len == 0) { return -1; } ... 2. utf8_to_utf16() should check whether input string is valid. Change-Id: I546138a7a8050681a524eabce9864219fc44f48e
Loading
Please register or sign in to comment