UCS-2 is considered an outdated version of UTF-16, so it's recommended to test both encodings (automatic detection of text encoding isn't foolproof).
If we know the source encoding, we can assume the target encoding is UTF-8 (which is a logical choice in 2021 and given that your question is tagged as UTF-8). This gives us all the necessary information.
The first step is to eliminate any non-standard raw byte manipulations (such as removing
str_replace("\x00", '', $data[7])
and similar code snippets) before proceeding with proper conversion. One potential approach using
mb_convert_encoding() could be:
$delimiter = "\t";
$fp = fopen($storagename, 'r');
while ( !feof($fp) ){
$line = mb_convert_encoding(fgets($fp, 2048), 'UTF-8', 'UCS-2LE');
$data = str_getcsv($line, $delimiter);
print_r($data);
}
You can refer to the list of supported encodings for more options.
One issue to consider is that str_getcsv() might not recognize UCS-2 line endings without explicit encoding specification.
Depending on the size of the CSV file, different solutions can be explored. For smaller files, direct conversion may suffice. For larger files, consider using stream_get_line():
This function is similar to fgets() but supports custom end-of-line delimiters beyond \n, \r, and \r\n, omitting the delimiter itself from the result.
An example implementation would look like this:
$ending = mb_convert_encoding("\n", 'UCS-2LE', 'UTF-8');
$line = mb_convert_encoding(stream_get_line($fp, 2048, $ending), 'UTF-8', 'UCS-2LE');
With this method, handling Unix line endings (\n) and Windows line endings (\r\n) should work seamlessly.