Everything you need to know about Unicode in Perl on Windows

I've now solved the display issue by making my Perl script change the codepage to 65001 just like CMD's `dir` does:

use feature ":5.16";  # Include say and Unicode string functions.
use open ':std', ':encoding(UTF-8)';  # Print Unicode without complaining about wide chars.
use utf8;  # Unicode source code.
use strict;  # Don't ignore undefined variables.
use Win32::Console;

Win32::Console::OutputCP(65001);  # Tell Windows Terminal to render Unicode (with unclustered graphemes in 1.23.13503.0 but they copy/paste ok).

my $file_name = "emoji_🎅🎉😀💢👩🏾‍🚀.txt";

say "Writing to ${file_name}";
open( my $fh, ">", $file_name ) or die "Can't write $file_name: $!";
binmode($fh, ":encoding(UTF-8)");  # encoding() also checks UTF-8 validity, unlike ":utf8".
say $fh "My name is \"$file_name\".";
close($fh);

say "Reading ${file_name}";
open( $fh, "<", $file_name ) or die "Can't read $file_name: $!";
binmode($fh, ":encoding(UTF-8)");
read($fh, my $buf, 100);
say $buf;
close($fh);

say "Done.";

This won't affect the codepage of the terminal after, just like `dir`: And despite the female black astronaut not rendering as a single character, it copy/pastes just fine:

15:23:45 C:\apps\StrawberryPerl\perl\bin>chcp
Active code page: 850

15:23:50 C:\apps\StrawberryPerl\perl\bin>perl C:\Users\C\Documents\code\Perl\test_unicode.pm
Writing to emoji_🎅🎉😀💢👩🏾‍🚀.txt
Reading emoji_🎅🎉😀💢👩🏾‍🚀.txt
My name is "emoji_🎅🎉😀💢👩🏾‍🚀.txt".

Done.

15:23:53 C:\apps\StrawberryPerl\perl\bin>chcp
Active code page: 850

Unfortunately, writing the emoji file name only appears to work correctly inside a shell inside VS Code; not inside a shell inside Windows Terminal 1.23.13503.0: Newer languages like Python do work.

file_name = "emoji_🎅🎉😀💢👩🏾‍🚀.py.txt"

print(f"Writing to {file_name}")
with open(file_name, "w", encoding="utf8") as f:
    f.write(f"My name is \"{file_name}\".")

print(f"Reading {file_name}")
with open(file_name, "r", encoding="utf8") as f:
    print(f.read())

print("Done.")

Keep your system and project tidy with a Python virtual environment folder

cees@cees-XPS-13-9380:~/code/space-monitor$ python3 -m venv venv
cees@cees-XPS-13-9380:~/code/space-monitor$ source venv/bin/activate
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ which python3
/home/cees/code/space-monitor/venv/bin/python3
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ python3 -m pip install logdna
Processing /home/cees/.cache/pip/wheels/af/ce/e9/fccf04ebc826e8cff5ec031f8ecc0c3b4092b44bc692f22a3f/logdna-1.5.3-py3-none-any.whl
Collecting requests
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Installing collected packages: certifi, idna, urllib3, chardet, requests, logdna
Successfully installed certifi-2020.4.5.1 chardet-3.0.4 idna-2.9 logdna-1.5.3 requests-2.23.0 urllib3-1.25.9
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ python3 -m pip freeze > requirements.txt
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ cat requirements.txt 
certifi==2020.4.5.1
chardet==3.0.4
idna==2.9
logdna==1.5.3
requests==2.23.0
urllib3==1.25.9
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ python3 -m pip install -r requirements.txt
Requirement already satisfied: certifi==2020.4.5.1 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (2020.4.5.1)
Requirement already satisfied: chardet==3.0.4 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: idna==2.9 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2.9)
Requirement already satisfied: logdna==1.5.3 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.5.3)
Requirement already satisfied: requests==2.23.0 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (2.23.0)
Requirement already satisfied: urllib3==1.25.9 in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.25.9)
(venv) cees@cees-XPS-13-9380:~/code/space-monitor$ deactivate
cees@cees-XPS-13-9380:~/code/space-monitor$ 

Using SharePoint 2010's REST API

Get 100 items after the first 100 items from the items of this year and month:

http://site.company.tld/_vti_bin/ListData.svc/YourList?$skip=100&$top=100&$select=Year,Month,ID&$filter=Year%20eq%20%272014%27%20and%20Month%20eq%20%27Mar%27


Here's a complete example in Python, TestSharePoint2010REST.py:

"""
SharePoint2010 REST test app by Cees Timmerman, 24jun14.
"""
# Fix Python 2.
from __future__ import print_function
try: input = raw_input
except: pass
try: import urllib.request as urllib2
except: import urllib2
try: from ntlm3 import HTTPNtlmAuthHandler
except: from ntlm import HTTPNtlmAuthHandler

import json, re, time

web_domain = "http://subdomain.company.tld/"
user = "company\\user"
password = "secret"
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, web_domain, user, password)
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman)
opener = urllib2.build_opener(auth_NTLM)
urllib2.install_opener(opener)

url = web_domain + "_vti_bin/ListData.svc/TimeRegistration?$filter=Modified%20gt%20datetime\'2014-04-30T00:00:00.000Z\'&$orderby=Modified%20asc&$select=Id,Year,Month,Hours,Project,Resource&$top=2050"

months = "Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec".split(",")
count = 0
while 1:
  print(url)
  request = urllib2.Request(url)
  request.add_header("accept", "application/json;odata=verbose")  # The default format is application/atom+xml. Example: "accept":"application/json;odata=verbose"
  response = urllib2.urlopen(request)
  headers = response.info()
  body = response.read().decode("utf-8")
  body = re.sub(r"(?<!\\)\\'", "'", body)  # Fix MicrosoftSharePointTeamServices 14.0.0.6029's incorrect JSON output. http://stackoverflow.com/questions/24390258/json-module-bug-in-python-3-4-1/24390987#comment37724338_24390290
  print("Headers: {}".format(headers))
  print("Body start: " + body[:800])
  print("==========================\nBody end: " + body[-800:])
  #err_loc=314020; print(">>>>>>Parse Error: {}<<<<<<".format(body[err_loc-40:err_loc+40])) # Debug output using value from thrown error.
  j = json.loads(body)
  results = j['d']['results'] if 'results' in j['d'] else j['d']
  print("Len: %r" % len(results))
  for item in results:
    count += 1
    print(item['Resource'].replace("company\\", ""), item['Project'], float(item['Hours']), "{}-{:02}-01".format(item['Year'], 1+months.index(item['Month'])), item['Id'])
  print("Count: %s" % count)
  
  # End loop or process next batch of 1000.
  url = None
  try: url = j['d']['__next']
  except: pass
  if not url: break

Patching bugs with SVN

I'm using Suds to work with SharePoint, but it was terribly slow due to broken caching, so i patched it:

I:\>cd temp

I:\temp>mkdir sudssvn

I:\temp>cd sudssvn

I:\temp\sudssvn>svn co http://svn.fedorahosted.org/svn/suds/trunk
A    trunk\LICENSE
A    trunk\tests
A    trunk\tests\rhq.py
A    trunk\tests\public.py
...

I:\temp\sudssvn>"C:\Program Files (x86)\Notepad++\notepad++.exe" trunk\suds\cache.py

I:\temp\sudssvn>svn di trunk
Index: trunk/suds/cache.py
===================================================================
--- trunk/suds/cache.py (revision 712)
+++ trunk/suds/cache.py (working copy)
@@ -190,7 +190,7 @@
     def put(self, id, bfr):
         try:
             fn = self.__fn(id)
-            f = self.open(fn, 'w')
+            f = self.open(fn, 'wb')
             f.write(bfr)
             f.close()
             return bfr
@@ -223,7 +223,7 @@
         try:
             fn = self.__fn(id)
             self.validate(fn)
-            return self.open(fn)
+            return self.open(fn, 'rb')
         except:
             pass

I:\temp\sudssvn>svn ci trunk
svn: E205007: Commit failed (details follow):
svn: E205007: Could not use external editor to fetch log message; consider setting the $SVN_EDITOR environment variable or using the --message (-m) or --file (-F) options
svn: E205007: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no 'editor-cmd' run-time configuration option was found

I:\temp\sudssvn> ...TortoiseSVN GUI:

Commit
Commit failed (details follow):
Can't create directory '/svn/suds/db/transactions/712-1.txn': Permission denied

Alas, a lack of project time/leadership stifles progress, but at least open source and free speech allows one to fork unmaintained projects. Torvalds' Git (simple guide) was made for decentralized development like that.

A Google result (and on closer inspection, previous error) tells me that instead of the GUI, i could've used the -m switch of svn:

I:\temp\sudssvn>svn ci trunk -m "fix caching on Windows"
svn: E000013: Commit failed (details follow):
svn: E000013: Can't create directory '/svn/suds/db/transactions/712-1.txn': Permission denied

The system environment variable PATH gives me easy access to "C:\Program Files\TortoiseSVN\bin\svn.exe".

Digits to Excel column letters

A little tricky, so i'll note it here:

>>> def getLetters(n):
...     o = ""
...     while 1:
...             n, m = divmod(n-1, 26)
...             o = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"[m] + o
...             if n < 1: break
...     return o
...
>>> getLetters(1)
'A'
>>> getLetters(25)
'Y'
>>> getLetters(26)
'Z'
>>> getLetters(27)
'AA'

And i might as well include my old getCol function:

def getCol(s):
 z = 0
 for i in range(1, len(s) + 1):
  c = s[-i]
  z += ("ABCDEFGHIJKLMNOPQRSTUVWXYZ".find(c) + 1) * 26**(i - 1)
 return z

Fixing Python 2.6+ to run Python 3 code:

# -*- coding: utf-8 -*-
from __future__ import print_function
__doc__ = """XLS2DB by Cees Timmerman"""
changelog = """
2012-06-28 v1.0
"""
import glob, locale, os, re, sys, time
#from pyXLSX.xlsx import workbook  # no MergedCells support and fails at Unicode.
from win32com.client import Dispatch  # PyWin32 or ActivePython required. Excel, too.
#from win32com.client.gencache import EnsureDispatch as Dispatch  # Also makes constants available.
#import Image  # PIL to resize images.
import pymysql   # Note: Python 3 doesn't support old db protocol.

print(__doc__)
print(changelog.split("\n")[-2])
print()

if sys.version[:2] == '2.':
 #print("Python 2 sucks at Unicode; use Python 3 to run this.")
 #sys.exit(2)

 # Fix Python 2. We should be running in Python 3, though.
 original_print = print
 def print(*args, **kwargs):
  #original_print("args: " + str(args))
  #original_print("kwargs: " + str(kwargs))
  new_args = []
  for arg in args:
   try:
    new_args.append(arg.encode('ascii', errors='xmlcharrefreplace'))
   except:
    new_args.append(arg)
  original_print(*new_args, **kwargs)
 
 try:
  input = raw_input
  str = unicode
 except: pass