Introducing pythondocs.xyz – live search for Python documentation

Winter is long here.

It is so long that I did an accidental software development after False Spring 3 – whoops!

pythondocs.xyz demo

pythondocs.xyz

pythondocs.xyz is a web tool that provides live search results for Python’s official documentation.

Please try it out and let me know what you think!

It’s at “beta” stage, which means it works pretty well but it’s not perfect.

It’s fast and it looks good and the results are… fine.

It did, however, survive the front page of HackerNews without going above 2% CPU usage, which I think is pretty good.

The next big feature will be better search results. In particular: improved prominence of important language features, like built-in functions, and refined full text search and ordering of results.

Here’s the tech stack as it currently stands, for those interested:

  • Parser: Beautiful Soup + Mozilla Bleach
  • Database: in-memory SQLite (aiosqlite) + SQLAlchemy
  • Web server: FastAPI + Uvicorn + Jinja2
  • Front end: Tailwind CSS + htmx + Alpine.js

This is my first big FastAPI project, and over the next few weeks I’ll blog about some of the tricks I used, especially to do with performance.

Python & VS Code: make Black and organize imports work together on save

Hello world. What a stable and positive time to be alive!

But at least we can fix our code editors.

I use Black to format my Python code, and I tell VS Code to organise imports on save.

Here’s the relevant bit of VS Code’s settings.json:

{
    "python.formatting.provider": "black",
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
        "source.organizeImports": true
    }
}

It’s a pretty pleasant experience, mostly. But this “flapping” behaviour when I save has been bugging me for weeks:

(It’s even more annoying when you’re part way down a file and the entire screen jerks up and down.)

Clearly, Black and the organize imports thing are fighting. But what is the organize imports thing? And what’s the fix?

It turns out that VS Code uses the isort library to sort imports, and isort has a Black compatability profile.

As a Poetry user, all I had to do is add these lines to the pyproject.toml file at the root of my project:

[tool.isort]
profile = "black"

Alternatively, create a file called .isort.cfg (note the leading dot) at the root of your project, with this content:

[settings]
profile=black

No more flapping, and at least one small corner of the universe is a little more sane.

Backup / migrate Microsoft To Do tasks with PowerShell and Microsoft Graph

Update: this post has a spiritual successor – Extract Microsoft To Do steps/sub-tasks from your web browser (with Asana import example)

For more than a year, I’ve foolishly been using a developer Office 365 subscription for some personal stuff. You know, the subscription where they delete your data if “development activity” isn’t detected every few months. As such, I’ve periodically had to fake some development activity in order to keep the clock ticking.

Not a sustainable situation, and it’s time to sort it out…

For me, this involves moving data from one subscription’s OneDrive to another. I’m fairly confident that Rclone will be able to handle this – it’s an excellent bit of software.

It also means moving Microsoft To Do tasks between subscriptions. Ah.

Not so easy

I couldn’t find an easy way of backing up To Do. There is mention of an Outlook backup option in the docs, but it’s missing on my account. And To Do will happily suck in data from Wunderlist but I can’t see an equivalent to get data out. Where’s the Justice Department when you need them?

Luckily Microsoft Graph has a To Do API in preview and I was able to put together a script to do the lifting for me.

Ironically, this has involved an intense burst of real developer activity…

Enter the Dragon

The full script is over on GitHub.

It provides two functions:

  • Export-MicrosoftTodo saves every Microsoft To Do list and task to an XML file.
  • Import-MicrosoftTodo loads this XML file and restores all lists and tasks.

This is what a backup looks like:

Here’s a restore:

And this is what the client sees:

You can see that the completed status of the tasks has been copied. This is also true of created/modified dates, reminders, notes, and so on.

The script is quite long, so I won’t paste the whole thing here, but here are a few interesting bits:

Emotional support

Thankfully, for the comfort of our technology-addled minds, Microsoft To Do lets you decorate your lists with little emojis. Internally, it looks like if the first character of the name is an emoji, it gets special treatment in the UI.

I was having difficulty creating these special list names but the fix was simply to add charset=utf-8 to Invoke-RestMethod‘s ContentType:

$params = @{
    "Method"         = "Post"
    "Uri"            = ($graphBaseUri + "/me/todo/lists")
    "Authentication" = "OAuth"
    "Token"          = $accessToken
    "Body"           = @{
        "displayName" = $list.displayName
    } | ConvertTo-Json
    # utf-8 makes emojis work. Life priorities are correct.
    "ContentType"    = "application/json; charset=utf-8"
}
Invoke-RestMethod @params

Before and after:

Jason Bateman

The basic aim of the script is to retrieve data from one API endpoint and to later submit the same data to another endpoint.

I found that PowerShell’s – er – adorable magic got in the way a bit. Specifically, the JSON (de-)serialisation done by Convert*-Json / Invoke-RestMethod didn’t preserve empty properties, and the conversion to/from a DateTime object didn’t match the format expected by the API – and I couldn’t see an easy way to override this behaviour.

My solution was to use an alternative JSON parser available in .NET to grab the appropriate bit of the HTTP response, remove a few properties, and store the resulting JSON as a string, to be later POSTed back to the API verbatim:

$response = Invoke-WebRequest -Uri $uri -Authentication OAuth -Token $accessToken
# Invoke-RestMethod / ConvertFrom-Json mangles the response, which I resent,
# so we're using an alternative parser and storing the JSON as a string
# https://stackoverflow.com/a/58169326/12055271
$json = [Newtonsoft.Json.JsonConvert]::DeserializeObject($response.Content)
ForEach ($task in $json.value) {
    # Don't need ID - Graph API can generate a new one
    $task.Remove("id") | Out-Null
    # Don't need ETag
    $task.Remove("@odata.etag") | Out-Null
    $results += $task.ToString()
}

I also chose to save the exported data to disk using PowerShell’s CLI XML format – rather than JSON – as an easy way of guarantee the string stays as it is.

Token effort

The script needs an OAuth2 token in order to authenticate with your Microsoft account.

An easy way to get going (and slightly hacky but fine for personal use) is to grant yourself all Tasks.* permissions in Graph Explorer and copy its token, as demoed here:

(Thanks GoToGuy for this blog post.)

Please read the following license agreement carefully

A few notes on the design of the script:

  • It worked well for me and 5000(!) tasks, but please do your own testing. You can create a test Microsoft account with a secondary email address, or make an Azure tenant for free.
  • Tested with PowerShell 7 only. Get with the times.
  • Export-MicrosoftTodo currently backs up every task and Import-MicrosoftTodo restores every task.
  • If you run Import-MicrosoftTodo twice you’ll end up with duplicates.
  • The account used for export/import is the one that generated the OAuth token. You can backup from one account and restore to another simply by providing different tokens.
  • The script does not currently migrate linkedResources – these “represent[…] an item in a partner application related to a todoTask.” Shrug.
  • Nor does it share any lists as part of data import.
  • Currently, the script needs to be run interactively, in order to receive the OAuth token and to confirm a restore.
  • I’d be open to making improvements in these areas if there’s interest! The script could backup individual lists, for example, or backup someone else’s account (with the appropriate permissions).
  • Unfortunately, I don’t think there’s currently any way to retain list groups.

And in conclusion

Thanks for reading!

This has been a fun project and hopefully you can get some use out of the methods used or the script itself.

Start Windows Terminal in WSL2 home directory

A little tip, partly written for the benefit of future-me…

As you may know, WSL2 filesystem performance is much improved over its predecessor – but only in the virtual disk of your Linux distro. The mounted Windows disk – /mnt/c/ – is still slow for disk-heavy operations, like Git clones, because of Linux-plus-NTFS something reasons.

Because of this, the first thing I do when I open my WSL2 Ubuntu distro in Windows Terminal is change to my Linux home directory. Every time.

There must be an easier way? Yep.

From the WSL docs:

bash ~ launches the bash shell into the user’s home directory. Similar to running cd ~.

To make this the default in Windows Terminal, open the Windows Terminal Settings, find your WSL2 profile, and add “commandline”: “bash.exe ~”

{
    "guid": "{2c4de342-38b7-51cf-b940-2309a097f518}",
    "hidden": false,
    "name": "Ubuntu",
    "source": "Windows.Terminal.Wsl",
    "commandline": "bash.exe ~"
}

(Note that the “source” line now needs a trailing comma.)

From this:

To this:

View all Windows Terminal colour schemes

Want to set the snazziest Windows Terminal colour scheme but not sure how to go about it?

Surely this is the top of everyone’s to-do list at this time of unprecedented calmness and serenity…

You could spend 5 minutes manually trying out the various options.

You could spend an hour hacking together a fairly rickety script that launches Windows Terminal with each of the built-in schemes.

Or you could let me do that for you and spend 20 seconds watching the results in video form:

Why not reward yourself for your time efficiency by spending 4 minutes 40 seconds scouring the Internet for Coronavirus morsels? Some suggested search terms:

  • potato long term storage tips
  • discourage social interaction clothing ideas obscene
  • maximum safe potato consumption
  • handshake alternatives obscene
  • where buy potato minneapolis

Stay safe out there.

Here’s the code:

# Script that cycles through all Windows Terminal color schemes
# Depends on there being a profile called "PowerShell" with a colorScheme already set
param(
    $profilesJsonPath = (Join-Path $env:LOCALAPPDATA 'Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState\profiles.json'),
    $defaultsJsonPath = (Join-Path $env:ProgramFiles 'WindowsApps\Microsoft.WindowsTerminal_0.10.761.0_x64__8wekyb3d8bbwe\defaults.json')
)

# Get all color schemes from default profiles file
$defaults = Get-Content $defaultsJsonPath | ConvertFrom-Json
$colorSchemes = $defaults.schemes.name

# Get profiles as text
$profiles = Get-Content $profilesJsonPath

# Fragile string stuff to find index of the colorScheme line in the PowerShell profile
# Easier than dealing with JSON + comments as objects?
$psNameIndex = ($profiles | Select-String '"name": "PowerShell",').LineNumber - 1
$sectionBoundaries = ($profiles | Select-String '},').LineNumber | ForEach-Object {$_ - 1} # Section delimited by '},'
$sectionStart = ($sectionBoundaries | Where-Object {$_ -lt $psNameIndex})[-1]
$sectionEnd = ($sectionBoundaries | Where-Object {$_ -gt $psNameIndex})[0]
$colorIndex = ($profiles[$sectionStart..$sectionEnd] | Select-String '"colorScheme":').LineNumber - 1 + $sectionStart

ForEach ($newScheme in $colorSchemes) {
    # Regex to insert new scheme
    $profiles[$colorIndex] = $profiles[$colorIndex] -replace '"colorScheme":.*[^,]' ,"`"colorScheme`": `"$newScheme`""
    # Write out JSON
    $profiles | Out-File $profilesJsonPath
    # Launch Windows Terminal
    $command = "Read-Host `"$newScheme`""
    $arguments = @{
        'FilePath' = 'wt.exe'
        'ArgumentList' = "-p `"PowerShell`" pwsh.exe -Command $command"
        'Wait' = $true
        'WindowStyle' = 'Maximized'
    }
    Start-Process @arguments
}