Freeze the big weight matrix and train two thin ones, B and A, whose product is the update: a fraction of the numbers, and merged into W for serving at no extra cost.
This interactive visualization needs JavaScript.