1. Disable Steam Overlay
1a. In your steam library and find Bioshock HD
1b. Right click on it and click properties
1c. Uncheck the steam overlay while in game
1d. Click apply

2. Change Havock Core to 1
Ensure Bioshock isn't running!
2a. In Explorer, browse to C:/Users/<username>/Appdata/Roaming/BioshockHD/Bioshock
2b. Open Bioshock.ini in notepad
2c. Search for the line HavokNumThreads=??? (could be any number, is often 2)
2d. Change this line to HavokNumThreads=1, and save the file.


3. Enable more memory for texture streaming in Bioshock2.
3a. In Explorer, browse to C:/Users/<username>/Appdata/Roaming/BioshockHD/Bioshock2
3b. Open Bioshock2SP.ini
3c. Search for TextureStreamingMemoryLimit= (for me it was set to 512, it's the amount of memory in MB the game can use for texture streaming)
3d. Depending on your graphics card's memory, change this value. I have a 3080ti mobile GPU, which has 16gb or memory, or 16000mb.
3e. I was able to change the line to TextureStreamingMemoryLimit=10000 Probably overkill but it made Bioshock2 Remastered run like a charm.

4. Enable more than 4gb of memory
4a. Download the 4gb patch program from here https://ntcore.com/?page_id=371
4b Double click the exe you downloaded, it will now want you to select the exe you want to patch.
4c. Within the explorer window that 4gb_patch.exe opened, browse to <yourSteamInstallationFolder>/steamapps/Bioshock/Remastered/Build/Final/ and click BioshockHD.exe
4d. The BioshockHD.exe will now be patched to use over 4gb of memory!
I think this fix is actually the most important!

5. Force Direct X 9
5a. Browse to <yourSteamInstallationFolder>/steamapps/Bioshock Remastered/Build/Final/
5b. right click Bioshock HD.exe and click create shortcut
5c. Drag this shortcut to your desktop or somewhere where you can use it to launch the game
5d. Right click the shortcut, click properties
5e. Make sure you are in the shortcut tab
5f. There is a text box labelled target. Add -dx9 to end of this target. It should look something like this:
"D:\Steam\steamapps\common\BioShock Remastered\Build\Final\BioshockHD.exe" -dx9
(your steam folder will probably be installed somewhere else)
5g. Click apply, and then OK.

6. Run Bioshock as Administrator
6a. Right click on the shortcut you created in the last section
6b. Click properties, and make sure you are in the shortcut tab
6c. Click Advanced
6d. Tick 'Run as Administrator'
6e. Click OK.
6f. Click Apply
6g. Click OK.


Always launch Bioshock via this shortcut, not the steam shortcut or steam itself! Otherwise the fixes won't take effect!

Bonus
Does Bioshock stay in windowed mode, even when you set it to fullscreen?

Download and Install Borderless Gaming here:
https://github.com/Codeusa/Borderless-Gaming/releases

Make sure the Bioshock process is in Borderless Gaming's preferences.

In conclusion, disable steam overlay, change havok cores to 1, enable more memory for texture streaming (bioshock 2 only), patch the exe to use more than 4gb of ram, force direct x 9 and run bioshock as administrator. Do these last two things via the shortcut.