What version of .NET are you using? The later versions you can us the webclient to download files async...
vb
Dim wc As New Net.WebClient
AddHandler wc.DownloadProgressChanged, AddressOf wc_DownloadProgressChanged
AddHandler wc.DownloadFileCompleted, AddressOf wc_DownloadFileCompleted
wc.DownloadFileAsync(New Uri("http://"), localFile)
To speed up your code, you could either cache more bytes before you write them out to the file or if the files are small enough cache all the bytes then write out the file after the download is complete. Also don't update the UI as often, maybe every 25k or 100k.
Right now every loop you are downloading 4k, writing 4k to a file, calculating progress, and updating the progress 3 times, twice to the same lable and once using a delegate to the progress bar. If you are running this in a background thread all UI updates should be done via a delegate, if you are not running this in a background thread I am not sure why you are just using the delegate for the progress bar update. Clean this up a bit and don't update the ui every iteration, nor would i write out to the file every iteration.
And what is this doing?
vb
blockSize = writeStream.Read(buffer, 0, 4096)
You are reading from the stream you are writing your downloaded file too every iteration? And you never use BlockSize.