Currently, we use PyDAP and talk to a NOAA Java OPENDAP server.
This works well ish, but:
Idea: drop PyDAP and write our own client
On the one hand, we need not speak the whole of the OPENDAP protocol. This would simplify things, require less requests to get the format of the data and so on. We know what format the wind data is in, and if it did change, then it would affect other areas of the predictor that expect a certain format anyway.
However: when it changes, if the expected form of wind data is hardcoded deep in the code, it may be a complete pain to change.
We could perform a manual DNS look up, and get a list of IP addresses that we can talk to, and then maintain some information on each node (if it's up, which datasets it has, …). Then if we want some data, we could hit all nodes simultaneously (for different portions of data) assuming that the limiting factor is not our network speed.
This requires more investigation; the NOAA OPENDAP servers have some strange behaviour depending on how much data you request in one go. Larger chunks seem to require it to think for a little while before it starts sending data over the network.
If we download larger chunks, more data that we don't need is downloaded - wasted time Downloading lots of small chunks will slow us down due to the overhead of many small requests.