http-cache with ureq

[mrchantey]

I'm working on a feature that requires the use of ureq as well as http caching, any tips or prior art for using http-cache with ureq?

[06chaynes]

When I had first looked into it some years ago ureq didn't appear to have middleware support, but I see now that has since changed. Given that it may be possible but I would need to play around with it to know for sure.

If I get some free time soon I'll look into what it would take to create a ureq client library for http-cache.

[06chaynes]

@mrchantey i noticed in the linked issue that one problem with using reqwest was the requirement of tokio.

Because http-cache is an async library we will need to deal with this somehow to use with ureq. Since this library would really start off for your use case do you have any input on how this might best be handled? It's my understanding we would need to import some async runtime to do this, but maybe there's another way I'm just not aware of.

Edit: to clarify this would be a requirement for the ureq http cache client library, not your code itself. But I want to check for any opinions on the matter before I attempt to throw something together.

[mrchantey]

Good question the requests are made in a non-tokio async block, its my understanding that bevy has some custom async implementation.

The first implementation used surf and http-cache-surf wich did work, code here, but when we tried replacing it with reqwest it seemed to require tokio async. The decision was made to use ureq because its lightweight, heres a bit more context in the discord.

I think @aevyrie would be a good person to ask about deeper bevy async stuff if required.

[06chaynes]

I think I might be able to use futures::executor::block_on for this. I will try to throw something together and see what happens.

[06chaynes]

Made an effort to start on this but running into some lifetime problems that I think are Send/Sync related on the ureq Middleware trait. Will look into this again as I have time but if any progress is made your way I'd also be happy to accept a PR.

[mrchantey]

Sounds good thanks for giving it a look.

[06chaynes]

I think what this may require me to do is to revisit the refactor I had attempted a few years ago, which was originally started to better support tower and therefore hyper. I hit a blocker with the tower service and the refactor didn't go anywhere. I ran into problems when trying to implement a ureq client library with things as they currently are, and I think this refactor may get around those issues.

I had pushed the code I had to the refactor branch so it wouldn't be lost but it would need some work before I could merge anything. To save time I think I will try to make the client against the refactor branch code and just see how far I get. If it does work out then I will pursue getting the refactor finished and updated with the changes that have been added in since the attempt was made.

[mrchantey]

For the Bevy use-case Ive added a very simple cache that never invalidates, its better than nothing but id still love to work out how to integrate http-cache.

[06chaynes]

Adding a ureq client as part of PR #116 but would love some feedback

[mrchantey]

fantastic there are a few competing prs currently open for the bevy use-case, once that settles I'll have something to base a http-cache branch on, looking forward to removing my naive impl :)

[mrchantey]

Nice straightforward api, it seems non-utf8 bodies produce the following error, have I missed something?

ERROR bevy_asset::server: Encountered an I/O error while loading asset: unexpected error while loading asset https://raw.githubusercontent.com/bevyengine/bevy/refs/heads/main/assets/branding/bevy_bird_dark.png: Cache error: HTTP error: Failed to read response body: io: stream did not contain valid UTF-8

Here is the draft pr, it can be run with

cargo run --example web_asset --features=https,web_asset_cache

To test different content types try changing the url in examples/asset/web_asset.rs to https://example.com or something, of course the image doesnt load but the caching seems to be working :)

[06chaynes]

Interesting, thanks for testing this! I will look into it and see what's up, I'm likely doing something wrong somewhere.