I'm getting a 404 (or other 40x errors) for my transformed image while other versions of that image work

Follow

Comments

30 comments

  • Avatar
    XO Group Inc.

    has anyone got a resolution to this? we're seeing it too

    0
    Comment actions Permalink
  • Avatar
    Itay Taragano

    Any error status code, including '404 Not Found' is cached for 1 hour for safety reasons. This means that after a full hour is passed, Cloudinary's service will be accessed for trying again to generate and deliver the requested image.

    Trying to access a derived image of a non existing public ID will result with a 404 status code that would be cached for an hour.

    0
    Comment actions Permalink
  • Avatar
    Aaron Craig

    While that is an answer, it's not really a solution, is it?

    That means that if there's a race condition or some other issue on Cloudinary's side that causes a 404 to be returned in error, the calling site will not be able to display the correct image size for an hour?  Also, that means the calling service gets an incorrect 404 error for an hour, which is doubly unfortunate.

    2
    Comment actions Permalink
  • Avatar
    Aaron Craig

    It also seems that the wrong 404 error is not cached if you try to access the image through the Cloudinary control panel.

    So, if an image (I'm using o8ix3qswdks9tdvfvrzp) gives a 404 on my server, going to the Media Library and searching for 'o8ix3qswdks9tdvfvrzp' returns the image.

    I'm using the Ruby library and the Cloudinary::Api.resource(image_id) method.

    It would be uber-awesome to sort this out :)

    0
    Comment actions Permalink
  • Avatar
    Itay Taragano

    Hi Aaron,

    Thank you for your comments. I apologize for the confusion.

    Trying to access the original resource (not a derivative) will return 404 only if it's currently deleted. The 404 is not cached in this case .

    The only issue is with accessing derived resources of non-existent resources. This is also not a race condition. This only happens when a derived resource is accessed before the upload call returns. Once the upload call returns, Cloudinary shouldn't have such errors.

    0
    Comment actions Permalink
  • Avatar
    Aaron Craig

    Thanks for getting back on this.

    Then my specific case is more troublesome.  I am occasionally getting 404 errors on the original resource, not a derivitave.  From my example above, in my Rails application, a call like:

    Cloudinary::Api.resource('o8ix3qswdks9tdvfvrzp')

    returns a 404 from the API, but going to the Media Library UI on your site and searching for 'o8ix3qswdks9tdvfvrzp' returns the image. 

     

    Any thoughts?

     

     

    2
    Comment actions Permalink
  • Avatar
    Itay Taragano

    Hi Aaron,

    Your file indeed exists, and should be returned correctly when calling 'Cloudinary::Api.resource('o8ix3qswdks9tdvfvrzp')'.

    Are you still getting this error now? In our logs we didn't see any 404s for your account. However, we did see several 420s (rate limited). Could this have been the error your got?

    Please note that by default Admin API calls are rate limited to 500 per hour (12,000 daily).

    Looking forward for your response.

    0
    Comment actions Permalink
  • Avatar
    Aaron Craig

    Yes, that sounds likely.  I'll investigate.  Thanks for getting back on this.

    0
    Comment actions Permalink
  • Avatar
    Warranteer Digital ltd.

    Hello,

    I'm trying to fetch and transform an image, and i'm getting a 404 and the header:
    X-Cld-Error: Resource not found - Error in loading <img_url> - HTML response.

    I have no problem opening the image (with the img_url) in my browser.

    What could be the problem here?

    Thanks!

    p.s.
    The image is hosted in google's cloudstorage

    0
    Comment actions Permalink
  • Avatar
    Richard Gieg

    Hi Warranteer Digital Team,

    I apologize for the delay in my response. Are you still noticing this issue? If so, would you be able to share the URL for this particular image so we can try reproduce the problem on our end?

    0
    Comment actions Permalink
  • Avatar
    Siphiwe

    Good day. I've got this same issue. One of the images is https://res.cloudinary.com/afroswag/image/upload/b_white,c_limit,e_saturation:50,r_5/a_0,q_auto/v1/vendors/botlefela/products/Newborn_crochet_shoes_assorted.jpg. I thought the issue was spaces but after replacing them with underscores I still get "x-cld-error:Resource not found - vendors/botlefela/products/Newborn_crochet_shoes_assorted"

    I must add that all the images were working until I recently rebuilt the database where the image public id's are stored. On the front end, the url looks correct (to the naked eye). Please do assist. Thanks a lot

    0
    Comment actions Permalink
  • Avatar
    Roee Ben Ari

    Hi Siphiwe, I see a valid image on https://res.cloudinary.com/afroswag/image/upload/b_white,c_limit,e_saturation:50,r_5/a_0,q_auto/v1/vendors/botlefela/products/Newborn_crochet_shoes_assorted.jpg. To make sure, I've manually invalidated this resource. Let us know if it works well now?

    0
    Comment actions Permalink
  • Avatar
    Siphiwe

    Hi Roee. Thanks for your assistance. Is there a requirement for public id's not to have spaces (or forward slashes) by any chance? I renamed the rest of the problematic images and they became visible. Also, the faulty image id's had additional (non-ascii) characters which I hadn't entered.

    0
    Comment actions Permalink
  • Avatar
    Roee Ben Ari

    While '?\%<>' are utterly forbidden, other special characters will be replaced with underscores '_' and preceding/trailing occurrences will be trimmed. Forward slashes are allowed (you can definitely have folders in your public_id) and while whitespaces are allowed as well, we recommend avoiding it.
    Other than that, the public ID should not begin nor end with whitespace or forward-slash (those will be omitted as well).

    0
    Comment actions Permalink
  • Avatar
    Robert Lightfoot

    I have 4 transformation URLs to the same PDF file.
    Each URL targets/extracts a specific range of pages in that pdf.
    e.g.,
    .../pg_1-4/...
    .../pg_5-9/...
    .../pg_10/...
    .../pg_11-14/...

    The first and third URLs return the targeted pages just fine.
    The second and fourth URLs return an error (404)

    Very confusing and frustrating.

    0
    Comment actions Permalink
  • Avatar
    Tobias Bulla

    I can understand that 404 results have to be cached for resources that arent there (or havent yet been uploaded). However there still is a bug! Cloudinary should of course purge all cached results for a resource when it is changed (or finally uploaded).

     

    So, please purge caches when a resources is uploaded and remove the 404s this way.

    2
    Comment actions Permalink
  • Avatar
    Tamar Meshulam

    This happens to our images as well. We haven't changed anything in the path but suddenly gmail returns 404 when trying to get them. It also seems to correlate with images in folders with spaces, but nothing was changed on our side.

    What is the resolution ? 

    0
    Comment actions Permalink
  • Avatar
    Jon mumm

    This seems crazy to me.  

    "accessing the transformed image before the original was uploaded would result in the error being cached for another period of 24 hours."

    WHYYYYY?!  If there is a legitimate reason for doing this, then at the very least you should provide a way to manually purge the cache.  As a developer in my dev environment, sometimes I WANT to see a 404 on a preview so that I know I've successfully changed the src url.  

    Please fix this or advise on a work-around.  Thanks.

    0
    Comment actions Permalink
  • Avatar
    Stephen Doyle

    Hi Jon,

    If you're testing what happens when you access a nonexistent derived image, (assuming you're not using something like our auto-upload feature to have the image automatically retrieved from a remote server in that case)  the initial cache of the error is much shorter, 60 seconds in the first instance, and the cache expiry is increased to that maximum of 24 hours only after repeated failures.

    You can quickly work around this in testing by requesting a different URL that resolves to the same image once you've uploaded the missing file. For example, you can add a transformation option that doesn't change the output like a_0.

    Regarding the wider question about why errors like this aren't cleared upon upload of the missing file, the main difficulty in clearing the cache automatically is that our invalidations are based on the public_id of the image in your account and on our database knowing which different derived versions exist and the corresponding URLs and cache keys - when the original file doesn't exist in your account we don't have such records to start from, and we don't create 'dummy' database entries for requested-but-not-existing files.

    To address why there's a cache of 4xx errors at all and why they're cached for up to 24 hours, the main reason is that in the vast majority of cases, requests which return a 404 error (or another 4xx error) are never resolved afterward so it's very important that we do cache them to protect our servers from that quantity of failing requests (successful requests are cached for 30-365 days depending on whether the account has a shared or private CDN distribution, for example)

    Such cases should be very rare outside of initial migration, especially if your workflow requires that an image is uploaded before you add it to a product or post, or add it to the webpage, etc.
    Regards,

    Stephen

    0
    Comment actions Permalink
  • Avatar
    Jon mumm

    Thank you for the context Stephen!  This is all very helpful in understanding what goes on behind the scenes and why.  There are two main things that I still don't understand after reading your response

    1. If I attempt to access a specific transformation after its specific url has been cached as a 404, but also after its image asset has been uploaded and is available, will the request cause the cache expiry double again?  In other words, if the image continues to be requested before the 404 cache can expire, will it just return a 404 forever even though the asset is available and other transformations are returning 200?
    2. Is there a way to manually purge the cache for a specific asset/transformation?

    In response to your suggestion that I simply use a different (but same) transformation to request the asset, that is not always an easy task.  We use a CMS that only accepts the image id, and then logic in the frontend code determines what transformation to request based on the client device, screen-size, pixel-ratio, etc.  This means that we have to edit how ALL images are requested just to skirt an issue caused by one asset viewed on one specific device type.  A much easier way to solve this problem would be to just be able to push a button that removes all entries in the 404 cache for an image with a specific id. 

    Now that I know what will happen, I should be able to avoid this issue in the future.  But it was a HUGE annoyance in the first place.  I continuously requested this image, because after I uploaded it I expected it to be there and so I kept trying to request it and request it to no avail.  Once I read this article, it became apparent that all my requests were just extending the length of the cache expiry, and that was sooooo frustrating to learn! Thanks for reading :)

    0
    Comment actions Permalink
  • Avatar
    Stephen Doyle

    Hi John,

    To answer your direct questions:
    1) No, the absolute worst case is if you try repeatedly to access a derived image until the cache is raised to 24 hours, and then upload it immediately after that 24 hour period 'resets' at the CDN that's handling your requests.
    The CDN caches the 404 response for up to 24 hours, but new failing requests to the CDN don't 'bump' the cache period. The important thing is whether the next request that the CDN passes to us [after the cache expires at the CDN] succeeds, so this can be a maximum of 23:59:59 after the image is uploaded, but usually shorter because, for example, you'll upload the file midway through the 24 hours

    2) Not in this specific case because of what I said about us not having a link from the 'missing' file to the URLs that should have returned it if it was present. We can do it manually from this side based on the exact URL you're trying to access, so if you ever have a production issue because of this we can clear it for you if you contact our support team.

    For what you've said about your use-case, where the CMS takes an image ID - one idea might be to use the Media Library Widget to select the image rather than copy/pasting the public_id - this guarantees that the image already exists before you try to use it and also lets your users upload a new file if they can't find what they're looking for in the gallery: https://cloudinary.com/documentation/media_library_widget 

    Regards,

    Stephen

    0
    Comment actions Permalink
  • Avatar
    TOH XUE ZUN

    hi, today i'm experiencing this also

    https://res.cloudinary.com/stashally/image/upload/w_300/q_auto/posts/prq5opqa2hfjhinb36mt.jpg

    is 500 error

    while 

    https://res.cloudinary.com/stashally/image/upload/w_400/q_auto/posts/prq5opqa2hfjhinb36mt.jpg

    is working fine

    note that my uploader only save the resulting identifier after successfully uploads the image, with eager transformation, so this should not be an "accessing derived images before actual image is done uploaded" case

    how to solve this?

    0
    Comment actions Permalink
  • Avatar
    Aleksandar Kostadinov

    Hi Toh Xue Zun,

    The first URL you shared is working for me when I access it now. Could you please try requesting it again?

    Please note that if you've seen a 500 error then that is not related to the error cache as that only covers 40x errors.

    If you come across any other URLs that are not working please share those with us. In addition, you can inspect the X-Cld-Error header that is returned in the response for unsuccessful requests and it shows more information as to why the request failed.

    Best regards,

    Aleksandar

    0
    Comment actions Permalink
  • Avatar
    TOH XUE ZUN

    hi, i have another error with url

    https://res.cloudinary.com/stashally/image/upload/q_auto/devel_posts/v1581411880/k5my5odhaqspirnhred1.png

    with error 400 and x-cld-error 'invalid transformation - devel' while devel_posts is my folder name

    and url

    https://res.cloudinary.com/stashally/image/upload/q_auto/devel_posts/k5my5odhaqspirnhred1.png

    produce 500 error with no x-cld-error

    while 

    https://res.cloudinary.com/stashally/image/upload/devel_posts/v1581411880/k5my5odhaqspirnhred1.png

    is available

    and changing q_auto to other transformation also produce error (this means original already there, but random transformation fail), and original image is only ~23kb

    0
    Comment actions Permalink
  • Avatar
    Aleksandar Kostadinov

    Hi,

    Thanks for sharing these.

    Looking at the first URL, the reason you're getting the error is that the URL is not valid. The folder is part of the public_id and should come after the version number. In your example, you are adding the devel_posts folder before the version therefore it is attempted to be used as a transformation which is invalid. If I set the folder after the version number the URL will work:

    https://res.cloudinary.com/stashally/image/upload/q_auto/v1581411880/devel_posts/k5my5odhaqspirnhred1.png

    The second URL does work for me:

    https://res.cloudinary.com/stashally/image/upload/q_auto/devel_posts/k5my5odhaqspirnhred1.png

    The last URL works with the wrong placement of the folder because there are no transformations.

    I'm not sure how exactly you are generating the URLs but I would recommend you to use one of our SDKs which will correctly build the URLs with you just supplying the parameters.

    0
    Comment actions Permalink
  • Avatar
    differential enterprises

    fyi, another problem we have noticed, after an image upload, sometimes the image can be downloaded in one geographic region (e.g. eastern US) but not in another (e.g. western US).  maybe this is due to trying to check if the image is uploaded (e.g. try to pull it down w/ http) immedately after (or perhaps during) the upload process?  and thereby getting the "cache of non existent image" as noted in earlier posts in this thread?

    0
    Comment actions Permalink
  • Avatar
    Shirly

    That situation can happen when the image exists in a CDN node. can you access this image with transformation? for example add `a_0` after the `upload/` in the URL and let us know if you see the image. If easier feel free to open a ticket in support@cloudinary.com and we can look at your account.

    0
    Comment actions Permalink
  • Avatar
    Daniel Mendoza

    @differential enterprises
    Please also see our response to your original post here:
    https://support.cloudinary.com/hc/en-us/community/posts/360007968360/comments/360001553200

    0
    Comment actions Permalink
  • Avatar
    MB

    Hi,

    after we upload an image to Cloudinary (often just a couple of fractions of a second later) we would rename it by calling `image/rename` API. The main reason for that is that we change the type from `authenticated` to `public` when user is ready to post it (but we upload preemptively).

    However quite frequently we see that Cloudinary API would return `404 Not Found` as response to rename call. Every time we check an image like this after the error, it is in fact present on Cloudinary, and the rename seemed to succeed. It's just a wrong response from Cloudinary. Any idea why this is?

    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    Hey MB, I've noticed that you've also opened this as a direct support ticket and that some answers were already supplied by our agents as part of the communication over the ticket's thread.
    If still in doubt, or have any further questions, please reply back on the support ticket's thread, and one of the agents will provide you with additional information regarding your followup questions.

     

    0
    Comment actions Permalink

Please sign in to leave a comment.